application hang

ODБOï

hi, im trying to read a big (10MB, 300K lines) text file and display it inside a QML TextArea, the reading method is running in separate thread, but after 15 - 20 seconds the application hangs.
im reading the file into a QString..

...
 Q_PROPERTY (QString fileContent  ...
...
 QFile file(filepath);
        if (file.open(QFile::ReadOnly))
        {
            if(!items.isEmpty())
            items.clear();
          
            float totalLineCount = 0;
            QTextStream code(&file);
            while ((code.atEnd() == false))
            {
                totalLineCount++;
                fileContent +=  code.readLine();
                fileContent+= "\n";
            }

the biggest files i can load in an acceptable time is a file of 2MB / 30 second
What are my options to be able to read bigger files quicker and show in QML?
thank you

JKSH

@LeLev said in application hang:

the reading method is running in separate thread, but after 15 - 20 seconds the application hangs.

How do you run the method in a separate thread?

 Q_PROPERTY (QString fileContent  ...
...
 QFile file(filepath);
        if (file.open(QFile::ReadOnly))
        {
            ...
            float totalLineCount = 0;
            QTextStream code(&file);
            while ((code.atEnd() == false))
            {
                totalLineCount++;
                fileContent +=  code.readLine();
                fileContent+= "\n";
            }

See if this is any faster:

QFile file(filepath);
if (file.open(QFile::ReadOnly|QFile::Text))
{
    QByteArray contents = file.readAll();
    int totalLineCount = contents.count('\n') + 1;
    fileContent = QString::fromUtf8(contents); // or fromLocal8Bit(), depending on your file encoding
}

P.S. Why is totalLineCount a float? You can't any fractions!

ODБOï

hi, thx for the answer
@JKSH said in application hang:

How do you run the method in a separate thread?

with QtConcurrentRun

@JKSH said in application hang:

See if this is any faster:

thak you!
i will have to adapt it because the code i pasted is simplified, In the real code, the QTextStream is used later like here, may i ask how would you iterate over fileContent or contents line by line in the second loop ?

void preProcessFile(QString filepath)
{
    QFile file(filepath);
    if (file.open(QFile::ReadOnly))
    {
        qDebug()<< "file found";
       if(!items.isEmpty())
        items.clear();

        setSource(QString(""));
        setLoading(true);

        QString fileContent("");

        float totalLineCount = 0;
        QTextStream code(&file);
        while ((code.atEnd() == false))
        {
            totalLineCount++;
            fileContent +=  code.readLine();
            fileContent+= "\n";
        }
        if (totalLineCount == 0)
            totalLineCount = 1;

        code.seek(0);

        double x = 0;
        double y = 0;
        double i = 0;
        double j = 0;
        bool arc = false;
        bool cw = false;
        bool mm = true;
        int index = 0;
        int g = 0;

        bool zeroInsert = false;
        do
        {
            QString strline = code.readLine();

            index++;

            trimToEnd(strline, '(');
            trimToEnd(strline, ';');
            trimToEnd(strline, '%');
            strline = strline.trimmed();

            if (strline.size() == 0)
            {....}//ignore comments
            else
            {
            ....
            }
        } while (code.atEnd() == false);

    }
    else
        return
}

@JKSH said in application hang:

Why is totalLineCount a float? You can't any fractions!

in the lib i use its is type of float, i found it strange also

JKSH

@LeLev said in application hang:

with QtConcurrentRun

Please show your code.

Did you apply QtConcurrent::run() to a QObject's class method? If so, have you made your method is thread-safe?

@LeLev said in application hang:

In the real code, the QTextStream is used later like here, may i ask how would you iterate over fileContent or contents line by line in the second loop ?

QTextStream can operate on a QByteArray* or QString*. I'm guessing that both of these options are faster than operating on a QFile* (but please benchmark to make sure)

in the lib i use its is type of float, i found it strange also

I see. There's not much we can do, then :)

ODБOï

@JKSH said in application hang:

Please show your code.

showPreviews is public slot of my class (classe base is QQuickPaintedItem)
it is running the preProcessFile() method with QtConcurrentRun
but preProcessFile() is accessing the member variable (items) of the class (is that a problem ?)

 void showPreviews(QString path){ // this is a public slot
        qDebug()<<"loading the file for the preview" << path;
        QFile file(path);
        if(!file.exists()){
            qDebug()<<"fichier introuvable..";
        }

        QtConcurrent::run([=]{
                  preProcessFile(path);
              });
// before i knew  QtConcurrent::run i dit with QThread::create
//        QThread *thread = QThread::create([=]{ preProcessFile(path);});
//        QObject::connect(thread,&QThread::finished,thread,&QThread::deleteLater);
//        QObject::connect(thread,&QThread::destroyed,[](){qDebug()<<"thread Deleted";});
//        thread->start();

    }

  void preProcessFile(QString filepath)
    {
        QFile file(filepath);
        if (file.open(QFile::ReadOnly))
        {
            qDebug()<< "file found";
           if(!items.isEmpty())
            items.clear();

            setSource(QString(""));
            setLoading(true);

            QString fileContent("");

            float totalLineCount = 0;
            QTextStream code(&file);
            while ((code.atEnd() == false))
            {
                totalLineCount++;
                fileContent +=  code.readLine();
                fileContent+= "\n";
            }
            if (totalLineCount == 0)
                totalLineCount = 1;

            code.seek(0);

            double x = 0;
            double y = 0;
            double i = 0;
            double j = 0;
            bool arc = false;
            bool cw = false;
            bool mm = true;
            int index = 0;
            int g = 0;

            bool zeroInsert = false;
            do
            {
                QString strline = code.readLine();

                index++;

                trimToEnd(strline, '(');
                trimToEnd(strline, ';');
                trimToEnd(strline, '%');

                strline = strline.trimmed();

                if (strline.size() == 0)
                {}//ignore comments
                else
                {
                    strline = strline.toUpper();
                    strline.replace("M6", "M06");
                    strline.replace(QRegExp("([A-Z])"), " \\1");
                    strline.replace(QRegExp("\\s+"), " ");
                    //if (strline.contains("G", Qt::CaseInsensitive))
                    {
                        if (processGCode(strline, x, y, i, j, arc, cw, mm, g))
                        {
                            if (!zeroInsert)
                            {
                                // insert 0,0 position
                                items.append(PosItem(0, 0, 0, 0, false, false, mm, 0)); // member access
                                zeroInsert = true;
                            }
                            items.append(PosItem(x, y, i, j, arc, cw, mm, index));
                        }
                    }
                }
            } while (code.atEnd() == false);
            setItems(items);
            file.close();
            setSource(fileContent);
            setLoading(false);
           
        }
        else
            printf("Can't open file\n");
    }

@JKSH said in application hang:

have you made your method is thread-safe?

no.. all i did is to make the member functions calls pass through Signal/Slots
instead of calling update() i created a signal doUpdate()

and i use Signal/Slots to handle it thread-safely
i will do the same for other method calls (eg processGCode(),trimToEnd() )
and also for the member access ( eg. items.append(PosItem(x, y, i, j, arc, cw, mm, index)); )

kshegunov

@LeLev said in application hang:

but preProcessFile() is accessing the member variable (items) of the class (is that a problem ?)

Yes.

PS. However it seems your statement is not correct - you're not touching the object from the preProcessFile function. Or if you are doing it, I'm not seeing it.

JKSH

@LeLev said in application hang:

showPreviews is public slot of my class (classe base is QQuickPaintedItem)
it is running the preProcessFile() method with QtConcurrentRun
but preProcessFile() is accessing the member variable (items) of the class (is that a problem ?)

Yes, it is a problem. You have 2 different threads that can read/write a variable but the access is not protected. This is not thread-safe and it can cause data corruption. See https://doc.qt.io/qt-5/threads-reentrancy.html

As @kshegunov pointed out, this is a race condition.

ODБOï

@kshegunov said in application hang:

However it seems your statement is not correct - you're not touching the object from the preProcessFile function

i do
items.append(PosItem(x, y, i, j, arc, cw, mm, index)); items is member, and :
processGCode(strline, x, y, i, j, arc, cw, mm, g) it is member method also

@JKSH said in application hang:

Yes, it is a problem. You have 2 different threads that can read/write a variable but the access is not protected. This is not thread-safe and it can cause data corruption. See https://doc.qt.io/qt-5/threads-reentrancy.html
As @kshegunov pointed out, this is a race condition.

Ok, thank you guys for the valuable advices, I will restart the whole thing from scratch and do it properly

ODБOï

@JKSH after reading Reentrancy and Thread-Safety i have a doubt,
here switchLoadingAndUpdate() is not thread safe, but if it is used only by showPreviews() it should not cause any problems no ?

class GCodeView :  public QQuickPaintedItem
{
    Q_OBJECT
public:
    GCodeView(QQuickItem *parent = 0);

public slots:
    void switchLoadingAndUpdate(){
       m_loading = !m_loading; // should i do this also through signal slot ?
      // emit newLoadingSet(!m_loading)
       emit needUpdate()
    }

void showPreviews(){  
    QtConcurrent::run([=]{
              switchLoading();
          });
    return;
}

void  doUpdate(){
  update();
}

  signals :
 void needUpdate();

protected:
    void paint( QPainter *painterp);
private:
    bool m_loading=false;
};
#endif // GCODEVIEW_H

//ctor
GCodeView::GCodeView(QQuickItem *parent): QQuickPaintedItem(parent)
{ 
    QObject::connect(this,&GCodeView::needUpdate,this,&GCodeView::doUpdate);
}

jsulm

@LeLev said in application hang:

switchLoadingAndUpdate()

it changes m_loading, if m_loading is used somewhere else you still could get a race condition.

ODБOï

@jsulm said in application hang:

used somewhere

in c++ ? it is only used in qml but never changed by anything. In this case it should not be a problem. I think i got it

GCodeView {
id:gView
}
BusyIndicator {
visible :  gView.loading
}