Reading large ASCII file with QFile

HB76

Hi everyone !

I'm trying to read data from a very large ASCII file (over 2 Gb) with QFile by reading the file line after line and processing each line before going to the next-one. To avoid application freeze during loading I used a Thread. My code works fine with smaller file (<500 Mb) but when I try to read large files, the program crash. Here is my code :

QFile file(file_name);
    if (!file.open(QIODevice::ReadOnly | QIODevice::Text))
        return;

    try
    {
        int size = int(file.size());
        bool end = false;

        while(!end)
        {
            QString line = file.readLine();

            //...
            // Processing line
            //...

            read += line.toUtf8().size();
            emit loadingValueChanged(int(100*double(read)/double(size)));
        }
    } catch (...) {
            emit error(tr("Error - data loading failed"));
    }

Should I use a memory mapped file or something like ?

My project is configured with Qt 5.9.8 MinGW 32 bits.

Thanks !

Christian Ehrlicher

@HB76 said in Reading large ASCII file with QFile:

int size = int(file.size());

This will not work when your file is larger than 2GB ...

mrjj

Hi
Its something else it crashes from than the file being big since you read it line by line.
Your code
while(!end) {

looks a bit odd.

Normally its done like.

QFile inputFile(fileName);
if (inputFile.open(QIODevice::ReadOnly))
{
   QTextStream in(&inputFile);
   while (!in.atEnd())
   {
      QString line = in.readLine();
      ...
   }
   inputFile.close();
}

I read a 100 GB file like this so we have to find out why you crash.

First. You have 100% sure its line in the files ?
else it will read it all in one go :)

HB76

@mrjj

I tried your code but unfortunately the problem remains, and I it looks like the file reading is slower than using directly QFile methods.
The strange thing is that the file loads correctly until the end (I connected a progressbar to the signal of my thread to visualize the loading progression), but when the loading is done, the thread send the error signal...

The file is ASCII with line for sure.

mrjj

@HB76
Hi

So the file reader is running in thread?

Try to disable the emit loadingValueChanged for a test and see if it still crashes.

You might be spamming the event loop which can crash.

else i see no reason to crash even if file was 1 TB as we just read it line by line. (and not store each line)

HB76

@mrjj

Even with th signal disabled the application continues to crash...
Is it possible that I run out of RAM ? Because each line can contain a lot of data (it is vectorial data file)

mrjj

@HB76 said in Reading large ASCII file with QFile:

Is it possible that I run out of RAM ?

How large is one line ?
Also how do you split it to values ?

Yes, must be something like that then.

is it a 32 bit or 64 bit app ?
also if you run with debugger on, what line makes it crash ?

HB76

The longest line can have more than 30'000 characters.
Each line contain a keyword at the beginning which is separated from data thanks to the "/" char.
After that, data are separated with coma.
So I use the split method to get a QStringList of data that I convert in values by using ToInt() method of QString class.

My compiler version in 32 bits. Do you think it can have an impact of the reading ?

SGaist

Hi,

Since you have a 32bit application you can't use more than 4GB of RAM. Is that the case with your application ?

kshegunov

@HB76 said in Reading large ASCII file with QFile:

My code works fine with smaller file (<500 Mb) but when I try to read large files, the program crash.

Please provide a backtrace of the crash.

HB76

@kshegunov

The exception returned is std::bad_alloc.
I should try with 64 bits version, but I need to add the 64 bits MinGW compiler since I don't have it configured in Qt.

mrjj

@HB76
Just install the Qt 64 bit MinGW Qt binary from the maintenance tool and you are good to go.

Also, have a look at
https://doc.qt.io/qt-5/qstringref.html
which might provide a huge boost since it will not copy the strings and all you want is to call toInt on it.

kshegunov

@HB76 said in Reading large ASCII file with QFile:

The exception returned is std::bad_alloc.

This doesn't tell me where the memory was not enough. But you have your answer anyway, you don't have enough memory.

HB76

Ok guys, I configured the project with an older MinGW version in 64 bits and it works ! It was just a simple problem... I will try to install the most recent MinGW version and it will be perfect.

Thanks all for your help !