Reading large ASCII file with QFile
-
I tried your code but unfortunately the problem remains, and I it looks like the file reading is slower than using directly QFile methods.
The strange thing is that the file loads correctly until the end (I connected a progressbar to the signal of my thread to visualize the loading progression), but when the loading is done, the thread send the error signal...The file is ASCII with line for sure.
@HB76
HiSo the file reader is running in thread?
Try to disable the emit loadingValueChanged for a test and see if it still crashes.
You might be spamming the event loop which can crash.
else i see no reason to crash even if file was 1 TB as we just read it line by line. (and not store each line)
-
@HB76
HiSo the file reader is running in thread?
Try to disable the emit loadingValueChanged for a test and see if it still crashes.
You might be spamming the event loop which can crash.
else i see no reason to crash even if file was 1 TB as we just read it line by line. (and not store each line)
-
Even with th signal disabled the application continues to crash...
Is it possible that I run out of RAM ? Because each line can contain a lot of data (it is vectorial data file)@HB76 said in Reading large ASCII file with QFile:
Is it possible that I run out of RAM ?
How large is one line ?
Also how do you split it to values ?Yes, must be something like that then.
is it a 32 bit or 64 bit app ?
also if you run with debugger on, what line makes it crash ? -
The longest line can have more than 30'000 characters.
Each line contain a keyword at the beginning which is separated from data thanks to the "/" char.
After that, data are separated with coma.
So I use the split method to get a QStringList of data that I convert in values by using ToInt() method of QString class.My compiler version in 32 bits. Do you think it can have an impact of the reading ?
-
Hi,
Since you have a 32bit application you can't use more than 4GB of RAM. Is that the case with your application ?
-
Hi everyone !
I'm trying to read data from a very large ASCII file (over 2 Gb) with QFile by reading the file line after line and processing each line before going to the next-one. To avoid application freeze during loading I used a Thread. My code works fine with smaller file (<500 Mb) but when I try to read large files, the program crash. Here is my code :
QFile file(file_name); if (!file.open(QIODevice::ReadOnly | QIODevice::Text)) return; try { int size = int(file.size()); bool end = false; while(!end) { QString line = file.readLine(); //... // Processing line //... read += line.toUtf8().size(); emit loadingValueChanged(int(100*double(read)/double(size))); } } catch (...) { emit error(tr("Error - data loading failed")); }Should I use a memory mapped file or something like ?
My project is configured with Qt 5.9.8 MinGW 32 bits.
Thanks !
@HB76 said in Reading large ASCII file with QFile:
My code works fine with smaller file (<500 Mb) but when I try to read large files, the program crash.
Please provide a backtrace of the crash.
-
@HB76 said in Reading large ASCII file with QFile:
My code works fine with smaller file (<500 Mb) but when I try to read large files, the program crash.
Please provide a backtrace of the crash.
The exception returned is std::bad_alloc.
I should try with 64 bits version, but I need to add the 64 bits MinGW compiler since I don't have it configured in Qt. -
The exception returned is std::bad_alloc.
I should try with 64 bits version, but I need to add the 64 bits MinGW compiler since I don't have it configured in Qt.@HB76
Just install the Qt 64 bit MinGW Qt binary from the maintenance tool and you are good to go.Also, have a look at
https://doc.qt.io/qt-5/qstringref.html
which might provide a huge boost since it will not copy the strings and all you want is to call toInt on it. -
The exception returned is std::bad_alloc.
I should try with 64 bits version, but I need to add the 64 bits MinGW compiler since I don't have it configured in Qt.@HB76 said in Reading large ASCII file with QFile:
The exception returned is std::bad_alloc.
This doesn't tell me where the memory was not enough. But you have your answer anyway, you don't have enough memory.
