Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error
-
hi everyone,
I try to read a file and store the data in the variable.
it's OK when the input file is small but when the file is big (more than 50,000 lines), it will cause
std::bad_alloc
error in the specific buildQt_5_15_2_MinGW_32_bit
.My test code is really simple:
#include "mainwindow.h" #include <QByteArray> #include <QDebug> #include <QFile> #include <QFileDialog> #include <QVector> #include "ui_mainwindow.h" MainWindow::MainWindow(QWidget *parent) : QMainWindow(parent) , ui(new Ui::MainWindow) { ui->setupUi(this); QVector<QByteArray> fileBuffer; QString fileName = QFileDialog::getOpenFileName(this, "open file", "./", "*.*"); QFile file; file.setFileName(fileName); if (file.open(QIODevice::ReadOnly)) { while (!file.atEnd()) { QByteArray line = file.readLine(); fileBuffer.append(line); } } qDebug() << "Read lines: " << fileBuffer.length(); } MainWindow::~MainWindow() { delete ui; }
Output of build
Qt_5_15_2_MinGW_32_bit
:19:17:30: Starting C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc
Output of build
Qt_5_15_2_MinGW_64_bit
:19:17:49: Starting C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_64_bit-Debug\debug\test.exe... Read lines: 70572
By the way, it seems
MinGW 8.1.0 32-bit
inQt 5.15.2
is the latest 32-bit build and no longer support offically? -
@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:
but when the file is big (more than 50,000 lines)
How big is the file (in bytes)?
Don't load whole file into memory if it is way too big... -
-
I doubt this app will crash due to reading 3.5mb
-
@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:
3.23MB, 70573 lines
This is not much.
Run your app through debugger and post the stack trace after the crash. -
15:08:38: Debugging C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe ... onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(121)\d3d9.dll!686B0C7E: (caller: 686766B4) ReturnHr(1) tid(2620) 8000FFFF �����Թ��� onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(98)\d3d9.dll!686B0BE5: (caller: 6867658C) ReturnHr(2) tid(2620) 8000FFFF �����Թ��� onecore\windows\directx\database\helperlibrary\lib\directxdatabasehelper.cpp(1410)\d3d9.dll!68676686: (caller: 68674DE6) ReturnHr(3) tid(2620) 8000FFFF �����Թ��� terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc
-
@jsulm what you see is no stack trace.
-
@Christian-Ehrlicher Yeah, you're right
-
@Christian-Ehrlicher Your are right! How could you know?
-
@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:
How could you know?
What do you mean? It's no stack trace. Please provide a proper one so we can take a look on it.
-
@jsulm Something wrong in this line:
fileBuffer.append(line);
If I comment this line, the app works fine.
And if the load file is less than 40,000 lines, it works fine too.
The content of the file doesn't matter, I tested with other files.Something wrong with
QVector
, it's OK in 64-bit build and NG in 32-bit build. -
@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:
NG in 32-bit build
Reading view MB of data into RAM is also fine on 32bit machine. I'm sure there is no problem with QVector. You still did not provide stack trace.
-
How long is one line? And how much lines are in the file?
-
3.23MB, 70573 lines.
Tail of my file:S315A0133A3049F5200AD45F54F0D80CB700040EA60F8C S315A0133A40B71F01FF3C02C2F814F03700E10314219B S315A0133A503701E1137685DF00F8FFDF01F6FFF602E3 S315A0133A60761382193C10D45F9130004FD9440C16AB S315A0133A7048346DFF8FCB9130004FD944300602F4F2 S315A0133A806DFF88CBDF091180D8069130004FD9443A S315A0133A903C0654FFD80B53CF22F0D4FFD4FF10FF0C S311A0133AA048946DFF77CB02920090FFFFB5 S6040113A93E S705A002004018
-
Print out the line number for each line you read in to see at which line it is crashing. Then look at this specific line in your file. Maybe also print out the length of the read QByteArray.