Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error

saerpa

hi everyone,

I try to read a file and store the data in the variable.

it's OK when the input file is small but when the file is big (more than 50,000 lines), it will cause std::bad_alloc error in the specific build Qt_5_15_2_MinGW_32_bit.

My test code is really simple:

#include "mainwindow.h"
#include <QByteArray>
#include <QDebug>
#include <QFile>
#include <QFileDialog>
#include <QVector>
#include "ui_mainwindow.h"

MainWindow::MainWindow(QWidget *parent)
    : QMainWindow(parent)
    , ui(new Ui::MainWindow)
{
    ui->setupUi(this);

    QVector<QByteArray> fileBuffer;
    QString fileName = QFileDialog::getOpenFileName(this, "open file", "./", "*.*");
    QFile file;
    file.setFileName(fileName);

    if (file.open(QIODevice::ReadOnly)) {
        while (!file.atEnd()) {
            QByteArray line = file.readLine();

            fileBuffer.append(line);
        }
    }

    qDebug() << "Read lines: " << fileBuffer.length();
}

MainWindow::~MainWindow()
{
    delete ui;
}

Output of build Qt_5_15_2_MinGW_32_bit:

19:17:30: Starting C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

Output of build Qt_5_15_2_MinGW_64_bit:

19:17:49: Starting C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_64_bit-Debug\debug\test.exe...
Read lines:  70572

By the way, it seems MinGW 8.1.0 32-bit in Qt 5.15.2 is the latest 32-bit build and no longer support offically?

jsulm

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

but when the file is big (more than 50,000 lines)

How big is the file (in bytes)?
Don't load whole file into memory if it is way too big...

JonB

@saerpa
Since you are Windows and calling readLine() you should include flag QIODevice::Text when opening the file. Probably not relevant to issue here but if you intend to use this code you ought do so, just in case it makes any difference.

saerpa

@jsulm In my case it's a .s19 file: 3.23MB, 70573 lines.
I try to load it into memory first and parse it line by line.

saerpa

@JonB I add the flag and try it again, the issue still exists.
Thanks for the advice! It's really important for me, now I know this flag is important :)

Christian Ehrlicher

I doubt this app will crash due to reading 3.5mb

jsulm

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

3.23MB, 70573 lines

This is not much.
Run your app through debugger and post the stack trace after the crash.

saerpa

@jsulm

15:08:38: Debugging C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe ...
onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(121)\d3d9.dll!686B0C7E: (caller: 686766B4) ReturnHr(1) tid(2620) 8000FFFF �����Թ���
onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(98)\d3d9.dll!686B0BE5: (caller: 6867658C) ReturnHr(2) tid(2620) 8000FFFF �����Թ���
onecore\windows\directx\database\helperlibrary\lib\directxdatabasehelper.cpp(1410)\d3d9.dll!68676686: (caller: 68674DE6) ReturnHr(3) tid(2620) 8000FFFF �����Թ���
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

jsulm

@saerpa This does not look like a crash in your code. Does your app also crash if you do not read that file?

Christian Ehrlicher

@jsulm what you see is no stack trace.

jsulm

@Christian-Ehrlicher Yeah, you're right

saerpa

@Christian-Ehrlicher Your are right! How could you know?

Christian Ehrlicher

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

How could you know?

What do you mean? It's no stack trace. Please provide a proper one so we can take a look on it.

saerpa

@jsulm Something wrong in this line:

fileBuffer.append(line);

If I comment this line, the app works fine.
And if the load file is less than 40,000 lines, it works fine too.
The content of the file doesn't matter, I tested with other files.

Something wrong with QVector, it's OK in 64-bit build and NG in 32-bit build.

jsulm

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

NG in 32-bit build

Reading view MB of data into RAM is also fine on 32bit machine. I'm sure there is no problem with QVector. You still did not provide stack trace.

saerpa

@jsulm

It won't trigger line 28 and exit.

Christian Ehrlicher

How long is one line? And how much lines are in the file?

JonB

@saerpa
Have you allowed it to crash in your screenshots, so we can see stack trace? It seems to me you have stopped it on a breakpoint at line #22. That is not what we are looking for.

saerpa

@Christian-Ehrlicher

3.23MB, 70573 lines.
Tail of my file:

S315A0133A3049F5200AD45F54F0D80CB700040EA60F8C
S315A0133A40B71F01FF3C02C2F814F03700E10314219B
S315A0133A503701E1137685DF00F8FFDF01F6FFF602E3
S315A0133A60761382193C10D45F9130004FD9440C16AB
S315A0133A7048346DFF8FCB9130004FD944300602F4F2
S315A0133A806DFF88CBDF091180D8069130004FD9443A
S315A0133A903C0654FFD80B53CF22F0D4FFD4FF10FF0C
S311A0133AA048946DFF77CB02920090FFFFB5
S6040113A93E
S705A002004018

Christian Ehrlicher

Print out the line number for each line you read in to see at which line it is crashing. Then look at this specific line in your file. Maybe also print out the length of the read QByteArray.