Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error

[saerpa]

@jsulm In my case it's a .s19 file: 3.23MB, 70573 lines.
I try to load it into memory first and parse it line by line.

[saerpa]

@JonB I add the flag and try it again, the issue still exists.
Thanks for the advice! It's really important for me, now I know this flag is important :)

[Christian Ehrlicher]

I doubt this app will crash due to reading 3.5mb

[jsulm]

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

3.23MB, 70573 lines

This is not much.
Run your app through debugger and post the stack trace after the crash.

[saerpa]

@jsulm

15:08:38: Debugging C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe ...
onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(121)\d3d9.dll!686B0C7E: (caller: 686766B4) ReturnHr(1) tid(2620) 8000FFFF �����Թ���
onecore\windows\directx\database\helperlibrary\lib\perappusersettingsqueryimpl.cpp(98)\d3d9.dll!686B0BE5: (caller: 6867658C) ReturnHr(2) tid(2620) 8000FFFF �����Թ���
onecore\windows\directx\database\helperlibrary\lib\directxdatabasehelper.cpp(1410)\d3d9.dll!68676686: (caller: 68674DE6) ReturnHr(3) tid(2620) 8000FFFF �����Թ���
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

[jsulm]

@saerpa This does not look like a crash in your code. Does your app also crash if you do not read that file?

[Christian Ehrlicher]

@jsulm what you see is no stack trace.

[jsulm]

@Christian-Ehrlicher Yeah, you're right

[saerpa]

@Christian-Ehrlicher Your are right! How could you know?

[Christian Ehrlicher]

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

How could you know?

What do you mean? It's no stack trace. Please provide a proper one so we can take a look on it.

[saerpa]

@jsulm Something wrong in this line:

fileBuffer.append(line);

If I comment this line, the app works fine.
And if the load file is less than 40,000 lines, it works fine too.
The content of the file doesn't matter, I tested with other files.

Something wrong with QVector, it's OK in 64-bit build and NG in 32-bit build.

[jsulm]

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

NG in 32-bit build

Reading view MB of data into RAM is also fine on 32bit machine. I'm sure there is no problem with QVector. You still did not provide stack trace.

[saerpa]

@jsulm

It won't trigger line 28 and exit.

[Christian Ehrlicher]

How long is one line? And how much lines are in the file?

[JonB]

@saerpa
Have you allowed it to crash in your screenshots, so we can see stack trace? It seems to me you have stopped it on a breakpoint at line #22. That is not what we are looking for.

[saerpa]

@Christian-Ehrlicher

3.23MB, 70573 lines.
Tail of my file:

S315A0133A3049F5200AD45F54F0D80CB700040EA60F8C
S315A0133A40B71F01FF3C02C2F814F03700E10314219B
S315A0133A503701E1137685DF00F8FFDF01F6FFF602E3
S315A0133A60761382193C10D45F9130004FD9440C16AB
S315A0133A7048346DFF8FCB9130004FD944300602F4F2
S315A0133A806DFF88CBDF091180D8069130004FD9443A
S315A0133A903C0654FFD80B53CF22F0D4FFD4FF10FF0C
S311A0133AA048946DFF77CB02920090FFFFB5
S6040113A93E
S705A002004018

[Christian Ehrlicher]

Print out the line number for each line you read in to see at which line it is crashing. Then look at this specific line in your file. Maybe also print out the length of the read QByteArray.

[saerpa]

@JonB I tried to get there but it's hard to catch the issue spot. In the screenshot I ignore 50,000 triggered in line22, but it will crash before 70,000.

@Christian-Ehrlicher
I modify the code like below.
I print the line number and the content.

    if (file.open(QIODevice::ReadOnly | QIODevice::Text)) {
        quint32 lineNumber{0};

        while (!file.atEnd()) {
            ++lineNumber;
            QByteArray line = file.readLine();

            qDebug() << "Line: " << lineNumber;
            qDebug() << "Data: " << line;
            qDebug() << "Data Length: " << line.length();

            fileBuffer.append(line);
        }
    }

I run it several times:

• It will exit in different lines, same file
• When std::bad_alloc occur it will continue to read several lines
• Exit places are different, it can happen after Data: output or Data Length: output or Line: output.

2 logs are listed below:

LogA:

...

Line:  12059
Data:  "S315A004F1B0CAFF3C03DA1334CF14DF5E4540C440F5DE\n"
Data Length:  47
Line:  12060
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Data:  "S315A004F1C06DFF8CFF14DF5E5540C440F51D007B0027\n"
Data Length:  47
Line:  12061
Data:  "S315A004F1D00090911000F739F08C35DF001A80144F97\n"
Data Length:  47
Line:  12062
Data:  "S315A004F1E0EE1709528409D45F0202142F34FF095280\n"
Data Length:  47

...


Line:  12092
Data:  "S315A004F3C008108920010840A402946D00F535DF02D7\n"
Data Length:  47
Line:  12093
Data:  "S315A004F3D0198014A02680DF001500D4CF921010FF48\n"
17:02:18: C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe exited with code 3

LogB:

...

Line:  7606
Data:  "S315A003DB60EE04D84114FF6E028214BA04911000F792\n"
Data Length:  47
Line:  7607
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Data:  "S315A003DB707868D9FF30D7F4AF49CE0D4A54FB49CFC5\n"
Data Length:  47
Line:  7608
Data:  "S315A003DB80383A14EAF8998B0F200249CF0C4A14FFAE\n"
Data Length:  47
Line:  7609
Data:  "S315A003DB9009C93839EE04DA0134FF3C51DF1A08000B\n"

...

Line:  7672
Data:  "S315A003DF80788A4CCD78894CCE78884CCF788709CFC0\n"
Data Length:  47
Line:  7673
Data:  "S315A003DF900019788609CF0419788509CF08197884DA\n"
Data Length:  47
17:02:56: C:\Users\alex\Documents\build-test-Desktop_Qt_5_15_2_MinGW_32_bit-Debug\debug\test.exe exited with code 3

[jsulm]

@saerpa said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

but it's hard to catch the issue spot

Why? Just remove the break point and run through debugger until it crashes, then post the stack trace...

Does this crash also happen if you don't read the file inside MainWindow constructor, but somewhere else (like when you click on a button)?

[saerpa]

@jsulm said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

Just remove the break point and run through debugger until it crashes, then post the stack trace...

That's what happened in screenshot 2:
Let the app run and crashed, line 28 not triggered and nothing show in the trace window

@jsulm said in Read big file and store the data into QVector<QByteArray> will cause std::bad_alloc error:

Does this crash also happen if you don't read the file inside MainWindow constructor, but somewhere else (like when you click on a button)?

Yes. That's my plan to process the file in a push button click slot function. And it crashed. So I move it to the MainWindow constructor to simplified the question.