QVector maximum size?

[Psnarf]

Why does qvector segfault when asize is 2^30 but not for 2^29? I can append 2gb of quint16 values with an empty ctor, yet when I try to pre-allocate that amount, segfault.

int fu  = (1 << 30) - 1; // throws a segfault at ctor
int bab = (1 << 29) - 1;  //throws a segfault at resize
QVector<qint16> dataSamples(fu);
dataSamples.resize(bab * 2);

(Slackware-current, Kde5/Plasma5, kernel 5.4.20, amd64 w/16g ram.)

[Christian Ehrlicher]

2^30 * sizeof(qint16) = 2GB - looks like you don't have 2 GB of contiguous (!) memory so the os can't allocate the memory.

[Psnarf]

@Christian-Ehrlicher
With Disk Cache consuming 4gb I have 4gb Free Physical Memory out of 14.7gb Total. Plenty of room for 2gb contiguous.
In qvector.h, asize is a signed int, with a test for negative values.

[Christian Ehrlicher]

@Psnarf said in QVector maximum size?:

Plenty of room for 2gb contiguous.

How will you know? Did you look into the internals of the os if there are 2GB of contiguous memory available?
Add a try/catch block around and you will see that's a memory (re)allocation error from the OS, nothing from Qt.

In qvector.h, asize is a signed int, with a test for negative values.

And what do you want to tell us with this? Yes this is correct.

[Psnarf]

@Christian-Ehrlicher
Kinfocenter showed Free physical memory bouncing between about 4gb and 2gb while this program was running:

QVector<qint16> bab;
QVector<qint16> bar;
for (int fu = 1073741810; fu < 1080000000; ++fu) {
    qDebug() << fu;
    QVector<qint16> bab(fu);
    bab.swap(bar);
}

The program terminates with fu=1073741812. (Initial values determined by binary trials.) What is magical about 0x3FFFFFF4, but not 0x3FFFFFF3?

[Christian Ehrlicher]

@Psnarf said in QVector maximum size?:

Kinfocenter showed Free physical memory bouncing between about 4gb and 2gb while this program was running:

And what do you want to tell me with this? Again: the os needs 2GB of contiguous memory otherwise the allocation will fail. There is nothing OS specific here it will more or less be the same with std::vector.

[Psnarf]

@Christian-Ehrlicher
This bit of code does not throw an error:

{ quint16 *fu = new quint16[0x3FFFFFFF];
delete [] fu; }

New quint16 probably does not allocate contiguous memory. Neither does QVector<T>(int size). KInfocenter does not indicate a discrete jump, but quickly climbs to 0x3FFFFFF4. Indeed, the OS does throw std::bad_alloc in qvector Q_CHECK_PTR(d) when it returns from qarraydata. I'll download the debug symbols and sources for further analysis, since the 5.14.1 docs do not include QArrayData.

[Christian Ehrlicher]

@Psnarf said in QVector maximum size?:

I'll download the debug symbols and sources for further analysis,

there is nothing more to debug - when the OS throws an out of memory error than there is nothing Qt can do.

[JonB]

@Psnarf
quint16 *fu = new quint16[0x3FFFFFFF];

New quint16 probably does not allocate contiguous memory. Neither does QVector<T>(int size).

Really? How do you think the new quint16[0x3FFFFFFF] is laid out then if not contiguous? :) It does allocate contiguously, and so for that matter does QVector<T>(int size).

What is magical about 0x3FFFFFF4, but not 0x3FFFFFF3?

Nothing per se. Except that presumably that is right at the boundary/limit of what can be allocated contiguously in your environment? Note that 0x3FFFFFF4 * 2 bytes [quint16] == 0x7FFFFFE8 ==~ 0x80000000 so it is very near the 2GB boundary. I make it 24 bytes short. Maybe there is some bookkeeping allocation involved which pushes this over the limit?

[mrjj]

Hi

QVector

Docs says
"Note: QVector and QVarLengthArray both guarantee C-compatible array layout. "

which means it does allocate as a contiguous block to guarantee this.