Problems with Open-Source Downloads read https://www.qt.io/blog/problem-with-open-source-downloads and https://forum.qt.io/post/638946

QVector maximum size?



  • Why does qvector segfault when asize is 2^30 but not for 2^29? I can append 2gb of quint16 values with an empty ctor, yet when I try to pre-allocate that amount, segfault.

    int fu  = (1 << 30) - 1; // throws a segfault at ctor
    int bab = (1 << 29) - 1;  //throws a segfault at resize
    QVector<qint16> dataSamples(fu);
    dataSamples.resize(bab * 2);
    

    (Slackware-current, Kde5/Plasma5, kernel 5.4.20, amd64 w/16g ram.)


  • Lifetime Qt Champion

    2^30 * sizeof(qint16) = 2GB - looks like you don't have 2 GB of contiguous (!) memory so the os can't allocate the memory.



  • @Christian-Ehrlicher
    With Disk Cache consuming 4gb I have 4gb Free Physical Memory out of 14.7gb Total. Plenty of room for 2gb contiguous.
    In qvector.h, asize is a signed int, with a test for negative values.


  • Lifetime Qt Champion

    @Psnarf said in QVector maximum size?:

    Plenty of room for 2gb contiguous.

    How will you know? Did you look into the internals of the os if there are 2GB of contiguous memory available?
    Add a try/catch block around and you will see that's a memory (re)allocation error from the OS, nothing from Qt.

    In qvector.h, asize is a signed int, with a test for negative values.

    And what do you want to tell us with this? Yes this is correct.



  • @Christian-Ehrlicher
    Kinfocenter showed Free physical memory bouncing between about 4gb and 2gb while this program was running:

    QVector<qint16> bab;
    QVector<qint16> bar;
    for (int fu = 1073741810; fu < 1080000000; ++fu) {
        qDebug() << fu;
        QVector<qint16> bab(fu);
        bab.swap(bar);
    }
    

    The program terminates with fu=1073741812. (Initial values determined by binary trials.) What is magical about 0x3FFFFFF4, but not 0x3FFFFFF3?


  • Lifetime Qt Champion

    @Psnarf said in QVector maximum size?:

    Kinfocenter showed Free physical memory bouncing between about 4gb and 2gb while this program was running:

    And what do you want to tell me with this? Again: the os needs 2GB of contiguous memory otherwise the allocation will fail. There is nothing OS specific here it will more or less be the same with std::vector.



  • @Christian-Ehrlicher
    This bit of code does not throw an error:

    { quint16 *fu = new quint16[0x3FFFFFFF];
    delete [] fu; }

    New quint16 probably does not allocate contiguous memory. Neither does QVector<T>(int size). KInfocenter does not indicate a discrete jump, but quickly climbs to 0x3FFFFFF4. Indeed, the OS does throw std::bad_alloc in qvector Q_CHECK_PTR(d) when it returns from qarraydata. I'll download the debug symbols and sources for further analysis, since the 5.14.1 docs do not include QArrayData.


  • Lifetime Qt Champion

    @Psnarf said in QVector maximum size?:

    I'll download the debug symbols and sources for further analysis,

    there is nothing more to debug - when the OS throws an out of memory error than there is nothing Qt can do.



  • @Psnarf
    quint16 *fu = new quint16[0x3FFFFFFF];

    New quint16 probably does not allocate contiguous memory. Neither does QVector<T>(int size).

    Really? How do you think the new quint16[0x3FFFFFFF] is laid out then if not contiguous? :) It does allocate contiguously, and so for that matter does QVector<T>(int size).

    What is magical about 0x3FFFFFF4, but not 0x3FFFFFF3?

    Nothing per se. Except that presumably that is right at the boundary/limit of what can be allocated contiguously in your environment? Note that 0x3FFFFFF4 * 2 bytes [quint16] == 0x7FFFFFE8 ==~ 0x80000000 so it is very near the 2GB boundary. I make it 24 bytes short. Maybe there is some bookkeeping allocation involved which pushes this over the limit?


  • Lifetime Qt Champion

    Hi

    QVector

    Docs says
    "Note: QVector and QVarLengthArray both guarantee C-compatible array layout. "

    which means it does allocate as a contiguous block to guarantee this.


Log in to reply