Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

Did QByteArray have memory limit?



  • Hi everyone,
    Today, my application suddenly crashed. It printed a information:

    Qt has caught an exception thrown from an event handler. Throwing
    exceptions from an event handler is not supported in Qt.
    You must not let any exception whatsoever propagate through Qt code.
    If that is not possible, in Qt 5 you must at least reimplement
    QCoreApplication::notify() and catch all exceptions there.
    
    Qt Concurrent has caught an exception thrown from a worker thread.
    This is not supported, exceptions thrown in worker threads must be
    caught before control returns to Qt Concurrent.
    

    And showed a window:
    捕获.PNG

    after a cup of hours, I may found the problem.

    I downloaded a file, file size is 2.1 Gb.
    And i use QbyteArray to store it, but when the size of QbyteArry is greater than (2 * 1024 * 1024 - 26), it crashed. And my memory is totally 16G and 8G is free.
    so I have this question.

    my complier is:
    环境.PNG

    test code:

    
    		QByteArray data;
    		data.reserve(1000);
    		qDebug() << data.capacity();
    //		data.reserve(1024 * 1024 * 1024 * 2 - 25); // crash
    		data.reserve(1024 * 1024 * 1024 * 2 - 26); // not crash
    		qDebug() << data.capacity();
    

    How can i save large file in memory with Qt?
    If you have any explains or suggestions , I will be appreciate;



  • @Mozzie
    Since you are crashing around 2GB there first thought would be if you were compiling 32-bit. But you seem to be full 64-bit, so that does not seem to be the issue.

    The next thing is that the memory allocated here must be contiguous. Even though you have 8GB free, we don't know whether it is laid out such that 2GB contiguous is not free. That could be a problem.

    Although you show different code which crashes/not crashes, you don't say but I'm thinking you do not mean it crashes here on this line? You mean, this line succeeds but later the program crashes?

    In which case, the question is what you then do with this array? The message implies you have sent it to an event handler?

    You should run this under the debugger (or click Retry from that error box and let debugger attach). It may show enough of a backtrace when it crashes to determine what methods it was calling when the error occurs.

    Finally, it depends why/what you are intending to do with this large file in memory as a QByteArray, but you might be better off using the memmap() set of C functions for handling large files on disk as though they were in memory.



  • @JonB said in Did QByteArray have memory limit?:

    Since you are crashing around 2GB there first thought would be if you were compiling 32-bit. But you seem to be full 64-bit, so that does not seem to be the issue.
    The next thing is that the memory allocated here must be contiguous. Even though you have 8GB free, we don't know whether it is laid out such that 2GB contiguous is not free. That could be a problem.
    Although you show different code which crashes/not crashes, you don't say but I'm thinking you do not mean it crashes here on this line? You mean, this line succeeds but later the program crashes?
    In which case, the question is what you then do with this array? The message implies you have sent it to an event handler?
    You should run this under the debugger (or click Retry from that error box and let debugger attach). It may show enough of a backtrace when it crashes to determine what methods it was calling when the error occurs.
    Finally, it depends why/what you are intending to do with this large file in memory as a QByteArray, but you might be better off using the memmap() set of C functions for handling large files on disk as though they were in memory.

    Thanks.

    I also don't know how to know whether it is laid out such that 2GB contiguous is not free. I think this is the most likely cause.

    I did not send this arry to event handler, the message is came from Qt:

    int QEventLoop::exec(ProcessEventsFlags flags)
    	{
    	    Q_D(QEventLoop);
    	    if (d->threadData->quitNow)
    	        return -1;
    
    	    if (d->inExec) {
    	        qWarning("QEventLoop::exec: instance %p has already called exec()", this);
    	        return -1;
    	    }
    	    d->inExec = true;
    	    d->exit = false;
    	    ++d->threadData->loopLevel;
    	    d->threadData->eventLoops.push(this);
    
    	    // remove posted quit events when entering a new event loop
    	    QCoreApplication *app = QCoreApplication::instance();
    	    if (app && app->thread() == thread())
    	        QCoreApplication::removePostedEvents(app, QEvent::Quit);
    
    	#if defined(QT_NO_EXCEPTIONS)
    	    while (!d->exit)
    	        processEvents(flags | WaitForMoreEvents | EventLoopExec);
    	#else
    	    try {
    	        while (!d->exit)
    	            processEvents(flags | WaitForMoreEvents | EventLoopExec);
    	    } catch (...) {
    	        qWarning("Qt has caught an exception thrown from an event handler. Throwing\n"
    	                 "exceptions from an event handler is not supported in Qt. You must\n"
    	                 "reimplement QApplication::notify() and catch all exceptions there.\n");
    
    	        // copied from below
    	        QEventLoop *eventLoop = d->threadData->eventLoops.pop();
    	        Q_ASSERT_X(eventLoop == this, "QEventLoop::exec()", "internal error");
    	        Q_UNUSED(eventLoop); // --release warning
    	        d->inExec = false;
    	        --d->threadData->loopLevel;
    
    	        throw;
    	    }
    	#endif
    
    	    // copied above
    	    QEventLoop *eventLoop = d->threadData->eventLoops.pop();
    	    Q_ASSERT_X(eventLoop == this, "QEventLoop::exec()", "internal error");
    	    Q_UNUSED(eventLoop); // --release warning
    	    d->inExec = false;
    	    --d->threadData->loopLevel;
    
    	    return d->returnCode;
    	}
    

    I just use memory to cache data from network, I don't want to write to disk frequently.
    Does memmap() will write to disk frequently? And memmap() seems need to know file size first?



  • @Mozzie said in Did QByteArray have memory limit?:

    QbyteArry is greater than (2 * 1024 * 1024 - 26), it crashed.

    @Mozzie said in Did QByteArray have memory limit?:

    data.reserve(1024 * 1024 * 1024 * 2 - 25); // crash
    data.reserve(1024 * 1024 * 1024 * 2 - 26); // not crash

    There is a ' * 1024' missing or too much.

    Just by doing

    QByteArray data;
    data.reserve(1024 * 1024 * 1024 * 2 - 25)
    

    I get an overflow compiler warning. because I'm trying to pass more than sizeof(int) as parameter
    warning: overflow in expression; result is -2147483622 with type 'int'
    (64bit GCC @ Ubuntu18.04, btw)

    Is it possible for you to split the ByteArray?



  • @Pl45m4
    Thanks
    It is just a warning, and if you use

    qDebug() << 1024 * 1024 * 1024 * 2 - 25;
    

    it is less than 2 ^ 32 / 2.

    I am thinking about how to split the QbyteArray and not write disk frequently


  • Lifetime Qt Champion

    Hi,

    As @JonB already suggested, mapping the file might be a better solution.


  • Lifetime Qt Champion

    @SGaist said in Did QByteArray have memory limit?:

    As @JonB already suggested, mapping the file might be a better solution.

    Afaiu the OP is downloading a file from the net - so mmap is not really possible. Simply writing it in chunks should work. QByteArray will not handle more than 2GB in Qt5.



  • @SGaist
    Thanks,
    If use mmap, It seems I need to create a file with the same size of download file, this is pretty slow and waste.



  • @Christian-Ehrlicher
    Thanks .

    Iis there any difference between write data in one time and write data in chunks on disk operation?
    I mean which way have small impact on disk.

    And it seems hard to make sure download is complete so that the file will be complete after write into disk .


  • Lifetime Qt Champion

    @Mozzie said in Did QByteArray have memory limit?:

    I mean which way have small impact on disk.

    If you've a good OS both is equal.



  • @Christian-Ehrlicher said in Did QByteArray have memory limit?:

    QByteArray will not handle more than 2GB in Qt5.

    Is there any document about "QByteArray will not handle more than 2GB in Qt5"?


  • Moderators



  • @J-Hilk
    Thank you. That's what I'm looking for.
    But, I did not see this in my qt-creator document, I'm using qt5.12.6


  • Lifetime Qt Champion

    @Mozzie said in Did QByteArray have memory limit?:

    Is there any document about "QByteArray will not handle more than 2GB in Qt5"?

    Just take a look at the ctors, the access functions and the resize/size ones and it's clear that it can't contain more than 2GB ...


Log in to reply