Did QByteArray have memory limit?

Mozzie

Hi everyone,
Today, my application suddenly crashed. It printed a information:

Qt has caught an exception thrown from an event handler. Throwing
exceptions from an event handler is not supported in Qt.
You must not let any exception whatsoever propagate through Qt code.
If that is not possible, in Qt 5 you must at least reimplement
QCoreApplication::notify() and catch all exceptions there.

Qt Concurrent has caught an exception thrown from a worker thread.
This is not supported, exceptions thrown in worker threads must be
caught before control returns to Qt Concurrent.

And showed a window:
捕获.PNG

after a cup of hours, I may found the problem.

I downloaded a file, file size is 2.1 Gb.
And i use QbyteArray to store it, but when the size of QbyteArry is greater than (2 * 1024 * 1024 - 26), it crashed. And my memory is totally 16G and 8G is free.
so I have this question.

my complier is:
环境.PNG

test code:


		QByteArray data;
		data.reserve(1000);
		qDebug() << data.capacity();
//		data.reserve(1024 * 1024 * 1024 * 2 - 25); // crash
		data.reserve(1024 * 1024 * 1024 * 2 - 26); // not crash
		qDebug() << data.capacity();

How can i save large file in memory with Qt?
If you have any explains or suggestions , I will be appreciate;

JonB

@Mozzie
Since you are crashing around 2GB there first thought would be if you were compiling 32-bit. But you seem to be full 64-bit, so that does not seem to be the issue.

The next thing is that the memory allocated here must be contiguous. Even though you have 8GB free, we don't know whether it is laid out such that 2GB contiguous is not free. That could be a problem.

Although you show different code which crashes/not crashes, you don't say but I'm thinking you do not mean it crashes here on this line? You mean, this line succeeds but later the program crashes?

In which case, the question is what you then do with this array? The message implies you have sent it to an event handler?

You should run this under the debugger (or click Retry from that error box and let debugger attach). It may show enough of a backtrace when it crashes to determine what methods it was calling when the error occurs.

Finally, it depends why/what you are intending to do with this large file in memory as a QByteArray, but you might be better off using the memmap() set of C functions for handling large files on disk as though they were in memory.

Mozzie

@JonB said in Did QByteArray have memory limit?:

Since you are crashing around 2GB there first thought would be if you were compiling 32-bit. But you seem to be full 64-bit, so that does not seem to be the issue.
The next thing is that the memory allocated here must be contiguous. Even though you have 8GB free, we don't know whether it is laid out such that 2GB contiguous is not free. That could be a problem.
Although you show different code which crashes/not crashes, you don't say but I'm thinking you do not mean it crashes here on this line? You mean, this line succeeds but later the program crashes?
In which case, the question is what you then do with this array? The message implies you have sent it to an event handler?
You should run this under the debugger (or click Retry from that error box and let debugger attach). It may show enough of a backtrace when it crashes to determine what methods it was calling when the error occurs.
Finally, it depends why/what you are intending to do with this large file in memory as a QByteArray, but you might be better off using the memmap() set of C functions for handling large files on disk as though they were in memory.

Thanks.

I also don't know how to know whether it is laid out such that 2GB contiguous is not free. I think this is the most likely cause.

I did not send this arry to event handler, the message is came from Qt:

int QEventLoop::exec(ProcessEventsFlags flags)
	{
	    Q_D(QEventLoop);
	    if (d->threadData->quitNow)
	        return -1;

	    if (d->inExec) {
	        qWarning("QEventLoop::exec: instance %p has already called exec()", this);
	        return -1;
	    }
	    d->inExec = true;
	    d->exit = false;
	    ++d->threadData->loopLevel;
	    d->threadData->eventLoops.push(this);

	    // remove posted quit events when entering a new event loop
	    QCoreApplication *app = QCoreApplication::instance();
	    if (app && app->thread() == thread())
	        QCoreApplication::removePostedEvents(app, QEvent::Quit);

	#if defined(QT_NO_EXCEPTIONS)
	    while (!d->exit)
	        processEvents(flags | WaitForMoreEvents | EventLoopExec);
	#else
	    try {
	        while (!d->exit)
	            processEvents(flags | WaitForMoreEvents | EventLoopExec);
	    } catch (...) {
	        qWarning("Qt has caught an exception thrown from an event handler. Throwing\n"
	                 "exceptions from an event handler is not supported in Qt. You must\n"
	                 "reimplement QApplication::notify() and catch all exceptions there.\n");

	        // copied from below
	        QEventLoop *eventLoop = d->threadData->eventLoops.pop();
	        Q_ASSERT_X(eventLoop == this, "QEventLoop::exec()", "internal error");
	        Q_UNUSED(eventLoop); // --release warning
	        d->inExec = false;
	        --d->threadData->loopLevel;

	        throw;
	    }
	#endif

	    // copied above
	    QEventLoop *eventLoop = d->threadData->eventLoops.pop();
	    Q_ASSERT_X(eventLoop == this, "QEventLoop::exec()", "internal error");
	    Q_UNUSED(eventLoop); // --release warning
	    d->inExec = false;
	    --d->threadData->loopLevel;

	    return d->returnCode;
	}

I just use memory to cache data from network, I don't want to write to disk frequently.
Does memmap() will write to disk frequently? And memmap() seems need to know file size first?

Pl45m4

@Mozzie said in Did QByteArray have memory limit?:

QbyteArry is greater than (2 * 1024 * 1024 - 26), it crashed.

@Mozzie said in Did QByteArray have memory limit?:

data.reserve(1024 * 1024 * 1024 * 2 - 25); // crash
data.reserve(1024 * 1024 * 1024 * 2 - 26); // not crash

There is a ' * 1024' missing or too much.

Just by doing

QByteArray data;
data.reserve(1024 * 1024 * 1024 * 2 - 25)

I get an overflow compiler warning. because I'm trying to pass more than sizeof(int) as parameter
warning: overflow in expression; result is -2147483622 with type 'int'
(64bit GCC @ Ubuntu18.04, btw)

Is it possible for you to split the ByteArray?

Mozzie

@Pl45m4
Thanks
It is just a warning, and if you use

qDebug() << 1024 * 1024 * 1024 * 2 - 25;

it is less than 2 ^ 32 / 2.

I am thinking about how to split the QbyteArray and not write disk frequently

SGaist

Hi,

As @JonB already suggested, mapping the file might be a better solution.

Christian Ehrlicher

@SGaist said in Did QByteArray have memory limit?:

As @JonB already suggested, mapping the file might be a better solution.

Afaiu the OP is downloading a file from the net - so mmap is not really possible. Simply writing it in chunks should work. QByteArray will not handle more than 2GB in Qt5.

Mozzie

@SGaist
Thanks,
If use mmap, It seems I need to create a file with the same size of download file, this is pretty slow and waste.

Mozzie

@Christian-Ehrlicher
Thanks .

Iis there any difference between write data in one time and write data in chunks on disk operation?
I mean which way have small impact on disk.

And it seems hard to make sure download is complete so that the file will be complete after write into disk .

Christian Ehrlicher

@Mozzie said in Did QByteArray have memory limit?:

I mean which way have small impact on disk.

If you've a good OS both is equal.

Mozzie

@Christian-Ehrlicher said in Did QByteArray have memory limit?:

QByteArray will not handle more than 2GB in Qt5.

Is there any document about "QByteArray will not handle more than 2GB in Qt5"?

J.Hilk

@Mozzie documentation
https://doc.qt.io/qt-5/qbytearray.html#maximum-size-and-out-of-memory-conditions

Mozzie

@J-Hilk
Thank you. That's what I'm looking for.
But, I did not see this in my qt-creator document, I'm using qt5.12.6

Christian Ehrlicher

@Mozzie said in Did QByteArray have memory limit?:

Is there any document about "QByteArray will not handle more than 2GB in Qt5"?

Just take a look at the ctors, the access functions and the resize/size ones and it's clear that it can't contain more than 2GB ...