QtCreator Debug Memory Allocation

webzoid · 8 Feb 2018, 07:57

I have a Windows application which runs for a few hours but then raises an exception when calling QSerialPort::readAll. Ultimately, the exception is coming from the QByteArray::reallocData call inside the readAll function via a qBadAlloc.

When looking at task manager, the application is currently reserving nearly 1700MB of RAM (there is a lot of data being consumed by the program) so my initial guess is that there is a pretty major memory leak going on somewhere.

Using the tools available within QtCreator, is there any way to see which objects are currently reserving the most amount of heap space?

I'm also trying to figure out why, when only the main application loop is executing, does the CPU usage regularly peak above 40%.

koahnig · 2 Aug 2018, 08:58

@webzoid

Are you increasing the size og QByteArray on a regular basis?

My personal guess is that you can watch the memory increasing over time in task manager. Which would point indeed towards a memory leak. Are you releasing the memory properly after your readall calls? Or are you storing it somewhere and basically keeping it the whole time?

Probably you need to post the code snippet of your reading procedure.

webzoid · 2 Aug 2018, 09:13

@koahnig I'm appending to a QByteArray on a very frequent basis. I'll try and explain more...

My application receives data from a number of serial port devices (like GPS, etc). Each device class has a readBuffer field which is a QByteArray and every time the QSerialPort::readyRead signal is emitted, I capture all available data (using QSerialPort::readAll) and append it to my readBuffer field ready for processing. Once data is processed, the processed bytes are removed from the readBuffer.

This all works fine and using qDebug I can see that the size of the readBuffer never goes about 100 bytes (buffer size increases while I wait for complete messages from the serial port).

Having done a bit more investigating over the past hour or so, there must be a memory leak elsewhere which is causing the RAM usage to shoot up and some point later, is causing the append function of the QByteArray to fail.

I'll dig further and post more later...

koahnig · 2 Aug 2018, 09:30

@webzoid

Out of personal experience I would recommend that you are checking and reporting the size of your QByteArray after each increase with qDebug. If those become really big you are up for trouble for sure.

I am not in the details of QByteArray, but each small increase of memory might require a complete copy of the already filled part after realloc of larger amount. Most likely QByteArray will use some smarter allocate process and predict ahead what is required. But those processes are typically predicting by duplication of what is already used. With time you always hit the limits.

On which OS and which compiler are you using?

Looks a bit like you are using a 32 bit compiler because you are getting closer to typical limits with your 1.7 GB memory use.

webzoid · 2 Aug 2018, 10:21

@koahnig I agree with the qDebug sentiment - as mentioned above, every time I append to the readBuffer, I print out the size of the QByteArray but also, I can see that its size never gets too huge.

I guess there could be merit in me allocating a fixed size array of, say, 1024 bytes so that any realloc should not occur?

I am running on Windows 10 and compiling for 32-bit.

koahnig · 2 Aug 2018, 10:25

@webzoid

QByteArray::reserve might help.

However, I see chances that you create memory leaks through repeated allocation, but no release on your side, which are ultimately causing the problem.

What size do you consider as not too huge?

webzoid · wrote on 2 Aug 2018, 10:25

@koahnig I've just added reserve with a length of 1024. I'll let this run for a while and see what happens.

As I mentioned above, the size never goes above 100 bytes (only simple NMEA GPS strings) so 1024 is overkill really.

koahnig · 2 Aug 2018, 10:39

I think you are looking at the wrong end. The returned QByteArray, which is filled with 100 Byte or so each time, shall bear no problem.

@webzoid said in QtCreator Debug Memory Allocation:

@koahnig I'm appending to a QByteArray on a very frequent basis. I'll try and explain more...

My application receives data from a number of serial port devices (like GPS, etc). Each device class has a readBuffer field which is a QByteArray and every time the QSerialPort::readyRead signal is emitted, I capture all available data (using QSerialPort::readAll) and append it to my readBuffer field ready for processing. Once data is processed, the processed bytes are removed from the readBuffer.

You wrote about appending on a frequent basis to a QByteArray. If you are receiving at 1 Hz 100 Byte you are going past the 1024 already 6 times minute. The issue should be with what you call there readBuffer. When you overwrite all the time, it should be fine. If you extend this all the time and it will grow and create the problem.

However, without some sort of comprehensive code snippet showing your intend it is hard to get the right names and understanding what you are trying to do.

webzoid · 2 Aug 2018, 11:01

@koahnig Sorry, I probably haven't explained this very well.

It is the readBuffer (i.e. my class field) which only every gets to about 100 bytes in size. As soon as I've processed the readBuffer and obtained all the information I need, I clean out the bytes which have been processed (using removeTo) and then go again.

The code I use to read from the QSerialPort is here:

void SerialDevice::readBytes() {
	// Read all available bytes from the port
	if (m_port.bytesAvailable()) {
		// Read the bytes into a buffer
		QByteArray buffer = m_port.readAll();
		// Append to the main read buffer
		this->readBuffer().append(buffer);
		// Process the read buffer
		processReadBuffer();
	}
}

I'm confident that there is no real issue with the code above.

jsulm · 2 Aug 2018, 11:03

@webzoid said in QtCreator Debug Memory Allocation:

this->readBuffer().append(buffer);

This looks strange: what does readBuffer() do?

webzoid · J jsulm 2 Aug 2018, 11:01

@jsulm It returns a reference to the QByteArray - its defined in a base class.

QByteArray& readBuffer() { return m_readBuffer; }

SGaist · 2 Aug 2018, 14:10

Hi,

What does processReadBuffer do ?

koahnig · wrote on 2 Aug 2018, 12:15

@webzoid said in QtCreator Debug Memory Allocation:

@koahnig Sorry, I probably haven't explained this very well.

It is the readBuffer (i.e. my class field) which only every gets to about 100 bytes in size. As soon as I've processed the readBuffer and obtained all the information I need, I clean out the bytes which have been processed (using removeTo) and then go again.

QByteArray does not have a routine called removeTo.

Is that routine from you as well?

webzoid · 2 Aug 2018, 14:45

@koahnig Apologies, I mean't the remove function, not removeTo.

@SGaist processReadBuffer converts the QByteArray to a QString and attempts to validate a NMEA string. If a valid string is found, the relevant bytes are removed from readBuffer ready for the next lot of data to arrive.

Either way though, I don't believe that this has anything to do with what I'm trying to find out.

I just want to know whether QtCreator allows inspection of heap sizes reserved by objects, or objects memory footprints. There is a memory leak, no doubt, but it's proving very hard to find: all local mallocs have their own free, all new instances are deleted where necessary, all QLists are cleared when required

koahnig · wrote on 2 Aug 2018, 14:45

@webzoid

For sure it is a nightmare with having many pointers and memory allocations and a memory leak in addition. However, there is typically the problem. Unfortunately, you are the only one to solve this.

As a general I would go for either malloc and free or for new and delete. My personal decision is to limit myself to new and delete and not mixing with malloc and delete. Some of the other fellows might correct me, but I believe they might coexist, but you have to careful for not confusing yourself.

In addition I am using shared pointers, which have the disadvantage of some overhead, but are taking away a lot of memory frustration. Nevertheless, they are the overall cure of memory leaks.

Personally I started with shared_ptr from boost which is now part of the C++ standards. There are also Qt alternatives such as QSharedPointer and its fellows.

jsulm · 3 Aug 2018, 06:31

@webzoid Take a look at http://doc.qt.io/qtcreator/creator-valgrind-overview.html

koahnig · 3 Aug 2018, 08:35

@jsulm said in QtCreator Debug Memory Allocation:

@webzoid Take a look at http://doc.qt.io/qtcreator/creator-valgrind-overview.html

For me as a windows driven developer is valgrind outside of the horizon. A port to windows started a while ago but was too slow to pick off AFAIK.

There is apparently something "new" with http://drmemory.org/ in open source for all platforms. I am wondering if this is any good.

webzoid · wrote on 3 Aug 2018, 08:35

@jsulm Thanks for the link, I've attempted to use Valgrind before but as I'm running on Windows, I got nowhere fast.

@koahnig I've tried DrMemory before but it didn't really help me. I just wish QtCreator (or an interested party) would bring along the Valgrind as its the only thing "missing"