Memory leaks with QQueue<QByteArray> from different threads

[goocreations]

I've come across a strange memory leak and I'm not sure if I'm missing some simple C++ principles.

I have my own thread-safe class containing a QQueue<QByteArray>

@class SharedQueue
{
public:

void enqueue(QByteArray array)
{
mMutex.lock();
mData.enqueue(array);
mMutex.unlock();
}

void clear()
{
mMutex.lock();
mData.clear();
mMutex.unlock();
}

private:

QMutex mMutex;
QQueue<QByteArray> mData;
}@

I have a first thread reading from a file and adding the data read to the SharedQueue

@QByteArray array = file.read(CHUNK_SIZE);
mSharedQueue->enqueue(array);@

For the purpose of this post, the second thread simply calls clear() on a pointer pointing to the same SharedQueue as the file thread.
So basically Thread1 reads the file and adds it to the queue and Thread2 removes everything from the queue the moment new data is added to it.

For some reasons there are major memory leaks (up to the size of the input file) and I have no idea what I'm doing wrong.

Interestingly if we change the enqueue function of SharedQueue to the following:

@void enqueue(QByteArray array)
{
mMutex.lock();
mData.enqueue(array);
mMutex.unlock();
clear();
}
@

the memory leaks disappear. The only thing that actually changes is that clear() is called from the same thread that calls enqueue.

Any idea what I'm doing wrong?

[rcari]

QQueue inherits from QList. The QList internal buffer only GROWS. Which means that it will NEVER shrink even if you remove all elements from it. This is a design choice. If you delete the SharedQueue instance, the memory will be properly released, no "real" leak.

[rcari]

You can take a look at that: http://qt-project.org/doc/qt-4.8/containers.html#growth-strategies
The QList does not have the squeeze() call, thus your impression of a leak.

[goocreations]

Nope, the problem still exists. I've tried using a QVector instead of the QQueue and after I cleared the vector I called squeeze(), but there is still some memory that is not "freed".

[mlong]

What method are you using to gauge your memory usage and memory leakage?

[goocreations]

I currently don't use any leakage tools (eg: Valgrind). I'm just using my system monitor (working under Ubuntu).

I had a related question about 2 years ago, and I was told that there are no leaks, but the memory is not immediately returned to the system (depends on OS). But I've now tried to run my example infinitely, and my memory usage just increases. So this cannot be the case.

[mlong]

I would highly recommend using Valgrind; it can give you specifics of where leaks are occurring and it removes any ambiguity that can be introduced by using just the system monitoring tools.

[goocreations]

Ok, I'll try figuring it out with Valgrind. But even if there are no "official" leaks, my program uses more and more memory (ran it for a couple of minutes, system monitor currently says 3.4GB used - for something that should not use more than 5mb).

I'll try Valgrind, but any suggestions are still welcome.

[goocreations]

I've found something interesting. I've not provided the complete code for what I'm actually doing. My enqueue function actually emits a signal:

@void enqueue(QByteArray array)
{
mMutex.lock();
mData.enqueue(array);
mMutex.unlock();
emit dataAvailable();
}
@

This signal is connected to the second thread. Hence when the signal is emitted, the second thread starts and calls clear on the ShareQueue:

@QObject::connect(mSharedQueue, SIGNAL(dataAvailable()), thread2, SLOT(start()));@

and the thread2's run function looks as follows:

@void MyThread2::run()
{
mSharedQueue.clear();
}@

If I make the above connection a blocking connection:

@QObject::connect(mSharedQueue, SIGNAL(dataAvailable()), thread2, SLOT(start()), Qt::BlockingQueuedConnection);@

everything works perfect and all memory is returned. However I can't use BlockingQueuedConnection in my case, since it causes deadlocks (as the docs state) in most cases. (The few times it didn't deadlock everything cleared correctly).

So I don't think this problem lies with QQueue/QList or QByteArray, but somewhere with the signals/slots and QThread.

[goocreations]

Ok, I've use Valgrind. Seems like there are no "real" leaks. Valgrind reports leaks way beyond Qt, somewhere in the GNU linux libraries.

[MuldeR]

[quote author="goocreations" date="1341655010"]Ok, I've use Valgrind. Seems like there are no "real" leaks. Valgrind reports leaks way beyond Qt, somewhere in the GNU linux libraries.[/quote]

Well, this doesn't mean much! After all, the call stack for every memory allocation (including those that are "leaks") will end up in C/C++ or OS library functions. It doesn't mean the cause of the problem is there tough. The problem usually is somewhere "above" in the call hierarchy. So only if the call stack for a memory leak's allocation neither originates from a Qt function nor passes through a Qt function, you can be sure that it's not Qt's fault. In the other case, if you see some Qt-related function somewhere on the call stack, it might be Qt's fault. But it might be as well your application code using the Qt library in the wrong way (e.g. constructing some Qt object but never destroying it). Further investigation will be needed along all the functions on the stack...