Memory Leak in Qt signal queue?

thielepaul

I have pretty much the same problem like the one described in this 3 year old thread:
Re: Memory leak when using queued signal&slot connections

As far as I understand, the memory used by the elements in the eventloop-queue should be freed once the all of them have been processed. However, it seems that the memory is freed only after calling malloc_trim(0);.

In the original thread it says that "when an application releases memory, the OS does not necessarily reclaim it immediately". For me it seems, like the memory is not reclaimed at all. When there is no more memory left, the OOM killer will kill other processes, but the memory is still not reclaimed.

For me, this is not a hypothetical problem but I see this issue in a productive application: there are have multiple threads, and if the thread with the slot is over-utilized, the queue length will grow and need more memory which is fine. However, I also see there the same problem that the small example below illustrates, that the memory is not freed once all elements from the queue have been processed.

It would be great, if you have some advice for me, how I can continue with this issue:

Is there something conceptually wrong to pass data between threads with qt signals and if yes, what should I use instead?
If using qt signals is fine, is it possible to limit the maximum queue length (similar like setting the capacity for a golang channel)?
Should I call malloc_trim regularly to make sure that the memory is freed?

main.cpp

#include "object.h"
#include <QCoreApplication>
#include <csignal>
#include <iostream>
#include <malloc.h>

int main(int argc, char **argv)
{
    signal(SIGINT, QCoreApplication::exit);
    signal(SIGTERM, QCoreApplication::exit);

    QCoreApplication app(argc, argv);
    Object object;

    QObject::connect(&object, &Object::signal, &object, &Object::slot, Qt::QueuedConnection);

    object.run();
    QCoreApplication::processEvents();
    std::cout << "processing done, press enter for executing malloc_trim" << std::endl;
    std::cin.get();
    // using 9 GB memory
    malloc_trim(0);
    // using few KB memory
    std::cout << "malloc_trim was executed" << std::endl;

    const auto status = app.exec();
    std::cout << "application finished" << std::endl;
    return status;
}

object.h

#include <QObject>
#include <iostream>

class Object : public QObject
{
    Q_OBJECT

signals:
    void signal(QByteArray) const;

public slots:
    void slot(QByteArray)
    {
    }

public:
    void run()
    {
        for (int i = 0; i < 100000; i++)
        {
            emit signal(QByteArray(100000, 'a'));
        };
        std::cout << "signaling done" << std::endl;
    }
};

CMakeLists.txt

cmake_minimum_required(VERSION 3.0)
project(memleak)
find_package(Qt6 REQUIRED COMPONENTS
    Core
)
set(CMAKE_AUTOMOC ON)
add_executable(${PROJECT_NAME}
    main.cpp object.h
)
target_link_libraries(${PROJECT_NAME}
    Qt6::Core
)

JonB

@thielepaul
My 2 cents. I stand to be corrected by someone more knowledgeable than me.

It sounds to me it's behaving just as I would expect. Because you have a tight loop emitting queued signal data you ask it to do millions of bytes of copy allocation. No frees as it goes. Then after processing Qt loop frees all these, millions of frees in a row. At this point I would expect all the allocated bytes to be returned to the malloc allocation pool. But not back to the OS, why should it? The fact that your malloc_trim(0); does return the used memory allocation to the OS does just what it should.

The hope with malloc/free is that malloc will be able to re-use freed blocks before asking for more OS memory. But with all frees after all mallocs that won't happen --- yet. Though hopefully it will when future malloc are issued.

Seems reasonable to me that Qt allocates memory ultimately via malloc (and free, quite possibly new/delete etc.), not Qt's issue how that behaves. If you want to call something which tells malloc to return its currently unused allocations to the OS, that's up to you.

Am I wrong?

thielepaul

@JonB Thanks for your reply! So if I get you right, if I want to make the previously used memory available to other processes, I should just call malloc_trim regularly? It seems wrong to me that the OOM killer starts killing processes, while another process still has a lot of unused memory allocated.

JonB

@thielepaul
I wouldn't do anything until I had tested what happens in practice with my actual code.

Your issue arises because you allocate 100,000 blocks of 100,000 bytes in a tight loop, without allowing the event loop to run and deallocate these. How realistic is that to your case?
In practice, the theory/hope is that other, unrelated allocations in your code will be able to re-use the blocks after they are freed. That may happen in real-life.
If you have reason to believe that your behaviour will be to allocate a huge amount of memory, then free it, not re-use it, and then want to run external processes which will be hampered by this, then maybe yes you could use your malloc_trim(). Personally I have never come across this used in a program.

thielepaul

@JonB
Thanks again!
Yes, I do see this problem in a real application: I have one thread that is receiving packets over the network and another one that is processing the data. The data is sent from one thread to the other via a signal-slot connection. If now more packets are coming in over the network than the second thread can handle, we see the same memory effects as with the toy-example above (e.g. the processing thread can handle 10 000 p/s, but data is coming in at a rate of 11 000 p/s, after a few minutes the same amount of memory is allocated as in the example).
I agree that it probably does not make sense to infinitely queue packets between the threads, but instead it probably makes sense to drop packets when the queue has reached a certain size. This is what my second question is aiming for: are there ways to monitor and/or limit the Qt-eventloop-queue-size?
Or do you have any other suggestions, what I could do to avoid that?

Christian Ehrlicher

Then don't use signals and slots but a proper queue handling system suitable for your needs. A simple queue + semaphores should do the trick.

thielepaul

@Christian-Ehrlicher
I already had a feeling that I was using qt signals in a way they are not intended to be used. Thanks for confirming that!