Weird memory leak on linux

fbg13

Sample application https://gitlab.com/g-fb/memory-extraction

I have an application that extracts archives to memory and when it finishes it returns a QVector<MemoryFile>. If I don't do anything with it there's no issue, but if I assign it to a member variable in my class then the memory is not released when I reopen the archive.

When I first extract an archive, the app's memory increases by the archive's size. If I load the same archive a second time, the memory increases again by the archive's size. If I keep opening the same file the memory keeps increasing.

On Windows it doesn't behave like this. After each extraction the memory stays about the same.

Here's the relevant code (full code)

...
void MainWindow::extractToMemory(QString archiveFile)
{
    qDebug() << archiveFile;

    using QArchive::MemoryExtractor;
    using QArchive::MemoryExtractorOutput;
    using QArchive::MemoryFile;

    auto extractor = new MemoryExtractor(archiveFile);
    extractor->setCalculateProgress(true);
    extractor->getInfo();
    extractor->start();

    connect(extractor, &MemoryExtractor::finished, this, [=](MemoryExtractorOutput *data) {
        QVector<MemoryFile> files = data->getFiles();
        images.clear();
        images = files;
        data->deleteLater();
        extractor->clear();

        qDebug() << "MemoryExtractor::finished \n";
    });
}

If I comment out the images.clear(); and images = files; there's no memory leak.

Library used for extraction https://github.com/antony-jr/QArchive

MemoryFile header

Christian Ehrlicher

Use e.g. valgrind to find out where/if exactly the memory is leaking. Don't see what the Qt library can do here though and don't think there is a real memory leak - how do you check it? ps is not the correct tool for this...

fbg13

@Christian-Ehrlicher said in Weird memory leak on linux:

Use e.g. valgrind to find out where/if exactly the memory is leaking.

Valgrind crashes on my system, I'll have to install another distro.

Don't see what the Qt library can do here though and don't think there is a real memory leak - how do you check it? ps is not the correct tool for this...

I'm using KDE's system monitor and task manager on Windows.

Also forgot to mention, KDE's system monitor has a detailed memory information window and there I can see this https://i.imgur.com/fYMlKFn.png

Christian Ehrlicher

@fbg13 said in Weird memory leak on linux:

Also forgot to mention, KDE's system monitor has a detailed memory information window

Still not much more than a ps - use a memory checker/leak analyzer to see where/if the memory leak comes from.

fbg13

@Christian-Ehrlicher said in Weird memory leak on linux:

Still not much more than a ps - use a memory checker/leak analyzer to see where/if the memory leak comes from.

Do you know if heaptrack is good for that?
I run it and this is what it reports:

peak heap memory consumption:
575,1MB after 27.358s
peak RSS (including heaptrack overhead):
582,3MB
total memory leaked:
1,1MB (328,7kB suppressed)

The archive's size was 200MB.

Christian Ehrlicher

I don't see anything in the heaptrack output which shows a massive leak anywhere. So I would guess valgrind will tell you the same.

fbg13

@Christian-Ehrlicher so is this a case of https://www.linuxatemyram.com?

SimonSchroeder

Where is extractor deleted? Instead of clear() I would call deleteLater() in this place (assuming this variable is not used anywhere else.

As long as we assume clear() to work correctly and release the memory of the files inside the archive this is not your problem.

However, there is a huge difference in physical and virtual memory. If you allocate memory for the first time it will get it from the OS. When you release the memory it might end up in an internal pool for use by further allocations. If MemoryExtractor allocates the memory for all files at once as one big chunk, other allocations steal from the internal pool and the second time MemoryExtractor still needs to get the same amount of physical memory. It is less likely, though, that this would happen a third time. Without the right tools this is impossible to figure out what actually happens.

fbg13

@SimonSchroeder I tried with deleteLater() too, same thing. Also, I kept loading a big file until my system froze and the app was killed.

No idea what's going on. I don't think it's a leak in the QArchive library since the memory goes down if I comment out the images = files;. And the same thing happened with KArchive when I tried storing each file's data as a QByteArray in a container like I did with QArchive.
In the end I figured out how to use KArchive to load only the file that I need and now everything's fine.

Would still like to understand what's happening though.

SimonSchroeder

It's been a while since I observed this, but my experience with Linux was that as soon as the virtual memory hits the limit of the physical memory the OS will kill the app. Say you have 16GB of RAM (including swap), once your virtual memory is 16GB Linux seems to recognize this as out of memory if you are trying to surpass this. Even if the physical memory associated with your app is just 10GB (or whatever). As I said: It's been a while since I've observed this, so I am not sure if this still applies.

Kent-Dorfman

I'm not sure what the backing of MemoryFile looks like. It is probably implemented differently between windoze and linux. Typically in the posix world files are mmap() to virtual memory space when accessed so that only needed pages are in real memory when needed, but all pages are continugously addressable to the process memory map. Perhaps this mechanism isn't being used effectively in the linux Qt implementation?