QDirIterator much slower on Qt 6.8.

Jammin44fm

Hey All,

I'm porting an application from Qt 6.5.3 to Qt 6.8.2.
In one section of code I need to iterate a large number of dirs and files, and get the filename, filesize and last modified time, and then construct a hashmap with those values.
The filepath is used as the key, and a class with the size and last modified time as the value.

The Qt code I'm using looks like this..

    QHash<QString, MyFileUtil::FileStat> result;
    for(QDirIterator it(sourcePath, QDir::Files, QDirIterator::Subdirectories); it.hasNext(); )
    {
        it.next();
        result.insert(it.filePath(), MyFileUtil::FileStat(it.fileInfo()));
    }

Since moving to Qt 6.8.2 this code is now running significantly slower.
In a simple example of 155 files it takes around 3.1ms when using Qt 6.5.3, however using the same code on Qt 6.8.2 it's now taking around 6.2ms - basically twice as long for the same amount of work!

A larger example only gets worse.
iterating over 85124 files takes around 820ms in Qt 6.5.3, However the same code on Qt 6.8.2
is taking a massive 4400ms!! like 5 times as long!

I've also tried the newer QDirListing Iterator, using the following code...

 QHash<QString, MyFileUtil::FileStat> result;
 const auto flags = QDirListing::IteratorFlag::FilesOnly | QDirListing::IteratorFlag::Recursive;
 for(const auto& dirEntry : QDirListing(sourcePath, flags))
 {
     result.insert(dirEntry.filePath(), MyFileUtil::FileStat(dirEntry.fileInfo()));
 }

However this produces similar timings to the QDirIterator based code.
I thought that creating the fileInfo object might be what taking so long in each iteration,
but I tried commenting that out, and replacing the MyFileUtil::FileStat copnstructor with just passing in 2 fixed values. ( So still doing the filePath() op, but not creating the fileinfo object to read the size and modified time)
However this only reduced the total runtime for the small example by about 1ms.

I know that saving 3ms doesn't seem like a big deal, but when dealing with a large number of files across many dirs, just this portion of the operation is taking 3 seconds longer it starts to be a real problem.

I'd love some help or insight to help me understand why this operation is suddenly so much slower than it used to be?

Currently I'm looking writing a windows specific path, just for windows.
The software needs to run on Win, Mac and Linux, so using Qt to provide a single solution is ideal.

Thanks,

James

Jammin44fm

OK I've done some more research, and found that in Qt 6.8 QDirIterator is just wrapping the newer, QDirListing.
However that the QDirListing is now making a call to GetFileAttributesExW, which was never happening in the older pure QDirListing version used in Qt 6.5.

Examining the hot path in each case make it pretty obvious that the calls to GetFileAttributesExW, are basically in addition to the existing to calls to FindNextFileW.
ie. the Qt 6.5 hot path contains FindNextFileW , but the Qt 6.8 hot path has the calls to GetFileAttributesExW, and calls to FindNextFileW exist, in a similar bnumber as 6.5, but no longer form part of the hot path.

Is there any way to make the new QDirListing, not make all the calls to GetFileAttributesExW? or operate similar to the old QDirListing?

Thanks,
James

Jammin44fm

Ok Some more findings...

I've moved my iterator to use QDirListing, and tried 2 different blocks of code inside the loop itself.
One that only reads the file path, and thus should not cause any calls to for the dirEntry object to crate a proper fileInfo object, which according to the docs should be faster, that code looks like...

    const auto flags = QDirListing::IteratorFlag::FilesOnly | QDirListing::IteratorFlag::Recursive;
    for(const auto& dirEntry : QDirListing(sourcePath, flags))
    {
        qint64 size = 0;
        qint64 lastModified = 99; 
        result.insert(dirEntry.filePath(), MyFileUtil::FileStat(size, lastModified));
    }

Sadly this code is still very slow, clocking in at around 4.9ms for the small case of 155 files,
and 3900ms for the large case - 85124 files.
Looking at a trace of the hotpath of the code it looks like it is still making calls to GetFileAttributesExW ,
for each iteration. I was hoping that by not making any calls that require a fileInfo object to be created i would avoid calls to GetFileAttributesExW?

I also tried a more complete example that looks like

    const auto flags = QDirListing::IteratorFlag::FilesOnly | QDirListing::IteratorFlag::Recursive;
    for(const auto& dirEntry : QDirListing(sourcePath, flags))
    {
        qint64 size = dirEntry.size();
        qint64 lastModified = dirEntry.lastModified(QTimeZone::LocalTime).toMSecsSinceEpoch();
        result.insert(dirEntry.filePath(), MyFileUtil::FileStat(size, lastModified));
    }

However this is even worse, with the small case being around 6ms, and the large case coming in around
4600ms.

FYI. the constructor for the MyFileUtil::FileStat is just doing a simple assignment to the 2 quint64 member vars,
so i dont think thats chewing up much time.

Also I have now tested against a windows specific code path, and that comes in around 1ms for the small case, and 298ms for the large case.
I know that Qt is more general, and that a platform specific code path will almost always be faster, but we are talking the Qt code being 15x slower than the platform specific version, which i think is a bit wrong,
AND it's a considerable slowdown form Qt 6.5 to 6.8, which is also problematic.

Hopefully I am mis-using the code, or not doing something correctly?

Thanks,
James

Asperamanca

This may be something for a bug report (https://bugreports.qt.io), or for the Qt mailing lists (https://lists.qt-project.org/)

Christian Ehrlicher

See https://bugreports.qt.io/browse/QTBUG-134699