[Solved] QDir::count() abnormally slow

KaiBT

Hi all,

I have the following issue with Qt 4.7.3, 4.7.4 and 4.8.0 for embedded Linux.

I have 9 small M3U files placed in a PLAYLIST folder of a USB HDD, at the root of a FAT32 partition. Once mounted, the resulting path is /media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST/*.m3u.

@
QDir playlistDir(path, QLatin1String("*"), QDir::Unsorted);
for (uint i = 0; i < playlistDir.count(); i++) {
...
}
@

If I run the above code with path set to "/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST", QDir::count() takes 13 seconds, for 9 files!!!

I have tried with a FAT32 SD card ("/media/sd/PLAYLIST"): same result.

I have tried to reach these paths through symlinks: same result.

I have tried to place the files on my root FS (UBIFS: "/root/PLAYLIST"): no issue.

I have tried a bind mount from my root FS: no issue.

For the cases triggering the issue, executing this code twice does not cause any significant latency on the 2nd run. Using system commands instead of QDir does not cause any issue.

So the common parameters in both configurations triggering the issue are: FAT32 and non-root FS mount point. I don't know which of these is causing the issue, but changing the FS type is not an option for my application.

Any idea of a trick that could help, or something?

Thanks in advance for your help.

Regards,
KaiBT

chriadam

I might be wrong, but I seem to recall that the QDir/QFile I/O classes in Qt 4.7 were based on a file system abstraction which had certain performance implications. In particular, the amount of stats which occurred was abnormally high in some situations. Are you able to use strace to see what in particular is taking the most time? Or, failing that, use callgrind or something.

KaiBT

The strace looks like:

@
337 13:55:43.696557 open("/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC <unfinished ...>
340 13:55:43.706249 <... nanosleep resumed> NULL) = 0 <0.020343>
340 13:55:43.706976 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000237>
340 13:55:43.708397 nanosleep({0, 20000000}, NULL) = 0 <0.020370>
340 13:55:43.729790 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000237>
340 13:55:43.731078 nanosleep({0, 20000000}, <unfinished ...>
337 13:55:43.734947 <... open resumed> ) = 23 <0.037507>
337 13:55:43.735617 statfs("/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST/", <unfinished ...>
340 13:55:43.753041 <... nanosleep resumed> NULL) = 0 <0.020446>
@

Then, a huge amount of:

@
340 13:55:43.754423 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000410>
340 13:55:43.755898 nanosleep({0, 20000000}, NULL) = 0 <0.020481>
@

And finally:

@
337 13:55:59.498871 <... statfs resumed> {f_type="MSDOS_SUPER_MAGIC", f_bsize=16384, f_blocks=7810801, f_bfree=7565231, f_bavail=0, f_files=0, f_ffree=16384, f_fsid={0, 0}, f_namelen=0, f_frsize=0}) = 0 <15.762265>
337 13:55:59.501756 getdents64(23, /* 11 entries */, 32768) = 424 <0.012794>
@

So the issue is clearly the statfs, which is totally useless at this point. df is also slow for these mounted drives, but it's useless to get the contents of a folder.

KaiBT

KaiBT

FYI, I read Qt's source code and found that statfs() was never invoked explicitly by Qt for QDir:count(). statfs() is actually called by EGLIBC as a consequence of the following line in QFileSystemIterator::QFileSystemIterator:

@
size_t maxPathName = ::pathconf(nativePath.constData(), _PC_NAME_MAX);
@

But this is a legitimate call required before readdir_r(), and readdir_r is required rather than readdir() to support reentrance.

So Qt is doing the right thing here. I will have to either cache a statfs() during some spare application time before QDir::count(), or to trust the usefree mount option.

KaiBT