Qt Forum

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Unsolved

    [Solved] QDir::count() abnormally slow

    Mobile and Embedded
    2
    4
    2913
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      KaiBT last edited by

      Hi all,

      I have the following issue with Qt 4.7.3, 4.7.4 and 4.8.0 for embedded Linux.

      I have 9 small M3U files placed in a PLAYLIST folder of a USB HDD, at the root of a FAT32 partition. Once mounted, the resulting path is /media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST/*.m3u.

      @
      QDir playlistDir(path, QLatin1String("*"), QDir::Unsorted);
      for (uint i = 0; i < playlistDir.count(); i++) {
      ...
      }
      @

      If I run the above code with path set to "/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST", QDir::count() takes 13 seconds, for 9 files!!!

      I have tried with a FAT32 SD card ("/media/sd/PLAYLIST"): same result.

      I have tried to reach these paths through symlinks: same result.

      I have tried to place the files on my root FS (UBIFS: "/root/PLAYLIST"): no issue.

      I have tried a bind mount from my root FS: no issue.

      For the cases triggering the issue, executing this code twice does not cause any significant latency on the 2nd run. Using system commands instead of QDir does not cause any issue.

      So the common parameters in both configurations triggering the issue are: FAT32 and non-root FS mount point. I don't know which of these is causing the issue, but changing the FS type is not an option for my application.

      Any idea of a trick that could help, or something?

      Thanks in advance for your help.

      Regards,
      KaiBT

      1 Reply Last reply Reply Quote 0
      • C
        chriadam last edited by

        I might be wrong, but I seem to recall that the QDir/QFile I/O classes in Qt 4.7 were based on a file system abstraction which had certain performance implications. In particular, the amount of stats which occurred was abnormally high in some situations. Are you able to use strace to see what in particular is taking the most time? Or, failing that, use callgrind or something.

        1 Reply Last reply Reply Quote 0
        • K
          KaiBT last edited by

          The strace looks like:

          @
          337 13:55:43.696557 open("/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC <unfinished ...>
          340 13:55:43.706249 <... nanosleep resumed> NULL) = 0 <0.020343>
          340 13:55:43.706976 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000237>
          340 13:55:43.708397 nanosleep({0, 20000000}, NULL) = 0 <0.020370>
          340 13:55:43.729790 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000237>
          340 13:55:43.731078 nanosleep({0, 20000000}, <unfinished ...>
          337 13:55:43.734947 <... open resumed> ) = 23 <0.037507>
          337 13:55:43.735617 statfs("/media/sata-hdd:1.0-scsi-0:0:0:0-part1/PLAYLIST/", <unfinished ...>
          340 13:55:43.753041 <... nanosleep resumed> NULL) = 0 <0.020446>
          @

          Then, a huge amount of:

          @
          340 13:55:43.754423 select(1, [0], NULL, NULL, {0, 0}) = 0 (Timeout) <0.000410>
          340 13:55:43.755898 nanosleep({0, 20000000}, NULL) = 0 <0.020481>
          @

          And finally:

          @
          337 13:55:59.498871 <... statfs resumed> {f_type="MSDOS_SUPER_MAGIC", f_bsize=16384, f_blocks=7810801, f_bfree=7565231, f_bavail=0, f_files=0, f_ffree=16384, f_fsid={0, 0}, f_namelen=0, f_frsize=0}) = 0 <15.762265>
          337 13:55:59.501756 getdents64(23, /* 11 entries */, 32768) = 424 <0.012794>
          @

          So the issue is clearly the statfs, which is totally useless at this point. df is also slow for these mounted drives, but it's useless to get the contents of a folder.

          KaiBT

          1 Reply Last reply Reply Quote 0
          • K
            KaiBT last edited by

            FYI, I read Qt's source code and found that statfs() was never invoked explicitly by Qt for QDir:count(). statfs() is actually called by EGLIBC as a consequence of the following line in QFileSystemIterator::QFileSystemIterator:

            @
            size_t maxPathName = ::pathconf(nativePath.constData(), _PC_NAME_MAX);
            @

            But this is a legitimate call required before readdir_r(), and readdir_r is required rather than readdir() to support reentrance.

            So Qt is doing the right thing here. I will have to either cache a statfs() during some spare application time before QDir::count(), or to trust the usefree mount option.

            KaiBT

            1 Reply Last reply Reply Quote 0
            • First post
              Last post