QTemporaryFile takes forever



  • Hi everyone, I am building an application that has to load a very big number of objects and store it in a QHash. Since loading them all in memory is impracticable (it would run out of RAM pretty fast) I thought I could store the individual objects in a QTemporaryFile and keep in memory just the path to the file. Now the problem is that as the number of QTemporaryFiles grows the execution time of the open() method becomes ridiculous.

     int main(int argc, char *argv[])
    {
        QList<QTemporaryFile*> allfiles;
        QTime timeCounter;
        int lastTime = 0;
        int currentTime=0;
        QFile file("C:/Temp/ChangeAverageTime.txt");
        file.open(QIODevice::WriteOnly | QIODevice::Text);
        QTextStream stream(&file);
        timeCounter.start();
        for (int i = 0; i < 2000;++i){
            if(i%100==0){
                lastTime = currentTime;
                currentTime = timeCounter.elapsed();
                stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(currentTime - lastTime) / 1000.0, 0, 'f', 2) << '\n';
            }
            allfiles.append(new QTemporaryFile("C:/temp/TempTest/"));
            allfiles.last()->open();
        }
        file.close();
        while (!allfiles.isEmpty())
            delete allfiles.takeLast();
         return 0;
    }
    

    Produces in output:
    0 Files. Time: 0.00
    100 Files. Time: 0.03
    200 Files. Time: 0.04
    300 Files. Time: 0.04
    400 Files. Time: 0.05
    500 Files. Time: 0.12
    600 Files. Time: 0.25
    700 Files. Time: 3.58
    800 Files. Time: 12.05
    900 Files. Time: 12.78
    1000 Files. Time: 13.38
    1100 Files. Time: 15.34
    1200 Files. Time: 15.79
    1300 Files. Time: 16.66
    1400 Files. Time: 24.11
    1500 Files. Time: 31.28
    1600 Files. Time: 31.95
    1700 Files. Time: 32.77
    1800 Files. Time: 35.13
    1900 Files. Time: 36.37

    Is it normal? 36s to create 100 files makes it straight unusable... Any suggestion on how to fix this?

    I'm running Qt 5.5.1 on Windows 7 x64 (the program is compiled as 32 bits executable)


  • Moderators

    So, you want to store each object in its own file?
    And you have "very big number of objects", right?
    Then you will create a very big number of temporary files, I don't think such an approach will deliver a good performance.



  • That's not exactly the point, if I simulate the same behaviour using QFile the results is MUCH faster and creating the file does not depend on the number of files already in the folder

         QList<QFile*> allfiles;
         QTime timeCounter;
         QString tempFileName;
         QFile file("C:/Temp/ChangeAverageTime.txt");
         file.open(QIODevice::WriteOnly | QIODevice::Text);
         QTextStream stream(&file);
         timeCounter.start();
         for (int i = 0; i < 2000; ++i) {
             if (i % 100 == 0) {
                 stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(timeCounter.restart()) / 1000.0, 0, 'f', 2) << '\n';
             }
             tempFileName.clear();
             do { // Generate random file name
                 for (int j = 0; j < 15; ++j) {
                     tempFileName.append(static_cast<char>(97 + (qrand() % 26)));
                 }
             } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
             allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
             allfiles.last()->open(QIODevice::WriteOnly);
         }
         file.close();
         while (!allfiles.isEmpty()) {
             auto tempDel = allfiles.takeLast();
             tempDel->remove();
             delete tempDel;
         }
         return 0;
    

    0 Files. Time: 0.00
    100 Files. Time: 0.03
    200 Files. Time: 0.03
    300 Files. Time: 0.03
    400 Files. Time: 0.03
    500 Files. Time: 0.03
    600 Files. Time: 0.03
    700 Files. Time: 0.03
    800 Files. Time: 0.03
    900 Files. Time: 0.03
    1000 Files. Time: 0.03
    1100 Files. Time: 0.03
    1200 Files. Time: 0.03
    1300 Files. Time: 0.03
    1400 Files. Time: 0.03
    1500 Files. Time: 0.03
    1600 Files. Time: 0.03
    1700 Files. Time: 0.03
    1800 Files. Time: 0.03
    1900 Files. Time: 0.03


  • Moderators

    I think creating temporary files is slower compared to creating new files, because Qt framework has to ensure that the file names are unique.



  • I'm doing that too in: while (QFile::exists("C:/temp/TempTest/" + tempFileName)); but it's lightning fast


  • Moderators

    Why shouldn't it be fast? You only check whether a file exists or not. What QTemporaryFile is doing: it creates a unique file name and ensures nobody else creates one with the same name in the same directory. This, for sure, involves some OS calls to make the creation of a temp file an atomic operation. Else several processes could create and use a temp file with same file name.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.