QTemporaryFile takes forever

[VRonin]

Hi everyone, I am building an application that has to load a very big number of objects and store it in a QHash. Since loading them all in memory is impracticable (it would run out of RAM pretty fast) I thought I could store the individual objects in a QTemporaryFile and keep in memory just the path to the file. Now the problem is that as the number of QTemporaryFiles grows the execution time of the open() method becomes ridiculous.

 int main(int argc, char *argv[])
{
    QList<QTemporaryFile*> allfiles;
    QTime timeCounter;
    int lastTime = 0;
    int currentTime=0;
    QFile file("C:/Temp/ChangeAverageTime.txt");
    file.open(QIODevice::WriteOnly | QIODevice::Text);
    QTextStream stream(&file);
    timeCounter.start();
    for (int i = 0; i < 2000;++i){
        if(i%100==0){
            lastTime = currentTime;
            currentTime = timeCounter.elapsed();
            stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(currentTime - lastTime) / 1000.0, 0, 'f', 2) << '\n';
        }
        allfiles.append(new QTemporaryFile("C:/temp/TempTest/"));
        allfiles.last()->open();
    }
    file.close();
    while (!allfiles.isEmpty())
        delete allfiles.takeLast();
     return 0;
}

Produces in output:
0 Files. Time: 0.00
100 Files. Time: 0.03
200 Files. Time: 0.04
300 Files. Time: 0.04
400 Files. Time: 0.05
500 Files. Time: 0.12
600 Files. Time: 0.25
700 Files. Time: 3.58
800 Files. Time: 12.05
900 Files. Time: 12.78
1000 Files. Time: 13.38
1100 Files. Time: 15.34
1200 Files. Time: 15.79
1300 Files. Time: 16.66
1400 Files. Time: 24.11
1500 Files. Time: 31.28
1600 Files. Time: 31.95
1700 Files. Time: 32.77
1800 Files. Time: 35.13
1900 Files. Time: 36.37

Is it normal? 36s to create 100 files makes it straight unusable... Any suggestion on how to fix this?

I'm running Qt 5.5.1 on Windows 7 x64 (the program is compiled as 32 bits executable)

[jsulm]

So, you want to store each object in its own file?
And you have "very big number of objects", right?
Then you will create a very big number of temporary files, I don't think such an approach will deliver a good performance.

[VRonin]

That's not exactly the point, if I simulate the same behaviour using QFile the results is MUCH faster and creating the file does not depend on the number of files already in the folder

     QList<QFile*> allfiles;
     QTime timeCounter;
     QString tempFileName;
     QFile file("C:/Temp/ChangeAverageTime.txt");
     file.open(QIODevice::WriteOnly | QIODevice::Text);
     QTextStream stream(&file);
     timeCounter.start();
     for (int i = 0; i < 2000; ++i) {
         if (i % 100 == 0) {
             stream << QString("%1 Files. Time: %2").arg(i).arg(static_cast<double>(timeCounter.restart()) / 1000.0, 0, 'f', 2) << '\n';
         }
         tempFileName.clear();
         do { // Generate random file name
             for (int j = 0; j < 15; ++j) {
                 tempFileName.append(static_cast<char>(97 + (qrand() % 26)));
             }
         } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
         allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
         allfiles.last()->open(QIODevice::WriteOnly);
     }
     file.close();
     while (!allfiles.isEmpty()) {
         auto tempDel = allfiles.takeLast();
         tempDel->remove();
         delete tempDel;
     }
     return 0;

0 Files. Time: 0.00
100 Files. Time: 0.03
200 Files. Time: 0.03
300 Files. Time: 0.03
400 Files. Time: 0.03
500 Files. Time: 0.03
600 Files. Time: 0.03
700 Files. Time: 0.03
800 Files. Time: 0.03
900 Files. Time: 0.03
1000 Files. Time: 0.03
1100 Files. Time: 0.03
1200 Files. Time: 0.03
1300 Files. Time: 0.03
1400 Files. Time: 0.03
1500 Files. Time: 0.03
1600 Files. Time: 0.03
1700 Files. Time: 0.03
1800 Files. Time: 0.03
1900 Files. Time: 0.03

[jsulm]

I think creating temporary files is slower compared to creating new files, because Qt framework has to ensure that the file names are unique.

[VRonin]

I'm doing that too in: while (QFile::exists("C:/temp/TempTest/" + tempFileName)); but it's lightning fast

[jsulm]

Why shouldn't it be fast? You only check whether a file exists or not. What QTemporaryFile is doing: it creates a unique file name and ensures nobody else creates one with the same name in the same directory. This, for sure, involves some OS calls to make the creation of a temp file an atomic operation. Else several processes could create and use a temp file with same file name.

[Mecanica]

I know this thread is too old, but I would resolve this by using an inmemory database like sqlite, where you can have two columns one for the file name an another for the file data stored as a QByteArray.

[JonB]

@Mecanica
Whatever the merits of your suggestion, that would not be a suitable replacement for a QTemporaryFile method in the Qt framework!

[JonB]

@VRonin said in QTemporaryFile takes forever:

     } while (QFile::exists("C:/temp/TempTest/" + tempFileName));
     allfiles.append(new QFile("C:/temp/TempTest/" + tempFileName));
     allfiles.last()->open(QIODevice::WriteOnly);

@VRonin I see that you wrote this in 2015. Leaving aside that if the timing you describe still holds then I am not sure why and yours is an interesting finding, looking at this code now would you recommend it for exclusively creating a file, in a multi-process environment? ;-)

[Christian Ehrlicher]

@JonB The code above is not correct and can create a race condition since another thread/process can create exactly the same filename between QFile::exists() and open(). That's also the reason why the Qt implementation is slower - it opens a file and if it fails it checks some error conditions.
Also the possible space for the file names is lower in QTemporaryFile (only six digits by default) while the other loop is using 15 characters, so the chance of a collision is much lower.

[JonB]

@Christian-Ehrlicher
It was the race condition that I had noticed. I was bringing that to @VRonin 's attention and seeing what he said about it, it's sometimes interesting to look back on your own code from years ago :)

[VRonin]

The bug was solved in Qt 5.8 so a while ago. I'll mark the thread as solved as well