[QFile] Why QFile::map QFile::unmap sometimes took a long time ?

david_wang

Hi all,

[Description]
I have a question about QFile::map and QFile::unmap
To accelerate the time highly frequently doing operation read/write files spent, I use the file-mapping method instead.

Everything seems to go well, but sometimes QFile::map and QFile::unmap took 100~200 times time (calling it Spike) than the average time they took.
e.g. QFile::map:
Average time took 75 microseconds, Spike took 18,000 microseconds !!

Here is my scenario and code snippet:

Step 1. Create a QFile Object (~38 MB)
Step 2. Calling map
Step 3. Modify content (~16 MB)
Step 4. Calling unmap

(From Step 1. to Step 4., Calling it one Loop.)

            const unsigned int kMB   = 1024*1024;
            const unsigned int kLoop = 1000; // Could be more
            QVector<QFile*> fileList;
            fileList.reserve(kLoop);
            for(size_t i = 0; i < kLoop; ++i) {
                // Create file
                QString fPath = QString("Z:/%1.mem").arg(i);
                fileList.push_back(new QFile(fPath));
                if(!fileList[i]->open(QIODevice::ReadWrite)) {
                    qDebug() << "File cannot open";
                }
                fileList[i]->resize(38*kMB); // 38 MB

                // map
                uchar* memory = fileList[i]->map(0, fileList[i]->size());

                // Modify
                memcpy(memory, pattern.data(), pattern.size()); // pattern: A QByteArray with random 16 MB data

                //unmap
                fileList[i]->unmap(memory);
            }

[Experiments]
To dig into what are the influencing factors resulting in Spike happening sometimes, there are experiments I tried.

CPU usage may be the factor.
Run the snippet above while quickly open/close Google Chrome page tab. With the CPU usage surge, both QFile::map and QFile::unmap took more time than average.
<Average>
QFile::map: 67
QFile::unmap:1173
<Spike>
QFile::map: 97
QFile::unmap:1472
Disk usage may be the factor leading to QFile::map Spike.
With frequent doing map/unmap operation, the disk usage keeps on a high level (~70%).
It is likely to happen QFile::map Spike while disk usage is running in a 100% busy state. In addition, it still happened when disk usage is running in a low workload(10~20%).
(The aforementioned example: average time took 75 microseconds, Spike took 18,000 microseconds.)
However, unmap is not significantly affected by disk usage.

[Environment]
OS: Window 10
CPU: AMD Ryzen 5 2400G
Disk: PCIe SSD 1TB
RAM: DDR4 32GB

Here is my question again:

Why QFile::map QFile::unmap sometimes took a long time ?
Is Spike is a "Normal" situation ?

Christian Ehrlicher

QFile::map() directly maps to the OS API so it's most likely an OS caching issue:

https://code.woboq.org/qt5/qtbase/src/corelib/io/qfiledevice.cpp.html#_ZN11QFileDevice3mapExxNS_14MemoryMapFlagsE

and then for Windows:

https://code.woboq.org/qt5/qtbase/src/corelib/io/qfsfileengine_win.cpp.html#_ZN20QFSFileEnginePrivate3mapExxN11QFileDevice14MemoryMapFlagsE

david_wang

Hi @Christian-Ehrlicher

Do you suggest this should ask Microsoft instead to know why the Spike happened ?

Thank you.

Christian Ehrlicher

Don't know if they will help you but I can't see what Qt can do against it. To be sure it's not Qt you can call Map/UnmapViewOfFile directly instead through Qt.
It may be some of your local caches too. Maybe the harddisk is slow or whatever but I'm pretty sure it's not Qt.

JonB

@david_wang
I don't know what you expect. When you map, and especially unmap after making changes, the "fast" in-memory changes you have been made have to be physically reflected back to the hard disk/storage. This can take a "long" time to do physical writes and flush buffers. Just like if you perform some disk-write-intensive operation, like a "backup", which starts off fast till it fills buffers and then goes slower at times as it physically flushes, it can feel "lumpy". map may be faster than explicit disk access, but it's not magic, and is still ultimately bound by the physical hardware behaviour.

david_wang

@Christian-Ehrlicher
Thank you for reply, I will give it a try and ask Microsoft

JonB

@david_wang said in [QFile] Why QFile::map QFile::unmap sometimes took a long time ?:

and ask Microsoft

:)

david_wang

@JonB
From the point of view of the snippet above, yes, exactly as you said, it looks confusing.

What I want to do is to simulate a storage device (use .mem as a storage capacity unit).
Initial idea is to use LRU (recent used .mem kept on memory (map), least used .mem write back to disk (unmap) ) method to decide my .mem strategy.

However, sometimes the QFile::map, QFile::unmap took a long time (Spike happened), that confused me.

SimonSchroeder

@david_wang said in [QFile] Why QFile::map QFile::unmap sometimes took a long time ?:

recent used .mem kept on memory (map), least used .mem write back to disk (unmap)

I am not sure that you can make these assumptions. The only thing you can assume is that you can access a mapped file as if in memory. There might be a few pitfalls which could be implementation dependent (these are just uninformed thought experiments; I don't have an particular knowledge in this area):

Once you map a file it does not have to be in memory immediately. The OS could implement that once you access a part of the file that is not in memory yet issues a page fault and loads the according page from file. Especially if you immediately access the file it might not be in memory right away. It might be still loading in the background...
There is nothing hindering the implementation of memory mapping to flush parts of the file to disk when idle. I am not sure if any implementation does this as this might increase wear on the disk. But, it could be possible. Especially when you are low on memory the OS could decide to write some memory pages belonging to your file back to disk directly instead of swap space.
I am not sure what triggers the actual write-back to disk. This does not have to be unmap. Unmap just tells the OS that it does not have to hold the file in memory any longer. Perhaps it is up to the OS when to write the file back to disk. Maybe the file is only flushed explicitly in the destructor of QFile? The spike could happen when the OS decides to write back other files before mapping another file. Still, this is all speculation.

You see from all these explanations that you might not have the full control of disk accesses as you expect.

david_wang

@SimonSchroeder

I agree your point that it's hard to have the full control of disk since there is an OS between disk and us.
Especially your 3rd point, I have the same thought either.

BTW, I gave up calling QFile::map and QFile::unmap but calling the CreateFileMapping+MapViewOfFile and UnmapViewOfFile respectively.

Good news is that CreateFileMapping is the culprit of taking up the most time of QFile::map, I mean spike is caused by CreateFileMapping API.
However, it's still not sure why it took so much time and keeping waiting for Microsoft's reply.