Is it possible to use or reuse Windows file handle as QFile
-
Hello folks,
Hope everyone is well.
I am having the following predicament:
In my application I need the ability to read/write a file unbuffered, which for Windows based systems does not seem to be supported by the QIODevice, as explicitly stated by the docs - https://doc.qt.io/qt-6/qiodevicebase.html#OpenModeFlag-enumDoes anyone know as to why to this is not supported?
Potential IO alignment implications associated with the non-cached IOs?Regardless, in a perfect world I would ideally want use to the QFile handle to do the reading and writing so I wouldn't have to write OS specific functions to handle the aforementioned scenario with the Windows filesystem cache, hence the question :)
I did find some articles which discuss the conversion of a QFile handle to a Windows based HANDLE(done in the code below) but not the opposite(which I need).
I also did try a potential ugly workaround to re-open the same handle with the Windows API and then use the QFile API, which did not work out, as expected. In short this is what I did try:
void testReadWrite() { QByteArray buff(1024*1024, '0'); QFile myFile("C:/temp/test/2.txt"); myFile.open(QIODevice::ReadWrite); HANDLE handle = (HANDLE) _get_osfhandle(myFile.handle()); ReOpenFile ( handle, GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, FILE_FLAG_NO_BUFFERING ); for (size_t i = 0; i < 10; i++) myFile.write(buff); }
FILE_FLAG_NO_BUFFERING is the ice topping.
The above was a desperate hope that ReOpenFile will actually reuse the same handle that QFile made, which I could then use with the coveted QFile API, but it actually did open the file a second time, thus creating a second handle(IMHO).
Needless to say, myFile.write used its own QFile handle, hence getting cached reads/writes.The Microsoft docs state that ReOpenFile works only for handles created by CreateFile, but I do get the exactly same behavior if I create a handle to a file with CreateFile, then call ReOpenFile on the same very same handle.
In other words, I do get IRP_MJ_CREATE regardless whether I call QFile::open() or CreateFile which is expected, then in both cases when I call ReOpenFile I get another IRP_MJ_CREATE which is also expected and fine, but it seems to be a different handle to the file altogether.
Could be very much wrong here, but at least my logic says that if the initial handle is actually reused the handle must be closed first and then re-used by the ReOpenFile function.
Getting a second handle from ReOpenFile also makes sense to me as to not implicitly kill anything that might be still happening with the first handle, which makes it more of a better defined behavior I guess.Anyways, point being is I obviously have gaps in the knowledge as to how all of this works under the hood so any advise here would be much appreciated.
Thanks in advance.
-
Hi,
Did you already test the open overloads taking a file handle or a file descriptor ?
-
Hi SGaist,
Yep, it did last night - failed miserably :)
Stared at the same overloads for a good 30 minutes today as well - still no luck.After seeing your reply just now an idea occurred to me, which seems to work as I need it.
I suppose I needed it to sink in and get a little nudge to try again. Thank you for that by the way :)Generally, I just did the opposite by casting the HANDLE to a file descriptor, though I am not sure if this indeed the right thing to do, and the right way to do it - the line between the comments in the function below.
If someone could confirm this approach is generally fine I would certainly sleep better :)void testReadWrite() { HANDLE hFile; hFile = CreateFile( L"C:\\temp\\2.zzz", GENERIC_READ | GENERIC_WRITE, FILE_SHARE_READ | FILE_SHARE_WRITE, NULL, OPEN_EXISTING, FILE_FLAG_NO_BUFFERING, NULL ); hFile == INVALID_HANDLE_VALUE ? qDebug() << "Could not open file. Error: " << GetLastError() : qDebug() << "File opened - Win API valid handle"; //////////////////////////////// int fd = _open_osfhandle(reinterpret_cast<intptr_t>(hFile), 0); //////////////////////////////// QFile myFile; myFile.open(fd, QIODevice::ReadWrite, QFileDevice::AutoCloseHandle); if(myFile.isOpen()) { QByteArray buff(1024*1024, '0'); for (size_t i = 0; i < 10; i++) myFile.write(buff); } if(CloseHandle(hFile)) qDebug() << "WinAPI handle closed"; myFile.close(); }
Weird thing is I am seeing only one IRP_MJ_CREATE now, despite theoretically opening the same file twice(like the first time), but could as well be normal when Qt is handling that.
Another weird one is if I comment the Qt part of the code, i.e. opening, writing and closing the file with QFile I do see the expected:
file open - IRP_MJ_CREATE, invoked by the CreateFile function
Then, upon the CloseHandle I get IRP_MJ_CLEANUP and IRP_MJ_CLOSEIf I uncomment the Qt part of the code I never see the IRP_MJ_CLOSE part, i.e. the filesystem calls end up to IRP_MJ_CLEANUP, regardless of the QFileDevice::AutoCloseHandle or QFileDevice::DontCloseHandle and whether I call myFile.close() at the end along with CloseHandle.
Technically speaking, from what I read IRP_MJ_CLEANUP implies that the file handle count for an object has reached 0, thus indicating the file is indeed closed by everything in user mode, but still the behavior kind of bugs me more or less, as it seems more of a delayed close instead of a well defined close.
Bottom line, I am getting the behavior I was after so I think this could be considered a win.
For the other overload with the file handle I did try:
FILE* fh = _fdopen(fd, "rb");
Where "fd" is coming from the same cast I did above, and then passed that the QFile open overload getting the same behavior as the file descriptor overload.
Again, no clue if any of this is remotely good or appropriate to do, despite getting the results I need.Pretty much both seem to get the job done, whereas the latter is one line more in terms of code.
This begs the question if there are any trade-offs I should be aware of.
If someone could expand on this whole topic in more detail it would be much appreciated.
-
@A-Newbie said in Is it possible to use or reuse Windows file handle as QFile:
int fd = _open_osfhandle(reinterpret_cast<intptr_t>(hFile), 0);
Did you check the fd value?
-
@A-Newbie said in Is it possible to use or reuse Windows file handle as QFile:
If someone could expand on this whole topic in more detail it would be much appreciated.
Well, at least I can explain why reopening of the file did not work. QFile::handle() returns an int which means it is a copy of the handle (not a reference). This also means that it can not be used to change the handle stored inside QFile. If you use your own handle to reopen the file the file connection associated with the handle (which is a number) gets destroyed and a new handle gets created. Your local handle gets assigned a new number for the new file connection. However, QFile still has the old number stored. But, this file handle number already got invalidated. Most likely your app will crash (or do something totally unexpected/undefined).
I don't see anything wrong with your current approach. This is exactly how I would expect that the overload taking a file handle should work. You first open the file using the Windows functions and then hand it over to QFile.
I am not sure if I got it right from your description: You are supposed to open the file only once! Don't open it using QFile (for the portable way) and then open it a second time using CreateFile and handing it to a different QFile. On Windows use the work around, on other OSes use QFile directly instead.
-
Thank you guys for chipping in.
@jsulm said in Is it possible to use or reuse Windows file handle as QFile:
@A-Newbie said in Is it possible to use or reuse Windows file handle as QFile:
int fd = _open_osfhandle(reinterpret_cast<intptr_t>(hFile), 0);
Did you check the fd value?
Not in this particular case, if you are referring to error checking, but I certainly will to ensure I am getting a valid(positive) value and not -1 so the code can then move onto the QFile opening.
@SimonSchroeder said in Is it possible to use or reuse Windows file handle as QFile:
@A-Newbie said in Is it possible to use or reuse Windows file handle as QFile:
If someone could expand on this whole topic in more detail it would be much appreciated.
Well, at least I can explain why reopening of the file did not work. QFile::handle() returns an int which means it is a copy of the handle (not a reference). This also means that it can not be used to change the handle stored inside QFile. If you use your own handle to reopen the file the file connection associated with the handle (which is a number) gets destroyed and a new handle gets created. Your local handle gets assigned a new number for the new file connection. However, QFile still has the old number stored. But, this file handle number already got invalidated. Most likely your app will crash (or do something totally unexpected/undefined).
Something here doesn't seem to add up.
I would agree that QFile::handle() returns something that is not the real handle opened by QFile itself, which is fine.
But provided it is a copy why would reopening the file using the copied handle close the original handle made by QFile?
Shouldn't the API be smart enough not to close the original handle considering it is still valid?
At least this is what I am seeing with the first function I posted since I see two opens, but any close happens after I explicitly close the file/handle at the end of the function.
Furthermore, presumably I am not seeing something in ProcMon, if the original handle was indeed closed then the writes wouldn't pass when I call them with QFile and I will get an error instead since the file is not opened to begin with.
1.png for referenceI don't see anything wrong with your current approach. This is exactly how I would expect that the overload taking a file handle should work. You first open the file using the Windows functions and then hand it over to QFile.
I am not sure if I got it right from your description: You are supposed to open the file only once! Don't open it using QFile (for the portable way) and then open it a second time using CreateFile and handing it to a different QFile. On Windows use the work around, on other OSes use QFile directly instead.
You are indeed correct, I do not want to have a file opened multiple times(unless I really do) and have potential leaking handles in the filesystem which might prevent further access to the file down the road.
The second function I posted opens the file the only once(2.png) as opposed to the first(1.png), which again brings us back to the discussion about the QFile::handle() and _open_osfhandle as both return the same number.What I fail to understand is the following:
Why in the first case(first post) when I used QFile.open() and then ReOpenFile the file is opened twice(1.png), provided ReOpenFile is referencing the number returned by QFile.handle() albeit copy, i.e. _get_osfhandle(myFile.handle()) returns 3?
I also do get do get two close calls for both handles, as expected.
In the second case scenario(second post) I do the opposite, namely creating a handle with a Windows function first(CreateFile), then casting the handle to get the number - _open_osfhandle(reinterpret_cast<intptr_t>(hFile), which again returns 3 as in the first case, but I do get only one open(2.png) and never a close, despite the fact I am explicitly calling both(just in case and for testing purposes) CloseHandle() and QFile.close().
I do get the CLEANUP which from what I understand from Microsoft's docs is some sort of a delayed close, as it seems that the file is considered closed for user mode, however kernel mode can still access it.Again, the case here is generally closed, but I am trying to wrap my head around as to what is happening and why, which to me is just as valuable as solving the initial problem.
Smaller picture is 2.png, larger is 1.png as it involves the cached writes which was the initial problem :)
-
You do have a lot of questions I don't have the answer to. There is one tiny bit of information I have found, though.
@A-Newbie said in Is it possible to use or reuse Windows file handle as QFile:
In the second case scenario(second post) I do the opposite, namely creating a handle with a Windows function first(CreateFile), then casting the handle to get the number - _open_osfhandle(reinterpret_cast<intptr_t>(hFile), which again returns 3 as in the first case, but I do get only one open(2.png) and never a close, despite the fact I am explicitly calling both(just in case and for testing purposes) CloseHandle() and QFile.close().
The documentation for the QFile::open() overload (https://doc.qt.io/qt-6/qfile.html#open-3) states the following:
When a QFile is opened using this function, behaviour of close() is controlled by the AutoCloseHandle flag. If AutoCloseHandle is specified, and this function succeeds, then calling close() closes the adopted handle. Otherwise, close() does not actually close the file, but only flushes it.
So, by default QFile::close() does not close the file. You need to specify the AutoCloseHandle flag for it to do something at all. Why CloseHandle() didn't work, I have no idea. Have you already investigated CreateFile()/CloseHandle() in combination without QFile in the middle (so just plain old Windows)? Does it behave in the same way?
-
-
Hey Simon,
Yes, any simple scenario like CreateFile -> CloseHandle or QFile.open() -> QFile.close() works as expected, and the file is opened, then closed. When I mix them together I never see the close happening irrespective of QFileDevice flags provided and if CloseHandle() or QFile.close() is called.
QFile.isOpened() returns false after the fact though.