Accessing named pipe (FIFO) on Linux as regular file

[KejPi]

Hi there, I want to read binary stream from named pipe (FIFO) like from regular file. It works fine except the case when it try to open it and there is no sender of data connected.

    if (!m_inputFile->open(QIODevice::ReadOnly))
    {
        qCCritical(rawFileInput) << "RAW-FILE: Unable to open file: " << m_fileName;
        delete m_inputFile;
        m_inputFile = nullptr;

        return false;
    }

In such case open() call never finishes. When sender connects, open() finishes successfully. I would like to avoid the deadlock and not to open if I cannot open (or set some timeout). Is there a way to do it? Or is there a better approach how to access named pipes on Linux?

Rationale behind: there are some tools capable of generating data to file. I want to read these data "on the fly" and using name pipe treating it as file from my app and from other tool seems to be elegant way how to exchange data.

[ankou29666]

I see nothing in QIODevice::open() that could help.

Question : do you delete the fifo after use ? or reuse the previous one everytime ?

[SGaist]

Hi,

Beside @ankou29666's good questions, which OS are you on ?

[KejPi]

I am reusing the FIFO and I guess users do too. I am developing and testing it on macOS but I assume Linux behaves the same way, but I can try.

[JonB]

@KejPi
I have looked into this. I do not see you will be able to achieve this with QFile.

First, based on https://man7.org/linux/man-pages/man2/open.2.html and https://man7.org/linux/man-pages/man7/fifo.7.html I do not believe Linux offers the possibility to "attempt" to open a FIFO file for read but fail if it is not presently open for write elsewhere. Nor to have open(2) timeout. Then per fifo(7): The normal behaviour of opening a FIFO for read is to block awaiting the writer to open. Opening for read and write might succeed, you might try that. Or by passing O_NONBLOCK you may be able to force the open for read to succeed but not wait. In either case the FIFO will be opened rather than not opening and returning failure as you would like. But maybe you could live with that.

But second there does not seem to be any way either to pass a flag like O_NONBLOCK to QFile::open() or to create a QFile from an already opened file descriptor. So the foregoing is all moot.

I stand to be corrected on this Linux behaviour if anyone knows better.

In summary I think you could try opening the named pipe QIODeviceBase::ReadWrite to see whether that returns without blocking, else I think you are in trouble.

[ankou29666]

I don't know what's your possibilities on the writer side, but, given what @JonB said, if this is ever possible for you, what I would do is

• create a new fifo every time before use from writer side
• delete it every time after use after use from reader side

such that open fails without blocking if you attempt to open a fifo that hasn't been created (instead of blocking). All you need is to handle that error for retry.

[KejPi]

The problem is that FIFO is actually handled by users trying to provide it to application instead of regular file. I understand it is not possible to handle it in a clean way using it as file, but what are other options?
To be more specific about the use-case, users us typically this:

odr-dabmod -a 0.6 -f fifo -F u8 input.eti

This command streams data to fifo file that is named pipe. And then they start the application and open FIFO as file input and play from it. The problem I am trying to solve is that my app remembers the last file and it tries to open it on start. This is no problem for normal file even if it does not exist but in case of FIFO without sender application application hangs trying to open it. I want to detect this case and report it as error, but other solutions to handle the use case are welcome.

[JonB]

@KejPi
So you tried QIODeviceBase::ReadWrite and that did not help/still blocked?

I already suggested that, under Linux at least, you will not be able to generate an error/failure if the other process is no longer using/writing to a fifo but left it in existence anyway. At best you will be able to open it without blocking, and you won't then know that the "sender" has gone away, other than that nothing will arrive on the pipe.

[KejPi]

It blocks too.

[JonB]

@KejPi
Shame, I see quite a few web posts indicating that opening a FIFO O_RDWR succeeds where O_RDONLY blocks.

Then I think you will need to (try to) open the FIFO in a secondary thread. Either the open blocks, and hence the thread but not anything else, or it succeeds/completes, at which point you signal that has happened. You'll have check it's OK to pass/access the QFile to the main thread after the open() call, I don't know whether there are any issues but I would have thought it's OK, unless someone says not?

[JonB]

@KejPi said in Accessing named pipe (FIFO) on Linux as regular file:

It blocks too.

Did you really try this/the right thing? If so what exact platform are you testing on?

I am Ubuntu 24.04. I created the named pipe via mkfifo fifofile. Then

QFile f("fifofile");
qDebug() << f.open(QIODevice::ReadOnly);

As you report, I confirm that this "hangs" (blocks) as expected. (Until another process opens the fifo file for write, e.g. echo hello > fifofile.) Then I change to

qDebug() << f.open(QIODevice::ReadWrite);

And as I suggested and expected this returns immediately with true.

I based this expectation on the https://man7.org/linux/man-pages/man7/fifo.7.html I referenced:

The FIFO must be opened on both ends (reading and writing) before data can
be passed. Normally, opening the FIFO blocks until the other end
is opened also.

A process can open a FIFO in nonblocking mode. In this case,
opening for read-only succeeds even if no one has opened on the
write side yet and opening for write-only fails with ENXIO (no
such device or address) unless the other end has already been
opened.

Under Linux, opening a FIFO for read and write will succeed both
in blocking and nonblocking mode. POSIX leaves this behavior
undefined. This can be used to open a FIFO for writing while
there are no readers available

The second paragraph tells us that for open read-only we would need nonblocking mode (O_NONBLOCK flag to low-level open()), which we cannot do with Qt QFile. But note the last paragraph. That tells us that even without this, in blocking mode, opening for read-write should succeed/not block under Linux. It should equally apply to "open a FIFO for reading while there are no writers available". So I expect it to work for you.

Of course, I have no idea whether that would apply under macOS. macOS != Linux. Ah, that is what you must be testing it on, and macOS is not behaving like Linux. So your situation must be worse than under Linux....

[JonB]

@KejPi
OK, I have spent quite a lot of time investigating this!

If you only need it to work on "proper" Linux opening QIODeviceBase::ReadWrite should return without blocking.

But that will (apparently from what you report) leave you unable to test/use it under MacOS. I played to find a solution assuming ReadWrite blocks just as much as ReadOnly.

It's less of a problem if you are prepared to do all your named pipe operations in their own thread. Then it never blocks the main thread. And it just never gets passed the open() block if there is no writer attached, and never does any reading. being blocked on an open() means the thread can never exit normally, it has to be terminate()d, which is not a great idea. And in any case I am suspecting you want to handle any reading from the pipe in your main thread, so this may not eb a starter.

Then the only solution I can find is to have a secondary thread open the named pipe for ReadWrite (or Append) and the main thread open it for read-only. I find this does work under Linux. Essentially, the fact that we have opened it for write in the thread allows main to open it for read without blocking. The secondary thread can close its write handle and then exit. It does not matter which order the two opens are performed in.

I have tested this with a "proof of concept", overriding QThread::run(). But to do it "properly" I would need to change it into the worker-thread-with-signals paradigm per the first example at https://doc.qt.io/qt-6/qthread.html#details. This is a bit of work. I would be prepared to do it for you if you truly intend to use it, but it's a chore if you decide you would not adopt this approach. Which I would understand as it's rather "hacky"/"hokey" and you may not want such a solution/workaround.

[Or, I have just realised I can do all of this in just a couple of lines and no secondary thread if your MacOS does not block on opening named pipe with O_NONBLOCK flag. But it would still be the same approach of doing a "dummy" open for write and then close so as to allow the QFile::open(QIODeviceBase::ReadOnly) not to block.]

If you do wish to pursue, please start by testing the following on your MacOS:

mkfifo fifofile

echo 'Hello world' >> fifofile &
cat < fifofile

cat < fifofile &
echo 'Goodbye world' >> fifofile

Do both of these cat commands succeed, outputting the other side's string, without blocking, on your MacOS?

[KejPi]

I am sorry for late reply. Of course you are right about QIODeviceBase::ReadWrite It is non blocking on both Linux and macOS. I have actually fell into a trap that after opening it I tried to read:

QDataStream in(m_inputFile);
in.readRawData(&ch, 1)

This actually blocks until there is some sender on the other side. I have changed my code in the way that I am trying to read only if (m_inputFile->size() > 0).
Unfortunately, I am facing an issue that I cannot recognise if I can read from FIFO. I am trying to use waitForReadyRead but this returns false even in case when sender is connected. If I try to read and sender is not connected then the app is stuck.

[JonB]

@KejPi said in Accessing named pipe (FIFO) on Linux as regular file:

I cannot recognise if I can read from FIFO. I am trying to use waitForReadyRead but this returns false even in case when sender is connected. If I try to read and sender is not connected then the app is stuck.

Yes this is correct. Your reads will block, even if you got your open to work. Which is why all your handling of this FIFO should probably be in a thread.

[KejPi]

Yes, I am doing that with watchdog functionality. But I do not like I need to terminate the thread to recover. Probably not the cleanest approach but it seems to work. I am making the issue as solved. Thank you for your help!

[JonB]

@KejPi said in Accessing named pipe (FIFO) on Linux as regular file:

But I do not like I need to terminate the thread to recover. Probably not the cleanest approach but it seems to work.

I agree. Easiest. If you really care open the named pipe and write to it in e.g. the main thread before you exit. That should unblock the the other thread's read. It's also messy/hacky. Once you had managed to open the file you could have used poll() or non-blocking reads, but that would not be with Qt code.