QVideoProbe stop working when processing time is long

[jsulm]

@leochan2009 My guess is that you get more frames than you can process in the same time. I suggest to put your frames into a queue and take them from there for processing in threads. But you could get out of memory then.

[leochan2009]

@jsulm
Hi, thanks for the prompt reply. Yes, the frame is faster than the processing. That is why i use the QFuturewatch and QFuture (https://doc.qt.io/qt-5/qfuture.html#details). From the description, the future function runs in another thread asynchronously. I have a QMutex implemented so when there is more video frame coming, the function just return none and don't trigger any calculation. see the slot function below:

connect(videoProbe, SIGNAL(videoFrameProbed(QVideoFrame)), controller, SLOT(receiveFrameFromCamera(QVideoFrame)));
void CameraControl::receiveFrameFromCamera(QVideoFrame frame)
{
     if(!m_imageProcessingInProgress.tryLock())
     {
           return;
     }
      m_futureWatcher->setFuture(QtConcurrent::run(processVideoFrame, frame));
}

The mutex only get released when the future function is finished.
in this way, even the frame is very fast, the processing thread can work on its own pace.

void CameraControl::imageProcessingFinished()
{
     m_imageProcessingInProgress.unlock();
}

however, the futurewatcher thread still interfere the videoprobe thread.

[SGaist]

Hi,

Shouldn't you rather have a queue and drop frames when it reaches a certain size ?

How long does your frame processing take with regard to the FPS of your video ?

[leochan2009]

@SGaist ,

I have the slot function "receiveFrameFromCamera" connected to the frame signal.
my understanding is that when frame is coming, the slot function "receiveFrameFromCamera" will react to the signal.

connect(videoProbe, SIGNAL(videoFrameProbed(QVideoFrame)), controller, SLOT(receiveFrameFromCamera(QVideoFrame)));
void CameraControl::receiveFrameFromCamera(QVideoFrame frame)
{
     if(!m_imageProcessingInProgress.tryLock())
     {
           return;
     }
      m_futureWatcher->setFuture(QtConcurrent::run(processVideoFrame, frame));
}

# when processVideoFrame finished,  imageProcessingFinished will be called to release the mutex
connect(m_futureWatcher, SIGNAL(finished()), this, SLOT(imageProcessingFinished()));
void CameraControl::imageProcessingFinished()
{
    m_imageProcessingInProgress.unlock();
}

As the there is a mutex lock, so the "receiveFrameFromCamera" slot will return immediately if the image processing is still working on the previous frame. The new coming frame should be automatically dropped, isn't it? How the QVideoProbe really work? is the emitted frame from the signal "videoFrameProbed" has to be processed? say, the video probe emits signal at 30 Hz, the image processing is only at 10 Hz. Although the incoming frames is faster than the processing speed, the mutex will prevent the frame from entering the image process. See the code:

 if(!m_imageProcessingInProgress.tryLock())
     {
           return;
     }

My point is: Although the image processing function "processVideoFrame" is not able to catch up with incoming frame, but the slot function receiveFrameFromCamera will be able to catch up with the emitted signal.
There is not need to queue the frame, if the processing is working on the previous frame, the new frame should be dropped by the slot function.

[SGaist]

I see. You're reasoning should be correct. I thought you wanted to be able to process as many frames as possible without dropping but indeed with a factor three in speed reduction that would not be possible.

Does the signal emission stop quickly ?
Do you have any other activity related to the video beside your processing ?

[leochan2009]

@SGaist exactly, i am fine with dropping frames. Although the processVideoFrame function is not able to process every frame, it is fine for me. The issue i am facing is that the process in the processVideoFrame function does interfere with the videoProbe, though it is running in an a-synchronized thread using the QFuture.
in the example i wrote here, there is sleep function:

void CameraControl::processVideoFrame(QVideoFrame frame)
{
    sleep(30000);
}

when i set the sleep time to be less than 30 ms, the application runs well. however, any value larger will stop the videoprobe from emitting signal.

My application is big, but there is not other activity related to the video. I might wrote a small application to demonstrate this issue so you can run from your side.
Thanks.

Best,
Longquan

[SGaist]

We know it should be the case but did you check that the processVideoFrame method does indeed run in a different thread ?

[leochan2009]

@SGaist
I will double check if it is running in a different thread. Thanks for the suggestion

[leochan2009]

@SGaist ,
double checked the thread, the processVideoFrame function does run in a different thread; below is the output of the thread id for the functions. When i didn't add sleep() call in the processVideoFrame, the program runs well. The imageProcessingFinished function got called and released the mutex.
receiveFrameFromCamera: 0x7f919da554c0
processVideoFrame: 0x7f90ca7f4700
imageProcessingFinished: 0x7f919da554c0

When i add the sleep() line, the imageProcessingFinished never get called. I think QFuture is interfered when there is a sleep call or other process that takes too long in the processVideoFrame thread.

[leochan2009]

@SGaist
I found an exactly same issue reported 3 years ago and you commented on the issue as well. the issue is unsolved
https://forum.qt.io/topic/85789/qfuturewatcher-s-signal-finished-does-not-get-emitted

[SGaist]

Would you be able to provide a minimal compilable example that shows this ?

In between, the explicit queue might be worth a try to allow you to go further.