Problems with Open-Source Downloads read https://www.qt.io/blog/problem-with-open-source-downloads and https://forum.qt.io/post/638946

QVideoProbe stop working when processing time is long



  • Hi I am working with QVideoProbe to emit the QVideoframe to other class for further processing, here is snippet of my code:

    videoProbe = new QVideoProbe;
    QScopedPointer<QCamera> camera;
    camera.reset(new QCamera(cameraInfo));
    CameraControl*  controller = new CameraControl() # CameraControl is a defined class to process the incoming frame.
    videoProbe->setSource(camera.data());
    
    connect(videoProbe, SIGNAL(videoFrameProbed(QVideoFrame)), controller, SLOT(receiveFrameFromCamera(QVideoFrame)));
    

    below is the implementation of CameraControl, i have put the processing into a FutureWatch class so they can run asynchronously. The incoming frame rate is 30HZ. when i set the sleep to 30000us, the program runs ok. however, if i increase the sleep time to 300000us, the videoprobe stop sending signals.

    void CameraControl::processVideoFrame(QVideoFrame frame)
    {
        sleep(30000);
    }
    
    void CameraControl::receiveFrameFromCamera(QVideoFrame frame)
    {
        if(!m_imageProcessingInProgress.tryLock())
        {
            return;
        }
        m_futureWatcher->setFuture(QtConcurrent::run(processVideoFrame, frame));
    }
    
    void CameraControl::imageProcessingFinished()
    {
        m_imageProcessingInProgress.unlock();
    }
    
    

    Any suggestions? thanks


  • Lifetime Qt Champion

    @leochan2009 My guess is that you get more frames than you can process in the same time. I suggest to put your frames into a queue and take them from there for processing in threads. But you could get out of memory then.



  • @jsulm
    Hi, thanks for the prompt reply. Yes, the frame is faster than the processing. That is why i use the QFuturewatch and QFuture (https://doc.qt.io/qt-5/qfuture.html#details). From the description, the future function runs in another thread asynchronously. I have a QMutex implemented so when there is more video frame coming, the function just return none and don't trigger any calculation. see the slot function below:

    connect(videoProbe, SIGNAL(videoFrameProbed(QVideoFrame)), controller, SLOT(receiveFrameFromCamera(QVideoFrame)));
    void CameraControl::receiveFrameFromCamera(QVideoFrame frame)
    {
         if(!m_imageProcessingInProgress.tryLock())
         {
               return;
         }
          m_futureWatcher->setFuture(QtConcurrent::run(processVideoFrame, frame));
    }
    

    The mutex only get released when the future function is finished.
    in this way, even the frame is very fast, the processing thread can work on its own pace.

    void CameraControl::imageProcessingFinished()
    {
         m_imageProcessingInProgress.unlock();
    }
    

    however, the futurewatcher thread still interfere the videoprobe thread.


  • Lifetime Qt Champion

    Hi,

    Shouldn't you rather have a queue and drop frames when it reaches a certain size ?

    How long does your frame processing take with regard to the FPS of your video ?



  • @SGaist ,

    I have the slot function "receiveFrameFromCamera" connected to the frame signal.
    my understanding is that when frame is coming, the slot function "receiveFrameFromCamera" will react to the signal.

    connect(videoProbe, SIGNAL(videoFrameProbed(QVideoFrame)), controller, SLOT(receiveFrameFromCamera(QVideoFrame)));
    void CameraControl::receiveFrameFromCamera(QVideoFrame frame)
    {
         if(!m_imageProcessingInProgress.tryLock())
         {
               return;
         }
          m_futureWatcher->setFuture(QtConcurrent::run(processVideoFrame, frame));
    }
    
    # when processVideoFrame finished,  imageProcessingFinished will be called to release the mutex
    connect(m_futureWatcher, SIGNAL(finished()), this, SLOT(imageProcessingFinished()));
    void CameraControl::imageProcessingFinished()
    {
        m_imageProcessingInProgress.unlock();
    }
    
    

    As the there is a mutex lock, so the "receiveFrameFromCamera" slot will return immediately if the image processing is still working on the previous frame. The new coming frame should be automatically dropped, isn't it? How the QVideoProbe really work? is the emitted frame from the signal "videoFrameProbed" has to be processed? say, the video probe emits signal at 30 Hz, the image processing is only at 10 Hz. Although the incoming frames is faster than the processing speed, the mutex will prevent the frame from entering the image process. See the code:

     if(!m_imageProcessingInProgress.tryLock())
         {
               return;
         }
    

    My point is: Although the image processing function "processVideoFrame" is not able to catch up with incoming frame, but the slot function receiveFrameFromCamera will be able to catch up with the emitted signal.
    There is not need to queue the frame, if the processing is working on the previous frame, the new frame should be dropped by the slot function.


  • Lifetime Qt Champion

    I see. You're reasoning should be correct. I thought you wanted to be able to process as many frames as possible without dropping but indeed with a factor three in speed reduction that would not be possible.

    Does the signal emission stop quickly ?
    Do you have any other activity related to the video beside your processing ?



  • @SGaist exactly, i am fine with dropping frames. Although the processVideoFrame function is not able to process every frame, it is fine for me. The issue i am facing is that the process in the processVideoFrame function does interfere with the videoProbe, though it is running in an a-synchronized thread using the QFuture.
    in the example i wrote here, there is sleep function:

    void CameraControl::processVideoFrame(QVideoFrame frame)
    {
        sleep(30000);
    }
    

    when i set the sleep time to be less than 30 ms, the application runs well. however, any value larger will stop the videoprobe from emitting signal.

    My application is big, but there is not other activity related to the video. I might wrote a small application to demonstrate this issue so you can run from your side.
    Thanks.

    Best,
    Longquan


  • Lifetime Qt Champion

    We know it should be the case but did you check that the processVideoFrame method does indeed run in a different thread ?


Log in to reply