RaspberryPI3 too slow thread with opencv algorithm

[davidino]

Goodmorning to all,
I'm trying to use ORB algorithm to detect an object in a room using a camera.
I'm already using a different thread with the purpose of detaching the GUI thread from the the one where I do calculation, as I reported in one of my previous topic "Custom Qlabel show videocamera on different thread".
Unfortunately, with this algorithm the CPU usage is more than 100%.

How do I know if the thread used for calculation is actually using a different processor, or it use the same as GUI thread? Is it possible to chooses which processor to use for one thread?
Since I saw that the most demanding part is when I detect and compute the keypoints and descriptors (code reported below), do you suggest to create another thread only for it?

vector<KeyPoint> sceneKeypoints, objectKeyPoints;
Mat sceneDescriptors, objectDescriptors;
Mat objectImg = imread("/home/root/objectToDetect.png", IMREAD_GRAYSCALE);

orb = ORB::create();
                
// EXTRACT KEYPOINTS AND DESCRIPTORS
orb->detect(objectImg, objectKeypoints);
orb->compute(objectImg, objectKeypoints, objectDescriptors);   
orb->detect(sceneImg, sceneKeypoints);
orb->compute(sceneImg, sceneKeypoints, sceneDescriptors);

Thank you very much.
Davidino

[mrdebug]

Hi. I have a situation similar to yours.

In order to know how much cpu each thread are using you can use top tool and press H. In this mode you will have the processes sorted by thread with the right thread name.

If you are using opencv maybe you are grabbing images from a local camera. This could be cpu expensive.

I'm working with an rtsp camera using ffmpeg to grap it. This is not cpu expensive. The grabbing is done by a dedicated non gui thread.
I have a second thread that (by using opencv) evaluate the frame.
The first thread uses the second thread only if the second thread is availabe to evaluate it.
In this way I have a good real time live video but the thread that are using opencv often drops frames.

[Pablo J. Rogina]

@davidino in addition to the good reply from @mrdebug, and although you're working with C++ (from your code snippet) you may want to take a look at this post, related to optimizing OpenCV and Python on the Raspberry Pi. Hopefully some hints may help you improve the performance of your environment.

[davidino]

Thank you both for your answer.
I worked something more and the performance has been improved a little bit. Before the frame rate was about 0.15 FPS!, now it's 1-1.8 FPS. Without the ORB algorithm, but simply grabbing the image from camera I can reach up to 40-45 FPS.
One of the thing that I changed was moving the detectAndCompute of the target image on the thread initialization (after all it's has to be done only one time, not continuously).

Hello mkdebug,
Regarding the top+H, the situation is shown below.

Yes, I'm using a usb camera, in fact I don't expect to reach the same frame rate as the pi camera, but I'd like to know as much as I can push to obtain the maximum.
Your solution is interesting, I'm trying to do the same as you. Thank you.

Hello, Oablo J. Rogina,
I saw the site that you suggest me, and it's very interesting, especially the deep learning part, unfortunately my OS image is made with yocto, and, at least for the moment, I cannot install programs as freely as with a standard image with apt-get command. But I'll keep it in mind, thank you.

[mrdebug]

Hi. In order to not make your cpu eaten by opencv you should insert a sleep step after every image. You should calculate the frame rate, for example 5 fpc could be enough.

[davidino]

Hello mrdebug,
thank you very much for your suggestion. I've done a second Qthread for calculation. so the cycle is the following.

• Read cam_image with one thread every QTimer event (200ms). I've inserted a QThread::msleep(100) in the first thread, for letting the second thread to work, as you suggested.
• Pass the the Mat data to the second thread for calculating ORB algorithm
• Pass back the Mat, where the algorithm has been applied, to the first thread.
• Convert Mat file to image and pass the QIMage to GUI thread.

The frame rate got an increasement up to 9 FPS (which is much more than before!).
For passing Mat object among threads, I had to register Mat type as follow:

....
Q_DECLARE_METATYPE(Mat)

int main(int argc, char *argv[])
{
    qmlRegisterType<VideoCamera>("VideocameraLib", 1, 0, "Videocamera");
    qRegisterMetaType<Mat>("Mat");
....

Concerning the CPU usage, the situation didn't change a lot, although the second thread and msleep.

If anyone could take a look at my other problem, I would really appreciate it:
https://forum.qt.io/topic/87039/pass-oncurrentindexchanged-event-to-an-object-in-page1/2

Thank you again.
Davidino