Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. QVideoFrame from custom buffer instead of memcpy()
Forum Updated to NodeBB v4.3 + New Features

QVideoFrame from custom buffer instead of memcpy()

Scheduled Pinned Locked Moved Unsolved General and Desktop
11 Posts 4 Posters 1.3k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M Offline
    M Offline
    malikcis
    wrote on 20 Jun 2023, 14:34 last edited by SGaist
    #1

    Hi,
    I created my own video sink inherited from QVideoSink in Qt 6. And I want to show content of this sink in QML side.
    For this, I copy a custom buffer to a QVideoFrame like this:

    void Producer::sendImageToQML(uchar *pData)
    {    
        QVideoFrame video_frame(QVideoFrameFormat(QSize(3840, 2160),QVideoFrameFormat::Format_YUYV));
        if(!video_frame.map(QVideoFrame::WriteOnly))
        {
            qWarning() << "QVideoFrame is not writable";
            return;
        }    
        **memcpy(video_frame.bits(0), pData, 16588800);**
        video_frame.unmap();
        m_videoSink->setVideoFrame(video_frame);
    }
    

    This works fine but CPU usage is high because of the memcpy operation.
    Is there a way to generate a QVideoFrame from a custom buffer pointer?
    Thx,
    PS: the rest of the code is prety much like this one.
    Malikcis

    1 Reply Last reply
    0
    • S Offline
      S Offline
      SGaist
      Lifetime Qt Champion
      wrote on 20 Jun 2023, 20:22 last edited by
      #2

      Hi,

      You would have a copy operation in any case.

      What is your data source ?

      One thing to take into account, you are moving 4K images around, this has a cost.

      Interested in AI ? www.idiap.ch
      Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

      M 1 Reply Last reply 21 Jun 2023, 08:12
      1
      • S SGaist
        20 Jun 2023, 20:22

        Hi,

        You would have a copy operation in any case.

        What is your data source ?

        One thing to take into account, you are moving 4K images around, this has a cost.

        M Offline
        M Offline
        malikcis
        wrote on 21 Jun 2023, 08:12 last edited by
        #3

        @SGaist Hi Samuel,
        I really appreciate your feedback.

        "What is your data source ?"
        My data source is an embedded CSI camera that I access via native Gstreamer "appsink" API. This is needed because I need to dynamically configure Gstreamer. Therefore I cannot use QT Mediaplayer or FFMPEG.

        "You would have a copy operation in any case."
        Unfortunately I don't agree with you here because I actually already managed to demonstrate custom pointer to QVideoFrame conversion in QT5 using QAbstractVideoBuffer (Unfortunately, this is not available in QT6 anymore).
        Here is the main code:

        int Gstpipeline::sendNewSample2QML(uchar *pData)
        {
            //gsize size;
            int video_with;
            int video_height;
            int video_size;
            int bytes_per_line;
        
            video_with = 480;
            video_height = 800;
        
            video_size = video_with * video_height * 4;
            bytes_per_line = video_with * 4;
        
            VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.
        
            emit newFrameAvailable(QVideoFrame(vbuffer, QSize(video_with, video_height), QVideoFrame::Format_BGR32));
        
            return 0;
        }
        

        I subclassed QAbstractVideoBuffer as can be seen here:

        //videoBuffer.h
        #ifndef VIDEOBUFFER_H
        #define VIDEOBUFFER_H
        
        #include <QAbstractVideoBuffer>
        
        class VideoBuffer : public QAbstractVideoBuffer
        {
        public:
            VideoBuffer(uchar *data, int numBytes, int bytesPerLine);
        
            virtual uchar *map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                               int *bytesPerLine) override;
        
            virtual QAbstractVideoBuffer::MapMode mapMode() const override;
        
            virtual void unmap() override;
        
            void setMap(uchar *data, int numBytes, int bytesPerLine) {
                m_data = data;
                m_numBytes = numBytes;
                m_bytesPerLine = bytesPerLine;
            }
        
        private:
            uchar *m_data;
            int m_numBytes;
            int m_bytesPerLine;
        };
        
        #endif//VIDEOBUFFER_H
        
        //############################
        //videoBuffer.cpp
        #include "videoBuffer.h"
        #include <QDebug>
        
        VideoBuffer::VideoBuffer(uchar *data, int numBytes, int bytesPerLine)
            : QAbstractVideoBuffer(QAbstractVideoBuffer::NoHandle)
            , m_data(data)
            , m_numBytes(numBytes)
            , m_bytesPerLine(bytesPerLine)
        {
        
        }
        
        uchar *VideoBuffer::map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                                int *bytesPerLine)
        {
            Q_UNUSED(mode)
        
            *numBytes = m_numBytes;
            *bytesPerLine = m_bytesPerLine;
        
            return m_data;
        }
        
        QAbstractVideoBuffer::MapMode VideoBuffer::mapMode() const
        {
            return QAbstractVideoBuffer::ReadOnly;
        }
        
        void VideoBuffer::unmap()
        {
           //delete m_data;
        }
        
        

        "One thing to take into account, you are moving 4K images around, this has a cost."
        Note that only 480x800 pixel image is used here. When using memcpy on this small images, my CPU (NXP imx8mp) goes high up to 80% whereas with above code I only use 7%. This is the reason why I need alternative solution.

        I was thinking of subclassing QVideoFrame in the same way as in the code above but I am not sure on how to do that. One will probably need to do a fake "mapping" that always uses the custom buffer.
        Any hints or ideas will be welcome.
        Thx, Malikcis

        B 1 Reply Last reply 21 Jun 2023, 08:44
        0
        • M malikcis
          21 Jun 2023, 08:12

          @SGaist Hi Samuel,
          I really appreciate your feedback.

          "What is your data source ?"
          My data source is an embedded CSI camera that I access via native Gstreamer "appsink" API. This is needed because I need to dynamically configure Gstreamer. Therefore I cannot use QT Mediaplayer or FFMPEG.

          "You would have a copy operation in any case."
          Unfortunately I don't agree with you here because I actually already managed to demonstrate custom pointer to QVideoFrame conversion in QT5 using QAbstractVideoBuffer (Unfortunately, this is not available in QT6 anymore).
          Here is the main code:

          int Gstpipeline::sendNewSample2QML(uchar *pData)
          {
              //gsize size;
              int video_with;
              int video_height;
              int video_size;
              int bytes_per_line;
          
              video_with = 480;
              video_height = 800;
          
              video_size = video_with * video_height * 4;
              bytes_per_line = video_with * 4;
          
              VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.
          
              emit newFrameAvailable(QVideoFrame(vbuffer, QSize(video_with, video_height), QVideoFrame::Format_BGR32));
          
              return 0;
          }
          

          I subclassed QAbstractVideoBuffer as can be seen here:

          //videoBuffer.h
          #ifndef VIDEOBUFFER_H
          #define VIDEOBUFFER_H
          
          #include <QAbstractVideoBuffer>
          
          class VideoBuffer : public QAbstractVideoBuffer
          {
          public:
              VideoBuffer(uchar *data, int numBytes, int bytesPerLine);
          
              virtual uchar *map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                                 int *bytesPerLine) override;
          
              virtual QAbstractVideoBuffer::MapMode mapMode() const override;
          
              virtual void unmap() override;
          
              void setMap(uchar *data, int numBytes, int bytesPerLine) {
                  m_data = data;
                  m_numBytes = numBytes;
                  m_bytesPerLine = bytesPerLine;
              }
          
          private:
              uchar *m_data;
              int m_numBytes;
              int m_bytesPerLine;
          };
          
          #endif//VIDEOBUFFER_H
          
          //############################
          //videoBuffer.cpp
          #include "videoBuffer.h"
          #include <QDebug>
          
          VideoBuffer::VideoBuffer(uchar *data, int numBytes, int bytesPerLine)
              : QAbstractVideoBuffer(QAbstractVideoBuffer::NoHandle)
              , m_data(data)
              , m_numBytes(numBytes)
              , m_bytesPerLine(bytesPerLine)
          {
          
          }
          
          uchar *VideoBuffer::map(QAbstractVideoBuffer::MapMode mode, int *numBytes,
                                  int *bytesPerLine)
          {
              Q_UNUSED(mode)
          
              *numBytes = m_numBytes;
              *bytesPerLine = m_bytesPerLine;
          
              return m_data;
          }
          
          QAbstractVideoBuffer::MapMode VideoBuffer::mapMode() const
          {
              return QAbstractVideoBuffer::ReadOnly;
          }
          
          void VideoBuffer::unmap()
          {
             //delete m_data;
          }
          
          

          "One thing to take into account, you are moving 4K images around, this has a cost."
          Note that only 480x800 pixel image is used here. When using memcpy on this small images, my CPU (NXP imx8mp) goes high up to 80% whereas with above code I only use 7%. This is the reason why I need alternative solution.

          I was thinking of subclassing QVideoFrame in the same way as in the code above but I am not sure on how to do that. One will probably need to do a fake "mapping" that always uses the custom buffer.
          Any hints or ideas will be welcome.
          Thx, Malikcis

          B Offline
          B Offline
          Bonnie
          wrote on 21 Jun 2023, 08:44 last edited by
          #4

          @malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.
          I've answered in https://forum.qt.io/post/749829 about how to use QAbstractVideoBuffer in Qt6, maybe you can see if that helps.

          M 2 Replies Last reply 21 Jun 2023, 09:21
          1
          • B Bonnie
            21 Jun 2023, 08:44

            @malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.
            I've answered in https://forum.qt.io/post/749829 about how to use QAbstractVideoBuffer in Qt6, maybe you can see if that helps.

            M Offline
            M Offline
            malikcis
            wrote on 21 Jun 2023, 09:21 last edited by
            #5

            @Bonnie said in QVideoFrame from custom buffer instead of memcpy():

            @malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.

            Hi Bonnie,
            Thank for this swift feedback. This is very exciting. I am going to read your post right now.

            1 Reply Last reply
            0
            • B Bonnie
              21 Jun 2023, 08:44

              @malikcis According to qvideoframe.h, there is still a public constructor function using QAbstractVideoBuffer but commented as "internal" and also undocumented since QAbstractVideoBuffer is now in private headers.
              I've answered in https://forum.qt.io/post/749829 about how to use QAbstractVideoBuffer in Qt6, maybe you can see if that helps.

              M Offline
              M Offline
              malikcis
              wrote on 22 Jun 2023, 16:10 last edited by malikcis
              #6

              @Bonnie Hi Bonnie,
              I could implement your suggestion in https://forum.qt.io/post/749829 and I get a correct video. However my CPU usage is 100% even though QVideoFrame only reuses video pointer. I think QVideoFrame will do internal memcpy.
              I put the entire project on github

              As you can see in the code on github, I subclassed QAbstractVideoBuffer pretty much in the same way as in previously committed code and reused pData buffer:

               VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.
                  m_videoSink->setVideoFrame(QVideoFrame(vbuffer, QVideoFrameFormat(QSize(video_with, video_height),QVideoFrameFormat::Format_RGBX8888)));
              

              Am I missing something?
              THX

              B 1 Reply Last reply 26 Jun 2023, 02:11
              0
              • M malikcis
                22 Jun 2023, 16:10

                @Bonnie Hi Bonnie,
                I could implement your suggestion in https://forum.qt.io/post/749829 and I get a correct video. However my CPU usage is 100% even though QVideoFrame only reuses video pointer. I think QVideoFrame will do internal memcpy.
                I put the entire project on github

                As you can see in the code on github, I subclassed QAbstractVideoBuffer pretty much in the same way as in previously committed code and reused pData buffer:

                 VideoBuffer *vbuffer = new VideoBuffer(pData, video_size, bytes_per_line);//there is lots of magic going on here. Unfortunatelly we cannot easilly pass simple pointer.
                    m_videoSink->setVideoFrame(QVideoFrame(vbuffer, QVideoFrameFormat(QSize(video_with, video_height),QVideoFrameFormat::Format_RGBX8888)));
                

                Am I missing something?
                THX

                B Offline
                B Offline
                Bonnie
                wrote on 26 Jun 2023, 02:11 last edited by
                #7

                @malikcis Sorry I haven't digged into that so much, you could compare the source code of Qt5 and Qt6 to find the difference.

                M 1 Reply Last reply 26 Jun 2023, 13:10
                0
                • B Bonnie
                  26 Jun 2023, 02:11

                  @malikcis Sorry I haven't digged into that so much, you could compare the source code of Qt5 and Qt6 to find the difference.

                  M Offline
                  M Offline
                  malikcis
                  wrote on 26 Jun 2023, 13:10 last edited by
                  #8

                  @Bonnie Thank you,
                  I indeed compared the code and I see that some internal QT code changed in QT6.
                  Also it seems memory copy is done internally. Anyway, I will continue investigations.

                  R 1 Reply Last reply 29 Dec 2023, 14:38
                  0
                  • M malikcis
                    26 Jun 2023, 13:10

                    @Bonnie Thank you,
                    I indeed compared the code and I see that some internal QT code changed in QT6.
                    Also it seems memory copy is done internally. Anyway, I will continue investigations.

                    R Offline
                    R Offline
                    Riptide
                    wrote on 29 Dec 2023, 14:38 last edited by
                    #9

                    Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6

                    M 2 Replies Last reply 10 Jan 2024, 11:39
                    0
                    • R Riptide
                      29 Dec 2023, 14:38

                      Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6

                      M Offline
                      M Offline
                      malikcis
                      wrote on 10 Jan 2024, 11:39 last edited by
                      #10

                      @Riptide
                      I am still using memcpy which consumes significant CPU resources.
                      I don't have alternative solution yet.

                      1 Reply Last reply
                      0
                      • R Riptide
                        29 Dec 2023, 14:38

                        Hello @malikcis, do you found some solution to this topic? I have the same problem with qt6

                        M Offline
                        M Offline
                        malikcis
                        wrote on 9 Oct 2024, 14:20 last edited by
                        #11

                        @Riptide
                        Hi,
                        Did you manage to solve this issue?
                        I am still looking for better solutions.

                        One solution I would like to explore now has recently been suggested here:
                        https://community.nxp.com/t5/i-MX-Graphics-Knowledge-Base/QT6-Qml-embedded-video-play-for-weston-with-Gstreamer/ta-p/1825985

                        1 Reply Last reply
                        0

                        • Login

                        • Login or register to search.
                        • First post
                          Last post
                        0
                        • Categories
                        • Recent
                        • Tags
                        • Popular
                        • Users
                        • Groups
                        • Search
                        • Get Qt Extensions
                        • Unsolved