Looking for example of how to use QVideoFrame with multiple plane pixel format (specifically P016 or P216 or NV16)
-
I am writing an app that receives NDI video frames from the network, writes them to a
QVideoFrame
, and thenQVideoSink
s them to aQVideoWidget
.https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/
Things work fine if I tell NDI to send me
NDIlib_recv_color_format_UYVY_BGRA
, which I process asQVideoFrameFormat::PixelFormat::Format_UYVY
and it renders correctly on the screen with an expected small loss of quality.When I tell NDI to send me
NDIlib_recv_color_format_best
, they send meNDIlib_FourCC_video_type_P216
which I process asQVideoFrameFormat::PixelFormat::Format_P016
.
P216 is a semi-planar format with 2 planes.
It is described as:// YCbCr color space using 4:2:2 in 16bpp. // In memory this is a semi-planar format. This is identical to a 16bpp version of the NV16 format. // The first buffer is a 16bpp luminance buffer. // Immediately after this is an interleaved buffer of 16bpp Cb, Cr pairs.
See also:
- https://wiki.videolan.org/YUV#Other_NV_formats
- https://chromium.googlesource.com/libyuv/libyuv/+/HEAD/docs/formats.md#nv12-and-nv21
- https://doc.qt.io/qt-6/qvideoframeformat.html#PixelFormat-enum
- https://en.wikipedia.org/wiki/FFmpeg#Pixel_formats
- https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats#422-formats
I have found zero Qt examples or Qt open source projects showing how to use QVideoFrame with multiple planes.
I wrote it how I think I understand it should go at:
https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/ndireceiverworker.cpp#L291
Problem with my code is that it crashes with either
heap corruption
orwrite access violation
just shortly after halfway throughmemcpy
ing the 2nd plane:https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/ndireceiverworker.cpp#L353
Before someone responds "You are obviously corrupting your heap or memory", the 2nd plane should be a contiguous 5,529,600 bytes, but I am crashing after writing only 2,858,994 bytes to it.
I think my problem has something to do with the
QVideoFrame::bits(int plane)
mapped buffers that I must be using wrong, but I am doing the best I know how considering I have been unable find any examples of how to use multiple planes.Or maybe I am incorrectly using P016 for a P216 pixel format?
If I write less than 2,858,995 bytes of the 2nd plane's 5,529,600 bytes the code no longer crashes and proves that it is mostly working, but it is clearly an incomplete raster:
Notice how the 2nd plane's U and V [color] components stretch down the screen because the write was incomplete.
If I write less bytes (ex: less thanbytesInPlane / 2
), it gets droopier.
It should look like this:So, how does one correctly write to multi-planar or semi-planar video frames and prevent it from corrupting the heap/memory?
Thanks!
-
I have learned two things:
- The crash is solved by changing:
memcpy(videoFrame.bits(plane), pNdiVideoFrame->p_data, bytesInPlane);
to
memcpy(videoFrame.bits(plane), pNdiVideoFrame->p_data, videoFrame.mappedBytes(planeNum));
- Format P016's layout is https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats#p016-and-p010:
Format P216's layout is https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats#p216-and-p210:
ThatH/2
explains the phenomenon that I am experiencing.
I need to either:
- trick the existing
QVideoFrame
code in to using P216
-or- - convert P216 (4:2:2) to P016 (4:2:0)...which I would prefer to not have to do.
-or- - implement my own P216 processing.
-
Hi,
I would add solution number 4: add support for P216 to QVideoFrame.
It would be a nice longterm option.
As for your current issue, you would indeed need to reprocess the YV part to fit P016.
-
@SGaist said in Looking for example of how to use QVideoFrame wiith multiple plane pixel format (specifically P016 or P216 or NV16):
Hi,
I would add solution number 3: add support for P216 to QVideoFrame.
It would be a nice longterm option.I already added to my followup and was about to post a new question specific to P216.
My OP was on the correct/recommended way to process multi/semi-planar pixel formats.
Think my solution is close? -
My number was wrong !
Number three might be the quickest to implement but likely not the most efficient.
Hence my number 4 suggestion.
-
I don't have it at hand but looking for color space conversion should give you some good results.
-
To follow up, I have temporarily mitigated the P216 -> P016 issue by using this code to scan every other UV line:
It still leaves a few small artifacts laying around, but it is passable while I start the conversation of getting P216 added to Qt.
Thanks!
Pv