Looking for example of how to use QVideoFrame with multiple plane pixel format (specifically P016 or P216 or NV16)

paulpv

I am writing an app that receives NDI video frames from the network, writes them to a QVideoFrame, and then QVideoSinks them to a QVideoWidget.

https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/

Things work fine if I tell NDI to send me NDIlib_recv_color_format_UYVY_BGRA, which I process as QVideoFrameFormat::PixelFormat::Format_UYVY and it renders correctly on the screen with an expected small loss of quality.

When I tell NDI to send me NDIlib_recv_color_format_best, they send me NDIlib_FourCC_video_type_P216 which I process as QVideoFrameFormat::PixelFormat::Format_P016.
P216 is a semi-planar format with 2 planes.
It is described as:

	// YCbCr color space using 4:2:2 in 16bpp.
	// In memory this is a semi-planar format. This is identical to a 16bpp version of the NV16 format.
	// The first buffer is a 16bpp luminance buffer.
	// Immediately after this is an interleaved buffer of 16bpp Cb, Cr pairs.

See also:

I have found zero Qt examples or Qt open source projects showing how to use QVideoFrame with multiple planes.

I wrote it how I think I understand it should go at:

https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/ndireceiverworker.cpp#L291

Problem with my code is that it crashes with either heap corruption or write access violation just shortly after halfway through memcpying the 2nd plane:

https://github.com/NightVsKnight/NDI-Monitor/blob/main/ndi-monitor-qt6/ndireceiverworker.cpp#L353

Before someone responds "You are obviously corrupting your heap or memory", the 2nd plane should be a contiguous 5,529,600 bytes, but I am crashing after writing only 2,858,994 bytes to it.

I think my problem has something to do with the QVideoFrame::bits(int plane) mapped buffers that I must be using wrong, but I am doing the best I know how considering I have been unable find any examples of how to use multiple planes.

Or maybe I am incorrectly using P016 for a P216 pixel format?

If I write less than 2,858,995 bytes of the 2nd plane's 5,529,600 bytes the code no longer crashes and proves that it is mostly working, but it is clearly an incomplete raster:

Notice how the 2nd plane's U and V [color] components stretch down the screen because the write was incomplete.
If I write less bytes (ex: less than bytesInPlane / 2), it gets droopier.
It should look like this:

So, how does one correctly write to multi-planar or semi-planar video frames and prevent it from corrupting the heap/memory?

Thanks!

paulpv

I have learned two things:

The crash is solved by changing:

memcpy(videoFrame.bits(plane), pNdiVideoFrame->p_data, bytesInPlane);

to

memcpy(videoFrame.bits(plane), pNdiVideoFrame->p_data, videoFrame.mappedBytes(planeNum));

Format P016's layout is https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats#p016-and-p010:

Format P216's layout is https://docs.microsoft.com/en-us/windows/win32/medfound/10-bit-and-16-bit-yuv-video-formats#p216-and-p210:

That H/2 explains the phenomenon that I am experiencing.

I need to either:

trick the existing QVideoFrame code in to using P216
-or-
convert P216 (4:2:2) to P016 (4:2:0)...which I would prefer to not have to do.
-or-
implement my own P216 processing.

SGaist

Hi,

I would add solution number 4: add support for P216 to QVideoFrame.

It would be a nice longterm option.

As for your current issue, you would indeed need to reprocess the YV part to fit P016.

paulpv

@SGaist said in Looking for example of how to use QVideoFrame wiith multiple plane pixel format (specifically P016 or P216 or NV16):

Hi,
I would add solution number 3: add support for P216 to QVideoFrame.
It would be a nice longterm option.

I already added to my followup and was about to post a new question specific to P216.
My OP was on the correct/recommended way to process multi/semi-planar pixel formats.
Think my solution is close?

SGaist

My number was wrong !

Number three might be the quickest to implement but likely not the most efficient.

Hence my number 4 suggestion.

paulpv

@SGaist Thanks.
Know much about efficiently converting 4:2:2 (what the unsupported P216 is) to 4:2:0 (what the supported P016 is)?
Seems like that should be pretty straight forward.

SGaist

I don't have it at hand but looking for color space conversion should give you some good results.

paulpv

To follow up, I have temporarily mitigated the P216 -> P016 issue by using this code to scan every other UV line:

It still leaves a few small artifacts laying around, but it is passable while I start the conversation of getting P216 added to Qt.

Thanks!

Pv