OpenGL: Displaying dynamic image content as texture

SGaist

In that case, did you consider implementing a camera backend for QtMultimedia ? That could simplify your life and make your image stream available to both widgets and QtQuick.

Riff Raff

I didn't know about QtMultimedia so far. Thank you for making me aware of it :-)

Although it supports 16 bit grayscale images, it seems to lack the support of custom GL shaders - does it? I use those shaders to do some pixel work on each input buffer ("frame") like line re-ordering. Also some frames contain multiple signals (up to 6 with 16 bit each), which will be calculated to a single frame.

OpenGl was my first choice, because it offers all those operations in hardware.

SGaist

Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface.

HSLee

Hi Riff,

I am actually looking for same solution as you are. I am also using QOpenglTexture+setData in QOpenGLWidget. My platform is Intel Atom E3800. The performance is way below what I expected in OpenGL. (40 fps 1080p. I wish to have 60 fps) I've googled but haven't found answer. Some say to lift up the performance is to "render-to-texture" (FBO) I am still looking for a good example for doing FBO.

Have you tried SGaist's suggestion (QVideoSurfaceGL painter implementation)? Would you please kindly share what you have found?

Riff Raff

Hello SGaist and HSLee,

I haven't had the time go get to that project, so I made no real progress. But before I had to switch, I took some notes about OpenGL buffer handling. I can only provide some links, but maybe it helps others:

// TODO: render to PBO/FBOs
// http://www.songho.ca/opengl/gl_pbo.html
// https://www.opengl.org/discussion_boards/showthread.php/184561-glTexSubImage2D-with-Buffer-Object-less-efficient
// https://www.opengl.org/discussion_boards/showthread.php/198787-Updating-textures-per-frame
// http://stackoverflow.com/questions/21577964/efficient-way-of-updating-texture-in-opengl
// http://stackoverflow.com/questions/9863969/updating-a-texture-in-opengl-with-glteximage2d
// http://stackoverflow.com/questions/3887636/how-to-manipulate-texture-content-on-the-fly/10702468#10702468
// http://hacksoflife.blogspot.de/2006/10/vbos-pbos-and-fbos.html

When I come up with a better solution than using glTexImage2D, I'll post here.

HSLee

Hi Riff-Raff,

Thank you for the link. This one is really helpful:
http://www.songho.ca/opengl/gl_pbo.html

Recently I got the frame rate boost to 54 fps by implementing PBO method:

void GLWidget::initializeGL()
{
initializeOpenGLFunctions();

glEnable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);

//Create texture
glGenTextures(1, &id_y);
glBindTexture(GL_TEXTURE_2D, id_y);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_nVideoW/1.5, m_nVideoH, 0, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, frame);
glBindTexture(GL_TEXTURE_2D, 0);
//Create 2 PBO
glGenBuffers(2, pbo);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[0]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[1]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

//Create and compile OpenGL GLSL shader programs
//vertex shader
m_pVSHader = new QOpenGLShader(QOpenGLShader::Vertex, this);
bool bCompile = m_pVSHader->compileSourceCode(vsrc);
if(!bCompile)
{
}
//fragment shader
m_pFSHader = new QOpenGLShader(QOpenGLShader::Fragment, this);
bCompile = m_pFSHader->compileSourceCode(fsrc_v210);
if(!bCompile)
{
}
m_pShaderProgram = new QOpenGLShaderProgram;
m_pShaderProgram->addShader(m_pFSHader);
m_pShaderProgram->addShader(m_pVSHader);
m_pShaderProgram->bindAttributeLocation("vertexIn", ATTRIB_VERTEX);
m_pShaderProgram->bindAttributeLocation("textureIn", ATTRIB_TEXTURE);
m_pShaderProgram->link();
m_pShaderProgram->bind();
wID = m_pShaderProgram->uniformLocation("width_gl");
hID = m_pShaderProgram->uniformLocation("height_gl");
GLint m_matrix = m_pShaderProgram->uniformLocation("matrix");
textureUniformY = m_pShaderProgram->uniformLocation("tex_y");

glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, vertexVertices);
glVertexAttribPointer(ATTRIB_TEXTURE, 2, GL_FLOAT, 0, 0, textureVertices);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glEnableVertexAttribArray(ATTRIB_TEXTURE);
glClearColor(0.3,0.3,0.3,0.0);

}

void GLWidget::paintGL()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
QMatrix4x4 m;
m.setToIdentity();
#ifdef ROTATE_90
m.rotate(90.0f, 0.0, 0.0, 1.0);
#endif
glUniform1f(wID, m_nVideoW);//1920.0);
glUniform1f(hID, m_nVideoH);//1080.0);
m_pShaderProgram->setUniformValue("matrix", m);
m_pShaderProgram->setUniformValue("u_tex", 0);

glActiveTexture(GL_TEXTURE0);
glBindTexture (GL_TEXTURE_2D, id_y);
int t = paint_pbo;
 if(paint_pbo)
{
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0);
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);
    paint_pbo = paint_pbo%2 + 1;
}
else
    paint_pbo = 1;

 glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]);
 glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, NULL, GL_STREAM_DRAW);
 GLvoid *pixelUnpackBuffer = glMapBufferRange(GL_PIXEL_UNPACK_BUFFER, 0, frameSize, GL_MAP_WRITE_BIT);
 if (!pixelUnpackBuffer)
 {
     qDebug () << "map buffer range failed.";
 }
 memcpy(pixelUnpackBuffer, m_pBufYuv, frameSize);
 glUnmapBuffer(GL_PIXEL_UNPACK_BUFFER);
 glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

}

However, I still can't make 60fps. It seems the bottleneck is:
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0);

According to Songho, copying pixel from PBO to texture object is by "glTexSubImage2D", and it's DMA. It should be fast. For somehow in my case, updating PBO is much faster than DMA data from PBO to texture. I wonder there's something wrong in my code.

To clarify, I've tried to render without uploading data and it can reach 60fps. I know the bottleneck is data transfering. Just don't know how to improve more.

SGaist

It's been some times since I did hardcore OpenGL but are you sure that GL_STREAM_DRAW is the right type ? Aren't you moving stuff within the GPU side ?

HSLee

Dear SGaist,

I've tried to changed to GL_STATIC_DRAW and GL_DYNAMIC_DRAW. The outcome is similar to GL_STREAM_DRAW. Seems in my case changing usage flag does not help.
I used GL_STREAM_DRAW because PBO content will be replaced by incoming frame data regularly (my case, preloaded 20 raw RGBA frames to RAM and loop copy the frame to PBO in sequence every 16ms)

I've googled this performance issue. Some suggested to draw with indices instead of vertex only. And I think I should buffered my vertex to GPU too. I'll post if I found method to improve.

HSLee

Hi guys,

I've tried to upload my vertex/texture vertices (vbo) and also create/upload indices (ibo) and do the render by "glDrawElements". It seems not help much. Seems the low performance causes by GPU ability. If I simply remove my shader conversion function (I am doing conversion from v210 to RGBA) to just output whatever "texture2D" output (video frame with wrong color). It can reach to 60 fps.

SGaist

From memory, IIRC, you should avoid as much as possible ifs in a shader thus do you currently need that check for de-interlacing in the shader ? Shouldn't you rather have two different shaders depending on the video source ?

The second thing I see, you are calling texture2D twice to access the exact same variable. Why not store the result of texture2D and use that variable in the colour computation function ?

Riff Raff

@SGaist: It seems you are refering to my original shaders, not the ones HSLee talked about.
Yes, the de-interlacing (by line) is required, it changes every image, so exchanging the shader every frame seems to be a slow approach to me. Also, thank you for the hint about texture2D()! I will give it a try.

EDIT: Just tried to access texture2D() only once, but "GPU Shader Analyzer" generates the same code, whether I export the result to a vec4 first or not.

Pietrko

Why aren't you using QOpenglTexture? I have similar problem (external device producing raw data i need to render as video).
@SGaist
"Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface."
Could you elaborate?

Is it the only class that one has to implement to get external video data provider?
How should one put the external data there? Construct a QVideoFrame object that contains it?

SGaist

@Pietrko, what do you mean by "external video data provider"?

Pietrko

@SGaist This was very unprecise expression.
What I meant is situation when you have a hardware device that provides data that is accessible only by calls to external (in relation to Qt) library.

SGaist

In that kind of cases, I usually write a backend for QtMultimedia to integrated that device within the multimedia workflow.