OpenGL: Displaying dynamic image content as texture
-
Hello all,
using Qt 5.4 currently, I'm trying to display changing image content as fast as possible on a widget. My attempt is using a QOpenGLWidget, which draws the actual image as a texture on a square (0.0 to 1.0 on both X and Y axis). The square consists of two triangles, of course.
The reason I've chosen OpenGl is that the input image is 16 bit and should be converted to 8 bit grayscale by defining values for brightness and contrast. The vertex shader applies those values to the texture ("fragShader").
The method setData() updates the texture whenever new data is received. Works, but it does similar things like the paintGl() method, so maybe it's not the correct way.
My OpenGL knowledge is "limited", so maybe can give me some hints on this topic:
- Is this way actually a good approach to display dynamic content ("video")?
- Whenever I change the texture data, I need to re-set vertices (triangles) and their texture coordinates. Is there a better way to do this?
Thank you for reading,
PauleVertex shader
attribute vec4 vertPos; /* IN: vertex position */ attribute vec2 vertTexCoord; /* IN: vertex texture coordinate */ uniform mat4 vertModelView; /* IN: vertex model-view-projection matrix */ varying vec2 fragTexCoord; /* OUT: vertex texture coordinate */ void main() { gl_Position = vertModelView*vertPos; fragTexCoord = vertTexCoord; }
Fragment shader
//precision mediump float; uniform sampler2D samplerTex1; /* IN: texture 1 sampler */ varying vec2 fragTexCoord; /* IN: vertex texture coordinate */ uniform float contrastMult; /* IN: contrast multiplier */ uniform float brightnessAdd; /* IN: brightness summand */ uniform vec3 deinterlace; /* IN: de-interlacing information: s .. interlacing offset [pixel/texture size] t .. texture size [pixel] p .. mirror y axis, if 1.0 */ const float fZero = 0.0; const float fOne = 1.0; const float fTwo = 2.0; const float fColScal = 256.0; /** * Uses texture coordinate provided by vertex shader instead of computing them here. See "Be Aware of Dynamic Texture Lookups" * in https://developer.apple.com/library/ios/documentation/3ddrawing/conceptual/opengles_programmingguide/BestPracticesforShaders/BestPracticesforShaders.html. */ void main() { vec2 texCoord; if (deinterlace.p == 0.0) { texCoord = vec2(deinterlace.s, fragTexCoord.y); } else { /* invert y */ texCoord = vec2(deinterlace.s, 1-fragTexCoord.y); } texCoord.x = clamp(texCoord.x+fragTexCoord.x, fZero, fOne); /* Adjust brightness and contrast. Make sure to first apply brightness, then contrast! */ float color = (fColScal * texture2D(samplerTex1, texCoord).a + texture2D(samplerTex1, texCoord).r + brightnessAdd) * contrastMult; color = clamp(color, fZero, fOne); gl_FragColor = vec4(color, color, color, fOne); }
VideoGlWidget.h
#pragma once #include <QOpenGLWidget> #include <QOpenGLFunctions> /* forward declaration(s) */ class QOpenGLDebugLogger; class QOpenGLShader; class QOpenGLShaderProgram; class QOpenGLTexture; /** * Displays single detector signal as image. */ class VideoGlWidget : public QOpenGLWidget, public QOpenGLFunctions { Q_OBJECT public: /** Constructor. */ VideoGlWidget( QWidget * parent = 0); /** Destructor. */ ~VideoGlWidget(); unsigned int frames; /** Total amount of rendered frames. */ unsigned int painted; /** Amount of calls to painGL() method. Increases every second, or if contrast/brightness changes or if new image is received. */ public slots: /** Set contrast. */ void setContrast( float contrast); /** Set brightness. */ void setBrightness( float brightness); /** Set data to display. */ void setData( char * data, int width, int height, bool invertY); protected: /** OpenGL initialization. */ virtual void initializeGL(); /** Paint OpenGL content. */ virtual void paintGL(); /** Called whenever widget is resized. */ virtual void resizeGL( int width, int height); /** Select shaders to use based on current setting of X function setting. */ void selectShaders(); protected: bool prepared; int remHeight; /** Height of image. [pixel] */ GLfloat invertY; /** Image is flipped horizontally, if non-zero. */ GLfloat contrastMult; /** Contrast multiplier. [0.0..65535.0] */ GLfloat brightnessAdd; /** Brightness summands. [-256.0..256.0] */ GLfloat deinterlace[3]; /** De-interlacing information for shader 0 .. de-interlacing offset [pixel/texture size] 1 .. texture size [pixel] 2 .. invert y, if 1 */ private: GLfloat mvpMatrix[16]; /** Model-view-projection matrix. */ QOpenGLShaderProgram * program; /** Shader program. */ QOpenGLShader * vertShader; /** Vertex shader. */ QOpenGLShader * fragShader; /** Fragment shader. */ int vertPosAttrId; /** Vertex shader attribute id of "vertPos". */ int vertTexCoordAttrId; /** Vertex shader attribute id of "vertTexCoord". */ int vertModelViewAttrId; /** Vertex shader attribute id of "vertModelView". */ int texSamplerId; /** Fragment shader uniform id of all texture samplers. */ GLuint signalTextureId; /** Texture ID per input signal. */ int contrastUniId; /** Fragment shader uniform id of contrast vector. */ int brightnessUniId; /** Fragment shader uniform id of brightness vector. */ int deinterlaceUniId; /** Fragment shader uniform id of de-interlace values. */ };
VideoGlWidget.cpp
#include "ui/VideoGlWidget.h" #include <QOpenGLDebugLogger> #include <QOpenGLShader> #include <QOpenGLShaderProgram> #include <cmath> /* local variables */ GLuint vertexAndTextureBufferIds[2]; /** Buffer IDs of vertex and texure VBOs. */ /** Retrieve orthogonal model-view-projection matrix. Near and far planes are set to -1 and 1. */ void ortho( float left, float right, float bottom, float top, GLfloat * matrix) { const float deltaX = right - left; const float deltaY = top - bottom; /* load identity to mvpMatrix */ memset(matrix, 0, 16*sizeof(GLfloat)); if (deltaX != 0.0f && deltaY != 0.0f) { matrix[0] = 2.0f / deltaX; /* 0;0 */ matrix[5] = 2.0f / deltaY; /* 1;1 */ matrix[10] = -1.0f; /* 2;2 */ matrix[15] = 1.0f; /* 3;3 */ matrix[12] = -(right + left) / deltaX; /* 3;0 */ matrix[13] = -(top + bottom) / deltaY; /* 3;1 */ } } /** * Constructor. */ VideoGlWidget::VideoGlWidget( QWidget * parent) : QOpenGLWidget(parent), prepared(false), vertShader(0), fragShader(0), program(0), vertPosAttrId(-1), vertTexCoordAttrId(-1), vertModelViewAttrId(-1), contrastUniId(-1), brightnessUniId(-1), deinterlaceUniId(-1), texSamplerId(0), signalTextureId(0), contrastMult(1.0f), brightnessAdd(1.0f), frames(0), painted(0) { } /** Destructor. */ VideoGlWidget::~VideoGlWidget() { } /** * Set contrast of single channels. * * @param[in] contrast contrast multiplier */ void VideoGlWidget::setContrast( float contrast) { /* adjust contrast */ if (contrast > 255.0) { contrast = 255.0; } else if (contrast <= 0.001) { contrast = 0.001; } this->contrastMult = contrast; /* redraw GL */ this->update(); } /** * Set brightness of single channel. * * @param[in] brightness new brightness value */ void VideoGlWidget::setBrightness( float brightness) { /* adjust contrast */ if (brightness < -255.0) { brightness = -255.0; } else if (brightness > 255.0) { brightness = 255.0; } this->brightnessAdd = brightness; /* redraw GL */ this->update(); } /** * Set data to display and mode. * * A display mode change will change OpenGL program and attached shaders. * * @param[in] data data buffer (16-bit values) * @param[in] width data width [px] * @param[in] height data height [px] * @param[in] invertY if true, flip image on Y axis */ void VideoGlWidget::setData( char * data, int width, int height, bool invertY) { /* set new REM image size */ this->remHeight = height; this->invertY = (invertY ? 1.0f : 0.0f); this->prepared = true; this->glUseProgram(this->program->programId()); /* load texture(s) */ this->glBindTexture( /* set type of texture (2D) */ GL_TEXTURE_2D, /* type of texture = 2D */ this->signalTextureId); /* texture ID */ this->glTexImage2D( GL_TEXTURE_2D, /* load 2D texture data */ 0, /* mip map level (0!) */ GL_LUMINANCE_ALPHA, /* internal format */ width, /* width */ height, /* height */ 0, /* border (0!) */ GL_LUMINANCE_ALPHA, GL_UNSIGNED_BYTE, /* data type (luminance and alpha both use 2 byte) */ data /* data */ ); GLenum lastError; while (GL_NO_ERROR != (lastError =this->glGetError())) { qDebug() << lastError; } /* two triangles create a square (GL_TRIANGLE_FAN) */ GLfloat vVertices[] = { 1.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 1.f, 0.f, 1.f, 1.f, 0.f }; GLfloat vTexCoord[] = { 1.f, 1.f, 0.f, 0.f, 1.f, 0.f, 0.f, 0.f, 0.f, 1.f, 0.f, 0.f }; // TODO: irgendwo eher setzen? /* load texture coordinates (same as vertices) */ this->glVertexAttribPointer(this->vertTexCoordAttrId, 2, GL_FLOAT, GL_FALSE, 3 * sizeof(GLfloat), vTexCoord); this->glEnableVertexAttribArray(this->vertTexCoordAttrId); /* load vertex data */ this->glVertexAttribPointer(this->vertPosAttrId, 3, GL_FLOAT, GL_FALSE, 0, vVertices); this->glEnableVertexAttribArray(this->vertPosAttrId); this->glDrawArrays(GL_TRIANGLE_FAN, 0, 4); /* always call update (even if ModeOff is active) */ this->update(); this->frames++; } /** * OpenGL initialization. * * Do not call: * - glEnable(GL_TEXTURE_2D) because it is only supported by fixed function pipeline */ void VideoGlWidget::initializeGL() { /* ----------------- GL / Qt state machine -------------------------------------------- */ this->initializeOpenGLFunctions(); /* cannot use any QOpenGLFunctions method before this call! */ this->glClearColor(0.0f, 0.0f, 0.0f, 1.0f); /* clear color buffer (GL_COLOR_BUFFER_BIT) to non-transparent black */ this->glPixelStorei(GL_UNPACK_ALIGNMENT, 1); /* pixel stores are aligned byte-wise (used for textures and others) */ this->glEnable(GL_CULL_FACE); /* front-facing elements are not drawn */ this->glDisable(GL_ALPHA_TEST); /* do not test fragments for their alpha value, because there's only one (top) fragment to be displayed */ this->glDisable(GL_BLEND); /* do not blend various fragments color with each other */ this->glDisable(GL_DEPTH_TEST); /* fragment depth testing off */ this->glDisable(GL_DITHER); /* color dithering off */ glDisable(GL_FOG); /* do not add fog color to distant objects */ glDisable(GL_LIGHTING); /* no light sources */ glDisable(GL_LOGIC_OP); /* disable logic operations for pixels */ glDisable(GL_STENCIL_TEST); /* disable stencil buffer, because full screen is drawn */ glDisable(GL_TEXTURE_1D); /* 1-dimensional texturing is not required, because fragment shader is enabled */ glDisable(GL_TEXTURE_2D); /* 2-dimensional texturing is not required, because fragment shader is enabled */ this->glCullFace(GL_FRONT); ortho(0.0f, 1.0f, 0.0f, 1.0f, this->mvpMatrix); /* define world coordinates */ this->program = new QOpenGLShaderProgram(); this->vertShader = new QOpenGLShader(QOpenGLShader::Vertex, (QObject *)0); this->fragShader = new QOpenGLShader(QOpenGLShader::Fragment, (QObject *)0); bool result; Q_ASSERT(glGetError() == GL_NO_ERROR); /* ---------------- data buffers ------------------------------------------------------- */ this->glGenTextures(1, &this->signalTextureId); /* generate IDs for texture data (maximum of 4) */ /* ---------------- shaders / program -------------------------------------------------- */ Q_INIT_RESOURCE(FpgaCommunication); /* vertex and fragment shaders are not checked for compiling errors; those are printed to console by Qt */ this->vertShader->compileSourceFile(":/shader/Resources/vertex"); this->fragShader->compileSourceFile(":/shader/Resources/fragment"); /* select correct shaders to use */ this->selectShaders(); } /** * Select shaders based on current setting of X function setting. */ void VideoGlWidget::selectShaders() { bool result; /* before adding any shaders, remove all of them */ this->program->removeAllShaders(); /* add shaders (depends on used ones) */ result = this->program->addShader(this->vertShader); result &= this->program->addShader(this->fragShader); Q_ASSERT(result); this->program->link(); Q_ASSERT(this->program->isLinked()); Q_ASSERT(glGetError() == GL_NO_ERROR); this->glUseProgram(this->program->programId()); /* ---------------- shader parameter IDs ----------------------------------------------- */ this->vertPosAttrId = this->program->attributeLocation("vertPos"); /* get shader attribute IDs */ this->vertTexCoordAttrId = this->program->attributeLocation("vertTexCoord"); this->vertModelViewAttrId = this->program->uniformLocation("vertModelView"); this->contrastUniId = this->program->uniformLocation("contrastMult"); this->brightnessUniId = this->program->uniformLocation("brightnessAdd"); this->deinterlaceUniId = this->program->uniformLocation("deinterlace"); this->texSamplerId = this->program->uniformLocation("samplerTex1"); /* get texture sampler ID */ /* ----------------- error check ------------------------------------------------------- */ Q_ASSERT(this->vertPosAttrId != -1); Q_ASSERT(this->vertTexCoordAttrId != -1); Q_ASSERT(this->vertModelViewAttrId != -1); Q_ASSERT(this->contrastUniId != -1); Q_ASSERT(this->brightnessUniId != -1); Q_ASSERT(this->deinterlaceUniId != -1); /* do _not_ check texture sampler IDs, some of them may not be set */ /* ----------------- static GL data ---------------------------------------------------- */ this->glUseProgram(this->program->programId()); /* model-view-projection matrix */ this->glUniformMatrix4fv(this->vertModelViewAttrId, 1, GL_FALSE, mvpMatrix); /* texture */ this->glActiveTexture(GL_TEXTURE0); /* following texture calls affect texture 0 */ this->glBindTexture( /* set type of texture (2D) */ GL_TEXTURE_2D, /* type of texture = 2D */ this->signalTextureId); /* texture ID */ this->glUniform1i( this->texSamplerId, /* texture sampler uniform ID */ 0); /* value of sampler */ glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST); /* set filter method to cheapest one available (performance-wise); wrapping is not required */ glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST); /* vertex data needs to be enabled once */ this->glEnableVertexAttribArray(this->vertTexCoordAttrId); this->glEnableVertexAttribArray(this->vertPosAttrId); } /** * Paint OpenGL content. */ void VideoGlWidget::paintGL() { #if 0 if (!this->prepared) { /* prepare video GL widget to display black image (will be called only once) */ unsigned short data[4] = {0, 0, 0, 0}; this->setData((char*) data, 2, 2, false, false); this->glClear(GL_COLOR_BUFFER_BIT); return ; } #endif this->painted++; GLenum lastError; while (GL_NO_ERROR != (lastError = this->glGetError())) { qDebug() << lastError; } // TODO: http://playcontrol.net/ewing/jibberjabber/opengl_vertex_buffer_object.html ?? /* two triangles create a square (GL_TRIANGLE_FAN) */ GLfloat vVertices[] = { 1.f, 0.f, 0.f, 0.f, 0.f, 0.f, 0.f, 1.f, 0.f, 1.f, 1.f, 0.f }; GLfloat vTexCoord[] = { 1.f, 1.f, 0.f, 0.f, 1.f, 0.f, 0.f, 0.f, 0.f, 1.f, 0.f, 0.f }; /* clear color buffer */ this->glClear(GL_COLOR_BUFFER_BIT); /* use the program object */ this->glUseProgram(this->program->programId()); /* set brightness/contrast */ this->glUniform1f(this->contrastUniId, this->contrastMult); this->glUniform1f(this->brightnessUniId, this->brightnessAdd); /* set some options */ this->deinterlace[0] = 0; this->deinterlace[1] = (GLfloat)this->remHeight; this->deinterlace[2] = this->invertY; this->glUniform3fv(this->deinterlaceUniId, 1, this->deinterlace); this->glBindTexture(GL_TEXTURE_2D, this->signalTextureId); // TODO: irgendwo eher setzen?? /* load texture coordinates (same as vertices) */ this->glVertexAttribPointer(this->vertTexCoordAttrId, 2, GL_FLOAT, GL_FALSE, 3 * sizeof(GLfloat), vTexCoord); this->glEnableVertexAttribArray(this->vertTexCoordAttrId); /* load vertex data */ this->glVertexAttribPointer(this->vertPosAttrId, 3, GL_FLOAT, GL_FALSE, 0, vVertices); this->glEnableVertexAttribArray(this->vertPosAttrId); this->glDrawArrays(GL_TRIANGLE_FAN, 0, 4); } /** Called whenever widget is resized. */ void VideoGlWidget::resizeGL( int width, int height) { this->glViewport(0, 0, width, height); }
-
Hi,
When you say video, do you mean actually a multimedia file ? If not, what is generating these images ?
-
The input data does not originate from a multimedia stream, but is a stream of single, uncompressed images, which are generated by a hardware device (FPGA). Frequency and resolution will vary (depending on the setting of the hardware device), but will not exceed 1k x 1k pixel at 25 fps.
The data consists of 16-bit samples, which can be interpreted as brightness levels from 0 (black) to 65535 (white).
-
So it's some sort of camera ?
-
In that case, did you consider implementing a camera backend for QtMultimedia ? That could simplify your life and make your image stream available to both widgets and QtQuick.
-
I didn't know about QtMultimedia so far. Thank you for making me aware of it :-)
Although it supports 16 bit grayscale images, it seems to lack the support of custom GL shaders - does it? I use those shaders to do some pixel work on each input buffer ("frame") like line re-ordering. Also some frames contain multiple signals (up to 6 with 16 bit each), which will be calculated to a single frame.
OpenGl was my first choice, because it offers all those operations in hardware.
-
Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface.
-
Hi Riff,
I am actually looking for same solution as you are. I am also using QOpenglTexture+setData in QOpenGLWidget. My platform is Intel Atom E3800. The performance is way below what I expected in OpenGL. (40 fps 1080p. I wish to have 60 fps) I've googled but haven't found answer. Some say to lift up the performance is to "render-to-texture" (FBO) I am still looking for a good example for doing FBO.
Have you tried SGaist's suggestion (QVideoSurfaceGL painter implementation)? Would you please kindly share what you have found?
-
Hello SGaist and HSLee,
I haven't had the time go get to that project, so I made no real progress. But before I had to switch, I took some notes about OpenGL buffer handling. I can only provide some links, but maybe it helps others:
// TODO: render to PBO/FBOs
// http://www.songho.ca/opengl/gl_pbo.html
// https://www.opengl.org/discussion_boards/showthread.php/184561-glTexSubImage2D-with-Buffer-Object-less-efficient
// https://www.opengl.org/discussion_boards/showthread.php/198787-Updating-textures-per-frame
// http://stackoverflow.com/questions/21577964/efficient-way-of-updating-texture-in-opengl
// http://stackoverflow.com/questions/9863969/updating-a-texture-in-opengl-with-glteximage2d
// http://stackoverflow.com/questions/3887636/how-to-manipulate-texture-content-on-the-fly/10702468#10702468
// http://hacksoflife.blogspot.de/2006/10/vbos-pbos-and-fbos.htmlWhen I come up with a better solution than using glTexImage2D, I'll post here.
-
Hi Riff-Raff,
Thank you for the link. This one is really helpful:
http://www.songho.ca/opengl/gl_pbo.htmlRecently I got the frame rate boost to 54 fps by implementing PBO method:
void GLWidget::initializeGL()
{
initializeOpenGLFunctions();glEnable(GL_DEPTH_TEST); glEnable(GL_TEXTURE_2D);
//Create texture
glGenTextures(1, &id_y);
glBindTexture(GL_TEXTURE_2D, id_y);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_nVideoW/1.5, m_nVideoH, 0, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, frame);
glBindTexture(GL_TEXTURE_2D, 0);
//Create 2 PBO
glGenBuffers(2, pbo);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[0]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[1]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);//Create and compile OpenGL GLSL shader programs
//vertex shader
m_pVSHader = new QOpenGLShader(QOpenGLShader::Vertex, this);
bool bCompile = m_pVSHader->compileSourceCode(vsrc);
if(!bCompile)
{
}
//fragment shader
m_pFSHader = new QOpenGLShader(QOpenGLShader::Fragment, this);
bCompile = m_pFSHader->compileSourceCode(fsrc_v210);
if(!bCompile)
{
}
m_pShaderProgram = new QOpenGLShaderProgram;
m_pShaderProgram->addShader(m_pFSHader);
m_pShaderProgram->addShader(m_pVSHader);
m_pShaderProgram->bindAttributeLocation("vertexIn", ATTRIB_VERTEX);
m_pShaderProgram->bindAttributeLocation("textureIn", ATTRIB_TEXTURE);
m_pShaderProgram->link();
m_pShaderProgram->bind();
wID = m_pShaderProgram->uniformLocation("width_gl");
hID = m_pShaderProgram->uniformLocation("height_gl");
GLint m_matrix = m_pShaderProgram->uniformLocation("matrix");
textureUniformY = m_pShaderProgram->uniformLocation("tex_y");glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, vertexVertices); glVertexAttribPointer(ATTRIB_TEXTURE, 2, GL_FLOAT, 0, 0, textureVertices); glEnableVertexAttribArray(ATTRIB_VERTEX); glEnableVertexAttribArray(ATTRIB_TEXTURE); glClearColor(0.3,0.3,0.3,0.0);
}
void GLWidget::paintGL()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
QMatrix4x4 m;
m.setToIdentity();
#ifdef ROTATE_90
m.rotate(90.0f, 0.0, 0.0, 1.0);
#endif
glUniform1f(wID, m_nVideoW);//1920.0);
glUniform1f(hID, m_nVideoH);//1080.0);
m_pShaderProgram->setUniformValue("matrix", m);
m_pShaderProgram->setUniformValue("u_tex", 0);glActiveTexture(GL_TEXTURE0); glBindTexture (GL_TEXTURE_2D, id_y); int t = paint_pbo; if(paint_pbo) { glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]); glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0); glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0); paint_pbo = paint_pbo%2 + 1; } else paint_pbo = 1; glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]); glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, NULL, GL_STREAM_DRAW); GLvoid *pixelUnpackBuffer = glMapBufferRange(GL_PIXEL_UNPACK_BUFFER, 0, frameSize, GL_MAP_WRITE_BIT); if (!pixelUnpackBuffer) { qDebug () << "map buffer range failed."; } memcpy(pixelUnpackBuffer, m_pBufYuv, frameSize); glUnmapBuffer(GL_PIXEL_UNPACK_BUFFER); glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0); glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
}
However, I still can't make 60fps. It seems the bottleneck is:
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0);According to Songho, copying pixel from PBO to texture object is by "glTexSubImage2D", and it's DMA. It should be fast. For somehow in my case, updating PBO is much faster than DMA data from PBO to texture. I wonder there's something wrong in my code.
To clarify, I've tried to render without uploading data and it can reach 60fps. I know the bottleneck is data transfering. Just don't know how to improve more.
-
It's been some times since I did hardcore OpenGL but are you sure that
GL_STREAM_DRAW
is the right type ? Aren't you moving stuff within the GPU side ? -
Dear SGaist,
I've tried to changed to GL_STATIC_DRAW and GL_DYNAMIC_DRAW. The outcome is similar to GL_STREAM_DRAW. Seems in my case changing usage flag does not help.
I used GL_STREAM_DRAW because PBO content will be replaced by incoming frame data regularly (my case, preloaded 20 raw RGBA frames to RAM and loop copy the frame to PBO in sequence every 16ms)I've googled this performance issue. Some suggested to draw with indices instead of vertex only. And I think I should buffered my vertex to GPU too. I'll post if I found method to improve.
-
Hi guys,
I've tried to upload my vertex/texture vertices (vbo) and also create/upload indices (ibo) and do the render by "glDrawElements". It seems not help much. Seems the low performance causes by GPU ability. If I simply remove my shader conversion function (I am doing conversion from v210 to RGBA) to just output whatever "texture2D" output (video frame with wrong color). It can reach to 60 fps.
-
From memory, IIRC, you should avoid as much as possible ifs in a shader thus do you currently need that check for de-interlacing in the shader ? Shouldn't you rather have two different shaders depending on the video source ?
The second thing I see, you are calling texture2D twice to access the exact same variable. Why not store the result of texture2D and use that variable in the colour computation function ?
-
@SGaist: It seems you are refering to my original shaders, not the ones HSLee talked about.
Yes, the de-interlacing (by line) is required, it changes every image, so exchanging the shader every frame seems to be a slow approach to me. Also, thank you for the hint about texture2D()! I will give it a try.EDIT: Just tried to access texture2D() only once, but "GPU Shader Analyzer" generates the same code, whether I export the result to a vec4 first or not.
-
Why aren't you using QOpenglTexture? I have similar problem (external device producing raw data i need to render as video).
@SGaist
"Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface."
Could you elaborate?Is it the only class that one has to implement to get external video data provider?
How should one put the external data there? Construct a QVideoFrame object that contains it? -
In that kind of cases, I usually write a backend for QtMultimedia to integrated that device within the multimedia workflow.