OpenGL: Displaying dynamic image content as texture

[Riff Raff]

Hello all,

using Qt 5.4 currently, I'm trying to display changing image content as fast as possible on a widget. My attempt is using a QOpenGLWidget, which draws the actual image as a texture on a square (0.0 to 1.0 on both X and Y axis). The square consists of two triangles, of course.

The reason I've chosen OpenGl is that the input image is 16 bit and should be converted to 8 bit grayscale by defining values for brightness and contrast. The vertex shader applies those values to the texture ("fragShader").

The method setData() updates the texture whenever new data is received. Works, but it does similar things like the paintGl() method, so maybe it's not the correct way.

My OpenGL knowledge is "limited", so maybe can give me some hints on this topic:

1. Is this way actually a good approach to display dynamic content ("video")?
2. Whenever I change the texture data, I need to re-set vertices (triangles) and their texture coordinates. Is there a better way to do this?

Thank you for reading,
Paule

Vertex shader

attribute vec4 vertPos;				/* IN: vertex position */
attribute vec2 vertTexCoord;		/* IN: vertex texture coordinate */
uniform   mat4 vertModelView;		/* IN: vertex model-view-projection matrix */
varying   vec2 fragTexCoord;		/* OUT: vertex texture coordinate */

void main()	{
	gl_Position  = vertModelView*vertPos; 
	fragTexCoord = vertTexCoord;
}

Fragment shader

//precision mediump float;
uniform	  sampler2D		samplerTex1;		/* IN: texture 1 sampler */
varying   vec2			fragTexCoord;		/* IN: vertex texture coordinate */

uniform	  float			contrastMult;		/* IN: contrast multiplier */
uniform	  float			brightnessAdd;		/* IN: brightness summand */			
uniform	  vec3			deinterlace;		/* IN: de-interlacing information:
												   s .. interlacing offset [pixel/texture size]
												   t .. texture size [pixel]
												   p .. mirror y axis, if 1.0 */


const     float			fZero	 = 0.0;
const     float			fOne	 = 1.0;
const	  float			fTwo	 = 2.0;
const     float			fColScal = 256.0;

/**
 * Uses texture coordinate provided by vertex shader instead of computing them here. See "Be Aware of Dynamic Texture Lookups"
 * in https://developer.apple.com/library/ios/documentation/3ddrawing/conceptual/opengles_programmingguide/BestPracticesforShaders/BestPracticesforShaders.html.
 */
void main() {

	vec2	texCoord;
	if (deinterlace.p == 0.0) {
		texCoord = vec2(deinterlace.s, fragTexCoord.y);
	} else {
		/* invert y */
		texCoord = vec2(deinterlace.s, 1-fragTexCoord.y);
	}
	
	texCoord.x = clamp(texCoord.x+fragTexCoord.x, fZero, fOne);
	
	/* Adjust brightness and contrast. Make sure to first apply brightness, then contrast! */
	float color = (fColScal * texture2D(samplerTex1, texCoord).a  + texture2D(samplerTex1, texCoord).r + brightnessAdd) * contrastMult;
	color = clamp(color, fZero, fOne);

	gl_FragColor =	vec4(color, color, color, fOne);
}      

VideoGlWidget.h

#pragma once

#include <QOpenGLWidget>
#include <QOpenGLFunctions>


/* forward declaration(s) */
class QOpenGLDebugLogger;
class QOpenGLShader;
class QOpenGLShaderProgram;
class QOpenGLTexture;


/**
 * Displays single detector signal as image.
 */
class VideoGlWidget : public QOpenGLWidget, public QOpenGLFunctions {

	Q_OBJECT

public:

	/** Constructor. */
	VideoGlWidget(
			 QWidget * parent = 0);
	/** Destructor. */
	~VideoGlWidget();


	unsigned int			frames;				/** Total amount of rendered frames. */
	unsigned int			painted;			/** Amount of calls to painGL() method. Increases every second, or if contrast/brightness changes or if new image is received. */

public slots:
	/** Set contrast. */
	void			setContrast(
		float			contrast);
	/** Set brightness. */
	void			setBrightness(
		float			brightness);
	/** Set data to display. */
	void			setData(
		char			* data,
		int				width,
		int				height,
		bool			invertY);
	

protected:
	/** OpenGL initialization. */
	virtual void	initializeGL();
	/** Paint OpenGL content. */
	virtual void	paintGL();
	/** Called whenever widget is resized. */
	virtual void	resizeGL(
			int			width,
			int			height);

	/** Select shaders to use based on current setting of X function setting. */
	void			selectShaders();

protected:
	bool					prepared;

	int						remHeight;			/** Height of image. [pixel] */
	GLfloat					invertY;			/** Image is flipped horizontally, if non-zero. */
	GLfloat					contrastMult;		/** Contrast multiplier. [0.0..65535.0] */
	GLfloat					brightnessAdd;		/** Brightness summands. [-256.0..256.0] */
	GLfloat					deinterlace[3];		/** De-interlacing information for shader
												    0 .. de-interlacing offset [pixel/texture size]
												    1 .. texture size [pixel]
													2 .. invert y, if 1 */

private:
	GLfloat					mvpMatrix[16];		/** Model-view-projection matrix. */

	QOpenGLShaderProgram	* program;						/** Shader program. */
	QOpenGLShader			* vertShader;					/** Vertex shader. */
	QOpenGLShader			* fragShader;					/** Fragment shader. */

	int						vertPosAttrId;					/** Vertex shader attribute id of "vertPos". */
	int						vertTexCoordAttrId;				/** Vertex shader attribute id of "vertTexCoord". */
	int						vertModelViewAttrId;			/** Vertex shader attribute id of "vertModelView". */
	int						texSamplerId;					/** Fragment shader uniform id of all texture samplers. */
	GLuint					signalTextureId;				/** Texture ID per input signal. */
	int						contrastUniId;					/** Fragment shader uniform id of contrast vector. */
	int						brightnessUniId;				/** Fragment shader uniform id of brightness vector. */
	int						deinterlaceUniId;				/** Fragment shader uniform id of de-interlace values. */
};

VideoGlWidget.cpp

#include "ui/VideoGlWidget.h"

#include <QOpenGLDebugLogger>
#include <QOpenGLShader>
#include <QOpenGLShaderProgram>
#include <cmath>


/* local variables */
GLuint			vertexAndTextureBufferIds[2];	/** Buffer IDs of vertex and texure VBOs. */



/** Retrieve orthogonal model-view-projection matrix. Near and far planes are set to -1 and 1. */
void
ortho(
		float			left,
		float			right,
		float			bottom,
		float			top,
		GLfloat			* matrix) {


    const float		deltaX = right - left;
    const float		deltaY = top - bottom;

	/* load identity to mvpMatrix */
	memset(matrix, 0, 16*sizeof(GLfloat));

	if (deltaX != 0.0f && deltaY != 0.0f) {
        
		matrix[0]  = 2.0f / deltaX;			/* 0;0 */
		matrix[5]  = 2.0f / deltaY;			/* 1;1 */
		matrix[10] = -1.0f;					/* 2;2 */
		matrix[15] = 1.0f;						/* 3;3 */

		matrix[12] = -(right + left) / deltaX;	/* 3;0 */
		matrix[13] = -(top + bottom) / deltaY;	/* 3;1 */
	}
}



/**
 * Constructor.
 */
VideoGlWidget::VideoGlWidget(
		QWidget		* parent)
	: QOpenGLWidget(parent),
	  prepared(false),

	  vertShader(0),
	  fragShader(0),
	  program(0),
	  vertPosAttrId(-1),
	  vertTexCoordAttrId(-1),
	  vertModelViewAttrId(-1),
	  contrastUniId(-1),
	  brightnessUniId(-1),
	  deinterlaceUniId(-1),
	  texSamplerId(0),
	  signalTextureId(0),

	  contrastMult(1.0f),
	  brightnessAdd(1.0f),

	  frames(0),
	  painted(0) {
  
}

/** Destructor. */
VideoGlWidget::~VideoGlWidget() {

}


/**
 * Set contrast of single channels.
 *
 * @param[in]	contrast		contrast multiplier
 */
void
VideoGlWidget::setContrast(
		float			contrast) {

	/* adjust contrast */
	if (contrast > 255.0) {
		contrast = 255.0;
	} else if (contrast <= 0.001) {
		contrast = 0.001;
	}
	this->contrastMult = contrast;

	/* redraw GL */
	this->update();
}
/**
 * Set brightness of single channel.
 *
 * @param[in]	brightness		new brightness value
 */
void
VideoGlWidget::setBrightness(
		float			brightness) {

	/* adjust contrast */
	if (brightness < -255.0) {
		brightness = -255.0;
	} else if (brightness > 255.0) {
		brightness = 255.0;
	}
	this->brightnessAdd = brightness;

	/* redraw GL */
	this->update();
}


/**
 * Set data to display and mode.
 *
 * A display mode change will change OpenGL program and attached shaders.
 *
 * @param[in]	data		data buffer (16-bit values)
 * @param[in]	width		data width [px]
 * @param[in]	height		data height [px]
 * @param[in]	invertY		if true, flip image on Y axis
 */
void
VideoGlWidget::setData(
		char			* data,
		int				width,
		int				height,
		bool			invertY) {


	/* set new REM image size */
	this->remHeight		= height;
	this->invertY		= (invertY ? 1.0f : 0.0f);
	this->prepared		= true;

	this->glUseProgram(this->program->programId());

	/* load texture(s) */
	this->glBindTexture(											/* set type of texture (2D) */
		GL_TEXTURE_2D,					/* type of texture = 2D */
		this->signalTextureId);			/* texture ID */

	
	this->glTexImage2D(			
		GL_TEXTURE_2D,			/* load 2D texture data */
		0,						/* mip map level (0!) */
		GL_LUMINANCE_ALPHA,		/* internal format */
		width,					/* width */
		height,					/* height */
		0,						/* border (0!) */
		GL_LUMINANCE_ALPHA,
		GL_UNSIGNED_BYTE,		/* data type (luminance and alpha both use 2 byte) */
		data					/* data */
	);

	GLenum lastError;
	while (GL_NO_ERROR != (lastError =this->glGetError())) {
		qDebug() << lastError;
	}

	/* two triangles create a square (GL_TRIANGLE_FAN) */
	GLfloat vVertices[] = {  1.f, 0.f, 0.f,
                             0.f, 0.f, 0.f,
							 0.f, 1.f, 0.f,
							 1.f, 1.f, 0.f };
	GLfloat vTexCoord[] = {  1.f, 1.f, 0.f,
							 0.f, 1.f, 0.f,
							 0.f, 0.f, 0.f,
							 1.f, 0.f, 0.f };



	// TODO: irgendwo eher setzen?


	/* load texture coordinates (same as vertices) */
	this->glVertexAttribPointer(this->vertTexCoordAttrId, 2, GL_FLOAT, GL_FALSE, 3 * sizeof(GLfloat), vTexCoord);
	this->glEnableVertexAttribArray(this->vertTexCoordAttrId);
	/* load vertex data */
	this->glVertexAttribPointer(this->vertPosAttrId, 3, GL_FLOAT, GL_FALSE, 0, vVertices);
	this->glEnableVertexAttribArray(this->vertPosAttrId);


	this->glDrawArrays(GL_TRIANGLE_FAN, 0, 4);

	/* always call update (even if ModeOff is active) */
	this->update();

	this->frames++;
}

/**
 * OpenGL initialization.
 *
 * Do not call:
 * - glEnable(GL_TEXTURE_2D) because it is only supported by fixed function pipeline
 */
void
VideoGlWidget::initializeGL() {

	/* ----------------- GL / Qt state machine -------------------------------------------- */

	this->initializeOpenGLFunctions();								/* cannot use any QOpenGLFunctions method before this call! */
	this->glClearColor(0.0f, 0.0f, 0.0f, 1.0f);						/* clear color buffer (GL_COLOR_BUFFER_BIT) to non-transparent black */
	this->glPixelStorei(GL_UNPACK_ALIGNMENT, 1);					/* pixel stores are aligned byte-wise (used for textures and others) */
	this->glEnable(GL_CULL_FACE);									/* front-facing elements are not drawn */

	this->glDisable(GL_ALPHA_TEST);									/* do not test fragments for their alpha value, because there's only one (top) fragment to be displayed */
    this->glDisable(GL_BLEND);										/* do not blend various fragments color with each other */
    this->glDisable(GL_DEPTH_TEST);									/* fragment depth testing off */
    this->glDisable(GL_DITHER);										/* color dithering off */


	glDisable(GL_FOG);												/* do not add fog color to distant objects */
    glDisable(GL_LIGHTING);											/* no light sources */
    glDisable(GL_LOGIC_OP);											/* disable logic operations for pixels */
    glDisable(GL_STENCIL_TEST);										/* disable stencil buffer, because full screen is drawn */
    glDisable(GL_TEXTURE_1D);										/* 1-dimensional texturing is not required, because fragment shader is enabled */		
    glDisable(GL_TEXTURE_2D);										/* 2-dimensional texturing is not required, because fragment shader is enabled */		

	this->glCullFace(GL_FRONT);
	ortho(0.0f, 1.0f, 0.0f, 1.0f, this->mvpMatrix);					/* define world coordinates */

	this->program				= new QOpenGLShaderProgram();
	this->vertShader			= new QOpenGLShader(QOpenGLShader::Vertex, (QObject *)0);
	this->fragShader			= new QOpenGLShader(QOpenGLShader::Fragment, (QObject *)0);


	bool result;
	Q_ASSERT(glGetError() == GL_NO_ERROR);

	/* ---------------- data buffers ------------------------------------------------------- */
	this->glGenTextures(1, &this->signalTextureId);	/* generate IDs for texture data (maximum of 4) */

	/* ---------------- shaders / program -------------------------------------------------- */

	Q_INIT_RESOURCE(FpgaCommunication);
	
	/* vertex and fragment shaders are not checked for compiling errors; those are printed to console by Qt */
	this->vertShader->compileSourceFile(":/shader/Resources/vertex");
	this->fragShader->compileSourceFile(":/shader/Resources/fragment");

	/* select correct shaders to use */
	this->selectShaders();
}

/**
 * Select shaders based on current setting of X function setting.
 */
void
VideoGlWidget::selectShaders() {

	bool result;

	/* before adding any shaders, remove all of them */
	this->program->removeAllShaders();

	/* add shaders (depends on used ones) */
	result =  this->program->addShader(this->vertShader);
	result &= this->program->addShader(this->fragShader);
	Q_ASSERT(result);

	this->program->link();

	Q_ASSERT(this->program->isLinked());
	Q_ASSERT(glGetError() == GL_NO_ERROR);

	this->glUseProgram(this->program->programId());

	/* ---------------- shader parameter IDs ----------------------------------------------- */

	this->vertPosAttrId			= this->program->attributeLocation("vertPos");						/* get shader attribute IDs */
	this->vertTexCoordAttrId	= this->program->attributeLocation("vertTexCoord");
	this->vertModelViewAttrId	= this->program->uniformLocation("vertModelView");
	this->contrastUniId			= this->program->uniformLocation("contrastMult");
	this->brightnessUniId		= this->program->uniformLocation("brightnessAdd");
	this->deinterlaceUniId		= this->program->uniformLocation("deinterlace");
	this->texSamplerId			= this->program->uniformLocation("samplerTex1");					/* get texture sampler ID */

	/* ----------------- error check ------------------------------------------------------- */

	Q_ASSERT(this->vertPosAttrId != -1);
	Q_ASSERT(this->vertTexCoordAttrId != -1);
	Q_ASSERT(this->vertModelViewAttrId != -1);
	Q_ASSERT(this->contrastUniId != -1);
	Q_ASSERT(this->brightnessUniId != -1);
	Q_ASSERT(this->deinterlaceUniId != -1);
	
	/* do _not_ check texture sampler IDs, some of them may not be set */

  	/* ----------------- static GL data ---------------------------------------------------- */

	this->glUseProgram(this->program->programId());
	/* model-view-projection matrix */
	this->glUniformMatrix4fv(this->vertModelViewAttrId, 1, GL_FALSE, mvpMatrix);

	/* texture */
	this->glActiveTexture(GL_TEXTURE0);								/* following texture calls affect texture 0 */
	this->glBindTexture(											/* set type of texture (2D) */
		GL_TEXTURE_2D,					/* type of texture = 2D */
		this->signalTextureId);			/* texture ID */
	this->glUniform1i(				
		this->texSamplerId,				/* texture sampler uniform ID */
		0);								/* value of sampler */

	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);	/* set filter method to cheapest one available (performance-wise); wrapping is not required */
	glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);

	/* vertex data needs to be enabled once */
	this->glEnableVertexAttribArray(this->vertTexCoordAttrId);
	this->glEnableVertexAttribArray(this->vertPosAttrId);
}



/**
 * Paint OpenGL content.
 */
void
VideoGlWidget::paintGL() {

  #if 0
	if (!this->prepared) {

		/* prepare video GL widget to display black image (will be called only once) */
		unsigned short	data[4] = {0, 0, 0, 0};
		this->setData((char*) data, 2, 2, false, false);

		this->glClear(GL_COLOR_BUFFER_BIT);
		return ;
	}
  #endif

	this->painted++;

	GLenum lastError;
	while (GL_NO_ERROR != (lastError = this->glGetError())) {
		qDebug() << lastError;
	}

     
	// TODO: http://playcontrol.net/ewing/jibberjabber/opengl_vertex_buffer_object.html ??
	/* two triangles create a square (GL_TRIANGLE_FAN) */
	GLfloat vVertices[] = {  1.f, 0.f, 0.f,
                             0.f, 0.f, 0.f,
							 0.f, 1.f, 0.f,
							 1.f, 1.f, 0.f };
	GLfloat vTexCoord[] = {  1.f, 1.f, 0.f,
							 0.f, 1.f, 0.f,
							 0.f, 0.f, 0.f,
							 1.f, 0.f, 0.f };


	/* clear color buffer */
	this->glClear(GL_COLOR_BUFFER_BIT);

	/* use the program object */
	this->glUseProgram(this->program->programId());

	/* set brightness/contrast */
	this->glUniform1f(this->contrastUniId, this->contrastMult);
	this->glUniform1f(this->brightnessUniId, this->brightnessAdd);
	/* set some options */
	this->deinterlace[0] = 0;
	this->deinterlace[1] = (GLfloat)this->remHeight;
	this->deinterlace[2] = this->invertY;
	this->glUniform3fv(this->deinterlaceUniId, 1, this->deinterlace);
	

	this->glBindTexture(GL_TEXTURE_2D, this->signalTextureId);

	// TODO: irgendwo eher setzen??

	/* load texture coordinates (same as vertices) */
	this->glVertexAttribPointer(this->vertTexCoordAttrId, 2, GL_FLOAT, GL_FALSE, 3 * sizeof(GLfloat), vTexCoord);
	this->glEnableVertexAttribArray(this->vertTexCoordAttrId);
	/* load vertex data */
	this->glVertexAttribPointer(this->vertPosAttrId, 3, GL_FLOAT, GL_FALSE, 0, vVertices);
	this->glEnableVertexAttribArray(this->vertPosAttrId);

	this->glDrawArrays(GL_TRIANGLE_FAN, 0, 4);
}






/** Called whenever widget is resized. */
void
VideoGlWidget::resizeGL(
		int		width,
		int		height) {

	this->glViewport(0, 0, width, height);
}

[SGaist]

Hi,

When you say video, do you mean actually a multimedia file ? If not, what is generating these images ?

[Riff Raff]

The input data does not originate from a multimedia stream, but is a stream of single, uncompressed images, which are generated by a hardware device (FPGA). Frequency and resolution will vary (depending on the setting of the hardware device), but will not exceed 1k x 1k pixel at 25 fps.

The data consists of 16-bit samples, which can be interpreted as brightness levels from 0 (black) to 65535 (white).

[SGaist]

So it's some sort of camera ?

[Riff Raff]

Yes, it generates images as if it were a camera.

[SGaist]

In that case, did you consider implementing a camera backend for QtMultimedia ? That could simplify your life and make your image stream available to both widgets and QtQuick.

[Riff Raff]

I didn't know about QtMultimedia so far. Thank you for making me aware of it :-)

Although it supports 16 bit grayscale images, it seems to lack the support of custom GL shaders - does it? I use those shaders to do some pixel work on each input buffer ("frame") like line re-ordering. Also some frames contain multiple signals (up to 6 with 16 bit each), which will be calculated to a single frame.

OpenGl was my first choice, because it offers all those operations in hardware.

[SGaist]

Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface.

[HSLee]

Hi Riff,

I am actually looking for same solution as you are. I am also using QOpenglTexture+setData in QOpenGLWidget. My platform is Intel Atom E3800. The performance is way below what I expected in OpenGL. (40 fps 1080p. I wish to have 60 fps) I've googled but haven't found answer. Some say to lift up the performance is to "render-to-texture" (FBO) I am still looking for a good example for doing FBO.

Have you tried SGaist's suggestion (QVideoSurfaceGL painter implementation)? Would you please kindly share what you have found?

[Riff Raff]

Hello SGaist and HSLee,

I haven't had the time go get to that project, so I made no real progress. But before I had to switch, I took some notes about OpenGL buffer handling. I can only provide some links, but maybe it helps others:

// TODO: render to PBO/FBOs
// http://www.songho.ca/opengl/gl_pbo.html
// https://www.opengl.org/discussion_boards/showthread.php/184561-glTexSubImage2D-with-Buffer-Object-less-efficient
// https://www.opengl.org/discussion_boards/showthread.php/198787-Updating-textures-per-frame
// http://stackoverflow.com/questions/21577964/efficient-way-of-updating-texture-in-opengl
// http://stackoverflow.com/questions/9863969/updating-a-texture-in-opengl-with-glteximage2d
// http://stackoverflow.com/questions/3887636/how-to-manipulate-texture-content-on-the-fly/10702468#10702468
// http://hacksoflife.blogspot.de/2006/10/vbos-pbos-and-fbos.html

When I come up with a better solution than using glTexImage2D, I'll post here.

[HSLee]

Hi Riff-Raff,

Thank you for the link. This one is really helpful:
http://www.songho.ca/opengl/gl_pbo.html

Recently I got the frame rate boost to 54 fps by implementing PBO method:

void GLWidget::initializeGL()
{
initializeOpenGLFunctions();

glEnable(GL_DEPTH_TEST);
glEnable(GL_TEXTURE_2D);

//Create texture
glGenTextures(1, &id_y);
glBindTexture(GL_TEXTURE_2D, id_y);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, m_nVideoW/1.5, m_nVideoH, 0, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, frame);
glBindTexture(GL_TEXTURE_2D, 0);
//Create 2 PBO
glGenBuffers(2, pbo);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[0]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[1]);
glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, 0, GL_STREAM_DRAW);
glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

//Create and compile OpenGL GLSL shader programs
//vertex shader
m_pVSHader = new QOpenGLShader(QOpenGLShader::Vertex, this);
bool bCompile = m_pVSHader->compileSourceCode(vsrc);
if(!bCompile)
{
}
//fragment shader
m_pFSHader = new QOpenGLShader(QOpenGLShader::Fragment, this);
bCompile = m_pFSHader->compileSourceCode(fsrc_v210);
if(!bCompile)
{
}
m_pShaderProgram = new QOpenGLShaderProgram;
m_pShaderProgram->addShader(m_pFSHader);
m_pShaderProgram->addShader(m_pVSHader);
m_pShaderProgram->bindAttributeLocation("vertexIn", ATTRIB_VERTEX);
m_pShaderProgram->bindAttributeLocation("textureIn", ATTRIB_TEXTURE);
m_pShaderProgram->link();
m_pShaderProgram->bind();
wID = m_pShaderProgram->uniformLocation("width_gl");
hID = m_pShaderProgram->uniformLocation("height_gl");
GLint m_matrix = m_pShaderProgram->uniformLocation("matrix");
textureUniformY = m_pShaderProgram->uniformLocation("tex_y");

glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, vertexVertices);
glVertexAttribPointer(ATTRIB_TEXTURE, 2, GL_FLOAT, 0, 0, textureVertices);
glEnableVertexAttribArray(ATTRIB_VERTEX);
glEnableVertexAttribArray(ATTRIB_TEXTURE);
glClearColor(0.3,0.3,0.3,0.0);

}

void GLWidget::paintGL()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
QMatrix4x4 m;
m.setToIdentity();
#ifdef ROTATE_90
m.rotate(90.0f, 0.0, 0.0, 1.0);
#endif
glUniform1f(wID, m_nVideoW);//1920.0);
glUniform1f(hID, m_nVideoH);//1080.0);
m_pShaderProgram->setUniformValue("matrix", m);
m_pShaderProgram->setUniformValue("u_tex", 0);

glActiveTexture(GL_TEXTURE0);
glBindTexture (GL_TEXTURE_2D, id_y);
int t = paint_pbo;
 if(paint_pbo)
{
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]);
    glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0);
    glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);
    paint_pbo = paint_pbo%2 + 1;
}
else
    paint_pbo = 1;

 glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pbo[paint_pbo-1]);
 glBufferData(GL_PIXEL_UNPACK_BUFFER, frameSize, NULL, GL_STREAM_DRAW);
 GLvoid *pixelUnpackBuffer = glMapBufferRange(GL_PIXEL_UNPACK_BUFFER, 0, frameSize, GL_MAP_WRITE_BIT);
 if (!pixelUnpackBuffer)
 {
     qDebug () << "map buffer range failed.";
 }
 memcpy(pixelUnpackBuffer, m_pBufYuv, frameSize);
 glUnmapBuffer(GL_PIXEL_UNPACK_BUFFER);
 glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);

glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

}

However, I still can't make 60fps. It seems the bottleneck is:
glTexSubImage2D(GL_TEXTURE_2D, 0, 0, 0, m_nVideoW/1.5, m_nVideoH, GL_RGBA, GL_UNSIGNED_INT_2_10_10_10_REV, 0);

According to Songho, copying pixel from PBO to texture object is by "glTexSubImage2D", and it's DMA. It should be fast. For somehow in my case, updating PBO is much faster than DMA data from PBO to texture. I wonder there's something wrong in my code.

To clarify, I've tried to render without uploading data and it can reach 60fps. I know the bottleneck is data transfering. Just don't know how to improve more.

[SGaist]

It's been some times since I did hardcore OpenGL but are you sure that GL_STREAM_DRAW is the right type ? Aren't you moving stuff within the GPU side ?

[HSLee]

Dear SGaist,

I've tried to changed to GL_STATIC_DRAW and GL_DYNAMIC_DRAW. The outcome is similar to GL_STREAM_DRAW. Seems in my case changing usage flag does not help.
I used GL_STREAM_DRAW because PBO content will be replaced by incoming frame data regularly (my case, preloaded 20 raw RGBA frames to RAM and loop copy the frame to PBO in sequence every 16ms)

I've googled this performance issue. Some suggested to draw with indices instead of vertex only. And I think I should buffered my vertex to GPU too. I'll post if I found method to improve.

[HSLee]

Hi guys,

I've tried to upload my vertex/texture vertices (vbo) and also create/upload indices (ibo) and do the render by "glDrawElements". It seems not help much. Seems the low performance causes by GPU ability. If I simply remove my shader conversion function (I am doing conversion from v210 to RGBA) to just output whatever "texture2D" output (video frame with wrong color). It can reach to 60 fps.

[SGaist]

From memory, IIRC, you should avoid as much as possible ifs in a shader thus do you currently need that check for de-interlacing in the shader ? Shouldn't you rather have two different shaders depending on the video source ?

The second thing I see, you are calling texture2D twice to access the exact same variable. Why not store the result of texture2D and use that variable in the colour computation function ?

[Riff Raff]

@SGaist: It seems you are refering to my original shaders, not the ones HSLee talked about.
Yes, the de-interlacing (by line) is required, it changes every image, so exchanging the shader every frame seems to be a slow approach to me. Also, thank you for the hint about texture2D()! I will give it a try.

EDIT: Just tried to access texture2D() only once, but "GPU Shader Analyzer" generates the same code, whether I export the result to a vec4 first or not.

[Pietrko]

Why aren't you using QOpenglTexture? I have similar problem (external device producing raw data i need to render as video).
@SGaist
"Then take a look at the QVideoSurfaceGLPainter implementation used for a custom QAbstractVideoSurface."
Could you elaborate?

Is it the only class that one has to implement to get external video data provider?
How should one put the external data there? Construct a QVideoFrame object that contains it?

[SGaist]

@Pietrko, what do you mean by "external video data provider"?

[Pietrko]

@SGaist This was very unprecise expression.
What I meant is situation when you have a hardware device that provides data that is accessible only by calls to external (in relation to Qt) library.

[SGaist]

In that kind of cases, I usually write a backend for QtMultimedia to integrated that device within the multimedia workflow.