Examples for Image Processing using GLSL on Qt.

[ryocchin]

This is a simple example to learn how to use GLSL for image processing on Qt. All source files can be obtained from this repository:
https://github.com/ryocchin/QtShaderExample

This tutorial includes three fundamental image filters, inverse(negative), mosaic and median filter.

[1] Initializing shader programs and frame buffer objects

The interface is simple. The main window has one widget for rendering OpenGL graphics, which is derived from QGLWidget. It needs to inherit QOpenGLFunctions, which provides cross-platform access to the OpenGL API.

@
class frag_pane : public QGLWidget, protected QOpenGLFunctions
{
private:
QOpenGLShaderProgram _copy_program;
QOpenGLShaderProgram _inverse_program;
...
QGLFramebufferObject *_in_fbo;
QGLFramebufferObject *_out_fbo;
@

I added two FBO(FrameBuffer Object) to store the original image and processed image, respectively. Actually, I used QGLFramebufferObject, Qt Class which encapsulates an OpenGL FBO.

@
initializeOpenGLFunctions( );
@

When this widget is initialized, initializeOpenGLFunctions() must be called. Shaders can be created in two ways. Programs can be embedded in the source files and complied as instance of QOpenGLShader. This is an example of an embedded shader program.

@
QOpenGLShader *vshader1 = NULL;
QOpenGLShader *fshader1 = NULL;

const char *vsrc1 =
"varying vec2 pos;\n"
"void main( void )\n"
"{\n"
...
"}\n";

const char *fsrc1 =
"uniform sampler2D image;\n"
"uniform int imgWidth;\n"
"uniform int imgHeight;\n"
...
" gl_FragColor.b = 1.0 - col.b;\n"
"}\n";

vshader1 = new QOpenGLShader(QOpenGLShader::Vertex, this);
fshader1 = new QOpenGLShader(QOpenGLShader::Fragment, this);

if( !vshader1->compileSourceCode(vsrc1) ) {
return false;
}
...
if( !_inverse_program.addShader(vshader1) ) {
return false;
}
...
if( !_inverse_program.link( ) ) {
return false;
}
@

Fragment and vertex shaders have to be compiled before they are added to shader programs and linked. The second way to create shader program is to add them from separate source files(I do not go into how to write GLSL).

@
//
// Fragment shader for mosaic image
//
uniform sampler2D image;
uniform int imgWidth;
uniform int imgHeight;
uniform int nPixels; // block size
varying vec2 pos;

void main(void)
{
vec2 texCoord;
int iCol = floor(pos.x / float(nPixels));
int iRow = floor(pos.y / float(nPixels));

int i, j;
vec4 sum = vec4(0.0);
int cnt = 0;
...
gl_FragColor.rgb = sum.rgb / (float(cnt));
}

//
// Vertex shader for mosaic image
//
varying vec2 pos;

void main(void)
{
pos = gl_Vertex.xy;
gl_Position = ftransform( );
}
@

These GLSL fragment and vertex programs are saved in two files, 'mosaic.frag' and 'mosaic.vert' and they are added to the program's resource using Qt Resource Editor(Qt Creator).

!https://raw.github.com/ryocchin/QtShaderExample/master/doc/resource_editor.jpg(Adding shaders to resource)!

Shader programs are complied from source and then linked to the shader object.

@
...
// Compile vertex shader
if( !pShader->addShaderFromSourceFile( QOpenGLShader::Vertex, strVertexFile ) ) {
return false;
}

// Compile fragment shader
if( !pShader->addShaderFromSourceFile( QOpenGLShader::Fragment, strFragmentFile ) ) {
return false;
}

if( !pShader->link( ) ) {
return false;
}
@

[2] Reading the image

Image is read into input FBO. FBO is (re-)created according to the dimension of the image and the read image is stored in the internal texture of the FBO.

@
...
reader.setFileName( strFile );
...
makeCurrent( );
_in_fbo = new QGLFramebufferObject( nWidth, nHeight, GL_TEXTURE_2D );
if( !_in_fbo ) {
goto PIX_EXIT;
}
...
_in_fbo->bind( );
glTexImage2D( GL_TEXTURE_2D, 0, GL_RGBA, nWidth, nHeight,
0, GL_RGBA, GL_UNSIGNED_BYTE, image );
...
_in_fbo->release();
@

[3] Image processing

Output FBO is chosen as the target of rendering. The texture in the input FBO is drawn to the output FBO in 'drawTexture(...)'. When this occurs, image enhancement algorithm defined in the bound fragment shader is applied. Shader has to be bound before the texture is copied. Shader program has to be released so that it no longer is used when texture is displayed on the screen. FBO's are also released.

@

makeCurrent( );
_out_fbo = new QGLFramebufferObject( nWidth, nHeight, GL_TEXTURE_2D );
if( !_out_fbo ) {
goto PIX_EXIT;
}

bindShader( );
_out_fbo->bind( );
...
_out_fbo->drawTexture( QPointF(0.0,0.0), _in_fbo->texture( ), GL_TEXTURE_2D );

releaseShader( );
_out_fbo->release();
@

[4] Displaying the image

In this example, the whole texture is simply shown at the lower left corner of the viewport. The texture in the output FBO is bound for rendering to the screen. Then, the entire area of the bound texture is rendered.

[5] Saving the Image

The processed image in the output FBO can be saved to the file. The texture in the output FBO is saved in one of the image formats supported by QImageWriter.

@
...
_out_fbo->bind( );
glEnable( GL_TEXTURE_2D );
glViewport( 0, 0, nWidth, nHeight );
...
glBindTexture( GL_TEXTURE_2D, _out_fbo->texture( ) );

image = new GLubyte[nHeightnWidth4];
if( !image ) {
return false;
}
glReadPixels( 0, 0, nWidth, nHeight, GL_RGB, GL_UNSIGNED_BYTE,
image );
...
writer.setFileName( strFile );
if( !writer.write( *img ) ) {
return false;
}
@

This is all this example has. I hope this is helpful and I appreciate any feedback.

[Vincent007]

There are some run-time errors.

QOpenGLShader::compile(Fragment): Fragment shader failed to compile with the fol
lowing errors:
ERROR: 1:19: error(#160) Cannot convert from: "float" to: "mediump int"
ERROR: 1:20: error(#160) Cannot convert from: "float" to: "mediump int"
ERROR: error(#273) 2 compilation errors. No code generated

*** Problematic Fragment shader source code ***
#define lowp
#define mediump
#define highp
//
// Fragment Shader for Image processing
//
// Mosaic Filter
//

//
// Fragment shader for mosaic image
//
uniform sampler2D image;
uniform int imgWidth;
uniform int imgHeight;
uniform int nPixels; // block size
varying vec2 pos;

void main(void)
{
vec2 texCoord;
int iCol = floor(pos.x / float(nPixels));
int iRow = floor(pos.y / float(nPixels));

// Mosaic by Average
int i, j;
vec4 sum = vec4(0.0);
int cnt = 0;
for(int j = iRownPixels ; j < iRownPixels + nPixels ; j++) {
for(int i = iColnPixels; i < iColnPixels + nPixels; i++) {
texCoord = vec2(float(i) / float(imgWidth),
float(j) / float(imgHeight));
sum += texture2D(image, texCoord);
cnt ++;
}
}

gl_FragColor.rgb = sum.rgb / (float(cnt));

}

---

Unable to compile fragment shader. Log: "Fragment shader failed to compile with
the following errors:
ERROR: 1:19: error(#160) Cannot convert from: "float" to: "mediump int"
ERROR: 1:20: error(#160) Cannot convert from: "float" to: "mediump int"
ERROR: error(#273) 2 compilation errors. No code generated

"
ERROR! Failed to Create Mosaic Shader.
ERROR! Failed to create shader programs.

[ryocchin]

Thank you, Vincent007.
I fixed mosaic.frag. Can you please retry?

I guess these two lines, which only issued warning but worked fine on my computer, were considered as fatal error on other system.

@
int iCol = floor(pos.x / float(nPixels));
int iRow = floor(pos.y / float(nPixels));
@

[Vincent007]

no run-time error now!! well done.

There is a minor issue. When I switch to another application, and then switch back to this application, I see a message is printed out "QOpenGLContext::swapBuffers() called without corresponding makeCurrent()".

[ryocchin]

The error was caused by these two lines in fragment shader:

@
int iCol = floor(pos.x / float(nPixels) );
int iRow = floor(pos.y / float(nPixels) );
@

I knew I should not use implicit cast in GLSL program. But, it was overlooked as it only issued following warnings when I compiled on my computer and the program actually worked as I intended.

@
0(46) : warning C7011: implicit cast from "int" to "float"
0(46) : warning C7011: implicit cast from "int" to "float"
@

Now, thanks to the feed back from Vincent, I learned that these are taken as fatal errors on some other systems and execution cannot continue;

@
ERROR: 1:19: error(#160) Cannot convert from: “float” to: “mediump int”
ERROR: 1:20: error(#160) Cannot convert from: “float” to: “mediump int”
ERROR: error(#273) 2 compilation errors. No code generated
@

So, I changed the above lines as below and now it seems to work fine.

@
int iCol = int( floor(pos.x / float(nPixels)) );
int iRow = int( floor(pos.y / float(nPixels)) );
@

I am not sure whether this difference arises because of difference Qt version or some other system settings. I appreciate if someone has any suggestion.

I will look into another warning Vincent mentioned.