Converting screen coordinates back to world coordinates for OpenGL picking

[dalishi]

Hi,

I am working on my OpenGL picking using ray casting. I am stuck at converting screen coordinates (mouse clicked 2D point) to 3D world coordinates.

I was trying to use QVector3D::unproject(), not sure if this is the correct way in Qt to do this. The question is how is the Z value I should use with the clicked X and Y for the conversion. From below code snippet you can see I get the depth value using glReadPixels with GL_DEPTH_COMPONENT even though this enum param is not supported by OpenGL ES2.0. The depth value I retrieved is always clamped into [0 1] and the resulted 3D coordinates is also not correct. Can anyone help me out with this? Thanks.

// read the depth value posZ of the clicked point (posX, posY) on screen
glReadPixels(posX, posY, 1, 1, GL_DEPTH_COMPONENT, GL_FLOAT, &posZ);
// calculate viewmodel matrix
QMatrix4x4 modelViewMatrix = viewMatrix * modelMatrix;
// get the viewport rectangle
const qreal retinaScale = devicePixelRatio();
QRect viewPortRect(0, 0, width() * retinaScale, height() * retinaScale);
// unproject to 3D
QVector3D clickedPointIn3DWorld =
            QVector3D(posX, posY, posZ).unproject(modelViewMatrix, projectionMatrix, viewPortRect);