QVector3D::project/unproject returning incorrect results between clipping planes

mayaknife

I'm finding that the QVector3D::project() and unproject() functions will give me the correct results for points which lie on or extremely close to the near and far clipping planes, but not for points in-between.

I've set up my projection and modelView matrices as follows:

QVector3D projection;
projection.setToIdentity();
projection.perspective(60.0f, 4.0f/3.0f, 0.1f, 100.0f);

QVector3D modelView, eye(15, 9, 15), centerOfInterest(0, 0, 0), up(0, 1, 0);
modelView.setToIdentity();
modelView.lookAt(eye, centerOfInterest, up);

The center of my view has pixel coordinates (459, 386). If I unproject that at depth 0 I get back a worldspace point on the near clipping plane:

QVector3D testPt(459, 386, 0);
QVector3D worldPt = testPt.unproject(modelView, projection, viewRect);

--> (14.9349, 8.96094, 14.9349)

viewRect is (0, 67) to (917, 704).

Similarly, unprojecting that same pixel coord at depth 1 gives me a valid point on the far clipping plane: (459, 386, 1) -> (-50.0951, -30.0575, -50.0951)

However, if I attempt to unproject that same pixel coord at any depth between 0.1 and 1 the point I get back is always very close to the near clipping plane. For example, unprojecting (459, 386, 0.9) should give me back a point close to the far clipping plane. Instead it gives me (14.3549, 8.61292, 14.3549), which is only slightly farther from the near clipping plane than was (459, 386, 0).

project() has the opposite problem: any worldspace position which isn't very close to the near plane gets projected to a point with the correct pixel coords, but a depth which is very close to the far plane. For example:

QVector3D worldPt(7.5, 4.5, 7.5);
QVector3D viewPt = worldPt.project(modelViewMatrix(), projectionMatrix(), viewRect);

--> (459, 386, 0.992313)

(7.5, 4.5, 7.5) is about one-eighth of the distance between the near and far planes, but the returned depth of 0.992313 is over 99% of the distance.

The source for QVector3D::project() and unproject() is as follows:

QVector3D QVector3D::project(const QMatrix4x4 &modelView, const QMatrix4x4 &projection, const QRect &viewport) const
{
    QVector4D tmp(*this, 1.0f);
    tmp = projection * modelView * tmp;
    if (qFuzzyIsNull(tmp.w()))
        tmp.setW(1.0f);
    tmp /= tmp.w();
    tmp = tmp * 0.5f + QVector4D(0.5f, 0.5f, 0.5f, 0.5f);
    tmp.setX(tmp.x() * viewport.width() + viewport.x());
    tmp.setY(tmp.y() * viewport.height() + viewport.y());
    return tmp.toVector3D();
}

QVector3D QVector3D::unproject(const QMatrix4x4 &modelView, const QMatrix4x4 &projection, const QRect &viewport) const
{
    QMatrix4x4 inverse = QMatrix4x4( projection * modelView ).inverted();
    QVector4D tmp(*this, 1.0f);
    tmp.setX((tmp.x() - float(viewport.x())) / float(viewport.width()));
    tmp.setY((tmp.y() - float(viewport.y())) / float(viewport.height()));
    tmp = tmp * 2.0f - QVector4D(1.0f, 1.0f, 1.0f, 1.0f);
    QVector4D obj = inverse * tmp;
    if (qFuzzyIsNull(obj.w()))
        obj.setW(1.0f);
    obj /= obj.w();
    return obj.toVector3D();
}

I've checked and the 'qFuzzyIsNull' calls are never true, so it's not a problem with degenerate w coords.

Does anyone have an idea of what's going on here? Is there something that I'm doing wrong?