I think I have found the problem. I went through the source code of the QGraphicsTextItem and came across the private _q_updateBoundingRect. In there there is a line:
if (size == qq->m_boundingRect.size() || pageSize.height() != -1)
return;
It looks like the pageSize.height() != -1 part is the problem. I have removed it and it seems to be working!