I can relate to the points you raise regarding the use of transformations. Some years ago, I have written pretty much what you describe, but I was never happy with the complex code that resulted.
It worked well, though, except for two issues which I couldn't find a solution for:
Cosmetic pens of width != 1 paint super slow (I needed them to keep a constant line width while zooming). Here the solution would be, as you write, to calculate everything up front and paint in screen space, but the code was too progressed to make that change
I used QGraphicsProxyWidget to integrate QScrollbars into my scene, which sometimes simply wouldn't repaint, no matter what I did (5.15.2, might have been fixed later)
Now, I am involved in coding a similar component from scratch, using QML and SceneGraph. QML for layouting and custom QML components written in C++ for the heavy lifting. It seems to work rather well, but documentation and exacmples on SceneGraph are thin.