AFAIK, the Qt 4.8 series has no release officially supporting such a recent version of macOS. Therefore you may have surprises. Note that Qt 4 has reached end of life so unless you are locked to that version, you should really consider updating to Qt 5 which is now at it's 5.9.0 release.
@raven-worx: while I'm not sure it is the right solution for what @tokafr wants to do, he can take ownership of the vertical scrollbar and have some influence this way. I have done it before that way.
@tokafr: Instead of trying to influence the hidden layout of the QTableView, you could mask the space between the header view and the vertical scrollbar to make it look like it being part of the header view. Stylesheets are probably the best place to try.