Did you solve your problem?
I also met same problem.
I use debian stretch and Qt 5.11. Even after setting up all possible flags on rpi3, I can't get the sound out from the jack in my Qt app. Command line tests play sound through jack....
AND : If I boot without HDMI cable, sound goes out from jack, even after pluging it into rasp and displaying qt app.