You're calling QDialog::exec which is for modal-only dialogs, and Qt::Popup isn't for dialogs, leave the window flags be. If you prepare a MWE (the download-and-build type) I can test on Linux (I have no Mac, sorry). Also you might consider filing a bug report if everything else fails.
I highly recommend re-read C++ book until you understand at least the basics or search for c++ novice forum. Your questions have nothing to do with Qt yet. This is not an offence. But I doubt you will be getting help you need here due to specialization of this forum.
You can't load Qt dlls when you want to simply because they are linked to your program and need to be resolved at startup. That's just how it works. If you were to load Qt dlls when you wanted (e.g. using QLibrary) you would have to also manually resolve addresses of all functions and classes. It's possible but just impractical at this scale.
But here are two ideas what you could do:
One way would be to split the app in two. First one would be the loader (the dialog) and it would use QProcess to start the main app when it finishes its work. Handy for updating the main app too as it's a separate file.
The other idea would be to make your main app a shared library and load it dynamically with QLibrary at the right time. Same benefit as above except an exe and a library instead of two exes.