Yes, you do not have to copy whole root file system.
"I haven't found any info of what is needed" - what is needed heavilly depends on what parts of Qt you want to build. What you can do is: run configure with your current sysroot, see which tests fail and for those you need check why they fail (check configure.log file) and install missing dev packages.
@ishi Visual Studio is of no interest here as you cannot use it to cross compile to ARM Linux as far as I know. If your app is using OpenCV you will either need to install it on your RPi (including the -dev package) and sync the sysroot, or cross-compile it.
FYI: the issues vanished with the previous version Qt 5.14.2 and the 7.5.0 Linaro toolchain (but not the others!). Suddenly the tests are being run. My cross development toolchain now is up and running, however, I still find it odd that there are no error messages in Qt's build system.