Build with mingw32-make *very* slow

aha_1980

Hi all,

I'm currently trying to compile a QMake subdirs project consisting of 10 .profiles with Qt 5.9.4 / MinGW 5.3 / Windows 7 x64 running as Guest in VirtualBox 5.2.6 (Guest additions are installed).

My problem is, that it takes 10...11 seconds for mingw32-make to go through all the Makefiles and recognize that nothing is changed (and therefore nothing has to be build). I tried to use a ramdisk but that does not change anything.

On my workstation running Windows 7 directly, the same takes approx. 3 seconds, which is acceptable.

Any ideas or hints for this? Thanks in advance.

14:20:05: Running steps for project xxxxControl...
14:20:05: Configuration unchanged, skipping qmake step.
14:20:05: Starting: "C:\Qt\Tools\mingw530_32\bin\mingw32-make.exe" -j 8
cd xxxxHAL\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHAL.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
mingw32-make[1]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
( if not exist Makefile.xxxxHalCanLibrary C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile.xxxxHalCanLibrary R:\Daten\xxxxControl\xxxxHAL\xxxxHalCanLibrary.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile.xxxxHalCanLibrary 
cd xxxxHalRemoteClient\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalRemoteClient\xxxxHalRemoteClient.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
( if not exist Makefile.xxxxHalLibrary C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile.xxxxHalLibrary R:\Daten\xxxxControl\xxxxHAL\xxxxHalLibrary.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile.xxxxHalLibrary 
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteClient'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteClient'
cd xxxxHalRemoteTest\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalRemoteTest\xxxxHalRemoteTest.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
cd xxxxHalRemoteSimple\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalRemoteSimple\xxxxHalRemoteSimple.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
cd xxxxHalTerminal\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalTerminal\xxxxHalTerminal.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
( if not exist Makefile.xxxxHalExample C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile.xxxxHalExample R:\Daten\xxxxControl\xxxxHAL\xxxxHalExample.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile.xxxxHalExample 
cd xxxxHalRemoteServer\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalRemoteServer\xxxxHalRemoteServer.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
cd xxxxHalRemoteServer-direct\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxHAL\xxxxHalRemoteServer-direct\xxxxHalRemoteServer-direct.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteSimple'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteSimple'
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalTerminal'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalTerminal'
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteTest'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteTest'
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteServer'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteServer'
mingw32-make[2]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteServer-direct'
mingw32-make[2]: Nothing to be done for 'first'.
mingw32-make[2]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL/xxxxHalRemoteServer-direct'
mingw32-make[1]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxHAL'
cd xxxxControl\ && ( if not exist Makefile C:\Qt\5.9.4\mingw53_32\bin\qmake.exe -o Makefile R:\Daten\xxxxControl\xxxxControl\xxxxControl.pro -spec win32-g++ "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Qt/Tools/mingw530_32/bin/mingw32-make -f Makefile 
mingw32-make[1]: Entering directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxControl'
mingw32-make[1]: Nothing to be done for 'first'.
mingw32-make[1]: Leaving directory 'R:/Daten/build-xxxxControl-Desktop_Qt_5_9_4_MinGW_32bit-Debug/xxxxControl'
14:20:15: The process "C:\Qt\Tools\mingw530_32\bin\mingw32-make.exe" exited normally.
14:20:15: Elapsed time: 00:11.```

raven-worx

@aha_1980
most probably this is a config issue?!
How many (virtual) CPUs do you have configured?
Did you enable CPU virtualization - if available (in BIOS settings)?
...

aha_1980

Hi @raven-worx,

thanks for your answer.

I just checked the BIOS settings (it's a Ryzen 7 1800X btw.):

PSS Support: Enabled (looks ACPI related)
NX Mode: Enabled (No-execute page protection)
SVM Mode: Enabled (CPU virtualization)
SMT Mode: Auto (simultaneous multithreading)
Core Leveling Mode: Automatic (number of cores per processor)

and I assigned 8 cores to the VM.

I don't mind that the compile speed is a bit slower, but the long turnaround times are quite annoying.

aha_1980

Update: I changed mingw32-make to jom in Creators build steps and that massively improved things.

The check for changed files now takes under 1 second!

I'm not sure if this setup is supported at all, but it looks promising.

koahnig

@aha_1980

May be you have checked already, but are you sure that enabled multiple cores for MinGW-make as well?
Respectively are you using parameter -j ?

kshegunov

@aha_1980 said in Build with mingw32-make *very* slow:

I'm not sure if this setup is supported at all, but it looks promising.

A tad late, but just for posterity:
I'm pretty sure mingw uses single core (due to its own limitations), so you have to use jom to get all of your cores to build. From what I've experienced jom is working quite fine and I haven't encountered any issues with it and Qt. In the end this is the part after qmake's run and it's rather unrelated to Qt's internals.

aha_1980

Hi, and thanks for all your answers.

@koahnig: Yes, I'm giving mingw32-make then -j 8 parameter to use my cores. But the slowness is already before compiling starts, when all the makefiles from my subdirs are checked. With mingw32-make you see the lines scrolling by and it seems to take a break at every 'Entering directory...' line. jom is just insanely fast.

The effect is much less visible if you have only one .pro file and therefore only one Makefile.

@kshegunov: Not too late at all :) Thanks, if you are going this way I'll try it too. I will see how it behaves if I'm working on some projects with it.

jsulm

@aha_1980 Could it be related to your anti-virus software?

aha_1980

Hi @jsulm,

no because there is no anti-virus ;) (these machines don't have physical internet connection so the risk is minimal)

aha_1980

I've marked this als solved, as I have a workaround for now. Thanks for all your suggestions!

koahnig

@aha_1980 said in Build with mingw32-make *very* slow:

@koahnig: Yes, I'm giving mingw32-make then -j 8 parameter to use my cores. But the slowness is already before compiling starts, when all the makefiles from my subdirs are checked. With mingw32-make you see the lines scrolling by and it seems to take a break at every 'Entering directory...' line. jom is just insanely fast.

That sounds interesting.
Are you simply overwritting mingw32-make with jom in qt creator?

kshegunov

@koahnig said in Build with mingw32-make *very* slow:

Are you simply overwritting mingw32-make with jom in qt creator?

This is how I've done it in the past - going to the build settings and putting jom in "Override make" in the build steps (and clean steps), then adding -j4 to the make arguments text fields. There may be a better way, though.

aha_1980

Hi @koahnig

Are you simply overwritting mingw32-make with jom in qt creator?

Yes, that's what I do.

There's also an old suggestion to have a jom checkbox for MinGW also. It was closed out of scope, but if there is enough interest maybe it can be reactived?

In contrast to @kshegunov I don't specify any -j X to jom as it takes all cores by default.

koahnig

@aha_1980 @kshegunov

Just to share my results.

I have a couple of layered "subdirs" summing up to 18 projects.

mingw32-make requires 25 seconds to check for a run when nothing has been changed.
jom about 4 seconds for same.

for rebuild the story is different
mingw32-make requires about 4:59
jom requires 6:11

The clean is definitely faster with jom, but apparently it looses later on time when starting for the different projects.
All checks are with compilation on ram disk with 8 cores

jom was not happy with my environment settings MAKEFLAGS=-j8

aha_1980

@koahnig: thanks for sharing.

The clean is definitely faster with jom, but apparently it looses later on time when starting for the different projects.
All checks are with compilation on ram disk with 8 cores

Interesting. As said, I'll do some more tests the next says and see how it behaves. For now jom is my winner.

jom was not happy my environment settings MAKEFLAGS=-j8

... but it does not need it on the other hand ;)

fem_dev

@aha_1980 said in Build with mingw32-make *very* slow:

I changed mingw32-make to jom in Creators build steps and that massively improved things.

How can I change the mingw32 to use JOM?

How can I install JOM? Is by MSYS2 console?

koahnig

@fem_dev said in Build with mingw32-make *very* slow:

@aha_1980 said in Build with mingw32-make *very* slow:

I changed mingw32-make to jom in Creators build steps and that massively improved things.

How can I change the mingw32 to use JOM?

You can change under build settings for the make step to jom. Do not remember if you have to do something in addition, but it was pretty straightforward at the time. However, Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

How can I install JOM? Is by MSYS2 console?
https://wiki.qt.io/Jom

fem_dev

@koahnig thank you,

Now I saw that the MSYS2 Qt Creator 4.12.4 installation already comes with jom inside the QtCreator\bin folder.
So, what I did:

1- Goto the Project's build settings and under the build steps category there is a "Make" item.
2- Expand the details on that and you'll find a "Override mingw32-make.exe:" item. Paste the jom location e.g. "C:\Qt\Tools\QtCreator\bin\jom.exe"

I follow this image that I found in the Stackoverfow:

After that, I deleted the build folder, cleaned my Qt project and rebuild it.

First Question

Qt Creator starts to use jom, but I got this compile time errors:

09:35:09: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited normally.
09:35:09: Starting: "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" -j4

jom 1.1.3 - empower your colors

cd App / && (test -e Makefile || C: /Qt/5.15.0/mingw81_64/bin/qmake.exe -o Makefile C: /Users/VM/Desktop/rotortest/App/App.pro -spec win32- g ++ CONFIG + = debug CONFIG + = qml_debug) && C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile
C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile.Debug
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/app.ui -o ui_app.h
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_app.h] Error 1
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/custom_widget/value_unit.ui -o ui_value_unit.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_assembly.ui -o ui_form_assembly.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_excitation.ui -o ui_form_excitation.h
The syntax of the file name, directory name, or volume label is incorrect.
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_assembly.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_value_unit.h] Error 1
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_excitation.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile [debug] Error 2
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ Makefile [sub-App-make_first] Error 2
09:35:10: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited with code 2.
Error while building / deploying project rotortest (kit: Desktop Qt 5.15.0 MinGW 64-bit)
When executing step "Make"

How can I fix it?

My system:

Qt Creator 4.12.4 (GCC based - downloaded by MSYS2)
Qt 5.15.0
Windows 10 x64
Mingw-w64
jom

Second question

@koahnig said in Build with mingw32-make *very* slow:

Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

Are you saying that you compiled your Qt project using the default MSYS2 Mingw-64 configuration (WITHOUT jom) and got a close time compilation speed compared with MSVC 2019?

koahnig

@fem_dev said in Build with mingw32-make *very* slow:

First Question

Qt Creator starts to use jom, but I got this compile time errors:

09:35:09: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited normally.
09:35:09: Starting: "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" -j4

jom 1.1.3 - empower your colors

cd App / && (test -e Makefile || C: /Qt/5.15.0/mingw81_64/bin/qmake.exe -o Makefile C: /Users/VM/Desktop/rotortest/App/App.pro -spec win32- g ++ CONFIG + = debug CONFIG + = qml_debug) && C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile
C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile.Debug
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/app.ui -o ui_app.h
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_app.h] Error 1
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/custom_widget/value_unit.ui -o ui_value_unit.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_assembly.ui -o ui_form_assembly.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_excitation.ui -o ui_form_excitation.h
The syntax of the file name, directory name, or volume label is incorrect.
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_assembly.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_value_unit.h] Error 1
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_excitation.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile [debug] Error 2
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ Makefile [sub-App-make_first] Error 2
09:35:10: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited with code 2.
Error while building / deploying project rotortest (kit: Desktop Qt 5.15.0 MinGW 64-bit)
When executing step "Make"

How can I fix it?

My system:

Qt Creator 4.12.4 (GCC based - downloaded by MSYS2)
Qt 5.15.0
Windows 10 x64
Mingw-w64
jom

Should be completely irrelevant. When your make file works with mingw-make, it has to work with jom as well. jomis a replacement for nmake because it did not allow -j parameter. AFAIK the MSVC does support directly multi-threading and it is not done through nmake.

Besides possible side effects through parallel access to the same file or whatever, there is no difference for compilation. You are listing compile errors which are due to the compiler you are using.

Second question

@koahnig said in Build with mingw32-make *very* slow:

Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

Are you saying that you compiled your Qt project using the default MSYS2 Mingw-64 configuration (WITHOUT jom) and got a close time compilation speed compared with MSVC 2019?

Nope. I do not use MSVC 2019 compiler at all. AFAIK there was an issue with an older MinGW-make not suporting multi-threading. Therefore this discussion and the solution with jom. After my reinstall I did not bother to set up jom again. At day's end some gain is possible by switching to jom, but it was not enough to go through the hassle again.

You seem to use a self-compiled Qt creator version. The output shows strange blanks around the backslashes. Possibly there is your problem with compilation.