Build with mingw32-make *very* slow

koahnig · wrote on 19 Feb 2018, 17:10

@aha_1980

May be you have checked already, but are you sure that enabled multiple cores for MinGW-make as well?
Respectively are you using parameter -j ?

kshegunov · wrote on 19 Feb 2018, 23:01

@aha_1980 said in Build with mingw32-make *very* slow:

I'm not sure if this setup is supported at all, but it looks promising.

A tad late, but just for posterity:
I'm pretty sure mingw uses single core (due to its own limitations), so you have to use jom to get all of your cores to build. From what I've experienced jom is working quite fine and I haven't encountered any issues with it and Qt. In the end this is the part after qmake's run and it's rather unrelated to Qt's internals.

aha_1980 · 20 Feb 2018, 07:56

Hi, and thanks for all your answers.

@koahnig: Yes, I'm giving mingw32-make then -j 8 parameter to use my cores. But the slowness is already before compiling starts, when all the makefiles from my subdirs are checked. With mingw32-make you see the lines scrolling by and it seems to take a break at every 'Entering directory...' line. jom is just insanely fast.

The effect is much less visible if you have only one .pro file and therefore only one Makefile.

@kshegunov: Not too late at all :) Thanks, if you are going this way I'll try it too. I will see how it behaves if I'm working on some projects with it.

jsulm · wrote on 20 Feb 2018, 07:56

@aha_1980 Could it be related to your anti-virus software?

aha_1980 · wrote on 20 Feb 2018, 08:04

Hi @jsulm,

no because there is no anti-virus ;) (these machines don't have physical internet connection so the risk is minimal)

aha_1980 · wrote on 20 Feb 2018, 09:01

I've marked this als solved, as I have a workaround for now. Thanks for all your suggestions!

koahnig · 20 Feb 2018, 11:50

@aha_1980 said in Build with mingw32-make *very* slow:

@koahnig: Yes, I'm giving mingw32-make then -j 8 parameter to use my cores. But the slowness is already before compiling starts, when all the makefiles from my subdirs are checked. With mingw32-make you see the lines scrolling by and it seems to take a break at every 'Entering directory...' line. jom is just insanely fast.

That sounds interesting.
Are you simply overwritting mingw32-make with jom in qt creator?

kshegunov · wrote on 20 Feb 2018, 11:50

@koahnig said in Build with mingw32-make *very* slow:

Are you simply overwritting mingw32-make with jom in qt creator?

This is how I've done it in the past - going to the build settings and putting jom in "Override make" in the build steps (and clean steps), then adding -j4 to the make arguments text fields. There may be a better way, though.

aha_1980 · wrote on 20 Feb 2018, 12:58

Hi @koahnig

Are you simply overwritting mingw32-make with jom in qt creator?

Yes, that's what I do.

There's also an old suggestion to have a jom checkbox for MinGW also. It was closed out of scope, but if there is enough interest maybe it can be reactived?

In contrast to @kshegunov I don't specify any -j X to jom as it takes all cores by default.

koahnig · 20 Feb 2018, 16:17

@aha_1980 @kshegunov

Just to share my results.

I have a couple of layered "subdirs" summing up to 18 projects.

mingw32-make requires 25 seconds to check for a run when nothing has been changed.
jom about 4 seconds for same.

for rebuild the story is different
mingw32-make requires about 4:59
jom requires 6:11

The clean is definitely faster with jom, but apparently it looses later on time when starting for the different projects.
All checks are with compilation on ram disk with 8 cores

jom was not happy with my environment settings MAKEFLAGS=-j8

aha_1980 · wrote on 20 Feb 2018, 16:17

@koahnig: thanks for sharing.

The clean is definitely faster with jom, but apparently it looses later on time when starting for the different projects.
All checks are with compilation on ram disk with 8 cores

Interesting. As said, I'll do some more tests the next says and see how it behaves. For now jom is my winner.

jom was not happy my environment settings MAKEFLAGS=-j8

... but it does not need it on the other hand ;)

fem_dev · 30 Jul 2020, 09:19

@aha_1980 said in Build with mingw32-make *very* slow:

I changed mingw32-make to jom in Creators build steps and that massively improved things.

How can I change the mingw32 to use JOM?

How can I install JOM? Is by MSYS2 console?

koahnig · 30 Jul 2020, 12:49

@fem_dev said in Build with mingw32-make *very* slow:

@aha_1980 said in Build with mingw32-make *very* slow:

I changed mingw32-make to jom in Creators build steps and that massively improved things.

How can I change the mingw32 to use JOM?

You can change under build settings for the make step to jom. Do not remember if you have to do something in addition, but it was pretty straightforward at the time. However, Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

How can I install JOM? Is by MSYS2 console?
https://wiki.qt.io/Jom

fem_dev · 30 Jul 2020, 14:05

@koahnig thank you,

Now I saw that the MSYS2 Qt Creator 4.12.4 installation already comes with jom inside the QtCreator\bin folder.
So, what I did:

1- Goto the Project's build settings and under the build steps category there is a "Make" item.
2- Expand the details on that and you'll find a "Override mingw32-make.exe:" item. Paste the jom location e.g. "C:\Qt\Tools\QtCreator\bin\jom.exe"

I follow this image that I found in the Stackoverfow:

After that, I deleted the build folder, cleaned my Qt project and rebuild it.

First Question

Qt Creator starts to use jom, but I got this compile time errors:

09:35:09: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited normally.
09:35:09: Starting: "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" -j4

jom 1.1.3 - empower your colors

cd App / && (test -e Makefile || C: /Qt/5.15.0/mingw81_64/bin/qmake.exe -o Makefile C: /Users/VM/Desktop/rotortest/App/App.pro -spec win32- g ++ CONFIG + = debug CONFIG + = qml_debug) && C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile
C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile.Debug
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/app.ui -o ui_app.h
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_app.h] Error 1
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/custom_widget/value_unit.ui -o ui_value_unit.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_assembly.ui -o ui_form_assembly.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_excitation.ui -o ui_form_excitation.h
The syntax of the file name, directory name, or volume label is incorrect.
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_assembly.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_value_unit.h] Error 1
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_excitation.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile [debug] Error 2
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ Makefile [sub-App-make_first] Error 2
09:35:10: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited with code 2.
Error while building / deploying project rotortest (kit: Desktop Qt 5.15.0 MinGW 64-bit)
When executing step "Make"

How can I fix it?

My system:

Qt Creator 4.12.4 (GCC based - downloaded by MSYS2)
Qt 5.15.0
Windows 10 x64
Mingw-w64
jom

Second question

@koahnig said in Build with mingw32-make *very* slow:

Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

Are you saying that you compiled your Qt project using the default MSYS2 Mingw-64 configuration (WITHOUT jom) and got a close time compilation speed compared with MSVC 2019?

koahnig · wrote on 30 Jul 2020, 14:05

@fem_dev said in Build with mingw32-make *very* slow:

First Question

Qt Creator starts to use jom, but I got this compile time errors:

09:35:09: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited normally.
09:35:09: Starting: "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" -j4

jom 1.1.3 - empower your colors

cd App / && (test -e Makefile || C: /Qt/5.15.0/mingw81_64/bin/qmake.exe -o Makefile C: /Users/VM/Desktop/rotortest/App/App.pro -spec win32- g ++ CONFIG + = debug CONFIG + = qml_debug) && C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile
C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe -f Makefile.Debug
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/app.ui -o ui_app.h
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_app.h] Error 1
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/custom_widget/value_unit.ui -o ui_value_unit.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_assembly.ui -o ui_form_assembly.h
'C: \ Qt \ 5.15.0 \ mingw81_64 \ bin \ uic.exe' ../../rotortest/App/form/form_excitation.ui -o ui_form_excitation.h
The syntax of the file name, directory name, or volume label is incorrect.
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_assembly.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_value_unit.h] Error 1
The syntax of the file name, directory name, or volume label is incorrect.
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile.Debug [ui_form_excitation.h] Error 1
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ App \ Makefile [debug] Error 2
jom: C: \ Users \ VM \ Desktop \ build-rotortest-Desktop_Qt_5_15_0_MinGW_64_bit-Debug \ Makefile [sub-App-make_first] Error 2
09:35:10: The process "C: \ Qt \ Tools \ QtCreator \ bin \ jom.exe" exited with code 2.
Error while building / deploying project rotortest (kit: Desktop Qt 5.15.0 MinGW 64-bit)
When executing step "Make"

How can I fix it?

My system:

Qt Creator 4.12.4 (GCC based - downloaded by MSYS2)
Qt 5.15.0
Windows 10 x64
Mingw-w64
jom

Should be completely irrelevant. When your make file works with mingw-make, it has to work with jom as well. jomis a replacement for nmake because it did not allow -j parameter. AFAIK the MSVC does support directly multi-threading and it is not done through nmake.

Besides possible side effects through parallel access to the same file or whatever, there is no difference for compilation. You are listing compile errors which are due to the compiler you are using.

Second question

@koahnig said in Build with mingw32-make *very* slow:

Mingw-make does support now multi-threading. I did some tests back then, but after a reinstall, there was no need anymore.

Are you saying that you compiled your Qt project using the default MSYS2 Mingw-64 configuration (WITHOUT jom) and got a close time compilation speed compared with MSVC 2019?

Nope. I do not use MSVC 2019 compiler at all. AFAIK there was an issue with an older MinGW-make not suporting multi-threading. Therefore this discussion and the solution with jom. After my reinstall I did not bother to set up jom again. At day's end some gain is possible by switching to jom, but it was not enough to go through the hassle again.

You seem to use a self-compiled Qt creator version. The output shows strange blanks around the backslashes. Possibly there is your problem with compilation.