Relationships between qmake, make, nmake, jom, and the compiler

Anonymous_Banned275

[Forked from https://forum.qt.io/topic/119138/high-cpu-usage/ --JKSH]

@Ketan__Patel__0011
For MSVC builds, Qt uses jom instead of nmake, to build in parallel, use more CPU and speed up the building.
You can refer to:
https://wiki.qt.io/Jom
https://www.qt.io/blog/2009/03/27/speeding-up-visual-c-qt-builds

For MSVC builds, Qt uses jom instead of nmake, to build in parallel, use more CPU and speed up the building.

Nice , but maybe little awkward , as far as Qt goes ,explanation.

Ever since i ventured into Qt land I have been "confused" with relation between "make" and "qmake" . And lack of configuring compiler/ linker options to boot. ( OK, they maybe there, but not very visible .)

I have been using "-j" option - in another IDE.
As far as I know, but I may be wrong, -j is standard (compiler) option, not sure why "jom" exists.

Now in QTCreator I am "making arguments" for "make" to use "-j" .

So "who is on first ? compiler or "Make" ?

Sorry for the rant.

Maybe somebody knows how to "explain" how setting "-jn" option to SPECIFIC n calculates the CPU usage.

To illustrate the original post - let's set the (optional !) n to 1 , just emulating the MSx behaviour on single core CPU.

PS
I hope my rant did to steer the discussion into wrong direction.

SGaist

@AnneRanch said in High CPU usage:

As far as I know, but I may be wrong, -j is standard (compiler) option, not sure why "jom" exists.

Because nmake (provided by VS) was more than bad at parallelising compilation when jom was created (it might still be but I haven't tested that recently)

It's not a gcc argument. As the screenshot you took says, it's a make argument. The -j argument is to instruct how many jobs to start in parallel. To use all available resources, you usually use the number of available cores of your machine as value. If you have hyper threading, you can multiply that value by two. In any case, never forget to put a number behind -j otherwise make is going to start working on everything it can so if you try to build for example the Linux kernel, you will have quite a lot of jobs started.

In any case, as all my fellow developers already wrote: using as much core as available to build an application is done to minimise build time. If it's really an issue for some reason, it's always possible to force single core use by passing the -j1.

Anonymous_Banned275

@SGaist said in High CPU usage:

otherwise make is going to start working on everything it can

OK , I do not want to hijack this , but please elaborate on the above quote. It may be just semantic / or my English translation .

"-j" lets "make" to utilize all cores - while sharing them with other processes as necessary

"-jn" gives "make" exclusive (?) access to n cores

So in case of 4 cores "-j" and "-j4" cannot be same - if my "exclusive " opinion correct.

JohanSolo

@AnneRanch said in High CPU usage:

@SGaist said in High CPU usage:

otherwise make is going to start working on everything it can

OK , I do not want to hijack this , but please elaborate on the above quote. It may be just semantic / or my English translation .

"-j" lets "make" to utilize all cores - while sharing them with other processes as necessary

"-jn" gives "make" exclusive (?) access to n cores

So in case of 4 cores "-j" and "-j4" cannot be same - if my "exclusive " opinion correct.

-j starts all jobs that can run in parallel (which most certainly will cause lots of swapping), this should in principle not be used. -jn or -j n allows up to n jobs to run in parallel.

JKSH

@AnneRanch

These tools read your high-level project description files and then produce Makefiles:

qmake (Qt-specific, reads *.pro files)
CMake (not Qt-specific, reads CMakeLists.txt files)

These tools read Makefiles and then run your compiler + linker:

make
nmake
jom

So "who is on first ? compiler or "Make" ?

Putting the info above together...

qmake/CMake is first
make/nmake/jom is second
Your compiler is third

Anonymous_Banned275

@JohanSolo said in Relationships between qmake, make, nmake, jom, and the compiler:

-j starts all jobs that can run in parallel (which most certainly will cause lots of swapping), this should in principle not be used. -jn or -j n allows up to n jobs to run in parallel.

This still makes little sense to me.

Let's define / start with "job" objective -the goal is to obtain maximum speed ( efficiency) while building the Qt project.
That implies , to me, to let tools to utilize all CPU resources - cores in multi core CPU.
Using "-j" or jom option lets the tool do such tasks "automatically".
If the build tool competes for resources ( swapping (?) ) it is pretty obvious , but still should be more efficient than choking the build process manually - such as specifying "n" in "-jn" .

Hence if the objective is efficiency it can be easy verified by observing compiler output last entry:

08:39:54: The process "/usr/bin/make" exited normally.
08:39:54: Elapsed time: 00:51.

Cheers

PS Very informative discussion, thanks to all participants.

SGaist

No it does not, -j allows for proper parallelization as long as you give it correct values. For example you might be doing some other stuff requiring also CPU power while building so you can limit make to use for example half the cores you have and it will happily generate the jobs. Make does not guess anything. "-j" without any argument just tells it to start absolutely everything at the same time. Make is not a smart orchestrator. That is not its job.

Anonymous_Banned275

"-j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run simultaneously. If there is more than one -j option, the last one is effective. If the -j option is given without an argument, make will not limit the number of jobs that can run simultaneously. "