Can't compile with jom in Windows - Makefile has bash syntax in it
-
This is relatively tangential, but is probably of interest to the crowd. For context, I'm using Qt compilation as a CPU benchmark. The configuration I have skips almost everything to keep the build time reasonable (i.e. "-no-dbus -no-open -skip this -skip that, etc etc")
After some informal tests on my own daily desktop, the build process (configure and make) with jom/MSVC takes around 28 minutes. Using MinGW 7.3 with a "-j 8" takes significantly longer, at around 39 minutes. That's almost a 40% difference. I did double-check that there were no errors and that a subsequent "make install"
I did verify that ming32-make was actually pulling in multiple jobs at the same time and kept an eye on Task Manager. (I've actually used make -j X before with an older Qt for the sampe exact purpose). From what I can gather, two things are different between jom/make.
The first one is that "configure" step with jom/MSVC seems to be multi-threaded itself, as it runs significantly faster compared to MinGW. That alone can easily count for a couple minutes' difference.
Then, the actual compilation with jom just seems to hit the CPU harder overall, for some reason, judging by the time delta and the machines's overall responsiveness while the compile job was running. For the record, Windows reports 100% CPU usage on both instances. I'll be running some tests on a dedicated box and report back.
-
This is relatively tangential, but is probably of interest to the crowd. For context, I'm using Qt compilation as a CPU benchmark. The configuration I have skips almost everything to keep the build time reasonable (i.e. "-no-dbus -no-open -skip this -skip that, etc etc")
After some informal tests on my own daily desktop, the build process (configure and make) with jom/MSVC takes around 28 minutes. Using MinGW 7.3 with a "-j 8" takes significantly longer, at around 39 minutes. That's almost a 40% difference. I did double-check that there were no errors and that a subsequent "make install"
I did verify that ming32-make was actually pulling in multiple jobs at the same time and kept an eye on Task Manager. (I've actually used make -j X before with an older Qt for the sampe exact purpose). From what I can gather, two things are different between jom/make.
The first one is that "configure" step with jom/MSVC seems to be multi-threaded itself, as it runs significantly faster compared to MinGW. That alone can easily count for a couple minutes' difference.
Then, the actual compilation with jom just seems to hit the CPU harder overall, for some reason, judging by the time delta and the machines's overall responsiveness while the compile job was running. For the record, Windows reports 100% CPU usage on both instances. I'll be running some tests on a dedicated box and report back.
Just in case you wonder: It is long known that MSVC is the fastest compiler for Windows. gcc on Linux is in the same league, probably a bit faster (the Linux file system is really good for tasks like compiling). MinGW, which is gcc on Windows is significantly slower. I don't know how Clang behaves on Windows, but IIRC, some people have great hopes in it.
Qt has to stay free or it will die.
-
Just in case you wonder: It is long known that MSVC is the fastest compiler for Windows. gcc on Linux is in the same league, probably a bit faster (the Linux file system is really good for tasks like compiling). MinGW, which is gcc on Windows is significantly slower. I don't know how Clang behaves on Windows, but IIRC, some people have great hopes in it.
-
@aha_1980, that's very useful information, thanks!
Other than having a benchmark that hits hard, another concern with this is ensuring it scales on many-core CPUs and the like. I'm guessing that'll depend as much on jom as it does on the compiler proper, but I should also probably MSVC to deliver on that front as well.
-
@aha_1980
Very interesting!the Linux file system is really good for tasks like compiling
Excellent! Why?
-
For the record, a while back we (site I work for) actually benchmarked compilation on HDDs vs SSDs. While I'm sure that things have changed in the meantime, we found that for the actual build step, I/O played a very, very minor part.
@morphine
Well using a ramdisk at work when compiling inside virtual machine
speed things up a lot. especially the linking part.
so while difference between spindisk (newer) and SSD might not
do much, upgrading to 6000/6000 MB/s ( ram disk speed)
did consistently work faster.
Its just a note. didn't really do much testing. -
We did not, in fact, test that scenario. You've piqued my interest, maybe I will at some point :)
@morphine
Well building Qt on ramdrive does take lots of ram :)
But real test could be fun. to see how much it matters. (for real)
for our project is just 20-25 secs. (out of 2 minutes) but
without proper testing it could be other factors. -
@morphine
Well using a ramdisk at work when compiling inside virtual machine
speed things up a lot. especially the linking part.
so while difference between spindisk (newer) and SSD might not
do much, upgrading to 6000/6000 MB/s ( ram disk speed)
did consistently work faster.
Its just a note. didn't really do much testing. -
Well using a ramdisk at work when compiling inside virtual machine
Now what is that "ramdisk" exactly? My Windows hosts Linux in a VirtualBox VM, are you saying I could have some sort of ramdisk in the Linux VM?
@JonB
well its a block device in ram. like a normal disk.
and yes inside the wm.
https://www.jamescoyle.net/how-to/943-create-a-ram-disk-in-linuxon windows i use
Softperfect ram disk. -
@aha_1980, that's very useful information, thanks!
Other than having a benchmark that hits hard, another concern with this is ensuring it scales on many-core CPUs and the like. I'm guessing that'll depend as much on jom as it does on the compiler proper, but I should also probably MSVC to deliver on that front as well.
@morphine said in Can't compile with jom in Windows - Makefile has bash syntax in it:
@aha_1980, that's very useful information, thanks!
Other than having a benchmark that hits hard, another concern with this is ensuring it scales on many-core CPUs and the like. I'm guessing that'll depend as much on jom as it does on the compiler proper, but I should also probably MSVC to deliver on that front as well.
When using Qt creator you require jom for MSVC for parallel compilation. IIRC it compensates the lack of nmake to do parallel compilations on its own. This is done by the MSVC IDE there.
Qt creator and MinGW does allow parallel compilation on windows. You need to use the -j parameter or environment settings. There no need to use jom with MinGW, but you can (see the link I had posted above for CPU comparison).
Also RAM disks help to speed up. I place my shadow on the ram disk.
There was already a discussion involving @mrjj https://forum.qt.io/topic/80933/qtcreator-shadow-build-directory-structure/19
However, I thought that there were some numbers given in comparison.
