Android Project - Make.exe parallel jobs

paolot

Hello everybody!

I'm currently working on a Qt project targeting Android and I noticed a really strange behavior: if I add a "-j" argument to the Make build step, the compilation (obviously) ends up requiring too much resources (in particular RAM) so I tried to change it to "-j4" or any "-jn" but it does not work, it spawns only a single process.

I investigated a little bit and saw that someone complains about the MinGW version of Make provided by the Android NDK. To be sure I made a quick test creating a Makefile calling two children Makefiles underneath and launching it with the same Make version and it works perfectly.

I suppose then that the issue is not related to Make.exe but probably something else; initially I thought to Qt Creator as the guilt but launching the build from command line reproduces the same issue.

Could it be something related to QMake?

My build command is

qmake.exe "C:\Projects\MyProject\project.pro" -r -spec android-clang "QMAKE_PROJECT_DEPTH=0" "CONFIG+=debug" -after DEFINES+=BLM_APP_BUILD_VERSION=\\\\\\\"\"000\\\\\\\"\" && C:\Tools\android\android-ndk-r18b\prebuilt\windows-x86_64\bin\make.exe -j 100

Details about the environment:

Project target: Android ARMv7
NDK: r18b
Qt version: 5.12.1
Qt Creator version: 4.9.2
Host operating system: Windows 10

Any suggestion? I don't see why "-j" should work while "-jn" doesn't...

Thanks a lot!

koahnig

@paolot

Hi and welcome to devnet forum

Apparently you are starting qmake outside of Qt creator. How do you that?
You should be able to check the actual make command being carried out. Alternatively, you can start the make in the command separately to ensure that qmake has no influence which I doubt anyway.

paolot

Hi, thanks for replying.

Looking at the "Compile Output" tab I see the following when I start the build:

13:08:39: Running steps for project stpalm...
13:08:39: Configuration unchanged, skipping qmake step.
13:08:39: Starting: "C:\Tools\android\android-ndk-r18b\prebuilt\windows-x86_64\bin\make.exe" -j12
cd utility\ && ( if not exist Makefile C:\Qt\5.12.1\android_armv7\bin\qmake.exe -o Makefile C:\Projects\MyProject\utility\utility.pro -spec android-clang "CONFIG+=debug" "CONFIG+=qml_debug" ) && C:/Tools/android/android-ndk-r18b/prebuilt/windows-x86_64/bin/make -f Makefile

As you can see this is just the first step since my project consists of several subfolders (one of which is "utility"), so the make command is run once for each subfolder.

The flag "-j12" is applied but does not work at all unfortunately.

koahnig

@paolot

Do you define a sequence of those subfolders?
If yes, it will follow the order you are giving. E.g. if subfolder3 depends on subfolder2, subfolder 2 depends on subfolder 1 and subfolder1 depends on subfolder, you cannot expect those subfolders re compiled in parallel. I have a similar structure and the subfolders will be processed in sequence.

Otherwise this is anyway something up to the make you are using. qmake is preparing only the different make files for processing by make.

paolot

I know that dependencies between projects have to be processed sequentially, in fact I'm talking about parallel build within the single project. It is working if I use just "-j" without specifying the number of jobs, so I would expect that it worked similarly even with "-j".

I already reached the Android NDK forum and they think that the issue could be related to QMake (obviously), but the most reasonable argument right now is that, as you already said, QMake just prepares the Makefiles, any issue with the parallel building should be caused by Make itself.

I'll try to find an example project to build on my machine in order to reproduce the issue, so that I can share with you and with the NDK developers.

Thanks a lot for the moment!

koahnig

@paolot

How do you set in creator the flag?

In creator on building settings page you can add -j12 or whatever you like that shall be used resp. in my creator 4.9.2 there is now the possibility to set the number of parallel jobs and it says 8. Never entered this number probably taken out of my environment.

IIRC in the past I did not want to add the flag per project/application. Therefore I have added in my windows environment

MAKEFLAGS=-j8

That may explain also why creator is recognizing the number of parallel jobs.

As it looks in your last post make is not started with an additional command line argument for parallel jobs.

paolot

@koahnig

Sorry, I made some confusion while writing the last post, what I meant is that:

Using "-j" to build works properly but it takes to many system resources
Using "-j12" or any other number does not work at all, using any number of jobs makes the build process to use just one job

I made some new tests that might be useful to understand what's going on under the hood. I made a full compilation from Qt Creator (running both QMake and Make), then I opened a command prompt, navigated to a subfolder of the project and ran the following commands:

Removed the bin and obj folders, then ran "make -j"; it took 35 seconds to complete
Removed again bin and obj, then ran "make"; it took 60 seconds
Removed again bin and obj, then ran "make -j12"; it took 36 seconds

That sounded promising, so I tried a similar approach using the parent Makefile, so I navigated to the project root and ran the following commands:

Removed the bin and obj folders only of one sub project (the same I tested before), then ran "make -j"; it took 3 minutes and 45 seconds
Removed again the bin and obj folders of the sub project, then ran "make"; it took 4 minutes and 5 seconds
Removed again the bin and obj folders of the sub project, then ran "make -j12"; it took 4 minutes and 5 seconds

In few words, after having generated the Makefiles with QMake, if I run "make -j12" from the command line for a single sub-project it works while running the same command for the main project makes the build being processed with a single job.

Any idea about the reason?

SGaist

Hi and welcome to devnet,

Since you are mentioning subproject. are you using the suddirs project template ?

paolot

@sgaist said in Android Project - Make.exe parallel jobs:

Hi and welcome to devnet,

Since you are mentioning subproject. are you using the suddirs project template ?

Hi @SGaist and thanks for your reply. Yes, I am using the subproject template, why are you asking?

SGaist

Can you show how you wrote the dependencies ?
If you use the ordered option, you might be killing the build parallelisation.

paolot

@sgaist said in Android Project - Make.exe parallel jobs:

Can you show how you wrote the dependencies ?
If you use the ordered option, you might be killing the build parallelisation.

It's exactly as you said:

TEMPLATE = subdirs

SUBDIRS = \
    proj1 \
    proj2 \
    main \
    test

CONFIG += ordered

I know that by using ordered I'm asking to compile projects sequentially, but I cannot explain why the build system is avoiding the parallelism within the sub-projects.

Moreover the same template is working perfectly when I use Desktop as target instead of Android...

Do you have any idea about how I could get rid of this behavior?

Edit: I use JOM to build the Desktop target.

koahnig

@paolot

Apparently the make.exe delivered by google with Android toolchain is not observing that flag.

I had noticed also that apparently the compilation took quite long. Certainly I am using subdir template which explains the sequential compilation of each dynamic lib, but also within a lib the compilation is sequential. That is due to make.exe as from Android SDK. For testing I have ensured that the parameter -j8 is set through additional make argument. This argument is set:

12:16:08: Starting: "C:\Users\xxx\AppData\Local\Android\Sdk\ndk-bundle\prebuilt\windows-x86_64\bin\make.exe" -j8

In order to allow a parallel compilation through creator an independend tool like jom.exe would be required in that case. It may be downloaded from https://wiki.qt.io/Jom . However, the page states explicitly that this is a replacement for nmake. Therefore, it might not be suitable for cross-compilation for Android.

paolot

@koahnig said in Android Project - Make.exe parallel jobs:

@paolot

Apparently the make.exe delivered by google with Android toolchain is not observing that flag.

I had noticed also that apparently the compilation took quite long. Certainly I am using subdir template which explains the sequential compilation of each dynamic lib, but also within a lib the compilation is sequential. That is due to make.exe as from Android SDK. For testing I have ensured that the parameter -j8 is set through additional make argument. This argument is set:
12:16:08: Starting: "C:\Users\xxx\AppData\Local\Android\Sdk\ndk-bundle\prebuilt\windows-x86_64\bin\make.exe" -j8
In order to allow a parallel compilation through creator an independend tool like jom.exe would be required in that case. It may be downloaded from https://wiki.qt.io/Jom . However, the page states explicitly that this is a replacement for nmake. Therefore, it might not be suitable for cross-compilation for Android.

That is what I thought at the beginning, but I started having doubts when I discovered that make.exe from the Android SDK works concurrently when launched from any sub-project folder. Are we sure there is not anything wrong with the project structure or the root level Makefile?

SGaist

The ordered keyword is a quick and easy way to build the subprojects in the order you want.

The correct way is described in the documentation but also on this nice article.

koahnig

@paolot

Basically you should be able to start the Android make on a command prompt outside of Qt creator at the same location where qt creator does. When it would be concurrent under these conditions it would point to some specifics of the Qt run environment for starting Android make.
I have not experienced such a behaviour so far. Therefore I consider as odd.

paolot

@sgaist said in Android Project - Make.exe parallel jobs:

The ordered keyword is a quick and easy way to build the subprojects in the order you want.

The correct way is described in the documentation but also on this nice article.

@SGaist thanks for the article, it is very interesting and useful, anyway even by switching to the suggested configuration (avoiding ORDERED) did not solve the issue for me.

@koahnig said in Android Project - Make.exe parallel jobs:

@paolot

Basically you should be able to start the Android make on a command prompt outside of Qt creator at the same location where qt creator does. When it would be concurrent under these conditions it would point to some specifics of the Qt run environment for starting Android make.
I have not experienced such a behaviour so far. Therefore I consider as odd.

I completely agree with you, my only fear is that QMake is doing something wrong when creating the top-level Makefile and it's somehow preventing the build process to go parallel. I could try writing a custom Makefile just building the subfolders, I'll let you know!