qmake and pre build, non parallel targets

ColossalTurtle

We have this project where we need to run a codegen step before any build, and we do that by

codegen.commands = ...
QMAKE_EXTRA_TARGETS += codegen
PRE_TARGETDEPS += codegen

But when doing a make -j {something larger than 1} this step is actually run in parallel with the rest of the project build, which will fail as most source files depend on the generated ones from the codegen step.

The solution I found was getting creative with the qmake custom targets and adding the make pseudo target to signal non parallel execution:

codegen.commands = ...
.NOTPARALLEL.depends = codegen
QMAKE_EXTRA_TARGETS += codegen '.NOTPARALLEL'
PRE_TARGETDEPS += codegen

Which does do the right Makefile generation, specifically:

.NOTPARALLEL: codegen

But, while working flawlessly, it does feel a bit hackish, and I wonder if there is a cleaner way of doing this that I missed in the documentation.

ColossalTurtle

Ok, so when I said "flawlessly" above I may have jumped the gun just a tiny bit...

The end result is that apparently not only the codegen target gets run outside the parallel job execution, but everything runs as if -j 1 had been passed, so no parallel jobs whatsoever.

This might be due to the majority of time being spent on the moc files building, and somehow QT not allowing these to run parallel, along with the fact my codegen will basically mark everything as dirty, where before the .NOTPARALLEL everything was first evaluated as existing and clean and thus not scheduled for rebuild, and only then codegen would run, which is sort of a more serious problem.

So, if someone has some idea on how to solve this conundrum I would really appreciate the feedback.

SGaist

Hi and welcome to devnet,

I don't have a direct answer but if using Qt Creator, you could add that step as part of your project setup.

Otherwise I am really not sure that it can be accomplished with qmake the way you need it.

Maybe cmake would be better suited.

ColossalTurtle

@SGaist Thank you for the kind welcome.

If I keep the codegen target on the .pro file, but remove the PRE_TARGETDEPS call, I can then add a make step between the qmake and the make that are created by default, calling 'make codegen' which will happen as a completely separate process, so not bothered by the parallel jobs of the following make call.

And yeah, it does seem to work, but that does not get carried across platforms, as that is part of the pro.user configuration. Maybe I can create a minimal .pro.user, I'm not sure, but if memory serves me right, it is not a good idea to add that file to version control as it is very environment specific.

We use QT Creator for development, but I'm coding CI and release supporting scripts now, so I need to be able to checkout a branch and compile it without user intervention.

So this solution would be perfect if I could make the default project configuration that has the extra build step so developers checking out a fresh branch would get that as part of the default configuration.

Using cmake will not really solve the issue, as within the scripts I'm creating I can manually and separately call codegen if need be, but I need this workflow to run in development too. Still, I might be missing something you were hinting at, it's been a long week.

Thank you.

SGaist

With the Ci it could be part of the build script but I understand the need of a "builtin" approach.

What I had in mind is the use of add_custom_command a bit like described here or here.

ColossalTurtle

I am definitely missing something here. How would I integrate that in the QT Creator workflow?

SGaist

If using add_custom_command with cmake ?

You would not need anything special in Qt Creator. It would automatically do the code generation so it's usable on the command line, with your CI and in Qt Creator.

ColossalTurtle

I should probably refrain from replying until I had a good night's sleep, but the more tired one is, the less self control one has, I guess :)

If I finally am getting what you are saying, the idea is replace qmake with cmake, which is enticing but I really have no idea what it entails. I will definitely have a go at it tomorrow.

I was operating under the impression QT Creator would only play nice with qmake.

SGaist

Recent versions of Qt Creator support cmake. It cannot do everything as integrated as with qmake but nevertheless, you can develop with it.

There's a Getting Started guide that might help you get started.

ColossalTurtle

Alright, slightly less asleep now, lets try one final effort to avoid having to move for cmake, as time is a bit crunched and I really need to get this over with to be able to afford the time needed for more drastic implementations, unless, of course, there is no other way.

I did find one way to convince make to do what I want, using order-only prerequisites, and it goes like this:

on Project.pro, we add the target but don't run it

codegen.commands = ...
'${OBJECTS}'.depends = '|codegen'
QMAKE_EXTRA_TARGETS += codegen '${OBJECTS}'

Which generates the following in the Makefile:

${OBJECTS}: |codegen

It basically says to run the compile targets after codegen, in such a way that does exactly what I want, except maybe for the compilation of MOCs that run before codegen which I believe have no dependency on the generated code, so that is fine for me, but also easy to fix if needed.

I'm going to run with this for now, and I leave it here in case someone else is struggling with the same, but I really appreciate all the help and pointers, and will definitely look into integrating cmake into the project as a whole in the near future!

Thanks

ColossalTurtle

And, again, I am humbled by how unnecessarily complex things get when we need to support the Windows platform.

So, yeah, everything works fine as far as qmake is concerned, but on windows QT is using jom and, well, this syntax isn't supported, which makes sense, since nmake does not support parallel execution and jom is nmake plus -j from what I read diagonally.

Oh, well, just leaving this here as FYI, will need to make time to try cmake sooner rather than later.