migration to cmake results in corrupt binaries

kshegunov

FWIW qmake performs decently for 'complex' projects juts as well. But anyhow, if something is crashing you can almost surely say it's not the build as such - the ordinary advice holds:
Run in debug, extract stack traces, use QT_DEBUG_PUGINS to see how plugins are loaded and what is missing/not loading, etc.

django.Reinhard

@kshegunov said in migration to cmake results in corrupt binaries:

use QT_DEBUG_PUGINS to see how plugins are loaded

Thanks! Didn't know that. I'll try next time.

Actually I'm testing with valgrind.
I have several occurrences of this kind:

==302243== Invalid read of size 8
==302243==    at 0x401D5FC: strncmp (strcmp.S:173)
==302243==    by 0x400604D: is_dst (dl-load.c:209)
==302243==    by 0x400861E: _dl_dst_substitute (dl-load.c:288)
==302243==    by 0x40088D1: fillin_rpath.isra.0 (dl-load.c:460)
==302243==    by 0x4008BE1: decompose_rpath (dl-load.c:631)
==302243==    by 0x4009745: cache_rpath (dl-load.c:673)
==302243==    by 0x4009745: cache_rpath (dl-load.c:654)
==302243==    by 0x4009745: _dl_map_object (dl-load.c:2074)
==302243==    by 0x400DDC0: openaux (dl-deps.c:64)
==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
==302243==    by 0x400E138: _dl_map_object_deps (dl-deps.c:248)
==302243==    by 0x4013DAA: dl_open_worker (dl-open.c:571)
==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
==302243==    by 0x40138F9: _dl_open (dl-open.c:837)
==302243==  Address 0xdffb211 is 1 bytes inside a block of size 8 alloc'd
==302243==    at 0x483877F: malloc (vg_replace_malloc.c:307)
==302243==    by 0x401C16A: strdup (strdup.c:42)
==302243==    by 0x4008B74: decompose_rpath (dl-load.c:606)
==302243==    by 0x4009745: cache_rpath (dl-load.c:673)
==302243==    by 0x4009745: cache_rpath (dl-load.c:654)
==302243==    by 0x4009745: _dl_map_object (dl-load.c:2074)
==302243==    by 0x400DDC0: openaux (dl-deps.c:64)
==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
==302243==    by 0x400E138: _dl_map_object_deps (dl-deps.c:248)
==302243==    by 0x4013DAA: dl_open_worker (dl-open.c:571)
==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
==302243==    by 0x40138F9: _dl_open (dl-open.c:837)
==302243==    by 0x8802257: dlopen_doit (dlopen.c:66)
==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)

I thought it happen at plugin loading, but that's not true.

I added QDebug messages to get rid of where that happens, but then it did not happen again.
Now I'm going to cleanup - let's see, if they come again then.

django.Reinhard

@kshegunov said in migration to cmake results in corrupt binaries:

use QT_DEBUG_PUGINS to see how plugins are loaded

Just tried it - and I wonder: are qt toolkit plugins loaded from a different thread, or is that plugin loading related to my code?
Mixture of my debug messages and plugin loading messages looks a bit strange too me.
It would make sense, if qt provided plugins get loaded by different thread.

@kshegunov said in migration to cmake results in corrupt binaries:

FWIW qmake performs decently for 'complex' projects juts as well.

Depends on what you understand with "complex".
qmake is wonderful to start with Qt. Its easy to learn and works well (at least in the beginning).
It may perform well with increasing projects too, but as soon as you start with subdirectories, qmake can not handle dependencies any more.

In my project I have static libraries, several plugins and the application. Plugins depend on libraries, as well as the application.
After changes at library, qmake does not build the plugins.
So I always had to manually clean and rebuild subprojects.
That was the reason why I startet with a different build system.

I coded the dependencies like this:

src.app.depends = src/baselib
src.lcLib.depends = src/baselib
src.statusInfo.CurCodes.depends = src/baselib
src.statusInfo.Position.depends = src/baselib
src.statusInfo.SpeedInfo.depends = src/baselib
src.statusInfo.ToolInfo.depends = src/baselib
src.plugPages.FixtureManager.depends = src/lcLib
src.plugPages.HelpView.depends = src/lcLib
src.plugPages.JogView.depends = src/lcLib
src.plugPages.LCToolTable.depends = src/lcLib
src.plugPages.MDIEditor.depends = src/lcLib
src.plugPages.PathEditor.depends = src/lcLib
src.plugPages.PrefsEditor.depends = src/baselib
src.plugPages.Preview3D.depends = src/lcLib
src.plugPages.SysEventView.depends = src/baselib
src.plugPages.ToolManager.depends = src/lcLib

May be I understood Qt docs wrong and all misbehaviour is my fault. Don't know.

What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!
With that variable I could organize sources depending on source categories and have the binaries organized for runtime without the need of calling install.

So in my oppinion dropping qmake support is the wrong way.

Christian Ehrlicher

@django-Reinhard said in migration to cmake results in corrupt binaries:

I coded the dependencies like this:

You don't need this at all with a recent build system - as soon as a library links against another one, those deps are created automatically.

mchinand

@django-Reinhard said in migration to cmake results in corrupt binaries:

What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!

I think CMAKE_RUNTIME_OUTPUT_DIRECTORY and CMAKE_ARCHIVE_OUTPUT_DIRECTORY are the equivalent to qmake's DESTDIR.

django.Reinhard

Thank you for that hint.

CMAKE_RUNTIME_OUTPUT_DIRECTORY seems to work, but only for executable application.

Whatever I tried, I can't get the plugins copied to some destination. Initially the plugins where declared as SHARED. After getting deeper into cmake docs I tried MODULE, which is said to be copied with runtime, but that does not work for me.

What am I doing wrong?

mchinand

Try setting the ~~CMAKE_ARCHIVE_OUTPUT_DIRECTORY~~ CMAKE_LIBRARY_OUTPUT_DIRECTORY for the plugin sub-projects and any other shared library sub-projects.

https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html#output-artifacts

kshegunov

@django-Reinhard said in migration to cmake results in corrupt binaries:

May be I understood Qt docs wrong and all misbehaviour is my fault.

Indeed.

What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!

... instead of providing install interface(s) ... *sigh*

Don't mess with the destination directories (neither in qmake, nor in cmake), use the install targets and pass your desired prefix on the command line.

https://cmake.org/cmake/help/latest/variable/CMAKE_INSTALL_PREFIX.html
and
https://cmake.org/cmake/help/latest/manual/cmake.1.html

(but also applicable to qmake)

django.Reinhard

@mchinand said in migration to cmake results in corrupt binaries:

Try setting the CMAKE_ARCHIVE_OUTPUT_DIRECTORY CMAKE_LIBRARY_OUTPUT_DIRECTORY for the plugin sub-projects

Already tried that without success.
Actually I use a custom command which calls a shell script, but the custom command is not always executed :(

@kshegunov said in migration to cmake results in corrupt binaries:

... instead of providing install interface(s) ... sigh

I do not know where I provoked your mockery. After all, I'm not a professional. I'll try my best, but after all, most is just guessing.

I thought, installing means publishing the build-results to user space - and that's not what I want here.
I used dest directories to have all in place for debug sessions, where I have paths coded in my application, where I search for plugins.

mchinand

You can install to any directory you want, it doesn't have to be a system directory. For testing, it can be one of your local directories.

kshegunov

@django-Reinhard said in migration to cmake results in corrupt binaries:

I do not know where I provoked your mockery. After all, I'm not a professional. I'll try my best, but after all, most is just guessing.

Yeah, I'm not a professional either, but I've wasted time dealing with people's overwriting of the build directories, so it's sort of a sore spot. Just follow the documentation is my best advice.

I thought, installing means publishing the build-results to user space - and that's not what I want here.

Well, not exactly no; what @mchinand said.

django.Reinhard

@kshegunov said in migration to cmake results in corrupt binaries:

dealing with people's overwriting of the build directories

Yes, last few days I did lot of stupid errors - including wiping out my complete project tree. But that happens if working too long.

@kshegunov said in migration to cmake results in corrupt binaries:

I thought, installing means publishing the build-results to user space - and that's not what I want here.

Well, not exactly no; what @mchinand said.

That leads to the next problem. Using DESTDIR with qmake just moves the targets to given location and its transparent to QtCreator.
Starting debug just takes the binaries from DESTDIR.

With cmake that's not possible. And I fear, using install with qmake would not change a bit.
I found no way to change path of the executable to run/debug in QtCreator. It just takes the target.
Using cmake - ignores custom commands, which may create a different target or move it ...

So I have to use kdevelop, where I can attach the debugger to whatever binary.

django.Reinhard

@Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

I coded the dependencies like this:

You don't need this at all with a recent build system - as soon as a library links against another one, those deps are created automatically.

After reading that, I thought - why not give qmake another try.

So I removed all that coded dependencies and did a fresh rebuild. Than I changed a source of a library and started rebuild.

Result: the changed source was compiled and that lib was rebuild. That was all!

no relinking of executable (which uses that lib)
no relinking of plugins (which use that lib)

So if qmake should be able to handle dependencies, what am I doing wrong?

On the other side: cmake does the expected work.

compiles the changed source
rebuilds the library
relinks the application
relinks every plugin that depends on that library

So from my point of view, there's a huge difference.

Beside from that:
Both debug builds produce slightly different sizes:
qmake:

executable about 43 MB
plugin about 33 MB

cmake:

executable about 35 MB
plugin about 28 MB

my very personal impression: qmake built is significantly faster

Christian Ehrlicher

@django-Reinhard said in migration to cmake results in corrupt binaries:

So if qmake should be able to handle dependencies, what am I doing wrong?

Where did I say that qmake handles this automatically? I would never call qmake a 'recent build system'...

The difference in the generated output size and also in the compile time results from different compiler options.

django.Reinhard

Hi Christian,

@Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

Where did I say that qmake handles this automatically? I would never call qmake a 'recent build system'...

So I got you wrong. As you replied to my qmake dependencies coding. I did that as mentioned in Qt docs.

So if qmake is not a 'recent build system', what are you using?

@Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

The difference in the generated output size and also in the compile time results from different compiler options.

Well, I did compare both build-systems command lines and I did not find any usage of compile options, which influence code generation (like -O2 or so). The main difference regarding code generation is the usage of different compiler names.

qmake uses g++
cmake uses c++

I can debug both binaries, so its not the difference between debug and release

kshegunov

@django-Reinhard said in migration to cmake results in corrupt binaries:

Yes, last few days I did lot of stupid errors - including wiping out my complete project tree. But that happens if working too long.

I didn't mean you personally. But anyhow.

That leads to the next problem. Using DESTDIR with qmake just moves the targets to given location and its transparent to QtCreator.
Starting debug just takes the binaries from DESTDIR.

With cmake that's not possible. And I fear, using install with qmake would not change a bit.

The build system already knows where the binaries are located. They're in the build folder where it'd placed it.

I found no way to change path of the executable to run/debug in QtCreator. It just takes the target.

Which target? There may be multiple targets, and so you can select in the interface which one you want to run.

Using cmake - ignores custom commands, which may create a different target or move it ...

It ignores no commands. I'd venture to say you're using it incorrectly; more specifically that you're trying to make it behave as you expect, instead of vice versa.

So I have to use kdevelop, where I can attach the debugger to whatever binary.

Which is again incorrect. I attach to running binaries no problem with Creator.

django.Reinhard

@kshegunov said in migration to cmake results in corrupt binaries:

It ignores no commands. I'd venture to say you're using it incorrectly; more specifically that you're trying to make it behave as you expect, instead of vice versa.

Allow me to ask an important question first: do you use cmake or do you just talk like that?

I've been struggling with cmake for over a week now and for me there's a big gap between cmake-documentation and what does work, or does not work for me. I'm tired fighting.

That's why I now settle for what does, even if I'm anything but happy with the status quo.

Meanwhile I found out how to debug a different executable with qtcreator. Popup-dialog is misleading :(

Actually I use a custom command to do some kind of install after build but inside the build tree. I execute a simple shell script and there I know what to do.

If you're using cmake, maybe you can help me with my last issue (regarding build system):

how can I add the language-files so that qtcreator knows them as part of the project (like he does with qmake projects)

Last comment to different file sizes: I changed cmake to use g++ - but that did not change filesize. So I suppose, cmake-build is still missing some action, that qmake does. Any idea, how I could find out, what is missing?

kshegunov

@django-Reinhard said in migration to cmake results in corrupt binaries:

Allow me to ask an important question first: do you use cmake or do you just talk like that?

Yes, I do use it, and also I just talk like that (in that manner).

I've been struggling with cmake for over a week now and for me there's a big gap between cmake-documentation and what does work, or does not work for me. I'm tired fighting.

Yeah, I feel you. I never said it was easy. The docs are decent if you're after a reference, but decent examples that are up to date and/or tutorials are n't much to be found.

That's why I now settle for what does, even if I'm anything but happy with the status quo.

That's fine by me in any case. Actually for what it's worth qmake is simpler and I like it more (also because I'm more used to it).

Actually I use a custom command to do some kind of install after build but inside the build tree. I execute a simple shell script and there I know what to do.

Eh, what I said before, why not add a target to do that instead of using some external shell script?
Also for cmake there may be better ways, depending on what exactly "some kind of install" means.

If you're using cmake, maybe you can help me with my last issue (regarding build system):

how can I add the language-files so that qtcreator knows them as part of the project (like he does with qmake projects)

Does that help:
https://doc.qt.io/qt-5/qtlinguist-cmake-qt5-add-translation.html

Qt adds a decent amount of macros/functions to cmake to make life easier.

Last comment to different file sizes: I changed cmake to use g++ - but that did not change filesize. So I suppose, cmake-build is still missing some action, that qmake does. Any idea, how I could find out, what is missing?

Inspect the actual compile and link lines, they may (even probably) differ.

django.Reinhard

@kshegunov said in migration to cmake results in corrupt binaries:

Actually for what it's worth qmake is simpler and I like it more (also because I'm more used to it).

Absolutely!

qmake is like riding a tricycle and cmake is a formula 1 rocket. Every kid mounts a tricycle and races - but you need several licenses to ride a formula 1 rocket.

If qmake could handle dependencies, I had no need to change. Well and custom commands are not well supported. Both build systems don't really support creating helpfiles. In cmake it was easier to embed a custom command in dependencies so that it builds automatically.

@kshegunov said in migration to cmake results in corrupt binaries:

Does that help:

Not really.

I already have that commands in my cmake files. But QtCreator does not show the sources in project tree (to start QLinguist).
And when I start QtCreator->external command->linguist->refresh translations I get the error message, that no ts-files are defined.
But they are - in subprojects.
... any way - I tried to define the ts-files in toplevel cmake file, but it made no difference.

So - its stupid, but actually the only way for me: I use the qmake-project to handle translations and the cmake file for application build.

kshegunov

This thread is really getting off topic, but as long as you are happy with it, it's fine by me.

@django-Reinhard said in migration to cmake results in corrupt binaries:

If qmake could handle dependencies, I had no need to change.

I assure you, it does. If it didn't one wouldn't be able to build Qt itself with it. Something is wrong with the way you've setup your subdir project and that's the reason. Try to run in verbose mode qmake -d to see what's getting parsed/processed.

Well and custom commands are not well supported. Both build systems don't really support creating helpfiles. In cmake it was easier to embed a custom command in dependencies so that it builds automatically.

Actually, it should be rather straightforward. Just the other day I stumbled on one custom command invocation in Qt itself. Here, take a look:
https://code.qt.io/cgit/qt/qtbase.git/tree/cmake/QtBuild.cmake?h=wip/cmake#n3887

I already have that commands in my cmake files. But QtCreator does not show the sources in project tree (to start QLinguist).

Ah, you mean in Creator. That's no fault of the build system, but rather the IDE's integration. There may be some way, but personally, I have no idea.