Important: Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

migration to cmake results in corrupt binaries



  • Hi,

    since qmake can't manage the new complexity of my project, i tried cmake.
    This was a big piece of work, because I had no idea about cmake. In the meantime the building of the application completes without errors.

    The positive changes are that the dependencies are better resolved, which is also reflected in the fact that all plugins can now be loaded. No more unresolvable externals.

    The bad news is that on loading a plugin the constructor is not executed, which means that all member pointers are not initialized (initialization is coded), which of course leads to initialization sequence errors and thus crashes.

    I have already summarized the most obvious changes in the cmake forum

    I suspect this is related to all the auto variables (e.g. CMAKE_AUTOMOC)

    Has anyone perhaps experienced something similar and knows how to fix it?


  • Moderators

    It seems you haven't configured yet which Qt to use. Try e.g.

    sudo apt-get install qt5-default
    

    Alternatively, you can also try to call

    lupdate-qt5
    


  • Thank you for your attention.

    Sure I did.

    I took me a while to get the right Qt toolkit selected. I have several toolkits installed from Qt binary installer - so no apt.
    In toplevel CMakeLists.txt I coded it like this:

    set(CMAKE_PREFIX_PATH "/opt/qt5.15/5.15.2/gcc_64" CACHE PATH "CMake search-path")
    
    find_package(Qt5 REQUIRED COMPONENTS
                 Gui
                 Widgets
                 Core
                 Sql
                 UiTools
                 Xml
                 LinguistTools
                 )
    

    cmake-gui showed that the found config files are those expected.


  • Lifetime Qt Champion

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    The bad news is that on loading a plugin the constructor is not executed,

    CMake is a build system tool, not a compiler. So it for sure will not change anything wrt if a ctor is called or not.



  • @Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

    CMake is a build system tool, not a compiler.

    Sure!

    but how do you call it, when the same source tree built with qmake works fine and crashes when built by cmake?

    I know, that qmake and cmake are build systems. But both execute different commands at different times. And created binaries differ substantially in size.



  • Are you sure you're not seeing the difference in build type (release vs debug) and not strictly qmake vs cmake?



  • Yes!

    I did not use any release build til now. At least not intentionally.


  • Moderators

    FWIW qmake performs decently for 'complex' projects juts as well. But anyhow, if something is crashing you can almost surely say it's not the build as such - the ordinary advice holds:
    Run in debug, extract stack traces, use QT_DEBUG_PUGINS to see how plugins are loaded and what is missing/not loading, etc.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    use QT_DEBUG_PUGINS to see how plugins are loaded

    Thanks! Didn't know that. I'll try next time.

    Actually I'm testing with valgrind.
    I have several occurrences of this kind:

    ==302243== Invalid read of size 8
    ==302243==    at 0x401D5FC: strncmp (strcmp.S:173)
    ==302243==    by 0x400604D: is_dst (dl-load.c:209)
    ==302243==    by 0x400861E: _dl_dst_substitute (dl-load.c:288)
    ==302243==    by 0x40088D1: fillin_rpath.isra.0 (dl-load.c:460)
    ==302243==    by 0x4008BE1: decompose_rpath (dl-load.c:631)
    ==302243==    by 0x4009745: cache_rpath (dl-load.c:673)
    ==302243==    by 0x4009745: cache_rpath (dl-load.c:654)
    ==302243==    by 0x4009745: _dl_map_object (dl-load.c:2074)
    ==302243==    by 0x400DDC0: openaux (dl-deps.c:64)
    ==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
    ==302243==    by 0x400E138: _dl_map_object_deps (dl-deps.c:248)
    ==302243==    by 0x4013DAA: dl_open_worker (dl-open.c:571)
    ==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
    ==302243==    by 0x40138F9: _dl_open (dl-open.c:837)
    ==302243==  Address 0xdffb211 is 1 bytes inside a block of size 8 alloc'd
    ==302243==    at 0x483877F: malloc (vg_replace_malloc.c:307)
    ==302243==    by 0x401C16A: strdup (strdup.c:42)
    ==302243==    by 0x4008B74: decompose_rpath (dl-load.c:606)
    ==302243==    by 0x4009745: cache_rpath (dl-load.c:673)
    ==302243==    by 0x4009745: cache_rpath (dl-load.c:654)
    ==302243==    by 0x4009745: _dl_map_object (dl-load.c:2074)
    ==302243==    by 0x400DDC0: openaux (dl-deps.c:64)
    ==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
    ==302243==    by 0x400E138: _dl_map_object_deps (dl-deps.c:248)
    ==302243==    by 0x4013DAA: dl_open_worker (dl-open.c:571)
    ==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
    ==302243==    by 0x40138F9: _dl_open (dl-open.c:837)
    ==302243==    by 0x8802257: dlopen_doit (dlopen.c:66)
    ==302243==    by 0x815C25F: _dl_catch_exception (dl-error-skeleton.c:208)
    

    I thought it happen at plugin loading, but that's not true.

    I added QDebug messages to get rid of where that happens, but then it did not happen again.
    Now I'm going to cleanup - let's see, if they come again then.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    use QT_DEBUG_PUGINS to see how plugins are loaded

    Just tried it - and I wonder: are qt toolkit plugins loaded from a different thread, or is that plugin loading related to my code?
    Mixture of my debug messages and plugin loading messages looks a bit strange too me.
    It would make sense, if qt provided plugins get loaded by different thread.

    @kshegunov said in migration to cmake results in corrupt binaries:

    FWIW qmake performs decently for 'complex' projects juts as well.

    Depends on what you understand with "complex".
    qmake is wonderful to start with Qt. Its easy to learn and works well (at least in the beginning).
    It may perform well with increasing projects too, but as soon as you start with subdirectories, qmake can not handle dependencies any more.

    In my project I have static libraries, several plugins and the application. Plugins depend on libraries, as well as the application.
    After changes at library, qmake does not build the plugins.
    So I always had to manually clean and rebuild subprojects.
    That was the reason why I startet with a different build system.

    I coded the dependencies like this:

    src.app.depends = src/baselib
    src.lcLib.depends = src/baselib
    src.statusInfo.CurCodes.depends = src/baselib
    src.statusInfo.Position.depends = src/baselib
    src.statusInfo.SpeedInfo.depends = src/baselib
    src.statusInfo.ToolInfo.depends = src/baselib
    src.plugPages.FixtureManager.depends = src/lcLib
    src.plugPages.HelpView.depends = src/lcLib
    src.plugPages.JogView.depends = src/lcLib
    src.plugPages.LCToolTable.depends = src/lcLib
    src.plugPages.MDIEditor.depends = src/lcLib
    src.plugPages.PathEditor.depends = src/lcLib
    src.plugPages.PrefsEditor.depends = src/baselib
    src.plugPages.Preview3D.depends = src/lcLib
    src.plugPages.SysEventView.depends = src/baselib
    src.plugPages.ToolManager.depends = src/lcLib
    

    May be I understood Qt docs wrong and all misbehaviour is my fault. Don't know.

    What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!
    With that variable I could organize sources depending on source categories and have the binaries organized for runtime without the need of calling install.

    So in my oppinion dropping qmake support is the wrong way.


  • Lifetime Qt Champion

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    I coded the dependencies like this:

    You don't need this at all with a recent build system - as soon as a library links against another one, those deps are created automatically.



  • @django-Reinhard said in migration to cmake results in corrupt binaries:

    What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!

    I think CMAKE_RUNTIME_OUTPUT_DIRECTORY and CMAKE_ARCHIVE_OUTPUT_DIRECTORY are the equivalent to qmake's DESTDIR.



  • Thank you for that hint.

    CMAKE_RUNTIME_OUTPUT_DIRECTORY seems to work, but only for executable application.

    Whatever I tried, I can't get the plugins copied to some destination. Initially the plugins where declared as SHARED. After getting deeper into cmake docs I tried MODULE, which is said to be copied with runtime, but that does not work for me.

    What am I doing wrong?



  • Try setting the CMAKE_ARCHIVE_OUTPUT_DIRECTORY CMAKE_LIBRARY_OUTPUT_DIRECTORY for the plugin sub-projects and any other shared library sub-projects.

    https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html#output-artifacts


  • Moderators

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    May be I understood Qt docs wrong and all misbehaviour is my fault.

    Indeed.

    What I really like at qmake (and miss in cmake) is the DESTDIR variable. That's so handy!

    ... instead of providing install interface(s) ... *sigh*

    Don't mess with the destination directories (neither in qmake, nor in cmake), use the install targets and pass your desired prefix on the command line.

    https://cmake.org/cmake/help/latest/variable/CMAKE_INSTALL_PREFIX.html
    and
    https://cmake.org/cmake/help/latest/manual/cmake.1.html

    (but also applicable to qmake)



  • @mchinand said in migration to cmake results in corrupt binaries:

    Try setting the CMAKE_ARCHIVE_OUTPUT_DIRECTORY CMAKE_LIBRARY_OUTPUT_DIRECTORY for the plugin sub-projects

    Already tried that without success.
    Actually I use a custom command which calls a shell script, but the custom command is not always executed :(

    @kshegunov said in migration to cmake results in corrupt binaries:

    ... instead of providing install interface(s) ... sigh

    I do not know where I provoked your mockery. After all, I'm not a professional. I'll try my best, but after all, most is just guessing.

    I thought, installing means publishing the build-results to user space - and that's not what I want here.
    I used dest directories to have all in place for debug sessions, where I have paths coded in my application, where I search for plugins.



  • You can install to any directory you want, it doesn't have to be a system directory. For testing, it can be one of your local directories.


  • Moderators

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    I do not know where I provoked your mockery. After all, I'm not a professional. I'll try my best, but after all, most is just guessing.

    Yeah, I'm not a professional either, but I've wasted time dealing with people's overwriting of the build directories, so it's sort of a sore spot. Just follow the documentation is my best advice.

    I thought, installing means publishing the build-results to user space - and that's not what I want here.

    Well, not exactly no; what @mchinand said.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    dealing with people's overwriting of the build directories

    Yes, last few days I did lot of stupid errors - including wiping out my complete project tree. But that happens if working too long.

    @kshegunov said in migration to cmake results in corrupt binaries:

    I thought, installing means publishing the build-results to user space - and that's not what I want here.

    Well, not exactly no; what @mchinand said.

    That leads to the next problem. Using DESTDIR with qmake just moves the targets to given location and its transparent to QtCreator.
    Starting debug just takes the binaries from DESTDIR.

    With cmake that's not possible. And I fear, using install with qmake would not change a bit.
    I found no way to change path of the executable to run/debug in QtCreator. It just takes the target.
    Using cmake - ignores custom commands, which may create a different target or move it ...

    So I have to use kdevelop, where I can attach the debugger to whatever binary.



  • @Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

    I coded the dependencies like this:

    You don't need this at all with a recent build system - as soon as a library links against another one, those deps are created automatically.

    After reading that, I thought - why not give qmake another try.

    So I removed all that coded dependencies and did a fresh rebuild. Than I changed a source of a library and started rebuild.

    Result: the changed source was compiled and that lib was rebuild. That was all!

    • no relinking of executable (which uses that lib)
    • no relinking of plugins (which use that lib)

    So if qmake should be able to handle dependencies, what am I doing wrong?

    On the other side: cmake does the expected work.

    • compiles the changed source
    • rebuilds the library
    • relinks the application
    • relinks every plugin that depends on that library

    So from my point of view, there's a huge difference.

    Beside from that:
    Both debug builds produce slightly different sizes:
    qmake:

    • executable about 43 MB
    • plugin about 33 MB

    cmake:

    • executable about 35 MB
    • plugin about 28 MB

    my very personal impression: qmake built is significantly faster


  • Lifetime Qt Champion

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    So if qmake should be able to handle dependencies, what am I doing wrong?

    Where did I say that qmake handles this automatically? I would never call qmake a 'recent build system'...

    The difference in the generated output size and also in the compile time results from different compiler options.



  • Hi Christian,

    @Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

    Where did I say that qmake handles this automatically? I would never call qmake a 'recent build system'...

    So I got you wrong. As you replied to my qmake dependencies coding. I did that as mentioned in Qt docs.

    So if qmake is not a 'recent build system', what are you using?

    @Christian-Ehrlicher said in migration to cmake results in corrupt binaries:

    The difference in the generated output size and also in the compile time results from different compiler options.

    Well, I did compare both build-systems command lines and I did not find any usage of compile options, which influence code generation (like -O2 or so). The main difference regarding code generation is the usage of different compiler names.

    • qmake uses g++
    • cmake uses c++

    I can debug both binaries, so its not the difference between debug and release


  • Moderators

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    Yes, last few days I did lot of stupid errors - including wiping out my complete project tree. But that happens if working too long.

    I didn't mean you personally. But anyhow.

    That leads to the next problem. Using DESTDIR with qmake just moves the targets to given location and its transparent to QtCreator.
    Starting debug just takes the binaries from DESTDIR.

    With cmake that's not possible. And I fear, using install with qmake would not change a bit.

    The build system already knows where the binaries are located. They're in the build folder where it'd placed it.

    I found no way to change path of the executable to run/debug in QtCreator. It just takes the target.

    Which target? There may be multiple targets, and so you can select in the interface which one you want to run.

    Using cmake - ignores custom commands, which may create a different target or move it ...

    It ignores no commands. I'd venture to say you're using it incorrectly; more specifically that you're trying to make it behave as you expect, instead of vice versa.

    So I have to use kdevelop, where I can attach the debugger to whatever binary.

    Which is again incorrect. I attach to running binaries no problem with Creator.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    It ignores no commands. I'd venture to say you're using it incorrectly; more specifically that you're trying to make it behave as you expect, instead of vice versa.

    Allow me to ask an important question first: do you use cmake or do you just talk like that?

    I've been struggling with cmake for over a week now and for me there's a big gap between cmake-documentation and what does work, or does not work for me. I'm tired fighting.

    That's why I now settle for what does, even if I'm anything but happy with the status quo.

    Meanwhile I found out how to debug a different executable with qtcreator. Popup-dialog is misleading :(

    Actually I use a custom command to do some kind of install after build but inside the build tree. I execute a simple shell script and there I know what to do.

    If you're using cmake, maybe you can help me with my last issue (regarding build system):

    • how can I add the language-files so that qtcreator knows them as part of the project (like he does with qmake projects)

    Last comment to different file sizes: I changed cmake to use g++ - but that did not change filesize. So I suppose, cmake-build is still missing some action, that qmake does. Any idea, how I could find out, what is missing?


  • Moderators

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    Allow me to ask an important question first: do you use cmake or do you just talk like that?

    Yes, I do use it, and also I just talk like that (in that manner).

    I've been struggling with cmake for over a week now and for me there's a big gap between cmake-documentation and what does work, or does not work for me. I'm tired fighting.

    Yeah, I feel you. I never said it was easy. The docs are decent if you're after a reference, but decent examples that are up to date and/or tutorials are n't much to be found.

    That's why I now settle for what does, even if I'm anything but happy with the status quo.

    That's fine by me in any case. Actually for what it's worth qmake is simpler and I like it more (also because I'm more used to it).

    Actually I use a custom command to do some kind of install after build but inside the build tree. I execute a simple shell script and there I know what to do.

    Eh, what I said before, why not add a target to do that instead of using some external shell script?
    Also for cmake there may be better ways, depending on what exactly "some kind of install" means.

    If you're using cmake, maybe you can help me with my last issue (regarding build system):

    • how can I add the language-files so that qtcreator knows them as part of the project (like he does with qmake projects)

    Does that help:
    https://doc.qt.io/qt-5/qtlinguist-cmake-qt5-add-translation.html

    Qt adds a decent amount of macros/functions to cmake to make life easier.

    Last comment to different file sizes: I changed cmake to use g++ - but that did not change filesize. So I suppose, cmake-build is still missing some action, that qmake does. Any idea, how I could find out, what is missing?

    Inspect the actual compile and link lines, they may (even probably) differ.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    Actually for what it's worth qmake is simpler and I like it more (also because I'm more used to it).

    Absolutely!

    qmake is like riding a tricycle and cmake is a formula 1 rocket. Every kid mounts a tricycle and races - but you need several licenses to ride a formula 1 rocket.

    If qmake could handle dependencies, I had no need to change. Well and custom commands are not well supported. Both build systems don't really support creating helpfiles. In cmake it was easier to embed a custom command in dependencies so that it builds automatically.

    @kshegunov said in migration to cmake results in corrupt binaries:

    Does that help:

    Not really.

    I already have that commands in my cmake files. But QtCreator does not show the sources in project tree (to start QLinguist).
    And when I start QtCreator->external command->linguist->refresh translations I get the error message, that no ts-files are defined.
    But they are - in subprojects.
    ... any way - I tried to define the ts-files in toplevel cmake file, but it made no difference.

    So - its stupid, but actually the only way for me: I use the qmake-project to handle translations and the cmake file for application build.


  • Moderators

    This thread is really getting off topic, but as long as you are happy with it, it's fine by me.

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    If qmake could handle dependencies, I had no need to change.

    I assure you, it does. If it didn't one wouldn't be able to build Qt itself with it. Something is wrong with the way you've setup your subdir project and that's the reason. Try to run in verbose mode qmake -d to see what's getting parsed/processed.

    Well and custom commands are not well supported. Both build systems don't really support creating helpfiles. In cmake it was easier to embed a custom command in dependencies so that it builds automatically.

    Actually, it should be rather straightforward. Just the other day I stumbled on one custom command invocation in Qt itself. Here, take a look:
    https://code.qt.io/cgit/qt/qtbase.git/tree/cmake/QtBuild.cmake?h=wip/cmake#n3887

    I already have that commands in my cmake files. But QtCreator does not show the sources in project tree (to start QLinguist).

    Ah, you mean in Creator. That's no fault of the build system, but rather the IDE's integration. There may be some way, but personally, I have no idea.



  • @kshegunov said in migration to cmake results in corrupt binaries:

    If qmake could handle dependencies, I had no need to change.

    I assure you, it does. If it didn't one wouldn't be able to build Qt itself with it.

    Well, not really.
    I wasn't aware, that qt is built by qmake too.
    But when I look at pro-files from qt, there's much additional coding. So much, that it does not look like a pro-file.
    So - its possible to achieve a correct build with qmake - but only if you spent lot of time in additional programming the build system.

    I already spent much more time with build-system than I wanted to.

    @kshegunov said in migration to cmake results in corrupt binaries:

    Just the other day I stumbled on one custom command invocation in Qt itself.

    ... but that's cmake. No problem there.
    I was talking about qmake. But as I saw the pro-files of qt - well, I underestimated qmake obviously.

    ... anyway - today I finally had the breakthrough I was looking for, so I'm happy with the level I've reached now. But that's a story for a new thread :)


  • Moderators

    @django-Reinhard said in migration to cmake results in corrupt binaries:

    So - its possible to achieve a correct build with qmake - but only if you spent lot of time in additional programming the build system.

    I was talking about qmake. But as I saw the pro-files of qt - well, I underestimated qmake obviously.

    qmake is old, it may even predate cmake, so yeah, Qt can be/used to be built with it. There was a special bootstrapping (actually I believe it's still in existence) done to build qmake first, before running it over the actual Qt tree. It is a great pain point and people still did it anyway. The support for qmake was dropped not because it's bad or can't do what you want, but because no one wants to pay for the maintenance. Maintaining code is expensive, and when there are de-facto standard tools that's much better from a the developer/company point of view.
    Qt's cmake files are no less complicated than the qmake project files, because there's a lot of pre, post and mid build tools that must be run so you can have a smooth experience as a user developer.

    But yeah, you must know the tools to wield them effectively. Blanket statements that this or that doesn't work are ... well, unproductive, for lack of a less offensive word. Of course you have to code the build system, code is code, and no build system is going to magically do what you want it to do, especially in some edge case ... so, sorry if I'm not convinced that not knowing the build system well enough should give you confidence to make bold statements that it can't do things or doesn't work.



  • Hi,

    @kshegunov said in migration to cmake results in corrupt binaries:

    Qt's cmake files are no less complicated than the qmake project files, because there's a lot of pre, post and mid build tools that must be run so you can have a smooth experience as a user developer.

    Agree completely.
    ... and I guess it will stil take a while until using cmake will be that comfortable as qmake was/is.

    @kshegunov said in migration to cmake results in corrupt binaries:

    I'm not convinced that not knowing the build system well enough should give you confidence to make bold statements that it can't do things or doesn't work.

    I do not want to be misunderstood - the bold statements were not bold to increase their truth content, but so that it is also read by people who only skim texts and then write answers that do not fit the question at all.

    If that came across wrong, then I'm sorry.
    I enjoyed the exchange with you (personally) very much


Log in to reply