Dynamically managing big projects with QtCreator



  • Hello.

    I just recently found the subdirs template to manage big projects that is composed of multiple subprojects/components.
    Unfortunately, I was unhappy to find out that I must still manually add each subproject/component directory to the SUBDIRS variable in the master project file.

    To make the matter's worse, some of subprojects might themselfs have some subprojects of their
    own (they are also template type: subdir).

    So I must manually also update their possible SUBDIRS too if i add/change/remove their components.

    Im talking about project with directory hierarchy at least 3 directories deep in worst case, and the combined number of subprojects+modules themselfs about 1000+

    My question now is:

    Are there any plans to modify the SUBDIRS variable behavior in future QtCreator versions so that you could use wilcards (or maybe even regular expressions) to define dynamically the subproject directories ?

    Like:

    SUBDIRS += directory1/* directory2/module* directory3/gui??/module* etc....

    That way I could just create/change/remove new module/project/component etc. without having to update and check every subdir type project-file SUBDIR variable separately.

    Also, could the subdir template be modified to accept other qmake variables too ?

    That way I could keep all the common stuff (linker flags, gcc options etc..) in one place and if needed, override them separately for indiviual components.

    For example:
    I could have all the QMAKE_CFLAGS and QMAKE_LDFLAGS in master project-file (which would be just subdir template) and if some of the subproject themselves are of type subdir I could if wanted to, override these easily in one place too, without having to modify hundreds or thousands of files.

    Thank you.



  • [quote author="SFroberg" date="1316790413"]Im talking about project with directory hierarchy at least 3 directories deep in worst case, and the combined number of subprojects+modules themselfs about 1000+[/quote]

    Are those already qmake projects?

    [quote author="SFroberg" date="1316790413"]Are there any plans to modify the SUBDIRS variable behavior in future QtCreator versions so that you could use wilcards (or maybe even regular expressions) to define dynamically the subproject directories ?[/quote]

    None that I know of. The easiest solution would be a shell script which builds the SUBDIRS structure once (I assume that you do not add/remove 1000+ projects on a daily basis).

    [quote author="SFroberg" date="1316790413"]That way I could just create/change/remove new module/project/component etc. without having to update and check every subdir type project-file SUBDIR variable separately.[/quote]

    When adding a new project you have to modify just one SUBDIRS variable - that one project level "above" the added project. To structure the project itself the shell script using to build to inital structure can be used.

    [quote author="SFroberg" date="1316790413"]That way I could keep all the common stuff (linker flags, gcc options etc..) in one place and if needed, override them separately for indiviual components.[/quote]

    Variables do not propagte through SUBDIRS projects (I'm still not quite sure if this is a good thing or a bad thing). This is usually solved by having a top-level .pri file which is included in all subprojects.



  • [quote author="Lukas Geyer" date="1316792150"][quote author="SFroberg" date="1316790413"]Im talking about project with directory hierarchy at least 3 directories deep in worst case, and the combined number of subprojects+modules themselfs about 1000+[/quote]

    Are those already qmake projects?[/quote]

    Yes, every single one of them.

    [quote author="Lukas Geyer" date="1316792150"]
    [quote author="SFroberg" date="1316790413"]Are there any plans to modify the SUBDIRS variable behavior in future QtCreator versions so that you could use wilcards (or maybe even regular expressions) to define dynamically the subproject directories ?[/quote]

    None that I know of. The easiest solution would be a shell script which builds the SUBDIRS structure once (I assume that you do not add/remove 1000+ projects on a daily basis).[/quote]
    [/quote]

    No, the usuall rate is just about 10 to 20 projects per month.
    These are mostly just new plugins to old subprojects but sometimes I have to categorise some of the new or old modules that share a common ground into a new subproject.

    [quote author="Lukas Geyer" date="1316792150"]
    [quote author="SFroberg" date="1316790413"]That way I could just create/change/remove new module/project/component etc. without having to update and check every subdir type project-file SUBDIR variable separately.[/quote]

    When adding a new project you have to modify just one SUBDIRS variable - that one project level "above" the added project. To structure the project itself the shell script using to build to inital structure can be used.[/quote][/quote]

    What if I have subdir project that contains 2 (or more) subdir projects(and other projects) themself?
    Let say I have subdir project A that's SUBDIR looks like this:

    A.pro

    SUBDIR += B C moduleA01 moduleA02 etc ..... up to say, moduleA56

    Now, let's say the directories B and C are in reality subdir projects too:

    B.pro

    SUBDIR += moduleB01 moduleB02 moduleB03 ... to moduleB45 dir1 dir2 plugin1 plugin2 plugin2 plugin4 lib/network lib/security lib/gui
    etc ....

    C.pro

    SUBDIR += moduleC1 plugin01 plugin02 etc ... up to plugin33

    It can get tedious to keep track of and manually update all the SUBDIR variables when you have a project that contains 60 subdirs and each of those subdirs can contain anything from tens to hundreds of qmake projects of various types.
    (One of the subdirs includes 744 plugin projects by itself)

    If dynamic update would be allowed A.pro would look like this:

    SUBDIR += B C moduleA??

    (or even just this if no fine grained control needed)

    SUBDIR += *

    B.pro would look like this:

    SUBDIR += moduleB?? dir? plugin? lib/*
    etc ....

    or just:

    SUBDIR += *

    and

    C.pro

    SUBDIR += moduleC1 plugin??

    or just:

    SUBDIR += *

    [quote author="Lukas Geyer" date="1316792150"]
    [quote author="SFroberg" date="1316790413"]That way I could keep all the common stuff (linker flags, gcc options etc..) in one place and if needed, override them separately for indiviual components.[/quote]

    Variables do not propagte through SUBDIRS projects (I'm still not quite sure if this is a good thing or a bad thing). This is usually solved by having a top-level .pri file which is included in all subprojects.[/quote]

    Yes, I know and it's really a shame.

    I spent 1 whole day just to put one line, "include(../common/common.pri)" that included all the common stuff like CFLAGS, LIBS etc... to every plugin module project file accross all the subprojects.

    And ofcourse I had to do the same thing again for those subdir projects under the master tree which needed different flags than the global common ones.

    If variables would be allowed to propagate down the hierarchy throught SUBDIRS projects, I could have done the whole thing in just under 5 seconds

    :)



  • Well, there has to be a reason why variables do not propagte down the hierachy. I still haven't found it - but there has to be one ;-)

    qmake hasn't received much love in recent years - improvement wise and documentation wise. Nevertheless Qt is open-source and open-gouvernance. Grab that latest code and start adding the functionality you miss!



  • [quote author="Lukas Geyer" date="1316808843"]Well, there has to be a reason why variables do not propagte down the hierachy. I still haven't found it - but there has to be one ;-)

    qmake hasn't received much love in recent years - improvement wise and documentation wise. Nevertheless Qt is open-source and open-gouvernance. Grab that latest code and start adding the functionality you miss![/quote]

    That's what I was afraid of :(

    However, I found an old thread from 2003 where someone asked exactly this same, variable propagation solution.

    http://lists.trolltech.com/qt-interest/2003-01/thread00187-0.html

    There it reads:

    "The best solution is to have .qmake.cache file in your toplevel directory,
    in that file set the variables that you want to be set for all your projects
    and then qmake will automatically use that cache file in all projects inside
    subdirectories of that directory. "

    and from:

    http://doc.qt.nokia.com/latest/qmake-environment-reference.html

    _"Cache File

    The cache file is a special file qmake reads to find settings not specified in the qmake.conf file, project files, or at the command line. If -nocache is not specified when qmake is run, it will try to find a file called .qmake.cache in parent directories of the current directory. If it fails to find this file, it will silently ignore this step of processing.

    If it finds a .qmake.cache file then it will process this file first before it processes the project file."
    _

    So, I created .qmake.cache file to master tree and just copied all my QMAKE_CFLAGS, QMAKE_LDFLAGS, etc... from my common.pri-file to it.
    Then I removed the "include(../common/common.pri)" from one of the subproject that had the least number of qmake-projects inside it and started rebuild.

    And it worked! :D

    Inspecting the compiler output log it clearly showed that it used the right settings from that .qmake.cache file for the subproject.

    :)



  • Fine! Thanks for sharing your solution.



  • Wait, I also found another hack for the SUBDIR problem with wildcards:

    http://www.qtcentre.org/wiki/index.php?title=Undocumented_qmake#SUBDIRS_projects

    I copied that addSubdirs function mentioned there to my .qmake.cache file and now I can finally do the equivalent of SUBDIRS += * with this line:

    addSubdirs(*)

    Yes!!!

    :-)


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.