Qt Libraries: why is something so simple made so complicated?
-
I think you are confusing Qt with VC++ Redist or something.
AFAIK Qt has no redistributeable runtime installation or anything, you only get the SDK for developers and nothing for the end user, that is the task of the developer I think.No offense but I think you have not much experience with how Qt works, what Chris Kawa said with the JPEg plugins etc are part of the Qt plugin system, which the build system doesn't know about. You never include anything for plugins, that is why Qt plugins are dynamically loaded when you use a jpeg image in you app the jpeg plugin gets loaded, if you use PNG images then not (for example), so there is no way an automated deployment can see what plugins are needed or not, all it can see it the default Qt module DLL's (QtCore, QtGUI etc).
Qt libs are usually bundled with the app or statically linked if you have a licence for that. I have never seen any Qt app where the library files are system wide installed (at least not on windows, maybe on unix systems).
If you compare Qt with VC++ redist packages, Qt has a lot of modules and usually you don't want to redistribute all Qt modules, but only the ones you needed. If you just copy everything in you app folder it should also work, but who want so many unnecessary files if they are never used?
Anyway what might help you with this problem is the qt.conf file you can put next to your binary and the config file contains paths where to find the Qt libs, so they don't have to be in the same folder as the app and you could do a system wide lookup if you wanted (I guess).
Check out the official documentation here: http://qt-project.org/doc/qt-5/qt-conf.html -
Yeah, I have to agree with Xander84. You're confusing many things with each other and bringing .Net examples into it only makes things less clear.
Qt doesn't install (nor has) any runtime package. All it does when you install SDK is place a buncha libs, dlls and headers in a folder. It doesn't make system "magically" know about these dlls.
The reason your program finds them when you run it from IDE (Qt Creator or other) is because that IDE runs your program in specifically crafted environment where it injects Qt locations into environment variables visible to your app. It's equivalent of adding Qt to PATH and creating QTDIR variable, except it's done per process, not system wide, which is a good thing because this way several Qt versions can live side by side and IDE injects them based on your Kit configuration. This info is not embedded in any way in the app .exe so when you ship it there is no way to tell which Qt dll it requires.MSVC compiler runtime is different in a sense, because MS chose different distribution policy for it. They provide runtime installers that adds their dlls to system wide locations. It's their product so they're allowed to do that. Others should not (and often don't) do that. You can either use their runtime installer(bundled with your app installer) or distribute single dlls within your app directory just as you would with Qt. For MinGW there's only second option as it doesn't provide runtime installer (as it's not MS product).
Qt is in a large part source compatible across versions (at least within a major version) and only partially binary compatible (across minor versions).
An app using function x() from lobrary y.dll only has symbols "x" and "y" embedded in it. System can hand it any library named "y" it finds. If function "x" is not in it or it is not binary compatible the app will crash at runtime and it has absolutely nothing to do with build process. It is totally different than what .Net does so any comparison here is not valid.As Xander84 said jpeg lib (also gifs, platform lib, databases and buncha other stuff) ARE part of the Qt SDK and have their own dlls in the plugins and platforms directory. They should also be distributed in your app, but there is no automatic way to figure out which will be used at runtime. An app will start and run but it will miss some functionality (like not displaying pictures).
There would be very little gain in automatically distributing everything. Most Qt based programs use only a handful of core modules and very little plugins (my apps usually take 4-6 dlls from among about 30 Qt provides). we don't want "hello world" program to be shipped with 50MB of dlls do we?
-
One more thing when comparing to what .Net does is that .Net runtime is a closed source package. When MS ships .Net 4.5.1 it is the same package everywhere.
When Qt ships version 5.2.1 it ships at least 5 or more of different builds of it (for different compilers and OpenGL build settings) and also a source, so there are countles different builds of the same version 5.2.1 that are incompatible between each other. There's no way to make a "universal" Qt runtime, unless you want it to be about 2GB in size and contain some of the more popular configurations(certainly not even close to all). But who would test and maintain (and use!) that? -
Small addition to the Qt plugin system: I noticed if you "forget" to distribute some plugins the app might silently exit (not crash). This happens at least with QML plugins but there are some special things because they are needed to render the window, so the app won't be able to display anything and just exists I guess. The important part here is that you won't get any error or crash, it looks like the app is just not starting at all which can be frustrating if you don't know why :)
-
No. I am not offended by your statements that I am not qt experienced. It is the truth. I stated that at the beginning of my original post. I appreciate your trying to help me understand a bit more about that. Your last two posts were helpful but I still don't follow why this cannot be done.
For one thing, the posts I have read say that this only happens on windows and linux. Apple products apparently have no problem picking out the right dynamic libraries to install (from what I have read).
The second thing is... so what you are saying is that there is no way for a class to be associated with a specific dll inside a specific sdk (and version) being used to provides the class as long as one knows the target platform? You are right. I don't get that at all. The linker knows which dll is necessary because it incorporates the filename, ordinals, etc. into the executable. So why can't the sdk figure it out? Especially because the developer of the SDK should know where the classes are located which is more than I can say for the linker... and the even the developer. How is a third party developer supposed to know which dll includes which class without doing an in-depth breakdown of the set of libraries? (I am NOT saying a developer shouldn't take the time to figure it out...only that IMO he shouldn't be required to.).
(Please keep dynamically linked plugins out of this discussion. Only basic core files such as qt5core.dll are applicable. I understand why this does not apply to plugins and so I am not addressing them.)
-
Well for the Qt module libs like Qt5Core etc it should be very easy for the developer to know what files need to be distributed, because you have to specify the used modules in the project file (.pro) to use with qmake!?
e.g.
@
QT = core gui network
@
so you are gonna need the QtCore, QtGui and QtNetwork libs, you just need to know that "core and gui is set by default. If you only use "QT += xml" for example the "QtCore and QtGUI libs are needed by default, that is all you need to know I think!?Of course the deployment of those files may be automated, but its not like this is much work for the developer to copy the base module libs himself, is it? :)
There might be reasons not to deploy all files automatically, also for mobile targets (Andorid, iOS etc) all libs are deployed by Qt Creator automatically.. and it takes forever to deploy that (like 1 minute for android), maybe that is one reason not to automate it haha, I don't know. -
Arguably it may not be a lot of work. But that depends on how compartmentalized the qt components/classes are. For example, math classes (assuming such classes exist) considered to be part of the "core" could be separated into a separate Qtmath.dll. Also, if it is a huge project with a myriad of other things to worry about (which the project for which I am using Qt will be), finding out that I have another on my plate is annoying.
I really don't know how Qt is structured and if it IS compartmentalized, it should be automated. (that is my main point... without an in depth look at the layout of the classes, there is no way for someone to know which dlls to include.) And as such, with all the other bells and whistles that come with this sdk I am really surprised that it doesn't include such a simple task.
NOTE: I use the Qt plugin for VS because I really don't like the Qt Creator layout for an IDE. (Although it is a perfect model for the kind of application I am trying to develop. The creator application GUI is one of the major things that sold me on using it for my project).
-
[quote author="primem0ver" date="1396895243"]
The second thing is... so what you are saying is that there is no way for a class to be associated with a specific dll inside a specific sdk (and version) being used to provides the class as long as one knows the target platform? You are right. I don't get that at all. The linker knows which dll is necessary because it incorporates the filename, ordinals, etc. into the executable. So why can't the sdk figure it out? [/quote]See, you're confusing things again. Linker has nothing to do with dlls(unless you're making a dll of course). Building has nothing to do with dlls. You can delete all Qt dlls from the system and your app will still compile and link just fine.
Linker deals with .lib files that contain a lot of info about where what is. When using MSVC .pdb files are also created (there's probably some counterpart for MinGW too) which also contain buncha paths and debugging info like function offsets, structures and classes layout etc.
I don't want to get too deep into that, but when a lib is linked an exe contains a name of the library and something called IAT (import address table) which is basically a list of offsets to functions in external library(dll). The dll itself is not needed for that because all necessary info is inside the .lib file. There's a .lib file for every .dll file.Dlls are linked at runtime (I'm not talking about dynamic LoadLibrary like plugins use). A starting app reports to system that it will use IATs for a library "X" and the system takes care of finding "X" and loading it. If it finds the wrong one then the offsets won't match and you have a crash.
So to summarize - app uses .lib files at link-time and .dlls at run-time.
[quote author="primem0ver" date="1396895243"]
I really don’t know how Qt is structured and if it IS compartmentalized, it should be automated. (that is my main point… without an in depth look at the layout of the classes, there is no way for someone to know which dlls to include.) And as such, with all the other bells and whistles that come with this sdk I am really surprised that it doesn’t include such a simple task.[/quote]
Qt is "very modularized":http://qt-project.org/doc/qt-5/qtmodules.html. If you're gonna use it it's a good idea to get to know where what is.I'll repeat myself - it's not that hard for a human but a hard task for automat. Core Qt modules(dlls) are just a part of what you need to deploy. there are lot of other dependencies that are not as easy to track as Qt core stuff: compiler runtime, plugins(some, like the platform plugin, are required to even start the app!!!), 3rd party libs like ICU, ANGLE, SSL.
Your app won't run correctly if you missed any of those so what is the point of making a tool that will automatically copy like 30% of what's needed?Anyway once you know what is where a correct deployment is a matter of 2-3 lines of "robocopy":http://en.wikipedia.org/wiki/Robocopy batch file in post-build event. I'm using this with VS all the time and it's a non-issue.
-
Chris... I am NOT confusing things. You keep talking down to me about things I am already aware of. Whether dlls are used or libs depend on if the dlls have a manifest of their functions built in. I realize that in standard C++ this does not apply so the lib files are the only ones that are necessary. However, this does NOT mean that dlls have nothing to do with it. The name of the dll and ordinal from the table within the lib is STILL written into the executable. I mentioned both of these in my last post. In response You said that there is a lib file for every dll. Libs don't ALWAYS have the same name as the dll. Yet if the dll is missing, the executable will trigger a missing dialog containing the right name for the dll. I will say again: If the linker can identify the dll name then the SDK can certainly identify which DLLS are necessary. The process doesn't really matter. The concept is still the same.
Furthermore, I am not going to keep entertaining your continuing to ignore the focus of this thread. I am already hand copying the dlls from OTHER libraries that I MYSELF have incorporated into the project. I have mentioned several times that they are not the focus of this thread. I have no problem with that because they are not part of a professional sdk, I know which dlls to use and I am personally and purposefully including them. That is my decision and my part of the workload.
In the case of Qt and every other large SDK however, the exact distribution and location of the classes within the SDK is not necessarily known and not necessarily easy to determine by the programmer without spending a good deal of overhead time. Does documentation even exist on these locations? I have not seen any! (And yes... I am already aware of the class documentation. But as far as I have read, this documentation does not report which dll it belongs too.)
I will restate and maintain my original objection:
It is unprofessional that the only way to learn which dlls I need to hand copy into the executable directory is for me to run the executable and one by one learn the names of the dlls I need to copy. (Or to download a separate utility to analyze the executable and list the dlls it needs). Solving this issue would NOT be a difficult task.
-
[quote author="primem0ver" date="1396902552"]I will say again: If the linker can identify the dll name then the SDK can certainly identify which DLLS are necessary. The process doesn't really matter. The concept is still the same.[/quote]
No, it can't. That's the point ;) But ok, I'm sorry. I will not comment further on the dll mechanism since that is not the topic of this thread, although you did confuse stuff and twist my words again ;)There is official guide on how to deploy your app to specific platform "here":https://qt-project.org/doc/qt-5/deployment.html. For Windows it's "here":https://qt-project.org/doc/qt-5/windows-deployment.html
There are also a couple of user wikis on this site about specific platforms and issues.
As for automated tools there is only one for Mac that I know of, but judging from the amount of topics about it on these forums it's more trouble than gain (I don't own a Mac so I'm just speculating).Running your app and adding dll one by one is not the only way(and certainly not easiest or most reliable). The easiest that I myself have tried are those:
Running the app in the IDE and inspecting modules window in your IDE (both Qt Creator and Visual Studio have one).
Running an external tool like "Dependency Walker":http://www.dependencywalker.com/ in profile mode and inspecting its output.
You can also refer to the links Xander84, JKSH and I gave you.I hope this helps.
If you need any more specific functionalities feel free to let us know and maybe someone on this forum will be able to help you. -
Oh, and you mentioned there is no way to tell which class is in which module. It's actually documented. For example list of "GUI C++ module classes":https://qt-project.org/doc/qt-5/qtgui-module.html
-
[quote author="Chris Kawa" date="1396904008"][quote author="primem0ver" date="1396902552"]I will say again: If the linker can identify the dll name then the SDK can certainly identify which DLLS are necessary. The process doesn't really matter. The concept is still the same.[/quote]
No, it can't. That's the point ;) But ok, I'm sorry. I will not comment further on the dll mechanism since that is not the topic of this thread, although you did confuse stuff and twist my words again ;)[/quote]If you want to talk about the core 30% you mentioned previously then I am ok with that. In order for me to illustrate what I mean I am going to use an example that isn't part of the core 30% but it illustrates either my point or my misunderstanding:
I want you to explain to me how if as you say "it can't", how does my executable know to ask for zlib1.dll when the lib file is named zdll.lib? Are you saying the executable didn't ask the OS for that dll by name? Because the zlib1.dll is not a registered dll and I did not install it. I simply downloaded a zip that contained all the files used for developing programs that use the zlib library. I manually copied that library over to the executable directory. When I remove it I get the error that zlib1.dll is missing.
-
P.S. Thanks for the info. I was able to copy the dlls using the output as you suggested (and did so before). FYI though, the included percent of dlls tied to the core library that I am using based on classes is more like 70% so far.
-
Sorry... your link about documentation is the same documentation I looked at earlier. It doesn't have any information on the library I need to copy that I was able to find.
-
[quote author="primem0ver" date="1396906436"]
I want you to explain to me how if as you say "it can't", how does my executable know to ask for zlib1.dll when the lib file is named zdll.lib? Are you saying the executable didn't ask the OS for that dll by name? Because the zlib1.dll is not a registered dll and I did not install it. I simply downloaded a zip that contained all the files used for developing programs that use the zlib library. I manually copied that library over to the executable directory. When I remove it I get the error that zlib1.dll is missing.
[/quote]
Ok, but remember you asked for it :) Name of the .lib file doesn't matter. Its contents do. And its contents say that the library is named "zlib1". You can rename the .lib file if you want but it won't change the name of the library. This information from the .lib is placed into the .exe file. When it starts it asks for "zlib1" so the system looks for zlib1.dll. It can be retrieved from any of the predefined system locations (app location, system32 folder, PATH environment etc. in specific order). It may or may not be the same zlib1.dll that you have downloaded, depending of what else you have on your computer and where it is located. You put the copy of zlib1.dll that you know is right inside your app dir so that it will be picked up first, before any other possible location.As for the 70% - see! Only two users and how the experience varies. Imagine that there's a guy with 10% and another with 90%. That's the essence of the problem!
Doc question - Yes it does. GUI module means you need Qt5Gui.dll, Core module means you need Qt5Core.dll etc. On each class doc page there's also info at the top like: qmake: QT += core, which means you need Qt5Core.dll etc.
-
[quote author="Chris Kawa" date="1396908495"]
Ok, but remember you asked for it :) Name of the .lib file doesn't matter. Its contents do. And its contents say that the library is named "zlib1". You can rename the .lib file if you want but it won't change the name of the library. This information from the .lib is placed into the .exe file. When it starts it asks for "zlib1" so the system looks for zlib1.dll. It can be retrieved from any of the predefined system locations (app location, system32 folder, PATH environment etc. in specific order). It may or may not be the same zlib1.dll that you have downloaded, depending of what else you have on your computer and where it is located. You put the copy of zlib1.dll that you know is right inside your app dir so that it will be picked up first, before any other possible location.[/quote]This is exactly as I understood. So how am I twisting your words? And as I stated before, if the linker is capable of picking out the files needed by using the lib, then the publisher of the specific sdk certainly can know which files to copy. That takes care of 70% of the needed dll files. I think your are assuming that since I am saying A hence B that I am automatically associating the occurrence of A with the occurrence of B. That is not true. I am only saying that if one is possible, then the other is possible and that it wouldn't be that hard. I, myself could (and may) do this.
My project will also eventually include a kind of SDK with its own IDE (assuming it becomes a popular application). I can tell you that I will be able to map the dependencies that will exist to the classes in my project. If it ever becomes necessary to include dlls other than the one being created by the IDE (in order to access other classes within the SDK), then you better believe that I will include functionality to do so.
[quote author="Chris Kawa" date="1396908495"]
As for the 70% - see! Only two users and how the experience varies. Imagine that there's a guy with 10% and another with 90%. That's the essence of the problem!
[/quote]No, that is not the problem. It is only a problem with the remaining percent (and that percent is out of the scope of this thread). This thread is only suggesting a solution for the 10% for your other in your example, the 30% for you, and the 70% for me. If the mappings were taken care of then these percentages would already taken care of and the programmer would have to worry about the rest.
[quote author="Chris Kawa" date="1396908495"]
Doc question - Yes it does. GUI module means you need Qt5Gui.dll, Core module means you need Qt5Core.dll etc. On each class doc page there's also info at the top like: qmake: QT += core, which means you need Qt5Core.dll etc.[/quote]True. But that only accounts for 20% of the 70% needed dlls I have listed based on the classes I used. This is because as I expected, the classes are more compartmentalized than as documented.
-
This is going nowhere. It's like me saying X and Z getting to you. I tried my best. Maybe I just can't explain stuff clear enough in which case I'm sorry and I give up.
Maybe you implementing this is the better way indeed so I encourage you. Maybe then you will see all the problems along the way and understand.
If you do manage to pull that off for Qt on Windows, spanning not only your specific use-cases, I'd be very glad and one of the first to use it, trust in that so good luck. I just hardly see that coming. -
I am not saying that I am going to do this for qt. I have my own project to worry about (which I may do this for).
I am still not convinced you understand what I am trying to say either. So that X and Z thing may go both ways. Let me tell you how I would approach solving this issue by using a non-specific case/analogy and you can go pick out the areas where it is flawed.
Let us say that I develop a set of classes that I deliver as an SDK. Let us also say that this set of classes is large.
First, as convention would have it, I would divide the classes in the SDK into namespaces depending on the purpose they serve. Let us say that I have 10 namespaces that include all classes. For simplicity let us call them A, B, C... all the way through J.
Now lets say that that because this project is so big, I decide to divide those namespaces across several libraries each because not everyone will need all classes within a single namespace for all projects. Again, for simplicity let us say that each namespace gets divided into 10 libraries each. On a windows platform, namespace A for example is divided into A0.dll, A1.Dll, A2.dll,.. all the way to A9.dll. That gives us a total of 100 libraries (dlls on windows) in our SDK's bin collection. That is 100 files that the end-developer has to worry about choosing between if they need to do it by hand. That is 100 files that the end-developer has to memorize the layout of (or use another tool to figure out which dlls to use).
Now let us say that each namespace has a certain number of classes divided among those dlls. For argument's sake let us also say that the sdk is open source so people are free to alter it as they please.
SOLUTION: As the developer of the SDK, I will already know which class is in which library because I wrote the code that compiled into those libraries. Even if someone has multiple versions of the library, some written by other people, at least for my own sdk, I can figure out which libraries to include since the specific SDK being used is mine (as expressed explicitly in the project settings). I can write an IDE extension that among other things, runs a program at compile time that takes the build type (including debug/non debug, 32 or 64 bit, OS type, and processor type) as an argument and parses the developers code looking for class/struct header includes. Since as the developer of the SDK I know which class (or header) is compiled into which library file and which set of library files go with each build, I can, in that extension, copy the files necessary for the build to the final build folder.
It is true that people can alter my code and create problems. But since the sdk/build differences are explicitly stated in the project settings, the only way the above solution would not work (that I can see) is if people over-wrote the files in MY SDK with files that were altered by someone else. In that case, it is the end-developers problem for doing something so careless.
It is also true that this only takes care of files from my SDK that are specific to the classes in question. However, as I have agreed earlier, any other classes are the end-developers responsibility.
I hope that you followed that well enough to critique.
-
One more thing: As the developer I also know which classes have which dependencies so if a class needs one of the other dlls as a dependency, that can still be mapped into the solution.
-
The windeployqt tool that comes with Qt 5.2 and later does automatic DLL selection.
