Implementing plugin dependencies

Martyn McFarquhar

@SGaist Thanks for the response. So I wonder how Qt Creator does it? It would be very restrictive to have plugins work like this. The idea was to have plugins work like packages in something like R or Python. Making each plugin entirely independent without the ability to use new functionality from one plugin in another kills this idea almost entirely. Are you certain that it’s not possible?

SGaist

Can you point at an example of creator plugin that should be doing this ?

Martyn McFarquhar

@SGaist Well all Qt Creator plug-ins depend on the Core plugin, but one example would be the vcsbase plugin, which depends on a number of other plugins: https://github.com/qt-creator/qt-creator/blob/master/src/plugins/vcsbase/vcsbase_dependencies.pri

kshegunov

Are we talking plugins or libraries (yes, plugins are libraries, but not vice versa)? I'm confused. A plugin is not going to link against another plugin, nor a library is going to do so. Plugins are to be loaded at runtime - either automatically or manually, so I really don't understand the LIBS variable being what it is.

Martyn McFarquhar

@kshegunov Thanks for the response. So we’re talking about plugins here. If I remove the LIBS declaration then the plugin won’t compile because it needs the source files from the plugin it depends on. If I include the source files then isn’t that just putting the dependent plugin sources into the binary, rather than them being loaded from the other plugin?

kshegunov

@Martyn-McFarquhar said in Implementing plugin dependencies:

So we’re talking about plugins here.

Good, then I don't see any reason for the plugins to be listed in the LIBS variable at all.

If I remove the LIBS declaration then the plugin won’t compile because it needs the source files from the plugin it depends on.

Why would it need source files? This sounds very, very wrong, beside the fact that LIBS has nothing to do with source files.

If I include the source files then isn’t that just putting the dependent plugin sources into the binary, rather than them being loaded from the other plugin?

Principally yes. And that probably sounds reasonable to many, that is until you start breaking the one definition rule, or make a great mess of some global symbol ...

A plugin is a self-contained unit, a binary on its own. A plugin should not just include anybody's sources, and a plugin shall not be responsible for its loading if it's dependent. I think you got this backwards is what I'm saying.
Say you have plugins A, B and C such that A and B have no dependencies on other plugins, but C has a dep on A. Then it's the application's responsibility to load A and B and it's A's responsibility to load C, it's not C's place to care who loaded it and why.

Basically loading goes like this:

Application -> A
    A -> C
Application -> B

Martyn McFarquhar

Perhaps it would be easier to understand what I'm trying to achieve (as it may not be possible) from the following example:

Say I am a developer who wants to make a new plugin and there's some useful function in another plugin that I want to include. I don't want to include the sources because then my code is tied to whatever version I compile with and any updates in the other plugin do not filter into mine. As such, I want those functions to be available through the loading of the other plugin. To do so, I should be able to somehow indicate that my plugin is dependent on the other plugin and then have some sort of higher level plugin manager take care of the loading order.

This is exactly the way Qt Creator works. In fact, if you look at the Qt Creator source code, the qtcreator.pri file seems to include plugin dependencies using LIBS (see lines 251-272: https://github.com/qt-creator/qt-creator/blob/master/qtcreator.pri)

@kshegunov said in Implementing plugin dependencies:

Why would it need source files? This sounds very, very wrong, beside the fact that LIBS has nothing to do with source files.

I understand what you're saying, but if I just include the headers then I get errors about missing vtables, presumably because the dependent plugin contains classes that need to be passed through MOC. Also, doing this would surely mean that the dependent plugin classes will get compiled into the binary, which could lead to the situation described above that I want to avoid.

Say you have plugins A, B and C such that A and B have no dependencies on other plugins, but C has a dep on A. Then it's the application's responsibility to load A and B and it's A's responsibility to load C, it's not C's place to care who loaded it and why.

Ok, so these seems more like an issue of design patterns, and I appreciate what you're saying. However, this just seems to be a problem of loading order, which I've already solved with a load queue based on the plugin dependencies. The issue is that simply loading the plugin doesn't seem to be enough. I don't see how the plugin would be able to use the functions from its dependency without linking them through LIBS, unless I need to go down the road of QLibrary and resolving the symbols manually?

It seems like what you are suggesting is counter to the way Qt Creator works. For instance, every plugin relies on the core plugin. But as far as I can tell, it is not necessary for every plugin to attempt to load the core when it runs. Again, it looks like Qt Creator does specifically link plugin dependencies to the main application using LIBS, unless I'm misunderstanding something?

kshegunov

@Martyn-McFarquhar said in Implementing plugin dependencies:

Say I am a developer who wants to make a new plugin and there's some useful function in another plugin that I want to include.

Which is a poor choice of architecture. @SGaist already said what's done usually in that case - have a library and link both plugins against it.

I don't want to include the sources because then my code is tied to whatever version I compile with and any updates in the other plugin do not filter into mine.

You also don't want to include the sources because:

It leads to (binary) code duplication
It can lead to global data duplication and/or global initialization bugs
You can easily violate the one definition rule

And these are reasons that, as already stated, much outweigh any convenience, 2 and 3 are actually bugs.

As such, I want those functions to be available through the loading of the other plugin. To do so, I should be able to somehow indicate that my plugin is dependent on the other plugin and then have some sort of higher level plugin manager take care of the loading order.

If that's what Creator is doing, maybe, however how that's wired I have no idea. Probably they have piles of custom code to facilitate it, for one there's nothing in Qt itself for to use ad hoc.

This is exactly the way Qt Creator works. In fact, if you look at the Qt Creator source code, the qtcreator.pri file seems to include plugin dependencies using LIBS (see lines 251-272: https://github.com/qt-creator/qt-creator/blob/master/qtcreator.pri)

I think you're missing a bit of "low level knowledge" on that one. Without claiming I know why they populate the LIBS like that, linking is not including code or dependencies, and quite importantly linking isn't the same as loading. Take windows as a case study, you have the import library (.lib) and the dynamic library (.dll), all the linker needs is the former - it needs to know what symbol is resolved how, that's all. You can happily link against a dll without having the actual dll. The loader on the other hand doesn't care about that implementation detail of the linker that's the import library, it just needs to load up the images, nothing more, so it only cares about the dlls.

I understand what you're saying, but if I just include the headers then I get errors about missing vtables, presumably because the dependent plugin contains classes that need to be passed through MOC.

moc has nothing to do with this. Errors about missing vtables means that the linker can't resolve the symbols, which on windows usually means someone forgot to export the class.

Ok, so these seems more like an issue of design patterns

No, this issue has nothing to do with design patterns, it's an issue of practicality and feasibility.

However, this just seems to be a problem of loading order, which I've already solved with a load queue based on the plugin dependencies.

And when you form a diamond, that is your dependency tree becomes a dependency graph? It's not purely a problem of loading order, by far.

The issue is that simply loading the plugin doesn't seem to be enough.

Well, that's going to map the image to memory, but to actually call something you'd need either the linker/loader (on linux the loader and the linker are basically the same thing) to resolve the symbols for you, or what you wrote:

I don't see how the plugin would be able to use the functions from its dependency without linking them through LIBS, unless I need to go down the road of QLibrary and resolving the symbols manually?

So either you need to link against the plugin or resolve the symbols one at a time at runtime (which is a massive PITA).

It seems like what you are suggesting is counter to the way Qt Creator works. For instance, every plugin relies on the core plugin. But as far as I can tell, it is not necessary for every plugin to attempt to load the core when it runs. Again, it looks like Qt Creator does specifically link plugin dependencies to the main application using LIBS, unless I'm misunderstanding something?

I don't know if you're misunderstanding, because I've not worked on Creator and I don't know how it does its plugin loading/management exactly. What I was suggesting is what Qt's own documentation is describing, I didn't invent it, but I agree with the approach. Personally, I find the idea of linking against a plugin rather abominable, even if not necessarily incorrect.

PS.
As far as I can understand what you're asking you're rather after a way to extend your application with libraries, instead of having plugins being loaded at runtime (which is also probably what Creator does).

Martyn McFarquhar

@kshegunov Thank you for the detailed response, I think I'm beginning to see more clearly how my intended design can be realised.

Let's start by not calling these entities "plugins" because I think there some issues with semantics that are making this discussion harder. So let's call them "modules" for now. So these "modules" are essentially dynamic libraries that share a common interface that allows them to communicate with the core application (add menu items, launch dialog windows etc.)

Let's say that there are a core set of modules that provide a lot of the functionality of the application. These are 1st-party modules that come bundled with the application. We still want them to be dynamic in the sense that we can pick and choose which modules to initialise at run-time (say we have different sets of modules for different "modes" of the application) but we also want to allow them to have dependencies on one another so the application functionality can be as modular as possible.

These core modules can then be linked with the application using LIBS because they are acting more like traditional dynamic libraries than plugins. The difference is largely that there is a common interface so the application can decide which modules to initialise.

Now, if another developer decides to design a new module that is provided as a *.dll, *.so or *.dylib then the application can use the same approach as the core modules in terms of initialisation. However, because these 3rd-party modules are not linked with the application they cannot have dependencies on each other, only on the modules included with the main application. In this sense, the 3rd-party modules act more like traditional plugins.

I guess my original confusion was in treating both these scenarios as if they were plugins, whereas really the issues with linking only correspond to 1st-party modules (which are acting more like libraries) with the above restrictions on 3rd-party modules. This wasn't helped by the fact that QtCreator seems to take a similar design philosophy and calls everything a plugin and talks about a "loading queue", which presumably just serves to check whether it is ok to initialise a "plugin" given the dependencies, rather than loading the "plugin" per se.

The solution to my original question was actually just to find a system for adding the modules to the LIBS variable of the main application so that they get linked correctly. Similarly, when deploying the application, any modules that are dependencies of another need to be specified as libraries. In my head it seemed wrong to be tying the "plugins" to the application in that way because I was still conceptualising these as independent units. In reality, these modules are really no different to other libraries, they just contain an interface for initialisation at runtime. The only drawback is that 3rd-party modules cannot depend on one another unless they are submitted for inclusion in the core pool of modules. But it seems like this is just the price to pay for this kind of system?

VRonin

I'm not discussing theory here, I'm far less qualified than @kshegunov so listen to him.

Coming back to the bare bones of your practical issue, I think what you need is just what this aricle describes

kshegunov

The only drawback is that 3rd-party modules cannot depend on one another unless they are submitted for inclusion in the core pool of modules. But it seems like this is just the price to pay for this kind of system?

Sorry for the long delay, I was traveling and had no time to actually look at the forum.
Yes, I would say that your summary is correct, and in that light I'd say what @VRonin had suggested as a way to build up the dependencies is probably best. That is - you have a set of core libraries that provide whatever they provide (which was what @SGaist mentioned in passing) and you stick to runtime loaded plugins being self-contained units that can't and/or won't depend on one another.