The cost of plugin scans

[elahav]

I investigated why it takes 14 seconds to start a simple Qt app on a particular system (QNX on RPi), and found out that it is the result of a scan Qt makes for plugin libraries. Every time an application starts, QFactoryLoader::update() will go through every file in both the application binary's path and the QPA plugin path, open and map the file, and then search for some magic value indicating it is a plugin. The app I was testing happened to be in the standard binary path and the QPA plugin path set to the standard library path, resulting in hundreds of files being opened and mapped for no good reason.

Putting the binary in its own folder and the Qt plugins in their own folder cuts down the load time to under one second.

Am I missing something? Is there some configuration option that avoids this scan? Surely on any system, and especially an embedded one, this kind of shooting in the dark in search of plugins is not the way to go.

--Elad

[SGaist]

Hi,

In this case I would say that the embedded system configuration is wrong. A normal Qt installation has the plugins stored in a specific path following a standard structure. The fact in yours the plugins have been put beside other libraries shows that something went wrong at some point in that regard.

[elahav]

@SGaist Thanks for the response. Sure, I can (and did) move the plugins to another location, but:

1. How is someone building a system supposed to know that?
2. All files in the directory where the executable resides still get scanned. I can have a 10GB database next to my executable, which will then be opened and mapped. Or I can have a thousand image files for the UI.

So, if I am not missing an option to configure Qt differently, I think it is misbehaving.

--Elad

[SGaist]

Not knowing how your custom system is built I can't really point you to the best resources for that.

1. By checking the Qt documentation (and possibly also what other projects like yocto do)
2. Is there some qt.conf file in your system ?

Unless some customization is done, Qt will only scan the appropriate folders that are created when you install it.

That said, do you have custom plugins for your application ?

[elahav]

No, only one plugin (the platform plugin), which is shared among all Qt applications.
The point I am trying to make (and perhaps belongs in a bug report) is that opening and mapping all files, even in a known directory, is bad practice in general, but especially on an embedded system. There has to be a better way to let Qt know which are the plugins of interest (e.g., using a manifest file, or just a naming convention).

--Elad

[SGaist]

I understand your issue.

To the best of my knowledge there is none currently. If memory serves well the mapping is used to get the plugin metadata in order to load the ones requested.

From a deployment point of view, what I would do (and have done in the past) is to customize the cross-compiled version to use only what was needed and thus also deploy only what was required. This minimized the footprint of Qt as well as the application.

[hskoglund]

Hi, one other possibility is if you can build a static version of Qt and build your app with that. Then all plugins are placed together with your code in the same .exe/.elf file.
That executable file will be bigger, say around 20 - 40 MB, but no file mappings/scans are needed for the plugins :-)

[elahav]

@hskoglund Sure, but I have many Qt apps (in fact a whole desktop system of Qt apps...). Some of these use QtWebkitWidgets, which is not a small library.

I can easily hack QFactoryLoader() to consider only files with a given name prefix/suffix. I posted here just to know if there is an existing way of avoiding the scan and if anyone else has noticed the problem.

--Elad

[hskoglund]

And I assume you also have tried forcing/resetting the scan of plugins to a specific folder by using QCoreApplication::setLibraryPaths just before your QApplication constructor call?

[elahav]

Good idea, but it doesn't help.

int main(int argc, char *argv[])
{
    QCoreApplication::setLibraryPaths(QStringList() << "/system/lib/plugins");

    QApplication app(argc, argv);
    qDebug() << app.libraryPaths();

produces

("/system/lib/qt", "/system/lib/plugins", "/system/opt/swmlogin")

The path from QT_QPA_PLATFORM_PLUGIN_PATH is still there but, more importantly, so is the path of the executable. The latter is mentioned in the documentation of setLibraryPaths() as still being searched. Not to mention having to change the code of every application I use is not a viable solution.

Anyway, I'll create a bug report at least to provide an option to suppress/restrict scanning.

--Elad

[hskoglund]

Hi, forgot one more option: create a qt.conf file next to your executable, with something like this inside:

[Paths]
Prefix=/system/lib
Plugins=/system/opt/swmlogin

i..e set Plugins to the path of your executable's folder, then Qt will look for the platforms subdirectory in that folder.
(I think Qt's factoryloader does a deduplication of the entries so that it will only scan the folder once)
And also no modification of source code needed :-)

[elahav]

Hmm... qt.conf seems to have no effect, Strange.

By the way, it looks like Windows only scans for files with the suffix .dll and Android only scans for files starting with libplugins_. From QFactoryLoader::update():

        QStringList plugins = QDir(path).entryList(
#if defined(Q_OS_WIN)
                    QStringList(QStringLiteral("*.dll")),
#elif defined(Q_OS_ANDROID)
                    QStringList(QLatin1String("libplugins_%1_*.so").arg(d->suffix)),
#endif
                    QDir::Files);

[elahav]

I rearranged things on the system:

1. Dropped QT_QPA_PLATFORM_PLUGIN_PATH
2. Set QT_PLUGIN_PATH to /system/lib/qt/plugins
3. Created a folder called platforms under the plugin path and moved the platform plugin there

Qt no longer maps any file other than the platform plugin, which is what I wanted from the start. I still don't understand the scan logic, but at least this problem is solved.

Thanks,
--Elad