Could not load the Qt platform plugin "xcb" in "" even though it was found.

JonB

I get the error message as per the title. As some of you know, I'm fairly good at helping others with their problems, now I'd appreciate some help with mine, as I am out of my depth/comfort area on this!

I start out with a working Qt 5.12.2 as supplied with Ubuntu 19.04. That is all fine.

I now receive a "project" I am supposed to work on from a third-party. It's for developing under VSCode, and is git-downloaded. It runs under a miniconda virtual environment which I have set up. When I try to run it I get the xcb failure. Setting export QT_DEBUG_PLUGINS=1 the relevant bit of the output is:

QFactoryLoader::QFactoryLoader() looking at "/home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so"
Found metadata in lib /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so, metadata=
{
    "IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
    "MetaData": {
        "Keys": [
            "xcb"
        ]
    },
    "archreq": 0,
    "className": "QXcbIntegrationPlugin",
    "debug": false,
    "version": 330752
}


Got keys from plugin meta data ("xcb")
QFactoryLoader::QFactoryLoader() checking directory path "/home/jon/miniconda3/envs/py37/bin/platforms" ...
Cannot load library /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so: (/home/jon/miniconda3/envs/py37/plugins/platforms/../../lib/libQt5XcbQpa.so.5: symbol _ZN11QFontEngine14bitmapForGlyphEj6QFixedRK10QTransformRK6QColor version Qt_5_PRIVATE_API not defined in file libQt5Gui.so.5 with link time reference)
QLibraryPrivate::loadPlugin failed on "/home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so" : "Cannot load library /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so: (/home/jon/miniconda3/envs/py37/plugins/platforms/../../lib/libQt5XcbQpa.so.5: symbol _ZN11QFontEngine14bitmapForGlyphEj6QFixedRK10QTransformRK6QColor version Qt_5_PRIVATE_API not defined in file libQt5Gui.so.5 with link time reference)"
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

This looks to me possibly like some Qt version mis-match?? A find in the miniconda/pkgs area shows many files containing qt-5.12.5, don't know if that is relevant.

The package supplied by the third-party is known to work under MacOS where they develop, precise Qt version unknown. They want it for Linux & Windows too, but probably have never tried it there. Note that it does not come with any kind of linuxdeployqt (or windeployqt) for setup.

How do I proceed now, please? Note that they will probably expect me to sort this out, as I don't think they have/want access to Linux.

JonB

Right, here is the reason/solution to this vexing problem:

I have already used pip to install PySide2, which goes into ~/.local/lib/python3.7/site-packages (Linux).
I am now trying to use a (supposedly) self-contained, standalone virtual environment (minconda) for a project. All its stuff got put into ~/miniconda3/envs/py37/lib/python3.7/site-packages, that includes (its own) PySide2.
When I strace where it decides to pick the libQt5Gui.so.5 mentioned in the error message, I see

openat(AT_FDCWD, "/home/jon/.local/lib/python3.7/site-packages/PySide2/libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/jon/.local/lib/python3.7/site-packages/PySide2/Qt/lib/libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = 4

so it's picking that up in the other PySide2's area.

If I rename so that ~/.local/lib/python3.7/site-packages/PySide2 directory cannot be found, the strace now shows

openat(AT_FDCWD, "/home/jon/miniconda3/envs/py37/lib/python3.7/site-packages/PySide2/../../../libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = 4

so it's picking it up in the desired PySide2's area. Lo & behold, the application finally works!!

Which is why, as I have said elsewhere, I just don't think these "virtual environments" are "robust"/"safe"/"clear" enough to feel comfortable with, at least from my experiences.

For anyone in a similar situation, I have found that running my python program under the virtual environment requires the -s command-line argument ("Don’t add the user site-packages directory to sys.path.") to prevent it from picking up that libQt5Gui.so.5 file in the ~/.local/lib/python3.7/site-packages area when it should be picking it up from where it is in the virtual environment area itself. Sad and frustrating, but true....

jsulm

@JonB said in Could not load the Qt platform plugin "xcb" in "" even though it was found.:

Cannot load library /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so: (/home/jon/miniconda3/envs/py37/plugins/platforms/../../lib/libQt5XcbQpa.so.5: symbol _ZN11QFontEngine14bitmapForGlyphEj6QFixedRK10QTransformRK6QColor version Qt_5_PRIVATE_API not defined in file libQt5Gui.so.5 with link time reference)

Do

ldd /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so

and check the output. As you already wrote you probably have other Qt libs there. You can try to set LD_LIBRARY_PATH to point to the directory containing your proper Qt libs and then start the app.

JonB

@jsulm
Already did the ldd. No "not founds". Most hits are in the miniconda3 area. the only ones which are not, if it's relevant, are:

	linux-vdso.so.1 (0x00007fff6bdd2000)
	libGL.so.1 => /lib/x86_64-linux-gnu/libGL.so.1 (0x00007fd916893000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd916872000)
	libX11-xcb.so.1 => /lib/x86_64-linux-gnu/libX11-xcb.so.1 (0x00007fd91686d000)
	libX11.so.6 => /lib/x86_64-linux-gnu/libX11.so.6 (0x00007fd916733000)
	libXrender.so.1 => /lib/x86_64-linux-gnu/libXrender.so.1 (0x00007fd9164f6000)
	libXext.so.6 => /lib/x86_64-linux-gnu/libXext.so.6 (0x00007fd9162e4000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fd916196000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd91613e000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd915dcb000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fd913b9a000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd91744a000)
	libGLX.so.0 => /lib/x86_64-linux-gnu/libGLX.so.0 (0x00007fd913b66000)
	libGLdispatch.so.0 => /lib/x86_64-linux-gnu/libGLdispatch.so.0 (0x00007fd913aa9000)

Haven't tried the LD_LIBRARY_PATH yet, not 100% where you want me to point that at?

jsulm

@JonB said in Could not load the Qt platform plugin "xcb" in "" even though it was found.:

not 100% where you want me to point that at?

In the terminal where you start your app after setting LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=PATH_TO_YOUR_QT_LIBS
./your_app

JonB

@jsulm
Sorry, my friend, but still not sure which/how many directories.

I have found that VSCode, which I am new to, is launching effectively via

env PYTHONPATH=/home/jon/Slicd/frontend PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /home/jon/miniconda3/envs/py37/bin/python -m src.frontend.main

so that's what I am using.

I have tried each of

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms    # location of `libqxcb.so`
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/    # location of `libQt5*.so`, inc. `libQt5XcbQpa.so.5` & `libQt5Gui.so.5`, I suspect this might be the one you mean?
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/qt5    # don't think it's this

but same result? Am I still using wrong path/need multiple ones?

jsulm

@JonB said in Could not load the Qt platform plugin "xcb" in "" even though it was found.:

not 100% where you want me to point that at?

Just realised I misunderstood that.
You can add all of them:

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms;/usr/lib/x86_64-linux-gnu/

But I'm not familiar with miniconda.
Does it contain its "own" Qt libs?

JonB

@jsulm
Stop-press! :) In your example, you mean a : where you have a ;, this is Linux not Windoze ;)

With both of those on my LD_LIBRARY_PATH, which I didn't know I needed, I now have an error message from the Python code, which I'm pretty sure means I have at last got past the "libraries" issue, and am good from here :) Sadly, no, see next post below....

Thank you so much! I don't really understand this area, having never had to look at it in previous projects without all this or the environment. Why do I need to put both those directories on LD_LIBRARY_PATH, how come it normally works/finds files in two different directories outside of the environment?

Yes, the miniconda virtual env has its own e.g. home/jon/miniconda3/envs/py37/lib/libQt5Gui.so.5 & /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so.

I'm unsure of the implications of mixing stuff from the venv and from my system install...?

JonB

@jsulm
Whoops, stop, it's not this simple!

I just found I was running from current directory ~/Slicd and that is when I get Python error. I had been running from ~/Slicd/frontend, which is where I thought I was supposed to (the instructions do not make this clear), and that still has the original error, regardless of LD_LIBRARY_PATH. So I may be no further forward, or I need clarity on where I'm supposed to run from.... Sigh :( :(

I'm afraid I can see from the Python error message that I am supposed to run from frontend (name of project) sub-directory, and that means the LD_LIBRARY_PATH has made no difference to behaviour (the Python error opening a log file path from the wrong starting directory must come before it tries to load whatever xcb is), we need to start again on resolution..... :(

Hmm, after

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/qt5/plugins/platforms:/usr/lib/x86_64-linux-gnu/
env PYTHONPATH=/home/jon/Slicd/frontend PYTHONIOENCODING=UTF-8 PYTHONUNBUFFERED=1 /home/jon/miniconda3/envs/py37/bin/python -m src.frontend.main

the error message still reports:

Cannot load library /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so: ...

and so on for the other file paths, just like originally, so it's still looking in the ~/miniconda/... area, not the system usr/lib/x86_64-linux-gnu/... area... ??

EDIT I'm just thinking: this is Python, there is no excutable for the Qt app it's interpreted Python, so these libraries (libqxcb.so etc.) are going to be dynamically loaded when required, not at start up like from C++ executable. Now I wonder whether Python helpfully(!) searches for these via something other than LD_LIBRARY_PATH...? But, say, https://stackoverflow.com/a/1100016/489865 implies it does use LD_LIBRARY_PATH for non-Python shared libs, I think.

So, for anyone reading this, I am still stuck with:

QLibraryPrivate::loadPlugin failed on "/home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so" :
"Cannot load library /home/jon/miniconda3/envs/py37/plugins/platforms/libqxcb.so:
(/home/jon/miniconda3/envs/py37/plugins/platforms/../../lib/libQt5XcbQpa.so.5:
symbol _ZN11QFontEngine14bitmapForGlyphEj6QFixedRK10QTransformRK6QColor version Qt_5_PRIVATE_API not defined in file libQt5Gui.so.5 with link time reference)"

I don't know what this (the last line) is telling me?

And changing LD_LIBRARY_PATH, if that's what I'm supposed to do, seems to have no effect on where Python/PySide2 dynamically loads these libraries from?

jsulm

@JonB said in Could not load the Qt platform plugin "xcb" in "" even though it was found.:

In your example, you mean a : where you have a ;, this is Linux not Windoze ;)

sure :-)

JonB

Right, here is the reason/solution to this vexing problem:

I have already used pip to install PySide2, which goes into ~/.local/lib/python3.7/site-packages (Linux).
I am now trying to use a (supposedly) self-contained, standalone virtual environment (minconda) for a project. All its stuff got put into ~/miniconda3/envs/py37/lib/python3.7/site-packages, that includes (its own) PySide2.
When I strace where it decides to pick the libQt5Gui.so.5 mentioned in the error message, I see

openat(AT_FDCWD, "/home/jon/.local/lib/python3.7/site-packages/PySide2/libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/jon/.local/lib/python3.7/site-packages/PySide2/Qt/lib/libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = 4

so it's picking that up in the other PySide2's area.

If I rename so that ~/.local/lib/python3.7/site-packages/PySide2 directory cannot be found, the strace now shows

openat(AT_FDCWD, "/home/jon/miniconda3/envs/py37/lib/python3.7/site-packages/PySide2/../../../libQt5Gui.so.5", O_RDONLY|O_CLOEXEC) = 4

so it's picking it up in the desired PySide2's area. Lo & behold, the application finally works!!

Which is why, as I have said elsewhere, I just don't think these "virtual environments" are "robust"/"safe"/"clear" enough to feel comfortable with, at least from my experiences.

For anyone in a similar situation, I have found that running my python program under the virtual environment requires the -s command-line argument ("Don’t add the user site-packages directory to sys.path.") to prevent it from picking up that libQt5Gui.so.5 file in the ~/.local/lib/python3.7/site-packages area when it should be picking it up from where it is in the virtual environment area itself. Sad and frustrating, but true....

SGaist

How did you create that conda environnement ?
How did you install PySide2 in it ?

JonB

@SGaist
So (trying to!) keeping it as brief as possible, the end result of my finding was:

I had initially installed PySide2 via pip, which is what you do. That meant it went into my ~/.local/... area. Other people should be in the same situation. This is a standard place for python to look for "site" packages. Note that this was not anything to do with a "virtual environment" at this stage, for right or for wrong.
To work on a separate project I was given a project using a .yml for its dependencies and a miniconda environment. That gave me a ~/miniconda/... with all python, PySide2, Qt stuff.
In theory, when I run the new app under conda environment it should find all its PySide2 & Qt libraries in the venv. That is what happens for another user of the project, who does not have a PySide2 or even Qt installed outside of the project/venv. If I rename my own ~/.local/.../PySide2 so it can't be found, the problem goes away.
Somehow, what happens is that libxcb.so is picked up in the conda environment, that loads libQt5XcbQpa.so also in the venv, but then that tries to load/access libQt5Gui.so, and picks up the one in my ~/.local, in preference to its own one in the venv (yes, its own is there, along with everything else). It will be that specific case, you'd have to look carefully at the error message.

It should not be the case that libraries in the virtual environment pick up outside-world ones in preference to their own, but that's what's happening. Which caused me a lot of grief to track down, and why my first experience with virtual environments has confirmed my worst fears! :(

I don't like pip either, but that's what PySide2 installs with, and I have had problems with pip-installed packages too in the past, where again they are not "robust" enough over their dependency searching, and have picked up inappropriate system files instead.

When I can do things with apt-get they just work, and it's easy to understand what is going on. Perhaps it's just me, these pips & venvs never seem to work out for my situations ;)

P.S.
BTW, just for fun, that's not quite the end of my woes. To work around without moving my outside-world .local/.../PySide2, I discovered running via python -s ... stops it reading from that site area. Great! Except: I have to use VSCode for this project, and I discover in VSCode you cannot control the command-line arguments to the python interpreter it runs! Oh no :( So I discover I can do this particular option instead via an environment variable I can export from VSCode, luckily.... Phew!