Definitive guide to building source from downloaded .run file?

SGaist

Hi,

@jsulm Just one small correction, you can also get the sources through the installer.

KH-219Design

@Tom-asso said in Definitive guide to building source from downloaded .run file?:

I've downloaded qt-opensource-linux-x64-5.14.2.run, and executed it on my ubuntu system. So now I have the source files as well as libraries on my system. Now I want to modify some of those sources and rebuild the libraries. This is turning out to be not so simple . Is there a definitive source that describes how to do this?

I would be fooling myself if I tried to call this "definitive", but it does comprehensively describe how I build from source on Linux:

https://github.com/219-design/build_qt_binaries/blob/09ad9369c8abfead615075f067f402fd89f/script.sh

^^ The above bash script is short (~100 lines), and not very complex.

It does all of:

clone the source from code.qt.io
run configure with a custom -prefix setting to install the binaries in a subfolder (not system /usr area)
make and make install

I use this to do exactly what you are asking about: to make modifications to some of the sources and then rebuild and test my modified Qt code. (Usually as a simple learning exercise, to gain a better understanding of the Qt internals.)

Tom asso

Thanks very much @jsulm and @SGaist
Created a ~/qt-download directory, downloaded official release qt-everywhere-src-5.14.2, extracted into ~/qt-download/. I see that qt-everywhere-src-5.14.2 has a top-level configure file, consistent with build instructions at https://doc.qt.io/qt-5/linux-building.html. Following those instructions, I executed:

% configure
% make

Now I want to link my application with the 5.14.2 libraries that I just built. So I go to my application project directory and try to generate a Makefile configured for those libraries. myProject.pro specifies:

QT += quick
QT += datavisualization

Then I execute the qmake in qt-everywhere-src-5.14.2, but it gives an error:

% ~/qt-download/qt-everywhere-src-5.14.2/qtbase/bin/qmake
Project ERROR: Unknown module(s) in QT: quick datavisualization

Why can’t qmake find quick and/or datavisualization? Building in qt-everywhere-src-5.14.2 certainly generated many qtquick libraries as well as libdatavisualizationqml2.so.

What am I missing?

Thanks!

KH-219Design

% ~/qt-download/qt-everywhere-src-5.14.2/qtbase/bin/qmake

^^ Regarding the qmake binary that you invoked...

Is this qmake binary:

one that was part of your initial download? Or
one that got compiled when you executed configure and then make?

It needs to be the latter.

In order to compile a project using the Qt binaries that you hand-built, you need to invoke the qmake that was also hand-built at the time you built Qt.

(Sorry if I am stating something that was already known to you. Just stating it out of an abundance of caution.)

Tom asso

Thanks @KH-219Design - that qmake was built during the configure step (thanks for the explanation of how 'qmake' should be used - I am still learning!)
I can see from the output of 'make' that many datavisualization and quick modules were compiled, so I don't understand why qmake is having problems...
Strange, I also see that 'make' apparently did not build qtcreator that I can find...

Thanks!

KH-219Design

Ok. I still have a slight inkling that you might be using a qmake that is other than the "ideal" one.

If you can decide what is the "outermost" enclosing directory (folder) that holds ALL you recent experimentation, then in a terminal window go to that outermost directory and run this:

find . -type f -name qmake

List back here all the qmake instances that were found. If there are more than one, then you make just be using the "wrong" one to build your project.

Tom asso

Now qmake runs without error on my application pro file. Not only do I have to directly execute the qmake that was built as part of the source download, but my $PATH must include other binary executables that were also built. So I ran 'make install' on the Qt source, which installed things in /usr/local/Qt5.14.2/bin, added that directory to PATH, and now qmake generates a Makefile.

KH-219Design

@Tom-asso said in Definitive guide to building source from downloaded .run file?:

Now qmake runs without error on my application pro file. Not only do I have to directly execute the qmake that was built as part of the source download, but my $PATH must include other binary executables that were also built. So I ran 'make install' on the Qt source, which installed things in /usr/local/Qt5.14.2/bin, added that directory to PATH, and now qmake generates a Makefile.

Hooray! Yes, that all sounds correct. Good on you for solving it, and thanks for posting the whole solution!

Tom asso

But now a new problem. After running qmake on my project .pro file, generating a Makefile, then running 'make' to build my application, my application executable ("mbgrdviz-2") does not run:

% ./mbgrdviz-2
qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in ""
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, webgl.

Aborted (core dumped)

Anyone know why? I built the Qt sources with configure, make, make install. Then built my app with qmake then make. What am I missing?

KH-219Design

I think you now need this next tip (referring back again to the Linux getting started notes from my own team):

If the app fails with a message such as qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "/home/daniel/Qt/5.14.0/gcc_64/plugins" even though it was found., you need to set QT_QPA_PLATFORM_PLUGIN_PATH to the correct location. And when you set that, you must ensure that LD_LIBRARY_PATH also points to the same Qt build.

 export QT_QPA_PLATFORM_PLUGIN_PATH="$(pwd)/build_qt_binaries/qt5_opt_install/plugins/platforms
 export LD_LIBRARY_PATH="$(pwd)/build_qt_binaries/qt5_opt_install/lib:$LD_LIBRARY_PATH"

^^ obviously you need to adjust the paths a bit, since on your setup the plugins and libs are not under "build_qt_binaries/qt5_opt_install"

Tom asso

@KH-219Design - I did 'make install' after building the source, which put stuff in /usr/local/Qt5.14.2, including in /usr/local/Qt5.14.2/ilbs and /usr/local/Qt5.14.2/plugins/platforms.

So:

% export LD_LIBRARY_PATH=/usr/local/Qt-5.14.2/lib
% export QT_QPA_PLATFORM_PLUGIN_PATH=/usr/local/Qt-5.14.2/plugins/platforms

But my app still fails to find the xcb plugin, and I can see why because 'make' of the Qt source code didn't build an xcb plugin and it wasn't installed:

ls /usr/local/Qt-5.14.2/plugins/platforms
libqeglfs.so       libqminimal.so    libqwayland-egl.so
libqlinuxfb.so     libqoffscreen.so  libqwayland-generic.so
libqminimalegl.so  libqvnc.so        libqwebgl.so

I.e. there is no "libxcb.so" or anything like that in there, although I see there are many source files with names containing "xcb". Per this page I tried running 'configure' with the -qt-xcb option:

./configure -qt-xcb
[...]
ERROR: Feature 'xcb' was enabled, but the pre-condition 'features.thread && features.xkbcommon && libs.xcb' failed.

Not sure what that means...
I tried to manually build libxcb within the Qt source directories - that built libxcb_static.a, but then failed:

% cd qtbase/src/plugins/platforms/xcb
% qmake   # Build Makefile from xcb.pro
% make
[...]
ar cqs ../../../../../lib/libxcb-static.a .obj/xfixes.o .obj/randr.o .obj/shm.o .obj/sync.o .obj/render.o .obj/shape.o .obj/xkb.o .obj/xinerama.o .obj/xinput.o .obj/xcb_aux.o .obj/atoms.o .obj/event.o .obj/xcb_image.o .obj/keysyms.o .obj/util.o .obj/icccm.o
make[1]: Leaving directory '/home/oreilly/qt-download/qt-everywhere-src-5.14.2/qtbase/src/plugins/platforms/xcb/xcb-static'
( test -e Makefile.xcb_qpa_lib || /home/oreilly/qt-download/qt-everywhere-src-5.14.2/qtbase/bin/qmake -o Makefile.xcb_qpa_lib /home/oreilly/qt-download/qt-everywhere-src-5.14.2/qtbase/src/plugins/platforms/xcb/xcb_qpa_lib.pro ) && make -f Makefile.xcb_qpa_lib 
Project ERROR: Could not find feature xcb-xinput.
Makefile:90: recipe for target 'sub-xcb_qpa_lib-pro-make_first-ordered' failed
make: *** [sub-xcb_qpa_lib-pro-make_first-ordered] Error 3

Man, I have no idea how you guys are able to get this to work ;-)

KH-219Design

Man, I have no idea how you guys are able to get this to work ;-)

Heh. I think it is like eating spicy food. First you develop a tolerance for it gradually, then you actually start to crave and enjoy it ;)

Hmmm....

I've got 2 thoughts now.

The most direct attack would be to get further into this:

./configure -qt-xcb
[...]
ERROR: Feature 'xcb' was enabled, but the pre-condition 'features.thread && features.xkbcommon && libs.xcb' failed.

That seems to me like a good approach, and there must be more info "somewhere" about that failure. Isn't there a config.log file somewhere, as an output from the configure? Its contents should look roughly like:

... truncated ...

executing config test linuxfb
+ cd /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/linuxfb && PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /opt/repositories/fc-build-01/qt5_dbg_configuration/qtbase/bin/qmake "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared warn_off console single_arch" "QMAKE_CFLAGS += --sysroot=/" "QMAKE_CXXFLAGS += --sysroot=/" "QMAKE_LFLAGS += --sysroot=/" -early "CONFIG += cross_compile" /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/linuxfb
+ cd /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/linuxfb && MAKEFLAGS= /usr/bin/make
> g++ -c -pipe --sysroot=/ --sysroot=/ -g -w -fPIC  -I. -I/opt/repositories/qt5_sources/qtbase/mkspecs/linux-g++ -o main.o main.cpp
> g++ --sysroot=/ --sysroot=/ -o linuxfb main.o      
test config.qtbase_gui.tests.linuxfb succeeded
looking for library mtdev
Trying source 0 (type pkgConfig) of library mtdev ...
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --exists --silence-errors mtdev
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --modversion mtdev
> 1.1.5
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --libs-only-L mtdev
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --libs-only-l mtdev
> -lmtdev
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --cflags mtdev
+ cd /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/mtdev && PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /opt/repositories/fc-build-01/qt5_dbg_configuration/qtbase/bin/qmake "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared warn_off console single_arch" "QMAKE_CFLAGS += --sysroot=/" "QMAKE_CXXFLAGS += --sysroot=/" "QMAKE_LFLAGS += --sysroot=/" -early "CONFIG += cross_compile" 'QMAKE_USE += mtdev' 'QMAKE_LIBS_MTDEV = -lmtdev' /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/mtdev
+ cd /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/mtdev && MAKEFLAGS= /usr/bin/make
> g++ -c -pipe --sysroot=/ --sysroot=/ -g -w -fPIC  -I. -I/opt/repositories/qt5_sources/qtbase/mkspecs/linux-g++ -o main.o main.cpp
> g++ --sysroot=/ --sysroot=/ -o mtdev main.o   -lmtdev   
 => source accepted.
test config.qtbase_gui.libraries.mtdev succeeded
looking for library openvg
Trying source 0 (type pkgConfig) of library openvg ...
+ PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /usr/bin/pkg-config --exists --silence-errors vg
pkg-config did not find package.
  => source produced no result.
Trying source 1 (type makeSpec) of library openvg ...
VG/openvg.h not found in [] and global paths.
  => source produced no result.
test config.qtbase_gui.libraries.openvg FAILED
executing config test qpa_default_platform
test config.qtbase_gui.tests.qpa_default_platform gave result xcb
looking for library harfbuzz
Trying source 0 (type inline) of library harfbuzz ...
+ cd /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/harfbuzz && PKG_CONFIG_SYSROOT_DIR=/ PKG_CONFIG_LIBDIR=//usr/lib/pkgconfig://usr/share/pkgconfig://usr/lib/x86_64-linux-gnu/pkgconfig /opt/repositories/fc-build-01/qt5_dbg_configuration/qtbase/bin/qmake "CONFIG -= qt debug_and_release app_bundle lib_bundle" "CONFIG += shared warn_off console single_arch" "QMAKE_CFLAGS += --sysroot=/" "QMAKE_CXXFLAGS += --sysroot=/" "QMAKE_LFLAGS += --sysroot=/" -early "CONFIG += cross_compile" 'QMAKE_USE += harfbuzz' 'QMAKE_LIBS_HARFBUZZ = -lharfbuzz' /opt/repositories/fc-build-01/qt5_dbg_configuration/config.tests/harfbuzz

... truncated ...

Hopefully there would be a more precise answer in there as to exactly what steps caused "features.xkbcommon" (and the others) to fail.

A less direct attack would be:

... if we assume the configure failed because you simply need more dev headers of more cascading dependencies to be available on your machine for the compiler to use...

Then if you try this and things get installed:

sudo sed -i -- 's/#deb-src/deb-src/g' /etc/apt/sources.list
sudo sed -i -- 's/# deb-src/deb-src/g' /etc/apt/sources.list
sudo apt-get update

sudo apt-get build-dep qt5-default

Then just try ./configure -qt-xcb again (or even a bare ./configure), and this time configure should be able to detect more build dependencies on the system.

Tom asso

@KH-219Design - I will try that. But in the meantime I actually built shared library libxcb.so "by hand" (since 'make' failed) and copied it into the plugins/platforms directory:

% g++ -shared -o libxcb.so *.o
% sudo cp libxcb.so /usr/shared/Qt5.14.2/plugins/platforms/.

I then rebuilt my app and tried to run it:

% qmake
% make clean
% make
% ./mbgrdviz-2
qt.qpa.plugin: Could not find the Qt platform plugin "xcb" in "/usr/local/Qt-5.14.2/plugins/platforms"
This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix this problem.

Available platform plugins are: eglfs (from /usr/local/Qt-5.14.2/plugins/platforms), linuxfb (from /usr/local/Qt-5.14.2/plugins/platforms), minimal (from /usr/local/Qt-5.14.2/plugins/platforms), minimalegl (from /usr/local/Qt-5.14.2/plugins/platforms), offscreen (from /usr/local/Qt-5.14.2/plugins/platforms), vnc (from /usr/local/Qt-5.14.2/plugins/platforms), wayland-egl (from /usr/local/Qt-5.14.2/plugins/platforms), wayland (from /usr/local/Qt-5.14.2/plugins/platforms), webgl (from /usr/local/Qt-5.14.2/plugins/platforms), eglfs, linuxfb, minimal, minimalegl, offscreen, vnc, wayland-egl, wayland, webgl.

So although the library I built, libxcb.so, is in /usr/shared/Qt5.14.2/plugsins/platforms, it is not detected as a "plugin" - why might that be? Is there some special step to "register" a plugin?
Thanks!

Tom asso

@KH-219Design - taking the "direct attack" by looking into the config.log file... I just installed the ubuntu package for xkbcommon (apt install libxkbcommon-dev), verified presence of /usr/include/xkbcommon/xkbcommon.h. Then re-ran "./configure -qt-xcb" which again gives "ERROR: Feature 'xcb' was enabled, but the pre-condition 'features.thread && features.xkbcommon && libs.xcb' failed." I look in config.logs and see these lines:

loaded result for library config.qtbase_gui.libraries.xkbcommon
Trying source 0 (type pkgConfig) of library xkbcommon ...
+ /usr/bin/pkg-config --exists --silence-errors xkbcommon '>=' 0.5.0
pkg-config did not find package.
  => source produced no result.
Trying source 1 (type inline) of library xkbcommon ...
xkbcommon/xkbcommon.h not found in [] and global paths.
  => source produced no result.
test config.qtbase_gui.libraries.xkbcommon FAILED

So configure cannot find /usr/include/xkbcommon/xkbcommon.h - even though it exists?

KH-219Design

regarding the question about not finding a plugin... I'm not really sure, but you might try running ldd on the so you built.

if ldd produces anything like either of these, then you would be missing load-time dependencies for your so:

bad outcome #1:

ldd libQt5Gui.so.5.14.0
# this message about the version not found is bad:
./libQt5Gui.so.5.14.0: /usr/lib/x86_64-linux-gnu/libQt5Core.so.5: version `Qt_5.14' not found (required by ./libQt5Gui.so.5.14.0)
	linux-vdso.so.1 (0x00007ffe7e3e5000)
	libQt5Core.so.5 => /usr/lib/x86_64-linux-gnu/libQt5Core.so.5 (0x00007f30cdac6000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f30cd8a7000)

bad outcome #2 ("not found"):

ldd libQt5Gui.so.5.14.0
	linux-vdso.so.1 (0x00007ffc16ba5000)
	libQt5Core.so.5 => not found                  # <-- anything "not found" is a problem
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f624ce6a000)
	libGL.so.1 => /usr/lib/x86_64-linux-gnu/libGL.so.1 (0x00007f624cbde000)
	libharfbuzz.so.0 => /usr/lib/x86_64-linux-gnu/libharfbuzz.so.0 (0x00007f624c940000)

(Note: I ran ldd on libQt5Gui only for the sake of generating sample output. I am not suggesting you ldd libQt5Gui, but rather that you ldd libxcb.so )

Regarding your results from the direct attack on configure failures:

+ /usr/bin/pkg-config --exists --silence-errors xkbcommon '>=' 0.5.0
pkg-config did not find package.

You could run /usr/bin/pkg-config --libs xkbcommon and see what you get.

You could also run apt-cache show libxkbcommon-dev and make sure the version is greater than 0.5.0

Racking my brain trying to think what else...

I guess just make sure any "left over cruft" from previous configure attempts are interfering... (but I could not enumerate what kinds of cruft you should be looking for).

Tom asso

Running pkg-config from the command line:

% /usr/bin/pkg-config --libs xkbcommon
-lxkbcommon

And "apt-cache show libxkbcommon-dev" shows version 0.8.0-1.
Thanks for all your help @KH-219Design !
I've been evaluating Qt for a seafloor-mapping project for a few months now - Qt has its share of bugs, there are many many moving pieces to it, many of those pieces barely documented or not documented at all - a very steep learning curve, at least for me. I'm grateful that you and others in the community provide invaluable help!

SGaist

What Ubuntu version are you running ?

KH-219Design

@Tom-asso said in Definitive guide to building source from downloaded .run file?:

xkbcommon.h not found in [] and global paths

I'm reading through this right now... seems like the same "xkbcommon" problem: https://forum.qt.io/topic/100908/qt-5-12-2-raspberry-pi-x11/16

KH-219Design

@KH-219Design said in Definitive guide to building source from downloaded .run file?:

seems like the same "xkbcommon" problem: https://forum.qt.io/topic/100908/qt-5-12-2-raspberry-pi-x11/16

Maybe I spoke too soon. Although the same error is mentioned, that thread seems entirely RPi specific in the end. Arggg...

KH-219Design

I also found this: https://github.com/RudolfCardinal/camcops/issues/3

... which seems to circle back to:

sudo sed -i -- 's/#deb-src/deb-src/g' /etc/apt/sources.list
sudo sed -i -- 's/# deb-src/deb-src/g' /etc/apt/sources.list
sudo apt-get update

sudo apt-get build-dep qt5-default

Has that been tried yet? I've lost track :)