Linker error while building qtwebkit statically against working qt 5.10.1 static build on windows

smihaila · wrote on 8 Mar 2018, 16:34

Hi,

I am getting linker errors at around 95% total progress when building the qtwebkit from sources, statically, against a working static qt-5.10.1 windows x64 build.

The qtwebkit source tree is Annulen's (Konstantin Tokarev) fork / backport, available from here:
https://github.com/annulen/webkit
http://code.qt.io/cgit/qt/qtwebkit.git/ (master aka stable aka 5.212 branch).

For the toolchain, I'm using Alexey's (alexpux) MSYS2 MingGW64 toolchain (gcc and g++ version 7.3.0).
The MSYS2 was installed under c:\msys64 and C:\msys64\mingw64\bin (where the toolchain binaries are) was added to win OS PATH.

I have managed to build OK the Qt 5.10.1 statically, from sources, on a windows 10 x64.
a. The configure + make options were the following (shadow build, command invoked via a cmd.exe prompt):

cd <my_empty_shadow_build_dir>
..\qt-everywhere-src-5.10.1\configure -v \
    -opensource -confirm-license \
    -release -static -static-runtime \
    -prefix c:\qt-5.10.1.win_x64_static \
    -platform win32-g++ \
    -nomake examples \
    -nomake tests \
    -opengl desktop
mingw32-make -j8
mingw32-make install

It seems that the qt5 build picked the std=c++1z dialect.

b. I have created a simple test app (exercising for example QWidgets and QSystemTray) and built it against the new static build qt-5.10.1. App worked and verified via depends.exe that I don't get undesired dynamic Qt5 libs.

And then I have attempted to continue with a static build for the QtWebKit:
a. The 1st thing I did was to apply Alexey's qtwebkit related patches from his MSYS2 git repo, available here:
https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-qtwebkit

b. But that did not seem enough.I had to do quite a few patches by hand to the Qt5 install dir by hand, such as:

.prl files having backslashes instead of forward slash
some of the .prl files not using the "lib" prefix in front of their names. Inside C:\qt5-5.10.1.win_x64_static\lib I copied Qt5Core.prl to libQt5Core.prl, same thing for Qt5Gui.prl, Qt5Network.prl, Qt5OpenGL.prl, Qt5Positioning.prl, Qt5PrintSupport, Qt5Sensors.prl, Qt5Test.prl, Qt5Widgets.prl. Even though the majority of the .prl files there do not have a "lib" name prefix.
c. missing Qt5Gui_PRIVATE_INCLUDE_DIRS paths in C:\qt-5.10.1.win_x64_static\lib\cmake\Qt5Gui\Qt5GuiConfig.cmake

c. Then I have kick-started a brand new qtwebkit build, using this (via a cmd.exe prompt):

cd <qtwebkit_local_git_repo>
REM to enable parallel builds, as qtwebkit builds take forever like 8 hours+
set MAKE_COMMAND="mingw32-make -j8"
perl Tools\Scripts\build-webkit --qt --release --cmakeargs="-Wno-dev -DCMAKE_PREFIX_PATH=c:/qt-5.10.1.win_x64_static -G\"MinGW Makefiles\""

Note 1: It seems that the qt5 build picked the std=c++11 dialect.
Note 2: ICU, libjpeg, libpng2, libxml2, libxslt and others seemed to have been picked from the MSYS2 installation (I have explicitly installed them via pacman, and always using the "mingw-w64-x86_64-" prefix for their package names.
So, I basically used this from a MSYS2 MING64 console:

pacman -S mingw-w64-x86_64-cmake mingw-w64-x86_64-libjpeg mingw-w64-x86_64-libpng mingw-w64-x86_64-icu mingw-w64-x86_64-libxml2 mingw-w64-x86_64-libxslt

8 hours later (time calculated without the -j8 make option), the build fails at approx 95%, during a linking phase.

The simplified output of the qtwebkit build is:

Checking Last-Modified date of qtwebkit-libs-win.zip...
Located a file of type application/octet-stream, of size 127044099.
Current qtwebkit-libs-win is up to date
Not searching for unused variables given on the command line.
-- Could NOT find Ruby (missing: RUBY_LIBRARY) (found suitable version "2.5.0", minimum required is "1.9")
-- Disabling ENABLE_QT_WEBCHANNEL since ENABLE_WEBKIT2 is disabled.
-- Found ICU header files in C:/msys64/mingw64/include
-- Found ICU libraries: C:/msys64/mingw64/lib/libicuuc.dll.a
-- Could NOT find Fontconfig (missing: FONTCONFIG_LIBRARIES FONTCONFIG_INCLUDE_DIR)
-- Could NOT find WebP (missing: WEBP_INCLUDE_DIRS WEBP_LIBRARIES)
-- Qt OpenGL implementation: GL
-- Qt OpenGL libraries: Qt5::Gui_glu32;Qt5::Gui_opengl32;Qt5::Gui_gdi32;Qt5::Gui_user32
-- Qt EGL libraries:
-- The following OPTIONAL packages have been found:

 * PkgConfig
 * Qt5OpenGL (required version >= 5.2.0)

-- The following REQUIRED packages have been found:

 * BISON (required version >= 2.1)
 * Gperf (required version >= 3.0.1)
 * Perl (required version >= 5.10.0)
 * PythonInterp (required version >= 2.7.0)
 * Sqlite
 * Threads
 * JPEG
 * PNG
 * ZLIB
 * ICU
 * LibXml2 (required version >= 2.8.0)
 * LibXslt (required version >= 1.1.7)
 * Qt5Core
 * Qt5Gui
 * Qt5Network
 * Qt5Widgets
 * Qt5Test
 * Qt5Positioning
 * Qt5Sensors
 * Qt5PrintSupport
   Required for ENABLE_PRINT_SUPPORT=ON
 * Qt5 (required version >= 5.2.0)
 * Qt5Multimedia (required version >= 5.2.0)
 * Qt5MultimediaWidgets (required version >= 5.2.0)
 * Ruby (required version >= 1.9)

-- The following OPTIONAL packages have not been found:

 * Fontconfig
 * WebP

-- Using platform-specific CMakeLists: C:/qtwebkit/Source/WTF/wtf/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Source/JavaScriptCore/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Source/JavaScriptCore/shell/PlatformQt.cmake
-- All-in-One build is enabled.
-- Using platform-specific CMakeLists: C:/qtwebkit/Source/WebCore/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Source/WebKit/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Source/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Tools/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Tools/DumpRenderTree/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Tools/ImageDiff/PlatformQt.cmake
-- Using platform-specific CMakeLists: C:/qtwebkit/Tools/TestWebKitAPI/PlatformQt.cmake
-- Enabled features:
--  ENABLE_ACCELERATED_2D_CANVAS ............ ON
--  ENABLE_ALLINONE_BUILD                     ON
--  ENABLE_API_TESTS ........................ ON
--  ENABLE_CSS_GRID_LAYOUT                    ON
--  ENABLE_DATABASE_PROCESS ................. ON
--  ENABLE_DATALIST_ELEMENT                   ON
--  ENABLE_DEVICE_ORIENTATION ............... ON
--  ENABLE_DRAG_SUPPORT                       ON
--  ENABLE_FULLSCREEN_API ................... ON
--  ENABLE_GAMEPAD_DEPRECATED                 OFF
--  ENABLE_GEOLOCATION ...................... ON
--  ENABLE_ICONDATABASE                       ON
--  ENABLE_INDEXED_DATABASE ................. ON
--  ENABLE_INSPECTOR_UI                       ON
--  ENABLE_LEGACY_WEB_AUDIO ................. OFF
--  ENABLE_LINK_PREFETCH                      ON
--  ENABLE_MEDIA_SOURCE ..................... OFF
--  ENABLE_NETSCAPE_PLUGIN_API                ON
--  ENABLE_OPENGL ........................... ON
--  ENABLE_PRINT_SUPPORT                      ON
--  ENABLE_QT_GESTURE_EVENTS ................ ON
--  ENABLE_QT_WEBCHANNEL                      OFF
--  ENABLE_SAMPLING_PROFILER ................ ON
--  ENABLE_SPELLCHECK                         ON
--  ENABLE_TOUCH_EVENTS ..................... ON
--  ENABLE_VIDEO                              ON
--  ENABLE_WEBKIT2 .......................... OFF
--  ENABLE_WEB_AUDIO                          OFF
--  ENABLE_X11_TARGET ....................... OFF
--  ENABLE_XSLT                               ON
--  USE_GSTREAMER ........................... OFF
--  USE_LIBHYPHEN                             OFF
--  USE_MEDIA_FOUNDATION .................... OFF
--  USE_QT_MULTIMEDIA                         ON
--  USE_SYSTEM_MALLOC ....................... 1
--  USE_WOFF2                                 ON
-- Configuring done
-- Generating done
-- Build files have been written to: C:/qtwebkit/WebKitBuild/Release
[  2%] Built target WTF
[  2%] Built target LLIntOffsetsExtractor
[  3%] Built target stageSharedScripts
[ 24%] Built target JavaScriptCore
[ 24%] Built target jsc
[ 26%] Built target ANGLESupport
[ 26%] Built target brotli
[ 27%] Built target woff2
[ 91%] Built target WebCore
[ 92%] Built target WebCoreTestSupport
[ 92%] Built target gtest
[ 92%] Automatic MOC for target WebKit
[ 92%] Built target WebKit_autogen
[ 94%] Built target WebKit
[ 94%] Automatic MOC for target WebKitWidgets
[ 94%] Built target WebKitWidgets_autogen
[ 95%] Built target WebKitWidgets
[ 95%] Automatic MOC for target tst_hybridPixmap
[ 95%] Built target tst_hybridPixmap_autogen
[ 95%] Linking CXX executable ..\..\..\..\tests\tst_hybridPixmap.exe
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qvideowidget.o):qvideowidget.cpp:(.text+0x2b5b): undefined reference to `QGLContext::currentContext()'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0xdfc): undefined reference to `QGLShaderProgram::removeAllShaders()'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x12b3): undefined reference to `QGLContext::getProcAddress(QString const&) const'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x23c2): undefined reference to `QGLShaderProgram::QGLShaderProgram(QGLContext const*, QObject*)'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x240e): undefined reference to `QGLContext::contextHandle() const'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x2f4a): undefined reference to `QGLShaderProgram::bind()'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x2f59): undefined reference to `QGLShaderProgram::enableAttributeArray(char const*)'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x2f8a): undefined reference to `QGLShaderProgram::setAttributeArray(char const*, float const*, int, int)'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x2fc3): undefined reference to `QGLShaderProgram::setUniformValue(char const*, float const (*) [4])'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x2ffb): undefined reference to `QGLShaderProgram::setUniformValue(char const*, int)'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x302a): undefined reference to `QGLShaderProgram::release()'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x3139): undefined reference to `QGLShaderProgram::setUniformValue(char const*, int)'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x369b): undefined reference to `QGLShaderProgram::addShaderFromSourceCode(QFlags<QGLShader::ShaderTypeBit>, char const*)'
C:/qt-5.10.1.win_x64_staticC:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x36c3): undefined reference to `QGLShaderProgram::link()'
C:/qt-5.10.1.win_x64_static/lib/libQt5MultimediaWidgets.a(qpaintervideosurface.o):qpaintervideosurface.cpp:(.text+0x3a50):
[...]
undefined reference to `QGLShaderProgram::~QGLShaderProgram()'
collect2.exe: error: ld returned 1 exit status
mingw32-make.exe[2]: *** [Source\WebKit\qt\tests\CMakeFiles\tst_hybridPixmap.dir\build.make:231: tests/tst_hybridPixmap.exe] Error 1
mingw32-make.exe[1]: *** [CMakeFiles\Makefile2:1106: Source/WebKit/qt/tests/CMakeFiles/tst_hybridPixmap.dir/all] Error 2
mingw32-make.exe: *** [Makefile:162: all] Error 2

Do you happen to know how to fix this?

I would love very much to have a static QtWebKit build working, so that I can implement an OAuth2-based login dialog in my desktop Qt5 app, which requires a web browser control to be embedded, only for that purpose alone (the rest is pure desktop app).

I am out of ideas and pulling my hair out as to why it fails the linking.

Other details that may be relevant:

I am also having in the system PATH "C:\Qt\5.10.1\Src\gnuwin32\bin" - for the bison and gperf (the C:\Qt\5.10.1 dir contains a standard, pre-build Qt installation, from Qt company - windows 32-bit mode, dynamic build).
I have also checked with objdump that C:\qt-5.10.1_win_x64_static\libs\libQt5MultimediaWidgets.a (which is about 184 KB in release mode) really contains the QGLContext and QGLShaderProgram C++ classes and their methods (I could see their C++ name manglings).

Would you think that it's the 7.3.0 mingw64 gcc/g++ toolchain creating this linker problems? But since all the compile and the linking always happens with the same toolchain, and my test app (linked against the qt5 5.10.1 static built libs) worked ok, I cannot explain.

Or, could it be that a difference in C++ dialect (c++2017 in qt5 vs. C++2011 in qtwebkit) creates a mismatch in bintools / .a static libs in regards to method signatures / c++ name mangling during linking time? I could hardly believe that the recent c++ standards are messing with the name mangling.

Finally, do you think it is worthwhile to repeat the experiment (both qt5 build and qtwebkit builds) using an older toolchain - such as mingw530_32 but the x64 flavor of it, which the official qt pre-build (dynamic mode) came with? Something that uses gcc / g++ 5.3.0? Does the 7.3.0 have issues / corruptions or trying to link in the same time both a dynamic and a static lib?

Thank you.

smihaila · wrote on 8 Mar 2018, 18:58

Maybe I should also apply these Qt5-related patches from Alexey's MINGW64 toolchain repo?
https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-qt5-static

But not all of them match. And some of them seem to mess the <qt5_install_dir>\lib\cmake*.cmake files to such extent that qtwebkits' configure phase would fail almost at the end, when it shows the submodules / features enabled or disabled (the summary).

The stats on those patches are as follows:

perfect match: 2, 3, 5, 11, 25, 28, 29, 31, 34, 35, 36, 38, 39, 40, 41, 43, 45, 46, 48, 51, 125, 300
partial match ("fuzz" level reported): 1-qt...., 6, 7, 8, 16, 17, 37, 49, 42
complete failure: 1, qml_static_plugin, 9, 10, 23, 24, 44, 50
these seemed to be already applied: 20, 30

It may very well be that the MinGW64 gcc / g++ 7.3.0 toolchain from Alexey is perfectly fine, but somehow the resulting qt5 build is not enabling 100% static libs? And those patches mentioned a few line above in this message, may help?

smihaila · wrote on 9 Mar 2018, 15:41

Some status updates:

I've got the qtwebkit to build ok, using these options:

perl Tools\Scripts\build-webkit \
	--qt \
	--release \
	--cmakeargs="-Wno-dev -DCMAKE_PREFIX_PATH=path_to_qt5_static_install_dir -G\"MinGW Makefiles\" -DENABLE_API_TESTS=OFF -DENABLE_TEST_SUPPORT=OFF" \
	--makeargs="-j8" \
	--install \
        --prefix=path_to_qt5_static_install_dir

(note the ENABLE_API_TESTS=OFF and ENABLE_TEST_SUPPORT=OFF defines (which seem to be effective in turning off some internal unit testing apps), plus the "--install" option.
The most important static .a libs that got built are: libJavaScriptCore, libWebCore, libWTF, libwoff2, libbrotli, libQt5WebKit and libQt5WebKitWidgets.

The install part of the build also seemed to follow the standard Qt "protocol" i.e. header, libs and .pri files landing in the right place.
The not-so-great news are:

The QtTestBrowser.exe app shows a message box like "Application failed to start because platform plugin “windows” is missing in path ''", even after copying sibling with it a "platform" dir with qt5 dll plugins etc. And then terminates. Considering the issues depicted in the points below, it is understandable why.
The mingw64 7.3.0 toolchain or qtwebkit build / cmake scripts themselves are doing something odd at linker time: what I have discovered was that the .a static files seemed to be "shallow", in the sense that instead of containing the actual .o / .obj object files, they contained only path strings / references to the actual .o files! Somehow the qt or qtwebkit cmake stuff, or the binutils (ar) part of the toolchain are performing in a very bizarre way. I have to re-assemble all the object files myself with the ar utility (3,000+ files) to finally see the static libs usable in my little test app.
My test app compiles ok, but fails to link - missing symbols are being reported in basic system libs brought by the toolchain itself (i.e. libws2_32.a, libshell32.a, libicu*.a, libxml2.a, libxslt.a, etc). Which is very strange because the libs are really there and are truly static i.e. I do see the allegedly missing symbols there. Because I saw __imp_xyz prefixes reported in those missing names.

I suspect that some mix-up occurs between static and import libs - very probably due to some of the libs brought by the qtwebkit itself inside "<qtwebkit_src_dir>\WebKitLibraries\win". Although those end up in .lib and not having the "lib" prefix in their name, so very probably they are msvc-based. And the qtwebkit's configure reported the dependent libs as coming from somewhere else i.e. the toolchain.

Very strange.
Today I'll redo all these steps with msvc - even if it fails at least it may give me more insights as to what happens under the hood with qtwebkit build script. And even if that doesn't work, it helps clear my mind a little and make me ready to tackle using an older gcc toolchain or do other patching in qt5 itself.

Thank you.