@SGaist I tried that but libqtlua still couldn't figured out to use uitools. I did got it to work by fixing the libqtlua.so library by adding /usr/local/src/qt5-build/qtbase/lib/libQt5UiTools.a to the generation of libqtlua.so via:
/usr/bin/c++ -fPIC -shared -Wl,-soname,libqtlua.so -o libqtlua.so CMakeFiles/qtlua.dir/qtluaconsole.cc.o CMakeFiles/qtlua.dir/qtluaenum.cc.o CMakeFiles/qtlua.dir/qtluaenumiterator.cc.o CMakeFiles/qtlua.dir/qtluafunction.cc.o CMakeFiles/qtlua.dir/qtluauseritem.cc.o CMakeFiles/qtlua.dir/qtluauseritemmodel.cc.o CMakeFiles/qtlua.dir/qtluauseritemselectionmodel.cc.o CMakeFiles/qtlua.dir/qtluauserlistitem.cc.o CMakeFiles/qtlua.dir/qtlualistiterator.cc.o CMakeFiles/qtlua.dir/qtluamember.cc.o CMakeFiles/qtlua.dir/qtluametacache.cc.o CMakeFiles/qtlua.dir/qtluamethod.cc.o CMakeFiles/qtlua.dir/qtluaplugin.cc.o CMakeFiles/qtlua.dir/qtluaproperty.cc.o CMakeFiles/qtlua.dir/qtluaqmetaobjecttable.cc.o CMakeFiles/qtlua.dir/qtluaqmetavalue.cc.o CMakeFiles/qtlua.dir/qtluaqmetaobjectwrapper.cc.o CMakeFiles/qtlua.dir/qtlualuamodel.cc.o CMakeFiles/qtlua.dir/qtluaqobjectiterator.cc.o CMakeFiles/qtlua.dir/qtluaqobjectwrapper.cc.o CMakeFiles/qtlua.dir/qtluaqtlib.cc.o CMakeFiles/qtlua.dir/qtluastate.cc.o CMakeFiles/qtlua.dir/qtluaitemviewdialog.cc.o CMakeFiles/qtlua.dir/qtluatablegridmodel.cc.o CMakeFiles/qtlua.dir/qtluatableiterator.cc.o CMakeFiles/qtlua.dir/qtluatabletreekeys.cc.o CMakeFiles/qtlua.dir/qtluatabletreemodel.cc.o CMakeFiles/qtlua.dir/qtluauserdata.cc.o CMakeFiles/qtlua.dir/qtluavaluebase.cc.o CMakeFiles/qtlua.dir/qtluavalue.cc.o CMakeFiles/qtlua.dir/qtluavalueref.cc.o CMakeFiles/qtlua.dir/qtluadispatchproxy.cc.o CMakeFiles/qtlua.dir/QtLua/moc_qtluaconsole.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluauseritemselectionmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluauseritemmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtlualuamodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluatabletreemodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluaitemviewdialog.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluatablegridmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluastate.cpp.o CMakeFiles/qtlua.dir/moc_qtluaqtlib.cpp.o /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.12.8 /usr/local/src/qt5-build/qtbase/lib/libQt5UiTools.a /usr/local/lib/liblua.a -lm -ldl /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5.12.8 /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.12.8 /usr/bin/c++ -fPIC -shared -Wl,-soname,libqtlua.so -o libqtlua.so CMakeFiles/qtlua.dir/qtluaconsole.cc.o CMakeFiles/qtlua.dir/qtluaenum.cc.o CMakeFiles/qtlua.dir/qtluaenumiterator.cc.o CMakeFiles/qtlua.dir/qtluafunction.cc.o CMakeFiles/qtlua.dir/qtluauseritem.cc.o CMakeFiles/qtlua.dir/qtluauseritemmodel.cc.o CMakeFiles/qtlua.dir/qtluauseritemselectionmodel.cc.o CMakeFiles/qtlua.dir/qtluauserlistitem.cc.o CMakeFiles/qtlua.dir/qtlualistiterator.cc.o CMakeFiles/qtlua.dir/qtluamember.cc.o CMakeFiles/qtlua.dir/qtluametacache.cc.o CMakeFiles/qtlua.dir/qtluamethod.cc.o CMakeFiles/qtlua.dir/qtluaplugin.cc.o CMakeFiles/qtlua.dir/qtluaproperty.cc.o CMakeFiles/qtlua.dir/qtluaqmetaobjecttable.cc.o CMakeFiles/qtlua.dir/qtluaqmetavalue.cc.o CMakeFiles/qtlua.dir/qtluaqmetaobjectwrapper.cc.o CMakeFiles/qtlua.dir/qtlualuamodel.cc.o CMakeFiles/qtlua.dir/qtluaqobjectiterator.cc.o CMakeFiles/qtlua.dir/qtluaqobjectwrapper.cc.o CMakeFiles/qtlua.dir/qtluaqtlib.cc.o CMakeFiles/qtlua.dir/qtluastate.cc.o CMakeFiles/qtlua.dir/qtluaitemviewdialog.cc.o CMakeFiles/qtlua.dir/qtluatablegridmodel.cc.o CMakeFiles/qtlua.dir/qtluatableiterator.cc.o CMakeFiles/qtlua.dir/qtluatabletreekeys.cc.o CMakeFiles/qtlua.dir/qtluatabletreemodel.cc.o CMakeFiles/qtlua.dir/qtluauserdata.cc.o CMakeFiles/qtlua.dir/qtluavaluebase.cc.o CMakeFiles/qtlua.dir/qtluavalue.cc.o CMakeFiles/qtlua.dir/qtluavalueref.cc.o CMakeFiles/qtlua.dir/qtluadispatchproxy.cc.o CMakeFiles/qtlua.dir/QtLua/moc_qtluaconsole.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluauseritemselectionmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluauseritemmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtlualuamodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluatabletreemodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluaitemviewdialog.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluatablegridmodel.cpp.o CMakeFiles/qtlua.dir/QtLua/moc_qtluastate.cpp.o CMakeFiles/qtlua.dir/moc_qtluaqtlib.cpp.o /usr/lib/x86_64-linux-gnu/libQt5Widgets.so.5.12.8 /usr/local/src/qt5-build/qtbase/lib/libQt5UiTools.a /usr/local/lib/liblua.a -lm -ldl /usr/lib/x86_64-linux-gnu/libQt5Gui.so.5.12.8 /usr/lib/x86_64-linux-gnu/libQt5Core.so.5.12.8
then type "make" and that's it. Once libqtlua.so has the Qt5UiTools symbols through the .a, then the problem is fixed. Wish it just worked out of the box though. Thanks.