QT 64 bit on Windows env? mingw 64bit support? bad_alloc on 32 bit...

mbruel

Hi,
I'm having an issue with the heap allocation under windows 32 bit architecture. I'm parsing a "big" xml (90 MB) with QDomDocument::setContent(QIODevice *dev, bool namespaceProcessing, QString *errorMsg = Q_NULLPTR, int *errorLine = Q_NULLPTR, int *errorColumn = Q_NULLPTR) and it seems to need around 2GB of memory which is refused by windows 32Bit and brake with a bad_allocation error message.

On my Linux 64bit, I'm arriving to 1890920 KB just for the load of the XML and 3050420KB once my objects are loaded. So for what I read there is no way to cope with such amount of memory needed for a process under windows 32bit.

I'm running QT 5.5.1 because I'm using webkit which is removed on 5.6
I've seen on the archive page this version: qt-opensource-windows-x86-msvc2013_64-5.5.1.exe
What is it exactly? x86 but with ms compiler on 64bit, I'm confused... is that a full 64bit realease or the QT dlls are still in 32bit?

If I understood correctly, mingw has also a 64bit support. Is there a way to compile with it in 64bit?

Thanks in advance for your answers :)

jsulm

@mbruel May be you should not use QDomDocument, the documentation says:

Note: The DOM tree might end up reserving a lot of memory if the XML document is big. For such documents, the QXmlStreamReader or the QXmlQuery classes might be better solutions.

An application consuming 2GB of RAM to parse a XML file is not really what I as user would expect.

mbruel

@jsulm
well yeah I know that it was not the best idea to use DOM but the project was started like this 5 years ago and they didn't imagine arriving to such big inputs.
There was a "reason" to use DOM : it makes it much easier the process to translate the XML into the complex data structure.
I'm just getting the project to finish it and rewrite all the parsing would take quite some time and so some money for the client which I don't think is a priority.
We just load the DOM and destruct it once the internal structure is loaded so it is not a major issue to have to use 2GB and then release them (It seems that maybe not everything ends up released but that's another issue...).
I've no problem running it under a linux 64bit environment with 8GB of RAM. The client is using Windows 64Bit with 8 to 16GB of RAM. I'm pretty convinced that if I could deliver them a version in 64 bit, they wouldn't have this allocation issue.

jsulm

@mbruel qt-opensource-windows-x86-msvc2013_64-5.5.1.exe should be the x64 version. I usually use the online installer - there you can select what ever you need.

mbruel

@jsulm
ok thanks, I'm going to try to install it and see... I'm just confused by the naming of the exe...
And would prefer to stick with mingw_64 but it doesn't look like we have this option...

I'm using the offline installer cause I'm behind a proxy, but I'm having issue as I can't connect to the QT account...

jsulm

@mbruel You can use MinGW 64bit, but then you have to build Qt from sources.

mbruel

hum. I'll check compiling from the sources...
I've just installed qt-opensource-windows-x86-msvc2013_64-5.5.1.exe and it seems QT dlls are in 32 bits, not 64...
qmake adds by default the option: -spec win32-msvc2013
I don't really understand the purpose of this release if the dlls are in 32bits...

jsulm

@mbruel win32 doesn't mean it is 32bit. It means the win32 API.

mbruel

@jsulm
well I don't know much windows but if I open QT5CORE.DLL with dependancy walker, I see that it uses c:\windows\system32\Kernel32.dll, user32.dll, shell32.dll ...
The dlls in c:\windows\system32 are not in 32bits?

Cause there is another folder: c:\windows\SysWow64\ where I can find of those dlls...

how can I know if a dll or an exe is compiled in 32bits or in 64bits? (like would the command file on linux)

Let me know if I'm wrong but a program compiled in 64 bits can't load 32 bits dll right?

Buckwheat

@mbruel Something that hasn't been mentioned. 32-bit Windows has a stone ceiling of 3GB. Now subtract out 1/4 to 1/2 GB for windows and however large your app is. This is a huge problem in CAD applications and point cloud applications as well.

I have seen about "building Qt" as 64 bit. You can install Qt (currently 5.6.x) and QtCreator (currently 4.0.3) from MSYS2 at sourceforge. This will give you both the 32-bit and 64-bit kits for static and dynamic builds. The current GNU toolchain is 6.1.0 so you will default to c++14 as a compiler. So, get as modern as you'd like!

I have posted instructions (as close as I could remember) in other posts about MinGW32 builds wanting to go 64bit. You can find MSYS2 project on sourceforge. It is also mentioned in the 64bit WIKI on qt.io. I chose this route because I want to focus on the development and not building Qt.

jsulm

@mbruel This link http://download.qt.io/official_releases/qt/5.7/5.7.0/qt-opensource-windows-x86-msvc2013_64-5.7.0.exe
is described as "Qt 5.7.0 for Windows 64-bit (VS 2013, 904 MB)" (see https://www.qt.io/download-open-source/#section-2). So, qt-opensource-windows-x86-msvc2013_64-5.5.1.exe should be 64bit.
Again, this 32 in system32.dll does not mean it is 32bits, it is coming from win32 API name.

mbruel

@jsulm
well I'm confused about how Windows deals with its 64bits and 32bits dlls...

$ file  /cygdrive/c/Qt/Qt5.5.1_64msvc/5.5/msvc2013_64/bin/Qt5Core.dll
/cygdrive/c/Qt/Qt5.5.1_64msvc/5.5/msvc2013_64/bin/Qt5Core.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows

$ file /cygdrive/c/windows/system32/KERNEL32.DLL
/cygdrive/c/windows/system32/KERNEL32.DLL: PE32+ executable (DLL) (console) x86-64, for MS Windows

$ file /cygdrive/c/windows/sysWoW64/kernel32.dll
/cygdrive/c/windows/sysWoW64/kernel32.dll: PE32 executable (DLL) (console) Intel 80386, for MS Windows

So the 64bits versions are in system32 and the 32bit in sysWow64 ? strange naming...

I've installed qt-opensource-windows-x86-msvc2013_64-5.5.1.exe and wanted to my project and compile it to give it a go but it seems that msvc2013_64 doesn't come with it and I'm kind of struggling to find it... It seems there was no 64bit version of msvc2013...

mbruel

@Buckwheat
Same here I'd prefer to focus on code, but I'm having this 32bit Windows limitation on heap/stack size...
I don't see on msys2 how to get QT directly...
I need a specific version of QT (5.5.1) for compatibility reasons... I think I'll just try to compile it from the source if I don't manage to make the msvc run on 64bits...

Rondog

I believe the best solution is to fix the source of the problem. You should consider re-writing the XML parser. Qt has options to deal with this.

We just load the DOM and destruct it once the internal structure is loaded so it is not a major issue to have to use 2GB and then release them (It seems that maybe not everything ends up released but that's another issue...).

Based on this comment you are only using it to read or write the XML data. I know that QDomDocument is convenient but since it has a narrow purpose it can be re-written without huge changes to the overall program.

Switching to a 64 bit version won't reduce the memory footprint of the application. If anything it will use more than the 32 bit version (the plus is that you have more available). This is not really a good solution to this problem.

Buckwheat

@mbruel Look for my other posts on MSYS2. I go through a complete (well mostly) walkthrough using pacman (the MSYS2 package manager) to download. The nice thing is that you do not have to worry about the /cygdrive/... stuff that is not portable. You also get the GDB debugger and do not have to go through the trouble of downloading and installing CDB for windows.

I admit, the naming conventions on windows does not make sense. There is a lot of legacy and as stated just means "not windows 3.x" which was 16 bit.

As a quickie:
pacman -Ss qt
shows all the qt libraries
pacman -Ss toolchain
shows the compiler toolchains.

to install,
pacman -Su ...

pacman works similar to YUM and DNF.

This is explained better in my other posts about MinGW

Chris Kawa

well I'm confused about how Windows deals with its 64bits and 32bits dlls...

Just to clear up some naming confusion:

x86 doesn't really mean 32bit. There are various flavors of x86 architecture, including x86_64, which is the 64bit architecture. Unfortunately although not technically correct it has become common to call 32bit builds x86 and 64bit builds x86_64 or amd64.
qt-opensource-windows-x86-msvc2013_64-5.5.1.exe means 64bit Qt build for Visual Studio 2013.
qt-opensource-windows-x86-msvc2013-5.5.1.exe means 32bit Qt build for Visual Studio 2013.

Learning the lessons of switching from 16bit to 32bit Microsoft decided not to change library names when switching to 64bit to allow easier porting of applications (you don't have to link to libs named differently for 32 and 64 bit). That made a mess on its own, to which another pile of confusion is the 32bit emulation layer in 64bit Windows called WOW64 (which stands for "Windows On Windows 64bit"). Combining these two facts you have:

On 32bit Windows:
C:/Windows/System32/kernel32.dll - this is a 32bit dll

On 64bit Windows:
C:/Windows/System32/kernel32.dll - this is a 64bit dll (yes, exactly the same as above)
C:/Windows/SysWOW64/kernel32.dll - this is a 32bit dll

and no, there is no kernel64.dll anywhere.

As for VS2013 64bit there is no such thing. The IDE itself is 32 bit but it comes with a compiler for both 32 and 64 bit. When you install VS Qt Creator automatically picks up both versions (and some others, like ARM). When you compile Qt from command line you can select the 32 or 64 bit toolchain by running a vcvarsall.bat batch file from the VS directory with a parameter x86 for 32bit or amd64 for 64bit.

mbruel

@Rondog
Well it would take a few days of work to change the parsing of the xml to use SAX instead of DOM. I agree it would be a good thing to do but it is not a priority for my client.
Nowadays, most laptops or computers have minimum 8GB of ram, so using 3GB at launch shouldn't be a problem. Anyway, even without the parsing I could end up to need more than 2GB of RAM so the move to a 64bit version is definitely needed.

mbruel

@Buckwheat
I may give it a try, but I've installed on my VM QT 64bit for msvc2013 and visual studio community 2013 and I've managed to compile my app in 64bit.
I don't have anymore the bad allocation error and I'm able to use more than 3GB of RAM :)
That's good enough for me. Plus in fact I guess that using msvc compiler instead of mingw could just be better for windows target (like my client is) as the system calls will be more direct (no need to include mingwm10.dll, libstdc++-6.dll, libgcc_s_dw-1.dll, libwinpthread-1.dll)

mbruel

@Chris-Kawa
Yes now it is more clear in my mind... I found the naming really confusing when you don't know windows environment... this system32 being 64bits and Wow64 being 32 bit adaptation...
On linux 64bit, the 64bits libs are in /usr/lib and the 32bit adaptation layer in /usr/lib32... This make more sense!

Concerning Visual Studio 2013, I've found this article on stack overflow that explain well the difference between x86_64 and amd64 compiler.
Visual Studio 2013 may be in 32bit but it has a 64 bit compiler and a 32bit one that can also generate a 64bit output.