_ITERATOR_DEBUG_LEVEL=0 breaks QString::toStdU16String()

[Perdrix]

I am trying to resolve some problems with performance of a debug build of my code which is heavily compute intensive and uses standard library iterators.

Based on some profiling work, followed by some internet searching, I set the C++ macro _ITERATOR_DEBUG_LEVEL=0 and rebuild my code. Problem is I then got an exception thrown at me which I discovered was thrown by a corrupt u16string in this code:

		QStringList files = fileDialog.selectedFiles();

		//
		// Before attempting to add the files prune out those that have already been loaded
		// and issue an error message
		//
		auto it = std::remove_if(files.begin(), files.end(),
			[&](const QString& s) {
				std::u16string u16s{ s.toStdU16String() };
				fs::path path(u16s);
				return fileAlreadyLoaded(path);
			});
		files.erase(it, files.end());

running under the debugger the variable u16s is garbage (and excessively long) after the line std::u16string u16s{ s.toStdU16String() }; The QString reference s is a valid 80 character string.

If I build the code without setting that pre-compiler macro it works correctly.

Puzzled!
David

[Perdrix]

I have worked around the problem by using boost::container::vector in the debug build instead of std::vector for a specific set of vectors that were massively impacted by the iterator checking code (the critical function was invoking std::lower_bound many millions of times).

Time comparisons:

Debug build:

std::vector > 18 minutes
boost::container::vector 45 seconds

Release build:

std::vector 1.64 seconds
boost::container::vector 30 seconds

So I now use boost::container::vector for the debug build and std::vector for the release build.

This means (sigh of relief) that I don't need to build a special version of Qt.

[Christian Ehrlicher]

@Perdrix said in _ITERATOR_DEBUG_LEVEL=0 breaks QString::toStdU16String():

I set the C++ macro _ITERATOR_DEBUG_LEVEL=0 and rebuild my code.

How?

I wonder that your program runs at all mixing this macro across different libraries most likely does not work

I am trying to resolve some problems with performance of a debug build of my code

performance and debug build do not match together.

QString::toStdU16String() doesn't do something fancy and is inlined so I doubt this is the real problem:

inline std::u16string QString::toStdU16String() const
{ return std::u16string(reinterpret_cast<const char16_t*>(data()), length()); }

And last but not least - why the hell to you mix Qt and std stuff here? Either files contains std::strings and you use std or it contains QString and you use QFile/QFileInfo/whatever. When you look at performance why do you convert between QString and std::string at all? stay with one of them,

[Chris Kawa]

Using that define is a bit tricky, because it has to be defined in all the code that uses standard library, otherwise you get incompatible implementations of standard containers.

All code means Qt too i.e. if you're using that macro in your own code and use Qt's methods that interact with standard containers (like toStdU16String for example) you have to compile Qt itself with that define. This goes for any other 3rd party library that you're using too.

I'm dealing with the same problem, because using debug version of the std without that define is a pain in demanding apps like realtime rendering, and since no package manager provides libraries pre-built with that define it's always a chore having to recompile everything myself.

Btw. if you use Qt in your entire app you should rather stay away from mixing Qt and std containers (strings). Qt has filesystem and path stuff support, so there's usually no need to convert to the standard paths.

@Christian-Ehrlicher said:

performance and debug build do not match together.

True, but it sometimes is a difference between slow and unusable. For example my game engine goes from 2FPS to 20FPS in debug build just by using that define. Sure, I loose some built-in debugging features of std, but without it the debug build is entirely useless anyway, because you can't even use the app. With it it's slow, but workable.

[Perdrix]

Indeed the difference between slow and unusable applies here.

I thought the difference between having that macro set to 2 (the default for a debug build) versus 0 was that extra debug code was added, not that it created incompatible versions of the code. Clearly I was wrong!

How hard is it to rebuild Qt with that define for MS VS2019/VS2022?

Thanks
D

[Chris Kawa]

I thought the difference between having that macro set to 2 (the default for a debug build) versus 0 was that extra debug code was added

It is. Some of that code is things like adding dead zones around allocations so that iterators can be validated, thus resulting in different object layouts.

How hard is it to rebuild Qt with that define for MS VS2019/VS2022?

If you've ever built Qt from source the difference is just adding -D_ITERATOR_DEBUG_LEVEL=0 as a parameter to the configure step.

[Perdrix]

Never built Qt from source b4 is there a "how to"?

BTW in this case the difference between Release build and Debug build for one particular part of the code was 1.64 seconds versus 18 minutes+ for a Debug build!!! Even ten seconds would be just about acceptable, but in this case the std library checking stuff really really hurt a lot.

D.

[Chris Kawa]

Here's the official guide: Building Qt from source, but I always find it a bit convoluted.

Here's my "just click" build.bat file that I use. Get Qt sources (the easiest is via online installer) and just place the bat next to the source directory (Src in my case) and run. Adjust the options at the top to your needs: 32/64 bit build, paths, configure options etc. See this list for available options of configure: Qt Configure Options.

SET VS_ENVIRONMENT="C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat"
SET PLATFORM=amd64
SET SRC_DIR=Src
SET BUILD_DIR=Build
SET INSTALL_DIR=msvc2022_64
SET CONFIGURE_OPTIONS=-c++std c++17 -D_ITERATOR_DEBUG_LEVEL=0 -debug-and-release -mp -opensource -confirm-license -optimized-tools -silent -nomake examples -nomake tests -opengl desktop

SET BAT_DIR=%~dp0
call %VS_ENVIRONMENT% %PLATFORM%
mkdir %BAT_DIR%%BUILD_DIR%
mkdir %BAT_DIR%%INSTALL_DIR%
cd %BAT_DIR%%BUILD_DIR%
call %BAT_DIR%%SRC_DIR%\configure -prefix %BAT_DIR%%INSTALL_DIR% %CONFIGURE_OPTIONS%
cmake --build . --parallel
cmake --install .
cmake --install . --config Debug
cd %BAT_DIR%%INSTALL_DIR%
rmdir %BAT_DIR%%BUILD_DIR% /Q /S
pause

It's also a good opportunity to exclude what you don't use from build e.g. I don't use QML or any web stuff, so excluding those (and a couple more) cuts the build time from an hour to just couple minutes. The result is also a lot smaller than the prebuilt package. You do that by adding -skip qt3d, -skip qtconnectivity etc. to the configure options. The module names are the directories inside the source dir.

[Perdrix]

Thank you sir

David

[Perdrix]

I have worked around the problem by using boost::container::vector in the debug build instead of std::vector for a specific set of vectors that were massively impacted by the iterator checking code (the critical function was invoking std::lower_bound many millions of times).

Time comparisons:

Debug build:

std::vector > 18 minutes
boost::container::vector 45 seconds

Release build:

std::vector 1.64 seconds
boost::container::vector 30 seconds

So I now use boost::container::vector for the debug build and std::vector for the release build.

This means (sigh of relief) that I don't need to build a special version of Qt.