You are aware of QCoreApplication::arguments(), I hope?

This is something we do in our command line applications that (on purpose) do not use any Qt. (Because of static linking which makes our lives way easier with our clients.)

QCoreApplication::arguments() works fine ;-)

Since ANSI C/C 99 it was stipulated that argv[argc] should be NULL. Code using argv could either look at argc or could iterate till argv[i] == NULL to visit all arguments. I have searched and cannot find anything which says this is no longer the case(?). So if you are passing your newly created argv replacement to other code you might verify it enters with argv[argc] == nullptr and then like to malloc((argc + 1) * sizeof(char*)) and argv[argc] = nullptr. :)

Little caveat: The setlocale trick does not fully solve command line arguments.

You are aware of QCoreApplication::arguments(), I hope?

The OP needs a std::string for a 3rd-party library. If the file access is done by the library, and not in his or her own code, it will probably not work on Windows if there are Unicode characters in the file name.

So, then it depends on the library. If it uses fstream internally, this approach will work, as setlocale() will change it for everything. If the library uses Windows' file API directly I would hope that there is a wide string interface as well because otherwise you are out of luck (I guess...).

@Christian-Ehrlicher Using the setlocale trick also switches the output via std::cout to UTF-8. You only have to make sure that your editor saves your source code files as UTF-8. And you have to tell the Microsoft compiler that the source is UTF-8 and that it should produce UTF-8 binaries (IIRC two separate switches).

Little caveat: The setlocale trick does not fully solve command line arguments. Suddenly, we had problem with certain file names/paths. Here is the work around that we are using for this to truly be fully inside UTF-8:

#ifdef _WIN32
int main(int argc_discard, char** argv_discard)
#else
int main(int argc, char** argv)
#endif
{
#ifdef _WIN32
    setlocale(LC_CTYPE, ".utf8");

    int argc;
    wchar_t** wargv = CommandLineToArgvW(GetCommandLineW(), &argc);
    char** argv;
    argv = malloc(argc * sizeof(char*));
    for (int i = 0; i < argc; ++i)
    {
        const int size = WideCharToMultiByte(CP_UTF8, 0, wargv[i], -1, NULL, 0, NULL, NULL);    // get length of string
        argv[i] = malloc(size + 1);
        WideCharToMultiByte(CP_UTF8, 0, wargv[i], -1, argv[i], size + 1, NULL, NULL);           // convert to UTF-8
    }
    LocalFree(wargv);
#endif
   ...
}

@Robert-Hairgrove said in QString encoding to std::string:

But on Windows, utf-16 is the native encoding.

This is wrong.

Well, not completely wrong, anyway. This is a quote directly from the "horse's mouth", as it were:

"UTF-16 is basically the de facto standard encoding used by Windows Unicode-enabled APIs. UTF-16 is the “native” Unicode encoding in many other software systems, as well. For example, Qt, Java and the International Components for Unicode (ICU) library, just to name a few, use UTF-16 encoding to store Unicode strings."

Source: https://learn.microsoft.com/en-us/archive/msdn-magazine/2016/september/c-unicode-encoding-conversions-with-stl-strings-and-win32-apis

By the way, thanks for your extremely helpful reply!

But on Windows, utf-16 is the native encoding.

This is wrong.

The #ifdef switching should be enough to ensure that it will continue to work normally under Linux, if it was working before:

#ifdef QT_OS_WINDOWS
setLocale(LC_ALL, ".UTF8");
#endif

We use the approach with setlocale(LC_ALL, ".UTF8") at the beginning of main(). You need to wrap it inside an #ifdef for Windows

I suggested this above to the OP, but they rejected it on the basis of

but unfortunately the suggestion from reddit won't be very useful because the system also has to run under Linux

! :)

We use the approach with setlocale(LC_ALL, ".UTF8") at the beginning of main(). You need to wrap it inside an #ifdef for Windows because it can crash on other systems (I think it was on macOS, but not on Linux). Now, you are in a world where on all systems you can open files using UTF-8 filenames.

For a while we had some strange problems with converting QString to std::string. Here is the way we do it now:

QString qstr("...something...");
std::string str = qstr.toUtf8().data();
// and the other way around
qstr = QString::fromUtf8(str.c_str());

This has worked reliably for us.

Though, there is technically a slight caveat concerning temporaries. To be on the safe side, you should save the result from toUtf8() to a temporary QByteArray and use data() on this variable. In very few cases we got our app to crash because of this. I think the C++ standard states something like this about temporaries.

This page might help:
https://newbedev.com/how-to-open-an-std-fstream-ofstream-or-ifstream-with-a-unicode-filename

Also, there is QFile::encodeName() and QFile::decodeName() which might help.

the suggestion from reddit won't be very useful because the system also has to run under Linux

I thought that Linux does use UTF-8 so there won't be the issue there, you are just looking for a solution for Windows. Anyway I don't know so over to you.

My (admittedly limited) understanding is that Windows does not use UTF8, at least for file paths. Are you saying that with toLatin1() does work while others like toUtf8() do not?

I find comments like https://www.reddit.com/r/cpp_questions/comments/ov1xqw/paths_with_nonascii_chars_on_windows/

std::string is an ANSI string. If you need to support non-ANSI characters, you need to use wstring or u8string (utf-8).

There is also a discussion there about

On newer versions of Windows 10 (v1903 and newer) it is possible to define a new setting called activeCodePage to UTF-8 in the manifest file of your application:

https://docs.microsoft.com/en-us/windows/win32/sbscs/application-manifests#activecodepage

If you are using mingw instead of MSVC, use GCC from ucrt repository. Call setlocale(LC_ALL, ".UTF8") in main() before anything else. After that most C library functions should assume that strings passed as char* are UTF-8. Use std::filesystem::path::u8string() to get UTF-8 encoded file name.

However take this with a pinch of salt, someone else may know better than I what your issue is/what to do about it....