UTF-8 characters acting strangely in Windows (but not Linux)

eggbertx

I'm trying to figure out why my file handling code works just fine in Linux but acts strangely in Windows. Both compile without errors or warnings but where the non-ASCII characters are displayed with qDebug() or written to a file in Windows I just get "????"

QTextStream out(&fileHandle);
QString string = QString::fromUtf8("おはよう。");
out << string;

Is there a better way to do this?

Chris Kawa

If you're using Visual Studio it doesn't, by default, treat your source code as encoded in utf-8. Unless there's a BOM it is interpreted using your current code page, which means that, by default, with Visual Studio the only "safe" character set is ASCII and extended utf-8 characters will come out garbage . If you want Visual Studio to treat your files as utf-8 you need to make it.

eggbertx

@Chris-Kawa I'm using Qt Creator in both Windows and Linux.

Chris Kawa

Qt Creator is an IDE. It doesn't matter. You could be using a Notepad and it's all the same. I'm talking about the compiler. Which Qt package have you installed?

eggbertx

@Chris-Kawa Well yes, I know that. I'm using what was provided by the Qt Creator installer.

Chris Kawa

There are many installers. There's an online installer which lets you install Qt packages for many different compilers and there are offline packages for a single specific compiler.
Please go to Tools -> Options ->Build & Run -> Kits tab, select te kit you're using and tell us what it says in the "Compiler" combo box.

eggbertx

@Chris-Kawa Microsoft Visual C++ Compiler 14.0 (amd64)

Chris Kawa

Right, so, as I said, you need to make it understand utf-8 with a switch. Add QMAKE_CXXFLAGS += /utf-8 to your .pro file and re-run qmake (Build ->Run qmake).
After that the source should be treated as utf-8, but that's just the first part.
The second part is setting the codec on the stream. The default coded is locale and platform specific, so you just got lucky without that on Linux (because most distros are utf-8 centric these days). In any case, when you're dealing with non-ASCII characters you should always try to specify the codec explicitly. It will spare you the trouble. To set a codec call out.setCodec("UTF-8"); before you write anything to the stream.

eggbertx

@Chris-Kawa I had already set QMAKE_CXXFLAGS += /utf-8 in my pro file and I ran qmake already. But using out.setCodec("UTF-8"); fixed the issue.

Solved UTF-8 characters acting strangely in Windows (but not Linux)