Converting accented characters to std::string returns mangled text

jsulm

@jacobP How do you show this string? In console? Could be just an issue with the font/encoding in your console.

shabdaclinic

This post is deleted!

jacobP

I am viewing the strings using the Visual Studio watcher while debugging. Below is the code I have currently:

std::string u8 = entry.toUtf8().constData();
auto u16 = entry.toStdU16String();
auto u32 = entry.toStdU32String();
auto wstd = entry.toStdWString();
auto stds = entry.toStdString();

The variables u8 and stds have the value WeÅ .txt (notice the space between the A and .txt) while the rest have WeŠ.txt.

I am trying to use C library that only takes const char* as inputs and it is currently crashing due to the strings being mangled.

mrjj

@jacobP
Hi I wonder what encoding the input is in ?
https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/

jacobP

@mrjj

They are Windows file names. From what I just looked up Windows file names are encoded in UTF-16.

mrjj

@jacobP
And you are sure they are not mangled from the source ?
As when you read them?

jacobP

@mrjj

Using the Visual Studio watcher,

entry

by itself will return WeŠ.txt but

entry.toUtf8()

will return WeÅ .txt.

I can see from my file explorer that the file name is WeŠ.txt.

jacobP

I also wanted to point this out: The character Š does not fit on a single byte. It's UTF-8 encoding is 197 160, Unicode is 352. Trying to fit this character in a char will result in 2 chars, Å(197) and <No break space>1(60) respectively.

mrjj

@jacobP
Yes, so i do wonder how to get correct ascii file name out of that.
Give it some time, some of the others might have inputs.

hskoglund

@jacobP That C library that only takes const char* as inputs, how old is it? Maybe it's for FAT file systems and not NTFS? (Qt and NTFS are about the same age (~25 years) that's why QString also uses UTF-16).

You could try the technology used before Unicode was invented: code pages. Pros: everything fits in single bytes. Cons: depending on what codepage you set your system for, different characters will be displayed for the same byte :-(

QString has a function for converting down from UTF-16 to your current Windows codepage: toLocal8Bit, example:
(also you need to #include "windows.h" to enable the ::GetACP() function)

QString s("WeŠ.txt");
qDebug() << s.toUcs4();

qDebug() << ::GetACP();
qDebug() << s.toLocal8Bit();

Output:

QVector(87, 101, 352, 46, 116, 120, 116)
1252
"We\x8A.txt"

First, I use toUcs4() to display the UTF-16 contents of the QString, and the Š is as you say 352. (Ucs4 is a bigger and newer brother to UTF-16).

Then I query Windows for which code page will be used for the toLocal8Bit() function, on my machine is 1252, this will vary from country to country.

The final line reveals that on code page 1252 the Š character has the code 0x8A (138 decimal), which fits into a byte. Try giving that QByteArray to your C library...

SGaist

Hi,

What C library is that ?