decode POST data with diacritic characters

Marek

Hi all,

I'm receiving POST data from server, server headers says it is:
("Content-Type", "application/x-www-form-urlencoded")
I want to decode, I'm using QByteArray::fromPercentEncoding(data), I have also tried Qurl::fromPercentEncoding
The problem is that in received string there is diacritic character which is decoded into sign "?"

When I display QByteArray after fromPercentEncoding I can see this:
diacritic\xC5\x82""character
it should be: diactiticłcharacter

So there is two byte sign '\xC5\x82'', checking utf-8 character table this is letter "ł" - this is what it should be.

I can't make it work, QByteArray is always decoded into QString diacritic?character

I assume that server is sending in UTF-8.

Please help ;)
Marek

JonB

@Marek
I don't know what your solution is, and I'm throwing myself in here, but any kind of fromPercentEncoding() sounds wrong. Percent encoding is used on a url. You have POST data, that is not a Url, and should never be treated as such for decoding etc.* I would have thought you'd be using some kind of fromUtf8() function.

*I may be wrong about this....! :(

Upon reflection, since I am indeed probably wrong and http://doc.qt.io/qt-5/qurl.html#fromPercentEncoding says:

Returns a decoded copy of input. input is first decoded from percent encoding, then converted from UTF-8 to unicode.

is it possible that when you talk about "display QByteArray" (e.g. qDebug()??) it's just that the display does not show the character correctly but it is actually correct when you treat it as a string?

EDIT Since I'm aware I have been unhelpful so far. This is not my area, but assuming this is down to some kind of language issue are you supposed to be using http://doc.qt.io/qt-5/qtextcodec.html during your decoding work to cater for that? I will shut up from now on....

Marek

@JonB Thanks for looking into it.
I think it is Percent encoding, when I display with qDebug without QByteArray::fromPercentEncoding I got this:
datetime=2018-10-04 170X0,0000000000221P-1022340X0,07FFF44DC44DP-102214
after I got this:
datetime=2018-10-04 17:34:14
So yes qDebug displays it somehow wrong, but data is urlencoded.

EDIT Oh man I spent whole afternoon chasing the problem that does not exists!
Diacritic character was simply not displayed by qDebug. Different problem was that the data from server changes space (" ") to (+) and (+) to %2B and fromPercentEncoding does not change (+) sign.
So first I need to QByteArray::replace(QByteArray("+"),QByteArray(" ")) and then fromPercentEncoding

Best,
Marek

JonB

@Marek

Diacritic character was simply not displayed by qDebug.

Of all my random, unhelpful, incorrect attempts at replying, this was indeed the one I had in mind when I asked whether you were using qDebug() and it might just be a display artefact, and the string would actually be correct "underneath"!

So first I need to QByteArray::replace(QByteArray("+"),QByteArray(" ")) and then fromPercentEncoding

Yes, although this wasn't your real problem, this is covered in various places. (It may also depend on whether you are Qt4 or 5, not sure?) http://doc.qt.io/qt-5/qurl.html says stuff like:

One special sequence to be aware of is that of the plus character ('+'). QUrl does not convert spaces to plus characters, even though HTML forms posted by web browsers do

You can also look at http://doc.qt.io/qt-5/qurl.html#ParsingMode-enum and https://stackoverflow.com/questions/36551109/qt-does-not-encode-sign. Basically it's all a bit abstruse, and you do have to do a little work on space & + yourself, as you show.

Marek

Using Qt5.

Yes, although this wasn't your real problem, this is covered in various places.

Yes, I have seen post on + sign earlier, I was calculating sha256 from server data for verification, when hash didn't agree I was chasing ? sign from qDebug ;)

Thanks a lot.
Marek