[Split] QString/QByteArray conversion problems



  • [EDIT: Split off as a new topic from "this thread":http://developer.qt.nokia.com/forums/viewthread/4433/, Volker]

    Hi
    please take a look I explain here perhaps it will help to both of us:

    I have some application wroten fully in Qt.
    In the application I take input from user from line edits and receive QStrings ( ->text() function of the QLineEdit class ). Then I need to save the input ( QStrings ) encrypted to .txt file on user machine’s hdd.
    For that purpose I decided to use simple XOR encryption. So I need to convert QString to QByteArray , after that make XOR between 2 byte arrays – input and encryption key and the result I need to save to .txt file so I convert the result vice versa to QString and put it to file.
    Also I need to read from file the encrypted data and to check it in some place in the application. For that purpose I read encrypted from previous user input from .txt file to QString and convert it to QByteArray make decryption on QByteArray, convert the result to QString – and very sorrow I do not receive the same input as before encryption – right side of user input lost!!
    Perhaps it is because of incorrect conversion QByteArray – QString and vice versa.
    What is strangeous – that only for large strings data lost and for number.
    I use UTF8 codec:

    @codec = QTextCodec::codecForName(“utf8”);@

    I get user input and write to file:

    @emailEntered = this->login->lineEdit_email->text();

    keyEntered = this->login->lineEdit_code->text();@

    I make conversion to QByteArray and encryption:

    @QString emailToSave = QString::fromUtf8(calculateXor(this->emailEntered.toUtf8(),encryptionKey));

    QString keyToSave = QString::fromUtf8(calculateXor(this->keyEntered.toUtf8(),encryptionKey));@

    I save to file:

    @QTextStream stream(&file);
    stream.setCodec(this->codec);
    stream<<‘‘<<emailToSave<<’\n’<<’’<<keyToSave<<’\n’<<’*’<<macAdressToSave<<’\n’;
    file.close();@

    I read from file:

    @QTextStream stream( &file );
    stream.setCodec(this->codec);
    emailFromFile.clear();
    keyFromFile.clear();
    macFromFile.clear();
    stream>>ch; // ‘
    stream>>ch; // email from file:
    while(ch!=’\n’)
    {
    emailFromFile.append(ch);
    stream>>ch;
    }
    stream>>ch;
    // ‘

    stream>>ch; // first digit of the coded license key
    while(ch!=’\n’)
    {
    keyFromFile.append(ch);
    stream>>ch;
    }
    stream>>ch; // ‘*’ stream>>ch;
    // first digit of mac from file
    while(!stream.atEnd()) // another while(stream!=’\n’)
    {
    macFromFile.append(ch);
    stream>>ch;
    }
    file.close();@

    I decrypt readed data from file:

    @QString result = QString::fromUtf8(calculateXor(this->emailFromFile.toAscii(),encryptionKey));

    QString result = QString::fromUtf8(calculateXor(this->keyFromFile.toAscii(),encryptionKey));@

    For email I get the same as before encryption, but for key ( another user input ) there is lost numbers at the right
    What is the problem? I try to fix it this time…



  • XORing the utf-8 bytes of a string with something else does NOT guarantee to reproduce a new valid string. You must save the result as a bunch of bytes (aka QByteArray) and put it into a file as data (use QDataStream). You must not reinterpret it as an utf-8 string and save in a QTextStream.



  • Hi Volker
    I read your reply.
    Yes, but I lose a part of data while converting QString to QByteArray and vice versa while there is no data lose should be while making xor encryption/decryption
    What should I tell to Qt by static methods, something, so as to convert any characters in both ways ( from byte array, to byte array ) without to lose any character?



  • You can take a look at the same post at QtCentre also:

    http://www.qtcentre.org/threads/39638-Convert-QString-into-QByteArray-and-vice-versa.



  • Pavel, please answer in the separate thread where your question was split off. This was not without intention - your question is unrelated to the OPs question in the other thread.

    There is no error with converting QStrings to QByteArrays and vice versa. You can do this happily as often as you want.

    BUT if you XOR the byte representation of your string, you manipulate the data in such a way, that it is not guaranteed to contain a valid utf-8 byte representation of any string. So calling fromUtf8() on this data is plain wrong. Everything afterwards leads to disaster.



  • The problem with data loss during encryption was solved my mine by
    a next way:

    I work with UTF8 codec always when I treat strings at any platform:

    I added next static "methods" to define codecs:

    @QTextCodec::setCodecForLocale(QTextCodec::codecForName("UTF-8"));
    QTextCodec::setCodecForCStrings(QTextCodec::codecForName("UTF-8"));@

    when I convert from QByteArray to QString I use next static "method" with next arguments:

    @macEnEn = calculateXor(fromFile.append(this->macFromFile),encryptionKey);
    QString result = QString::fromUtf8(macEnEn.data(),macEnEn.size());@

    The proble was - you should provide second argument an exact size of array to static "method" QString::fromUtf8(...)

    it works!
    You can take any character set, convert it, encode using bitwise manipulations, store or transmit
    and decode to any character set using your symmetrical key!



  • Ok Thank you very much, sorry if I've wrote a little not ordered.
    I will write ordered.



  • any disaster Volker! - I have solved the problem you can try it by yourself - the things work!
    Here you have simple encryption method for any character set.



  • Oh. My. God.

    You will not understand, will you?

    If you manipulate bytes with XOR you are very likely to create a byte sequence, that is NOT a valid utf-8 encoded string. How often shall I repeat this again until it gets into you brain?!?

    You need a sample? Here we go:

    Let's have input char 'A' = 0x41 = 0100 0001
    Now you happen to XOR that 'A' with 0x41: the result 0x00 and you blew up your encoding. And all this does not count in 2, 3 and 4 byte encoded characters.

    And don't tell me this won't happen. You cannot guarantee. Period.

    Again: do not use fromUtf8() with your manipulated bytes!



  • Very interesting.
    Yes it can happen ( I do not understand deeply now but I think...), but it can happen when you convert to bytes, make xor, convert to string, store and convert vice versa. It happens when you put xored bytes coded of utf to QString for some of cases not always, am I right?

    Edit:
    What 8 bit character codec to use that has characters “coded” for any 0 to 2^8 – 1 numbers?

    Edit: merged posts. Please add to your previous comment if you hit the Post button before thinking if this is really all you wanted to say; Andre



  • For encryption, it really doesn't matter. As long as you are consistent. I would probably use UTF-8. After XOR-ing, you end up with a byte array containing binairy data, not a string (as Volker has tried to point out to you repeatingly). You could use QByteArray::toBase64 to create a QString you can store in a text file. Then, you can do the reverse on your way back: read in the base64 encoded byte array, XOR with your secret key, and then create a QString with fromUtf8 based on that decoded byte array.



  • Yes Yes Yes
    Thank you
    If I use QByteArray::toBase64 what kind of QString then?

    Edit:
    I am looking for simplest way to encrypt any characters set
    Perhaps to use QCryptographicHash?

    Edit: merged two consecutive messages again. Please do that yourself next time; Andre



  • Since base64 by definition only uses ASCII characters, any encoding that has those at the right places will do. UTF-8 is one of them, but of course QString::fromAscii() would do better, I think.

    Edit: QCryptographicHash does, as the name implies, hashing. That is not the same as encryption. Hashing is (in theory) a one-way, non reversible operation, encryption a two-way operation (that is, you can reverse it). You might look into QCA.



  • After I checked and thought here it is:

    There is a need to convert each character to 8 bit such way that any 8 bit number has it's representation in characters - one to one from 0000 0000 to 1111 1111 and then it is Ok to make xor and vice versa?

    So what coding system ( standard ) provides 8 bit number one to one to characted mapping?
    ASCII?



  • if I use QByteArray QCryptographicHash::hash ( const QByteArray & data, Algorithm method )
    and then should I put the QByteArray to text file ( if I need for example to store encrypted data on hdd ) as QDataStream or can I use QByteArray::toBase64 ? What codec Qt supports crossplatform that provides one - to - one coding of characters to bytes ( 8 bit numbers ) ?
    Pavel
    14.03.2011



  • This also can help to any who is interesting in fast and reliable encryption:

    http://wiki.forum.nokia.com/index.php/How_to_Encrypt_a_text_using_QCryptographicHash_in_Qt_for_Symbian



  • because ASCII has no period and UTF8 does have?



  • I am sorry, but: you really don't get it, do you?

    You still keep on posting multiple messages in a row, instead of adding to your previous messages with the edit link on the right side of your last message

    I already explained to you that hashing is not the same as encryption, no matter what that misguided wiki entry tells you

    I already explained that the character encoding you use for your string is inconsequential

    QByteArray already can be thought off as an array of 8-bit (yes, that is a byte in the x86 world) characters. Do your xor magic on that.



  • Sorry, ASCII is one to one character to 8 bit conversion in all 0 - 255 range?
    I am not sure
    I need Qt supported codec that converts characters one to one to 8 bit number in 0 -255 range, another worlds each 8 bit number in byte array has it's unique character.
    What is the codec?



  • Andre I understand - Ok I won't post multiple but by using edit.
    Ok Thanks a Lot. Using hash can I make "unhash" with the same key? How can I get unhashing one - to -one?



  • [quote author="Pavel Mazniker" date="1300124063"]Andre I understand - Ok I won't post multiple but by using edit.[/quote]

    You just did.



  • [quote author="Pavel Mazniker" date="1300122778"]What codec Qt supports crossplatform that provides one - to - one coding of characters to bytes ( 8 bit numbers ) ?[/quote]

    The whole thing about codecs is to map all the big bunch of non-ASCII characters into some 8-bit representation. This can reduce the mappable characters (like in ISO-8859-1) or it can lead to a representation consisting of multiple 8-bit bytes (like in UTF-8).

    All conversion methods that return QByteArray give you an array of 8bit bits.



  • [quote author="Volker" date="1300125063"]
    [quote author="Pavel Mazniker" date="1300122778"]What codec Qt supports crossplatform that provides one - to - one coding of characters to bytes ( 8 bit numbers ) ?[/quote]

    The whole thing about codecs is to map all the big bunch of non-ASCII characters into some 8-bit representation. This can reduce the mappable characters (like in ISO-8859-1) or it can lead to a representation consisting of multiple 8-bit bytes (like in UTF-8).

    All conversion methods that return QByteArray give you an array of 8bit bits.[/quote]

    Hi
    Thank you very much for your help.
    Your explanations here at the forum really helped me.
    It seems this time I fixed the problem finally - encryption/decryption works for letters,numbers and @ sign,

    • I've checked for some inputs and it is ok and also theoretically shoud not be any problem( not for any possible character ).So as to be sure 100% one should understand
      how Ascii,Base64 map char to byte.
      I encrypt by manipulating bytes of byte array: QString someString.toAscii(),
      after that I convert it .toBase64() and save it using QDataStream to .dat file.
      When I decrypt I open .dat file with QDataStream,convert readed data from Base64 using QByteArray::fromBase64()
      and make byte manipulations needed for decryption on the byte array decoded from file.
      Pavel


  • Sounds reasonable.

    One hint: If you use QDataStream you can save yourself the base64 encoding/decoding step, but write and read a QByteArray directly.

    Also I would convert the original string to utf8(), as toAscii() and toLatin1() are conversions that can loose characters (ie. it is a conversion that is not reversible).

    Just try:

    @
    // a cyrillic char and the euro sign
    QString t = QString::fromUtf8("д€");
    qDebug() << "ascii" << t.toAscii().toHex();
    qDebug() << "latin1" << t.toLatin1().toHex();
    qDebug() << "utf8" << t.toUtf8().toHex();
    @

    the output is

    @
    ascii "3f3f"
    latin1 "3f3f"
    utf8 "d0b4e282ac"
    @

    Hex code 3f is question mark; so both chars translate to '?' when using toAscii(), and you will get back to question marks, when re-transforming the bytes to a string.

    And please, please stop thinking in characters when you called any of the toXXX() methods of QString. What you then have is just a bunch of bytes. Treat them as bytes, not as chars!



  • I understand UTF8 provides support for wider characters set that Ascii
    So I save QString converted to UTF8 code as byte array ( manipulated ) but when I decrypt I can loose characters once UTF8 is not (one) -to -(one) (character) -to - (8bit number) mapping?



  • [quote author="Pavel Mazniker" date="1300196573"]I understand UTF8 provides support for wider characters set that Ascii
    So I save QString converted to UTF8 code as byte array ( manipulated ) but when I decrypt I can loose characters once UTF8 is not (one) -to -(one) (character) -to - (8bit number) mapping?[/quote]

    No. You can just re-create a QString from your QByteArray containing UTF-8. The operation is completely lossless.



  • The following snippet is guaranteed to work with any value for x (ie. x and y are always equal in the end):

    @
    String x = "some fancy string";
    QByteArray ba = x.toUtf8();
    String y = QString::fromUtf8(ba);
    if(x == y)
    qDebug() << "strings are equal";
    @

    If you replace toUtf8() with toAscii() or toLatin1() and replace fromUtf8() with fromAscii() or fromLatin1(), it is not guaranteed that x and y are equal in the end.

    For some further explanations, please ask Google or your favorite search engine on the topic (text encoding, charsets, etc). That's beyond the topic for a Qt forum.



  • @Pavel:

    Perhaps this "Wiki article":http://developer.qt.nokia.com/wiki/Simple_encryption I just published is of use to you? It presents a class I wrote for simple encryption and decryption of QStrings and QByteArrays with a symmetric 64 bit key.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.