Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Struggling to stream byte array .toUtf8() out a TextStream
Forum Updated to NodeBB v4.3 + New Features

Struggling to stream byte array .toUtf8() out a TextStream

Scheduled Pinned Locked Moved Solved General and Desktop
13 Posts 3 Posters 1.3k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • R Offline
    R Offline
    rhb327
    wrote on last edited by
    #1

    Suppose you have a string like this:

    Filename: 222runㅁㄴㅇ (32323272756ee38581e384b4e38587 <<< UTF8 hex)

    qDebug() << runfileName << runfileName.length() << runfileName.toUtf8() << runfileName.toUtf8().length();
    

    will yield: "222run???" 9 "222run\xE3\x85\x81\xE3\x84\xB4\xE3\x85\x87" 15

    I want to append the 15 bytes into a QTextStream and the below fails miserably:

    TextStreaming << tr("Run Filename") << "," << runfileName.toUtf8() << "\n"
    

    This appears to be due to the fact the << operator will take the QByteArray produced by .toUtf8() and will end up calling QString::fromUtf8(runfileName) and I push the 9 incorrect bytes vs. the 15 correct bytes I want!

    What's the proper way to "push" a UTF8 series of bytes into a QTextStream?

    Thanks,
    -Rich

    1 Reply Last reply
    0
    • B Offline
      B Offline
      Bonnie
      wrote on last edited by Bonnie
      #2

      @rhb327 said in Struggling to stream byte array .toUtf8() out a TextStream:

      I want to append the 15 bytes into a QTextStream

      Why? QTextStream takes QString, a.k.a, unicode wide char string, not bytes. There's no need to, also you shouldn't, encode it to a 8-bit string.
      To set the encoding codec, use QTextStream::setCodec instead of encoding the string by yourself.
      If it is for binary data storage, you may consider to use QDataStream.

      1 Reply Last reply
      2
      • R Offline
        R Offline
        rhb327
        wrote on last edited by
        #3

        Thanks for the input. I just tested << runfileName and << runfileName.toUtf8() and both yield expected behavior on Linux. Prior post was on Win10. Also, the TextStream already had setCodec("UTF-8"). Thus, I think this might be a Desktop Win10 issue not an issue on Linux Desktop (tested for this response) and on my embedded Linux platform.

        Not sure why Win10 would be different but per my hex editor it definitely appears to be.

        Thanks,
        -Rich

        JKSHJ 1 Reply Last reply
        0
        • B Offline
          B Offline
          Bonnie
          wrote on last edited by
          #4

          QTextStream should work fine in win10.
          I don't know what your issue is, but perhaps it is not due to QTextStream.

          1 Reply Last reply
          0
          • R Offline
            R Offline
            rhb327
            wrote on last edited by
            #5

            Yeah, not sure but moving on for now as it's ok on the target OS. Trusting my hex editor on matters like this. Thanks for your inputs.

            1 Reply Last reply
            0
            • R rhb327

              Thanks for the input. I just tested << runfileName and << runfileName.toUtf8() and both yield expected behavior on Linux. Prior post was on Win10. Also, the TextStream already had setCodec("UTF-8"). Thus, I think this might be a Desktop Win10 issue not an issue on Linux Desktop (tested for this response) and on my embedded Linux platform.

              Not sure why Win10 would be different but per my hex editor it definitely appears to be.

              Thanks,
              -Rich

              JKSHJ Offline
              JKSHJ Offline
              JKSH
              Moderators
              wrote on last edited by JKSH
              #6

              @rhb327 said in Struggling to stream byte array .toUtf8() out a TextStream:

              Not sure why Win10 would be different but per my hex editor it definitely appears to be.

              On Linux, the default system codec is UTF-8. This is not the case on Windows.

              I just tested << runfileName and << runfileName.toUtf8() and both yield expected behavior on Linux.

              This is just a coincidence.

              [EDIT: The following explanation is wrong, sorry. --JKSH]

              Like @Bonnie said, QTextStream takes a QString; it does not take a QByteArray. When you pass a QByteArray into it, your program first decodes the QByteArray into a QString using your system's default codec.

              So how does this affect your tests? Well, on Linux, your UTF8-encoded byte array was converted to a QString using a UTF-8 codec first, and then it is passed into QTextStream. Since the encoding and the codec match, everything works fine.

              On Window,s your UTF8-encoded byte array was converted to a QString using a non-UTF8 codec, so you get corrupted data.

              What's the proper way to "push" a UTF8 series of bytes into a QTextStream?

              I agree with @Bonnie: You should not push bytes into QTextStream. You should push text into QTextStream.

              Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

              1 Reply Last reply
              1
              • R Offline
                R Offline
                rhb327
                wrote on last edited by
                #7

                Thanks for explaining why I see the difference on Linux vs. Win10. And I understand the << operator of QTextStream will process my runfileName.toUtf8() QByteArray through QString::fromUtf8() per the documentation and I'm guessing that's where the system default encoding you mention occurs and that this is a poor approach to accomplish what I desire which is a UTF8 string in the TextStream.

                So what's still missing for me is:

                1. I have a QString with the correct data
                2. I configure a QTextStream with setCodec("UTF-8")
                3. I run TextStream << runfileName on Win10 and I don't see the 15-bytes I expect

                So here I'm passing a QString and it's not correct, why? Even the original qDebug() in my first post shows that UTF8 encoding is not working with qDebug() << runfiIeName which I think you would say is because of Win10s default. But for the TextStream I've set the codec to UTF8 so why only 9-bytes in the text file with 3Fh bytes vs. the 15-byte UTF8 message I expect when I've set the codec?

                Thanks,
                -Rich

                B JKSHJ 2 Replies Last reply
                0
                • R rhb327

                  Thanks for explaining why I see the difference on Linux vs. Win10. And I understand the << operator of QTextStream will process my runfileName.toUtf8() QByteArray through QString::fromUtf8() per the documentation and I'm guessing that's where the system default encoding you mention occurs and that this is a poor approach to accomplish what I desire which is a UTF8 string in the TextStream.

                  So what's still missing for me is:

                  1. I have a QString with the correct data
                  2. I configure a QTextStream with setCodec("UTF-8")
                  3. I run TextStream << runfileName on Win10 and I don't see the 15-bytes I expect

                  So here I'm passing a QString and it's not correct, why? Even the original qDebug() in my first post shows that UTF8 encoding is not working with qDebug() << runfiIeName which I think you would say is because of Win10s default. But for the TextStream I've set the codec to UTF8 so why only 9-bytes in the text file with 3Fh bytes vs. the 15-byte UTF8 message I expect when I've set the codec?

                  Thanks,
                  -Rich

                  B Offline
                  B Offline
                  Bonnie
                  wrote on last edited by
                  #8

                  @rhb327
                  Can you post your actual code for doing this?
                  I just tried using QTextStream to write the QString you provide to a file, and the file size is exactly 15 bytes.
                  I'm also using Win10.

                  1 Reply Last reply
                  0
                  • R rhb327

                    Thanks for explaining why I see the difference on Linux vs. Win10. And I understand the << operator of QTextStream will process my runfileName.toUtf8() QByteArray through QString::fromUtf8() per the documentation and I'm guessing that's where the system default encoding you mention occurs and that this is a poor approach to accomplish what I desire which is a UTF8 string in the TextStream.

                    So what's still missing for me is:

                    1. I have a QString with the correct data
                    2. I configure a QTextStream with setCodec("UTF-8")
                    3. I run TextStream << runfileName on Win10 and I don't see the 15-bytes I expect

                    So here I'm passing a QString and it's not correct, why? Even the original qDebug() in my first post shows that UTF8 encoding is not working with qDebug() << runfiIeName which I think you would say is because of Win10s default. But for the TextStream I've set the codec to UTF8 so why only 9-bytes in the text file with 3Fh bytes vs. the 15-byte UTF8 message I expect when I've set the codec?

                    Thanks,
                    -Rich

                    JKSHJ Offline
                    JKSHJ Offline
                    JKSH
                    Moderators
                    wrote on last edited by JKSH
                    #9

                    @rhb327 said in Struggling to stream byte array .toUtf8() out a TextStream:

                    I configure a QTextStream with setCodec("UTF-8")

                    Can you double-check that it has been configued correctly? What does stream.codec()->name() return?

                    This code gives me 15 bytes:

                    auto ba = QByteArray::fromHex("32323272756ee38581e384b4e38587");
                    auto str = QString::fromUtf8(ba);
                    
                    QFile fout("output.txt");
                    fout.open(QFile::Text|QFile::WriteOnly);
                    
                    QTextStream stream(&fout);
                    
                    qDebug() << stream.codec()->name(); // "System"
                    stream.setCodec("UTF-8");
                    qDebug() << stream.codec()->name(); // "UTF-8"
                    
                    stream << str;
                    

                    P.S. I realized that my previous explanation was wrong, sorry. Please ignore it.

                    Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                    1 Reply Last reply
                    1
                    • R Offline
                      R Offline
                      rhb327
                      wrote on last edited by
                      #10

                      Thanks for helping me. This is a hook I put at the top of the function back on Win10 right now and running 5.12.0:

                      void Utilities::exportRunFile(quint8 logFileType, QString runfileName) {
                          qDebug() << "DEBUG: " << runfileName << runfileName.length() << runfileName.toUtf8() << runfileName.toUtf8().length();
                          QSaveFile tempFile;
                          QTextStream txtStream;
                          txtStream.setCodec("UTF-8");
                          tempFile.setFileName("C:/Users/richard.bair/Downloads/test.csv");
                          tempFile.open(QIODevice::WriteOnly);
                          txtStream.setDevice(&tempFile);
                          txtStream << tr("Run Filename") << "," << runfileName << "\n";
                          txtStream.flush();
                          tempFile.commit();
                      

                      qDebug output is below...filename happens to have some Korean characters in it...just part of a quick test.

                      DEBUG:  "222run???" 9 "222run\xE3\x85\x81\xE3\x84\xB4\xE3\x85\x87" 15
                      

                      Here's the file in a hex editor and Notepad++:

                      25fc44dc-132f-49d0-bb36-a8d3c247a881-image.png
                      fdbea824-e651-4270-b574-4a2e3296a134-image.png

                      1 Reply Last reply
                      0
                      • R Offline
                        R Offline
                        rhb327
                        wrote on last edited by
                        #11

                        Added extra debug I missed...

                        void Utilities::exportRunFile(quint8 logFileType, QString runfileName) {
                            qDebug() << "DEBUG: " << runfileName << runfileName.length() << runfileName.toUtf8() << runfileName.toUtf8().length();
                            QSaveFile tempFile;
                            QTextStream txtStream;
                            txtStream.setCodec("UTF-8");
                            tempFile.setFileName("C:/Users/richard.bair/Downloads/test.csv");
                            tempFile.open(QIODevice::WriteOnly);
                            txtStream.setDevice(&tempFile);
                            txtStream << tr("Run Filename") << "," << runfileName << "\n";
                            txtStream.flush();
                            tempFile.commit();
                            qDebug() << txtStream.codec()->name();
                        

                        @JKSH, I think you're on to something...why is System showing?

                        60eb5e66-8203-475b-8c87-a4bcf096e527-image.png

                        JKSHJ 1 Reply Last reply
                        0
                        • R Offline
                          R Offline
                          rhb327
                          wrote on last edited by
                          #12

                          Check this out...

                          void Utilities::exportRunFile(quint8 logFileType, QString runfileName) {
                              qDebug() << "DEBUG: " << runfileName << runfileName.length() << runfileName.toUtf8() << runfileName.toUtf8().length();
                              QSaveFile tempFile;
                              QTextStream txtStream;
                              qDebug() << txtStream.codec()->name();
                              txtStream.setCodec("UTF-8");
                              qDebug() << txtStream.codec()->name();
                              tempFile.setFileName("C:/Users/richard.bair/Downloads/test.csv");
                              tempFile.open(QIODevice::WriteOnly);
                              txtStream.setDevice(&tempFile);
                              qDebug() << txtStream.codec()->name();
                              txtStream << tr("Run Filename") << "," << runfileName << "\n";
                              txtStream.flush();
                              qDebug() << txtStream.codec()->name();
                              tempFile.commit();
                              qDebug() << txtStream.codec()->name();
                          

                          d0f5c891-976d-4063-b9fd-6c61932b387b-image.png

                          1 Reply Last reply
                          0
                          • R rhb327

                            Added extra debug I missed...

                            void Utilities::exportRunFile(quint8 logFileType, QString runfileName) {
                                qDebug() << "DEBUG: " << runfileName << runfileName.length() << runfileName.toUtf8() << runfileName.toUtf8().length();
                                QSaveFile tempFile;
                                QTextStream txtStream;
                                txtStream.setCodec("UTF-8");
                                tempFile.setFileName("C:/Users/richard.bair/Downloads/test.csv");
                                tempFile.open(QIODevice::WriteOnly);
                                txtStream.setDevice(&tempFile);
                                txtStream << tr("Run Filename") << "," << runfileName << "\n";
                                txtStream.flush();
                                tempFile.commit();
                                qDebug() << txtStream.codec()->name();
                            

                            @JKSH, I think you're on to something...why is System showing?

                            60eb5e66-8203-475b-8c87-a4bcf096e527-image.png

                            JKSHJ Offline
                            JKSHJ Offline
                            JKSH
                            Moderators
                            wrote on last edited by
                            #13

                            @rhb327 said in Struggling to stream byte array .toUtf8() out a TextStream:

                            @JKSH, I think you're on to something...why is System showing?

                            I'm not sure. Maybe calling setDevice() resets the codec somehow?

                            Try calling setCodec() after setDevice(), or pass the device in the QTextSteram constructor, like the code in my previous post.

                            Qt Doc Search for browsers: forum.qt.io/topic/35616/web-browser-extension-for-improved-doc-searches

                            1 Reply Last reply
                            0

                            • Login

                            • Login or register to search.
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • Users
                            • Groups
                            • Search
                            • Get Qt Extensions
                            • Unsolved