Qt Forum

    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Search
    • Unsolved

    Solved QByteArray to string?

    Language Bindings
    python3 pyqt5
    4
    29
    24664
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • JonB
      JonB last edited by A Former User

      With Python 3/PyQt 5.x, I have a QByteArray (as it happens, returned by QProcess.readAllStandardOutput(), though that shouldn't matter). I want to get the Python "string" of it, to display.

      Things like, say, a straightforward str(qba) or str(qba.data()) did not deliver the correct result. To be specific, for an output of

      Hello
      world
      

      (nothing special, just two lines of output) all of these varieties produce

      b'Hello\nworld'
      

      exactly as that string (verified in debugger and on display), i.e. the content is enclosed by literal b' and ', and newlines are representation \n characters, not embedded actual newlines. (This must be some representation of the QByteArray content, like "these are the bytes".)

      After searching around the web (nothing covered it), and several attempts, I found that the only thing which works correctly is:

      str(qba.data(), encoding='utf-8')
      

      (well, you can play with which encoding a bit, but I'm thinking UTF-8 is the best thing)

      I am surprised such a non-obvious construct is not commented upon elsewhere. Do Pythonistas here agree this is what is required?

      1 Reply Last reply Reply Quote -1
      • JonB
        JonB last edited by JonB

        [This post cross-posed to https://forum.qt.io/topic/85493/unicodedecodeerror-with-output-from-windows-os-command/18 ]

        For the record, I have done exhaustive investigation, and there is only one solution which "correctly" displays the £ character under Windows. I am exhausted so will keep this brief:

        1. To create a file name with a £ in it: Go into, say, Notepad and use its Save to name a file like abc£.txt. This is in the UK, using a UK keyboard and a standard UK-configured Windows.

        2. Note that at this point if you view the filename in either Explorer or, say, via dir you do see a £, not some other character. That's what my user will want to see in the output of the command he will run.

        3. Run an OS command like robocopy or even dir, which will include the filename in its output.

        4. Read the output with QProcess.readAllStandardOutput(). I'm saying the £ character will arrive as a single byte of value 0x9c.

        5. For the required Python/PyQt decoding bytes->str (QByteArray->QString) line, the only thing which works (does not raise an exception) AND represents the character as a £ is: ba.bytes().decode("cp850").

        That is the "Code Page 850", used in UK/Western Europe (so I'm told). It is the result output of you open a Command Prompt and execute just chcp.

        Any other decoding either raises UnicodeDecodeError (e.g. if utf-8) or decodes but represents it with another character (e.g. if windows_1252 or cp1252).

        I still haven't found a way of getting that cp850 encoding name programatically from anywhere --- if you ask Python for, say, the "system encoding" or "user's preferred encoding" you get the cp1252 --- so I've had to hard-code it. [EDIT: If you want it, it's ctypes.cdll.kernel32.GetConsoleOutputCP().]

        So there you are. I don't have C++ as opposed to Python for Qt, but I have a suspicion that if anyone tries it using the straight C++ Qt way of text = QString(process.readAllStandardOutput()) they'll find they do not actually get to see the £ symbol....

        1 Reply Last reply Reply Quote 1
        • SGaist
          SGaist Lifetime Qt Champion last edited by SGaist

          Hi,

          IIRC, you can also use QString, something like: string = QString(qba).

          Not with PyQt 5 as it seems

          [edit: updated after getting new informations SGaist]

          Interested in AI ? www.idiap.ch
          Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

          JonB 2 Replies Last reply Reply Quote 1
          • JonB
            JonB @SGaist last edited by JonB

            @SGaist
            Oh! That would be much more obvious!
            I thought PyQt (possibly PyQt5) deliberately does not implement QString and you can only use str? Doing the conversion "at the Qt side" would be simpler than "at the Python side". I will have to investigate....

            1 Reply Last reply Reply Quote -1
            • SGaist
              SGaist Lifetime Qt Champion last edited by

              You can find information about string handling in the PyQt5 gotchas article.

              Interested in AI ? www.idiap.ch
              Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

              JonB 1 Reply Last reply Reply Quote 1
              • JonB
                JonB @SGaist last edited by

                @SGaist said in Python3/PyQt5.x QByteArray to string?:

                You can find information about string handling in the PyQt5 gotchas article.

                Yep, I even read that earlier, and apparently did not understand I can do QString if I want to, I thought I tried and found QString not defined in PyQt and so you can only use str. I need to re-try this....

                1 Reply Last reply Reply Quote -1
                • JonB
                  JonB @SGaist last edited by JonB

                  @SGaist said in Python3/PyQt5.x QByteArray to string?:

                  Hi,

                  IIRC, you can also use QString, something like: string = QString(qba).

                  Yeah, you see, I checked and I don't see how you can do this? Where is the QString to use from PyQt, that's exactly the point?? from PyQt5.QtCore import QString doesn't find anything... I'd love to be able to do it that way, but it's back to my understanding that PyQt deliberately hides QString and you have to use Python str instead?

                  Like see the depressing answer to https://riverbankcomputing.com/pipermail/pyqt/2015-September/036470.html:

                  is there some other way to get a hold of a real QString with PyQt5?

                  No.

                  Elsewhere I have come across from PyQt4.QtCore import QString. So I'm thinking PyQt5 actually removed that ability... :(

                  https://riverbankcomputing.com/pipermail/pyqt/2014-January/033561.html informs us the reason is:

                  It's more Pythonic to use Python strings than Qt strings.

                  Great! :(

                  1 Reply Last reply Reply Quote -1
                  • SGaist
                    SGaist Lifetime Qt Champion last edited by

                    My bad, I based my answer on the gotchas article and the QString constructor list. It would be worth mentioning in that article that QString has been remove from PyQt5.

                    By the way QStringList is also in this case.

                    All in all, your trick seems to be the best solution. You can also post in the Riverbank mailing to see if they have any suggestions as best practice to convert QByteArray to string.

                    Interested in AI ? www.idiap.ch
                    Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                    JonB 1 Reply Last reply Reply Quote 1
                    • JonB
                      JonB @SGaist last edited by JonB

                      @SGaist
                      Yeah, that's precisely the conclusion I came to too! Hence the question for confirmation it really is that obscure :)

                      Riverbank have obviously already made clear what they want to do, having explicitly apparently changed it from PyQt4 to PyQt5, so there seems little point in asking them. Unless I am mistaken, they won't much actively respond to a PyQt query from me?

                      1 Reply Last reply Reply Quote 0
                      • SGaist
                        SGaist Lifetime Qt Champion last edited by

                        Why wouldn't they ? You are not asking them to bring back QString, just what they recommend as best practice to convert a QByteArray to a python string following your example.

                        Interested in AI ? www.idiap.ch
                        Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                        JonB 1 Reply Last reply Reply Quote 0
                        • JonB
                          JonB @SGaist last edited by

                          @SGaist
                          I meant, I thought I have tried posting to PyQt forum before, and just no replies. Maybe I'm mistaken though...

                          1 Reply Last reply Reply Quote -1
                          • jazzycamel
                            jazzycamel last edited by

                            The standard python3 way to handle this would be:

                            print(qba.decode('utf8'))
                            

                            This is actually to do with changes between Python versions 2 and 3. In Python 2 str could be used for both text and binary data and was considered 'brittle' by the core devs., so in Python 3 it was decided that str (incl. unicode) would be used only for text and bytes would be used for binary (see more here). Therefore, in PyQt5, when a Qt type containing binary data (QByteArray, the clue is in the name) is converted to a native type, bytes is used rather than str giving the developer the choice of which encoding to use if it is string data.

                            Also, as an active member of the PyQt mailing list, I can say it is normally pretty responsive and helpful so, in future, please think of giving us a second chance :).

                            Hope this helps :)

                            For the avoidance of doubt:

                            1. All my code samples (C++ or Python) are tested before posting
                            2. As of 23/03/20, my Python code is formatted to PEP-8 standards using black from the PSF (https://github.com/psf/black)
                            JonB 1 Reply Last reply Reply Quote 3
                            • JonB
                              JonB @jazzycamel last edited by JonB

                              @jazzycamel
                              Thank you, but I'm sorry, I don't see how. The whole point is the PyQt 4 to 5 changes document (or is it Python 2 to 3, I can't recall) is that it says QByteArray.decode() method was removed? It's not there if I try to use it. Have you tried your suggestion with PyQt5/Python3?

                              1 Reply Last reply Reply Quote -1
                              • jazzycamel
                                jazzycamel last edited by

                                @JNBarchan
                                Apologies, that should have been

                                print(qba.data().decode('utf8'))
                                

                                (That'll teach me to read things properly...!)

                                For the avoidance of doubt:

                                1. All my code samples (C++ or Python) are tested before posting
                                2. As of 23/03/20, my Python code is formatted to PEP-8 standards using black from the PSF (https://github.com/psf/black)
                                JonB 1 Reply Last reply Reply Quote 3
                                • JonB
                                  JonB @jazzycamel last edited by JonB

                                  @jazzycamel
                                  OK, that does work, thank you! Now then, may I ask:

                                  1. QByteArray.data() returns bytes. Where was I supposed to come across documentation for bytes.decode() (e.g. in PyQt?)? [EDIT: I'm a newbie to both Python & Qt. I spend my time looking around the Qt documentation to do this stuff. I'm beginning to guess this is a Python issue, not Qt, but it's a lot to take in!]

                                  2. (Because of #1) I don't know the arguments to decode(). I have used my utf-8 and your utf8 and as far as I can see both work the same. Which is "right"/"preferable"?

                                  3. Can you comment (briefly :) ) on why decode() vs str(encoding=...) is preferable/nicer/more Pythonic?

                                  1 Reply Last reply Reply Quote 0
                                  • jazzycamel
                                    jazzycamel last edited by jazzycamel

                                    1. bytes is a python standard type and is fully documented in the python docs, the particular information you require re. bytes.decode() can be found here.
                                    2. In the documentation linked above you will find a link to Standard Encodings (also part of the python docs) which will tell you all you ever wanted to know about encodings (and more!). utf-8 and utf8 are simply aliases of one another, both are perfectly acceptable (as detailed/listed in the docs) as are U8 and UTF (I think...!).
                                    3. Semantics, but Python is considered to be primarily an object-oriented language and therefore you should use an objects own methods (yes, bytes and str are objects as are all 'types' in Python) rather than a function. In fact, the str() function just invokes an objects own __str__() method as that defines how the object should be represented as a string (true for all types).

                                    For the avoidance of doubt:

                                    1. All my code samples (C++ or Python) are tested before posting
                                    2. As of 23/03/20, my Python code is formatted to PEP-8 standards using black from the PSF (https://github.com/psf/black)
                                    JonB 1 Reply Last reply Reply Quote 3
                                    • JonB
                                      JonB @jazzycamel last edited by

                                      @jazzycamel
                                      Yep, all good stuff, makes sense, thank you very much!

                                      As I edited against #1, I now realise that certain things from Qt via PyQt require me to look at Python documentation rather than Qt.

                                      Since you happen to be here, and are so kind, would you care to comment on one issue which was raised in posts above. In PyQt 4, apparently, you could go s = QString() if you wanted to. Is it indeed correct that in PyQt 5 there really is no such thing as QString anywhere, and you have to deal in Python types like str in every situation? (Doubtless same applies to, say, QByteArray type and bytes, and for other such Qt types where you have decided only to allow the Python type.)

                                      Finally, don't suppose you could make Python be just like C# instead for me, then I'd be much happier? ;-)

                                      jazzycamel 1 Reply Last reply Reply Quote 0
                                      • SGaist
                                        SGaist Lifetime Qt Champion last edited by

                                        @jazzycamel long time no see ! Thanks for the thorough explanation :-)
                                        Parts of it would be a welcome addition to the PyQt5 documentation.

                                        Interested in AI ? www.idiap.ch
                                        Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                                        1 Reply Last reply Reply Quote 0
                                        • jazzycamel
                                          jazzycamel @JonB last edited by

                                          @JNBarchan
                                          There is indeed no such thing as QString() in PyQt5. It shouldn't be necessary as the library takes care of type marshalling between the Python and Qt (C++) types. In fact, while there is a QVariant(), its generally not necessary to use it for the same reason. QByteArray() does exist also, but I would steer clear of it if possible and let PyQt5 deal with via bytes().

                                          No, I will never (and no one else should!) ever make Python like C#!! :)

                                          For the avoidance of doubt:

                                          1. All my code samples (C++ or Python) are tested before posting
                                          2. As of 23/03/20, my Python code is formatted to PEP-8 standards using black from the PSF (https://github.com/psf/black)
                                          JonB 1 Reply Last reply Reply Quote 3
                                          • JonB
                                            JonB @jazzycamel last edited by

                                            @jazzycamel , or anyone else

                                            Having implemented qba.data().decode('utf8') as directed, I have now come across a situation where the QByteArray data returned by QProcess.readAllStandardOutput() from an OS command run under Windows causes the Python/PyQt code to generate a UnicodeDecodeError error, as detailed in my post https://forum.qt.io/topic/85493/unicodedecodeerror-with-output-from-windows-os-command

                                            This makes it impossible to convert the data, blocking the whole behaviour of my usage.

                                            My belief is that this would not be happening at all from C++ where I would simply use whatever methods of QByteArray/QString or the language. The problem is precisely is that I am being forced to use a "Python/PyQt" way of doing this, causing the error in Python/PyQt only, which is exactly why I didn't want to have to do that but cannot get access to the necessary types/methods of Qt from PyQt...?

                                            1 Reply Last reply Reply Quote -1
                                            • SGaist
                                              SGaist Lifetime Qt Champion last edited by

                                              Can you show the code you use ?

                                              Interested in AI ? www.idiap.ch
                                              Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                                              JonB 1 Reply Last reply Reply Quote 0
                                              • First post
                                                Last post