QByteArray not initializing



  • I thought I had a handle on QByteArrays, but there's obviously something I'm missing. Here's a clip from a debug session:
    0_1507825416359_qbytearray.PNG

    Note the value of qMac is "<not accessible>" and indeed if I try to access it (with an assignment, an append, a resize, etc). I get an error.

    Am I defining it wrong? Odd how qKey and qTxt work just fine. Thanks...


  • Qt Champions 2016

    Hi
    Do you use the qMac variable ?
    It could be compiler optimized it out.
    you mean if you do
    qMac.append(xxx)
    you get error ?
    What error?



  • @mrjj yes I do. I try to use it in several ways, and I get an error (usually a segmentation fault).

    I'm wondering whether I need to initialize it on construction or something...


  • Qt Champions 2016

    @mzimmers
    That sounds strange.
    You do try to use it in same function ? ( int test() )

    and you did test with something that is 100% valid?
    the char *mac could be be pointing to invalid stuff.

    so if this crashes
    qmac.append("the test");

    Then there is something really odd going on.



  • @mrjj I'm appending a character at a time:

        uint8_t c;
        n = strlen(mac);
        for (uint32_t i = 0; i < n; i +=2)
        {
            p = mac + i;
            sscanf(p, "%2x", (int *) &c);
            qMac.append((char) c);
        }
    

    I wonder if it's objecting to the content of my characters?

    The behavior is unpredictable; sometimes I get this error instead of a segmentation fault:

    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    terminate called after throwing an instance of 'std::bad_alloc'
      what():  std::bad_alloc
    

  • Qt Champions 2016

    @mzimmers said in QByteArray not initializing:

    std::bad_alloc'

    that is normally for an out of memory situation ?

    the crash could be from sscanf.
    also i think you can go out of bounds with this code
    n = strlen(mac);
    for (uint32_t i = 0; i < n; i +=2)

    if n is not even and you increase with 2 would that not count wrongly ?



  • @mrjj everything you say is true, but it's not crashing on the sscanf() call. And it's not getting far enough into the mac string to overrun. I'm doing the identical thing with 2 other QByteArrays before this one, and they work perfectly.

    I even thought there might be something wrong with my variable name, so I refactored...same thing.



  • Without seeing the rest of the code, I would guess that something in that method is blowing the stack.
    Look for a buffer overrun.


  • Qt Champions 2016

    @mzimmers
    Hmm. if you single step the loop, cant you see what is wrong when
    it crashes ?

    you do the same thing with qText and qKey?



  • @mrjj here's the code in its entirety:

    #include <QByteArray>
    #include <QDebug>
    #include <QCoreApplication>
    #include <QMessageAuthenticationCode>
    #include <QCryptographicHash>
    
    int test(char *key, char *txt, char *mac);
    
    int main(int argc, char *argv[])
    {
        QCoreApplication a(argc, argv);
    
        char  Key8[] =          "60e431591ee0b67f0d8a26aacbf5b77f"  //  arbitrary key
                                "8e0bc6213728c5140546040f0ee37f54";
        char  Txt8[] =                                              //  Len = 1054 BYTES
                    "e5098b6a0b1cfc57c6a76537104a39c48baecb15c6bbb46fbb0b745f9c9e5c05"
                    "cfcfabb33786f7b7b5b0ce74eeec9eb84f87d2494fab3ec1f4d3bd9c99821890"
                    "ee352a1d40964264fbf2c93c6ded2583cc75dcb27bf4fdb489cabcf97bfa5cc6"
                    "4b2352cfb0b3a707a0579eb713b697cd0b5e3377d1feb9f181d7b89cc86dee4f"
                    "ed8269f10e44ec48adc6940c6badbb40122c1dc2d9323920e4e1fbad0b4397d4"
                    "dc38b8ade3b3dace2926f464fa3b5b82ebc5e3b81cf647e8bbd2cb55c9e31ffd"
                    "212f8729b66739421c6106e64ac83d3b9e13cd8321b3a9f10d9171bb8cb74e71"
                    "c34d1e8d0fc8d14b8e5e12bbe2bd2a1431fc224b70d228e4e2063509db26ecd9"
                    "ca7cc402763e69928805600a4a80eab4ae6a2c3792b98c6942195e643f98c0dc"
                    "3fa3c2b07431cbbe113e38fc0b7b45c51c4431700ed29d2736b236f63f759323"
                    "29aa60be9009bd7832f1e1b9ac1503ec84727a1e6c8423c7c5b903e763262d55"
                    "9078e654532e0868f206a468b5b5ebd3eddb4f673536e5f0f8160e5f3311561b"
                    "7cf79c9c440974355965c931aec5c7225f69f776f052ac4bd6b19f85389fd61d"
                    "f60ecabbeb00c8886ff7983d20ac5d81e303bc71253f40806772fd81f9387402"
                    "05a5b7dcd07cce083da258b493d275967f91e4815d656936b342727cfe45f973"
                    "b2a5ac257ce64c5eca4f53be8d9fd90c3dfcb8cd1e2cef15c307449ed02c2e17"
                    "04f4f1be76a40b311ee7cf81987b5089252a807ef3fc99c79eabbc0ef657d897"
                    "037bced04620d32a425015283bcea1b53e0484bb613d30f14c1422f5f82cc29a"
                    "b7228b8375c06bf13d746dd9ff00953a90720badf2577d3ed62cbe7a5f15b3c9"
                    "29d26ffe8aee9d2d17391ebc6a79f4bd235d5f7b2db2455343d9d7c6b27972cc"
                    "6071c36a0d112f86d98972fb06a186e900abc64e9ab653db9b05b70079c0c84a"
                    "64e8cfee8690eaa68a4bafbb5be112632e46894ec2cc6e7ce697a4513d517deb"
                    "3e20dbb37ed5963232671e27ef9f62d6b514f0a22c5a5dde2d77e7e184965958"
                    "f5002fe17d47fbd5d9c407644d443ce89eff427360cae9aa788dc8d7d9f62439"
                    "916f139f094ee035884cb29dfa396941f0eec9e8e782da88cdc18e5bc1d9a535"
                    "1b57ce15ac520ffa47e666f87fe5b18ab3c8cb2a48ecf81f36fb8397c6a7a5f5"
                    "9a9fa96cedbb4ecd1c7a6d9d65afdb6bef7791600b6e0a18ba23edb06fc9ec21"
                    "162feccc54f2665611f10db53401b18bade263b3b972da1a612115d144a54260"
                    "97efdf5c6a5d1f3c2d318f687242f993f5f1884bd95f2ece34dd4320cea46f5a"
                    "26c7c945b665402778233bdda9d97c2acd8c4a4ff39dcfdc3a3fbfc5942e3ab8"
                    "ca9ff4aec17293c1fbaf583d603002f93f9befe8909485eb7c30d0e91fac6c22"
                    "8c5fa6c011eddeafbdbe30af20ae53a85206c03d37ac17a30096bfb4f584cd3f"
                    "72ef28a3303cea9cc636095c70bb36de0eb50577704d4faed05bd54da020";
    
        char  Mac8[] =                          //  HMAC_SHA256 7.3728=410ms 18.4320=
                    "4a0ce2f8ffca9e77443e661c88ccdc04ba6ac9bb901642fadae7fe8e4c2d50f7";
    
        test(Key8, Txt8, Mac8);
        return a.exec();
    }
    
    int test(char *key, char *txt, char *mac)
    {
        QByteArray qKey;
        QByteArray qTxt;
        QByteArray qMac;
        uint8_t c;
        char *p;
        uint32_t n;
    
        n = strlen(txt);
        for (uint32_t i = 0; i < n; i += 2)
        {
            p = txt + i;
            sscanf(p, "%2x", (int *) &c);
            qTxt.append((char) c);
        }
    
        n = strlen(key);
        for (uint32_t i = 0; i < n; i +=2)
        {
            p = key + i;
            sscanf(p, "%2x", (int *) &c);
            qKey.append((char) c);
        }
    
        n = strlen(mac);
        for (uint32_t i = 0; i < n; i +=2)
        {
            p = mac + i;
            sscanf(p, "%2x", (int *) &c);
            qMac.append((char) c);
        }
    
        QMessageAuthenticationCode auth(QCryptographicHash::Sha256, qKey);
        auth.addData(qTxt, 1054);
        qMac = auth.result();
    
        return (strcmp(qMac.data(), mac) == 0);
    }
    

    qTxt and qKey work perfectly.

    EDIT:

    I'm aware that the strlen calls might not work right all the time, but they seem to be OK in this example. There's something wrong with my qMac variable, at least in the eyes of the debugger.


  • Qt Champions 2016

    super, let me crash it ;)



  • @mzimmers: Your problem with Mac8 is most likely that the array has an unequal number of chars...

    PS: I already said in your other thread that you can use QByteArray::fromHex().



  • @aha_1980 not sure what you mean by "an unequal number of chars".

    I saw your other response, but didn't understand it. I'll take another look.



  • @mzimmers said in QByteArray not initializing:

    @aha_1980 not sure what you mean by "an unequal number of chars".

    strlen(Mac8) == 65

    Your loop is iterating with steps of two. So you're reading over the end of Mac8 with sscanf().

    I'm not sure what QByteArray::fromHex() will do with this invalid input, but most likely it will not crash. (If it does, we create a bugreport).



  • @aha_1980 this crash is occurring on the second iteration through the loop, so I don't think it's an array/string overrun.


  • Qt Champions 2016

    Oddly enough i dont crash here
    it just converts to 32 values each time.

    alt text


  • Qt Champions 2016

    @mzimmers
    Hi
    Can you try
    // QCoreApplication a(argc, argv);
    // a.exec();
    and just return 0;

    You are not sending it a close signal so i think crash is due to killing it with debugger.

    At least i cannot crash it without running the coreapp and exec().



  • @mrjj yeah, I'm pretty sure my loops are OK. I honestly don't know what's going on -- maybe some weird bug in the constructor.

    I removed the core app stuff...same results.

    I think I'll implement aha_1980's suggestion. I took another look, and it's a lot simpler than this. I'll report back on how I do with it.


  • Qt Champions 2016

    @mzimmers
    Ok. i checked the loop and its 64 in length and result is 32 in bytearray.
    no crashes at any time.
    Very odd then.
    Im on 64 bit.



  • I'm on Windows 10, using MinGW, FWIW. Just tried null terminating the string myself, and still bombs in the loop (though aha's suggestion works).


  • Qt Champions 2016

    @mzimmers
    Hmm visual studio here.
    Maybe its related to mingw.
    I will try tomorrow at work with mingw and see.
    I dont like when crashes are not reproducible.



  • @mzimmers said in QByteArray not initializing:

    @mrjj here's the code in its entirety:

    uint8_t c;
    
    sscanf(p, "%2x", (int *) &c);
    

    c holds a 8-bit int, right? Firstly that's not right for (int *) &c, and secondly look at sscanf("%2x", it will expect the address of a whole int (32-bit? 64-bit?) to store its result. So you're overwriting the local stack space, hence possible crashing? Or, uint8_t is allocated on an odd address boundary, which won't be good.

    Make c be a proper int, or fix your sscanf.

    Your understanding of QByteArray is fine, but not of scanf :-)

    Also, if you use QByteArray::fromHex(), you won't need to use sscanf :)



  • @JNBarchan the pointer should be OK, as far as I can see. Your point about the sscanf() overwriting the output buffer may be valid, though it's really odd that the first two loops worked fine.


  • Qt Champions 2016

    @JNBarchan
    Thats a really good observation and might explain why
    no crashing here 64bit. Completely overlooked it :)
    alt text



  • @mzimmers said in QByteArray not initializing:

    @JNBarchan the pointer should be OK

    What pointer, and what do you mean by "OK"?

    P.S. I can explain why it goes through sometimes and fails others, but it's going to mean quite a bit of typing, do you really want to know?



  • @mrjj
    If that sscanf supports %hhx that should work.



  • @JNBarchan in this line of code:

           sscanf(p, "%2x", (int *) &c);
    

    the address formed by the cast should be fine, as far as I can tell.

    I agree the problem is in the sscanf. This line of code:

            sscanf(p, "%hhx", &c);
    

    compiles but doesn't work right (c keeps getting 0xff). I'm going to go with aha_1980's approach and call it a day.

    Thanks to everyone who looked at this.



  • @mzimmers said in QByteArray not initializing:

    @JNBarchan in this line of code:

           sscanf(p, "%2x", (int *) &c);
    

    the address formed by the cast should be fine, as far as I can tell.

    The address is indeed a valid address, but it points to an area reserved to hold one byte and you're storing 4 or 8 bytes there, which is very naughty! Hope you understand.


  • Qt Champions 2016

    @JNBarchan said in QByteArray not initializing:

    which is very naughty!

    Buffer overflows - the pinnacle of C programming ... ;)



  • @kshegunov I expected the "%2x" in the sscanf to restrict output to one byte (2 hex characters), but I guess it doesn't work that way.


  • Qt Champions 2016

    Not really, no. There's no compile-time type checking with variadic functions (which scanf and related are). It will expect you to give it a proper integer to write to. But really your error is the cast itself. You can't cast pointers like this. You have a char on the stack and you cast its address to a int *. From there on the compiler will consider that this pointer is referencing an integer, meaning you have "hijacked" 3 bytes from the nearby variables in the stack. If you modify your code like this:

    uint8_t c, c2, c3, c4;
    // ...
    sscanf(p, "%2x", (int *) &c);
    // ...
    

    You'd discover that all those variables - c2, c3 and c4 are modified alongside your original c because sscanf overruns the given buffer and writes outside of it.



  • The original code has:

        QByteArray qMac;
        uint8_t c;
    

    My guess is qMac is stored above c in the local stack. When you write to &c you cobble what's in qMac, so the bug doesn't show up till you reach the qMac loop. Then qMac.append((char) c); gets evaluated and crashes. Am I right?

    Behaviour could vary across 32/64 size and specific compiler allocation or register code generated.



  • @JNBarchan that explanation makes sense. I'll have to brush up on how sscanf works (though I'd rather use something else) to avoid similar problems in the future.

    Thanks to all who participated.



  • @kshegunov said in QByteArray not initializing:

    uint8_t c, c2, c3, c4;
    // ...
    sscanf(p, "%2x", (int *) &c);
    // ...
    

    You'd discover that all those variables - c2, c3 and c4 are modified alongside your original c because sscanf overruns the given buffer and writes outside of it.

    @kshegunov Purely OOI, did you actually test this code? I was actually thinking, assuming local variables allocated down the stack in order they're encountered, you'd have to sscanf into c4 rather than c to get that behaviour from the way they're laid out...?


  • Qt Champions 2016

    @JNBarchan said in QByteArray not initializing:

    @kshegunov Purely OOI, did you actually test this code?

    Not at the time, no.

    I was actually thinking, assuming local variables allocated down the stack in order they're encountered, you'd have to sscanf into c4 rather than c to get that behaviour from the way they're laid out...?

    Depends on many things in fact, but most important of those would be the compiler (also which way the stack goes address-wise). Don't forget the compiler's allowed to reorder the variables and memory access freely (with exception of memory fences and volatile, but that's another discussion) if it finds it convenient to do so. Meaning, if you don't reference a variable it may even strip it from the assembly. That is for release, though, I've noticed compilers (at least g++) will not remove things from the binary in debug mode even if redundant/unnecessary.

    Anyway, I just made a quick test with g++ (7.2):

    uint8_t c = 0, c2 = 0, c3 = 0, c4 = 0;
    *reinterpret_cast<int *>(&c) = -1; // c = c2 = c3 = c4 = 0xFF
    

    and

    uint8_t c = 0, c2 = 0, c3 = 0, c4 = 0, c5 = 0;
    *reinterpret_cast<int *>(&c) = -1; // c = c3 = c4 = c5 = 0xFF; c2 = 0x00
    

    and the assembly for the second one:

            8 [1]	    uint8_t c = 0, c2 = 0, c3 = 0, c4 = 0, c5 = 0;
    0x5555555547c5  <+0x000b>        c6 45 fb 00        movb   $0x0,-0x5(%rbp)
    0x5555555547c9  <+0x000f>        c6 45 ff 00        movb   $0x0,-0x1(%rbp)
    0x5555555547cd  <+0x0013>        c6 45 fe 00        movb   $0x0,-0x2(%rbp)
    0x5555555547d1  <+0x0017>        c6 45 fd 00        movb   $0x0,-0x3(%rbp)
    0x5555555547d5  <+0x001b>        c6 45 fc 00        movb   $0x0,-0x4(%rbp)
            9 [1]	    *reinterpret_cast<int *>(&c) = -1;
    0x5555555547d9  <+0x001f>        48 8d 45 fb        lea    -0x5(%rbp),%rax
    0x5555555547dd  <+0x0023>        c7 00 ff ff ff ff  movl   $0xffffffff,(%rax)
    

    As you can see the variable c was actually reordered by the compiler to occupy an address deeper in the stack.



  • @kshegunov you'll see reordering in other areas of code as well. For bit-bangers like me, the most notorious is the realignment of scalars within a structure. Some of the compilers for embedded systems have ways to prevent it, but you usually have to take care of it yourself.

    In retrospect it's pretty obvious that sscanf would behave the way it did. At the time, I was just hustling through a problem and didn't think it through.

    One of the challenges I'm having on this project is all the necessary data type conversions for data streams. As a C++ programmer, I'm partial to the string object, but I'm using an AES facility that likes uints, an authentication utility that uses char arrays, and then there's Qt's QStrings and QByteArrays. The compiler catches a lot of mismatches, but there are still a lot of ways to hit the weeds.


  • Qt Champions 2016

    @mzimmers said in QByteArray not initializing:

    For bit-bangers like me, the most notorious is the realignment of scalars within a structure.

    Do you mean structure padding? If so, that's quite known and well documented. And usually the compiler does that for a good reason, but this is not reordering as such. Here the compiler actually switched the memory layout of the variables for whatever reason, and you don't see that with structures, it may only include padding between the members but order is fixed.



  • @kshegunov I did a little research; it looks like the problem has been obviated in C++11:

    link text


  • Qt Champions 2016

    Good job! I didn't know that particular peculiarity, simply because it never occurred to me one could use a struct/class with interleaving access specifiers for POD. Perhaps that's the reason I had never hit it and thus me not knowing it. Anyway, thanks for the info, it's appreciated!



  • Do you mean structure padding? If so, that's quite known and well documented.

    In rare cases you may have to use the #pragma pack directive
    https://stackoverflow.com/questions/3318410/pragma-pack-effect


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.