Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Data Manipulation

Data Manipulation

Scheduled Pinned Locked Moved Solved General and Desktop
14 Posts 5 Posters 4.0k Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • 6thC6 Offline
    6thC6 Offline
    6thC
    wrote on last edited by
    #2

    It's been a while since I was doing some checksum work... Not really sure if I'm even close but no one else has said anything yet so... Is something like this any good for you?

    int iFieldWidth = 2;
    int iNumberBase = 16;
    hexDataSum = QString("%1").arg(parsedValue, iFieldWidth, iNumberBase, QLatin1Char( '0' )).toUpper();
    

    It's part of a pretty specific section of my message handling needs, but it might help?

    1 Reply Last reply
    1
    • M MScottM

      Hi All,

      I am working on a project as a way to refresh on programming, and learn the Qt platform.

      The project is to take a log-file (a text file with comma delimited data) generated by a monitoring system and trying to make it possible to view the data graphically in different ways - mainly make it friendly for human viewing. I've been doing pretty well so far - when I hit a problem, I've been able to solve it by reading the Qt documentation and searching the forums. I've gotten to the point where, through my GUI, I can browse to, and select a file and read in the data I want in QByteArrays.

      Now though, I'm at the point where I need to do something with the data and I can't seem to figure out what the problem is, after a week or so of trying different things. I'm sure it's something fundamental that I'm missing, but I'm getting frustrated and it's time to reach out for help.

      The data is all recorded in hex values from different instruments - here is the file header:

      Date,Time(UTC),Bus Name,PGN#,PGN Priority,PGN Src. Addr.,PGN Dest. Addr.,Data Length,DB0,DB1,DB2,DB3,DB4,DB5,DB6,DB7

      And two rows of sample data:

      20151029,140040,NMEA2000,127250,2,01,ff,8,ff,0a,bd,ff,7f,ff,7f,fd
      20151029,140040,NMEA2000,129025,2,03,ff,8,ab,23,8e,13,0e,8f,6b,b9

      Most of the data manipulation involves concatenating two or more of the hex values, converting to decimal, and some math operations. For instance: In the sample data above, to make the vessel heading (PGN# 127250) human readable, I have to concatenate DB2 with DB1 (in the data above: '0a, bd' needs to become 'bd0a'), change the resulting hex value to decimal (bd0a = 48,394), multiply by 0.0001 (=4.8394) which gets me the heading in radians, then convert to degrees (4.8397*57.29= compass heading of 277.3 degrees).

      It seems simple, but I can't figure out how to do the concatenation and conversion from string to hex - I am stuck, stuck stuck.

      Here is the section of code that creates the QByteArrays:

      while (!file.atEnd()) {
                  QByteArray columnRead = file.readLine();
                  timeStamp.append(columnRead.split(',').at(1));
                  PGNList.append(columnRead.split(',').at(3));
                  DB1.append(columnRead.split(',').at(9));
                  DB2.append(columnRead.split(',').at(10));
                  }
      

      This gets me the data like this - "0a" "bd"

      then I try to convert from string to hex and concatenate, but it just isn't working like I think it should be:

      void MainWindow::on_btnHeading_clicked()
      {
          QStringList HeadingList;
      
          int DB1Count = DB1.count();
          for (i=0; i<DB1Count; ++i) {
              QString DB1Data=(DB1.at(i));
              DB1hex=DB1Data.toInt(&ok, 16);
              DB1Array.append(DB1hex);
          }
      
          int DB2Count = DB2.count();
          for (i=0; i<DB2Count; ++i) {
              QString DB2Data=(DB2.at(i));
              DB2hex=DB2Data.toInt(&ok, 16);
              DB2Array.append(DB2hex);
          }
      
          qDebug() << "DB1Array" << DB1Array.at(3) << DB1Array.at(4);
      
          PGNListCount = PGNList.count();
          for (i=0;i<PGNListCount;i++) {
              if (PGNList.at(i).contains("127250")) {
                  HeadingList.append(PGNList.at(i));
                  DB21Data.append(DB2Array + DB1Array);
                  
              }
      
       }
      
      qDebug() << "DB21Data" << DB21Data.at(3);
      
      }
      

      The last qDebug statment just prints the ascii symbols of the hex (this isn't from the sample data above):

      Debugging starts
      DB1Raw "fd" "2e"
      DB1Array ý .
      DB21Data w
      Debugging has finished

      Any help would be appreciated! I'm happy to post all of my code if it helps - was trying to keep it short.

      jsulmJ Offline
      jsulmJ Offline
      jsulm
      Lifetime Qt Champion
      wrote on last edited by
      #3

      @MScottM What is the type of DB1hex?
      Also you should check ok after converting from hex string to integer.

      https://forum.qt.io/topic/113070/qt-code-of-conduct

      1 Reply Last reply
      1
      • M MScottM

        Hi All,

        I am working on a project as a way to refresh on programming, and learn the Qt platform.

        The project is to take a log-file (a text file with comma delimited data) generated by a monitoring system and trying to make it possible to view the data graphically in different ways - mainly make it friendly for human viewing. I've been doing pretty well so far - when I hit a problem, I've been able to solve it by reading the Qt documentation and searching the forums. I've gotten to the point where, through my GUI, I can browse to, and select a file and read in the data I want in QByteArrays.

        Now though, I'm at the point where I need to do something with the data and I can't seem to figure out what the problem is, after a week or so of trying different things. I'm sure it's something fundamental that I'm missing, but I'm getting frustrated and it's time to reach out for help.

        The data is all recorded in hex values from different instruments - here is the file header:

        Date,Time(UTC),Bus Name,PGN#,PGN Priority,PGN Src. Addr.,PGN Dest. Addr.,Data Length,DB0,DB1,DB2,DB3,DB4,DB5,DB6,DB7

        And two rows of sample data:

        20151029,140040,NMEA2000,127250,2,01,ff,8,ff,0a,bd,ff,7f,ff,7f,fd
        20151029,140040,NMEA2000,129025,2,03,ff,8,ab,23,8e,13,0e,8f,6b,b9

        Most of the data manipulation involves concatenating two or more of the hex values, converting to decimal, and some math operations. For instance: In the sample data above, to make the vessel heading (PGN# 127250) human readable, I have to concatenate DB2 with DB1 (in the data above: '0a, bd' needs to become 'bd0a'), change the resulting hex value to decimal (bd0a = 48,394), multiply by 0.0001 (=4.8394) which gets me the heading in radians, then convert to degrees (4.8397*57.29= compass heading of 277.3 degrees).

        It seems simple, but I can't figure out how to do the concatenation and conversion from string to hex - I am stuck, stuck stuck.

        Here is the section of code that creates the QByteArrays:

        while (!file.atEnd()) {
                    QByteArray columnRead = file.readLine();
                    timeStamp.append(columnRead.split(',').at(1));
                    PGNList.append(columnRead.split(',').at(3));
                    DB1.append(columnRead.split(',').at(9));
                    DB2.append(columnRead.split(',').at(10));
                    }
        

        This gets me the data like this - "0a" "bd"

        then I try to convert from string to hex and concatenate, but it just isn't working like I think it should be:

        void MainWindow::on_btnHeading_clicked()
        {
            QStringList HeadingList;
        
            int DB1Count = DB1.count();
            for (i=0; i<DB1Count; ++i) {
                QString DB1Data=(DB1.at(i));
                DB1hex=DB1Data.toInt(&ok, 16);
                DB1Array.append(DB1hex);
            }
        
            int DB2Count = DB2.count();
            for (i=0; i<DB2Count; ++i) {
                QString DB2Data=(DB2.at(i));
                DB2hex=DB2Data.toInt(&ok, 16);
                DB2Array.append(DB2hex);
            }
        
            qDebug() << "DB1Array" << DB1Array.at(3) << DB1Array.at(4);
        
            PGNListCount = PGNList.count();
            for (i=0;i<PGNListCount;i++) {
                if (PGNList.at(i).contains("127250")) {
                    HeadingList.append(PGNList.at(i));
                    DB21Data.append(DB2Array + DB1Array);
                    
                }
        
         }
        
        qDebug() << "DB21Data" << DB21Data.at(3);
        
        }
        

        The last qDebug statment just prints the ascii symbols of the hex (this isn't from the sample data above):

        Debugging starts
        DB1Raw "fd" "2e"
        DB1Array ý .
        DB21Data w
        Debugging has finished

        Any help would be appreciated! I'm happy to post all of my code if it helps - was trying to keep it short.

        J.HilkJ Offline
        J.HilkJ Offline
        J.Hilk
        Moderators
        wrote on last edited by
        #4

        @MScottM

        Hi,

        there are a couple of ways to do what you want to do. I'll post a code snippet from one of my projects that should help you:

        //for converting 2* 8bit to int
        QByteArray buffer;
        int value = static_cast<int>(buffer[i+1]) + static_cast<int>(buffer[i])*256;
        

        Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


        Q: What's that?
        A: It's blue light.
        Q: What does it do?
        A: It turns blue.

        jsulmJ 1 Reply Last reply
        1
        • J.HilkJ J.Hilk

          @MScottM

          Hi,

          there are a couple of ways to do what you want to do. I'll post a code snippet from one of my projects that should help you:

          //for converting 2* 8bit to int
          QByteArray buffer;
          int value = static_cast<int>(buffer[i+1]) + static_cast<int>(buffer[i])*256;
          
          jsulmJ Offline
          jsulmJ Offline
          jsulm
          Lifetime Qt Champion
          wrote on last edited by
          #5

          @J.Hilk Please be aware that he is trying to convert a string containing hex numbers - not two bytes to int.

          https://forum.qt.io/topic/113070/qt-code-of-conduct

          J.HilkJ 1 Reply Last reply
          1
          • jsulmJ jsulm

            @J.Hilk Please be aware that he is trying to convert a string containing hex numbers - not two bytes to int.

            J.HilkJ Offline
            J.HilkJ Offline
            J.Hilk
            Moderators
            wrote on last edited by
            #6

            @jsulm Ok, fair enough, I was a bit confused by him placing the String in a QByteArray in the first place.

            but thats even easier, and my code can be adjusted to that:

            QString part1, part2;
            
            //Either concat them like you wanted to
            part1.append(part2);
            int value = part1.toInt(checkBool/*or nullptr*/,16);
            
            //Or convert them first to int similair to my first post
            int value = part1.toInt(nullptr,16)*256 + part2.toInt(nullptr,16);
            

            Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


            Q: What's that?
            A: It's blue light.
            Q: What does it do?
            A: It turns blue.

            1 Reply Last reply
            0
            • VRoninV Offline
              VRoninV Offline
              VRonin
              wrote on last edited by VRonin
              #7

              you should really use QTextStream instead. Streams are Qt's preferred way of interacting with devices.

              instead of while (!file.atEnd()) {/* ... */ } (P.S. no need to call split multiple times) make sure you have file opened in text mode, something like file.open(QIODevice::ReadOnly | QIODevice::Text)):

              QTextStream fileStream(&file);
              QString line;
              while (fileStream.readLineInto(&line)) {
              const QStringList lineParts = line.split(',',QString::KeepEmptyParts);
              timeStamp << lineParts.at(1);
              PGNList<< lineParts.at(3);
              DB1<< lineParts.at(9);
              DB2<< lineParts.at(10);
              const QString concatenated =  DB1.last() + DB2.last();
              qDebug() << concatenated << concatenated.toInt(Q_NULLPTR,16);
              }
              

              "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
              ~Napoleon Bonaparte

              On a crusade to banish setIndexWidget() from the holy land of Qt

              1 Reply Last reply
              2
              • M Offline
                M Offline
                MScottM
                wrote on last edited by MScottM
                #8

                Thank you all for your input and code snippets! I'll be studying and digesting today, thanks!!

                @jsulm - I had declared DB1hex first as long, then tried as int.

                @VRonin - Are QTextStream's faster? These log files can be big, running to 50-60 megabytes and 150k plus lines. I've been worried about how long it will eventually take to parse out a whole file (I've been working with a file that I cut down to 1000 lines).

                Edit

                @J-Hilk - if anything in my code doesn't make sense, it's 'cause I'm coding challenged :) I appreciate any input that helps me learn the right way of doing things.

                Best regards

                VRoninV 1 Reply Last reply
                0
                • M MScottM

                  Thank you all for your input and code snippets! I'll be studying and digesting today, thanks!!

                  @jsulm - I had declared DB1hex first as long, then tried as int.

                  @VRonin - Are QTextStream's faster? These log files can be big, running to 50-60 megabytes and 150k plus lines. I've been worried about how long it will eventually take to parse out a whole file (I've been working with a file that I cut down to 1000 lines).

                  Edit

                  @J-Hilk - if anything in my code doesn't make sense, it's 'cause I'm coding challenged :) I appreciate any input that helps me learn the right way of doing things.

                  Best regards

                  VRoninV Offline
                  VRoninV Offline
                  VRonin
                  wrote on last edited by
                  #9

                  @MScottM said in Data Manipulation:

                  Are QTextStream's faster?

                  No. are they slower, no.

                  I parsed the entire Hamlet play in 3 seconds so it should not be that bad

                  "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                  ~Napoleon Bonaparte

                  On a crusade to banish setIndexWidget() from the holy land of Qt

                  1 Reply Last reply
                  0
                  • M Offline
                    M Offline
                    MScottM
                    wrote on last edited by
                    #10

                    @VRonin Okay - I rewrote the code to use QTextStream, and the data in the qDebug statement in your code snippet is exactly what I'm looking for (thanks!!), but when I try to put it into some kind of array that I can access from somewhere else, I'm getting errors - conversion errors, mismatch errors, ugh! I think I just don't have some syntax right.

                    Shouldn't the converted variable 'concatenated' (which is now 'int') be able to go into a QByteArray?

                    concatenated.toInt(&ok, 16);
                    QByteArray concatData = concatenated;
                    

                    Thanks for the input so far - I'm getting close to getting this figured out!

                    -Scott

                    VRoninV 1 Reply Last reply
                    0
                    • M MScottM

                      @VRonin Okay - I rewrote the code to use QTextStream, and the data in the qDebug statement in your code snippet is exactly what I'm looking for (thanks!!), but when I try to put it into some kind of array that I can access from somewhere else, I'm getting errors - conversion errors, mismatch errors, ugh! I think I just don't have some syntax right.

                      Shouldn't the converted variable 'concatenated' (which is now 'int') be able to go into a QByteArray?

                      concatenated.toInt(&ok, 16);
                      QByteArray concatData = concatenated;
                      

                      Thanks for the input so far - I'm getting close to getting this figured out!

                      -Scott

                      VRoninV Offline
                      VRoninV Offline
                      VRonin
                      wrote on last edited by VRonin
                      #11

                      @MScottM said in Data Manipulation:

                      Shouldn't the converted variable 'concatenated' (which is now 'int') be able to go into a QByteArray?

                      Yes but you have to specify the format, for example QByteArray concatData = concatenated.toUtf8();

                      @MScottM said in Data Manipulation:

                      concatenated.toInt(&ok, 16);

                      toInt() is const so unless you use the return value it does nothing.

                      "La mort n'est rien, mais vivre vaincu et sans gloire, c'est mourir tous les jours"
                      ~Napoleon Bonaparte

                      On a crusade to banish setIndexWidget() from the holy land of Qt

                      1 Reply Last reply
                      1
                      • M Offline
                        M Offline
                        MScottM
                        wrote on last edited by MScottM
                        #12

                        Okay, feeling pretty dense...sorry! I think I'm not understanding something basic. Nothing I've tried so far (trying to learn from the hints above) allows me to put the converted variable into an array for manipulating in another area of the code.

                        It fails with "invalid conversion from 'int' to 'const char*' [-fpermissive"]

                        1 Reply Last reply
                        0
                        • M Offline
                          M Offline
                          MScottM
                          wrote on last edited by
                          #13

                          YES! Two steps forward and one step back.

                          The code now runs and I can see the concatenated data as a string, but when I ask for the count of the arrays, PGNList gives me 1000 which is equal to the number of lines in the log file (and what I expect), but the QByteArray concatData says 4. I'm guessing that is the length of the concatenated string going into it (like CDAB)?

                          So do I have to define the size of the array element somehow? Just guessing, anyway, I'm further along than I was a week ago!

                          1 Reply Last reply
                          0
                          • M Offline
                            M Offline
                            MScottM
                            wrote on last edited by
                            #14

                            I finally figured it out - thanks to all for the hints and tips.

                            Now on to the part that I was really trying to get to: displaying the data in a useful way. I think I'm going to try to graph this particular data (heading) with heading on the Y axis and a timestamp on the X.

                            Best regards!

                            1 Reply Last reply
                            0

                            • Login

                            • Login or register to search.
                            • First post
                              Last post
                            0
                            • Categories
                            • Recent
                            • Tags
                            • Popular
                            • Users
                            • Groups
                            • Search
                            • Get Qt Extensions
                            • Unsolved