Data Manipulation

MScottM

Hi All,

I am working on a project as a way to refresh on programming, and learn the Qt platform.

The project is to take a log-file (a text file with comma delimited data) generated by a monitoring system and trying to make it possible to view the data graphically in different ways - mainly make it friendly for human viewing. I've been doing pretty well so far - when I hit a problem, I've been able to solve it by reading the Qt documentation and searching the forums. I've gotten to the point where, through my GUI, I can browse to, and select a file and read in the data I want in QByteArrays.

Now though, I'm at the point where I need to do something with the data and I can't seem to figure out what the problem is, after a week or so of trying different things. I'm sure it's something fundamental that I'm missing, but I'm getting frustrated and it's time to reach out for help.

The data is all recorded in hex values from different instruments - here is the file header:

Date,Time(UTC),Bus Name,PGN#,PGN Priority,PGN Src. Addr.,PGN Dest. Addr.,Data Length,DB0,DB1,DB2,DB3,DB4,DB5,DB6,DB7

And two rows of sample data:

20151029,140040,NMEA2000,127250,2,01,ff,8,ff,0a,bd,ff,7f,ff,7f,fd
20151029,140040,NMEA2000,129025,2,03,ff,8,ab,23,8e,13,0e,8f,6b,b9

Most of the data manipulation involves concatenating two or more of the hex values, converting to decimal, and some math operations. For instance: In the sample data above, to make the vessel heading (PGN# 127250) human readable, I have to concatenate DB2 with DB1 (in the data above: '0a, bd' needs to become 'bd0a'), change the resulting hex value to decimal (bd0a = 48,394), multiply by 0.0001 (=4.8394) which gets me the heading in radians, then convert to degrees (4.8397*57.29= compass heading of 277.3 degrees).

It seems simple, but I can't figure out how to do the concatenation and conversion from string to hex - I am stuck, stuck stuck.

Here is the section of code that creates the QByteArrays:

while (!file.atEnd()) {
            QByteArray columnRead = file.readLine();
            timeStamp.append(columnRead.split(',').at(1));
            PGNList.append(columnRead.split(',').at(3));
            DB1.append(columnRead.split(',').at(9));
            DB2.append(columnRead.split(',').at(10));
            }

This gets me the data like this - "0a" "bd"

then I try to convert from string to hex and concatenate, but it just isn't working like I think it should be:

void MainWindow::on_btnHeading_clicked()
{
    QStringList HeadingList;

    int DB1Count = DB1.count();
    for (i=0; i<DB1Count; ++i) {
        QString DB1Data=(DB1.at(i));
        DB1hex=DB1Data.toInt(&ok, 16);
        DB1Array.append(DB1hex);
    }

    int DB2Count = DB2.count();
    for (i=0; i<DB2Count; ++i) {
        QString DB2Data=(DB2.at(i));
        DB2hex=DB2Data.toInt(&ok, 16);
        DB2Array.append(DB2hex);
    }

    qDebug() << "DB1Array" << DB1Array.at(3) << DB1Array.at(4);

    PGNListCount = PGNList.count();
    for (i=0;i<PGNListCount;i++) {
        if (PGNList.at(i).contains("127250")) {
            HeadingList.append(PGNList.at(i));
            DB21Data.append(DB2Array + DB1Array);
            
        }

 }

qDebug() << "DB21Data" << DB21Data.at(3);

}

The last qDebug statment just prints the ascii symbols of the hex (this isn't from the sample data above):

Debugging starts
DB1Raw "fd" "2e"
DB1Array ý .
DB21Data w
Debugging has finished

Any help would be appreciated! I'm happy to post all of my code if it helps - was trying to keep it short.

6thC

It's been a while since I was doing some checksum work... Not really sure if I'm even close but no one else has said anything yet so... Is something like this any good for you?

int iFieldWidth = 2;
int iNumberBase = 16;
hexDataSum = QString("%1").arg(parsedValue, iFieldWidth, iNumberBase, QLatin1Char( '0' )).toUpper();

It's part of a pretty specific section of my message handling needs, but it might help?

jsulm

@MScottM What is the type of DB1hex?
Also you should check ok after converting from hex string to integer.

J.Hilk

@MScottM

Hi,

there are a couple of ways to do what you want to do. I'll post a code snippet from one of my projects that should help you:

//for converting 2* 8bit to int
QByteArray buffer;
int value = static_cast<int>(buffer[i+1]) + static_cast<int>(buffer[i])*256;

jsulm

@J.Hilk Please be aware that he is trying to convert a string containing hex numbers - not two bytes to int.

J.Hilk

@jsulm Ok, fair enough, I was a bit confused by him placing the String in a QByteArray in the first place.

but thats even easier, and my code can be adjusted to that:

QString part1, part2;

//Either concat them like you wanted to
part1.append(part2);
int value = part1.toInt(checkBool/*or nullptr*/,16);

//Or convert them first to int similair to my first post
int value = part1.toInt(nullptr,16)*256 + part2.toInt(nullptr,16);

VRonin

you should really use QTextStream instead. Streams are Qt's preferred way of interacting with devices.

instead of while (!file.atEnd()) {/* ... */ } (P.S. no need to call split multiple times) make sure you have file opened in text mode, something like file.open(QIODevice::ReadOnly | QIODevice::Text)):

QTextStream fileStream(&file);
QString line;
while (fileStream.readLineInto(&line)) {
const QStringList lineParts = line.split(',',QString::KeepEmptyParts);
timeStamp << lineParts.at(1);
PGNList<< lineParts.at(3);
DB1<< lineParts.at(9);
DB2<< lineParts.at(10);
const QString concatenated =  DB1.last() + DB2.last();
qDebug() << concatenated << concatenated.toInt(Q_NULLPTR,16);
}

MScottM

Thank you all for your input and code snippets! I'll be studying and digesting today, thanks!!

@jsulm - I had declared DB1hex first as long, then tried as int.

@VRonin - Are QTextStream's faster? These log files can be big, running to 50-60 megabytes and 150k plus lines. I've been worried about how long it will eventually take to parse out a whole file (I've been working with a file that I cut down to 1000 lines).

Edit

@J-Hilk - if anything in my code doesn't make sense, it's 'cause I'm coding challenged :) I appreciate any input that helps me learn the right way of doing things.

Best regards

VRonin

@MScottM said in Data Manipulation:

Are QTextStream's faster?

No. are they slower, no.

I parsed the entire Hamlet play in 3 seconds so it should not be that bad

MScottM

@VRonin Okay - I rewrote the code to use QTextStream, and the data in the qDebug statement in your code snippet is exactly what I'm looking for (thanks!!), but when I try to put it into some kind of array that I can access from somewhere else, I'm getting errors - conversion errors, mismatch errors, ugh! I think I just don't have some syntax right.

Shouldn't the converted variable 'concatenated' (which is now 'int') be able to go into a QByteArray?

concatenated.toInt(&ok, 16);
QByteArray concatData = concatenated;

Thanks for the input so far - I'm getting close to getting this figured out!

-Scott

VRonin

@MScottM said in Data Manipulation:

Shouldn't the converted variable 'concatenated' (which is now 'int') be able to go into a QByteArray?

Yes but you have to specify the format, for example QByteArray concatData = concatenated.toUtf8();

@MScottM said in Data Manipulation:

concatenated.toInt(&ok, 16);

toInt() is const so unless you use the return value it does nothing.

MScottM

Okay, feeling pretty dense...sorry! I think I'm not understanding something basic. Nothing I've tried so far (trying to learn from the hints above) allows me to put the converted variable into an array for manipulating in another area of the code.

It fails with "invalid conversion from 'int' to 'const char*' [-fpermissive"]

MScottM

YES! Two steps forward and one step back.

The code now runs and I can see the concatenated data as a string, but when I ask for the count of the arrays, PGNList gives me 1000 which is equal to the number of lines in the log file (and what I expect), but the QByteArray concatData says 4. I'm guessing that is the length of the concatenated string going into it (like CDAB)?

So do I have to define the size of the array element somehow? Just guessing, anyway, I'm further along than I was a week ago!

MScottM

I finally figured it out - thanks to all for the hints and tips.

Now on to the part that I was really trying to get to: displaying the data in a useful way. I think I'm going to try to graph this particular data (heading) with heading on the Y axis and a timestamp on the X.

Best regards!