More data help

jnewing

the binary data is pulled from a file
fields are separated by 0x7c "|" char

the data
@46 49 4C 45 48 45 41 44 45 52 8C 00 00 00 30 00 00 00 7C 53 54 52 49
4E 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 7C 03 00 7C 73 74 72 7C 0B 00
7C 61 6E 6F 74 68 65 72 20 73 74 72 7C 1E 00 00 00 7C 09 00 7C 30 30
30 2E 34 33 2E 32 39 7C
@

data breakdown;
breakdown of the data is as follows;
@uchar[0x0a] "46 49 4C 45 48 45 41 44 45 52 " (this is a constant file header)
ulong[0x4] "8C 00 00 00" (unsigned long value)
ulong[0x4] "30 00 00 00" (unsigned long value)
uchar[0x1] "7C" (field separator)
uchar[0x3f] "53 54 52 49
4E 47 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00" (constant char[])
uchar[0x1] "7C" (field separator)
ushort[0x2] "03 00" (length of the next char[] after the field separator)
uchar[0x1] "7C" (field separator)
uchar[0x3] "73 74 72" (char[] length defined by previous field NOT including the field separator)@

currently what I'm doing is reading the data one char at a time looking for field separator 0x7c and put prior to that into a variable.
I know this is not very efficient etc... this is why I'm looking for a better way would a struct be an option?
@
struct foo {
unsigned char f1[0x0a];
unsigned long f2[0x4];
...
unsigned shot f3_len[0x2];
unsigned char f3[f3_len];
...
} bar;

infile.read(bar, sizeof(bar));
@

any help / feedback is appreciated - thanks

jnewing

Also another question I had, didn't feel it warranted a new thread,

Say I had the above data in a QByteArray, know say I know that at offset 0x53, the next two bytes make up a qint16 integer, what would be the best way to go about reading that into an int?

currently I'm using QString to turn it into a string and then toInt from there, this seems a little much, i though about using memcopy however really was hoping there is a nicer way of doing it.

-thx

vsorokin

Using of structure is better way.
Try look on QByteArray().data() method, it returned const char* on data, may be you can add your offset for this and get needed bytes

goetz

QByteArray has some method that could be of use for you:

indexOf with a single char can help you finding the field separators
mid() can extract the snippets you're in need of (use in conjunction with the indexes found with the previous function)
toInt, toLong might be of use for the ints, but watch out for endianess!

If you create the file you read of with Qt too, you should have a look at QDataStream. It contains nice platform independent conversion methods for various types.

jnewing

volker oddly enough this does not work

@qint16 foo = data.mid(0x64, 2).toInt();@

so to test i added some watches and did the following
@QByteArray foo = data.mid(0x64, 2);
qint16 bar = foo.toInt(); // note i also tried toShort();
@

foo does indeed get the data that is expected "0x03, 0x00" however it does not convert this
to a int, sort or anything else. they all return 0

goetz

Ah, sorry, I misread the docs. You'll have to convert the data to an int yourself:

@
qint16 bar = ba.at(0x64) + (ba.at(0x64 +1) << 8);
@

QDataStream is not an option?

jnewing

I was unable to get datastream to work :/ ether, have you tried it?

andre

QDataStream works reliably for me. If you have the option of using it, I suggest you do.

Just make sure that if you stream integers in and out, you use one of Qts explicit types: quint16, qint64, etc., so you have well defined items in your binairy file. These types are guaranteed to be the same on all supported platforms. Also, it is wise to explicitly set a version for the datastream at both ends.

You can even overload the operator<<() and operator>>() for your class or struct, and stream the whole block of data in one go that way.

goetz

[quote author="jnewing" date="1303253359"]I was unable to get datastream to work :/ ether, have you tried it?[/quote]

It works perfectly. But you must create the file using QDataStream for writing too!

jnewing

I found a quick simple way, here is an example:

@
QDataArray foo; // is filled with binary data
qint16 bar = static_cast<qint16>(data[0x5e]);
@

that works quite efficiently.

thanks: Volker, Andre

goetz

This code is not platform independent (big/small endian!). You will run into some troubles eventually. Do yourself a favor and do it the right way from the very beginning!

jnewing

[quote author="Volker" date="1303300686"]This code is not platform independent (big/small endian!). You will run into some troubles eventually. Do yourself a favor and do it the right way from the very beginning![/quote]

Yah your right, thx again.

tobias.hunger

"QtEndian":http://doc.qt.nokia.com/4.7/qtendian.html has some helper functions for endianess issues. A little known corner of Qt:-)

jnewing

[quote author="Tobias Hunger" date="1303308248"]"QtEndian":http://doc.qt.nokia.com/4.7/qtendian.html has some helper functions for endianess issues. A little known corner of Qt:-)[/quote]

So using this tasty little tidbit and because I already had my data into a QByteArray I wrote some functions in a untility class as follows;

@
qint16 ByteUtils::extractUInt16(const QByteArray& data, int offset, bool bigendian)
{
if (bigendian)
return qFromBigEndian<qint16>((uchar*)data.mid(offset, 0x2).data());
else
return qFromLittleEndian<qint16>((uchar*)data.mid(offset, 0x2).data());

}

qint32 ByteUtils::extractUInt32(const QByteArray& data, int offset, bool bigendian)
{
if (bigendian)
return qFromBigEndian<qint32>((uchar*)data.mid(offset, 0x4).data());
else
return qFromLittleEndian<qint32>((uchar*)data.mid(offset, 0x4).data());

}
@

works like a charm ;) thanks all

tobias.hunger

I'd just decide on one byte order to use in the file and stick with that. That avoids the bigendian bool and the if/else. Branches are always slow and avoiding them in a method that is called all the time is a good idea: Extracting ints from a data file smells like something that will happen rather often.

Why do you need the data.mid()? That constructs a temporary object that is not necessary. @data.constData() + offset@ should work just as well as @data.mid(offset, 2).data()@ without creating a temporary object.

I'd also try to use constData whenever possible. Using data() the way you do the compiler might think it needs to copy the temporary object you created since somebody might want to write into those bytes. Using constData is a very clear hint that this is not going to happen and the compiler can optimize the copy out. Of course it might be intelligent enough to do it for the data() case, too, but who wants to bet on the compiler being intelligent? ;-)

jnewing

[quote author="Tobias Hunger" date="1303314614"]I'd just decide on one byte order to use in the file and stick with that. That avoids the bigendian bool and the if/else. Branches are always slow and avoiding them in a method that is called all the time is a good idea: Extracting ints from a data file smells like something that will happen rather often.

Why do you need the data.mid()? That constructs a temporary object that is not necessary. @data.constData() + offset@ should work just as well as @data.mid(offset, 2).data()@ without creating a temporary object.

I'd also try to use constData whenever possible. Using data() the way you do the compiler might think it needs to copy the temporary object you created since somebody might want to write into those bytes. Using constData is a very clear hint that this is not going to happen and the compiler can optimize the copy out. Of course it might be intelligent enough to do it for the data() case, too, but who wants to bet on the compiler being intelligent? ;-)
[/quote]

I have no control over the file and it has both bigendian and littleendian values (just to make my life hard) as it was a pc game that was ported to xbox 360 :s

Also you're 100% right I'll use the data.constData() + offset as I have no real need for a temp object :) thanks again.