SizeOf Struct giving false number.

ivan.todorovich · wrote on 14 Dec 2010, 16:10

Hi people,

I've encountered a problem that's driving me crazy.

Take a look at this example code:

@#include <QtCore/QCoreApplication>
#include <QDebug>

struct dbfFieldInfo {

char            name[11];
unsigned char   type;
quint32         displacement;
unsigned char   length;
unsigned char   precision;
unsigned char   flags;
quint32         nextid;
unsigned char   step;
char            reserved[8];

};

int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
qDebug() << "Struct SizeOf .. : " << sizeof(dbfFieldInfo);
qDebug() << "-------------------------";
qDebug() << "name[11] ....... : " << sizeof(dbfFieldInfo::name);
qDebug() << "type ........... : " << sizeof(dbfFieldInfo::type);
qDebug() << "displacement ... : " << sizeof(dbfFieldInfo::displacement);
qDebug() << "length ......... : " << sizeof(dbfFieldInfo::length);
qDebug() << "precision ...... : " << sizeof(dbfFieldInfo::precision);
qDebug() << "flags .......... : " << sizeof(dbfFieldInfo::flags);
qDebug() << "nextid ......... : " << sizeof(dbfFieldInfo::nextid);
qDebug() << "step ........... : " << sizeof(dbfFieldInfo::step);
qDebug() << "reserved[8] .... : " << sizeof(dbfFieldInfo::reserved);
qDebug() << "-------------------------";
qDebug() << "SUM ............ : " <<
sizeof(dbfFieldInfo::name)
+ sizeof(dbfFieldInfo::type)
+ sizeof(dbfFieldInfo::displacement)
+ sizeof(dbfFieldInfo::length)
+ sizeof(dbfFieldInfo::precision)
+ sizeof(dbfFieldInfo::flags)
+ sizeof(dbfFieldInfo::nextid)
+ sizeof(dbfFieldInfo::step)
+ sizeof(dbfFieldInfo::reserved);

return a.exec&#40;&#41;;

}@

Output:

@Struct SizeOf .. : 36

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
nextid ......... : 4
step ........... : 1
reserved[8] .... : 8

SUM ............ : 32@

Can anyone tell me why sizeof(dbfFieldInfo) isn't 32 ???

Moreover, I've tried to trace the problem and it seems this data member is causing it
@quint32 nextid;@

Look what happens if I remove it:

@#include <QtCore/QCoreApplication>
#include <QDebug>

struct dbfFieldInfo {

char            name[11];
unsigned char   type;
quint32         displacement;
unsigned char   length;
unsigned char   precision;
unsigned char   flags;
//quint32         nextid;
unsigned char   step;
char            reserved[8];

};

int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
qDebug() << "Struct SizeOf .. : " << sizeof(dbfFieldInfo);
qDebug() << "-------------------------";
qDebug() << "name[11] ....... : " << sizeof(dbfFieldInfo::name);
qDebug() << "type ........... : " << sizeof(dbfFieldInfo::type);
qDebug() << "displacement ... : " << sizeof(dbfFieldInfo::displacement);
qDebug() << "length ......... : " << sizeof(dbfFieldInfo::length);
qDebug() << "precision ...... : " << sizeof(dbfFieldInfo::precision);
qDebug() << "flags .......... : " << sizeof(dbfFieldInfo::flags);
//qDebug() << "nextid ......... : " << sizeof(dbfFieldInfo::nextid);
qDebug() << "step ........... : " << sizeof(dbfFieldInfo::step);
qDebug() << "reserved[8] .... : " << sizeof(dbfFieldInfo::reserved);
qDebug() << "-------------------------";
qDebug() << "SUM ............ : " <<
sizeof(dbfFieldInfo::name)
+ sizeof(dbfFieldInfo::type)
+ sizeof(dbfFieldInfo::displacement)
+ sizeof(dbfFieldInfo::length)
+ sizeof(dbfFieldInfo::precision)
+ sizeof(dbfFieldInfo::flags)
//+ sizeof(dbfFieldInfo::nextid)
+ sizeof(dbfFieldInfo::step)
+ sizeof(dbfFieldInfo::reserved);

return a.exec(&#41;;

}@

Output:

@Struct SizeOf .. : 28

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
step ........... : 1
reserved[8] .... : 8

SUM ............ : 28@

stukdev · wrote on 14 Dec 2010, 16:12

Sometimes the compiler optimize the struct and not give the correct size, try by using:
@
#pragma pack (1)
@
at the top of the file.

ivan.todorovich · wrote on 14 Dec 2010, 16:14

[quote author="stuk" date="1292343122"]Sometimes the compiler optimize the struct and not give the correct size, try by using a
#pragma pack 1 at the top of the file.[/quote]

same result :(

@#pragma pack 1

#include <QtCore/QCoreApplication>
#include <QDebug>

...@

Output:

@Struct SizeOf .. : 36

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
nextid ......... : 4
step ........... : 1
reserved[8] .... : 8

SUM ............ : 32@

stukdev · wrote on 14 Dec 2010, 16:15

Sorry i make mistake i use

@
#pragma pack (1)
@

giesbert · wrote on 14 Dec 2010, 16:20

Hi,

using #pragma pack has much influence on the time. It slows down the machine, as the processor can't work anymore on natural boundaries. So think about this, if you want to use #pragma pack. It reduces the size of your struct, as not all members are layed out on 32 bit borders (or whatever the machine has, you compile for) but that means, the processor has to jump between 32 bit and less bit boundaries.

Zlatomir · wrote on 14 Dec 2010, 16:21

"Recommended reading":http://en.wikipedia.org/wiki/Data_structure_alignment

And don't forget about that pack, especially if you build a library (or other way use the objects in other places that pack the data structure differently)

stukdev · wrote on 14 Dec 2010, 16:31

Yes, slow memory Vs slow speed.
I use #pragma pack when i deploy to AVR cpu because the memory is important ( < 256k ) and the speed that you lost is low significative.

[quote author="Gerolf Reinwardt" date="1292343651"]Hi,

using #pragma pack has much influence on the time. It slows down the machine, as the processor can't work anymore on natural boundaries. So think about this, if you want to use #pragma pack. It reduces the size of your struct, as not all members are layed out on 32 bit borders (or whatever the machine has, you compile for) but that means, the processor has to jump between 32 bit and less bit boundaries.[/quote]

ivan.todorovich · wrote on 14 Dec 2010, 16:35

Thanks for all the responses.

I now understand how struct's sizes work. And solved the problem in a different manner: changing the read function. Instead of reading sizeof(dbfFieldInfo) bytes, now I explicitly read 32 bytes.

tobias.hunger · wrote on 14 Dec 2010, 16:42

The compiler is free to add "holes" into any struct to speed up accessing individual members. So it will always place ints, etc. at proper word boundaries. Smaller data members will get "grouped" as there is little to gain by aligning them on word boundaries.

So this is how your data structure looks like:

@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
quint32 nextid; // align on word boundary, add 1 byte
unsigned char step; // one byte
char reserved[8]; // Align on word boundary again, so 3 bytes of padding needed.
@

Try changing it to this:
@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
unsigned char step; // one byte, total of 4 bytes!
quint32 nextid; // align on word boundary, no padding needed.
char reserved[8]; // Align on word boundary again, no padding needed.
@

Be aware that this layout might differ on a 64bit machine.

ivan.todorovich · wrote on 14 Dec 2010, 17:22

[quote author="Tobias Hunger" date="1292344923"]The compiler is free to add "holes" into any struct to speed up accessing individual members. So it will always place ints, etc. at proper word boundaries. Smaller data members will get "grouped" as there is little to gain by aligning them on word boundaries.

So this is how your data structure looks like:

@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
quint32 nextid; // align on word boundary, add 1 byte
unsigned char step; // one byte
char reserved[8]; // Align on word boundary again, so 3 bytes of padding needed.
@

Try changing it to this:
@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
unsigned char step; // one byte, total of 4 bytes!
quint32 nextid; // align on word boundary, no padding needed.
char reserved[8]; // Align on word boundary again, no padding needed.
@

Be aware that this layout might differ on a 64bit machine.[/quote]

Wow!
Really good explanation, thanks!

goetz · wrote on 14 Dec 2010, 18:25

[quote author="Tobias Hunger" date="1292344923"]The compiler is free to add "holes" into any struct to speed up accessing individual members. So it will always place ints, etc. at proper word boundaries. Smaller data members will get "grouped" as there is little to gain by aligning them on word boundaries.

So this is how your data structure looks like:

@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
quint32 nextid; // align on word boundary, add 1 byte
unsigned char step; // one byte
char reserved[8]; // Align on word boundary again, so 3 bytes of padding needed.
@
[/quote]

Sorry to nitpick a bit here :-)
At least on gcc (Mac), the last array is directly appended to the quint32 and the padding seems to be done at the end of the array:

@
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
dbfFieldInfo i;

qDebug() << "Struct SizeOf .. : " << sizeof(i)              << (quintptr)&i;
qDebug() << "-------------------------";

qDebug() << "member.......... :"
         << "size"
         << "address"
         << "expectedAddress"
         << "offset";

qDebug() << "name[11] ....... : "
         << sizeof(i.name)
         << (quintptr)&i.name;
qDebug() << "type ........... : "
         << sizeof(i.type)
         << (quintptr)&i.type
         << ((quintptr)&i.name + sizeof(i.name))
         << ((quintptr)&i.type - (quintptr)&i.name - sizeof(i.name));
qDebug() << "displacement ... : "
         << sizeof(i.displacement)
         << (quintptr)&i.displacement
         << ((quintptr)&i.type + sizeof(i.type))
         << ((quintptr)&i.displacement - (quintptr)&i.type - sizeof(i.type));
qDebug() << "length ......... : "
         << sizeof(i.length)
         << (quintptr)&i.length
         << ((quintptr)&i.displacement + sizeof(i.displacement))
         << ((quintptr)&i.length - (quintptr)&i.displacement - sizeof(i.displacement));
qDebug() << "precision ...... : "
         << sizeof(i.precision)
         << (quintptr)&i.precision
         << ((quintptr)&i.length + sizeof(i.length))
         << ((quintptr)&i.precision - (quintptr)&i.length - sizeof(i.length));
qDebug() << "flags .......... : "
         << sizeof(i.flags)
         << (quintptr)&i.flags
         << ((quintptr)&i.precision + sizeof(i.precision))
         << ((quintptr)&i.flags - (quintptr)&i.precision - sizeof(i.precision));
qDebug() << "nextid ......... : "
         << sizeof(i.nextid)
         << (quintptr)&i.nextid
         << ((quintptr)&i.flags + sizeof(i.flags))
         << ((quintptr)&i.nextid - (quintptr)&i.flags - sizeof(i.flags));
qDebug() << "step ........... : "
         << sizeof(i.step)
         << (quintptr)&i.step
         << ((quintptr)&i.nextid + sizeof(i.nextid))
         << ((quintptr)&i.step - (quintptr)&i.nextid - sizeof(i.nextid));
qDebug() << "reserved[8] .... : "
         << sizeof(i.reserved)
         << (quintptr)&i.reserved
         << ((quintptr)&i.step + sizeof(i.step))
         << ((quintptr)&i.reserved - (quintptr)&i.step - sizeof(i.step));

qDebug() << "-------------------------";
qDebug() << "SUM ............ : " <<
            sizeof(i.name)
            + sizeof(i.type)
            + sizeof(i.displacement)
            + sizeof(i.length)
            + sizeof(i.precision)
            + sizeof(i.flags)
            + sizeof(i.nextid)
            + sizeof(i.step)
            + sizeof(i.reserved);
return a.exec&#40;&#41;;

}

/*
Struct SizeOf .. : 36 3221223552

member.......... : size address expectedAddress offset
name[11] ....... : 11 3221223552
type ........... : 1 3221223563 3221223563 0
displacement ... : 4 3221223564 3221223564 0
length ......... : 1 3221223568 3221223568 0
precision ...... : 1 3221223569 3221223569 0
flags .......... : 1 3221223570 3221223570 0
nextid ......... : 4 3221223572 3221223571 1
step ........... : 1 3221223576 3221223576 0
reserved[8] .... : 8 3221223577 3221223577 0

SUM ............ : 32
*/
@

dangelog · wrote on 14 Dec 2010, 19:12

[quote author="Volker" date="1292351143"][quote author="Tobias Hunger" date="1292344923"]The compiler is free to add "holes" into any struct to speed up accessing individual members. So it will always place ints, etc. at proper word boundaries. Smaller data members will get "grouped" as there is little to gain by aligning them on word boundaries.

So this is how your data structure looks like:

@
char name[11]; // first 11 bytes, starts on word boundary
unsigned char type; // 12 byte, so word boundary here (on 32 bit, not necessarily on a 64 bit mashine!)
quint32 displacement; // placed on word boundary, no padding needed here
unsigned char length;
unsigned char precision;
unsigned char flags; // OK, 3 bytes for these members
quint32 nextid; // align on word boundary, add 1 byte
unsigned char step; // one byte
char reserved[8]; // Align on word boundary again, so 3 bytes of padding needed.
@
[/quote]

Sorry to nitpick a bit here :-)
At least on gcc (Mac), the last array is directly appended to the quint32 and the padding seems to be done at the end of the array:
[/quote]

That's what he meant: "nextid" is aligned on word boundary (and ends on the next one), then we have 9 chars, therefore we need 3 bytes of padding at the end of the struct (to pad the whole struct to a word boundary). If we consider also the byte added before the "nextid" field, we get the 4 bytes of difference between sizeof() and the sum of fields.

And by the way, you can use the offsetof() macro to pick up offsets inside a struct.

goetz · wrote on 14 Dec 2010, 19:25

[quote author="peppe" date="1292353920"]That's what he meant: "nextid" is aligned on word boundary (and ends on the next one), then we have 9 chars, therefore we need 3 bytes of padding at the end of the struct (to pad the whole struct to a word boundary). If we consider also the byte added before the "nextid" field, we get the 4 bytes of difference between sizeof() and the sum of fields.

And by the way, you can use the offsetof() macro to pick up offsets inside a struct.
[/quote]

Sorry, I misinterpreted it as padding between step and reserved to have reserved start on a word boundary - my fault. Good to know of offsetof() - I hope I will never have to use it myself in live code ;-)

SizeOf Struct giving false number.

@Struct SizeOf .. : 36

name[11] ....... : 11 type ........... : 1 displacement ... : 4 length ......... : 1 precision ...... : 1 flags .......... : 1 nextid ......... : 4 step ........... : 1 reserved[8] .... : 8

@Struct SizeOf .. : 28

name[11] ....... : 11 type ........... : 1 displacement ... : 4 length ......... : 1 precision ...... : 1 flags .......... : 1 step ........... : 1 reserved[8] .... : 8

@Struct SizeOf .. : 36

name[11] ....... : 11 type ........... : 1 displacement ... : 4 length ......... : 1 precision ...... : 1 flags .......... : 1 nextid ......... : 4 step ........... : 1 reserved[8] .... : 8

/* Struct SizeOf .. : 36 3221223552

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
nextid ......... : 4
step ........... : 1
reserved[8] .... : 8

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
step ........... : 1
reserved[8] .... : 8

name[11] ....... : 11
type ........... : 1
displacement ... : 4
length ......... : 1
precision ...... : 1
flags .......... : 1
nextid ......... : 4
step ........... : 1
reserved[8] .... : 8

/*
Struct SizeOf .. : 36 3221223552