Copy constructor question: memcpy, QString/QByteArray

primem0ver

@VRonin: Yes. Thanks

I have changed the status back to unsolved because the std::variant has the same problem as QVariant. It uses too much memory space for a large amount of simple numeric data.

JKSH

@primem0ver Back to your original question:

You cannot put a QString/QByteArray in a union.
You can create a QString/QByteArray using new and store the pointer in a union, but you must then free the memory manually.

VRonin

@JKSH said in Copy constructor question: memcpy, QString/QByteArray:

but you must then free the memory manually

Which is normally a problem when using unions

It uses too much memory space for a large amount of simple numeric data

Can you elaborate on the order of magnitude?

JonB

@VRonin

It uses too much memory space for a large amount of simple numeric data

Can you elaborate on the order of magnitude?

https://stackoverflow.com/questions/45575892/why-is-sizeofstdvariant-the-same-size-as-a-struct-with-the-same-members

In practice, the OP's "simple numeric data" will be 8 bytes, plus an 8 byte overhead for the std::variant, == 16 bytes. Thus twice the size of the data he wishes to store. Whether he regards double this amount of storage as "too much memory space" is unknown. Assuming in some shape or form he wants a union plus a "flag" for the content type, he will be hard-pressed to reduce this....

primem0ver

I am making use of CRTP's, virtual methods, and static class data to make sure that the content type "flag" is not a necessary part of an instance of my variant ("union") class. This means that any class instance holding one data item will only require 8 bytes and still have the functionality of the other variant types that people have suggested. The only sacrifice is the time it takes to look up a virtual function which then calls a static function. Technically this allows me to create a custom "variant" class that only occupies 8 bytes; which is enough to hold a QString /QByteArray. Since QVariant and std:variant make use of what is essentially a union for their contents (and both allow QString/QByteArray), I am not certain why @JKSH says it cannot/should not be done.

fcarney

I read through the responses in this thread and I don't quite understand why a class system could not be used to achieve the same thing.

The base class could be virtual, contain no data, and only specify interaction methods with the data.
Each derived class could contain a maximum of 8 bytes of storage for whatever data type is required. Each class could easily be verified to not exceed this constraint by a simple sizeof check. The inheritance would then be limited to a maximum of 1 depth.

Types could be identified by an enum defined in the base class. Each class could have a method that is defined to return the enum value. You would just have to be diligent about not mixing up enum values between classes. Again these would be all defined in the base class so all classes would have access to the enums. Also, casting could be used to determine which class is being used as well.

Each class would be responsible for defining derived methods that interact with the data. These methods could accept and return variants that are then interpreted in the class for storage of its unique data.

Maybe some kind of explanation of the constraints might shed some light on better ways to approach this?

fcarney

Just a note. Many containers have some overhead over and above the size of the data they store.

So to get the size of the container is sizeof(<datatype>). However to get the size of the data stored in the container you have to be more creative:

QVector<qint32> qvecqint32;  // a typical container holding a qint32

//put some values in there
qint32 vals[] = {0,1,2,3,4,5,6,7};
for(auto val: vals){
        vecqint32.push_back(val);
        qvecqint32.push_back(val);
}

qInfo() << "vecqint32 sizeof:" << sizeof(qvecqint32);
qInfo() << "vecqint32:" << vecqint32.size() * sizeof(decltype(vecqint32)::value_type);

The output of this is:

//your code here
vecqint32 sizeof: 4
vecqint32: 32

So the container itself is 4 bytes with no data in it. Plus the data was added and it is 32 bytes. So 36 bytes. I have not tested, but I imagine QString and other containers may have some overhead as well.

primem0ver

@fcarney
I don't think you have really followed the purpose here at all. Why are you bringing up QVector??? (That has nothing to do with this conversation).

The idea is to have a basic container for basic (fundamental) data that does not exceed 8 bytes in size (the size it takes to store any of the types which have been discussed). QVariant has a size of 16 bytes; not 8. I already have a "working" class which will do this. My only concern is with the result of temporarily copying data in my custom variant class (whose goal is to only use enough space to contain the value data of a pointer, basic number, or a QString/QByteArray) in order to "compile" the data into a different format which is stored in the same space (turning the string "3, 4, 5" into a pointer to a 3D vector for example). I have given the code for that compile method in the 6th post of this thread (including the OP). Keep in mind that the only data that is passed on the stack (or as part of the "QString" class) is a pointer to the reference counted QString data.

Since both QString and QByteArray (which in this case is being used as a container for an ANSI string) are reference counted variables, I was wondering how temporarily copying the data when it contains one of these types would affect the reference count. (Hence the title and content of the OP)

@everyone:
I think the bottom line is that the only way this would be an issue is if I changed the copy and then assigned it to the original. Since I am not doing that (the point is to change it into another data type entirely), I don't think the reference count will be incorrectly represented except possibly when the copy goes out of scope. Is this true?

If it is, I can probably correct for it by copying the QString/QByteArray explicitly (since this is probably what happens in other variants when the copy constructor is used.

fcarney

I mentioned QVector because it is a container just like QString and QByteArrary. There is overhead:

QByteArray strdata; // empty string
qInfo() << "sizeof:" << sizeof(strdata); // overhead of a container
strdata = "01234567"; // 8 characters
qInfo() << "sizeof:" << sizeof(strdata); // same overhead, but data is now stored by container
qInfo() << "Actual data stored size:" << strdata.size() * sizeof(decltype(strdata)::value_type);
qInfo() << "Actual overhead is:" << strdata.size() * sizeof(decltype(strdata)::value_type) + sizeof(strdata);

Attempting to store 8 bytes in a QByteArrary ends up storing 12 bytes. The QVariant would only be used to return a generic value from a base class, not to store data.

JKSH

@primem0ver said in Copy constructor question: memcpy, QString/QByteArray:

Since QVariant and std:variant make use of what is essentially a union for their contents (and both allow QString/QByteArray)...

QVariant does not store QString/QByteArray in a union. It only stores numerics and pointers in a union: https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qvariant.h.html#QVariant::Private::Data More specifically, QString gets stored in a QVariant::PrivateShared structure, and a pointer to this struct gets stored in the union.

I was wondering how temporarily copying the data when it contains one of these types would affect the reference count. (Hence the title and content of the OP)

memcpy()-ing a QString/QByteArray will increase the number of copies but won't increase the reference count.

I don't think the reference count will be incorrectly represented except possibly when the copy goes out of scope. Is this true?

(If there's only 1 copy) When the QString/QByteArray destructor is called on either the copy or the original, the shared string data will be deleted -- this means the other object will become something like a dangling pointer.

If it is, I can probably correct for it by copying the QString/QByteArray explicitly (since this is probably what happens in other variants when the copy constructor is used.

Always use the QString/QByteArray's copy constructor. Don't use memcpy().

primem0ver

@fcarney said in Copy constructor question: memcpy, QString/QByteArray:

Attempting to store 8 bytes in a QByteArrary ends up storing 12 bytes. The QVariant would only be used to return a generic value from a base class, not to store data.

Right. This is all I am trying to do. I am not trying to store the actual array in the 8 bytes. For anything but a basic value that requires 8 bytes I am storing a pointer to the object. The ultimate point is to "compile" the basic data stored in a tree structure (parsed from a text file or script) to create either a "factory" template, information/data to be used in a "factory" template, or pseudocode from a script. The application I am building is a platform that will host lots of "user" created content (assuming the prototype performs as imagined). So I need to do a lot of compartmentalized generic coding in order to accomplish the tasks for the myriad of use cases that it will be applied to.

@JKSH thanks for the clear explanation.