Copy constructor question: memcpy, QString/QByteArray

primem0ver

Another helpful piece of information:

In the above code, the "string" case is where the string is meant as a literal and should be preserved. In all other cases below that line, the string is meant to be discarded upon being compiled.

Asperamanca

Both QString and QByteArray only hold pointers to their data. So all you would copy would be pointers. If you want to go that low-level, you probably need to write your own PODs to hold a byte array or string, or find them in another library.

VRonin

valueStrings = &oldData.string().split(','); will not work. Just use QStringList's copy constructor, Qt uses lazy copying anyway so it's super cheap. I this case it probably even triggers the move constructor so even cheaper.
You are trying to reinvent std::variant. There are people way smarter than me whose job is to come up with the most efficient implementation of that class... I'd say use that, you can't go wrong.

primem0ver

@VRonin

I am not sure who's not following who here. Why wouldn't it work? The QStringList is only used in special cases where the string to be parsed should contain a numeric tuple such as 2d or 3d coordinates (e.g. "3, 4, 5"). Why would I use QStringList's copy constructor for this when I want to split the contents of the qstring returned by oldData.string() in order to parse the contained values into a tuple of integer or float values?
Except that QVariant takes twice as much memory to do the same job and I only need this for the aformentioned use case. I will not be holding anything more complex than a pointer or a number (or the QString data pointer as suggested by Asperamanca). My application will be doing a lot of script and XML parsing so I want to keep things as efficient and as compartmentalized as possible.

@Asperamanca said in Copy constructor question: memcpy, QString/QByteArray:

If you want to go that low-level, you probably need to write your own PODs to hold a byte array or string, or find them in another library.

Actually I have considered this option (for a few reasons) and am on the fence. The problem is that I am using a LOT of functionality from the Qt library and having to constantly convert between Qt and another library (even if it is my own) would end up being rather inefficient.

J.Hilk

@primem0ver said in Copy constructor question: memcpy, QString/QByteArray:

Except that QVariant takes twice as much memory to do the same job and I only need this for the aformentioned use case. I will not be holding anything more complex than a pointer or a number (or the QString data pointer as suggested by Asperamanca). My application will be doing a lot of script and XML parsing so I want to keep things as efficient and as compartmentalized as possible.

What @VRonin meant is not QVariant but std::variant

Those two are very different and rhe std version is a lot more lightweight than what Qt does.

primem0ver

@J.Hilk
Ooops... my bad. I should have payed attention to the std::

primem0ver

It looks like the std::variant may be a workable solution. I will see how it works. Thanks VRonin. Haven't really looked at the new stuff yet in C++ 17. I am still curious about your QStringList comment for the purpose of clarity.

VRonin

@primem0ver said in Copy constructor question: memcpy, QString/QByteArray:

I am not sure who's not following who here. Why wouldn't it work?

&oldData.string().split(','); returns the address of a temporary item. It will go out of scope on the following line. Your debugger might keep it alive so you don't notice it but it will come back to bite you in the backside. Only const references extend the lifetime of temporary variables. change it to QStringList valueStrings; and then valueStrings = oldData.string().split(','); It will use this constructor that is just 2 pointer assignments

primem0ver

@VRonin
The reason I used the pointer is because I didn't want to cause an unnecessary call to the default constructor. I guess it isn't that big of a deal but doesn't declaring QStringList valueStrings a priori call the default constructor and then again if the copy constructor is used? I suppose if I kept the pointer and use the copy constructor on the returned value, it would still use two constructors though... so I guess the default is more efficient. Does the compiler know not to initialize an unused instance of a variable?

VRonin

@primem0ver said in Copy constructor question: memcpy, QString/QByteArray:

The reason I used the pointer is because I didn't want to cause an unnecessary call to the default constructor.

Fair enough. QStringList* valueStrings; and then valueStrings = new QStringList(oldData.string().split(',')); and then remember to call delete valueStrings; at the end. or use std::unique_ptr. Hope it's clear you can't ever store the address of a temporary variable such as a return value

primem0ver

@VRonin: Yes. Thanks

I have changed the status back to unsolved because the std::variant has the same problem as QVariant. It uses too much memory space for a large amount of simple numeric data.

JKSH

@primem0ver Back to your original question:

You cannot put a QString/QByteArray in a union.
You can create a QString/QByteArray using new and store the pointer in a union, but you must then free the memory manually.

VRonin

@JKSH said in Copy constructor question: memcpy, QString/QByteArray:

but you must then free the memory manually

Which is normally a problem when using unions

It uses too much memory space for a large amount of simple numeric data

Can you elaborate on the order of magnitude?

JonB

@VRonin

It uses too much memory space for a large amount of simple numeric data

Can you elaborate on the order of magnitude?

https://stackoverflow.com/questions/45575892/why-is-sizeofstdvariant-the-same-size-as-a-struct-with-the-same-members

In practice, the OP's "simple numeric data" will be 8 bytes, plus an 8 byte overhead for the std::variant, == 16 bytes. Thus twice the size of the data he wishes to store. Whether he regards double this amount of storage as "too much memory space" is unknown. Assuming in some shape or form he wants a union plus a "flag" for the content type, he will be hard-pressed to reduce this....

primem0ver

I am making use of CRTP's, virtual methods, and static class data to make sure that the content type "flag" is not a necessary part of an instance of my variant ("union") class. This means that any class instance holding one data item will only require 8 bytes and still have the functionality of the other variant types that people have suggested. The only sacrifice is the time it takes to look up a virtual function which then calls a static function. Technically this allows me to create a custom "variant" class that only occupies 8 bytes; which is enough to hold a QString /QByteArray. Since QVariant and std:variant make use of what is essentially a union for their contents (and both allow QString/QByteArray), I am not certain why @JKSH says it cannot/should not be done.

fcarney

I read through the responses in this thread and I don't quite understand why a class system could not be used to achieve the same thing.

The base class could be virtual, contain no data, and only specify interaction methods with the data.
Each derived class could contain a maximum of 8 bytes of storage for whatever data type is required. Each class could easily be verified to not exceed this constraint by a simple sizeof check. The inheritance would then be limited to a maximum of 1 depth.

Types could be identified by an enum defined in the base class. Each class could have a method that is defined to return the enum value. You would just have to be diligent about not mixing up enum values between classes. Again these would be all defined in the base class so all classes would have access to the enums. Also, casting could be used to determine which class is being used as well.

Each class would be responsible for defining derived methods that interact with the data. These methods could accept and return variants that are then interpreted in the class for storage of its unique data.

Maybe some kind of explanation of the constraints might shed some light on better ways to approach this?

fcarney

Just a note. Many containers have some overhead over and above the size of the data they store.

So to get the size of the container is sizeof(<datatype>). However to get the size of the data stored in the container you have to be more creative:

QVector<qint32> qvecqint32;  // a typical container holding a qint32

//put some values in there
qint32 vals[] = {0,1,2,3,4,5,6,7};
for(auto val: vals){
        vecqint32.push_back(val);
        qvecqint32.push_back(val);
}

qInfo() << "vecqint32 sizeof:" << sizeof(qvecqint32);
qInfo() << "vecqint32:" << vecqint32.size() * sizeof(decltype(vecqint32)::value_type);

The output of this is:

//your code here
vecqint32 sizeof: 4
vecqint32: 32

So the container itself is 4 bytes with no data in it. Plus the data was added and it is 32 bytes. So 36 bytes. I have not tested, but I imagine QString and other containers may have some overhead as well.

primem0ver

@fcarney
I don't think you have really followed the purpose here at all. Why are you bringing up QVector??? (That has nothing to do with this conversation).

The idea is to have a basic container for basic (fundamental) data that does not exceed 8 bytes in size (the size it takes to store any of the types which have been discussed). QVariant has a size of 16 bytes; not 8. I already have a "working" class which will do this. My only concern is with the result of temporarily copying data in my custom variant class (whose goal is to only use enough space to contain the value data of a pointer, basic number, or a QString/QByteArray) in order to "compile" the data into a different format which is stored in the same space (turning the string "3, 4, 5" into a pointer to a 3D vector for example). I have given the code for that compile method in the 6th post of this thread (including the OP). Keep in mind that the only data that is passed on the stack (or as part of the "QString" class) is a pointer to the reference counted QString data.

Since both QString and QByteArray (which in this case is being used as a container for an ANSI string) are reference counted variables, I was wondering how temporarily copying the data when it contains one of these types would affect the reference count. (Hence the title and content of the OP)

@everyone:
I think the bottom line is that the only way this would be an issue is if I changed the copy and then assigned it to the original. Since I am not doing that (the point is to change it into another data type entirely), I don't think the reference count will be incorrectly represented except possibly when the copy goes out of scope. Is this true?

If it is, I can probably correct for it by copying the QString/QByteArray explicitly (since this is probably what happens in other variants when the copy constructor is used.

fcarney

I mentioned QVector because it is a container just like QString and QByteArrary. There is overhead:

QByteArray strdata; // empty string
qInfo() << "sizeof:" << sizeof(strdata); // overhead of a container
strdata = "01234567"; // 8 characters
qInfo() << "sizeof:" << sizeof(strdata); // same overhead, but data is now stored by container
qInfo() << "Actual data stored size:" << strdata.size() * sizeof(decltype(strdata)::value_type);
qInfo() << "Actual overhead is:" << strdata.size() * sizeof(decltype(strdata)::value_type) + sizeof(strdata);

Attempting to store 8 bytes in a QByteArrary ends up storing 12 bytes. The QVariant would only be used to return a generic value from a base class, not to store data.

JKSH

@primem0ver said in Copy constructor question: memcpy, QString/QByteArray:

Since QVariant and std:variant make use of what is essentially a union for their contents (and both allow QString/QByteArray)...

QVariant does not store QString/QByteArray in a union. It only stores numerics and pointers in a union: https://code.woboq.org/qt5/qtbase/src/corelib/kernel/qvariant.h.html#QVariant::Private::Data More specifically, QString gets stored in a QVariant::PrivateShared structure, and a pointer to this struct gets stored in the union.

I was wondering how temporarily copying the data when it contains one of these types would affect the reference count. (Hence the title and content of the OP)

memcpy()-ing a QString/QByteArray will increase the number of copies but won't increase the reference count.

I don't think the reference count will be incorrectly represented except possibly when the copy goes out of scope. Is this true?

(If there's only 1 copy) When the QString/QByteArray destructor is called on either the copy or the original, the shared string data will be deleted -- this means the other object will become something like a dangling pointer.

If it is, I can probably correct for it by copying the QString/QByteArray explicitly (since this is probably what happens in other variants when the copy constructor is used.

Always use the QString/QByteArray's copy constructor. Don't use memcpy().