Does QVariant know when it has reached its memory bounds?



  • Hello everyone.

    I have a bit of a unique question, I think, as I wasn't able to find a satisfactory answer elsewhere, or find a way to formulate my problem that yielded good answers through google and QtProject.
    I'm using QDataStream to stream certain types to a file for persistence. This works fine and well, until I realize that the datamembers of these types are very likely to change in the future. I'd like to continue using QDataStream for this, however, since it's very fast and flexible in other areas.

    The scenarios are as follows:
    Suppose we have a project saved with a specific version number. This project contains a number of items that each have a QString somewhere in the middle of their data structure. When they get exported and imported, this string needs to be written and read in the same order, otherwise all hell breaks loose and you never know what you'll get when the bytes are converted to the proper type. If those QStrings are removed at a later time, for some reason, or a different datamember is added before them (and written/read at that position), then what would be gotten from those streams is undefined. There doesn't seem to be a clean fix for this, but I also don't require one as we decided to organize the types in such a manner that data subject to change is kept as late as possible and can be held as 'reserved' should they ever be removed.
    However, one problem that does need fixing it when datamembers are added to the ends of these structures. So, suppose a newer version of the software added a QString to the back of the aforementioned types and also attempts to read these from files saved by an older version of the software (backward compatibility is key!), the result is undefined. However, I attempted to encapsulate everything, including nested lists of other structures, into QVariants, and read/write those to and from the datastream. The result surprised me that the QVariant seemed to know when it was in danger of stampeding over the memory of another object in the structure.

    My question is, does QVariant know when the bytes allotted to it in a buffer end? If so, does it default construct members it cannot take from a buffer that, from its point of view, is empty? I did some tests and QString seems to always get initialized to "", while a double gets initialized to 0.0. So as far as I can tell, it seems to work. However, this code is crucial, so I want to understand what's happening before I commit to any one solution.

    I hope this question wasn't too long-winded and I apologize if it's already been answered elsewhere, but I couldn't find a good answer in any documentation or other question.

    Thank you in advance.


  • Moderators

    your problem can be handled easily by introducing a "magic number" and a version into your resulting binary. This version number will always have the same size (for example quint32).

    So when you readin your binary data you always check the magic number if the file is really your data type file. Right after that you read the version number which lets you know which types of data and in which order they come. So whenever your data structure changes you need to increment the version number. This lets you easily guarantee backward compatibility.

    For example you can specify different methods for reading the data versions:
    @
    int readDataFile()
    {
    QFile file("file.xxx");
    file.open(QIODevice::ReadOnly);
    QDataStream in(&file);

    // Read and check the header
    quint32 magic;
    in >> magic;
    if (magic != 0xA0B0C0D0) //your very own unique magic number code
    return XXX_BAD_FILE_FORMAT;

    // Read the version
    qint32 version;
    in >> version;
    if (version < 100)
    return XXX_BAD_FILE_TOO_OLD;
    if (version > 123)
    return XXX_BAD_FILE_TOO_NEW;

    switch( version )
    {
    case 100:
    return readDataV100(in, ...);
    break;
    case 101:
    return readDataV101(in, ...);
    break;
    }

    int readDataV100(QDataStream & --in, ...)
    {
    //do some version specific initialization?
    in.setVersion(QDataStream::Qt_3_2);

      //do read the data from the stream
     in >> ...;
    

    }
    @

    This is also a noticed in the "QDataStream docs":http://qt-project.org/doc/qt-4.8/qdatastream.html#versioning.



  • I read that documentation before, but, we want to avoid having multiple different reading versions for each different version of the file. This file type changes quite significantly with each software version due to the nature of the application. I've thought of this method before, but the problem is that, in our case, we'll end up with half a dozen different input versions that just linger in the code indefinitely. Hence why we've decided not to care about variables being removed from the middle of the data structure.


  • Moderators

    well that's not possible...you can't have one without the other ... at least without using pure magic.
    The better you think your structure through the more you could optimize and thus minimize your code to support multiple version.

    About what data structures are we talking exactly?

    If you only write nested QVariants into the data stream it should work. Then your program just needs to handle through the QVariant structure. Similiar like traversing XML file ... not a very pleasant work to do IMHO :) ... but working

    So get your data into QVariants and write a single QVariant at the end into the data stream should do it.



  • Yes, I know, it's choosing the lesser of two evils. It's not an easy decision to make, but I definitely understand where you're coming from, don't get me wrong. I'd also prefer to use a version and magic number to do some early checking and delegate the loading to the proper method. But at the same time, I don't want all sorts of messy input/output code clogging up the files just because old versions still need handling.

    As it stands, I'm only writing standard types and QVariants, but the interfaces hide all the nasty conversions from the user of the structure so they don't need to worry about what happens internally. It's not pleasant though, that's true.

    It's good to know QVariant keeps this into account, so we have two working solutions, we just need to decide on which one to go for.

    Any idea what the most common solution for backward compatibility is? Is it the first you suggested, with keeping 'old' load and save logic in the program?


  • Moderators

    [quote author="RenegadeVile" date="1385118868"]
    Any idea what the most common solution for backward compatibility is? Is it the first you suggested, with keeping 'old' load and save logic in the program?[/quote]
    i would say the first one i suggested IMHO. But it's basically just a matter of the developer's preference, since he has to maintain the code in the future.
    To keep the code clean and structured you could write plugins with your own interface.
    So your main program has the plugin routine (which most probably wont change very often) to find a plugin which supports the reading of the data at runtime.

    The plug-in interface could provide a method to ask if a certain version number is supported. If it returns true you can pass the data stream and let the plugin read the data from it. So you can easily drop support for a certain version by just removing the plugin if you want to.

    Theoretically (without knowing your business logic) when the plugin interface stays the same you can even add forward compatibility by just copying the plugin to your old application version.

    So it's a matter of what effort you do want to put into the solution to fit your needs the most.



  • The plug-in interface is a very interesting idea, I might look into it just for my own enjoyment; but it's currently not feasible for our application to put that much time into initially setting up such a system.
    However, you have convinced me to use a version number and proceed from there.

    Thank you for the helpful advice and answers.

    (I'd +1 everything, but I haven't the faintest where to do so)


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.