500 Variables in class how to get/set/store in json efficiently?

StudentScripter

@JonB What kind of model would be a good way of storing this? Currently im storing the data for each of my qgraphicsitems in the qgraphicsitem subclass. (i dont have 500 parameters yet but it will grow somewhere near that)

Currently im retrieving and setting the data like that from json on load:

    // Parse the JSON document
    QJsonParseError jsonError;
    QJsonDocument doc = QJsonDocument::fromJson(data, &jsonError);
    if (jsonError.error != QJsonParseError::NoError) {
        qDebug() << "Error parsing JSON: " << jsonError.errorString();
        return;
    }
    
    
    
    // Access the first object in the array, which contains the SceneID Counter
    QJsonObject objCounter = doc.array().at(0).toObject();
    setSceneItemCounter(objCounter["SceneID Counter"].toInt());


    // Iterate through each QJsonObject in the QJsonArray
    foreach (QJsonValue val, doc.array()) {
        QJsonObject obj = val.toObject();


        
        // Create a new ResizablePixmapItem with the data from the QJsonObject
        ResizablePixmapItem *mPixmap = new ResizablePixmapItem(QPixmap());

        mPixmap->setSceneID(obj["SceneID"].toInt());
        mPixmap->setControlName(obj["ControlName"].toString());
        mPixmap->setImageFilePath(obj["ImageFilePath"].toString());
        
        mPixmap->setPos(obj["posX"].toDouble(), obj["posY"].toDouble());
        mPixmap->setSelectorFrameBounds(QRectF(obj["x"].toDouble(), obj["y"].toDouble(), obj["width"].toDouble(), obj["height"].toDouble()));
        
        mPixmap->setFrameWidth(obj["FrameWidth"].toInt());
        mPixmap->setFrameHeight(obj["FrameHeight"].toInt());
        

        // Add the ResizablePixmapItem to the QGraphicsScene
        addItem(mPixmap);

        mPixmap->setPixmap(mPixmap->getImageFilePath());
        mPixmap->setFlags(QGraphicsItem::ItemIsMovable |QGraphicsItem::ItemIsSelectable);


        this->update();
    }

I store it on save like that:

    // Erstellen Sie ein QJsonArray
    QJsonArray array;
    
    //Speichert den Stand des SzeneItem Counters
    QJsonObject objC;
    objC["SceneID Counter"] = SceneItemCounter;
    array.append(objC);
    
    
    // Durchlaufen Sie jedes QGraphicsItem in Ihrer QGraphicsScene
    foreach (QGraphicsItem *item, items()) {
        if(item == GroupSelectionRect){
            //Überspringe das GroupSelectionRect
            continue;
        }
        // Casten Sie das QGraphicsItem zu einem ResizablePixmapItem
        ResizablePixmapItem *mPixmap = dynamic_cast<ResizablePixmapItem *>(item);

        if (mPixmap) {
            // Erstellen Sie ein QJsonObject für jedes QGraphicsItem
            QJsonObject obj;

            obj["SceneID"] = mPixmap->getSceneID();
            obj["ControlName"] = mPixmap->getControlName();
            obj["ImageFilePath"] = mPixmap->getImageFilePath();
            
            obj["posX"] = mPixmap->scenePos().x();
            obj["posY"] = mPixmap->scenePos().y();
            obj["x"] = mPixmap->selectorFrameBounds().x();
            obj["y"] = mPixmap->selectorFrameBounds().y();
            obj["width"] = mPixmap->selectorFrameBounds().width();
            obj["height"] = mPixmap->selectorFrameBounds().height();
    
            obj["FrameWidth"] = mPixmap->getFrameWidth();
            obj["FrameHeight"] = mPixmap->getFrameHeight();
            

            // Fügen Sie das QJsonObject zum QJsonArray hinzu
            array.append(obj);
        }
    }

    // Erstellen Sie ein QJsonDocument mit dem QJsonArray
    QJsonDocument doc(array);

JonB

@StudentScripter
"Model" really means any kind of model, where all your variable values are stored as a collection (so they can be iterated, and saved/restored as a whole) and can be accessed somehow individually so that you know what each one means. It could be, say, a row model with a mapping between row numbers and what data is stored there, or your QMap where you can index by a key string.

What you show is OK if you want to use it. You have to put into code the variable/value name as a string and which C++ variable or function call you want to get/set it. You do that explicitly, once when you are saving and once when you are restoring.

If you wish, you can make it a bit nicer by writing a getter and setter for each "variable" which maps to/from your map.

int Mapper::sceneID() const { return obj["SceneID"].toInt(); }
void Mapper::setSceneID(int id) { obj["SceneID"] = id; }
...

so now you are using the serializable map as your data store instead of individual variables, but presenting a getter/setter interface so you can code as though they were variables.

If you are saving/restoring all the properties of widgets or graphics items you show then, yes, your number of items is going to get "large". Usually people find a way to provide some kind of "model" for the "back-end" from which the "front-end" can be reconstructed. But if you really need to save/restore frame positions, bounds, widths, heights etc. then so be it. You can reduce the number of lines of code if you write "helper" functions to convert between, say, a QRect (which can be stored in a QVariant so OK for your QMap) for selectorFrameBounds() and a QJsonObject with keys for the members, and re-use that.

Don't forget there is QJsonObject QJsonObject::fromVariantMap(const QVariantMap &map) and QVariantMap QJsonObject::toVariantMap() const to convert between JSON objects and a QMap of QVariants in one go.

SGaist

Hi,

Do you mean your JSON is going to be a single dictionary with 500 entries or will it rather be a dict/list of dicts ?

StudentScripter

@SGaist every graphicsitem has its own json array where all these values are stored for this item.

JonB

@StudentScripter I think you mean JSON object not array.

StudentScripter

@JonB Well yeah, one entry for every graphicsitems holding all up to 500 values.

So for 1000 items it would be 500.000 entrys inside the json document. But i guess that shouldn't be that big of a problem as every variable written is just a few bytes. Accumulating to around 24 megabyte if you really have thousand graphicsitems and only longer strings.

JonB

@StudentScripter
It may be OK but why do you have so much data to serialize? Why do you have or need 500 values for each graphics item, and all need to be persisted?

If you are doing something like Serializing QGraphicsScene contents, my feeling is like the response there saying

Serializing QGraphicsItem is not a good idea. A QGrahpicsScene is supposed to represent your data. Instead of serializing the representation, it's better to serialize the data/model of your application.

There might be some hints in other posts there if you do wish to serialize graphics objects nonetheless. Or others if you Google for qgrapchicsscene serialize if that is what you are trying to achieve.

StudentScripter

@JonB Yes all most of the data on which interfaces/nodes to show are stored on a per item basis as each item can have a different set of nodes with completely different propertys. So when you click on a item i have to update the interface to match that. Hope that makes sense.

JonB

@StudentScripter Your post crossed with my last. Just take on board the

Serializing QGraphicsItem is not a good idea. A QGrahpicsScene is supposed to represent your data. Instead of serializing the representation, it's better to serialize the data/model of your application.

But only you know what your application needs.

StudentScripter

@JonB Hm i see but i really wouldn't know how else to serialize the data. As said it surely needs to be seperated so i have data for the interface according to which item is clicked. I surely could store most of the data in a seperate class, only providing the id of the graphicsitem the data belongs to.

Or what else would you maybe suggest? Thanks for all the good hints along my way, wouldn't know what i would do without the help of yours, SGaist and PI45m4. <3 Really appreciate that as not that experienced programmer.

I see you asked something like that some time ago: https://forum.qt.io/topic/109305/serializing-qgraphicsscene-and-json/8

JonB

@StudentScripter said in 500 Variables in class how to get/set/store in json efficiently?:

I see you asked something like that some time ago: https://forum.qt.io/topic/109305/serializing-qgraphicsscene-and-json/8

Yeah, that was in my youth ;) I had no choice there, it was to do with Python JSON handling and I was working for someone else, not my own stuff.

I can't really say more than I have. I'm not sure how you accumulate 500 different items/variables of information to associate with up to 1,000 items. How did this data get there? How are these graphics items created? Did someone sit and create them or did they come from some other underlying data model? Why do you have to look at a pixmap and save information about its "frame"? But like I said, we are not going to understand or go through all your design decisions. You may be right and for your needs this is all reasonable and necessary data.

SGaist

Well, 500 properties per QGraphicsItem does sound like quite a lot.

As @JonB suggested: are you sure you are really needing all of them ?

You should take a step back and start by doing some drawing and design to have a clear picture of what the data you need are. What makes sense of keeping, do you really need to load them all within every item ? And also, do they all make sense together ?

As an example, if your item represent some control in a system, the properties of that control will have nothing to do with for example the position of the item in the scene, hence it would make little sense to smoosh them together.

You might also want to ensure that JSON is the correct data structure you need. Maybe a simple SQLite database might be more suited. Or maybe multiple JSON files rather than one super massive file.

StudentScripter

@SGaist Well thanks for the good hints:

Yes i need all these variables, as every variable stores the data of a slider, button, lineedit for the specific item
Of course i could seperate these data out to an extra subclass, but what would it be worth? I still would have to store this data and provide a relation to the item somehow on which the data on the interface needs to change.
May a question: what is faster/better:
.......1. when clicking on a item reading 500 values from the json file for that item live and setting them to the slidersOR
.......2. reading all 500 values * (at maximum) 1000 items at startup from the file and saving them in the subclasses variables so when the item is clicked it just reads from the already set variables
I could have a json file for each item with 500 values but i guess having 1000 json files wouldn't be that great either. So i guess i stick with 1 big file?
Regarding SQLite i haven't used it before and from what i watched i do not exactly know/understand how it could save me time/would be more suitable than a json file i can relatively easy dump all these data into. But may enlighten me. Im always eager to learn. :D

Y'all have a nice sunday.

JonB

@StudentScripter said in 500 Variables in class how to get/set/store in json efficiently?:

Yes i need all these variables, as every variable stores the data of a slider, button, lineedit for the specific item

We have tried to explain: you are supposed/better to have a model for this data, and be saving/restoring that rather than the data of each UI widget. For example, where does the value of a slider or the text of a line edit come from? That should really be a model. I think you have none and are storing the values of your UI widgets as your only "model" instead. Anyway we have said this, it is up to you.

May a question: what is faster/better:

.......1. when clicking on a item reading 500 values from the json file for that item live and setting them to the slidersOR

.......2. reading all 500 values * (at maximum) 1000 items at startup from the file and saving them in the subclasses variables so when the item is clicked it just reads from the already set variables

A JSON file is no kind of "random access" database (unlike, say, SQLite). You cannot read parts of it a time, you have to read the whole thing and let JSON parse it. So your #1 is not possible, you cannot "reading 500 values from the json file for that item live". I imagine the most time, whatever that is, will be reading & parsing the 500 x 1000 item's-worth of text from the JSON file, rather than the time then taken to "transfer" those as settings to the widgets.

So you will need to go for #2, in some shape or form. Certainly read the whole file. Then either get the "transfer" done at that point for all values to all widgets, or if that takes too long leave the data in the in-memory JSON structure and go access an individual widget's settings the first time you need it. That can be done efficiently in-memory since your JSON object hierarchy can be indexed by key or index (whichever way you chose to store it).

StudentScripter

@JonB Well thanks i don't quite get what you mean exactly with model?

So SQLite may would be a better option probably if i want to save data for 500 sliders, buttons on a per item basis.

JonB

@StudentScripter
A database (such as SQLite) might be a better choice for half a million data items. And more so if the quantity grows. You can access any item directly. You can alter one item without having to re-save all the other items as you do for a JSON file.

Having said that, computers are fast and have lots of memory these days. JSON for 0.5M items might be fine for the way you are going to use it. You seem to know where you are with JSON, it's quite a bit of learning to change over to a database. If you want to learn by all means go ahead. If you want to just proceed with your app try it with the JSON approach you have and see whether speed (or memory) is an issue.

A big factor is what you might want to do with the data outside of you and your application. If you have requirements there, e.g. for interfacing with other uses, that is a different matter and may dictate what you use.

StudentScripter

@JonB Thank you very much. Well i guess i do not exactly need SQLite but it seems like a cool thing to learn. As its a personal project i can take the time. :D And also who doesn't like over optimised good running programs.

Im not planning to do anything with the data outside my application. So yeah i guess json would be fine here too, but as said i like to learn new stuff. Have a nice evening.

SGaist

Then I would insist on taking the time to design things properly even for a personal project.

As @JonB wrote, you seem to want to dump quite a lot of stuff that may or may not be related to your items. Hence the first step is to properly model your stuff and then decide which data backend is the best fit.

StudentScripter

@SGaist @JonB So just wanted to add onto this, to get a short feedback on whether im on the right track or not.
This is how i create my database as a test. Is that a good way or is there a better practice on dynamically creating a database with SQLLite? :) Have a nice day.



MainWindow::MainWindow(QWidget *parent)
    : QMainWindow(parent)
{


    QSqlDatabase m_db = QSqlDatabase::addDatabase("QSQLITE");
    m_db.setDatabaseName("C:/Users/Musik/Desktop/TestSQLPATH/database.db");

    if (!m_db.open()) {
        qDebug() << "Error: connection with database failed";
    } else {
        qDebug() << "Database: connection ok";
    }

    // Sample QVariantMap (replace this with your actual QVariantMap)
    QVariantMap dataMap;
    dataMap["name"] = "John";
    dataMap["age"] = 30;
    dataMap["city"] = "New York";
    dataMap["son"] = "Paul";

      QString tableName = "MyCustomTableName";  // Table name

    // Drop the table if it exists
    if (m_db.tables().contains(tableName)) {
        QSqlQuery dropQuery;
        if (!dropQuery.exec("DROP TABLE " + tableName)) {
            qDebug() << "Error dropping table:" << dropQuery.lastError();
        }
    }



    if (m_db.isOpen()) {  // Überprüfe, ob die Datenbank geöffnet ist



        QString createString = "CREATE TABLE " + tableName + "(";
        QString insertString1 = "INSERT INTO " + tableName + "(";
        QString insertString2 = " VALUES (";

        for (auto it = dataMap.begin(); it != dataMap.end(); ++it) {
            const QVariant& value = it.value();
            QString mykey = it.key();

            if (value.typeId() == QMetaType::Type::QString) {
                createString += mykey + " TEXT";
            } else if (value.typeId() == QMetaType::Type::Int) {
                createString += mykey + " INTEGER";
            } else if (value.typeId() == QMetaType::Type::Double) {
                createString += mykey + " DOUBLE";
            } // Add more types as needed



            insertString1 += mykey;
            insertString2 += ":" + mykey;



            if (it != --dataMap.end()) {
                createString += ", "; // Add comma if not the last element
                insertString1 += ", ";
                insertString2 += ", ";
            }

            qDebug() << "Key:" << it.key() << ", Value:" << it.value();
        }

        createString += ")";
        insertString1 += ")";
        insertString2 += ")";
        QString insertfullstring = insertString1 += insertString2;
        qDebug() << createString << insertfullstring;




        // Überprüfe, ob die Tabelle existiert, bevor du einfügen
        if (!m_db.tables().contains(tableName)) {
          QSqlQuery createQuery;

            if (!createQuery.exec(createString)) {
                qDebug() << "Error creating table:" << createQuery.lastError();
            }
        }



        QSqlQuery query;
        query.prepare(insertfullstring);


        bindValuesToQuery(query, dataMap);



        if (query.exec()) {
            qDebug() << "Person hinzugefügt";
        } else {
            qDebug() << "Fehler beim Hinzufügen einer Person:"
                     << query.lastError();
        }

   m_db.close();  // Close the database when done
    } else {
        qDebug() << "Datenbank ist nicht geöffnet.";
    }



}

MainWindow::~MainWindow() {}

void MainWindow::bindValuesToQuery(QSqlQuery &query, const QVariantMap &map)
{
    for (auto it = map.begin(); it != map.end(); ++it) {
        const QVariant& value = it.value();
        QString key = it.key();

        query.bindValue(":" +key, value);

    }
}

JonB

@StudentScripter
Your code itself is OK per se.

However, I am not at all sure you have thought through the content/architecture. You dynamically create a table to correspond to the column type in one QVariantMap. I guess that will be OK to store multiple rows if your have an array/list of QVariantMaps where each & every item/row has exactly the same number of columns holding values of the same type in the same order. But useless if that is not the case, one item/row has any kind of different data. Only you know whether that is indeed the case.

Without a lot of thought/planning a database does not have the flexibility that JSON has for an arbitrary number of properties/columns, possibly of varying data types. It is more suited when the number of columns is known in advance and each column will have a known, fixed data type.

I did suggest earlier that, given where you were at and your knowledge, you might test whether the JSON file worked OK/fast enough to stick with that.

Both @SGaist & I suggested that usually you do not try to save/restore visual widget attributes. Rather you have some model behind what the UI shows and that is known and can be serialized/deserialized. Your 1,000 items with up to 500 attributes each must "come from somewhere", it does not pop up in a vacuum. Something told you to create a pixmap, set its position, alter its frame size/bounds, etc. That is supposed to be your "model", from which if you save you can re-create corresponding widgets with properties.

Also, if you have 500 arbitrary variables to save as columns in each row of a database that is not great. SQLite may or may not let you create a table with 500 columns, I don't know, but it's probably not a good architecture.

I have a feeling you are now going to be asking all sorts of questions about databases. @SGaist may wish to answer further, but for my part I feel I must leave you to it. There is a lot to learn about databases, particularly on the architecture/design/concept side, if you are going to start using them.