500 Variables in class how to get/set/store in json efficiently?

StudentScripter

@JonB Well yeah, one entry for every graphicsitems holding all up to 500 values.

So for 1000 items it would be 500.000 entrys inside the json document. But i guess that shouldn't be that big of a problem as every variable written is just a few bytes. Accumulating to around 24 megabyte if you really have thousand graphicsitems and only longer strings.

JonB

@StudentScripter
It may be OK but why do you have so much data to serialize? Why do you have or need 500 values for each graphics item, and all need to be persisted?

If you are doing something like Serializing QGraphicsScene contents, my feeling is like the response there saying

Serializing QGraphicsItem is not a good idea. A QGrahpicsScene is supposed to represent your data. Instead of serializing the representation, it's better to serialize the data/model of your application.

There might be some hints in other posts there if you do wish to serialize graphics objects nonetheless. Or others if you Google for qgrapchicsscene serialize if that is what you are trying to achieve.

StudentScripter

@JonB Yes all most of the data on which interfaces/nodes to show are stored on a per item basis as each item can have a different set of nodes with completely different propertys. So when you click on a item i have to update the interface to match that. Hope that makes sense.

JonB

@StudentScripter Your post crossed with my last. Just take on board the

Serializing QGraphicsItem is not a good idea. A QGrahpicsScene is supposed to represent your data. Instead of serializing the representation, it's better to serialize the data/model of your application.

But only you know what your application needs.

StudentScripter

@JonB Hm i see but i really wouldn't know how else to serialize the data. As said it surely needs to be seperated so i have data for the interface according to which item is clicked. I surely could store most of the data in a seperate class, only providing the id of the graphicsitem the data belongs to.

Or what else would you maybe suggest? Thanks for all the good hints along my way, wouldn't know what i would do without the help of yours, SGaist and PI45m4. <3 Really appreciate that as not that experienced programmer.

I see you asked something like that some time ago: https://forum.qt.io/topic/109305/serializing-qgraphicsscene-and-json/8

JonB

@StudentScripter said in 500 Variables in class how to get/set/store in json efficiently?:

I see you asked something like that some time ago: https://forum.qt.io/topic/109305/serializing-qgraphicsscene-and-json/8

Yeah, that was in my youth ;) I had no choice there, it was to do with Python JSON handling and I was working for someone else, not my own stuff.

I can't really say more than I have. I'm not sure how you accumulate 500 different items/variables of information to associate with up to 1,000 items. How did this data get there? How are these graphics items created? Did someone sit and create them or did they come from some other underlying data model? Why do you have to look at a pixmap and save information about its "frame"? But like I said, we are not going to understand or go through all your design decisions. You may be right and for your needs this is all reasonable and necessary data.

SGaist

Well, 500 properties per QGraphicsItem does sound like quite a lot.

As @JonB suggested: are you sure you are really needing all of them ?

You should take a step back and start by doing some drawing and design to have a clear picture of what the data you need are. What makes sense of keeping, do you really need to load them all within every item ? And also, do they all make sense together ?

As an example, if your item represent some control in a system, the properties of that control will have nothing to do with for example the position of the item in the scene, hence it would make little sense to smoosh them together.

You might also want to ensure that JSON is the correct data structure you need. Maybe a simple SQLite database might be more suited. Or maybe multiple JSON files rather than one super massive file.

StudentScripter

@SGaist Well thanks for the good hints:

Yes i need all these variables, as every variable stores the data of a slider, button, lineedit for the specific item
Of course i could seperate these data out to an extra subclass, but what would it be worth? I still would have to store this data and provide a relation to the item somehow on which the data on the interface needs to change.
May a question: what is faster/better:
.......1. when clicking on a item reading 500 values from the json file for that item live and setting them to the slidersOR
.......2. reading all 500 values * (at maximum) 1000 items at startup from the file and saving them in the subclasses variables so when the item is clicked it just reads from the already set variables
I could have a json file for each item with 500 values but i guess having 1000 json files wouldn't be that great either. So i guess i stick with 1 big file?
Regarding SQLite i haven't used it before and from what i watched i do not exactly know/understand how it could save me time/would be more suitable than a json file i can relatively easy dump all these data into. But may enlighten me. Im always eager to learn. :D

Y'all have a nice sunday.

JonB

@StudentScripter said in 500 Variables in class how to get/set/store in json efficiently?:

Yes i need all these variables, as every variable stores the data of a slider, button, lineedit for the specific item

We have tried to explain: you are supposed/better to have a model for this data, and be saving/restoring that rather than the data of each UI widget. For example, where does the value of a slider or the text of a line edit come from? That should really be a model. I think you have none and are storing the values of your UI widgets as your only "model" instead. Anyway we have said this, it is up to you.

May a question: what is faster/better:

.......1. when clicking on a item reading 500 values from the json file for that item live and setting them to the slidersOR

.......2. reading all 500 values * (at maximum) 1000 items at startup from the file and saving them in the subclasses variables so when the item is clicked it just reads from the already set variables

A JSON file is no kind of "random access" database (unlike, say, SQLite). You cannot read parts of it a time, you have to read the whole thing and let JSON parse it. So your #1 is not possible, you cannot "reading 500 values from the json file for that item live". I imagine the most time, whatever that is, will be reading & parsing the 500 x 1000 item's-worth of text from the JSON file, rather than the time then taken to "transfer" those as settings to the widgets.

So you will need to go for #2, in some shape or form. Certainly read the whole file. Then either get the "transfer" done at that point for all values to all widgets, or if that takes too long leave the data in the in-memory JSON structure and go access an individual widget's settings the first time you need it. That can be done efficiently in-memory since your JSON object hierarchy can be indexed by key or index (whichever way you chose to store it).

StudentScripter

@JonB Well thanks i don't quite get what you mean exactly with model?

So SQLite may would be a better option probably if i want to save data for 500 sliders, buttons on a per item basis.

JonB

@StudentScripter
A database (such as SQLite) might be a better choice for half a million data items. And more so if the quantity grows. You can access any item directly. You can alter one item without having to re-save all the other items as you do for a JSON file.

Having said that, computers are fast and have lots of memory these days. JSON for 0.5M items might be fine for the way you are going to use it. You seem to know where you are with JSON, it's quite a bit of learning to change over to a database. If you want to learn by all means go ahead. If you want to just proceed with your app try it with the JSON approach you have and see whether speed (or memory) is an issue.

A big factor is what you might want to do with the data outside of you and your application. If you have requirements there, e.g. for interfacing with other uses, that is a different matter and may dictate what you use.

StudentScripter

@JonB Thank you very much. Well i guess i do not exactly need SQLite but it seems like a cool thing to learn. As its a personal project i can take the time. :D And also who doesn't like over optimised good running programs.

Im not planning to do anything with the data outside my application. So yeah i guess json would be fine here too, but as said i like to learn new stuff. Have a nice evening.

SGaist

Then I would insist on taking the time to design things properly even for a personal project.

As @JonB wrote, you seem to want to dump quite a lot of stuff that may or may not be related to your items. Hence the first step is to properly model your stuff and then decide which data backend is the best fit.

StudentScripter

@SGaist @JonB So just wanted to add onto this, to get a short feedback on whether im on the right track or not.
This is how i create my database as a test. Is that a good way or is there a better practice on dynamically creating a database with SQLLite? :) Have a nice day.



MainWindow::MainWindow(QWidget *parent)
    : QMainWindow(parent)
{


    QSqlDatabase m_db = QSqlDatabase::addDatabase("QSQLITE");
    m_db.setDatabaseName("C:/Users/Musik/Desktop/TestSQLPATH/database.db");

    if (!m_db.open()) {
        qDebug() << "Error: connection with database failed";
    } else {
        qDebug() << "Database: connection ok";
    }

    // Sample QVariantMap (replace this with your actual QVariantMap)
    QVariantMap dataMap;
    dataMap["name"] = "John";
    dataMap["age"] = 30;
    dataMap["city"] = "New York";
    dataMap["son"] = "Paul";

      QString tableName = "MyCustomTableName";  // Table name

    // Drop the table if it exists
    if (m_db.tables().contains(tableName)) {
        QSqlQuery dropQuery;
        if (!dropQuery.exec("DROP TABLE " + tableName)) {
            qDebug() << "Error dropping table:" << dropQuery.lastError();
        }
    }



    if (m_db.isOpen()) {  // Überprüfe, ob die Datenbank geöffnet ist



        QString createString = "CREATE TABLE " + tableName + "(";
        QString insertString1 = "INSERT INTO " + tableName + "(";
        QString insertString2 = " VALUES (";

        for (auto it = dataMap.begin(); it != dataMap.end(); ++it) {
            const QVariant& value = it.value();
            QString mykey = it.key();

            if (value.typeId() == QMetaType::Type::QString) {
                createString += mykey + " TEXT";
            } else if (value.typeId() == QMetaType::Type::Int) {
                createString += mykey + " INTEGER";
            } else if (value.typeId() == QMetaType::Type::Double) {
                createString += mykey + " DOUBLE";
            } // Add more types as needed



            insertString1 += mykey;
            insertString2 += ":" + mykey;



            if (it != --dataMap.end()) {
                createString += ", "; // Add comma if not the last element
                insertString1 += ", ";
                insertString2 += ", ";
            }

            qDebug() << "Key:" << it.key() << ", Value:" << it.value();
        }

        createString += ")";
        insertString1 += ")";
        insertString2 += ")";
        QString insertfullstring = insertString1 += insertString2;
        qDebug() << createString << insertfullstring;




        // Überprüfe, ob die Tabelle existiert, bevor du einfügen
        if (!m_db.tables().contains(tableName)) {
          QSqlQuery createQuery;

            if (!createQuery.exec(createString)) {
                qDebug() << "Error creating table:" << createQuery.lastError();
            }
        }



        QSqlQuery query;
        query.prepare(insertfullstring);


        bindValuesToQuery(query, dataMap);



        if (query.exec()) {
            qDebug() << "Person hinzugefügt";
        } else {
            qDebug() << "Fehler beim Hinzufügen einer Person:"
                     << query.lastError();
        }

   m_db.close();  // Close the database when done
    } else {
        qDebug() << "Datenbank ist nicht geöffnet.";
    }



}

MainWindow::~MainWindow() {}

void MainWindow::bindValuesToQuery(QSqlQuery &query, const QVariantMap &map)
{
    for (auto it = map.begin(); it != map.end(); ++it) {
        const QVariant& value = it.value();
        QString key = it.key();

        query.bindValue(":" +key, value);

    }
}

JonB

@StudentScripter
Your code itself is OK per se.

However, I am not at all sure you have thought through the content/architecture. You dynamically create a table to correspond to the column type in one QVariantMap. I guess that will be OK to store multiple rows if your have an array/list of QVariantMaps where each & every item/row has exactly the same number of columns holding values of the same type in the same order. But useless if that is not the case, one item/row has any kind of different data. Only you know whether that is indeed the case.

Without a lot of thought/planning a database does not have the flexibility that JSON has for an arbitrary number of properties/columns, possibly of varying data types. It is more suited when the number of columns is known in advance and each column will have a known, fixed data type.

I did suggest earlier that, given where you were at and your knowledge, you might test whether the JSON file worked OK/fast enough to stick with that.

Both @SGaist & I suggested that usually you do not try to save/restore visual widget attributes. Rather you have some model behind what the UI shows and that is known and can be serialized/deserialized. Your 1,000 items with up to 500 attributes each must "come from somewhere", it does not pop up in a vacuum. Something told you to create a pixmap, set its position, alter its frame size/bounds, etc. That is supposed to be your "model", from which if you save you can re-create corresponding widgets with properties.

Also, if you have 500 arbitrary variables to save as columns in each row of a database that is not great. SQLite may or may not let you create a table with 500 columns, I don't know, but it's probably not a good architecture.

I have a feeling you are now going to be asking all sorts of questions about databases. @SGaist may wish to answer further, but for my part I feel I must leave you to it. There is a lot to learn about databases, particularly on the architecture/design/concept side, if you are going to start using them.

SGaist

@StudentScripter please take a step back.

You seem to want to use SQLite as a drop-in replacement of your original JSON structure which is the wrong move.

As @JonB and I suggested on several occasions now: get back to the drawing board and do some scribbling to describe your your data, application, model the data itself, properly separate the business logic from the UI.
Without a clearer picture it's just going to be technology thrown around without purpose.

Even if it's only for learning purposes, getting the design first before coding is a crucial step.

StudentScripter

@SGaist @JonB
Well thanks for leading me the way. I really want to improve instead of annoy and so tried to listen what you guys where telling. But I wanted to confirm that i got it the right way now.

So here is what i do now:

I divided my GUI into smaller groups (classes), so every group consist of only 10-15 widgets
For every group on runtime i store the values into a map. So every map holds 10-15 values with a specific key per value.
i want to create a different SQLLite table for each group, so every table has around 10-15 columns --> i would than transfer the last values of the maps to the corresponding table [I somehow want to set as row the id of my qgraphicsitems, cause as said the data does not directly belong to the graphicsitem but has to be shown when the specific item is selected]
So everytime when i select a different graphicsitem the data of the old graphicsitem is saved/transfered into the correct rows of the tables and the data of the now selected graphicsitem gets read from the table and stored in the map (or maybe 2 different maps 1 for save one for load)

Well so these are my new thoughts on a so i think way better structural design of my project. But let me know what you think. Im always open for further suggestions and corrections. :)

Thanks for the time and have a nice evening.

SGaist

@StudentScripter there's still one crucial information that is missing: what are you modeling ? What is it that requires 10 to 15 widgets to manage ? Based on your initial numbers (500 * 1000), it means you might be trying to create around 50000 widgets in your application.

StudentScripter

@SGaist Well thats may a misunderstanding. These where approximations on the values i have to store, but in fact there will be only around 500 widgets that actually display the data. Not all 50000 values are needed at once only around 500 at a time.

As said i graphicsitems and when one item is clicked a specific set of 500 values has to be displayed in these widgets (lineedits, spinboxes...).

I divided as the interface into sections so around 15 of these widgets have there own class and sql tabel: (just a quick sketch in word on how i imagine it, all example no real data)

As you can see the items decode/are the key to retrieve the data but besides that they don't have a connection to the data. It's just that i want to show this data when a certain item is clicked.
So for example with 9 different widget groups, 1000 graphicsitems, and 15 Widgets per widget group i would have:

15 Tables with 1000 rows (filled with the graphicsitem ids) and 15 columns each.

Hope that helps to understand what i want to do. Would that be a good way to approach that or not? (All numbers are not facts but just approximations)

Thanks for helping me out.

JonB

@StudentScripter
Hi. @SGaist and I have asked you over and over the same question, but you do not respond to it, not once!

This is my last attempt to glean the following information from you: what is your "model" behind the widgets? For example, where does the "specific set of 500 values has to be displayed in these widgets" actually come from? Do you just hard-code these 500 values/widgets in your code?

We would expect the data for those 500 widgets to come from some kind of "model". Not just appear as 500 lines of code. Then why do you have to save all of these? You should be able to reconstruct the widgets from this "model" at a later date, not to have to save the widgets.

On a separate note: it is hard to imagine how or why there would be 500 widgets which need to be constructed/displayed when a graphics item is clicked. It's just too many. How do you even lay them out? A better interface would probably be: show the 500 widget values in a table of consecutive lines. Note this is showing the widgets' values (as text), not the actual widgets. When the user clicks on one of these rows to edit a value, at that point go into edit mode and create the suitable corresponding widget. QTableView editing works like this. For example, if a value is text you might create a QLineEdit, if it is a number you might create a QSpinBox, etc. So only one widget is ever actually displayed, and only so long as the user interacts with it to enter a value. Then the editing widget disappears and is replaced again by the non-editable text value in the row.