Append Model to VXYModelMapper is very slow

neda

Hi,
I use "VXYModelMapper" and "QAbstractTableModel" for draw real time chart.
I save points of chart in JSON file.
I draw real time chart like below example:
https://forum.qt.io/topic/77439/can-someone-provide-a-working-example-of-vxymodelmapper-with-lineseries-chart/2

I have a lot of points (more than 1,000,000 points) and insert rows in QAbstractTableModel(append point to QVector<QPair<double,double>>) is very slow.

QJsonArray npcArray=json["points"].toArray();
    chartModel->clearChartModel();
    QVector<QPointF> vecPoints;
    
    foreach(const QJsonValue & val, npcArray){
        double xVal=val.toObject().value("x").toDouble();
        double yVal=val.toObject().value("y").toDouble();
        
        QPointF point=QPointF(xVal,yVal);
        vecPoints.append(point);
        chartModel->appendData(xVal*unitXCoefficient,yVal*unitYCoefficient);

    }

void ChartModel::appendData(double const xVal,double const yVal)
{
    setXMin(qMin(m_xMin, xVal));
    setXMax(qMax(m_xMax, xVal));
    setYMin(qMin(m_yMin, yVal));
    setYMax(qMax(m_yMax, yVal));
    int const vecSize = vecChartPoints.size();
    beginInsertRows(QModelIndex (), vecSize, vecSize);
    vecChartPoints.append(qMakePair<double, double>(xVal, yVal));

    endInsertRows();
}

Christian Ehrlicher

When inserting bulk data you should not use begin/endInsertRow for every single line - this will cause a complete update for every line. Just write a proper appendData() which takes all new lines and call begin/endInsertRows once.

neda

@Christian-Ehrlicher

I change code but it is not benefit.

QJsonArray npcArray=json["points"].toArray();
    clearChartModel();
    QVector<QPointF> vecPoints;
    
    beginInsertRows(QModelIndex(),0,npcArray.size()-1);
    foreach(const QJsonValue & val, npcArray){
        double xVal=val.toObject().value("x").toDouble();
        double yVal=val.toObject().value("y").toDouble();
        QPointF point=QPointF(xVal,yVal);
        vecPoints.append(point);

        xVal*=unitXCoefficient;
        yVal*=unitYCoefficient;

        setXMin(qMin(m_xMin, xVal));
        setXMax(qMax(m_xMax, xVal));
        setYMin(qMin(m_yMin, yVal));
        setYMax(qMax(m_yMax, yVal));
        vecChartPoints.append(qMakePair<double, double>(xVal,yVal));
    }
    endInsertRows();

VRonin

I don't think the append is the bottleneck at all. The real problem I think is the chart trying to draw >1m points (and it needs to refresh them all every time due to how QCharts is implemented). You might want to put a proxy model inbetween that performs some sampling of the 1m points

JonB

@neda
I believe it is likely that the bottleneck is as @VRonin says, it's in the chart point drawing not the model.

Temporarily comment out lines in your code. Try it so the points are added to the model but not the chart (disconnect chart from model) and compare speed.

You should consider point sampling to reduce the number of points, and also whether you should be deleting old points as you add new ones in a "sliding window".

neda

@JonB
I understood what is happened and I accept it's in the chart point drawing not the model but I do not know what do I do.
I reduce the number of points to 5000 but not benefit.
I try disconnect chart from model with blow code but not benefit.

foreach(const QJsonValue & val, npcArray){
        double xVal=val.toObject().value("x").toDouble();
        double yVal=val.toObject().value("y").toDouble();
        QPointF point=QPointF(xVal,yVal);
        vecPoints.append(point);

        xVal*=unitXCoefficient;
        yVal*=unitYCoefficient;

        //        setXMin(qMin(m_xMin, xVal));
        //        setXMax(qMax(m_xMax, xVal));
        //        setYMin(qMin(m_yMin, yVal));
        //        setYMax(qMax(m_yMax, yVal));
        _vecChartPoints.append(qMakePair<double, double>(xVal,yVal));
    }
 
 beginInsertRows(QModelIndex(),0,npcArray.size()-1);
 vecChartPoints=_vecChartPoints;

endInsertRows();

Last Line of code(endInsertRows) causes delay.

JonB

@neda

I reduce the number of points to 5000 but not benefit.

Well, there must be some benefit when you reduce the number of points from 1,000,000 to 5,000, surely, surely?? If not something else is way wrong....

So you are claiming that if you scrap all your code and just test adding 5,000 rows to a QAbstractTableModel it is "slow"? Just how "slow" is "slow" for 5,000 rows?

I see you calling beginInsertRows() & append() yourself. For QAbstractTableModel the docs (http://doc.qt.io/qt-5/qabstracttablemodel.html#details) tell you:

Models that provide interfaces to resizable data structures can provide implementations of insertRows(), removeRows(), insertColumns(), and removeColumns(). When implementing these functions, it is important to call the appropriate functions so that all connected views are aware of any changes:
.
An insertRows() implementation must call beginInsertRows() before inserting new rows into the data structure, and it must call endInsertRows() immediately afterwards.
.
... etc.

Where are you doing that implementation?

Have you tried a QStandardItemModel (http://doc.qt.io/qt-5/qstandarditemmodel.html, inherits from QAbstractItemModel) to see how that behaves on your insert/append timings? If that does it noticeably faster than you then it would indicate your implementation is not good.

Winz

Take a look at this article, I think the proposed solution should solve your problem https://www.kdab.com/a-speed-up-for-charting-on-embedded/

neda

@JonB said in Insert rows in QAbstractTableModel is very slow:

Well, there must be some benefit when you reduce the number of points from 1,000,000 to 5,000, surely, surely??

I've only reduced the number of points to see If the number of points is less, the code will work or not :(
Usually charts have more than 100,000 points and I have to consider up to 1,000,000 points.

@JonB said in Insert rows in QAbstractTableModel is very slow:

So you are claiming that if you scrap all your code and just test adding 5,000 rows to a QAbstractTableModel it is "slow"? Just how "slow" is "slow" for 5,000 rows?

I don't know why, but my program is not responding (work just for about 100 points).

I changed the codes and use "VXYModelMapper" and "QStandardItemModel".

VXYModelMapper {
                            id: modelMapper
                            model: myChartClass.newMyChartModel
                            series: lineSeries
                            xColumn: 0
                            yColumn: 1
                        }
void MyChartClass::setMyChartModel(QStandardItemModel *model)
{

    newMyChartModel= model;
    emit myChartModelChanged(model);
}

double m_xMin=0;
    double m_xMax=0;
    double m_yMin=0;
    double m_yMax=0;
   QStandardItemModel* lineModel=new QStandardItemModel(npcArray.size(), 2);

    foreach(const QJsonValue & val, npcArray){
        double xVal=val.toObject().value("x").toDouble();
        double yVal=val.toObject().value("y").toDouble();
                
        QStandardItem *item1 = new QStandardItem(QString::number(xVal));
        lineModel->setItem(i, 0, item1);

        QStandardItem *item2 = new QStandardItem(QString::number(yVal));
        lineModel->setItem(i, 1, item2);
         i+=1;
        
        m_xMin = qMin(m_xMin, xVal);
        m_xMax = qMax(m_xMax, xVal);
        m_yMin = qMin(m_yMin, yVal);
        m_yMax =  qMax(m_yMax, yVal);
    }
   
    myChartClass->setXMin(m_xMin);
    myChartClass->setXMax(m_xMax);
    myChartClass->setYMin(m_yMin);
    myChartClass->setYMax(m_yMax);

//To this line everything is very good and fast   

    QElapsedTimer timer;
    timer.start();
    myChartClass->setMyChartModel(lineModel);
    qDebug() << "The slow operation took" << timer.elapsed() << "millisecond";

Output:

The slow operation took 123045 millisecond.

Total point count: 5188

It is very slow :(

neda

I consume 123045 milliseconds if I do it after showing the window, if I do it before it consumes 0 millisecond.
I need the user to select a chart from the list to display it. So the code runs after showing the window.

I use QTimer but not benefit.
Please guide me.

JonB

@neda
2 minutes is of course not right for 5000-odd points!

What does your emit myChartModelChanged(model); cause to be called?

Winz

Could you please paste a minimal working code. Did you analyze performance with a profiler tool to identify the bottleneck?

neda

@JonB said in Append Model to VXYModelMapper is very slow:

What does your emit myChartModelChanged(model); cause to be called?

I do not think so.
I think the problem is displaying because if I do it before showing the window it consumes 0 millisecond.

neda

@Winz

Did you analyze performance with a profiler tool to identify the bottleneck?

No, What are the tools for doing this?

Winz

It depends on your OS. On linux you can try valgrind. It's a collection of tools for dynamic analysis integrated to QtCreator : Under QtCreator, in release mode, launch Analyze menu > valgrind function profiler. Test your app and on closing QtCreator will display profiling information. For QML profiling, check Projects > build > Build steps details > enable QML debugging and Projects > run > Debugger settings > enable QML and launch Analyze menu > QML profiling. then test your app and on closing QtCreator will display profiling information