Application not responding when export huge data to CSV.

mangekoyu

Hello. I have a function for export data to csv. It's working well but when I try to export huge data like 100K. application not responding. I have no idea why I got this error. Can someone helo me please?

Here is export function.

void MainWindow::exportArraysToCSV(QList < QList < double >> dataColums) {
  QString filters("CSV files (*.csv);;All files (*.*)");
  QString defaultFilter("CSVi files (*.csv)");
  QString fileName = QFileDialog::getSaveFileName(0, "Save file", QCoreApplication::applicationDirPath(),
    filters, & defaultFilter);
  QFile file(fileName);

  if (file.open(QFile::WriteOnly | QFile::Append)) {
    QTextStream data( & file);
    QStringList strList;
    QString formula;



  data << strList.join(",") << "\n";

    int maxRowCount = 0;
    foreach(auto column, dataColums)
    maxRowCount = qMax(maxRowCount, column.count());

    for (int i = 5; i < maxRowCount; ++i) // rows
    {

      strList.clear();
      for (int j = 0; j < 2; ++j) // columns
      {


        if (i < dataColums[j].count()){
          
            strList.append(QString::number(dataColums[j][i], 'f'));
            
        }


        else
          strList.append("\"\" ");
      }

      data << strList.join(";") + "\n";


    }

    file.close();
  }
}

JonB

@mangekoyu
What "error"? It presumably takes this long to build the lists and output them to file. During that time your UI will be blocked, unless you do a bit at a time on a timer or in a separate, non-UI thread.

mangekoyu

@JonB said in Application not responding when export huge data to CSV.:

UI thread.

I'm waited around 1 hour but st'll saying program not responding. How can I meke it like you said?
My problem ıs it's taking long time to export and user can say Oh program crashed.

JonB

@mangekoyu
It should not take an hour! :)

I cannot see a fault in that code which would cause it to loop indefinitely. Why don't you put some qDebug() statements in so you can see where it is getting, or step through it in debugger?

mangekoyu

@JonB I tried put qDebug() to all lines in function but when application says not responding qDebug() I cannot see any debug :(

JonB

@mangekoyu
What platform are you on? Windows you (apparently) have to take special action to see debug messages?

But can't you see them in Creator-debugger? Are you even sure you hit this function? Why don't you step through it in debugger?

piervalli

@JonB You can not block UI than gran of 1m , you should move on secondary thread with qconcurrency

SGaist

Hi,

@mangekoyu said in Application not responding when export huge data to CSV.:

maxRowCount = qMax(maxRowCount, column.count());

Why is your max row count tied to the column count ?

What does 100k represent ?
How many columns do you have ?

Your logic is also not clear, you have some magic numbers here and there which makes it hard to really follow what should go into your CSV file.

JonB

@SGaist said in Application not responding when export huge data to CSV.:

Why is your max row count tied to the column count ?

Instead of the normal, with rows having columns, and maybe not all rows having the same number of columns, the OP seems to have a list of columns which hold the rows.... The mac row count seems to be the max rows among all columns....

mangekoyu

@SGaistThere will be 2 columns and
each will have a maximum of 170,000 rows.

JonB

@mangekoyu
That is a lot of rows, which will take a bit of time, but still not 1 hour/hangs/crashes. Like I said, put in a qDebug() or step through to see what is happening.

Separate points:

data << strList.join(",") << "\n";: this always just puts a single \n, nothing else, into data. What's the point?
~~data << strList.join(";") + "\n";: this outputs one line with the columns joined by ;. Normally CSV files use , as the column joiner/separator, hope you know what you are doing.~~ [ @J-Hilk has explained to me that CSVs use locale numbers.]

J.Hilk

@JonB said in Application not responding when export huge data to CSV.:

Normally CSV files use , as the column joiner/separator, hope you know what you are doing.

in the English speaking world maybe, over here, its ; as default separator because , is usually the "decimal point"

JonB

@J-Hilk
Ohhhh, I didn't know that! [I will correct that point in my answer.] In that case, CSV files differ by locale, but don't say which locale they use, so goodness knows how it works across countries!

UPDATE
Does this apply to CSV files though? [Wikipedia states](https://en.wikipedia.org/wiki/Comma-separated_values#:~:text=A comma-separated values (CSV,name for this file format.):

A comma-separated values (CSV) file is a delimited text file that uses a comma to separate values. Each line of the file is a data record. Each record consists of one or more fields, separated by commas. The use of the comma as a field separator is the source of the name for this file format.

Alternative delimiter-separated files are often given a ".csv" extension despite the use of a non-comma field separator. This loose terminology can cause problems in data exchange.

Hmm. "This loose terminology can cause problems in data exchange" :)

J.Hilk

@JonB are you sure ? I have a really strong déjà vu here!

@mangekoyu
print the maxRowCount after you finished calculating it. See how big this actually is!

foreach(auto column, dataColums)
     maxRowCount = qMax(maxRowCount, column.count());
qDebug() << maxRowCount;

mangekoyu

@J-Hilk @JonB It works as you said. I must wait untill finish write all rows and columns. I think I can put progressbar for make user wait. Thank you for all answers.

J.Hilk

@mangekoyu there should be ways to make this faster,

don't use join and not int he frequency you do, it is a heavy operation, and you actually don't need it
I don't know what dataColums type is, but if its a QtContainer, consider using at() for fetching read only const reference, it should be faster
drop strList all together, you have a QTextStream operator, that one excepts nearly everything, especially integers, so no need to call QString::Number either

  if (file.open(QFile::WriteOnly | QFile::Append)) {
    QTextStream data( & file);

    data << QChar::LineFeed;

    int maxRowCount = 0;
    foreach(auto column, dataColums)
        maxRowCount = qMax(maxRowCount, column.count());

    for (int row = 5; row < maxRowCount; ++i) {
       for (int col = 0; col < 2; ++j) {
            if (row < dataColums.at(col).count()) {
                data << dataColums[j][i];
            } else {
               data << QChar('"') << QChar('"');
            }
        }
        data << QChar(';') << QChar::LineFeed;
    }

    file.close();
  }