Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB

Christian Ehrlicher

Ok, some c++20 magic to use c++ instead c, but not really optimized

testFile.open(QFile::ReadOnly);
instr.tickList.clear();
instr.tickList.reserve(1000000);
QElapsedTimer parseTimer1;
parseTimer1.start();
const QByteArrayList allData = testFile.readAll().split('\n');
for (const auto &line : allData)
{
    const QByteArrayList data = line.split(';');
    std::string str = data.at(0).data();   // TODO: use data.at(0).toStdString()
    str[15] = '.';                         // sadly needed for correct msec parsing
    std::istringstream stream(str);
    std::chrono::sys_time<std::chrono::milliseconds> tTimePoint;
    std::chrono::from_stream(stream, "%Y%m%d %H%M%S", tTimePoint);
    instr.tickList.push_back({
        QDateTime::fromMSecsSinceEpoch(tTimePoint.time_since_epoch().count()),
        data.at(1).toDouble(),
        data.at(2).toDouble(),
        data.at(3).toDouble(),
        data.at(4).toInt()
    });
}
qDebug().noquote() << QString("Qt parse time: %1ms").arg(parseTimer1.elapsed());```

compared to @J-Hilk 's version:

compiled with debug:
Qt parse time: 32125ms
Qt parse time: 154511ms

compiled with release:
Qt parse time: 17926ms
Qt parse time: 157748ms

Attention: Qt debug libs, the emplace_back() doesn't help much.

TheLumbee

@Christian-Ehrlicher Before I ever used Qt, I was facing the same issue with C++. I initially believed it was a filesystem difference and posted in a forum here: https://cplusplus.com/forum/general/254030/

Near the end of the thread you'll see that others noticed the same issue with Windows. I just find it odd that this blatant difference has never been noticed, at least in a major way.

JonB

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Before I ever used Qt, I was facing the same issue with C++.

I do think this thread is getting confused. You certainly seem to talk in this thread about various different aspects of your speed with Rust/C++/Qt/Windows/file I/O all mixed into one. One has to deal with these separately. The issue @Christian-Ehrlicher and I, at least, are discussing now is specifically what to do about QDateTime::fromString(), which is by far the major contributor to your efficiency compared to Rust, other items are minor. The proposal is if one has Qt 6.4+ and C++ 20 then std::chrono can be used to parse the string input to a "naive datetime" (and I have a hunch that is what Rust uses) and that converted to a QDateTime in condirably better time that QDateTime::fromString().

This is quite distinct from e.g. the time taken to read the large file under Windows.

TheLumbee

@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

The issue @Christian-Ehrlicher and I, at least, are discussing now is specifically what to do about QDateTime::fromString(), which is by far the major contributor to your efficiency compared to Rust, other items are minor. The proposal is if one has Qt 6.4+ and C++ 20 then std::chrono can be used to parse the string input to a "naive datetime" (and I have a hunch that is what Rust uses) and that converted to a QDateTime in condirably better time that QDateTime::fromString().

Apologies. I don't have C++20, but I can set up an environment to test it. But even without the DateTime, Rust is parsing the file 2-3x faster than C++, with just floats and ints. Maybe C++20 has some improvements in that domain, but I'll set up an environment to test this.

JonB

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

But even without the DateTime, Rust is parsing the file 2-3x faster than C++, with just floats and ints.

I do understand this. But I suggest this is a separate issue from the QDateTime. You started with 40x faster. Dealing with QDateTime is the first priority. File reading or parsing ints and floats is a separate issue requiring its own solution.

TheLumbee

@JonB I agree. I'll do some testing with this today and provide an update.

Thanks again for the help.

TheLumbee

To provide an update, the solution provided by @JonB 's "final offering" in this post parses files a little quicker. Although better than my initial approach, still not nearly as fast as the Rust solution. For now, I'm sticking with the Rust lib I wrote but I do think this points out some performance enhancements that can be made on the C++ side of things.

DerReisende

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Although

For MSVC++ you should write a bug report for them with test data. There seems to be a major performance issue with the MS c++ library as it is much slower than other solutions. Maybe they can provide an update in a future version or hint to workarounds to improve performance on windows.

Ahmad Samir

Hello. I know this is an old topic.

I couldn't test as the test file doesn't seem accessible anymore.

Assuming you were testing on Linux with glibc, could you try setting the TZ environment variable? e.g.:
export TZ=":/etc/localtime"

See also:
https://sourceware.org/bugzilla/show_bug.cgi?id=24004
https://bugreports.qt.io/browse/QTBUG-77948

GregB

I just wanted to give this thread kudos. I copied the following function from this thread into my code:

auto DateTimeParser =[](const QStringRef & string) ->QDateTime {
    const QDate date(string.left(4).toInt(), string.mid(4,2).toInt(), string.mid(6.2).toInt());
    const QTime time(string.mid(9,2).toInt(), string.mid(11,2).toInt(), string.mid(13.2).toInt(), string.mid(15.3).toInt());
    QDateTime dt (date,time);
    return dt;
};

and the performance improved dramatically. I was parsing approximately 1 million datetimes and it took 90-100 seconds using QDateTime::fromString(). Using this function it now takes 23.5 seconds. 4x increase in performance.

Thank you!!

FeRDNYC

Necromancy, sorry, but there was a point made a while back that kind of zipped by without comment, and I wanted to circle back to it:

@Christian-Ehrlicher said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

The problem with QDateTime is that for every call the internal format parser (QDateTimeParser, private class) is re-created and needs to re-evaluate the string. This takes a lot of time.

Given how expensive it is to re-create that parser for every call, I wonder if Qt would consider making the parser public. instantiable, and reusable, so that when parsing a lot of same-formatted date strings, it could be created just once and applied to all of them? Kind of like Python's re.compile() for regular expressions.

e.g. something like:

QDateTimeStringParser* dateParser = new QDateTimeStringParser(
    "yyyyMMdd HHmmss zzz0000"
);
QElapsedTimer* parseTimer1 = new QElapsedTimer();
parseTimer1->start();
for (int ii = 0; ii < allData.size() - 1; ii++)
{
    QByteArrayList data = allData.at(ii).split(';');
    t.dt = dateParser->parse(data.at(0));
    t.last = data.at(1).toDouble();
    t.bid = data.at(2).toDouble();
    t.ask = data.at(3).toDouble();
    t.volume = data.at(4).toInt();
    instr.tickList.append(t);
}
qDebug().noquote() << QString("Qt parse time: %1ms")
        .arg(parseTimer1->elapsed());
dateParser->deleteLater();

I wonder if that could make it significantly faster? parse() would have to be reentrant and stateless, of course (beyond the stored, immutable format string), which could still be tricky for a complex parser.