Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB

[JonB]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue.

Do you have access to both Qt 6.4 and C++20? If so, can you try combining
https://doc.qt.io/qt-6/qdatetime.html#fromStdTimePoint-1 (QDateTime QDateTime::fromStdTimePoint(const std::chrono::local_time<std::chrono::milliseconds> &time))
https://en.cppreference.com/w/cpp/chrono/parse (std::chrono::parse())
to see whether the datetime conversion part now matches Rust's?

For right or for wrong, I have appended a post into https://bugreports.qt.io/browse/QTBUG-97489 for this whole datetime parsing to QDateTime issue, as I am concerned it is a "show-stopper" if you have a large amountof string datetime data you need to get into Qt's QDateTime.

[Christian Ehrlicher]

Ok, some c++20 magic to use c++ instead c, but not really optimized

testFile.open(QFile::ReadOnly);
instr.tickList.clear();
instr.tickList.reserve(1000000);
QElapsedTimer parseTimer1;
parseTimer1.start();
const QByteArrayList allData = testFile.readAll().split('\n');
for (const auto &line : allData)
{
    const QByteArrayList data = line.split(';');
    std::string str = data.at(0).data();   // TODO: use data.at(0).toStdString()
    str[15] = '.';                         // sadly needed for correct msec parsing
    std::istringstream stream(str);
    std::chrono::sys_time<std::chrono::milliseconds> tTimePoint;
    std::chrono::from_stream(stream, "%Y%m%d %H%M%S", tTimePoint);
    instr.tickList.push_back({
        QDateTime::fromMSecsSinceEpoch(tTimePoint.time_since_epoch().count()),
        data.at(1).toDouble(),
        data.at(2).toDouble(),
        data.at(3).toDouble(),
        data.at(4).toInt()
    });
}
qDebug().noquote() << QString("Qt parse time: %1ms").arg(parseTimer1.elapsed());```

compared to @J-Hilk 's version:

compiled with debug:
Qt parse time: 32125ms
Qt parse time: 154511ms

compiled with release:
Qt parse time: 17926ms
Qt parse time: 157748ms

Attention: Qt debug libs, the emplace_back() doesn't help much.

[TheLumbee]

@Christian-Ehrlicher Before I ever used Qt, I was facing the same issue with C++. I initially believed it was a filesystem difference and posted in a forum here: https://cplusplus.com/forum/general/254030/

Near the end of the thread you'll see that others noticed the same issue with Windows. I just find it odd that this blatant difference has never been noticed, at least in a major way.

[JonB]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Before I ever used Qt, I was facing the same issue with C++.

I do think this thread is getting confused. You certainly seem to talk in this thread about various different aspects of your speed with Rust/C++/Qt/Windows/file I/O all mixed into one. One has to deal with these separately. The issue @Christian-Ehrlicher and I, at least, are discussing now is specifically what to do about QDateTime::fromString(), which is by far the major contributor to your efficiency compared to Rust, other items are minor. The proposal is if one has Qt 6.4+ and C++ 20 then std::chrono can be used to parse the string input to a "naive datetime" (and I have a hunch that is what Rust uses) and that converted to a QDateTime in condirably better time that QDateTime::fromString().

This is quite distinct from e.g. the time taken to read the large file under Windows.

[TheLumbee]

@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

The issue @Christian-Ehrlicher and I, at least, are discussing now is specifically what to do about QDateTime::fromString(), which is by far the major contributor to your efficiency compared to Rust, other items are minor. The proposal is if one has Qt 6.4+ and C++ 20 then std::chrono can be used to parse the string input to a "naive datetime" (and I have a hunch that is what Rust uses) and that converted to a QDateTime in condirably better time that QDateTime::fromString().

Apologies. I don't have C++20, but I can set up an environment to test it. But even without the DateTime, Rust is parsing the file 2-3x faster than C++, with just floats and ints. Maybe C++20 has some improvements in that domain, but I'll set up an environment to test this.

[JonB]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

But even without the DateTime, Rust is parsing the file 2-3x faster than C++, with just floats and ints.

I do understand this. But I suggest this is a separate issue from the QDateTime. You started with 40x faster. Dealing with QDateTime is the first priority. File reading or parsing ints and floats is a separate issue requiring its own solution.

[TheLumbee]

@JonB I agree. I'll do some testing with this today and provide an update.

Thanks again for the help.

[TheLumbee]

To provide an update, the solution provided by @JonB 's "final offering" in this post parses files a little quicker. Although better than my initial approach, still not nearly as fast as the Rust solution. For now, I'm sticking with the Rust lib I wrote but I do think this points out some performance enhancements that can be made on the C++ side of things.

[DerReisende]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Although

For MSVC++ you should write a bug report for them with test data. There seems to be a major performance issue with the MS c++ library as it is much slower than other solutions. Maybe they can provide an update in a future version or hint to workarounds to improve performance on windows.

[Ahmad Samir]

Hello. I know this is an old topic.

I couldn't test as the test file doesn't seem accessible anymore.

Assuming you were testing on Linux with glibc, could you try setting the TZ environment variable? e.g.:
export TZ=":/etc/localtime"

See also:
https://sourceware.org/bugzilla/show_bug.cgi?id=24004
https://bugreports.qt.io/browse/QTBUG-77948