Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB

[TheLumbee]

@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

@TheLumbee Why don't you just parse the date string in a rust function which returns the number of msecs sinc midnight 1970? This way you only have to store a qint64 variable. If you need a QDateTime instance then just call QDateTime::fromMSecsSinceEpoch when needed. This should be fast.

Don't know why I didn't think of that. I'll write the Rust lib, and hopefully have some great results. I'll let you know.

[Christian Ehrlicher]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Just seems insane that the performance between Rust and C++ is so different when it comes to parsing these files.

Simply read my explanation on what needs to be done when parsing a datetime string. This can be optimized in Qt but if someone needs speed other solutions are much better (see @J-Hilk 's solution by bypassing QDateTime::fromString() at all) so noone is doing it.

[JonB]

@TheLumbee
From where you are now. If you comment out the datetime handling in the C++ (and the Rust if you like), is your performance timing for the Qt/C++ acceptable compared to the Rust? So it is only the QDateTime parsing which is the issue for you?

[Christian Ehrlicher]

@J-Hilk said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Not sure if they optimised split in Qt6 or simply dropped splitRef because hardly any one used it 🤷‍♂️

QStringRef is gone and so is splitRef().
You should use QStringView and it's split() version.

[DerReisende]

Here is a version of @J-Hilk 's program modified to run with Qt 6.4:

int main(int argc, char **argv) {
    QFile *testFile = new QFile(R"(C:\Users\aea_t\Downloads\TestFile.txt)");


    testFile->open(QFile::ReadOnly);
    Instrument instr;
    Tick t;

    QElapsedTimer *parseTimer2 = new QElapsedTimer();
    parseTimer2->start();
    testFile->reset();
    instr.tickList.reserve(1000000);

    auto DateTimeParser = [](const QStringView string) -> QDateTime {
        const QDate date(string.left(4).toInt(), string.mid(4, 2).toInt(), string.mid(6, 2).toInt());
        const QTime time(string.mid(9, 2).toInt(), string.mid(11, 2).toInt(), string.mid(13, 2).toInt(),
                         string.mid(15, 3).toInt());
        QDateTime dt(date, time);
        return dt;
    };

    QTextStream readFile(testFile);
    QString line;
    line.reserve(100);
    const QChar semicolon(';');
    while (!readFile.atEnd()) {
        if (!readFile.readLineInto(&line, 100)) {
            break;
        }
        const auto result = line.split(semicolon);
        instr.tickList.append(
                {DateTimeParser(result.at(0)), result.at(1).toFloat(), result.at(2).toFloat(), result.at(3).toFloat(),
                 result.at(3).toUInt()});
    }
    qDebug().noquote() << QString("Qt parse time: %1ms")
            .arg(parseTimer2->elapsed());
}

Before we continue complaining about Qt it would be great to know on what OS @TheLumbee is running the code?
All solution seem to run much slower on Windows than Linux and macOS.
I tried the above version on my Win 11 machine VS2022 and it takes around 32 seconds to complete (compared to his 1.4 seconds on macOS). I don't have a mingw installation for comparison unfortunately :(

[Christian Ehrlicher]

@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

const auto result = line.split(semicolon);

Please read my last post about QStringView

[DerReisende]

@Christian-Ehrlicher It is just a port of @J-Hilk code to run with 6.4. I do not intend to rework it for QStringView - and my first attempt was incorrect anyways.

[Christian Ehrlicher]

@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

It is just a port of @J-Hilk code to run with 6.4.

But the port is wrong - you're now creating useless QString objects ...

[DerReisende]

@Christian-Ehrlicher Maybe I don't get the problem:
.splitRef() does not exist anymore in 64. .split() returns a QStringList which is implicitly shared according to the doc. Where is the mistake and how would you come to a QStringRef/QStringView version on Qt 6.4?

[Christian Ehrlicher]

@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

Where is the mistake and how would you come to a QStringRef/QStringView version on Qt 6.4?

QString::splitRef() - create a QList/Vector with references to parts of a QString, no QString objects are created, nothing gets copied
QString::split() - create a QList/Vector with newly create QString objects

To avoid the creation of the QString objects and get the same behavior as before, use QStringView::split()0

[TheLumbee]

@DerReisende I'm running on Ubuntu 22.04 64-bit. I'm currently creating the Rust lib. Got the shared lib and C-header file generated, but having an issue linking it all. So hopefully that will be resolved soon.

[TheLumbee]

@JonB Unfortunately, no. The solutions provided are by far leaps and bounds better than before, but the file sizes I'm expecting will definitely cause an issue if it's all done in Qt. I'm currently writing a Rust lib to resolve the problem for now, but even a 2-3x increase with Rust will make a monumental difference.

[DerReisende]

@Christian-Ehrlicher
Ok I modified it with the following:

        //const auto result = line.split(semicolon);
        const QStringView sv{line};
        const auto result = sv.split(semicolon);

Doesn't make a difference in runtime. And looking at the QString split code I am almost sure that line.split already does this optimization through implicit sharing through QStringList.

[Christian Ehrlicher]

@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

I am almost sure that line.split already does this optimization through implicit sharing through QStringList.

No, implicit sharing of a container has nothing to do with creating new QString objects - QStringList is a list of string objects, not a list of QStringViews ...

[JoeCFD]

@TheLumbee That is what I did with Java code before. When I got any bottleneck in my Java apps, I tried to use C/C++ code to do the jobs. Good luck!

[TheLumbee]

Update

I successfully created the Rust lib and it's passing all the data to Qt. Rust converts the dateTime portion to mSecsSinceEpoch and then it passes that to Qt to create a DateTime object. Much faster that way.

This wasn't the preferred choice, but it does the job faster than I've ever seen. I do appreciate everyone's input and help. Hopefully in the future Qt would be able to match the performance of Rust with this particular issue.

Also, I'm going to run some tests with this Rust lib on Windows to see if it's still glacial compared to Unix.

[TheLumbee]

Windows Update

After testing on Windows, I'm actually getting the same performance as Linux. So, it must be a C++ issue with Windows. I've tested with MinGW and MSVC with parsing and it's nearly unusable.

[Christian Ehrlicher]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

So, it must be a C++ issue with Windows

Again - use plain C++ and not Qt - the QDateTime parsing is painful slow...

[JonB]

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue.

Do you have access to both Qt 6.4 and C++20? If so, can you try combining
https://doc.qt.io/qt-6/qdatetime.html#fromStdTimePoint-1 (QDateTime QDateTime::fromStdTimePoint(const std::chrono::local_time<std::chrono::milliseconds> &time))
https://en.cppreference.com/w/cpp/chrono/parse (std::chrono::parse())
to see whether the datetime conversion part now matches Rust's?

For right or for wrong, I have appended a post into https://bugreports.qt.io/browse/QTBUG-97489 for this whole datetime parsing to QDateTime issue, as I am concerned it is a "show-stopper" if you have a large amountof string datetime data you need to get into Qt's QDateTime.

[Christian Ehrlicher]

Ok, some c++20 magic to use c++ instead c, but not really optimized

testFile.open(QFile::ReadOnly);
instr.tickList.clear();
instr.tickList.reserve(1000000);
QElapsedTimer parseTimer1;
parseTimer1.start();
const QByteArrayList allData = testFile.readAll().split('\n');
for (const auto &line : allData)
{
    const QByteArrayList data = line.split(';');
    std::string str = data.at(0).data();   // TODO: use data.at(0).toStdString()
    str[15] = '.';                         // sadly needed for correct msec parsing
    std::istringstream stream(str);
    std::chrono::sys_time<std::chrono::milliseconds> tTimePoint;
    std::chrono::from_stream(stream, "%Y%m%d %H%M%S", tTimePoint);
    instr.tickList.push_back({
        QDateTime::fromMSecsSinceEpoch(tTimePoint.time_since_epoch().count()),
        data.at(1).toDouble(),
        data.at(2).toDouble(),
        data.at(3).toDouble(),
        data.at(4).toInt()
    });
}
qDebug().noquote() << QString("Qt parse time: %1ms").arg(parseTimer1.elapsed());```

compared to @J-Hilk 's version:

compiled with debug:
Qt parse time: 32125ms
Qt parse time: 154511ms

compiled with release:
Qt parse time: 17926ms
Qt parse time: 157748ms

Attention: Qt debug libs, the emplace_back() doesn't help much.