Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB

TheLumbee

I have written the same file parsing implementation in Rust and in Qt, but Rust completes the task unbelievably quicker. I'd love to just implement this in C++ because I haven't taken the time to create a Rust lib for the file mainly because I'm not extremely proficient in the language. But I'm curious if anyone has any answers/solutions to solving this issue. It's actually mind-blowing the difference, but there has to be a C++ solution that is comparable in performance.

Rust code here:

use chrono::NaiveDateTime;
use howlong::HighResolutionTimer;
use std::env;
use std::fs::File;
use std::io::prelude::*;
use std::io::BufReader;

#[derive(PartialEq)]
pub struct Tick
{
    pub dt: NaiveDateTime,
    pub last: f32,
    pub bid: f32,
    pub ask: f32,
    pub volume: u32
}

impl Tick
{
    pub fn init(dt: &str, last: f32, bid: f32, ask: f32,
        volume: u32) -> Tick
    {
        let date_time = NaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f")
            .expect("Invalid time signature");
        Tick { dt: date_time, last, bid, ask, volume }
    }
}

#[derive(PartialEq)]
pub struct Instrument
{
    pub tick_list: Vec<Tick>,
}

impl Instrument
{
    pub fn new() -> Instrument
    {
        Instrument { tick_list: Vec::new() }
    }

    fn add_tick_from_str(&mut self, data: &str)
    {
        let temp: Vec<&str> = data.split(';').collect();
        if temp.len() != 5
        {
            println!("Tick data does not contain enough information.");
            return;
        }
        
        let tick = Tick::init(temp[0], temp[1].parse().expect("Unable to parse last price"),
            temp[2].parse().expect("Unable to parse bid price"), temp[3].parse()
            .expect("Unable to parse ask price"), temp[4].parse().expect("Unable to parse volume"));
        self.tick_list.push(tick);
    }

    pub fn import_tick_data(&mut self, file_name: &str)
    {
        let file = File::open(file_name)
            .expect("Tick file not found to import instrument.");
        let buf_reader = BufReader::new(file);
        for line in buf_reader.lines()
        {
            self.add_tick_from_str(&line.unwrap());
        }
    }
}

fn main() 
{
    let timer = HighResolutionTimer::new();
    let args: Vec<String> = env::args().collect();
    let mut instr = Instrument::new();
    instr.import_tick_data(&args[1].as_str());
    println!("Rust parse time: {:?}", timer.elapsed());
}

Qt Code here:

#include <QByteArray>
#include <QDateTime>
#include <QDebug>
#include <QElapsedTimer>
#include <QFile>

struct Tick
{
    QDateTime dt;
    double last,
        bid,
        ask;
    quint32 volume;
};

struct Instrument
{
    QList<Tick> tickList;
};

int main(int argc, char** argv)
{
    QElapsedTimer* parseTimer = new QElapsedTimer();
    parseTimer->start();
    if (argc != 2)
    {
        qDebug().noquote() << "File argument is required";
        return -1;
    }

    QFile* testFile = new QFile(argv[1]);
    if (!testFile->exists())
    {
        qDebug().noquote() << "Test file does not exist";
        return -2;
    }

    testFile->open(QFile::ReadOnly);
    QByteArrayList allData = testFile->readAll().split('\n');
    Instrument instr;
    Tick t;
    for (int ii = 0; ii < allData.size() - 1; ii++)
    {
        QByteArrayList data = allData.at(ii).split(';');
        t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000");
        t.last = data.at(1).toDouble();
        t.bid = data.at(2).toDouble();
        t.ask = data.at(3).toDouble();
        t.volume = data.at(4).toInt();
        instr.tickList.append(t);
    }

    qDebug().noquote() << QString("Qt parse time: %1ms")
        .arg(parseTimer->elapsed());
}

Results here:

Rust parse time: 722.340226ms
Qt parse time: 30452ms

File used for parsing:

https://drive.google.com/file/d/1A7mqAeC238yMoveg9ausZ5DRu5kNfKRp/view?usp=sharing

JonB

@TheLumbee
I know nothing about Rust(!), but find it hard to believe its "parsing" code will be faster than native C++ code. For the C++ are you compiling for Release or for Debug --- debug can be a lot slower. A couple of observations on your code:

testFile->readAll().split('\n');:
If the file is "large" reading all its content and splitting into lines for processing uses memory. An efficient parser would process as at read line-by-line. However, I doubt this is significant.

QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000");:
This is my only thought for "slowness". A while ago we discussed something about Qt's handling of parsing datetimes from string, and something was found to be slow to do with local time conversions or similar. Can you comment out just this line and see what effect it has on your overall timing?

J.Hilk

@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

I doubt this is significant

it is, depending on Qt Version, if 5.15 than yes, using splitRef should be significantly faster. Not sure if they optimised split in Qt6 or simply dropped splitRef because hardly any one used it 🤷‍♂️

JonB

@J-Hilk
Hmm, I just had a look at the OP's linked file used to parse, expecting it to be "not large". However, after a while of scrolling I gave up trying to reach the bottom, it's taking too long! So I guess it is "larger" than I had imagined.

I am still not sure that reading into memory and splitting would cause the significant difference in timing the OP shows though. The OP should try my previous "datetime" suggestion. Nonetheless he should bear your comment in mind. I see that the Rust uses for line in buf_reader.lines(), which we can guess means it reads a line at a time into a buffer. To equate the Qt should also read a line at a time (e.g. a QTextStream?) rather than readAll().split('\n').

Also the issue of release vs debug build is critical, check that.

J.Hilk

@JonB just noticed, the OP is operating purely on QByteArrays, that one never had a SplitRef -> always hard copies :D

To equate the Qt should also read a line at a time (e.g. a QTextStream?)

yes, would be my approach

besides that, there are tons of optimisations still possible.

Believe it or not, qDebug() calls are time intensive. Calling parseTimer->elapsed() and storing that in an int before qDebug() can make make difference in 100's of ms.

Irrelevant if you compare inside Qt, but not outside

JonB

@TheLumbee
OK, since I have nothing better to do, I decided to look at timings on my version of Qt/C++. Obviously I can only look at these, I don't have Rust.

I run Qt 5.12.8 under a VM on Windows 7. My machine is old and slow! The VM has 3GB out of 8GB and 2 CPUs out of 4. We are only interested in comparative timings. So far I have only done initial ones, on your code as-is --- I will change code and do some more --- but the results are already fascinating!

I started out running under debugger, compiled for debug. I gave up waiting for the file (68MB, 1.3 million lines) to load. After more than a minute it was still on just the starting QByteArrayList allData = testFile->readAll().split('\n') statement, it had not even finished that!
Still compiled for debug, I did a Run from Creator instead of a Debug. It completed in 59 seconds. That compares to your timing of 30 seconds.
Then all I did was comment out the QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000");, following my hunch. And guess what? 60 seconds got reduced to... 2 seconds!!

I will do some more investigations and improvements, and post my results later. But I think you can see now where the main culprit lies... :)

DerReisende

Is there a reason why your Tick class's rust impl uses 32bit floats and your c++ code 64bit doubles?

DerReisende

@JonB I had a performance problem with QDate::fromString and ended up implementing a lookup cache that increased performance by factor 30. I think it is also mentioned either in source or doc that these conversion functions are slow.

JonB

@DerReisende
Indeed, that is what I guessed the issue would be, and the whole point is that the timings show this is by far the main factor, costing ~97% of the total time!! We need to find the bug report for this QDate::fromString() which I recall. See whether there is a suggested workaround, or which Qt version it got fixed in...

OK, it's here: https://bugreports.qt.io/browse/QTBUG-97489. And see also https://forum.qt.io/topic/131317/speed-issue-on-qt-6-2-0-vs-qt-6-1-3/2 . The bug says "Fixed in
6.3.0 Alpha". I don't see a Qt workaround for earlier versions....

and ended up implementing a lookup cache that increased performance by factor 30

Your "factor 30" is just what my timings are showing. If the user is earlier than Qt 6.3, could this be applied to his case? You might like to show what you did for your "lookup cache".

JonB

@TheLumbee
Next update. Changed over to compiling the Qt for Release instead of Debug:

With the QDate::fromString() still in the code it "made no difference". Still 59 seconds. This is because around 97% of the time is spent in that function, which is unaffected by how i compile my own code. Not surprising no improvement.
Without the QDate::fromString() in the code, reduced 2 seconds to 1.5 seconds. Consistent, but figures are low to be certain about conclusion. Implies 25% time reduction for Release vs Debug build.

DerReisende

@JonB
I did the following for parsing QDate objects in one of my projects

QDate TreeModel::convert_string_to_date(QString &inDateStr,
                                        tsl::hopscotch_map<QString, QDate> &date_cache) noexcept(false) {
    if (date_cache.contains(inDateStr))
        return date_cache.at(inDateStr);
    //else we need to compute
    QDate result;
    if (inDateStr.contains('.')) {
        // we have dd.mm.yyy
        result = QDate::fromString(inDateStr, "dd.MM.yyyy");
    } else if (inDateStr.contains('-')) {
        //we have yyyy-mm-dd aka. ISO-8601 format
        result = QDate::fromString(inDateStr, Qt::ISODate);
    } else {
        throw DateConversionException("Unknown Date Type: " + inDateStr.toStdString());
    }
    date_cache.insert({inDateStr, result});

    return result;
}

with:

tsl::hopscotch_map<QString, QDate> date_cache{DATE_CACHE_SIZE};

I used tsl::hopscotch_map as well as it is faster than std::unordered_map but it could be easily replaced.

Using something for the date parsing on col 1 of the testfile should greatly enhance runtime performance sind cache hit rate there should be really high.

JonB

@DerReisende
Ah, OK, you mean you are just caching previously met dates in the cache. The OP's file has "thousands" of different dates with times. Actually, inspecting its million entries cover a 24:00:00 hour period from 6:00am to 5:59am. He could think about adapting your date principle, don't know if he wants to allow for local time which could have an hour "daylight" skip potentially or can ignore that or times are UTC? Anyway there is certainly something which could be done with these million entries to avoid having to fromString(data.at(0), "yyyyMMdd HHmmss zzz0000") on each one...!

DerReisende

@JonB My quick and dirty take on the problem which brings runtime on my machine from 37seconds down to 6 seconds:

struct Tick {
    QDateTime dt;
    float last{};
    float bid{};
    float ask{};
    quint32 volume{};
};

struct Instrument {
    QList<Tick> tickList;
};

static QDate convert_string_to_date(const QString& inDateStr) {
    static std::unordered_map<QString, QDate> date_cache;
    if (date_cache.contains(inDateStr))
        return date_cache.at(inDateStr);
    //else we need to compute
    QDate result = QDate::fromString(inDateStr, "yyyyMMdd");
    date_cache.insert({inDateStr, result});

    return result;
}

static QTime convert_string_to_time(const QString& inDateStr) {
    static std::unordered_map<QString, QTime> time_cache;

    //else we need to compute
    const auto list = inDateStr.split(' ');
    QTime baseTime;
    if (time_cache.contains(list[0]))
        baseTime = time_cache.at(list[0]);
    else {
        baseTime = QTime::fromString(list[0], "HHmmss");
        time_cache.insert({list[0], baseTime});
    }

    return baseTime.addMSecs(QTime::fromString(list[1], "zzz0000").msec());
}

static QDateTime convert_string_to_datetime(QString &inDateStr)
{
    const auto list = inDateStr.split(' ');
    const QDate date = convert_string_to_date(list[0]);
    const QTime time = convert_string_to_time(list[1] + ' ' + list[2]);
    QDateTime result(date, time);

    return result;
}

int main(int argc, char **argv) {
    QElapsedTimer parseTimer;
    parseTimer.start();
    QFile testFile("TestFile.txt");
    testFile.open(QFile::ReadOnly);
    QByteArrayList allData = testFile.readAll().split('\n');

    Instrument instr;
    for (int ii = 0; ii < allData.size() - 1; ii++)
    {
        QByteArrayList data = allData.at(ii).split(';');
        Tick t;
        QString dt_str = data.at(0);
        t.dt = convert_string_to_datetime(dt_str);
        /*const auto orig = QDateTime::fromString(dt_str, "yyyyMMdd HHmmss zzz0000");
        if (t.dt != orig) {
            qDebug() << "orig: " << orig << " versus: " << t.dt;
            std::exit(99);
        }*/
        t.last = data.at(1).toFloat();
        t.bid = data.at(2).toFloat();
        t.ask = data.at(3).toFloat();
        t.volume = data.at(4).toInt();
        instr.tickList.append(t);
    }

    qDebug().noquote() << QString("Qt parse time: %1ms")
            .arg(parseTimer.elapsed());

    return 0;
}

Still a lot slower than rust :(

JonB

@TheLumbee
OK, here is my final code offering.

I use QTextStream on the file to read a line at a time. As suspected, this makes little difference to timing over reading the whole file into memory and splitting it, but is a lot gentler on memory. It is also more favorable to e.g. Windows text files. I assume it's what the Rust will be doing, and I would do same if I were you.

I cannot use a "fixed" QDataTime::fromString(), and i am assuming you cannot (if you can, try it). If you look at your Rust you have dt: NaiveDateTime. That "naive" means it does not do any local time/daylight etc, conversions. (https://docs.rs/chrono/latest/chrono/naive/struct.NaiveDateTime.html# "ISO 8601 combined date and time without timezone"). I can't use std::chrono::parse as that is C++20, I can't use strptime() as that is available for Windows cross-platform. I can't be bothered to try to write a "cached lookup", I don't know whether Rust internals use anything like that, and it does not "scale" well if we can't make assumptions about what dates we will meet. For now/illustration I have just gone for sscanf() to read your fixed-format date/time field inputs (don't know whether QString::toInt() on QString::mid() for each fixed-length section might be faster, didn't try).

My code --- with your stuff commented out so you can see what i have changed --- reads:

    Instrument instr;
    Tick t;
    testFile->open(QFile::ReadOnly);
//    QByteArrayList allData = testFile->readAll().split('\n');
    QTextStream ts(testFile);
    QString line;
    while (ts.readLineInto(&line))
//    for (int ii = 0; ii < allData.size() - 1; ii++)
    {
//        QByteArrayList data = allData.at(ii).split(';');
        QStringList data = line.split(';');
//        t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000");
        int year, month, day, hour, minute, second, milli;  
        sscanf(data.at(0).toLocal8Bit().constData(), "%4d%2d%2d %2d%2d%2d %3d0000", &year, &month, &day, &hour, &minute, &second, &milli);
        t.dt = QDateTime(QDate(year, month, day), QTime(hour, minute, second, milli), Qt::UTC);
        t.last = data.at(1).toDouble();
        t.bid = data.at(2).toDouble();
        t.ask = data.at(3).toDouble();
        t.volume = data.at(4).toInt();
        instr.tickList.append(t);
    }

Compiled for release this take 2.5 seconds. I think your machine is twice my speed, so that might equate to 1.25 seconds for you.

Compared to your Rust 0.72 seconds it's not bad! Not as good as I would like, but it should get you going! By all means look into alternative ways of converting that datetime string for improvement.

Good luck, and hope this helps! :)

J.Hilk

int main(int argc, char** argv)
{


    QFile* testFile = new QFile(":/TestFile.txt");


    testFile->open(QFile::ReadOnly);
    QByteArrayList allData = testFile->readAll().split('\n');
    Instrument instr;
    Tick t;

    QElapsedTimer* parseTimer1 = new QElapsedTimer();
    parseTimer1->start();
    for (int ii = 0; ii < allData.size() - 1; ii++)
    {
        QByteArrayList data = allData.at(ii).split(';');
        t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000");
        t.last = data.at(1).toDouble();
        t.bid = data.at(2).toDouble();
        t.ask = data.at(3).toDouble();
        t.volume = data.at(4).toInt();
        instr.tickList.append(t);
    }

    qDebug().noquote() << QString("Qt parse time: %1ms")
        .arg(parseTimer1->elapsed());

    QElapsedTimer* parseTimer2 = new QElapsedTimer();
    parseTimer2->start();
    testFile->reset();
    instr.tickList.clear();
    instr.tickList.reserve(1000000);

    auto DateTimeParser =[](const QStringRef & string) ->QDateTime {
        const QDate date(string.left(4).toInt(), string.mid(4,2).toInt(), string.mid(6.2).toInt());
        const QTime time(string.mid(9,2).toInt(), string.mid(11,2).toInt(), string.mid(13.2).toInt(), string.mid(15.3).toInt());
        QDateTime dt (date,time);
        return dt;
    };

    QTextStream readFile(testFile);
    QString line; line.reserve(100);
    const QChar semicolon(';');
    while(!readFile.atEnd()){
        if(!readFile.readLineInto(&line,100)){
            break;
        }
        const auto result = line.splitRef(semicolon);
        instr.tickList.append({DateTimeParser(result.at(0)),result.at(1).toDouble(), result.at(2).toDouble(), result.at(3).toDouble(),result.at(3).toInt()});
    }
    qDebug().noquote() << QString("Qt parse time: %1ms")
        .arg(parseTimer2->elapsed());
}

result: 1440 ms, but your pc might be faster :D

Christian Ehrlicher

@J-Hilk said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

QByteArrayList allData = testFile->readAll().split('\n');
...
QByteArrayList data = allData.at(ii).split(';');

Sadly QByteArrayView doesn't have yet a split() function as QStringView has - the two lines above are painful slow for big datasets.

TheLumbee

@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.

So thank you for that!

I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.

I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.

I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:

That "naive" means it does not do any local time/daylight etc, conversions.

If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.

J.Hilk

@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:

f any of you are interested, I'll test each of your solutions and provide an update

sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D

But this actually woke me up and got me excited to start my day so thank you

👍 thumbsup

TheLumbee

Also, I'd like to note that I did run both Rust and Qt in release mode. Tried to give Qt the best shot I could. My initial thoughts were some type of buffering/caching Rust did internally but I've attempted tests with C++ on that front and can't match it.

TheLumbee

@DerReisende Didn't actually notice that until now. But with doubles, the performance is almost the same.