Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB
-
@JonB
I did the following for parsing QDate objects in one of my projectsQDate TreeModel::convert_string_to_date(QString &inDateStr, tsl::hopscotch_map<QString, QDate> &date_cache) noexcept(false) { if (date_cache.contains(inDateStr)) return date_cache.at(inDateStr); //else we need to compute QDate result; if (inDateStr.contains('.')) { // we have dd.mm.yyy result = QDate::fromString(inDateStr, "dd.MM.yyyy"); } else if (inDateStr.contains('-')) { //we have yyyy-mm-dd aka. ISO-8601 format result = QDate::fromString(inDateStr, Qt::ISODate); } else { throw DateConversionException("Unknown Date Type: " + inDateStr.toStdString()); } date_cache.insert({inDateStr, result}); return result; }with:
tsl::hopscotch_map<QString, QDate> date_cache{DATE_CACHE_SIZE};I used tsl::hopscotch_map as well as it is faster than std::unordered_map but it could be easily replaced.
Using something for the date parsing on col 1 of the testfile should greatly enhance runtime performance sind cache hit rate there should be really high.
wrote on 30 Sept 2022, 10:43 last edited by@DerReisende
Ah, OK, you mean you are just caching previously met dates in the cache. The OP's file has "thousands" of different dates with times. Actually, inspecting its million entries cover a 24:00:00 hour period from 6:00am to 5:59am. He could think about adapting your date principle, don't know if he wants to allow for local time which could have an hour "daylight" skip potentially or can ignore that or times are UTC? Anyway there is certainly something which could be done with these million entries to avoid having tofromString(data.at(0), "yyyyMMdd HHmmss zzz0000")on each one...! -
@DerReisende
Ah, OK, you mean you are just caching previously met dates in the cache. The OP's file has "thousands" of different dates with times. Actually, inspecting its million entries cover a 24:00:00 hour period from 6:00am to 5:59am. He could think about adapting your date principle, don't know if he wants to allow for local time which could have an hour "daylight" skip potentially or can ignore that or times are UTC? Anyway there is certainly something which could be done with these million entries to avoid having tofromString(data.at(0), "yyyyMMdd HHmmss zzz0000")on each one...!wrote on 30 Sept 2022, 11:46 last edited by@JonB My quick and dirty take on the problem which brings runtime on my machine from 37seconds down to 6 seconds:
struct Tick { QDateTime dt; float last{}; float bid{}; float ask{}; quint32 volume{}; }; struct Instrument { QList<Tick> tickList; }; static QDate convert_string_to_date(const QString& inDateStr) { static std::unordered_map<QString, QDate> date_cache; if (date_cache.contains(inDateStr)) return date_cache.at(inDateStr); //else we need to compute QDate result = QDate::fromString(inDateStr, "yyyyMMdd"); date_cache.insert({inDateStr, result}); return result; } static QTime convert_string_to_time(const QString& inDateStr) { static std::unordered_map<QString, QTime> time_cache; //else we need to compute const auto list = inDateStr.split(' '); QTime baseTime; if (time_cache.contains(list[0])) baseTime = time_cache.at(list[0]); else { baseTime = QTime::fromString(list[0], "HHmmss"); time_cache.insert({list[0], baseTime}); } return baseTime.addMSecs(QTime::fromString(list[1], "zzz0000").msec()); } static QDateTime convert_string_to_datetime(QString &inDateStr) { const auto list = inDateStr.split(' '); const QDate date = convert_string_to_date(list[0]); const QTime time = convert_string_to_time(list[1] + ' ' + list[2]); QDateTime result(date, time); return result; } int main(int argc, char **argv) { QElapsedTimer parseTimer; parseTimer.start(); QFile testFile("TestFile.txt"); testFile.open(QFile::ReadOnly); QByteArrayList allData = testFile.readAll().split('\n'); Instrument instr; for (int ii = 0; ii < allData.size() - 1; ii++) { QByteArrayList data = allData.at(ii).split(';'); Tick t; QString dt_str = data.at(0); t.dt = convert_string_to_datetime(dt_str); /*const auto orig = QDateTime::fromString(dt_str, "yyyyMMdd HHmmss zzz0000"); if (t.dt != orig) { qDebug() << "orig: " << orig << " versus: " << t.dt; std::exit(99); }*/ t.last = data.at(1).toFloat(); t.bid = data.at(2).toFloat(); t.ask = data.at(3).toFloat(); t.volume = data.at(4).toInt(); instr.tickList.append(t); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer.elapsed()); return 0; }Still a lot slower than rust :(
-
I have written the same file parsing implementation in Rust and in Qt, but Rust completes the task unbelievably quicker. I'd love to just implement this in C++ because I haven't taken the time to create a Rust lib for the file mainly because I'm not extremely proficient in the language. But I'm curious if anyone has any answers/solutions to solving this issue. It's actually mind-blowing the difference, but there has to be a C++ solution that is comparable in performance.
Rust code here:
use chrono::NaiveDateTime; use howlong::HighResolutionTimer; use std::env; use std::fs::File; use std::io::prelude::*; use std::io::BufReader; #[derive(PartialEq)] pub struct Tick { pub dt: NaiveDateTime, pub last: f32, pub bid: f32, pub ask: f32, pub volume: u32 } impl Tick { pub fn init(dt: &str, last: f32, bid: f32, ask: f32, volume: u32) -> Tick { let date_time = NaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f") .expect("Invalid time signature"); Tick { dt: date_time, last, bid, ask, volume } } } #[derive(PartialEq)] pub struct Instrument { pub tick_list: Vec<Tick>, } impl Instrument { pub fn new() -> Instrument { Instrument { tick_list: Vec::new() } } fn add_tick_from_str(&mut self, data: &str) { let temp: Vec<&str> = data.split(';').collect(); if temp.len() != 5 { println!("Tick data does not contain enough information."); return; } let tick = Tick::init(temp[0], temp[1].parse().expect("Unable to parse last price"), temp[2].parse().expect("Unable to parse bid price"), temp[3].parse() .expect("Unable to parse ask price"), temp[4].parse().expect("Unable to parse volume")); self.tick_list.push(tick); } pub fn import_tick_data(&mut self, file_name: &str) { let file = File::open(file_name) .expect("Tick file not found to import instrument."); let buf_reader = BufReader::new(file); for line in buf_reader.lines() { self.add_tick_from_str(&line.unwrap()); } } } fn main() { let timer = HighResolutionTimer::new(); let args: Vec<String> = env::args().collect(); let mut instr = Instrument::new(); instr.import_tick_data(&args[1].as_str()); println!("Rust parse time: {:?}", timer.elapsed()); }Qt Code here:
#include <QByteArray> #include <QDateTime> #include <QDebug> #include <QElapsedTimer> #include <QFile> struct Tick { QDateTime dt; double last, bid, ask; quint32 volume; }; struct Instrument { QList<Tick> tickList; }; int main(int argc, char** argv) { QElapsedTimer* parseTimer = new QElapsedTimer(); parseTimer->start(); if (argc != 2) { qDebug().noquote() << "File argument is required"; return -1; } QFile* testFile = new QFile(argv[1]); if (!testFile->exists()) { qDebug().noquote() << "Test file does not exist"; return -2; } testFile->open(QFile::ReadOnly); QByteArrayList allData = testFile->readAll().split('\n'); Instrument instr; Tick t; for (int ii = 0; ii < allData.size() - 1; ii++) { QByteArrayList data = allData.at(ii).split(';'); t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000"); t.last = data.at(1).toDouble(); t.bid = data.at(2).toDouble(); t.ask = data.at(3).toDouble(); t.volume = data.at(4).toInt(); instr.tickList.append(t); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer->elapsed()); }Results here:
Rust parse time: 722.340226ms Qt parse time: 30452msFile used for parsing:
https://drive.google.com/file/d/1A7mqAeC238yMoveg9ausZ5DRu5kNfKRp/view?usp=sharing
wrote on 30 Sept 2022, 11:58 last edited by JonB@TheLumbee
OK, here is my final code offering.I use
QTextStreamon the file to read a line at a time. As suspected, this makes little difference to timing over reading the whole file into memory and splitting it, but is a lot gentler on memory. It is also more favorable to e.g. Windows text files. I assume it's what the Rust will be doing, and I would do same if I were you.I cannot use a "fixed"
QDataTime::fromString(), and i am assuming you cannot (if you can, try it). If you look at your Rust you havedt: NaiveDateTime. That "naive" means it does not do any local time/daylight etc, conversions. (https://docs.rs/chrono/latest/chrono/naive/struct.NaiveDateTime.html# "ISO 8601 combined date and time without timezone"). I can't usestd::chrono::parseas that is C++20, I can't usestrptime()as that is available for Windows cross-platform. I can't be bothered to try to write a "cached lookup", I don't know whether Rust internals use anything like that, and it does not "scale" well if we can't make assumptions about what dates we will meet. For now/illustration I have just gone forsscanf()to read your fixed-format date/time field inputs (don't know whetherQString::toInt()onQString::mid()for each fixed-length section might be faster, didn't try).My code --- with your stuff commented out so you can see what i have changed --- reads:
Instrument instr; Tick t; testFile->open(QFile::ReadOnly); // QByteArrayList allData = testFile->readAll().split('\n'); QTextStream ts(testFile); QString line; while (ts.readLineInto(&line)) // for (int ii = 0; ii < allData.size() - 1; ii++) { // QByteArrayList data = allData.at(ii).split(';'); QStringList data = line.split(';'); // t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000"); int year, month, day, hour, minute, second, milli; sscanf(data.at(0).toLocal8Bit().constData(), "%4d%2d%2d %2d%2d%2d %3d0000", &year, &month, &day, &hour, &minute, &second, &milli); t.dt = QDateTime(QDate(year, month, day), QTime(hour, minute, second, milli), Qt::UTC); t.last = data.at(1).toDouble(); t.bid = data.at(2).toDouble(); t.ask = data.at(3).toDouble(); t.volume = data.at(4).toInt(); instr.tickList.append(t); }Compiled for release this take 2.5 seconds. I think your machine is twice my speed, so that might equate to 1.25 seconds for you.
Compared to your Rust 0.72 seconds it's not bad! Not as good as I would like, but it should get you going! By all means look into alternative ways of converting that datetime string for improvement.
Good luck, and hope this helps! :)
-
int main(int argc, char** argv) { QFile* testFile = new QFile(":/TestFile.txt"); testFile->open(QFile::ReadOnly); QByteArrayList allData = testFile->readAll().split('\n'); Instrument instr; Tick t; QElapsedTimer* parseTimer1 = new QElapsedTimer(); parseTimer1->start(); for (int ii = 0; ii < allData.size() - 1; ii++) { QByteArrayList data = allData.at(ii).split(';'); t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000"); t.last = data.at(1).toDouble(); t.bid = data.at(2).toDouble(); t.ask = data.at(3).toDouble(); t.volume = data.at(4).toInt(); instr.tickList.append(t); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer1->elapsed()); QElapsedTimer* parseTimer2 = new QElapsedTimer(); parseTimer2->start(); testFile->reset(); instr.tickList.clear(); instr.tickList.reserve(1000000); auto DateTimeParser =[](const QStringRef & string) ->QDateTime { const QDate date(string.left(4).toInt(), string.mid(4,2).toInt(), string.mid(6.2).toInt()); const QTime time(string.mid(9,2).toInt(), string.mid(11,2).toInt(), string.mid(13.2).toInt(), string.mid(15.3).toInt()); QDateTime dt (date,time); return dt; }; QTextStream readFile(testFile); QString line; line.reserve(100); const QChar semicolon(';'); while(!readFile.atEnd()){ if(!readFile.readLineInto(&line,100)){ break; } const auto result = line.splitRef(semicolon); instr.tickList.append({DateTimeParser(result.at(0)),result.at(1).toDouble(), result.at(2).toDouble(), result.at(3).toDouble(),result.at(3).toInt()}); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer2->elapsed()); }result: 1440 ms, but your pc might be faster :D
-
int main(int argc, char** argv) { QFile* testFile = new QFile(":/TestFile.txt"); testFile->open(QFile::ReadOnly); QByteArrayList allData = testFile->readAll().split('\n'); Instrument instr; Tick t; QElapsedTimer* parseTimer1 = new QElapsedTimer(); parseTimer1->start(); for (int ii = 0; ii < allData.size() - 1; ii++) { QByteArrayList data = allData.at(ii).split(';'); t.dt = QDateTime::fromString(data.at(0), "yyyyMMdd HHmmss zzz0000"); t.last = data.at(1).toDouble(); t.bid = data.at(2).toDouble(); t.ask = data.at(3).toDouble(); t.volume = data.at(4).toInt(); instr.tickList.append(t); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer1->elapsed()); QElapsedTimer* parseTimer2 = new QElapsedTimer(); parseTimer2->start(); testFile->reset(); instr.tickList.clear(); instr.tickList.reserve(1000000); auto DateTimeParser =[](const QStringRef & string) ->QDateTime { const QDate date(string.left(4).toInt(), string.mid(4,2).toInt(), string.mid(6.2).toInt()); const QTime time(string.mid(9,2).toInt(), string.mid(11,2).toInt(), string.mid(13.2).toInt(), string.mid(15.3).toInt()); QDateTime dt (date,time); return dt; }; QTextStream readFile(testFile); QString line; line.reserve(100); const QChar semicolon(';'); while(!readFile.atEnd()){ if(!readFile.readLineInto(&line,100)){ break; } const auto result = line.splitRef(semicolon); instr.tickList.append({DateTimeParser(result.at(0)),result.at(1).toDouble(), result.at(2).toDouble(), result.at(3).toDouble(),result.at(3).toInt()}); } qDebug().noquote() << QString("Qt parse time: %1ms") .arg(parseTimer2->elapsed()); }result: 1440 ms, but your pc might be faster :D
@J-Hilk said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
QByteArrayList allData = testFile->readAll().split('\n');
...
QByteArrayList data = allData.at(ii).split(';');Sadly QByteArrayView doesn't have yet a split() function as QStringView has - the two lines above are painful slow for big datasets.
-
wrote on 30 Sept 2022, 12:56 last edited by
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
-
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
f any of you are interested, I'll test each of your solutions and provide an update
sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D
But this actually woke me up and got me excited to start my day so thank you
👍 thumbsup
-
wrote on 30 Sept 2022, 12:59 last edited by
Also, I'd like to note that I did run both Rust and Qt in release mode. Tried to give Qt the best shot I could. My initial thoughts were some type of buffering/caching Rust did internally but I've attempted tests with C++ on that front and can't match it.
-
Is there a reason why your Tick class's rust impl uses 32bit floats and your c++ code 64bit doubles?
wrote on 30 Sept 2022, 13:02 last edited by@DerReisende Didn't actually notice that until now. But with doubles, the performance is almost the same.
-
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
wrote on 30 Sept 2022, 14:04 last edited by JonB@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Thanks for all the responses! Didn't actually expect much here.
Can't speak for other fora, but we are quality here in this forum :)
If any of you are interested, I'll test each of your solutions and provide an update.
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there. -
@DerReisende Didn't actually notice that until now. But with doubles, the performance is almost the same.
wrote on 30 Sept 2022, 14:06 last edited by DerReisende@TheLumbee Using floats instead of doubles should minimize memory usage. And my solution was made with Qt 6.4
-
@TheLumbee Using floats instead of doubles should minimize memory usage. And my solution was made with Qt 6.4
wrote on 30 Sept 2022, 14:09 last edited by@DerReisende
Purely OoI: with your 6.4 did you just try the originalQDateTime::fromString()as-was, did that have a significant improvement over the "buggy" previous one? -
@DerReisende
Purely OoI: with your 6.4 did you just try the originalQDateTime::fromString()as-was, did that have a significant improvement over the "buggy" previous one?wrote on 30 Sept 2022, 14:18 last edited by@JonB
QDateTime::fromStringwithout any optimization takes 37 seconds to complete --> slow as hell.
I tested performance ofQDate::fromStringin all Qt6 versions on windows and it NEVER got any faster - thats why I am using the posted cache approach to increase the performance in my app. -
wrote on 30 Sept 2022, 14:29 last edited by JoeCFD
Rust is made for quicker performance. Qt is made for HMI and speed is not the key feature.
If there are bottlenecks in your app, you can add Rust lib to handle them. If you know Rust well, the big banks will like to have you for high-speed trading apps. Rust is getting popular now and it is something: good to know. -
The problem with QDateTime is that for every call the internal format parser (QDateTimeParser, private class) is re-created and needs to re-evaluate the string. This takes a lot of time.
-
The problem with QDateTime is that for every call the internal format parser (QDateTimeParser, private class) is re-created and needs to re-evaluate the string. This takes a lot of time.
wrote on 30 Sept 2022, 14:55 last edited by@Christian-Ehrlicher
This is true, but you have to reflect on Rust is using itsNaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f")for this and is 30x-odd faster. That is not an "acceptable" difference, and the question is what to do from Qt to get acceptable performance regardless of the reasons? -
@Christian-Ehrlicher
This is true, but you have to reflect on Rust is using itsNaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f")for this and is 30x-odd faster. That is not an "acceptable" difference, and the question is what to do from Qt to get acceptable performance regardless of the reasons?Lifetime Qt Championwrote on 30 Sept 2022, 14:57 last edited by Christian Ehrlicher@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
nd the question is what to do from Qt to get acceptable performance regardless of the reasons?
compare rust with std::get_time() or similar functions. And maybe open a bug report with the findings here.
-
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
wrote on 30 Sept 2022, 15:49 last edited by@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
Just tested my solution on windows 11 with VS 2022...
My cached solution finishes in 19 seconds (compared to approx. 7 on macOS), the original version takes 575 seconds...OMG! Release mode with AVX2 enabled. -
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
f any of you are interested, I'll test each of your solutions and provide an update
sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D
But this actually woke me up and got me excited to start my day so thank you
👍 thumbsup
wrote on 30 Sept 2022, 16:52 last edited by@J-Hilk said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D
Thanks for the answer, but QStringRef is not available in Qt 6.4. Can't test this solution. Although I did replace with QString, but the performance was middling.
-
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Thanks for all the responses! Didn't actually expect much here.
Can't speak for other fora, but we are quality here in this forum :)
If any of you are interested, I'll test each of your solutions and provide an update.
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there.wrote on 30 Sept 2022, 16:54 last edited by TheLumbee@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there.Of all the solutions, this one has the best performance, for Qt at least. Results are around 3900ms. Still quite a bit slower than Rust. Issue is, I'm looking at parsing files 1GB+ in size so that really adds up when its 5x the time to complete.
21/64