Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB
-
Also, I'd like to note that I did run both Rust and Qt in release mode. Tried to give Qt the best shot I could. My initial thoughts were some type of buffering/caching Rust did internally but I've attempted tests with C++ on that front and can't match it.
-
Is there a reason why your Tick class's rust impl uses 32bit floats and your c++ code 64bit doubles?
@DerReisende Didn't actually notice that until now. But with doubles, the performance is almost the same.
-
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Thanks for all the responses! Didn't actually expect much here.
Can't speak for other fora, but we are quality here in this forum :)
If any of you are interested, I'll test each of your solutions and provide an update.
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there. -
@DerReisende Didn't actually notice that until now. But with doubles, the performance is almost the same.
@TheLumbee Using floats instead of doubles should minimize memory usage. And my solution was made with Qt 6.4
-
@TheLumbee Using floats instead of doubles should minimize memory usage. And my solution was made with Qt 6.4
@DerReisende
Purely OoI: with your 6.4 did you just try the originalQDateTime::fromString()as-was, did that have a significant improvement over the "buggy" previous one? -
@DerReisende
Purely OoI: with your 6.4 did you just try the originalQDateTime::fromString()as-was, did that have a significant improvement over the "buggy" previous one?@JonB
QDateTime::fromStringwithout any optimization takes 37 seconds to complete --> slow as hell.
I tested performance ofQDate::fromStringin all Qt6 versions on windows and it NEVER got any faster - thats why I am using the posted cache approach to increase the performance in my app. -
Rust is made for quicker performance. Qt is made for HMI and speed is not the key feature.
If there are bottlenecks in your app, you can add Rust lib to handle them. If you know Rust well, the big banks will like to have you for high-speed trading apps. Rust is getting popular now and it is something: good to know. -
The problem with QDateTime is that for every call the internal format parser (QDateTimeParser, private class) is re-created and needs to re-evaluate the string. This takes a lot of time.
Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
Visit the Qt Academy at https://academy.qt.io/catalog -
The problem with QDateTime is that for every call the internal format parser (QDateTimeParser, private class) is re-created and needs to re-evaluate the string. This takes a lot of time.
@Christian-Ehrlicher
This is true, but you have to reflect on Rust is using itsNaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f")for this and is 30x-odd faster. That is not an "acceptable" difference, and the question is what to do from Qt to get acceptable performance regardless of the reasons? -
@Christian-Ehrlicher
This is true, but you have to reflect on Rust is using itsNaiveDateTime::parse_from_str(dt, "%Y%m%d %H%M%S %f")for this and is 30x-odd faster. That is not an "acceptable" difference, and the question is what to do from Qt to get acceptable performance regardless of the reasons?@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
nd the question is what to do from Qt to get acceptable performance regardless of the reasons?
compare rust with std::get_time() or similar functions. And maybe open a bug report with the findings here.
-
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
So thank you for that!
I've tested this with versions 512, 5.15, 6.0, 6.2.4, and 6.4. Never noticed a major difference between them regarding this issue. Current machine: i7-6700 with 32GB RAM. So not sure what y'all are working with but the results seem promising.
I was previously streaming into a QTextStream then reading line-by-line but came across this post: https://forum.qt.io/topic/98282/parsing-large-big-text-files-quickly and a couple of others that suggested that is more expensive that using a QByteArray. I didn't notice much difference to be quite honest.
I did comment out the QDateTime parsing just to check and it was a significant improvement. Not quite like Rust but I'll attribute that to @JonB comment:
That "naive" means it does not do any local time/daylight etc, conversions.If any of you are interested, I'll test each of your solutions and provide an update. But this actually woke me up and got me excited to start my day so thank you.
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
Just tested my solution on windows 11 with VS 2022...
My cached solution finishes in 19 seconds (compared to approx. 7 on macOS), the original version takes 575 seconds...OMG! Release mode with AVX2 enabled. -
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
f any of you are interested, I'll test each of your solutions and provide an update
sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D
But this actually woke me up and got me excited to start my day so thank you
👍 thumbsup
@J-Hilk said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
sure, feedback is always appreciated! Nothing more discouraging than getting ghosted after providing an answer :D
Thanks for the answer, but QStringRef is not available in Qt 6.4. Can't test this solution. Although I did replace with QString, but the performance was middling.
-
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Thanks for all the responses! Didn't actually expect much here.
Can't speak for other fora, but we are quality here in this forum :)
If any of you are interested, I'll test each of your solutions and provide an update.
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there.@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Please do :) Note that mine will be least the code for the greatest speed benefit :) You get your money back if you don't think so ;-) Note that really the file reading, buffering etc. is marginal to the whole, the single most important thing is that
QDateTime::fromString()is "unusably bad" for performance, unless you move to Qt 6.3+ and say it's a lot better there.Of all the solutions, this one has the best performance, for Qt at least. Results are around 3900ms. Still quite a bit slower than Rust. Issue is, I'm looking at parsing files 1GB+ in size so that really adds up when its 5x the time to complete.
-
Rust is made for quicker performance. Qt is made for HMI and speed is not the key feature.
If there are bottlenecks in your app, you can add Rust lib to handle them. If you know Rust well, the big banks will like to have you for high-speed trading apps. Rust is getting popular now and it is something: good to know.@JoeCFD said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Rust is made for quicker performance. Qt is made for HMI and speed is not the key feature.
If there are bottlenecks in your app, you can add Rust lib to handle them. If you know Rust well, the big banks will like to have you for high-speed trading apps. Rust is getting popular now and it is something: good to know.I'm beginning to notice that. But I haven't even found a C/C++ implementation that competes, at least in this aspect alone, with what Rust is doing.
I am actually considering just creating a Rust lib to handle this. Not sure what other options I have if I'm looking to get this kind of performance. I know Rust well enough so let the big banks know I'm available ;-).
Given that I'm expecting much larger file sizes, using Rust would be the best choice I think. Maybe a solution will arise with C++ that can handle it.
-
@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
nd the question is what to do from Qt to get acceptable performance regardless of the reasons?
compare rust with std::get_time() or similar functions. And maybe open a bug report with the findings here.
@Christian-Ehrlicher said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
nd the question is what to do from Qt to get acceptable performance regardless of the reasons?
compare rust with std::get_time() or similar functions. And maybe open a bug report with the findings here.
I'll try to get to this at some point. Just seems insane that the performance between Rust and C++ is so different when it comes to parsing these files.
-
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@JonB @J-Hilk @DerReisende Thanks for all the responses! Didn't actually expect much here. I apologize for not providing more details. I've been dealing with this file parsing issue in C++ for years. Same code in Windows takes >100x times to complete rather than using Linux for some odd reason which I've posted in C++ forums prior to using Qt, but what you've provided is actually the first significant improvement I've ever seen.
Just tested my solution on windows 11 with VS 2022...
My cached solution finishes in 19 seconds (compared to approx. 7 on macOS), the original version takes 575 seconds...OMG! Release mode with AVX2 enabled.@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Just tested my solution on windows 11 with VS 2022...
My cached solution finishes in 19 seconds (compared to approx. 7 on macOS), the original version takes 575 seconds...OMG! Release mode with AVX2 enabled.Yeah, there has to be something internally with the way Windows handles I/O for the massive difference. I've been trying to narrow down the issue for years without success. Interesting point though.
-
One issue I'm facing is still QDateTime parsing. Even if I create the Rust lib, it doesn't have QDateTime and std::chrono interoperability. So I was thinking of returning a Tick array with a const char* for DateTime but then I'd still have to go through the entire Tick array and convert all the values to QDateTimes anyway. This sort of bring back the initial problems.
Honestly, this performance thing is the big reason I'm struggling to decide between this, https://github.com/fzyzcjy/flutter_rust_bridge, and Qt/QML for projects. I love Qt but performance is huge for me. Just looking for a justification here.
-
One issue I'm facing is still QDateTime parsing. Even if I create the Rust lib, it doesn't have QDateTime and std::chrono interoperability. So I was thinking of returning a Tick array with a const char* for DateTime but then I'd still have to go through the entire Tick array and convert all the values to QDateTimes anyway. This sort of bring back the initial problems.
Honestly, this performance thing is the big reason I'm struggling to decide between this, https://github.com/fzyzcjy/flutter_rust_bridge, and Qt/QML for projects. I love Qt but performance is huge for me. Just looking for a justification here.
@TheLumbee Why don't you just parse the date string in a rust function which returns the number of msecs sinc midnight 1970? This way you only have to store a qint64 variable. If you need a QDateTime instance then just call
QDateTime::fromMSecsSinceEpochwhen needed. This should be fast. -
@TheLumbee Why don't you just parse the date string in a rust function which returns the number of msecs sinc midnight 1970? This way you only have to store a qint64 variable. If you need a QDateTime instance then just call
QDateTime::fromMSecsSinceEpochwhen needed. This should be fast.@DerReisende said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@TheLumbee Why don't you just parse the date string in a rust function which returns the number of msecs sinc midnight 1970? This way you only have to store a qint64 variable. If you need a QDateTime instance then just call
QDateTime::fromMSecsSinceEpochwhen needed. This should be fast.Don't know why I didn't think of that. I'll write the Rust lib, and hopefully have some great results. I'll let you know.
-
@Christian-Ehrlicher said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
@JonB said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
nd the question is what to do from Qt to get acceptable performance regardless of the reasons?
compare rust with std::get_time() or similar functions. And maybe open a bug report with the findings here.
I'll try to get to this at some point. Just seems insane that the performance between Rust and C++ is so different when it comes to parsing these files.
@TheLumbee said in Rust file parsing significantly faster than Qt/C++ file parsing. Solutions for Qt implementation wanted. File size: 68.5 MB:
Just seems insane that the performance between Rust and C++ is so different when it comes to parsing these files.
Simply read my explanation on what needs to be done when parsing a datetime string. This can be optimized in Qt but if someone needs speed other solutions are much better (see @J-Hilk 's solution by bypassing QDateTime::fromString() at all) so noone is doing it.
