QString::indexOf(QRegularExpression) is slower than indexOf(QRegExp)

johnson54

In Qt 5.15.2, "str" is a QString, the length of "str" is commonly 32KBytes, when I use str.indexOf, I found
str.indexOf(QRegularExpression(splitStr)) is much more slowly than str.indexOf(QRegExp(splitStr)), is there something wrong?

JonB

@johnson54
Can you give your comparative timings so we understand the difference? And you do understand that means for many repeats, not just for one off?

johnson54

Here is my code and result. the ‘gnss.log’ file is about 12MB。

QRegExp  pkg_num =  1229 elapsed =  141 ms
QRegularExpression  pkg_num =  1229 elapsed =  9168 ms

#include <QCoreApplication>
#include <QtDebug>
#include <QFile>
#include <QRegularExpression>
#include <QRegExp>
#include <QElapsedTimer>

constexpr int READ_SIZE = 32*1024*1024;

int TestSplit(int b_use_reg_exp)
{
    QFile file("E:/gnss.log");
    QString split_str("GGA");
    if (file.open(QIODevice::ReadOnly) == false)
        return -1;
    auto p_buffer = new char[READ_SIZE];
    int finish_flag = 0;
    QString str_buff;
    int last_pos = 0;
    int pkg_num = 0;
    int pos = 0;
    while (1)
    {
        auto file_size = file.read(p_buffer, READ_SIZE);
        if (file_size < READ_SIZE)
            finish_flag = 1;
        str_buff.append(QString(QLatin1String(p_buffer, file_size)));
        if (b_use_reg_exp)
            pos = str_buff.indexOf(QRegExp(split_str));
        else
            pos = str_buff.indexOf(QRegularExpression(split_str));

        int offset_pos = -1;
        while (pos != -1)
        {
            offset_pos = pos;
            pkg_num++;
            if (b_use_reg_exp)
                pos = str_buff.indexOf(QRegExp(split_str), pos+1);
            else
                pos = str_buff.indexOf(QRegularExpression(split_str), pos+1);
        }
        str_buff.remove(0, offset_pos+1);
        if (offset_pos != -1)
            last_pos += offset_pos + 1;
        if (finish_flag == 1)
            break;
    }
    file.close();
    delete []p_buffer;
    return pkg_num;
}

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    QElapsedTimer t_time;
    int pkg_num;
    t_time.start();
    pkg_num = TestSplit(true);
    qDebug() << "QRegExp " << "pkg_num = " << pkg_num << "elapsed = " << t_time.elapsed() << "ms";
    t_time.start();
    pkg_num = TestSplit(false);
    qDebug() << "QRegularExpression " << "pkg_num = " << pkg_num << "elapsed = " << t_time.elapsed() << "ms";
    return a.exec();
}

JonB

@johnson54
Well that does seem to be a pretty considerable difference!

What you did not say, but show now, is that your splitter regular expression is not a regular expression, just a constant string ("GGA"). I am not defending the timing, but it might be interesting to know whether this kind of difference also applies when the regular expression actually has some work to do? I assume you realise for the splitter you have you can just use int QString::indexOf(QLatin1String str, int from = 0, Qt::CaseSensitivity cs = Qt::CaseSensitive) const, and you ought time that too.

The other thought I have, and I don't know if this is a "thing". You construct the regular expression each time as a parameter to indexOf(). Regular expression construction can be expensive. Try taking the QRegExp(split_str) & QRegularExpression(split_str) outside the loop and use the already-constructed reg exp each time. Any difference? This should be done anyway when the reg exp does not change in the loop.

johnson54

@JonB
I changed the split_str, make it regular expression (GG.,), and make the construction of reg exp outside the loop.

QRegExp  pkg_num =  1229 elapsed =  63 ms
QRegularExpression  pkg_num =  1229 elapsed =  7716 ms

#include <QCoreApplication>
#include <QtDebug>
#include <QFile>
#include <QRegularExpression>
#include <QRegExp>
#include <QElapsedTimer>

constexpr int READ_SIZE = 32*1024*1024;

int TestSplit(int b_use_reg_exp)
{
    QFile file("E:/gnss.log");
    QString split_str("GG.,");
    if (file.open(QIODevice::ReadOnly) == false)
        return -1;
    auto p_buffer = new char[READ_SIZE];
    int finish_flag = 0;
    QString str_buff;
    int last_pos = 0;
    int pkg_num = 0;
    int pos = 0;
    auto reg_exp = QRegExp(split_str);
    auto regular_expression = QRegularExpression(split_str);
    while (1)
    {
        auto file_size = file.read(p_buffer, READ_SIZE);
        if (file_size < READ_SIZE)
            finish_flag = 1;
        str_buff.append(QString(QLatin1String(p_buffer, file_size)));
        if (b_use_reg_exp)
            pos = str_buff.indexOf(reg_exp);
        else
            pos = str_buff.indexOf(regular_expression);

        int offset_pos = -1;
        while (pos != -1)
        {
            offset_pos = pos;
            pkg_num++;
            if (b_use_reg_exp)
                pos = str_buff.indexOf(reg_exp, pos+1);
            else
                pos = str_buff.indexOf(regular_expression, pos+1);
        }
        str_buff.remove(0, offset_pos+1);
        if (offset_pos != -1)
            last_pos += offset_pos + 1;
        if (finish_flag == 1)
            break;
    }
    file.close();
    delete []p_buffer;
    return pkg_num;
}

int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);
    QElapsedTimer t_time;
    int pkg_num;
    t_time.start();
    pkg_num = TestSplit(true);
    qDebug() << "QRegExp " << "pkg_num = " << pkg_num << "elapsed = " << t_time.elapsed() << "ms";
    t_time.start();
    pkg_num = TestSplit(false);
    qDebug() << "QRegularExpression " << "pkg_num = " << pkg_num << "elapsed = " << t_time.elapsed() << "ms";
    return a.exec();
}

johnson54

I use QRegExp instead of QRegularExpression in Qt 5.15, when porting to Qt 6, the QString::indexOf(QRegExp) is deprecated. I changed to QString::indexOf(QRegularExpression), but the performance is terrible. So I tested both in Qt 5.15 and found there is considerable difference in performance. However, many people said QRegularExpression enjoys better performance, I want to know if I use it wrong.

JonB

@johnson54
Interesting findings :)

It's nice to see that taking either regular expression construction out of the loop saves around a second, isn't it?!

But your difference remains, and now it's an even greater percentage. More than 100x is not good! So much so that I hope one of our Qt experts will care to comment further....

However, many people said QRegularExpression enjoys better performance, I want to know if I use it wrong.

Not from your example, apparently! You are using it fine.

SGaist

Hi,

Do you have the same performance hit with both Qt 6 and Qt 5 ?

Which version of Qt 6 did you try ?

On which platform ?

jeremy_k

Calling QRegularExpression::optimize() prior to entering the loop might help. Using QRegularExpression::globalMatch() instead of QString::indexOf() is also worth investigating.

JonB

I note we are told:

The QRegularExpression class introduced in Qt 5 is a big improvement upon QRegExp, in terms of APIs offered, supported pattern syntax and speed of execution

:)

@johnson54
You have not said: are you running these tests compiled for debug? If so, can you see what the timings are if you compile for release?

johnson54

@JonB
In Qt 5.15.2, Windows 10, MSVC 2019-32bit
Release Mode:

QRegExp  pkg_num =  1229 elapsed =  44 ms
QRegularExpression  pkg_num =  1229 elapsed =  4142 ms

Debug Mode:

QRegExp  pkg_num =  1229 elapsed =  63 ms
QRegularExpression  pkg_num =  1229 elapsed =  7716 ms

johnson54

@SGaist
In Qt6, QRegularExpression has the same performance as Qt 5.15.2 (Windows 10, MSVC 2019-32bit).
I use Qt 6.2, Windows 10, MSVC 2019-64bit, Release Mode, the result is

QRegularExpression  pkg_num =  1229 elapsed =  4385 ms

Although I added "greaterThan(QT_MAJOR_VERSION, 5): QT += core5compat" in .pro
I cannot use QString::indexOf(QRegExp) in Qt 6.2

johnson54

@jeremy_k I put 'regular_expression.optimize();' before the whole loop, and it makes no help.