[SOLVED] Suggest a faster way for replacing custom parameters in a string

SGaist

Hi,

Can you show an example of string that your are building ?

T3STY

The string could be anything, really. It could include any number of parameters and other text. This is one:

"[%HH:%mm:%s.%zzz] operation completed (%EE) %M"

And that's the code I'm using at the moment:

void consoleWidget::print(QVariant message, errorLevel EL){
    // using a QVariant to allow printing both strings and numeric data types 
    // get current local time
    QTime curTime(QTime::currentTime());
    
    // error level messages; el_msg[0] must not be empty (using a space) to prevent QString::at() crashing the application ->
    const QString el_msg[4] = {" ", "info", "warning", "error"};
    
    // replace parameters with data with the format specified in _format
    // _format is a private (pointer to) QString storing the message format to print, such as:
    // "[%HH:%mm:%s.%zzz] operation completed (%EE) %M"
    QString tmpMsg(*_format);
    // hour, leading zero, 00-23 or 01-12AM/PM
    tmpMsg.replace("%hh", curTime.toString("hh"));
    // hour, 0-23 or 1-12AM/PM
    tmpMsg.replace("%h",  curTime.toString("h") );
    // hour, 0-23
    tmpMsg.replace("%HH", curTime.toString("HH"));
    // minutes, 0-59
    tmpMsg.replace("%H",  curTime.toString("H") );
    // minutes, leading 0, 00-59
    tmpMsg.replace("%mm", curTime.toString("mm"));
    // hour, leading 0, 00-23
    tmpMsg.replace("%m",  curTime.toString("m") );
    // seconds, leading 0, 00-59
    tmpMsg.replace("%ss", curTime.toString("ss"));
    // seconds, 0-59
    tmpMsg.replace("%s",  curTime.toString("s") );
    // milliseconds, leading 0, 000-999
    tmpMsg.replace("%zzz",curTime.toString("zzz"));
    // milliseconds, 0-999
    tmpMsg.replace("%z",  curTime.toString("z") );
    // am/pm (lowercase)
    tmpMsg.replace("%a",  curTime.toString("a") );
    // AM/PM (UPPERCASE)
    tmpMsg.replace("%A",  curTime.toString("A") );
    // timezone
    tmpMsg.replace("%t",  curTime.toString("t") );
    
    // error level, lowercase, extended:    error, warning, info, (empty)
    tmpMsg.replace("%ee",  el_msg[EL]);
    // error level, lowercase, abbreviated: e, w, i, (empty)
    tmpMsg.replace("%e" ,  el_msg[EL].at(0));
    // error level, UPPERCASE, extended:    ERROR, WARNING, INFO, (empty)
    tmpMsg.replace("%EE",  el_msg[EL].toUpper());
    // error level, UPPERCASE, abbreviated: E, W, I, (empty)
    tmpMsg.replace("%E" ,  el_msg[EL].at(0).toUpper());
    
    // replace user message parameter with data
    // Most QVariant stored types can be converted to QString, it's safe not to check for conversion availability
    tmpMsg.replace("%M", message.toString());
    
    // will be replaced with appendHtml()
    this->appendPlainText(tmpMsg);
}

mrjj

@T3STY said:

"[%HH:%mm:%s.%zzz] operation completed (%EE) %M"

If that is a typically length of your source string then something like

tmpMsg.replace("%a", " <b>" + curTime.toString("a") + "</b>");

Should be fast enough ?
Or Am I completely misunderstanding the question ?

T3STY

That is what I was thinking to use. However, I was asking if there was anything that would replace all the parameters at once.
Replacing strings so many times with QString::replace() causes a lot of memory reallocation in order to expand the string size to fit requirements, which also causes more CPU usage. While increasing the string size to twice every allocation is fast enough, doing so 3 or 4 times for 30 strings causes a huge performance loss.
So, if there was any way to tell QString::replace() (or some other method) about more things to replace at once I think it would definitely be faster and cheaper for the memory.

BTW, The format string could be made of even 20 parameters with extra (non parametric) words. Consider:

"h: %h | hh: %hh | H: %h | HH: %HH | m: %m | mm: %mm | s: %s | ss: %ss | z: %z | zzz: %zzz | a: %a | A: %A |  | t: %t | e: %e | E: %E | ee: %ee | EE: %EE |  | %M"

which includes only part of the parameters I want to implement. Also %M is a user message that could be even hundred of characters long. There is no typical string length that I could consider to reserve memory in advance.

Chris Kawa

I'd say sum up the total length of the parameters and call reserve on the string before doing the replacements.

T3STY

That's a good idea! Thanks :)
But it will still not solve the number replacements I should make... I have to think about some other strategy...

onek24

@T3STY

I can't provide any idea or solution, but maybe i can give you a hint - something i would try: Check out the source-code of the QString class and see what ::args(...) does. Maybe you can use Qt's code and implement the functionality to have different kinds of parameter-types(%s, %EE, ...).

T3STY

Unfortunately, modifying Qt's source is not a viable option because of the license I'm using (LGPL). But I think I found some strategy to maintain some const-ness in the initial string length. I am using pre-formatted strings with "<FONT class="something">%1</FONT>" html tags, so most formatting is done with CSS ( QPlainTextEdit::document::setDefaultStyleSheet() ). This will reduce the number of reallocations (before replacement) to 1 (the %1 argument) with QString::arg(), and then I replace all data in the final string. I'm also thinking about other possible tricks like asking the time to QTime::getCurrentTime() with a separate string, so I would only replace time once instead of for each parameter.

Chris Kawa

But it will still not solve the number replacements I should make

You'll need to convert the numbers to strings anyway, so you can do that first and get the length of these strings.
The problem here is that numeric conversions in Qt like Qstring::number return a new string (thus allocate memory).
You could allocate a handful of QStrings once, reserve some space in them and then use QTextStream to make the conversions "in place".

T3STY

@Chris-Kawa The number to string conversion is done via QVariant::toString() anyway, so that is not a problem. And reserving space is not necessary anymore if I use the tricks above :)

onek24

@T3STY said:

Unfortunately, modifying Qt's source is not a viable option because of the license I'm using (LGPL).

No need to modify Qt's source. Just check what Qt does in such a case and develop your own algorithm/method.

SGaist

What about just doing a good old

QString message = "{" QTime::currentTime().toString() + el_msg[] etc.

?

T3STY

@SGaist that would make it hard to follow a pre-format string. Yes, it would be faster and, to some extent, easier. But definitely not the solution in my case.

Anyway, eventually I went for the HTML pre-formatted strings method that I wrote about in an earlier post. It's the easier and faster solution for me and seems to have the lightest impact on performance of all methods I have tried.

Thank you all for help :)

SGaist

If you want to be sure about the performance: QBENCHMARK ;)

Happy coding !