QTextStream performance?

[Perdrix]

A user of the code I develop is complaining that the writes to a file (which in his case is to a network drive) are pretty slow and he can see the file growing as it is written. He says:

"wouldn't it be a good idea to buffer the write content internally and flush it in 1 shot. I can see the file size of the txt file growing in the explorer when I refresh, and as a programmer this looks like an easy-to-fix bottle neck."

The code I use to write the file is pretty simple and I include all of it here:

bool CRegisteredFrame::SaveRegisteringInfo(const fs::path& szInfoFileName)
{
	bool bResult = false;
	QFile data(szInfoFileName);
	if (!data.open(QFile::WriteOnly | QFile::Truncate))
		return false;

	QTextStream fileOut(&data);	
	{
		fileOut << QString("OverallQuality = %1").arg(m_fOverallQuality, 0, 'f', 2) << Qt::endl;
		fileOut << paramString(QualityParam, " = %1").arg(this->quality, 0, 'f', 2) << Qt::endl;
		fileOut << "RedXShift = 0.0" << Qt::endl;
		fileOut << "RedYShift = 0.0" << Qt::endl;
		fileOut << "BlueXShift = 0.0" << Qt::endl;
		fileOut << "BlueYShift = 0.0" << Qt::endl;
		if (m_bComet)
			fileOut << QString("Comet = %1, %2").arg(m_fXComet, 0, 'f', 2).arg(m_fYComet, 0, 'f', 2) << Qt::endl;
		fileOut << QString("SkyBackground = %1").arg(m_SkyBackground.m_fLight, 0, 'f', 4) << Qt::endl;
		fileOut << paramString(ThresholdParam, " = %1").arg(100.0 * this->usedDetectionThreshold, 0, 'f', 3) << Qt::endl;
		fileOut << "NrStars = " << m_vStars.size() << Qt::endl;

		for (int i = 0; const CStar& star : this->m_vStars)
		{
			fileOut << "Star# = " << i << Qt::endl;
			fileOut << QString("Intensity = %1").arg(star.m_fIntensity, 0, 'f', 2) << Qt::endl;
			fileOut << QString("Quality = %1").arg(star.m_fQuality, 0, 'f', 2) << Qt::endl;
			fileOut << QString("MeanRadius = %1").arg(star.m_fMeanRadius, 0, 'f', 2) << Qt::endl;
			fileOut << paramString(CircularityParam, " = %1").arg(star.m_fCircularity, 0, 'f', 2) << Qt::endl;
			fileOut << "Rect = " << star.m_rcStar.left << ", "
				<< star.m_rcStar.top << ", "
				<< star.m_rcStar.right << ", "
				<< star.m_rcStar.bottom << Qt::endl;
			fileOut << QString("Center = %1, %2").arg(star.m_fX, 0, 'f', 2).arg(star.m_fY, 0, 'f', 2) << Qt::endl;
			fileOut << QString("Axises = %1, %2, %3, %4, %5")
				.arg(star.m_fMajorAxisAngle, 0, 'f', 2)
				.arg(star.m_fLargeMajorAxis, 0, 'f', 2)
				.arg(star.m_fSmallMajorAxis, 0, 'f', 2)
				.arg(star.m_fLargeMinorAxis, 0, 'f', 2)
				.arg(star.m_fSmallMinorAxis, 0, 'f', 2) << Qt::endl;
			++i;
		}
		bResult = true;
	}
	return bResult;
}

Is there anything I can do to improve that?

Can I create the text stream on a QString/Byte array and then write the file object using QFile::write()?

[jsulm]

@Perdrix said in QTextStream performance?:

Can I create the text stream on a QString/Byte array

Yes, it's right there in the documentation:

• https://doc.qt.io/qt-6/qtextstream.html#QTextStream-3
• https://doc.qt.io/qt-6/qtextstream.html#QTextStream-4

[Perdrix]

OK - not enough coffee :)

[JonB]

@Perdrix
And did removing the QTextStream from do the writing really make a difference? I would have expected its buffering/flushing to be reasonably efficient anyway. The Qt::endls should not be flushing as you are writing to file (not e.g. terminal).

[Perdrix]

I can't see a difference locally. I will ask the originator to test in his environment.

[JonB]

@Perdrix It will be interesting, I am "unconvinced" it will (much), but we shall see :)

[Perdrix]

The file in question (in his case) is over 10000 lines which is a somewhat extreme case. I sent him a test build and he says:

"the performance on writing the txt file is ways faster now."

I've asked if he can quantify that...

[JonB]

@Perdrix
Fair enough, I can't argue with that!

The implication (to me, of such a noticeable speed improvement) is that QTextStream is doing many more (small?) buffer writes than a single QFile::write(). I would have expected QTextStream's defaults would be fine for this, even across a network.

Now I have a feeling that Qt::endl / std::endl do actively flush. I was not expecting that, writing \n to a file device would not do so.

If you are interested and you/your user can spare the time, change your code temporarily to output a \n (I'm assuming you're Linux) where you have Qt::endl, above all in the loop. My bet is that will show the same "fast" behaviour as you have now when you use a single QFile::write()? If that is so we need to find an alternative to endl which does not flush (to file), and I shall never use endl again, that's terrible default behaviour (IMHO)!

Yep, confirmed Googling. std::endl does flush as well as newline

Inserts a newline character into the output sequence os and flushes it as if by calling os.put(os.widen('\n')) followed by os.flush().

Qt::endl:

Qt::endl Same as operator<<('\n') and flush().

Not Qt's fault, but a choice with consequences from C++/std. I see I am not the only person who thinks this is a bad decision! I would bet there are millions of lines of C++ code out there flushing to file quite unnecessarily :( Moral: if you care about speed when writing to a file don't ever use endl!

I see there is a clang-tidy for this, performance-avoid-endl

Checks for uses of std::endl on streams and suggests using the newline character '\n' instead.

Rationale: Using std::endl on streams can be less efficient than using the newline character '\n' because std::endl performs two operations [....]

[Perdrix]

FYI here's his timing for an extreme case (200000 records) - the timing includes more than just writing the file so you should only take into account the difference in the two timings:

Old code: 4 mins 16.955
New code: 20.473s

So the old code took 3m 56.4s longer to write the file than the new code.

So most definitely Qt::endl (and std::endl) kill performance when writing to a file.

D.

[JonB]

@Perdrix said in QTextStream performance?:

So Qt::endl (and std::endl) kill performance ...

Yes they do indeed per my findings detailed above. As I wrote

Moral: if you care about speed when writing to a file don't ever use endl!

I think its flush() behaviour is a disastrous addition and as I wrote I shall never be using endl again, only \n, as it's way too easy to miss what it's doing.

If I were you I would rewrite to use \n and still use the QTextStream to file, rather than building your own in-memory string to send in one go to QFile::write(). If your data was "large" (a lot larger than your 10k lines) why fill memory with it before writing it out?

[SimonSchroeder]

@JonB said in QTextStream performance?:

change your code temporarily to output a \n (I'm assuming you're Linux)

It doesn't matter if it's Linux. The C++ standard specifies that if the file is opened in text mode \n is automatically converted to the correct line ending for your platform. (Only disadvantage of text mode is that you cannot properly use seek() on the file anymore. Anyways, I purposfully always use \n on all platforms for my text files. Every text editor, even on Windows, can handle that properly.)

[JonB]

@SimonSchroeder said in QTextStream performance?:

It doesn't matter if it's Linux. The C++ standard specifies that if the file is opened in text mode \n is automatically converted to the correct line ending for your platform

It does matter since you can see the OP wrote

if (!data.open(QFile::WriteOnly | QFile::Truncate))

which is why I assumed OP was under Linux. If the OP correctly included QIODeviceBase::Text then it would not matter, but there is only so much I am prepared to type into an answer....

Every text editor, even on Windows, can handle that properly.

Not Notepad in my experience.

[Perdrix]

Actually this code is running on Windows - from what you wrote it would seem I need to add that flag - it will be done.

[JonB]

@Perdrix said in QTextStream performance?:

Actually this code is running on Windows

:) Always helps to say, especially if using text files! Under Windows you should always include QFile::Text when opening text files for either read or write, I'm surprised you have gotten away without this for so long... It will certainly be relevant if changing from endl to \n in order to get the speed increase. Under Linux it does not matter/it's a NO-OP.