Solved Data file is full of zeros after OS crash or power outage

[Youda008]

I have an application that stores its state in a data file and updates that file every 60 seconds.

I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
{
	QString newFilePath = origFilePath+".new";
	QFile newFile( newFilePath );
	if (!newFile.open( QIODevice::WriteOnly ))
	{
		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
	}

	newFile.write( newContent );
	if (newFile.error() != QFile::NoError)
	{
		return "Could not write to file "%newFilePath%": "%newFile.errorString();
	}

	newFile.close();

	QFile oldFile( origFilePath );
	if (oldFile.exists())
	{
		if (!oldFile.remove())
		{
			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
		}
	}

	if (!newFile.rename( origFilePath ))
	{
		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
	}

	return "";
}

This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

What else can i do to prevent this problem?

[J.Hilk]

@Youda008 said in Data file is full of zeros after OS crash or power outage:

hat is the best practice for saving data files to reduce the occurences of this to minimum?

have you heard of QSaveFile ?

https://doc.qt.io/qt-6/qsavefile.html#details

[Christian Ehrlicher]

@Youda008 said in Data file is full of zeros after OS crash or power outage:

Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

Not Qt but your OS or more exactly the file system driver. Nothing Qt can do here - you have to use the OS api to make sure the fs write buffer gets flushed (if it's possible at all though).

[ChrisW67]

@Youda008 QFile::close() calls flush(), so everything is handed off to the operating system at that point as @Christian-Ehrlicher points out. The OS software, the controller hardware, and disk drive involved can all be buffering. This is why dedicated file storage devices often have short term power backup to ensure buffers are flushed to disk.

There's also the possibility that your code deliberately wrote a file full of zeroes. This could be the result of an error on your part, an earlier failure (to allocate memory for example), or the result of the user's "OS crash" causing errant behaviour.

"OS Crash" when described by a user has a very broad range of possibility. It seems unlikely that such an event would always occur in the split second every minute this specific code was executing, unless triggered by this activity. Could be the user's storage driver is what is crashing the operating system (e.g Windows BSOD).

[Youda008]

I forgot to mention that i have both Windows users and Linux users reporting this problem.

What is the best practice for saving data files to reduce the occurences of this to minimum? I don't remember anyone ever complaining that they lost for example user settings of their favourite text editor due to a power outtage.

[JonB]

@Youda008
The best you could add to your code is to call OS sync(2) (fsync(2) if you know what you are doing, but sync() is simplest). You might call that after newFile.close();, and maybe again after newFile.rename(). Whether it will make any difference is anyone's guess.

If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

[J.Hilk]

@Youda008 said in Data file is full of zeros after OS crash or power outage:

hat is the best practice for saving data files to reduce the occurences of this to minimum?

have you heard of QSaveFile ?

https://doc.qt.io/qt-6/qsavefile.html#details

[Youda008]

@JonB

If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

Ideally yes, but that is not under my control. I write the code, others execute it. It does not happen to me.

[Youda008]

@J-Hilk

have you heard of QSaveFile ?

I haven't o_O Looks very good. Thanks!

[JonB]

@Youda008
It is not clear whether/how the code you show (which seems fine) could lead either to a "crash" or to file being "filled with zeros", so you may not be able to do anything to this code to affect that behaviour. Like I said, try a sync() FWIW.

QSaveFile may be convenient, but ought not be related to the behaviour your describe.