Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Data file is full of zeros after OS crash or power outage
Forum Updated to NodeBB v4.3 + New Features

Data file is full of zeros after OS crash or power outage

Scheduled Pinned Locked Moved Solved General and Desktop
9 Posts 5 Posters 594 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Youda008Y Offline
    Youda008Y Offline
    Youda008
    wrote on last edited by
    #1

    I have an application that stores its state in a data file and updates that file every 60 seconds.

    I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

    QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
    {
    	QString newFilePath = origFilePath+".new";
    	QFile newFile( newFilePath );
    	if (!newFile.open( QIODevice::WriteOnly ))
    	{
    		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
    	}
    
    	newFile.write( newContent );
    	if (newFile.error() != QFile::NoError)
    	{
    		return "Could not write to file "%newFilePath%": "%newFile.errorString();
    	}
    
    	newFile.close();
    
    	QFile oldFile( origFilePath );
    	if (oldFile.exists())
    	{
    		if (!oldFile.remove())
    		{
    			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
    		}
    	}
    
    	if (!newFile.rename( origFilePath ))
    	{
    		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
    	}
    
    	return "";
    }
    
    

    This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

    Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

    What else can i do to prevent this problem?

    Christian EhrlicherC C Youda008Y JonBJ 4 Replies Last reply
    0
    • Youda008Y Youda008

      I forgot to mention that i have both Windows users and Linux users reporting this problem.

      What is the best practice for saving data files to reduce the occurences of this to minimum? I don't remember anyone ever complaining that they lost for example user settings of their favourite text editor due to a power outtage.

      J.HilkJ Online
      J.HilkJ Online
      J.Hilk
      Moderators
      wrote on last edited by
      #6

      @Youda008 said in Data file is full of zeros after OS crash or power outage:

      hat is the best practice for saving data files to reduce the occurences of this to minimum?

      have you heard of QSaveFile ?

      https://doc.qt.io/qt-6/qsavefile.html#details


      Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


      Q: What's that?
      A: It's blue light.
      Q: What does it do?
      A: It turns blue.

      Youda008Y 1 Reply Last reply
      4
      • Youda008Y Youda008

        I have an application that stores its state in a data file and updates that file every 60 seconds.

        I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

        QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
        {
        	QString newFilePath = origFilePath+".new";
        	QFile newFile( newFilePath );
        	if (!newFile.open( QIODevice::WriteOnly ))
        	{
        		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
        	}
        
        	newFile.write( newContent );
        	if (newFile.error() != QFile::NoError)
        	{
        		return "Could not write to file "%newFilePath%": "%newFile.errorString();
        	}
        
        	newFile.close();
        
        	QFile oldFile( origFilePath );
        	if (oldFile.exists())
        	{
        		if (!oldFile.remove())
        		{
        			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
        		}
        	}
        
        	if (!newFile.rename( origFilePath ))
        	{
        		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
        	}
        
        	return "";
        }
        
        

        This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

        Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

        What else can i do to prevent this problem?

        Christian EhrlicherC Offline
        Christian EhrlicherC Offline
        Christian Ehrlicher
        Lifetime Qt Champion
        wrote on last edited by
        #2

        @Youda008 said in Data file is full of zeros after OS crash or power outage:

        Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

        Not Qt but your OS or more exactly the file system driver. Nothing Qt can do here - you have to use the OS api to make sure the fs write buffer gets flushed (if it's possible at all though).

        Qt Online Installer direct download: https://download.qt.io/official_releases/online_installers/
        Visit the Qt Academy at https://academy.qt.io/catalog

        1 Reply Last reply
        1
        • Youda008Y Youda008

          I have an application that stores its state in a data file and updates that file every 60 seconds.

          I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

          QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
          {
          	QString newFilePath = origFilePath+".new";
          	QFile newFile( newFilePath );
          	if (!newFile.open( QIODevice::WriteOnly ))
          	{
          		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
          	}
          
          	newFile.write( newContent );
          	if (newFile.error() != QFile::NoError)
          	{
          		return "Could not write to file "%newFilePath%": "%newFile.errorString();
          	}
          
          	newFile.close();
          
          	QFile oldFile( origFilePath );
          	if (oldFile.exists())
          	{
          		if (!oldFile.remove())
          		{
          			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
          		}
          	}
          
          	if (!newFile.rename( origFilePath ))
          	{
          		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
          	}
          
          	return "";
          }
          
          

          This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

          Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

          What else can i do to prevent this problem?

          C Offline
          C Offline
          ChrisW67
          wrote on last edited by
          #3

          @Youda008 QFile::close() calls flush(), so everything is handed off to the operating system at that point as @Christian-Ehrlicher points out. The OS software, the controller hardware, and disk drive involved can all be buffering. This is why dedicated file storage devices often have short term power backup to ensure buffers are flushed to disk.

          There's also the possibility that your code deliberately wrote a file full of zeroes. This could be the result of an error on your part, an earlier failure (to allocate memory for example), or the result of the user's "OS crash" causing errant behaviour.

          "OS Crash" when described by a user has a very broad range of possibility. It seems unlikely that such an event would always occur in the split second every minute this specific code was executing, unless triggered by this activity. Could be the user's storage driver is what is crashing the operating system (e.g Windows BSOD).

          1 Reply Last reply
          0
          • Youda008Y Youda008

            I have an application that stores its state in a data file and updates that file every 60 seconds.

            I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

            QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
            {
            	QString newFilePath = origFilePath+".new";
            	QFile newFile( newFilePath );
            	if (!newFile.open( QIODevice::WriteOnly ))
            	{
            		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
            	}
            
            	newFile.write( newContent );
            	if (newFile.error() != QFile::NoError)
            	{
            		return "Could not write to file "%newFilePath%": "%newFile.errorString();
            	}
            
            	newFile.close();
            
            	QFile oldFile( origFilePath );
            	if (oldFile.exists())
            	{
            		if (!oldFile.remove())
            		{
            			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
            		}
            	}
            
            	if (!newFile.rename( origFilePath ))
            	{
            		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
            	}
            
            	return "";
            }
            
            

            This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

            Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

            What else can i do to prevent this problem?

            Youda008Y Offline
            Youda008Y Offline
            Youda008
            wrote on last edited by
            #4

            I forgot to mention that i have both Windows users and Linux users reporting this problem.

            What is the best practice for saving data files to reduce the occurences of this to minimum? I don't remember anyone ever complaining that they lost for example user settings of their favourite text editor due to a power outtage.

            J.HilkJ 1 Reply Last reply
            0
            • Youda008Y Youda008

              I have an application that stores its state in a data file and updates that file every 60 seconds.

              I wrote the following function to prevent data loss when the application or the OS is interrupted during writing.

              QString updateFileSafely( const QString & origFilePath, const QByteArray & newContent )
              {
              	QString newFilePath = origFilePath+".new";
              	QFile newFile( newFilePath );
              	if (!newFile.open( QIODevice::WriteOnly ))
              	{
              		return "Could not open file "%newFilePath%" for writing: "%newFile.errorString();
              	}
              
              	newFile.write( newContent );
              	if (newFile.error() != QFile::NoError)
              	{
              		return "Could not write to file "%newFilePath%": "%newFile.errorString();
              	}
              
              	newFile.close();
              
              	QFile oldFile( origFilePath );
              	if (oldFile.exists())
              	{
              		if (!oldFile.remove())
              		{
              			return "Could not delete the previous file "%origFilePath%": "%oldFile.errorString();
              		}
              	}
              
              	if (!newFile.rename( origFilePath ))
              	{
              		return "Could not rename the new file "%newFilePath%" back to "%origFilePath%": "%newFile.errorString();
              	}
              
              	return "";
              }
              
              

              This code should only remove the old file after the new file is successfully written and closed. Yet my users still keep comming to me with an issue that their data file is full of zero bytes after their OS crashed. I even tried adding sleep() after the newFile.close() but it had no effect.

              Why is this happening? Does Qt do some caching or asynchronous I/O that lies to me that the data was successfully written when it wasn't?

              What else can i do to prevent this problem?

              JonBJ Offline
              JonBJ Offline
              JonB
              wrote on last edited by
              #5

              @Youda008
              The best you could add to your code is to call OS sync(2) (fsync(2) if you know what you are doing, but sync() is simplest). You might call that after newFile.close();, and maybe again after newFile.rename(). Whether it will make any difference is anyone's guess.

              If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

              Youda008Y 1 Reply Last reply
              0
              • Youda008Y Youda008

                I forgot to mention that i have both Windows users and Linux users reporting this problem.

                What is the best practice for saving data files to reduce the occurences of this to minimum? I don't remember anyone ever complaining that they lost for example user settings of their favourite text editor due to a power outtage.

                J.HilkJ Online
                J.HilkJ Online
                J.Hilk
                Moderators
                wrote on last edited by
                #6

                @Youda008 said in Data file is full of zeros after OS crash or power outage:

                hat is the best practice for saving data files to reduce the occurences of this to minimum?

                have you heard of QSaveFile ?

                https://doc.qt.io/qt-6/qsavefile.html#details


                Be aware of the Qt Code of Conduct, when posting : https://forum.qt.io/topic/113070/qt-code-of-conduct


                Q: What's that?
                A: It's blue light.
                Q: What does it do?
                A: It turns blue.

                Youda008Y 1 Reply Last reply
                4
                • JonBJ JonB

                  @Youda008
                  The best you could add to your code is to call OS sync(2) (fsync(2) if you know what you are doing, but sync() is simplest). You might call that after newFile.close();, and maybe again after newFile.rename(). Whether it will make any difference is anyone's guess.

                  If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

                  Youda008Y Offline
                  Youda008Y Offline
                  Youda008
                  wrote on last edited by
                  #7

                  @JonB

                  If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

                  Ideally yes, but that is not under my control. I write the code, others execute it. It does not happen to me.

                  JonBJ 1 Reply Last reply
                  0
                  • J.HilkJ J.Hilk

                    @Youda008 said in Data file is full of zeros after OS crash or power outage:

                    hat is the best practice for saving data files to reduce the occurences of this to minimum?

                    have you heard of QSaveFile ?

                    https://doc.qt.io/qt-6/qsavefile.html#details

                    Youda008Y Offline
                    Youda008Y Offline
                    Youda008
                    wrote on last edited by
                    #8

                    @J-Hilk

                    have you heard of QSaveFile ?

                    I haven't o_O Looks very good. Thanks!

                    1 Reply Last reply
                    0
                    • Youda008Y Youda008 has marked this topic as solved on
                    • Youda008Y Youda008

                      @JonB

                      If I were you I should be more worried about why the OS is "crashing" and/or files are being filled with zeros than the behaviour of this program! :)

                      Ideally yes, but that is not under my control. I write the code, others execute it. It does not happen to me.

                      JonBJ Offline
                      JonBJ Offline
                      JonB
                      wrote on last edited by JonB
                      #9

                      @Youda008
                      It is not clear whether/how the code you show (which seems fine) could lead either to a "crash" or to file being "filled with zeros", so you may not be able to do anything to this code to affect that behaviour. Like I said, try a sync() FWIW.

                      QSaveFile may be convenient, but ought not be related to the behaviour your describe.

                      1 Reply Last reply
                      0

                      • Login

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Users
                      • Groups
                      • Search
                      • Get Qt Extensions
                      • Unsolved