Concurrent problems when reading a CSV file.

pepita

Hello everyone;

I have a problem that leads my app to crash after a certain period of time.

I have a robot-application with a microcontroller that is writing a CSV file every certain time (a new row every 2 to 14 seconds, depending on the robot's task). This application it's completely independent, and the data are coming through RS-232.

Then, I have also a Qt desktop application that reads this CSV file, and interprets its data to make tables, maps and statistics.
In this application, I've set a timer to read the file every second, to make a kind of real-time display. When I go into this auto-refresh mode, is when the app crashes...

As the CSV file is written and read concurrently by two different applications, and because of the timing in both programms, the writting and reading process can concur in time, I think that the app is crashing because of this. I don't know how windows handle this things, if the file is being blocked when writting, and the Qt app crashes because of trying to access a locked file,.. no idea, I'm not an expert.

I've read about mutex and semaphores, and if the problem is really the concurrency, seems to be the solution...

Or,... could it be a memory problem, that after a certain time I run out of memory and the app crashes? I'm managing quite large files, with about 40 000 rows...

Anyways I'm asking for some help from you guys because I don't know what's going on, and I'm a little bit lost with this...

Thank you very much in advance,

Best regards.

Edit: What I want is, to modify my Qt application to control the access to the file, i.e., to check whether the file is being written and wait until this process finish to read it.

Chris Kawa

Mutexes and semaphores won't help you as these are mechanisms for different threads in a single process. What you've got are two different processes.

How do you open file for reading/writing? Do you check what the open() method returns? I assume it's ok for the reading process to skip a turn or two if it can't ope the file. Is it ok for the writer too?

If not then there are couple of strategies. You could write to a temp file and rename it when you finally close it. The reader would then never try to read the opened temporary file. Another solution is to create a "marker" file. When either process reads or writes to a log file it creates some other temporary file and deletes it after it's done. When the other process needs to read or write if waits until there is no temporary to access the log. Yet another way is to use some "guard" variable in a memory shared across processes eg. by using "QSharedMemory":http://qt-project.org/doc/qt-5/qsharedmemory.html

pepita

Hi Chris, thanks for the reply;

So this is how I'm openning the file in the Qt app:

@
void MainApp::openCsvFile ()
{
if (_autoRefreshOn == false)
{
_openedFile = QFileDialog::getOpenFileName (0, "Open CSV file",QDir::currentPath(),"CSV Files(*.csv)");

    if (!_openedFile.endsWith (".csv"))
    {
        QMessageBox::warning (0,"Error","Selected file is not a CSV","Ok");
        return;
    }
    clearView ();
}

QFile file &#40;_openedFile&#41;;

if (file.open(QIODevice::ReadOnly))
{
    QString data = file.readAll();
    data.remove( QRegExp("\r") );
    QChar character;
    QTextStream textStream(&data);

    while (!textStream.atEnd())
    {
        textStream >> character;

        if (character == ';')
        {
            _cellData << _readBuffer;             
            _readBuffer.clear();                     
            _countCells ++;
        }
        else if (character == '\n')
        {
            _cellData << _readBuffer;              
            _readBuffer.clear();                
            _countCells ++;
            _countRows ++;
        }
        else if (textStream.atEnd())
        {
            _readBuffer.append(character);   
        }
        else
            _readBuffer.append(character);   
    }
}
else
    return;

   [...]

@

Basically I'm openning the file in two different ways: with a push button to open the file for the first time, and automatically every second if in auto-refresh mode.

You're completely right, it's Ok for the app to skip the needed turns just to avoid crashing, at the end this only will make the real-time display a little bit less "real-time", but the app stable, which is my goal.

And it's also Ok for the writter, which let's say, the rights of the file belongs completely to it (I don't know how manages the file, I'm assuming that it writes the data, and closes the file). Then the writter is the master, and the Qt app that reads the file is the slave. Shouldn't interfere in the writting process, and always be waiting for the file to be ready.

So basically I think that checking if the file is opened by other process, or being written, or... everytime the Qt application opens the file would make the deal.

How can I check this? Is there a method in Qt, like Mutexes and Semaphores, but for external processes? or is there a method inside QFile class and I'm missing something in my if statement?

Do you also discard the memory lack problem after a period of time running?

Thanks for the help.

Chris Kawa

bq. So basically I think that checking if the file is opened by other process, or being written, or… everytime the Qt application opens the file would make the deal.

Although there are ways to do that in WinAPI I think Qt doesn't expose these and just keeps to the portable subset of them. There are some extensions to the Qt library that may allow you to do that. One such is "QtLockedFile":http://doc.qt.digia.com/solutions/4/qtlockedfile/qtlockedfile.html but I don't know the the current status of that extension

bq. Is there a method in Qt, like Mutexes and Semaphores, but for external processes?

You can use one of the methods I mentioned. Closest to a mutex would be the QSharedMemory with explicit locks but the file solutins could be easier to implement.

bq. Do you also discard the memory lack problem after a period of time running?

I assumed you already determined that the problem lies in the concurrent access to the file? Have you run your app in the debugger to confirm that?

Not a solution, but one thing you can do to help is minimizing the file blocking time. Right now you open a file, read in all of its contents and proceed to processing it while the file is still opened till the file variable goes out of scope. A minor optimization here would be to close() the file immediately after you read the contents and not to block it longer than necessary.

pepita

Hi Chris.

I've installed the packages you mentioned, QtLockedFile, and also read carefully the docu.

For me seems that this only works with two processes using both the QtLockedFile extension, so one can write a "flag" (QtLockedFile::LockMode) whether the file is locked or not, and the other read this "flag" to see if it's allowed to open the file, and viceversa.

I do not think that the other process is writting anything that I can check in my app to see if the file is locked or not, with the methods from the QtLockedFile extension.

Am I wrong?

Is there anything to do with the windows-way to treat files? I mean, is like when you try to delete a file which is open, windows promt up a error message telling us that we cannot delete that, because it's open...

PS: Then I'm having another problem, SIGFPE arithmetical exception, but I will open a new thread if I can not manage to solve it...

Chris Kawa

bq. For me seems that this only works with two processes using both the QtLockedFile extension, so one can write a “flag” (QtLockedFile::LockMode) whether the file is locked or not, and the other read this “flag” to see if it’s allowed to open the file, and viceversa.

Yes, isn't that what you wanted?

bq. I do not think that the other process is writting anything that I can check in my app to see if the file is locked or not, with the methods from the QtLockedFile extension.

I thought that you can also modify the code of the writing app. Is that not the case?

Windows file modes "are complicated":http://msdn.microsoft.com/en-us/library/windows/desktop/aa365430.aspx. They offer sharing scenarios that are not available on other platforms and thus QFile doesn't expose these.

You should stop thinking about "checking if you can open a file". This is a common concurrency trap. Consider this:
@
if(someCheckToSeeIfICanOpen("file.txt")) //suppose this returns true
file.open() //between the previous line and this the file might have been locked by another process so this can still fail
@
It's just not the way to go. You can't get away with this without some locking. A flagging mechanism like the QtLockedFile or a temporary file are a good solution.