Reading file with QTextStream looping over file

Woody

Situation:
In my program, I want to read a file. So I put my file in a QTextStream.
@if(file.open(QIODevice::ReadOnly | QIODevice::Text))
{
QTextStream in(&file);
in.setAutoDetectUnicode(true);

    while(!in.atEnd())
    {
        progressDialog.setValue(in.pos());
        QString line = QString();
        line = QString(in.readLine());
        if (!line.isEmpty())
        {
             ...
        }
    }

}@

Problem:
The file loops over the first 154 lines (of 235). And it doesn't stop.
So it reads 65% of my file, loops over that 65% and just skips the other 35%.

Anyone got an idea about this problem?

sidewinder

Is there anything special at line 154? Does it happen with any file?

Woody

I tried it with 2 files, one where he got to 87% and one to 65%

There is nothing special on that line, I even tried to delete line 153, 154 and 155, but nothing else happened.

I can't post the file, cause it contains confedential information. A screenshot with dummytext is possible:
!http://wouterverbruggen.be/dummy.jpg(Dummy.jpg)!

Even if I copy those 3 line to the first line, it does the same.

DKvita

Try to remove QIODevice::Text option.

Woody

Just tried it and isn't really working.

If I don't put it in a QTextStream it works, but I need special characters to be read. So I need
@in.setAutoDetectUnicode(true);@

sidewinder

If you expect to have unicode characters why don't you set codec to one of the utf's? Try "setting codec":http://qt-project.org/doc/qt-4.8/qtextstream.html#setCodec for your QTextStream object to UTF-8. Unless you set one globally before...

DKvita

Can you send me your file 'as is'.

Woody

That doesn't help with the fact I don't have to use a QTextStream. I think (read: I'm quiet sure) that the problem has to do with the QTextStream. But where that is, I really haven't got a clue.

Edit: I can't really send you the file, sorry. But there isn't really something wrong with the line, nor the file itself.

DKvita

Ok, maybe, you may post which special characters appear in the file?

Woody

In this file the only special characters are ë (on Daniël, Joëlle, ...), é (on Conté, Romanée, ...), ö (on Buddenböhmer), ü (on Büller)

But none of these characters are on line 153, 154, 155, 156

DerManu

Does this:
@
while(true)
qDebug() << in.readLine();
@

read past that magic line 154?
Have you validated the strings that readLine returns? Maybe 154 is actually the last line in the file because some apparent multiple lines in the file are read joined together (for whatever crazy unicode reason).

bq. I tried it with 2 files, one where he got to 87% and one to 65%

I suppose the other file was also such a name list with that (maybe broken) unicode encoding. Do the following:
Open a text editor and type by hand some lines that reproduce the visual appearance (i.e. which characters come when) in one of the unicode files. This is to make sure there aren't any strange unicode characters inside. Then either compare them in a hex editor or just try to read that hand-crafted file (maybe multiply the lines by copy-pasting to 200 or so).
Now that should absolutely work, because your code is very simple. And if it does, you know that the original files contain crazy unicode control characters that need to be rectified first. It may be an effect due to the automatic unicode detection. If it's not detected correctly, who knows what happens. Why don't you set the codec yourself as proposed already?

DKvita

Do you sure the file is really unicode? It seems you have ASCII file that detected by QTextCodec as UTF-8.

Woody

I don't think my decoding is the problem here.
If I don't try to decode it, it does exactly the same as with the autoDetect and the setCodec("utf-8").

I'll try to write some of it again, but that's not a solution for 30k lines-file :)

Woody

Going to update Qt, I know have 4.7.4, my colleague has 4.8.2.
And on his PC it works fine.

sidewinder

It seems you may have been affected by bug "QTBUG-9814":https://bugreports.qt-project.org/browse/QTBUG-9814.

Woody

Mmh that indeed could be possible. Thnx for the tip!

Woody

It is 'solved'. I installed Qt 4.8.2 and it does it's job just fine. Takes the whole file and treats every line properly.

I don't think I need to put [SOLVED] in the title cause the problem in 4.7.4 isn't solved.