Unsolved Qt 5.14.1 QSqlModel and French Characters.
-
@ad5xj said in Qt 5.14.1 QSqlModel and French Characters.:
But it does seem logical, given the code , that the data is unpacked from the supplied string
Besides, the origin should not matter.It does matter where this string comes from - do you read it from a file or from a e.g. QLineEdit? If the encoding there is wrong then it will be put wrong in the database.
-
@Christian-Ehrlicher I am obviously not communicating well.
Regardless whether the data comes from a file or a QLineEntry it is assigned to a QString in order to be a segment of the QString f. In this case the offending data is part of data loaded from an external source in text format.
Are you saying that assignment will not change the data to UTF-8?
-
@ad5xj said in Qt 5.14.1 QSqlModel and French Characters.:
Are you saying that assignment will not change the data to UTF-8?
If you convert it wrong then QString will be wrong, yes.
This is working fine for me:
int main(int argc, char **argv) { QApplication app(argc, argv); QSqlDatabase db(QSqlDatabase::addDatabase("QSQLITE")); db.setDatabaseName(":memory:"); db.open(); QSqlQuery q; q.exec("CREATE TABLE tmp (name varchar);"); q.exec("INSERT INTO tmp (name) VALUES ('Ren\u00e9');"); q.exec("SELECT * FROM tmp;"); q.next(); QString str = q.value(0).toString(); QLabel lbl(str); lbl.show(); return app.exec(); }
-
@Christian-Ehrlicher Not sure how to answer your last reply.
The data is read into a QByteArray from a text file with a QIODeice as a QFile object opened as text, using the readAll() method. The resulting QByteArray is appended to a QString. That allows the data to be split into lines on the "\n" of each record.
Since the incoming data is delimited text, each line of text is parsed for each data field. The data in each field becomes the correct data type as detected by the parser.
So unless there is something being overlooked at the time the data is read from the input, I am not sure what else to tell you.
-
@ad5xj said in Qt 5.14.1 QSqlModel and French Characters.:
The resulting QByteArray is appended to a QString.
Again: how and what's the encoding of the file? Please show us the code!
-
@Christian-Ehrlicher OK, but I don't think you will learn any more than I have already posted.
void ReadFile(QString fil) QByteArray allRecs; fil = "textfile.txt" QStringLIst fields; // get the file to parse QFile infile(fil); if ( !infile.open(QIODevice::ReadOnly | QIODevice::Text)) { QMessageBox mb; mb.setWindowTitle(QApplication::tr("Text Import Failure")); msg = __FILE__; msg += __FUNCTION__; msg += QString("%1 ").arg(__LINE__); msg += QApplication::tr("Cannot read file "); msg += QString("%1:").arg(fil); msg += QString("\n%2 ").arg(infile.errorString()); mb.setText(msg); mb.exec(); return; } data = ""; lines.clear(); // init the lines var strLineSeg = ""; // First read all the text file records into memory allRecs = infile.readAll(); // read everything into buffer file buffer infile.close(); // close the input file data.append(allRecs); // change from required QByteArray to QString lines = data.split("\n",QString::KeepEmptyParts); // split file records into lines on line break fields.clear(); for (int p = pos; p < y; ++p ) { linestr = lines[p].trimmed().toLocal8Bit(); if ( linestr.length() > 0 ) { // not a blank line // make a list of all fields in the line fields = linestr.split("<",QString::KeepEmptyParts); fields << linestr.trimmed().split("<",QString::SkipEmptyParts); cnt = fields.count(); parse(fields); } fields.clear(); } void parse(QStringList flds) { If (fldname == "Name") { QString datastr = flds[1]; QString name = datastr.trimmed(); // All fields read now make the new record from the data QString strRec = ""; strRec = name + "|"; . . . addNewQSO(strRec); }
The rest of the database save code you have already.
BTW the database was created with the SQLite PRAGMA encoding = 'UTF-8'; -
@ad5xj UPDATE:
Some progress...I have been able to use various text editors to determine the incoming data is text encoded as Windows-1258. Using the QTextCodec to convert the QByteArray to QString as UTF-8 it fixed the problem.This is only a short-term solution since I will not always know what the encoding is. So now, how do I detect the character encoding from the raw text file in order to use the right conversion codec?
A search of this forum is inconclusive.
-
@ad5xj There is no complete solution for detecting the encoding. For example UTF-8 and Latin1 are identical for the first 127 characters. See https://forum.qt.io/topic/12319/how-to-automatically-detect-the-codec-text-file
-
With some research, I have come upon a command-line utility called UTRAC which does at least a partial job of text encoding detection. It is not a complete list and is not 100% effective but does beg the question; why QFile and QTextStream have not at least attempted this. If the UTRAC guys have come this far, it is at least feasible. If it requires a feature request for QT 5.15 then I will put one in.
-
@ad5xj said in Qt 5.14.1 QSqlModel and French Characters.:
why QFile and QTextStream have not at least attempted this
Why should they (especially QFile)? QFile is simply a file handle with some operation for basic file operations like reading/writing. Most of the time you do not have to guess as you know what you're writing and reading. And also, as you already discovered, there is no 100% solution for this and doing this checking is time consuming (you need to read the whole file and check each character), so it is an expensive operation.