[SOLVED] Encoding UTF-8, QStandardItem and cross-compiling

[paulo.hubert]

Hello all.

I'm developing a desktop application on Qt 5.5. I'm on a Ubuntu 14.04 environment, but I need the application to work on both Ubuntu and Windows. For that, I am using MinGW, which works fine.
My application is accessing some web services which return xml files, encoded in UTF-8, with a lot of non-english characters. I parse the xml using rapidxml, and (tipically) store some information on a QMap<QString, QString> sctructure. The code I use for that is something like this:

mapaOrgaos[oName.item(i)->text().toUtf8()] = oID.item(i)->text().toUtf8();

Here both oName and oID are instances of QStandardItemModel.
After that, I add the map keys to a Combo box, like this:

ui->ComboOrgaos->addItems(mapaOrgaos.keys());

On Ubuntu, everything works fine. But when I cross compile for Windows, the texts shown in the combo box are not correctly encoded (the text shows quotation marks over a black background for non-english characters).
I've been searching the web for a week trying to find out what might be the problem. One important thing to add is that the encoding works fine for strings that do not pass through a QStandardItemModel object. For example, literal strings (used in labels) or xml results passed directly to a text box show correctly encoded text.
Any ideas for what can be causing this?

[Leonardo]

I don't understand why are you calling toUtf8(). QString always stores its data as Unicode. Once you've parsed your xml, you shouldn't care about its original encoding anymore.

[paulo.hubert]

Leonardo,

yes, you're right. Actually I've put the "toUtf8()" hoping that it might solve the problem, which of course it doesn't.
Same problem if I just use

oName.item(i)->text()

[Leonardo]

Could you us show some of your parsing code?

[paulo.hubert]

This post is deleted!

[paulo.hubert]

@Leonardo

Sure.
As I mentioned before, I use rapidxml for the parsing job. It works with c strings, so I created an interface between the parser and the Qt framework, to handle the conversion between QStrings and char *. It goes something like this:

int clParser::parseXml(QString qstrDoc)  {// qstrDoc holds the XML 
   // I found that the easiest way to handle the conversion is through a QByteArray
   bytes = new QByteArray(qstrDoc.toLocal8Bit().constData(), qstrDoc.toLocal8Bit().size());
   bytes_allocated = true;

   try {
      oXml.parse<parse_validate_closing_tags>(bytes->data());
   } catch(parse_error e) {
      return -1;
   }
   if(oXml.first_node()->first_node() == 0) {
      return -2;
   } else {
      return 0;
   }
}

Now for the actual data extraction part. I created a logic to translate a XML document into a table. The method below feeds a QStandardItemModel with values from the column "qstrColName", belonging to entity "qstrEntName", whose parent is "qstrParentName".

void clParser::getColumn(QString qstrParentName, QString qstrEntName, QString qstrColName, QStandardItemModel &oCol) {

    int i = 0;
    int iDepth;

    // Again, the QByteArrays are used to convert between QString and char *
    QByteArray byteParent(qstrParentName.toLocal8Bit().constData(), qstrParentName.toLocal8Bit().size());
    QByteArray byteEnt(qstrEntName.toLocal8Bit().constData(), qstrEntName.toLocal8Bit().size());
    QByteArray byteCol(qstrColName.toLocal8Bit().constData(), qstrColName.toLocal8Bit().size());

    char *strColName = byteCol.data();
    char *strParentName = byteParent.data();
    char *strEntName = byteEnt.data();

    // Look for the first ocurrence of  entity
    xml_node<> *nodeFirst = oXml.first_node();
    xml_node<> *nodeEnt = find_first(strEntName, iDepth);
    xml_node<> *nodeParent = 0;
    if(nodeEnt==0) { 
       return;
    }

    xml_node<> *nodeCol;
    xml_attribute<> *attr;
    while(nodeEnt != 0) {
        // Look for a node with the same name as the column
        nodeCol = nodeEnt->first_node(strColName);
        if(nodeCol == 0) {
            // No node found, I loop through the attributes
            attr = nodeEnt->first_attribute(strColName,byteCol.size(), false);
            if(attr != 0) {
                i++;
                // HERE IS WHERE THE DATA GETS ACTUALLY TRANSFERED
                oCol.appendRow(new QStandardItem());
                std::string strtmp(attr->value());
                oCol.item(i-1)->setText(QString::fromStdString(strtmp));
            } else {
                nodeEnt++;
            }
        } else {
            while(nodeCol != 0) {
                i++;
                oCol.appendRow(new QStandardItem());
                if(nodeCol->value_size() == 0)
                    // Missing data
                    oCol.item(i-1)->setData(QString("NA"));
                else {
                    // HERE, AGAIN, DATA GETS TRANSFERED
                    std::string strtmp(nodeCol->value());
                    oCol.item(i-1)->setText(QString::fromStdString(strtmp));
                }

                nodeCol = nodeCol->next_sibling(strColName);
            }
        }

        // To treat cases where there are entities and parents with the same name
        if(i > 0) {
            if(nodeEnt->next_sibling(strEntName) == 0 && qstrParentName.compare(QString("")) != 0) {
                nodeParent = nodeEnt->parent();
                while(nodeParent != nodeFirst && strcmp(nodeParent->name(), strParentName) != 0) {
                    nodeParent = nodeParent->parent();
                }
                if(nodeParent == nodeFirst){
                    return;
                }
                nodeParent = nodeParent->next_sibling();
                if(nodeParent) {
                    nodeEnt = first(strEntName, nodeParent);
                    while(nodeEnt == 0) {
                        nodeParent = nodeParent->next_sibling();
                        if(nodeParent == 0) {
                            break;
                        }
                        nodeEnt = first(strEntName, nodeParent);
                    }
                } else {
                    break;
                }
            } else {
                nodeEnt = nodeEnt->next_sibling(strEntName);
            }
        }
    }
}

[Leonardo]

Hi. Why are you calling toLocal8Bit() and not toUtf8() when converting your xml from QString to char*? As you know for sure it's utf8, you shouldn't rely on the system to get the proper encoding.

http://doc.qt.io/qt-5/qstring.html#toLocal8Bit

[paulo.hubert]

Oh well. I didn't have a reason for doing that, it was just plain ignorance.
I corrected that, and also one more thing. I eliminate the std::string from

std::string strtmp(attr->value());
oCol.item(i-1)->setText(QString::fromStdString(strtmp));

Standard c++ strings might not cope well with UTF-8 encoding. So I changed it to

QString strtmp(attr->value());
oCol.item(i-1)->setText(strtmp);

These two changes combined have solved it. Thank you very much!