Issue opening a QFile on Windows when the file name is read from command line and contains accents...

[mbruel]

Hi,
My app is reading from command line the name of a file that I store in a QString.
If the file name contains accents (like a French é) I don't manage to neither open the file in a QFile nor printing out again the name properly in the command line.
I've tried to play with QTextCodec::setCodeForLocale(QTextCodec::codecForName("utf-8")) or using toLocale8Bit() when storing my QString but none works...
Any help?

PS: I'm reading from command line using QString QCommandLineParser::value(const QString &optionName) const

[SGaist]

Hi,

Which OS are you using ?
Could you provide a minimal compilable example to try it out ?

[mbruel]

@SGaist it's on the title, the issue is on Windows only. (I'm using 7 but it was reported on 10 too)
I don't have a minimal example but alright I gonna make it, shouldn't be so long.

[mbruel]

@SGaist so here is a minimal project: testQFileWithAccentsInName
it's just a main:

#include <QCoreApplication>
#include <QCommandLineParser>
#include <iostream>
#include <QTextStream>
#include <QFile>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    const QString optInput = "input";
    const QList<QCommandLineOption> sCmdOptions = {
        {{"i", optInput}, "input file to upload (single file or directory), you can use it multiple times", optInput}
    };
    QCommandLineParser parser;
    parser.setApplicationDescription("testQFileWithAccentsInName");
    parser.addOptions(sCmdOptions);

    // Process the actual command line arguments given by the user
    QStringList args;
    for (int i = 0; i < argc; ++i)
        args << argv[i];

    QTextStream cout(stdout);
    if (!parser.parse(args) || !parser.isSet(optInput))
    {
        cout << QString("Error syntax... it should be: testQFileWithAccentsInName -i <path_to_file>\n");
        return 1;
    }

    std::cout << "argv[2] using stdio: " << argv[2] << "\n";
    cout << "argv[2] using QTextStream: " << argv[2] << "\n";

    QString filePath = parser.value(optInput);
    cout << "Try to open file " << filePath << "\n" << Qt::flush;
    QFile file(filePath);
    int res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    filePath = argv[2];
    cout << "Try to open file2 " << filePath << "\n" << Qt::flush;
    QFile file2(filePath);
    res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;


    return res ? 0 : 1;
}

So you need one command line option using -i then path to an existing file
basically: testQFileWithAccentsInName.exe -i filePath

If you run it a file named for example tést.txt it will fail to open it... (on Windows)

[jeremy_k]

Does the application create a QCoreApplication instance prior to reading the string? If not, try that. QCoreApplication::QCoreApplication initializes Qt's locale support.

[mbruel]

@jeremy_k well it was, I removed it in the example cause it had the same result.
anyway I put it back in the code.

[jsulm]

@mbruel What do https://doc.qt.io/qt-5/qfiledevice.html#error and https://doc.qt.io/qt-5/qiodevice.html#errorString return?

[mbruel]

@jsulm code updated in the previous post or here on github
The error is that it doesn't find the file... although it exists and has been provided using completion in the cmd.

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
Try to open file c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

my Windows7 is installed in French in a VM using British keyboard on a Debian host with locale en_GB.UTF-8
cf snapshot.

[JonB]

@mbruel
I don't know the answer, but may I throw out a couple of comments:

• I would start by seeing exactly what I receive in argv[2], and then relate that to QString/QCommandLineParser.

• I note that you type in e-acute character, yet when the code outputs the error message to the terminal it shows U-acute where it should show e-acute. Is there any significance/problem there?

[mbruel]

@JonB I've updated the code above and here
Here is the output

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....
Try to open file2 c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

not sure why I see the e-acute é well when using completion and then as Ú when it is printed out...
no idea how works Windows locale.
as said it's a French Win7. I've the issue whatever keyboard (layout) I use on Windows (French or English). My laptop has a British keyboard and again the host is a Debian with LANG=en_GB.UTF-8

it doesn't seem related to QCommandLineParser as I've the same issue trying to open another QFile using directly a QString initialized with argv[2].

[JonB]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

same issue trying to open another QFile using directly a QString initialized with argv[2]

Personally I would want to try opening the file by, say, fopen(argv[2]). Like you I have "no idea how works Windows locale." and never have done :) But it would help me understand whether I have some QString/locale problem or something more fundamental about receiving the right characters from the command line/prompt.

[mbruel]

@JonB code updated, it's opening the file read in the cmd when I use directly argv[2] on fopen

#include <QCoreApplication>
#include <iostream>
#include <cstdio>

#include <QCommandLineParser>
#include <QTextStream>
#include <QFile>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    const QString optInput = "input";
    const QList<QCommandLineOption> sCmdOptions = {
        {{"i", optInput}, "input file to upload (single file or directory), you can use it multiple times", optInput}
    };
    QCommandLineParser parser;
    parser.setApplicationDescription("testQFileWithAccentsInName");
    parser.addOptions(sCmdOptions);

    // Process the actual command line arguments given by the user
    QStringList args;
    for (int i = 0; i < argc; ++i)
        args << argv[i];

    QTextStream cout(stdout);
    if (!parser.parse(args) || !parser.isSet(optInput))
    {
        cout << QString("Error syntax... it should be: testQFileWithAccentsInName -i <path_to_file>\n");
        return 1;
    }

    std::cout << "argv[2] using stdio: " << argv[2] << "\n";
    cout << "argv[2] using QTextStream: " << argv[2] << "\n";

    QString filePath = parser.value(optInput);
    cout << "Try to open file " << filePath << "\n" << Qt::flush;
    QFile file(filePath);
    int res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    filePath = argv[2];
    cout << "\nTry to open file2 " << filePath << "\n" << Qt::flush;
    QFile file2(filePath);
    res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    FILE *cFile = fopen(argv[2], "r");
    cout << "\nTry to open cFile using fopen on argv[2]\n" << Qt::flush;
    cout << (cFile ? "OK\n" : "ERROR\n") << Qt::flush;
    if (cFile)
        fclose(cFile);

    return res ? 0 : 1;
}

output:

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

Try to open file2 c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

Try to open cFile using fopen on argv[2]
OK

So to me its a locale / QString issue. Following this link I've tried to play with QTextCodec::setCodeForLocale(QTextCodec::codecForName("utf-8")) or QString::toLocale8Bit() without success.
That's why I came ask for help here ;)

PS: I'm having the issue with the e-acute é but the issue was raised by Spanish people on their windows (without VM)

[JonB]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

So to me its a locale / QString issue.

Indeed at least you have proved that! If it's doable via fopen(argv[2]) it must be doable via QFile(QString), somehow.

That's why I came ask for help here ;)

Hopefully someone else will answer, because I have never got my head around locales, Latin/UTF-8 characters et al... :)

[mbruel]

@JonB said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

Hopefully someone else will answer, because I have never got my head around locales, Latin/UTF-8 characters et al... :)

same always been kind of a nightmare to me... haha

[Christian Ehrlicher]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

QStringList args;
for (int i = 0; i < argc; ++i)
    args << argv[i];

wtf?

Use QCoreApplication::arguments() instead. What you do can't work at all since the command line does not pass the values utf-8 encoded.

[JonB]

@mbruel
I knew we needed a Qt expert here! @Christian-Ehrlicher understands things like UTF, Latin etc. ! ;-)
Take his advice.

[mbruel]

@Christian-Ehrlicher said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

QStringList args;
for (int i = 0; i < argc; ++i)
    args << argv[i];

wtf?

Use QCoreApplication::arguments() instead. What you do can't work at all since the command line does not pass the values utf-8 encoded.

Cool! first time I hear about QCoreApplication::arguments()...
problem solved. (indeed my loop looked a bit dodgy/hacky... haha)
Thanks \o/

Updated code.

#include <QCoreApplication>
#include <iostream>
#include <cstdio>

#include <QCommandLineParser>
#include <QTextStream>
#include <QFile>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    const QString optInput = "input";
    const QList<QCommandLineOption> sCmdOptions = {
        {{"i", optInput}, "input file to upload (single file or directory), you can use it multiple times", optInput}
    };
    QCommandLineParser parser;
    parser.setApplicationDescription("testQFileWithAccentsInName");
    parser.addOptions(sCmdOptions);

    QStringList args = QCoreApplication::arguments();
    QTextStream cout(stdout);
    if (!parser.parse(args) || !parser.isSet(optInput))
    {
        cout << QString("Error syntax... it should be: testQFileWithAccentsInName -i <path_to_file>\n");
        return 1;
    }

    std::cout << "argv[2] using stdio: " << argv[2] << "\n";
    cout << "argv[2] using QTextStream: " << argv[2] << "\n";

    QString filePath = parser.value(optInput);
    cout << "Try to open file using QCommandLineParser: " << filePath << "\n" << Qt::flush;
    QFile file(filePath);
    int res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;
    if (res)
        file.close();

    filePath = args.at(2);
    cout << "\nTry to open file using directly QCoreApplication::arguments: " << filePath << "\n" << Qt::flush;
    QFile file2(filePath);
    res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    FILE *cFile = fopen(argv[2], "r");
    cout << "\nTry to open cFile using fopen on argv[2]\n" << Qt::flush;
    cout << (cFile ? "OK\n" : "ERROR\n") << Qt::flush;
    if (cFile)
        fclose(cFile);

    return res ? 0 : 1;
}

and output:

C:\Users\mb>cd Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug\

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file using QCommandLineParser: c:\Users\mb\Downloads\tÚst.nzb
OK

Try to open file using directly QCoreApplication::arguments: c:\Users\mb\Downloads\tÚst.nzb
OK

Try to open cFile using fopen on argv[2]
OK

[ChrisW67]

I think you have been bitten by a character encoding issues.

The file name looks like "tést.nzb" when viewed in the QtCreator execution parameters field.
Internally this will be using a QString with the following Unicode code points (hex):

t    é    s    t    .    n    z    b 
0074 00E9 0073 0074 002E 006E 007A 0062

The same logical characters can be encoded in Windows-1252 (Western Euro) or UTF-8:

t  é     s  t  .  n  z  b
74 E9    73 74 2E 6E 7A 62  Windows-1252, 8-bits per character
74 C3 A9 73 74 2E 6E 7A 62  UTF-8, 8 to 32-bits per character

It is these conversions that Qt toLocal8Bit() and fromLocal8Bit() and friends do.
If a program expecting a Windows-1252 encode string was given bytes for the UTF-8 version you might see:

tést.nzb

Conversely, a Windows-1252 string fed to a program expecting UTF-8 would see that the byte E9 is not a valid UTF-8 encoding and substitute a "bad character" placeholder, often a "?".

t?st.nzb

This second option matches what your screen shot displays. This may not be exactly what is happening in your case, but it is indicative of the types of problem that can arise.

[SimonSchroeder]

Here are a few things I have learned over the years:

1. Make sure your source code files are stored as UTF-8 (if you have non-ASCII characters in there).
2. Make sure you let the Microsoft compiler know that you are using UTF-8 (I don't remember the switches; there might be one for the source files and another one for the object files).
3. On Windows add the code line setlocale(LC_ALL, ".UTF8"); which will make sure that std::cin, std::cout and all file I/O will be UTF-8 (also filenames!).

The last one is the most important one. I guess this would also tell your program that your command line parameters are UTF-8.