Issue opening a QFile on Windows when the file name is read from command line and contains accents...

[mbruel]

@JonB I've updated the code above and here
Here is the output

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....
Try to open file2 c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

not sure why I see the e-acute é well when using completion and then as Ú when it is printed out...
no idea how works Windows locale.
as said it's a French Win7. I've the issue whatever keyboard (layout) I use on Windows (French or English). My laptop has a British keyboard and again the host is a Debian with LANG=en_GB.UTF-8

it doesn't seem related to QCommandLineParser as I've the same issue trying to open another QFile using directly a QString initialized with argv[2].

[JonB]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

same issue trying to open another QFile using directly a QString initialized with argv[2]

Personally I would want to try opening the file by, say, fopen(argv[2]). Like you I have "no idea how works Windows locale." and never have done :) But it would help me understand whether I have some QString/locale problem or something more fundamental about receiving the right characters from the command line/prompt.

[mbruel]

@JonB code updated, it's opening the file read in the cmd when I use directly argv[2] on fopen

#include <QCoreApplication>
#include <iostream>
#include <cstdio>

#include <QCommandLineParser>
#include <QTextStream>
#include <QFile>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    const QString optInput = "input";
    const QList<QCommandLineOption> sCmdOptions = {
        {{"i", optInput}, "input file to upload (single file or directory), you can use it multiple times", optInput}
    };
    QCommandLineParser parser;
    parser.setApplicationDescription("testQFileWithAccentsInName");
    parser.addOptions(sCmdOptions);

    // Process the actual command line arguments given by the user
    QStringList args;
    for (int i = 0; i < argc; ++i)
        args << argv[i];

    QTextStream cout(stdout);
    if (!parser.parse(args) || !parser.isSet(optInput))
    {
        cout << QString("Error syntax... it should be: testQFileWithAccentsInName -i <path_to_file>\n");
        return 1;
    }

    std::cout << "argv[2] using stdio: " << argv[2] << "\n";
    cout << "argv[2] using QTextStream: " << argv[2] << "\n";

    QString filePath = parser.value(optInput);
    cout << "Try to open file " << filePath << "\n" << Qt::flush;
    QFile file(filePath);
    int res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    filePath = argv[2];
    cout << "\nTry to open file2 " << filePath << "\n" << Qt::flush;
    QFile file2(filePath);
    res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    FILE *cFile = fopen(argv[2], "r");
    cout << "\nTry to open cFile using fopen on argv[2]\n" << Qt::flush;
    cout << (cFile ? "OK\n" : "ERROR\n") << Qt::flush;
    if (cFile)
        fclose(cFile);

    return res ? 0 : 1;
}

output:

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

Try to open file2 c:\Users\mb\Downloads\t?st.nzb
ERROR #5: Le fichier spÚcifiÚ est introuvable....

Try to open cFile using fopen on argv[2]
OK

So to me its a locale / QString issue. Following this link I've tried to play with QTextCodec::setCodeForLocale(QTextCodec::codecForName("utf-8")) or QString::toLocale8Bit() without success.
That's why I came ask for help here ;)

PS: I'm having the issue with the e-acute é but the issue was raised by Spanish people on their windows (without VM)

[JonB]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

So to me its a locale / QString issue.

Indeed at least you have proved that! If it's doable via fopen(argv[2]) it must be doable via QFile(QString), somehow.

That's why I came ask for help here ;)

Hopefully someone else will answer, because I have never got my head around locales, Latin/UTF-8 characters et al... :)

[mbruel]

@JonB said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

Hopefully someone else will answer, because I have never got my head around locales, Latin/UTF-8 characters et al... :)

same always been kind of a nightmare to me... haha

[Christian Ehrlicher]

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

QStringList args;
for (int i = 0; i < argc; ++i)
    args << argv[i];

wtf?

Use QCoreApplication::arguments() instead. What you do can't work at all since the command line does not pass the values utf-8 encoded.

[JonB]

@mbruel
I knew we needed a Qt expert here! @Christian-Ehrlicher understands things like UTF, Latin etc. ! ;-)
Take his advice.

[mbruel]

@Christian-Ehrlicher said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

@mbruel said in Issue opening a QFile on Windows when the file name is read from command line and contains accents...:

QStringList args;
for (int i = 0; i < argc; ++i)
    args << argv[i];

wtf?

Use QCoreApplication::arguments() instead. What you do can't work at all since the command line does not pass the values utf-8 encoded.

Cool! first time I hear about QCoreApplication::arguments()...
problem solved. (indeed my loop looked a bit dodgy/hacky... haha)
Thanks \o/

Updated code.

#include <QCoreApplication>
#include <iostream>
#include <cstdio>

#include <QCommandLineParser>
#include <QTextStream>
#include <QFile>
int main(int argc, char *argv[])
{
    QCoreApplication a(argc, argv);

    const QString optInput = "input";
    const QList<QCommandLineOption> sCmdOptions = {
        {{"i", optInput}, "input file to upload (single file or directory), you can use it multiple times", optInput}
    };
    QCommandLineParser parser;
    parser.setApplicationDescription("testQFileWithAccentsInName");
    parser.addOptions(sCmdOptions);

    QStringList args = QCoreApplication::arguments();
    QTextStream cout(stdout);
    if (!parser.parse(args) || !parser.isSet(optInput))
    {
        cout << QString("Error syntax... it should be: testQFileWithAccentsInName -i <path_to_file>\n");
        return 1;
    }

    std::cout << "argv[2] using stdio: " << argv[2] << "\n";
    cout << "argv[2] using QTextStream: " << argv[2] << "\n";

    QString filePath = parser.value(optInput);
    cout << "Try to open file using QCommandLineParser: " << filePath << "\n" << Qt::flush;
    QFile file(filePath);
    int res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;
    if (res)
        file.close();

    filePath = args.at(2);
    cout << "\nTry to open file using directly QCoreApplication::arguments: " << filePath << "\n" << Qt::flush;
    QFile file2(filePath);
    res = file.open(QIODevice::ReadOnly|QIODevice::Text);
    cout << (res ? QString("OK\n")
                 : QString("ERROR #%1: %2...\n").arg(file.error()).arg(file.errorString()))
         << Qt::flush;

    FILE *cFile = fopen(argv[2], "r");
    cout << "\nTry to open cFile using fopen on argv[2]\n" << Qt::flush;
    cout << (cFile ? "OK\n" : "ERROR\n") << Qt::flush;
    if (cFile)
        fclose(cFile);

    return res ? 0 : 1;
}

and output:

C:\Users\mb>cd Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug\

C:\Users\mb\Desktop\github\build-testQFileWithAccentsInName-Desktop_Qt_5_15_0_MinGW_64_bit-Debug\debug>testQFileWithAccentsInName.exe -i c:\Users\mb\Downloads\tést.nzb
argv[2] using stdio: c:\Users\mb\Downloads\tÚst.nzb
argv[2] using QTextStream: c:\Users\mb\Downloads\tÚst.nzb
Try to open file using QCommandLineParser: c:\Users\mb\Downloads\tÚst.nzb
OK

Try to open file using directly QCoreApplication::arguments: c:\Users\mb\Downloads\tÚst.nzb
OK

Try to open cFile using fopen on argv[2]
OK

[ChrisW67]

I think you have been bitten by a character encoding issues.

The file name looks like "tést.nzb" when viewed in the QtCreator execution parameters field.
Internally this will be using a QString with the following Unicode code points (hex):

t    é    s    t    .    n    z    b 
0074 00E9 0073 0074 002E 006E 007A 0062

The same logical characters can be encoded in Windows-1252 (Western Euro) or UTF-8:

t  é     s  t  .  n  z  b
74 E9    73 74 2E 6E 7A 62  Windows-1252, 8-bits per character
74 C3 A9 73 74 2E 6E 7A 62  UTF-8, 8 to 32-bits per character

It is these conversions that Qt toLocal8Bit() and fromLocal8Bit() and friends do.
If a program expecting a Windows-1252 encode string was given bytes for the UTF-8 version you might see:

tést.nzb

Conversely, a Windows-1252 string fed to a program expecting UTF-8 would see that the byte E9 is not a valid UTF-8 encoding and substitute a "bad character" placeholder, often a "?".

t?st.nzb

This second option matches what your screen shot displays. This may not be exactly what is happening in your case, but it is indicative of the types of problem that can arise.

[SimonSchroeder]

Here are a few things I have learned over the years:

1. Make sure your source code files are stored as UTF-8 (if you have non-ASCII characters in there).
2. Make sure you let the Microsoft compiler know that you are using UTF-8 (I don't remember the switches; there might be one for the source files and another one for the object files).
3. On Windows add the code line setlocale(LC_ALL, ".UTF8"); which will make sure that std::cin, std::cout and all file I/O will be UTF-8 (also filenames!).

The last one is the most important one. I guess this would also tell your program that your command line parameters are UTF-8.