OpenSSL AES encryption and decryption functions don't work as expected in Qt GUI

[DoubleC122]

I have two AES encryption and decryption functions using OpenSSL and if called in main, they work 100%, as I tested. An unfortunate behavior arises when I use them in a Qt gui.

First, the two functions:

void zipUtils::encryptAES(std::string file, QByteArray passph)
{

    FILE *iF = fopen(file.c_str(), "rb");
    std::string oFile = file+".qenc";
    FILE *oF = fopen(oFile.c_str(), "wb");

    fseek(iF, 0L, SEEK_END);
    int fsize = ftell(iF);
    fseek(iF, 0L, SEEK_SET);

    int out = 0;
    int out2 = 0;
    unsigned char *inD = ( unsigned char * )malloc(fsize);
    unsigned char *outD = ( unsigned char * )malloc(fsize*2);

    QCryptographicHash hash = QCryptographicHash(QCryptographicHash::Sha3_256);
    hash.addData(passph);
    QByteArray res = hash.result().toHex();

    QByteArray initIV = randomBytes(256);

    unsigned const char* ckey = (unsigned const char*)res.toStdString().c_str();
    unsigned const char* ivec = (unsigned const char*)initIV.toStdString().c_str();

    fread(inD,sizeof(char),fsize, iF);

    EVP_CIPHER_CTX* ctx = EVP_CIPHER_CTX_new();

    if(!EVP_EncryptInit(ctx,EVP_aes_256_cbc(),ckey,ivec))
    {
        std::cout<<"EVP_EncryptInit() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_EncryptUpdate(ctx,outD,&out,inD,fsize))
    {
        std::cout<<"EVP_EncryptUpdate() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_EncryptFinal(ctx,outD+out,&out2))
    {
        std::cout<<"EVP_EncryptFinal() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    fwrite(outD,sizeof(char),out+out2,oF);
    free(inD);
    free(outD);
    fclose(iF);
    fclose(oF);
} 

And the other one:

void zipUtils::decryptAES(std::string file, QByteArray passph)
{
    FILE *iF = fopen(file.c_str(), "rb");
    boost::filesystem::path oFile(file);
    FILE *oF = fopen(oFile.stem().c_str(), "wb");

    fseek(iF, 0L, SEEK_END);
    int fsize = ftell(iF);
    fseek(iF, 0L, SEEK_SET);

    int out = 0;
    int out2 = 0;
    unsigned char *inD = ( unsigned char * )malloc(fsize);
    unsigned char *outD = ( unsigned char * )malloc(fsize*2);

    QCryptographicHash hash = QCryptographicHash(QCryptographicHash::Sha3_256);
    hash.addData(passph);
    QByteArray res = hash.result().toHex();

    QByteArray initIV = randomBytes(256);

    unsigned const char* ckey = (unsigned const char*)res.toStdString().c_str();
    unsigned const char* ivec = (unsigned const char*)initIV.toStdString().c_str();

    fread(inD,sizeof(char),fsize, iF); // Read Entire File

    EVP_CIPHER_CTX * ctx = EVP_CIPHER_CTX_new();

    if(!EVP_DecryptInit(ctx,EVP_aes_256_cbc(),ckey,ivec))
    {
        std::cout<<"EVP_DecryptInit() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_DecryptUpdate(ctx,outD,&out,inD,fsize))
    {
        std::cout<<"EVP_DecryptUpdate() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    if(!EVP_DecryptFinal(ctx, outD+out, &out2))
    {
        std::cout<<"EVP_DecryptFinal() error: "<<ERR_error_string(ERR_get_error(), NULL);
    }

    fwrite(outD,sizeof(char),out+out2,oF);
    free(inD);
    free(outD);
    fclose(iF);
    fclose(oF);
}

As I stated, these worked perfectly if used in main(). Now, in my gui, I use a contextual menu and once I click the specific action, I trigger a slot that executes inside it the encryption or decryption respectively.

The encryption slot is this:

void MainWindow::encrypt(QString file)
{
    boost::filesystem::path path(file.toStdString());
    chdir(path.parent_path().c_str());
    QByteArray pass;
    bool ok;
    QInputDialog dg;
    QString text = dg.getText(this, tr("Encrypt"),
                            tr("Add passphrase:"), QLineEdit::Normal,
                            QDir::home().dirName(), &ok);

    if (ok && !text.isEmpty())
    {
        pass = QByteArray::fromStdString(text.toStdString());
    }
    else dg.close();

    zipUtils::encryptAES(path.filename().string(), pass);
} 

And the decryption slot:

void MainWindow::decrypt(QString file)
{
    boost::filesystem::path path(file.toStdString());
    chdir(path.parent_path().c_str());
    QByteArray pass;
    bool ok;
    QInputDialog dg;
    QString text = dg.getText(this, tr("Decrypt"),
                            tr("Enter passphrase:"), QLineEdit::Normal,
                            QDir::home().dirName(), &ok);

    if (ok && !text.isEmpty())
    {
        pass = QByteArray::fromStdString(text.toStdString());
    }
    else dg.close();

    zipUtils::decryptAES(path.filename().string(), pass);
}

Then, I have a slot that pops the contextual menu, and and I add the encryption action like this: QAction* action5 = myMenu.addAction("Encrypt"); and the decryption action is just in the same way.

Then, I use a QSignalMapper to connect the slot to my action:

connect(action5, SIGNAL(triggered(bool)), mapper5, SLOT(map()));
connect(mapper5, SIGNAL(mapped(QString)), this, SLOT(encrypt(QString)));
mapper5->setMapping(action5, Fpath);

The action above is similar for both encryption and decryption. Now, the real issue comes after all of this. When I click on a file (an archive in my case for instance) I call the encrypt() slot, which indeed creates the encrypted version of that archive. But then, when I click on that encrypted file, of course the slot decrypt() is called, and it indeed decrypts that specific file, but in the archive there are no more files.

If instead I try to call both encryptAES() and decryptAES() in one of the two slots (this was done purely as an experiment), for instance encrypt(), it works, the archive is restored from the encrypted file and still has all its contents.

I could place the two functions anywhere in my gui code, and they would work, but the problem is when I call them from those slots, they stop working properly.

Also, after the those operations, if I close the gui, I get the following error from the decrypt function in my console: EVP_DecryptFinal() error: error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decryptPress

[SGaist]

Hi,

First thing I'd do is cleanup the types you are using, there's a mix of boost::filesystem generated path from QString, then again there's back and forth between QByteArray and QString using std::string which doesn't really make sense.

Did you check that at every stage you are creating the correct data to pass to your encryption/decryption functions ?

[DoubleC122]

Yes, sorry for the mess but I use QByteArray to get the hash result from QCryptographicHash. Anyhow, I found out that the key in the form of unsigned const char* ckey somehow did not store the key properly and was printing garbage. But by declaring the key as unsigned char ckey[256]; and setting the value with memcpy appeared to have solved the issue. To be honest, I never thought this was the issue because the functions worked perfectly in a separated test in main(). Probably I was too tired and that made me overlook all of this. Thank you for your reply.

[SGaist]

@DoubleC122 said in OpenSSL AES encryption and decryption functions don't work as expected in Qt GUI:

unsigned const char* ckey = (unsigned const char*) = res.toStdString().c_str();

That kind of construct is also dangerous. You are storing a pointer to data generated from a temporary variable. Which means it can be invalid at any time.

[DoubleC122]

@SGaist That probably also explains why I was getting garbage