QSslSocket Behaviour Linux vs macOS (possibly LibreSSL incompatibilities?)

[Markus M.]

Hello all!

Disclaimer: in case I'm in the wrong forums, please move this accordingly... thx

I am rather new in Qt development; I developed/am developing an application using QTcpServer along with a client app. Instead of QTcpSockets QSslSocket is used for obvious reasons.

Now I am experiencing issues with QSslSocket - while my server runs fine under Linux I get errors (QAbstractSocket Errors 20/21: Internal SSL error and invalid data, respectively.
Now, as said: the server runs fine in Linux (OpenSuSE/Ubuntu); I believe all parameters have been set as required (evidenced by the absence of errors in Linux).

I tried all sorts of things (e.g. invoking QSslSocket::ignoreSslErrors) - nothing seems to work.
In the end I suspect the problem indeed lies in the openssl implementation used on the Mac (latest High Sierra; apparently LibreSSL 2.2.7 is in place there).

By any chance - did anyone stumble across similar issues and can point me in the right direction? Any ideas on workarounds/solving greatly appreciated.

[SGaist]

Hi and welcome to devnet,

What version of Qt are you using ?
Is it self-built ?
Since a few versions, Qt on macOS is built using a backend that uses the Secure Transport framework from Apple.

[Markus M.]

@SGaist

Thank you....

I'm using the latest Qt 5.10 installed via the official online installer. So, no, not self built.

Edit: I might add: The implementation by and large seems rather picky. Adding / using certificates only works when using .pem endoded certificates. While using .crt is no problem in Linux, these result in error 13 (SSLHandshakeError) on the Mac

[SGaist]

Can you provide a minimal compilable example that shows that behaviour ?

[Markus M.]

@SGaist

I guess this method is where the magic happens:

void PCITcpConnections::accept(qintptr handle, PCITcpConnection *connection) {
QSslSocket *socket = new QSslSocket(this);

connect(socket, &QSslSocket::encrypted, connection, &PCITcpConnection::encrypted);
connect(socket,&QSslSocket::disconnected, this, &PCITcpConnections::disconnected);
connect(socket,static_cast<void (QSslSocket::*)(QAbstractSocket::SocketError)>(&QSslSocket::error),this,&PCITcpConnections::error);

const auto cert = QSslCertificate::fromPath("perceptronics_cert.pem", QSsl::Pem, QRegExp::Wildcard);
const auto CAcert = QSslCertificate::fromPath("CA_cert.pem", QSsl::Pem, QRegExp::Wildcard);

socket->addCaCertificates(CAcert);
socket->setPrivateKey("perceptronics_key.pem");
socket->setLocalCertificate(cert.at(0));
socket->setPeerVerifyMode(QSslSocket::VerifyNone);

QSslError error(QSslError::SelfSignedCertificate, cert.at(0));
QList<QSslError> expectedSslErrors;
expectedSslErrors.append(error);
socket->ignoreSslErrors(expectedSslErrors);

qDebug() << " ... accepting connection";

if(socket->setSocketDescriptor(handle)) {
    socket->startServerEncryption();
} else {
    qWarning() << this << "could not accept connection" << handle;
    connection->deleteLater();
    return;
}

qDebug() << " Mode: " << socket->mode();

connection->moveToThread(QThread::currentThread());
connection->setSocket(socket);

tcpConnections.insert(socket,connection);
qDebug() << this << "clients = " << tcpConnections.count();
emit socket->connected();

}

[Markus M.]

Follow up:

I compiled Qt from source using the latest openssl version (also compiled from source). Unfortunately no change in behaviour.

[SGaist]

I see you are using relative paths which means that these files might not be found at run time. For testing you should put the complete path to each of them.

[Markus M.]

@SGaist You are right - however, in the debugger I see the files are loaded. Except maybe the key file.
I noticed something weird I am about to dig into: even though I set the peerVerifyName in my client, the peerVerifyName on the server is empty.

Something must be wrong there. Still does not explain why it does work under Linux but fails on the Mac.
I also noticed: on the server the connected state is preserved despite the mentioned errors, whereas on the client a network timeout is experienced after some seconds.

[SGaist]

Might be a silly question but did you explicitly select the OpenSSL backend when compiling Qt ? Also, did you linked to OpenSSL or load at runtime ?

[Markus M.]

@SGaist There are no silly questions - just maybe silly answers :-)

1: yes, I set the OpenSSL backend when compiling.
2: I linked.

That said this is also something to look into; I am not fully clear how Qt handles SSL and if I did it correctly. Assuming its an SSL issue I wonder why my Mac client can connect to the Linux server without issues.
But in general I agree, this might be the root cause. Unfortunately the error messages are not exactly verbose - it just says "internal SSL error"... could be anything

[SGaist]

1. Can you check with otool -L the resulting QtNetwork framework ? Just to ensure that it indeed used your version of OpenSSL and not the one from the system.

Are you using a custom made certificate ?

[Markus M.]

@SGaist Here's the otool output:

@rpath/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.10.0, current version 5.10.0)

Yes, I use self signed certificates. That is, a self signed CA certificate which in turn have been used to sign the server cert. According to the KeyChain app the certs are ok (ie trusted)

[SGaist]

It looks incomplete. That looks like just the library id.

I get:

@rpath/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.10.0, current version 5.10.1)
@rpath/QtCore.framework/Versions/5/QtCore (compatibility version 5.10.0, current version 5.10.1)
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
/System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.31.2)
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1450.15.0)
/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 822.19.0)
/System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 963.30.1)
/usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
/System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork (compatibility version 1.0.0, current version 893.13.1)

[Markus M.]

@SGaist Silly me; I only pasted the one line affecting QtNetwork. Sorry for that. Here we go:

@rpath/QtWidgets.framework/Versions/5/QtWidgets (compatibility version 5.10.0, current version 5.10.0)
@rpath/QtGui.framework/Versions/5/QtGui (compatibility version 5.10.0, current version 5.10.0)
@rpath/QtCore.framework/Versions/5/QtCore (compatibility version 5.10.0, current version 5.10.0)
/System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/IOKit.framework/Versions/A/IOKit (compatibility version 1.0.0, current version 275.0.0)
@rpath/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.10.0, current version 5.10.0)
/System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
/System/Library/Frameworks/AGL.framework/Versions/A/AGL (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

fyi: I reverted back to the standard Qt version, ie not using the self compiled one for now. Hence the output above might not be what you expected.

For now, I try to figure out a couple of things (e.g why my peerValidName etc are not set on the socket) and how to load ssl at runtime

[SGaist]

Indeed, it was the one from your self-compiled Qt that is of interest.

As for loading at run time, you pass the -openssl-runtime option.

[Markus M.]

@SGaist Thank you for now... appreciated.

I am about to re-compile Qt which will take a couple of hours (about 4 to 5 hours yesterday) on a 2017 MBP 15...

Will keep you advised on my efforts

[SGaist]

Don't compile all of Qt. For your testing you likely only need qtbase. If more, then build only the modules you need after qtbase (or pass a list of -skip options for all modules you don't use in your application).

[Markus M.]

@SGaist I am aware, thank you. That is exactly what I am researching right now - what modules to compile. Don't want to be too restrictive though 'cause if successful I'll use this version for my client and other apps, too.

[SGaist]

What modules are you using for it currently ?

[Markus M.]

@SGaist

core, widgets and network. This is only for the server, the client also requires gui.

Its just crazy, just "make clean" takes forever