@silau It looks good but ldd doesn't always report exactly what is actually happening when you run your exe. It's a good start though. If you really wanted to trace it further you would have to ldd on each of those libs to see what they link. One of them could be using the 5.6 or whatever other version you may have. That being said I find that to be a bit unlikely though.
When I get a little bit of time later today I will run your code using Arch linux and 5.11 and see what happens. That will help determine if maybe it's a bug in Qt (which I highly doubt for something that big).
If it works for me the next steps for you would be to run it in the debugger and step through the working 5.6 one during socket connections and then the broken 5.11 one and see what is different.
Another option is take it down to the simplest form and just use the chat example or something from Qt and see if it works with 5.11 and then try to see what is different.
I'll let you know what I find out on my linux/Qt 5.11 in a bit.
Edit: Didn't get a chance today, was really busy. I will run a quick test tomorrow some time though.
Don't get this the wrong way, but are you coming from Java? I'm asking because this:
Doesn't make any sense in C++. What you probably want is to use the scope operator - :: to access one of the enum values. As for the error, please provide the full implementation of your TCP server and mention if you use threading, if so how. Also as mentioned by @koahnig
new QTcpSocket(new QObject());
This leaks 2 objects one socket and one generic QObject. You need to provide a valid pointer from the object tree to make use of the automatic deletion Qt supplies for child objects.
I currently have a very similar problem. I'm implementing a send mail client which should gracefully fall back to an unencrypted connection if encryption fails (and the user has decided to go ahead anyway). For that purpose I've connected to the QAbstractSocket::error() signal.
While I do get the signal it's ultimately moot since the code in question (QSslSocketBackendPrivate::startHandshake() in qsslsocket_openssl.cpp) immediately closes the socket by calling QSslSocket::abort() after emitting the signal. Because of that calling QSslSocket::ignoreSslErrors() is never an option.
I currently have no workaround in place. Maybe the only solution is to connect to the disconnected() signal and then check whether the connection was closed because of a handshake error. If it was, then open a new unencrypted socket/connection. I'm open to suggestions, though... :-)
We would need the whole implementation and debug it. But I guess it would be more productive when you watch some tutorial about high performance TCP server designs and stuff. This is a nice tutorial I can recommend: https://www.youtube.com/watch?v=zMgbVN3uQ-E