Don't get this the wrong way, but are you coming from Java? I'm asking because this:
Doesn't make any sense in C++. What you probably want is to use the scope operator - :: to access one of the enum values. As for the error, please provide the full implementation of your TCP server and mention if you use threading, if so how. Also as mentioned by @koahnig
new QTcpSocket(new QObject());
This leaks 2 objects one socket and one generic QObject. You need to provide a valid pointer from the object tree to make use of the automatic deletion Qt supplies for child objects.
I currently have a very similar problem. I'm implementing a send mail client which should gracefully fall back to an unencrypted connection if encryption fails (and the user has decided to go ahead anyway). For that purpose I've connected to the QAbstractSocket::error() signal.
While I do get the signal it's ultimately moot since the code in question (QSslSocketBackendPrivate::startHandshake() in qsslsocket_openssl.cpp) immediately closes the socket by calling QSslSocket::abort() after emitting the signal. Because of that calling QSslSocket::ignoreSslErrors() is never an option.
I currently have no workaround in place. Maybe the only solution is to connect to the disconnected() signal and then check whether the connection was closed because of a handshake error. If it was, then open a new unencrypted socket/connection. I'm open to suggestions, though... :-)
We would need the whole implementation and debug it. But I guess it would be more productive when you watch some tutorial about high performance TCP server designs and stuff. This is a nice tutorial I can recommend: https://www.youtube.com/watch?v=zMgbVN3uQ-E