QTcpServer Design



  • Hello, I am looking to design a high throughput web server(millions to billions of hits a day). Each hit is going to have to parse the http header, get information for a database, construct a response, and send out a response. I have created a server, although I don't believe I am designing it right, as it is crashing under heavy load. A simple example of my server right now is as follows:

    @TCPInterface::TCPInterface(QObject *parent) : QTcpServer(parent)
    {
    listen(QHostAddress::Any,80);

    connect(this, SIGNAL(toDB(QTcpSocket*)),
    App::getInstance()->getDatabase(), SLOT(fromInterface(QTcpSocket*)),
    Qt::QueuedConnection);
    connect(App::getInstance()->getDatabase(), SIGNAL(sendToInterface(QTcpSocket*)),
    this, SLOT(fromDB(QTcpSocket*)),
    Qt::QueuedConnection);
    }

    void TCPInterface::incomingConnection(int socket)
    {
    QTcpSocket *s = new QTcpSocket(this);
    connect(s, SIGNAL(readyRead()), this, SLOT(readClient()));
    connect(s, SIGNAL(disconnected()), this, SLOT(discardClient()));
    connect(s, SIGNAL(destroyed()), this, SLOT(nullSocket()));
    s->setSocketDescriptor(socket);
    }

    void TCPInterface::readClient()
    {
    QTcpSocket* socket = (QTcpSocket*)sender();

    emit toDB(socket);
    }

    void TCPInterface::discardClient()
    {
    QTcpSocket* socket = qobject_cast<QTcpSocket*>(sender());
    if(socket != NULL)
    socket->deleteLater();
    }

    void TCPInterface::fromDB(QTcpSocket *socket)
    {
    QString tmp = "HTTP/1.0 200 Ok\r\n";
    tmp += "Content-Type: text/html; charset="utf-8"\r\n";
    tmp += "\r\nTest\n";

    if(socket != NULL && socket->isWritable() && socket->isOpen() && socket->state() == QTcpSocket::ConnectedState)
    socket->write(tmp.toAscii());
    if(socket != NULL && socket->state() == QTcpSocket::ConnectedState)
    socket->disconnectFromHost();
    }

    void TCPInterface::nullSocket()
    {
    QTcpSocket socket = qobject_cast<QTcpSocket>(sender());
    if(socket != NULL)
    socket = NULL;
    }

    void DatabaseInterface::fromInterface(QTcpSocket *socket)
    {
    emit sendToInterface(socket);
    }@

    I am not exactly sure what my problem is, but I believe it happens when a client disconnects from the server, and I try and use a socket that no longer exists. I do have back traces printing out, however they are not always the same, here are a few I have gotten:

    @Error: signal 11:
    ./SimpleTCPServer[0x401ce6]
    /lib64/libc.so.6[0x3c14032920]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN9QIODevice5writeEPKcx+0x22)[0x7fac6b5a6d22]
    ./SimpleTCPServer[0x401fb6]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN7QObject5eventEP6QEvent+0x38e)[0x7fac6b62cbae]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN16QCoreApplication14notifyIntern alEP7QObjectP6QEvent+0x8c)[0x7fac6b61a30c]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN23QCoreApplicationPrivate16sendP ostedEventsEP7QObjectiP11QThreadData+0x3d3)[0x7fac6b61ebf3]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(+0x1c7cf3)[0x7fac6b64bcf3]
    /lib64/libglib-2.0.so.0(g_main_context_dispatch+0x22e)[0x3c15438f0e]
    /lib64/libglib-2.0.so.0[0x3c1543c938]
    /lib64/libglib-2.0.so.0(g_main_context_iteration+0x7a)[0x3c1543ca3a]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN20QEventDispatcherGlib13processE ventsE6QFlagsIN10QEventLoop17ProcessEventsFlagEE+0 x73)[0x7fac6b64b833]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN10QEventLoop13processEventsE6QFl agsINS_17ProcessEventsFlagEE+0x32)[0x7fac6b618ec2]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN10QEventLoop4execE6QFlagsINS_17P rocessEventsFlagEE+0x164)[0x7fac6b619334]
    /home/root/QtSDK/Desktop/Qt/4.8.1/gcc/lib/libQtCore.so.4(_ZN16QCoreApplication4execEv+0xb9)[0x7fac6b61efc9]
    ./SimpleTCPServer[0x401c95]
    /lib64/libc.so.6(__libc_start_main+0xfd)[0x3c1401ecdd]
    ./SimpleTCPServer[0x401b59]@

    @Error: signal 11:
    SimpleTCPServer[0x4200b6]
    /lib64/libc.so.6[0x3cd1232920]
    SimpleTCPServer[0x444a18]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtCore.so.4(+0x89c5b)[0x7fd32140dc5b]
    /lib64/libpthread.so.0[0x3cd1a07851]
    /lib64/libc.so.6(clone+0x6d)[0x3cd12e76dd]@

    @Error: signal 11:
    SimpleTCPServer[0x4200b6]
    /lib64/libc.so.6[0x3cd1232920]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtNetwork.so.4(_ZNK15QAbstractSocket5stateEv+0x4)[0x7fd218e85ee4]
    SimpleTCPServer[0x4449fc]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtCore.so.4(+0x89c5b)[0x7fd218975c5b]
    /lib64/libpthread.so.0[0x3cd1a07851]
    /lib64/libc.so.6(clone+0x6d)[0x3cd12e76dd]@

    @Error: signal 11:
    SimpleTCPServer[0x4200b6]
    /lib64/libc.so.6[0x3cd1232920]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtNetwork.so.4(+0xba512)[0x7f576d893512]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtNetwork.so.4(_ZN15QAbstractSocket19setSocketDescriptorEiNS_11SocketStateE6QFlagsIN9QIODevice12OpenModeFlagEE+0xb3)[0x7f576d89fd03]
    SimpleTCPServer[0x429996]
    /root/qt-everywhere-opensource-src-4.8.1/lib/libQtNetwork.so.4(+0xcfad8)[0x7f576d8a8ad8]@

    Any advice on how to redesign my server would be greatly appreciated or if you happen to know why my server is crashing.

    Thank you


  • Moderators

    You can check the "state of the socket":http://qt-project.org/doc/qt-4.8/qabstractsocket.html#state prior to use.
    I am not using sender(). I believe someone noted lately that it is not very reliable.



  • Hi, I tried using that to see the state, and one of two things happen. As you can in the code I posted here:

    @if(socket != NULL && socket->isWritable() && socket->isOpen() && socket->state() == QTcpSocket::ConnectedState)
    socket->write(tmp.toAscii());
    if(socket != NULL && socket->state() == QTcpSocket::ConnectedState)
    socket->disconnectFromHost();@

    I do check the state.

    On this simple sample server, this will usually result in a back trace saying

    @QIODevice::write(char const*, long long);@

    was a problem, which I am assuming is the

    @socket->write(tmp.toAscii());@

    line. However, on my actual server, where there is a slightly bigger delay from database access, I get a back trace(third back trace in orginal post) including:

    @QAbstractSocket::state() const@

    Which I believe is from checking the state of a deleted socket pointer, even though all my my state checks a prefaced with a check to see if they have been deleted or not(socket != NULL). So I assuming that I am getting bad timing, where my sockets are being deleted by deleteLater just as I check my sockets state, although it seems unlikely that this timing(After NULL check but before state check) happens as often as it does, so it may be cause by something else.


  • Moderators

    Sorry, did not read careful enough your listings.
    Unfortunately, I could not dig out the post referring to the issue when using sender(). I am not experienced enough with linux to be of help with your listings.



  • I see, I would love to know the alternative to sender(), I tried searching around, but could not find anything. Does the flow of my server seem right? Unfortunately I am fairly new to Qt programming, so I might have a basic fundamental wrong. Should I be able to pass a QTcpSocket pointer between threads, and if necessary call some functions from it?

    I also read somewhere that sockets run asynchronously. Does this mean if I subclass the QTcpSocket class, that any work they do will run like a thread, and I can do all of my work from that class and it will utilize all of my cores?

    I am just trying to get more knowledge about how Qt and the QTcpServer/QTcpSocket work in general, so I can make my design work. I think the errors are coming from my design, rather than the code bugs.


  • Moderators

    This is the basis of my implementations with a wrapper to QTcpSocket. Since the dataReady signal gives you already the socket, you do not need to use sender().

    I have also a wrapper for QTcpServer which basically has a container to hold all sockets connected. However, my servers have to send the same information to all sockets connected, which are only a few.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.