QTcpSocket not receiving POST data from web page



  • Hi everyone!
    I'm writting a "web server" for an application I'm developing and when I send a POST to the web server it only gets the following:

    @"POST /login HTTP/1.1
    Host: localhost
    Connection: keep-alive
    Content-Length: 49
    Accept: /
    Origin: http://localhost
    X-Requested-With: XMLHttpRequest
    User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko)
    Chrome/33.0.1750.154 Safari/537.36
    Content-Type: application/x-www-form-urlencoded; charset=UTF-8
    Referer: http://localhost/inicio.html
    Accept-Encoding: gzip,deflate,sdch
    Accept-Language: es-ES,es;q=0.8

    "@

    The missing data can be read only when new post request arrives:

    @"user=admin&pass=admin&token=1234POST /login HTTP/1.1"@

    The code that reads the socket is:

    @void MyClass::readClient() // Slot connected to readReady signal of the QTcpSocket
    {
    if (disabled)
    return;

    QTcpSocket* socket = (QTcpSocket*)sender();
    
    if (socket->canReadLine()) {
    
        QString data(socket->readLine());
    
        processRequest(data, socket); // parse the data and send the response through the socket
    }
    
    qDebug() << "Bytes Available before writing" << socket->bytesAvailable();
    socket->waitForBytesWritten();
    qDebug() << "Bytes Available after writing" << socket->bytesAvailable();
    qDebug() << "Data missed:" << QString(socket->readAll());
    
    socket->close();
    
    QtServiceBase::instance()->logMessage("Wrote to client");
    
    if (socket->state() == QTcpSocket::UnconnectedState) {
        delete socket;
        QtServiceBase::instance()->logMessage("Connection closed");
    }
    

    }@

    The output of this codes is:

    @"Bytes Available before writing 0
    Bytes Available after writing 32
    Data missed: "user=admin&pass=admin&token=1234""@

    It seems that until the socket is flushed (in a way) I'm not being able to read the data which doesn't finish with a new line.

    I hope I was clear enough
    Thanks in advance



  • I think canReadLine() in line 8 waits until a line becomes available which happens when data arrive. Try to use bytesAvailable() instead and see if it will help. Also readLine() will read only one line.
    A line is a string that ends with '\r' I think. So it better to use readAll() and use a while() loop
    @
    QBytesArray data;
    while(socket->bytesAvailable()) {
    data.append(socket->readAll())
    }

    process(data);

    @



  • Hi,
    Sorry but something went wrong in the copy-paste step.
    I'm using the following code:

    @
    ...
    QString data(socket->readAll());

        processRequest(data, socket); // parse the data and send the response through the socket
    

    ...@

    The readLine() its been used to check if it is a GET request or a POST request. My code is not actually this one but it basically does what it is posted. socket->bytesAvailable() returns 0 before socket->waitForBytesWritten() and 32 after that line



  • Hi, the data can be transferred with multiple (TCP) packets depending on the size, so usually you write the data to a buffer until you have received the whole data and only after that process it or you might end up with partial or corrupted data.
    I am not an HTTP expert but in most cases the packet size will be prepended to the actual data, so if you send a string for example it will first send the length of the string and than the characters so you can simply check if the whole string has been received before you process it.
    I believe the Qt socket example explain that and also use a buffer until all data is received, in case you don't know what i mean. If its not to much data you can let the socket itself buffer it until you've received enough data to process, so you might get a few calls to the "readyRead" slot before all data is available.



  • Hi Xander84,
    Thanks for the reply but unfortunately I've tried that. The result was that the process was blocked trying to read the 32 bytes, but they were never available (or I'm assuming they were not available). With the chrome debugger (CTRL + SHIFT + J) I can see that the full post request with the data is being sent.
    I think that as the data (final part of the message) is missing a '\r' or '\n' character, the QTcpSocket library can't read it



  • You are closing socket at the end of readClient().
    Do you reopen it again on each incoming packet?
    It is not right because you are loosing incoming buffer when you close socket.
    Next packet will come as new connection and all data that were in the buffer after first readLine() or readAll() is gone.



  • andreyc is right, why are you closing the socket anyway? most web servers leave the socket open for some minutes maybe. It's way faster if you leave it open unless you know you have only one request every 5 minutes or something.
    Also the header "Connection: keep-alive" is good for something, the client doesn't want you to close the socket :D



  • Hi andreyc, thanks for the response
    The problem is that the data is available in the first readClient() call, but after sending some data in response (lines 15, 16, and 17). Nevertheless, I've tried not closing the socket with the same result



  • I've tried this myself now with a very small example code and it works as it should be!? here my code:
    @
    QTcpServer *server = new QTcpServer;
    connect(server, &QTcpServer::newConnection, [=]{
    QTcpSocket *client = server->nextPendingConnection();
    qDebug() << client->peerAddress() << client->peerPort();
    connect(client, &QTcpSocket::readyRead, [=]{
    qDebug() << client->bytesAvailable() << client->readAll();
    });
    });
    qDebug() << server->listen(QHostAddress::LocalHost, 80);
    @
    I just print everything to the debug console.
    example output send from chrome/postman extension as post data
    @
    true
    QHostAddress( "127.0.0.1" ) 7549
    0 "POST / HTTP/1.1

    Host: localhost

    Connection: keep-alive

    Content-Length: 19

    Cache-Control: no-cache

    Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop

    Content-Type: application/x-www-form-urlencoded

    User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36

    Postman-Token: 0d5e263b-dd15-a29d-6621-06d89c52fafa

    Accept: /

    Accept-Encoding: gzip,deflate,sdch

    Accept-Language: de,en;q=0.8

    user=admin&pass=123"
    QHostAddress( "127.0.0.1" ) 7550
    @
    see the post data at the end? so everything is there I just used this two parameters for the test. Also notice I get two TCP connection, sometimes even three. Don't ask me why but the only one is sending the data. I am a little bit confused because the "client->bytesAvailable()" return 0 but as you can see the whole package is there right after that with "client->readAll()"!?



  • Xander84,
    It is strange. Now I'm doing a peek of all data at the beginning of the slot and I'm not getting the data before the server response but after. What version of Qt are you using? Mine is 4.7.4
    Do not worry about the multiple connections, I have the same problem. It must be a HTML issue



  • I'm using Qt 5.2.1, I don't know if that is a bug in your Qt version :/
    Maybe you can just use a newer version to verify if your code is working there?
    also did you see I get always a 0 with socket->bytesAvailable(), I've used sockets before and never had this problem, so i'm a little confused about that :D



  • geageagea - in case of POST you have to read data from the socket until you will collect number of bytes provided in "Content-Length" header. You can't rely on temporary status - if there are any data or not. Otherwise you will stop reading in case of any glitch in the network or delay caused by "Nagle's algorithm":http://en.wikipedia.org/wiki/Nagle's_Algorithm

    POST section is after all headers - so first you have to read all headers: line by line, till 2 newlines one after another (AFAIR).

    Then you can start reading POST section. To do that you have to change your code from ->readAll() to something like that:
    @
    while(collectedBytes<expectedBytes)
    {
    // good to have any wait for data ie: waitForReadyRead(ms);
    tmpQByteArray = socket->read(expectedBytes-collectedBytes);
    collectedBytes+=tmpQByteArray.size();
    //and now add data from tmpQByteArray to your final buffer with POST
    // or
    // readed = socket->read( (char*) &tmpVector[collectedBytes], expectedBytes - collectedBytes );
    // collectedBytes += readed;
    .
    .
    .
    }@

    expectedBytes should be based on "Content-Length" and allready received bytes.



  • Xander84, thanks I'll try with a newer version, however I have to use this version, for now.
    Damos,
    I've tried that option, the thing is that the tmpQByteArray has always 0 length so my program gets stuck in an infinite loop trying to read the expectedBytes.

    I've tried this alternative but it always return 0 bytes read:
    @quint8 rawBuffer[2048];
    memset(rawBuffer, 0, 2048);
    qDebug() << "Bytes read" << recv(socket->socketDescriptor(), rawBuffer, 2048, 0);@
    I'm on Windows 7
    Thanks in advance



  • Ok, geageagea.
    Could you tell me pls, what client is sending data to your server? Is it regular browser like ie. Chrome or FF or your own client? Yes, I saw headers but just to be sure... :)

    I'm asking because in your header you have information:
    @Content-Length: 49@

    when example of missing content:
    @"user=admin&pass=admin&token=1234POST /login HTTP/1.1"@

    has only 32 bytes of data (before next POST request).
    So - looks like request isn't send properly?

    BTW - Could you share source of your inicio.html page?



  • Damos,
    Yes, I'm using Chrome 34.
    I did a modification to the header I've posted. I was trying no to make public information that I shouldn't. The content length is correct so it is not a problem.

    inicio.html:

    @<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
    <html >
    <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Form Test</title>
    <style type="text/css">
    body {
    background-image: url(imagenes/fondohome.jpg);
    background-repeat: no-repeat;
    background-position: center;
    background-position: 122px top;
    }
    .celdacampotexto {
    left: 50px;
    margin-left: 70px;
    }
    .fondocampotexto {
    background-image: url(imagenes/fondodegraderosa.jpg);
    background-repeat: repeat-x;
    height: auto;
    width: 228px;
    }
    .confirmar {
    background-image: url(imagenes/confirmar.jpg);
    font-family: Verdana, Geneva, sans-serif;
    height: 27px;
    background-repeat: no-repeat;
    }
    </style>
    [removed]

    document.onkeypress = MenuKeyPress;
    function MenuKeyPress(event)
    {
    var chCode = event.which;

    // Si se apreta la 'r' ó 'R'
    if(chCode == 13)
    {
    event.preventDefault();
    $('#Confirmar').trigger("click");
    }
    }

    function MM_swapImgRestore() { //v3.0
    var i,x,a=document.MM_sr; for(i=0;a&&i<a.length&&(x=a[i])&&x.oSrc;i++) x.src=x.oSrc;
    }
    function MM_preloadImages() { //v3.0
    var d=document; if(d.images){ if(!d.MM_p) d.MM_p=new Array();
    var i,j=d.MM_p.length,a=MM_preloadImages.arguments; for(i=0; i<a.length; i++)
    if (a[i].indexOf("#")!=0){ d.MM_p[j]=new Image; d.MM_p[j++].src=a[i];}}
    }

    function MM_findObj(n, d) { //v4.01
    var p,i,x; if(!d) d=document; if((p=n.indexOf("?"))>0&&parent.frames.length) {
    d=parent.frames[n.substring(p+1)].document; n=n.substring(0,p);}
    if(!(x=d[n])&&d.all) x=d.all[n]; for (i=0;!x&&i<d.forms.length;i++) x=d.forms[i][n];
    for(i=0;!x&&d.layers&&i<d.layers.length;i++) x=MM_findObj(n,d.layers[i].document);
    if(!x && d.getElementById) x=d.getElementById(n); return x;
    }

    function MM_swapImage() { //v3.0
    var i,j=0,x,a=MM_swapImage.arguments; document.MM_sr=new Array; for(i=0;i<(a.length-2);i+=3)
    if ((x=MM_findObj(a[i]))!=null){document.MM_sr[j++]=x; if(!x.oSrc) x.oSrc=x.src; x.src=a[i+2];}
    }
    [removed]
    </head>

    <body background="imagenes/fondohome.jpg" >
    <table width="1033" height="550" border="0">
    <tr>
    <td width="610" height="182"> </td>
    <td width="375"> </td>
    <td width="319"> </td>
    </tr>
    <tr>
    <td height="247" rowspan="4"> </td>
    <td height="60"><label></label> </td>
    <td rowspan="4"> </td>
    </tr>
    <tr>
    <td width="375" height="57"><span class="celdacampotexto">
    <input name="Password2" type="text" class="fondocampotexto" id="Password2" value="" autofocus/>
    </span></td>
    </tr>
    <tr>
    <td height="61"><form action="" method="post" name="form1" class="celdacampotexto" id="form1">
    <label for="Password"></label>
    <input name="Password" type="password" class="fondocampotexto" id="Password" onkeypress="MenuKeyPress()"/>
    </form></td>
    </tr>
    <tr>
    <td height="73" align="center">
    <a id='btnlogin' >
    <img src="imagenes/botonconfirmar1.png" alt="Confirmar" name="Confirmar" width="90" height="27" border="0" id="Confirmar" />
    </a>
    </td>
    </tr>
    <tr>
    <td> </td>
    <td> </td>
    <td> </td>
    </tr>
    </table>
    <div align="center"></div>

    <div id='cargando' >
    <img src="imagenes/loading.gif" style='margin:0px auto;'/>
    <p id='estadoprogreso' ></p>
    </div>

    </body>
    </html>
    [removed][removed]
    [removed][removed]@

    index.js:

    @
    $('#btnlogin').click(function (event)
    {
    event.preventDefault();

    if($('#Password2').val().length > 0 && $('#Password').val().length > 0)
    {
    login($('#Password2').val(), $('#Password').val());
    }
    })

    function login(username, password)
    {
    $('#cargando').show();
    $('#estadoprogreso').html("Logueando...");

    try{
    $.ajax('login',
    {
    type: 'POST',
    data:
    {
    user:username, pass:password, token:1234
    },
    success: function (data){
    $('#cargando').hide();
    $('#estadoprogreso').html("");
    alert(data);
    }
    });
    }
    catch(err)
    {
    alert(err.message);
    }
    }
    @

    Maybe it is too complicated for what it does, but basically there are 2 inputs and a button that executes the code in index.js



  • Hi everyone,

    I've finally solved this problem. I was mistaken to think that with the first readyRead signal I could read the entire post. So what I'm doing is to evaluate if the data is missing before processing the request. If I don't have the data, I simply wait for the next signal of readyRead to be emitted and complete the request.

    Thanks to all



  • That is exactly what I said in my first answer!?
    the packet might be divided due to TCP packet size or other factors, but good you solved it at least :)



  • Xander84,
    Yes, I was so stubborn because sometimes I received the whole package and sometimes I didn't. The thing that was making crazy is that the data was available after responding the request.
    Anyway, thanks a lot for your help



  • Late to the party, but I have written a web application framework that might suit your needs. It might be definately worth for you to take a look at it:
    https://github.com/cybercatalyst/qtwebserver


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.