QTcpSocket not receiving POST data from web page

Xander84

Hi, the data can be transferred with multiple (TCP) packets depending on the size, so usually you write the data to a buffer until you have received the whole data and only after that process it or you might end up with partial or corrupted data.
I am not an HTTP expert but in most cases the packet size will be prepended to the actual data, so if you send a string for example it will first send the length of the string and than the characters so you can simply check if the whole string has been received before you process it.
I believe the Qt socket example explain that and also use a buffer until all data is received, in case you don't know what i mean. If its not to much data you can let the socket itself buffer it until you've received enough data to process, so you might get a few calls to the "readyRead" slot before all data is available.

geageagea

Hi Xander84,
Thanks for the reply but unfortunately I've tried that. The result was that the process was blocked trying to read the 32 bytes, but they were never available (or I'm assuming they were not available). With the chrome debugger (CTRL + SHIFT + J) I can see that the full post request with the data is being sent.
I think that as the data (final part of the message) is missing a '\r' or '\n' character, the QTcpSocket library can't read it

andreyc

You are closing socket at the end of readClient().
Do you reopen it again on each incoming packet?
It is not right because you are loosing incoming buffer when you close socket.
Next packet will come as new connection and all data that were in the buffer after first readLine() or readAll() is gone.

Xander84

andreyc is right, why are you closing the socket anyway? most web servers leave the socket open for some minutes maybe. It's way faster if you leave it open unless you know you have only one request every 5 minutes or something.
Also the header "Connection: keep-alive" is good for something, the client doesn't want you to close the socket :D

geageagea

Hi andreyc, thanks for the response
The problem is that the data is available in the first readClient() call, but after sending some data in response (lines 15, 16, and 17). Nevertheless, I've tried not closing the socket with the same result

Xander84

I've tried this myself now with a very small example code and it works as it should be!? here my code:
@
QTcpServer *server = new QTcpServer;
connect(server, &QTcpServer::newConnection, [=]{
QTcpSocket *client = server->nextPendingConnection();
qDebug() << client->peerAddress() << client->peerPort();
connect(client, &QTcpSocket::readyRead, [=]{
qDebug() << client->bytesAvailable() << client->readAll();
});
});
qDebug() << server->listen(QHostAddress::LocalHost, 80);
@
I just print everything to the debug console.
example output send from chrome/postman extension as post data
@
true
QHostAddress( "127.0.0.1" ) 7549
0 "POST / HTTP/1.1

Host: localhost

Connection: keep-alive

Content-Length: 19

Cache-Control: no-cache

Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop

Content-Type: application/x-www-form-urlencoded

User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.116 Safari/537.36

Postman-Token: 0d5e263b-dd15-a29d-6621-06d89c52fafa

Accept: /

Accept-Encoding: gzip,deflate,sdch

Accept-Language: de,en;q=0.8

user=admin&pass=123"
QHostAddress( "127.0.0.1" ) 7550
@
see the post data at the end? so everything is there I just used this two parameters for the test. Also notice I get two TCP connection, sometimes even three. Don't ask me why but the only one is sending the data. I am a little bit confused because the "client->bytesAvailable()" return 0 but as you can see the whole package is there right after that with "client->readAll()"!?

geageagea

Xander84,
It is strange. Now I'm doing a peek of all data at the beginning of the slot and I'm not getting the data before the server response but after. What version of Qt are you using? Mine is 4.7.4
Do not worry about the multiple connections, I have the same problem. It must be a HTML issue

Xander84

I'm using Qt 5.2.1, I don't know if that is a bug in your Qt version :/
Maybe you can just use a newer version to verify if your code is working there?
also did you see I get always a 0 with socket->bytesAvailable(), I've used sockets before and never had this problem, so i'm a little confused about that :D

Damos

geageagea - in case of POST you have to read data from the socket until you will collect number of bytes provided in "Content-Length" header. You can't rely on temporary status - if there are any data or not. Otherwise you will stop reading in case of any glitch in the network or delay caused by "Nagle's algorithm":http://en.wikipedia.org/wiki/Nagle's_Algorithm

POST section is after all headers - so first you have to read all headers: line by line, till 2 newlines one after another (AFAIR).

Then you can start reading POST section. To do that you have to change your code from ->readAll() to something like that:
@
while(collectedBytes<expectedBytes)
{
// good to have any wait for data ie: waitForReadyRead(ms);
tmpQByteArray = socket->read(expectedBytes-collectedBytes);
collectedBytes+=tmpQByteArray.size();
//and now add data from tmpQByteArray to your final buffer with POST
// or
// readed = socket->read( (char*) &tmpVector[collectedBytes], expectedBytes - collectedBytes );
// collectedBytes += readed;
.
.
.
}@

expectedBytes should be based on "Content-Length" and allready received bytes.

geageagea

Xander84, thanks I'll try with a newer version, however I have to use this version, for now.
Damos,
I've tried that option, the thing is that the tmpQByteArray has always 0 length so my program gets stuck in an infinite loop trying to read the expectedBytes.

I've tried this alternative but it always return 0 bytes read:
@quint8 rawBuffer[2048];
memset(rawBuffer, 0, 2048);
qDebug() << "Bytes read" << recv(socket->socketDescriptor(), rawBuffer, 2048, 0);@
I'm on Windows 7
Thanks in advance

Damos

Ok, geageagea.
Could you tell me pls, what client is sending data to your server? Is it regular browser like ie. Chrome or FF or your own client? Yes, I saw headers but just to be sure... :)

I'm asking because in your header you have information:
@Content-Length: 49@

when example of missing content:
@"user=admin&pass=admin&token=1234POST /login HTTP/1.1"@

has only 32 bytes of data (before next POST request).
So - looks like request isn't send properly?

BTW - Could you share source of your inicio.html page?

geageagea

Damos,
Yes, I'm using Chrome 34.
I did a modification to the header I've posted. I was trying no to make public information that I shouldn't. The content length is correct so it is not a problem.

inicio.html:

@<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html >
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Form Test</title>
<style type="text/css">
body {
background-image: url(imagenes/fondohome.jpg);
background-repeat: no-repeat;
background-position: center;
background-position: 122px top;
}
.celdacampotexto {
left: 50px;
margin-left: 70px;
}
.fondocampotexto {
background-image: url(imagenes/fondodegraderosa.jpg);
background-repeat: repeat-x;
height: auto;
width: 228px;
}
.confirmar {
background-image: url(imagenes/confirmar.jpg);
font-family: Verdana, Geneva, sans-serif;
height: 27px;
background-repeat: no-repeat;
}
</style>
[removed]

document.onkeypress = MenuKeyPress;
function MenuKeyPress(event)
{
var chCode = event.which;

// Si se apreta la 'r' ó 'R'
if(chCode == 13)
{
event.preventDefault();
$('#Confirmar').trigger("click");
}
}

function MM_swapImgRestore() { //v3.0
var i,x,a=document.MM_sr; for(i=0;a&&i<a.length&&(x=a[i])&&x.oSrc;i++) x.src=x.oSrc;
}
function MM_preloadImages() { //v3.0
var d=document; if(d.images){ if(!d.MM_p) d.MM_p=new Array();
var i,j=d.MM_p.length,a=MM_preloadImages.arguments; for(i=0; i<a.length; i++)
if (a[i].indexOf("#")!=0){ d.MM_p[j]=new Image; d.MM_p[j++].src=a[i];}}
}

function MM_findObj(n, d) { //v4.01
var p,i,x; if(!d) d=document; if((p=n.indexOf("?"))>0&&parent.frames.length) {
d=parent.frames[n.substring(p+1)].document; n=n.substring(0,p);}
if(!(x=d[n])&&d.all) x=d.all[n]; for (i=0;!x&&i<d.forms.length;i++) x=d.forms[i][n];
for(i=0;!x&&d.layers&&i<d.layers.length;i++) x=MM_findObj(n,d.layers[i].document);
if(!x && d.getElementById) x=d.getElementById(n); return x;
}

function MM_swapImage() { //v3.0
var i,j=0,x,a=MM_swapImage.arguments; document.MM_sr=new Array; for(i=0;i<(a.length-2);i+=3)
if ((x=MM_findObj(a[i]))!=null){document.MM_sr[j++]=x; if(!x.oSrc) x.oSrc=x.src; x.src=a[i+2];}
}
[removed]
</head>

</body>
</html>
[removed][removed]
[removed][removed]@

index.js:

@
$('#btnlogin').click(function (event)
{
event.preventDefault();

if($('#Password2').val().length > 0 && $('#Password').val().length > 0)
{
login($('#Password2').val(), $('#Password').val());
}
})

function login(username, password)
{
$('#cargando').show();
$('#estadoprogreso').html("Logueando...");

try{
$.ajax('login',
{
type: 'POST',
data:
{
user:username, pass:password, token:1234
},
success: function (data){
$('#cargando').hide();
$('#estadoprogreso').html("");
alert(data);
}
});
}
catch(err)
{
alert(err.message);
}
}
@

Maybe it is too complicated for what it does, but basically there are 2 inputs and a button that executes the code in index.js

geageagea

Hi everyone,

I've finally solved this problem. I was mistaken to think that with the first readyRead signal I could read the entire post. So what I'm doing is to evaluate if the data is missing before processing the request. If I don't have the data, I simply wait for the next signal of readyRead to be emitted and complete the request.

Thanks to all

Xander84

That is exactly what I said in my first answer!?
the packet might be divided due to TCP packet size or other factors, but good you solved it at least :)

geageagea

Xander84,
Yes, I was so stubborn because sometimes I received the whole package and sometimes I didn't. The thing that was making crazy is that the data was available after responding the request.
Anyway, thanks a lot for your help

cybercatalyst

Late to the party, but I have written a web application framework that might suit your needs. It might be definately worth for you to take a look at it:
https://github.com/cybercatalyst/qtwebserver