QTcpServer with multiple persistent connections
-
@Dick-Balaska said in QTcpServer with multiple persistent connections:
[1] You handle message fragmentation and reconnects, right?
Of course I don't – or at least, I don't know – or I'm not sure ;-)
Would you please explain what you mean? If a "reconnect" is loss of a client connection followed by a new connect, I handle it (the client is removed from the server's client list as soon as it disconnects and is simply added like a new one when it reconnects).
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf.
ChatClient::onReadyRead()).I don't need encryption and such, the protocol will only be used in a closed LAN consisting of two or three computers, the data itself is not confidential, and I also don't need to protect against manipulation etc. (somebody trying this would shoot himself in the foot as the program probably wouldn't work as expected in this case).
@l3u_ I see class
ChatClientreferenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.WebSockets guarantees you get a whole message at a time. A regular socket (QTcpSocket) you just know you have x bytes available and it's up to you to frame them in a message. (Like http knows that \n\n is the end of the request headers).
Clearly you're well beyond that.
If you're unsure about message fragmentation, make a class with a >2k byte object and see if you get it in one message. (hardwired lan TCP is 1536 bytes per packet. Wifi is about 400 bytes per packet.) -
@l3u_ I see class
ChatClientreferenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.WebSockets guarantees you get a whole message at a time. A regular socket (QTcpSocket) you just know you have x bytes available and it's up to you to frame them in a message. (Like http knows that \n\n is the end of the request headers).
Clearly you're well beyond that.
If you're unsure about message fragmentation, make a class with a >2k byte object and see if you get it in one message. (hardwired lan TCP is 1536 bytes per packet. Wifi is about 400 bytes per packet.)@Dick-Balaska said in QTcpServer with multiple persistent connections:
I see class ChatClient referenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
@l3u_ said in QTcpServer with multiple persistent connections:
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf. ChatClient::onReadyRead()).
Yes, this is correct.
-
@Dick-Balaska said in QTcpServer with multiple persistent connections:
I see class ChatClient referenced in two examples (linux Qt 5.11.2), bluetooth, which I don't think you're doing, and WebChannel, which actually sits on top of WebSockets. So, I'm not sure what your base is.
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
@l3u_ said in QTcpServer with multiple persistent connections:
If "message fragmentation" is that "you can't expect all data to arrive in one piece", I think this is already handled by the chat example (cf. ChatClient::onReadyRead()).
Yes, this is correct.
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
-
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
@Dick-Balaska said in QTcpServer with multiple persistent connections:
But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has.
This is incorrect. It determines end-of-message based on the
QByteArrayit receives. This isn't strictly a text protocol, as the buffer's size is known beforehand (you haveQDataStreamwithQByteArrayon both sides). If the JSON parsing (after the message is read) fails, then the server just continues its merry way, it doesn't even report the error, which it probably should ... -
@kshegunov said in QTcpServer with multiple persistent connections:
Scroll up. The OP is using as a base the WIP example from the wiki I sourced in my first post.
Ok. Wrapping "read the socket" in a transaction is a neat trick. But, hmm, it determines end-of-message-received by whether it can parse valid JSON out of what it has. It seems to me that if you send a malformed message, the socket becomes dead. There is no way to clear the existing buffer. It will continue growing the buffer ad infinitum. I'd rather see it buffer until it receives a marker like \0\0, with a limit of 64KB on a message. Then if the JSON doesn't parse you can note the error and move on. Unless there is some application level feedback (ACK) involved, so the client can kill and restart the socket, neither side will know this socket is on its way to tipping over the app.
I know, you shouldn't be sending malformed JSON, but doing so shouldn't be fatal.
PS.
You raise a good point for the general case, however - how to validate messages that come through a binary stream. But I fear this is a topic that goes beyond the purpose of this rather simple example.Read and abide by the Qt Code of Conduct
-
PS.
You raise a good point for the general case, however - how to validate messages that come through a binary stream. But I fear this is a topic that goes beyond the purpose of this rather simple example.@kshegunov said in QTcpServer with multiple persistent connections:
But I fear this is a topic that goes beyond the purpose of this rather simple example.
Keep on with your conversation if you don't mind – I (and everybody else reading this) is definitely learning all the stuff I wished to find somewhere before starting my project!
-
@kshegunov said in QTcpServer with multiple persistent connections:
But I fear this is a topic that goes beyond the purpose of this rather simple example.
Keep on with your conversation if you don't mind – I (and everybody else reading this) is definitely learning all the stuff I wished to find somewhere before starting my project!
Well the conventional wisdom when working with a developed binary protocol (which the aforementioned example does not pretend to posses), is to add a message header (usually a fixed size one) and a checksum for it. So you know when reading the header and calculating the checksum if your header's valid. Thereafter you can read the payload somewhat safely, for which (if necessary) you can also add a hash to be sure data's valid. After that it can be parsed, if that's required.
Read and abide by the Qt Code of Conduct
-
Well the conventional wisdom when working with a developed binary protocol (which the aforementioned example does not pretend to posses), is to add a message header (usually a fixed size one) and a checksum for it. So you know when reading the header and calculating the checksum if your header's valid. Thereafter you can read the payload somewhat safely, for which (if necessary) you can also add a hash to be sure data's valid. After that it can be parsed, if that's required.
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we? Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
-
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we? Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
@l3u_ said in QTcpServer with multiple persistent connections:
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we?
Well, yes. But you're not protected from a malformed message (i.e. some client you didn't write sending you a ridiculous message).
Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
Yes, it adds the
QByteArray's size before sending the actual contents of the byte array. However you have no way of knowing if this size (or the data) isn't corrupted somehow. That's really rare, but it's a possibility.For example we don't check the byte array's size before trying to read the data. If someone sends you an integer indicating a message of 10GB, the code will just eat it. It doesn't check if that's too big.
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
That is correct.
Read and abide by the Qt Code of Conduct
-
@l3u_ said in QTcpServer with multiple persistent connections:
@kshegunov But after all, we can still be sure that a message is processed completely by using QDataStream transactions, can't we?
Well, yes. But you're not protected from a malformed message (i.e. some client you didn't write sending you a ridiculous message).
Connecting my server using telnet from a console, the JSON greeting is prepended by an additional character, so I think the datastream also adds some header and knows if it's complete?
Yes, it adds the
QByteArray's size before sending the actual contents of the byte array. However you have no way of knowing if this size (or the data) isn't corrupted somehow. That's really rare, but it's a possibility.For example we don't check the byte array's size before trying to read the data. If someone sends you an integer indicating a message of 10GB, the code will just eat it. It doesn't check if that's too big.
If I read the code correctly, the chat example makes sure that a whole message is received (and the socket buffer is emptied after reading it out) and tries to parse the JSON data inside it. If it's actually JSON, and it's valid, some action is triggered, if not, nothing is done – but the buffer is still emptied. Is this correct?
That is correct.
@kshegunov Okay, then I'm safe. It's a protocol to be used by two or three computers in a LAN, and everybody using it wants it to function – so nobody will try to fake a message or trick it with wrong headers.
I protect the integrity against whatever may go wrong with a checksum: Each change request contains a checksum the whole data has to have afterwards. If the checksum is correct after a client applied the change, everything is fine. If not, the connection is closed and the client is told to re-connect, as a syncing with the server is done when a new client connects. Ans this one also contains the checksum verification.
At least, I hope that I'm safe this way ;-)
-
@kshegunov Okay, then I'm safe. It's a protocol to be used by two or three computers in a LAN, and everybody using it wants it to function – so nobody will try to fake a message or trick it with wrong headers.
I protect the integrity against whatever may go wrong with a checksum: Each change request contains a checksum the whole data has to have afterwards. If the checksum is correct after a client applied the change, everything is fine. If not, the connection is closed and the client is told to re-connect, as a syncing with the server is done when a new client connects. Ans this one also contains the checksum verification.
At least, I hope that I'm safe this way ;-)
-
@kshegunov Nice! Man, you all really, really helped me with this. I never did TCP stuff until now, but I'm pretty sure I can implement it in a proper way for my usecase with your help :-)
