Tcp server for high volume traffic
-
Hi
I have TCP server basically created as from the example of Threaded Fortune Server, I use this scheme for many of my projects, works well and stable.
However someone wanted me to run the server for 10.000 users simultaneosly. So I quickly created server, and client application to test it out, client when it starts it creates 2500 threads, connects to the server and sends/receive JSON payload every 5 sec, there were 4 clients running on different Linux machines located on the Internet
With 5000 connections I have about 17 mln. files open on my Linux VPS (8 cores 16GB RAM) and I fill like this is the limit for stable work. When I try 10.000 simultaneous connections, server drops some of the connections, or I can't connect.
I can't use REST API approach for that because the goal is that server sends some data to all users on specific time intervals.I have been shown that some other company made server with 30.000 simultaneous users connected and sending data to all of them took server 3 seconds. My test was a bit different becuse each testing thread was sending JSON query every 5 sec. but I got the feeling that the problem was with the number of simultaneos connections.
I have made a repo with server example and screens from top command running on Linux
https://github.com/frankipl/load-server.gitDetails of my test:
10.000 user records in PostgresSQL database, server on startup reads them all, so for reading queries I serve data from server memory, only for update or write I update postgres data
Each thread has 88 queries of read data, 10 queries to update data, 1 query add data, 1 query sends small PDF file (315 kb), so there is 100 queries, each testing thread draws a query until it sends them all and repeats again.Question is: How to make it work better ? And especially for a server with 8 cores
Best Regards
Marek -
Hi
I have TCP server basically created as from the example of Threaded Fortune Server, I use this scheme for many of my projects, works well and stable.
However someone wanted me to run the server for 10.000 users simultaneosly. So I quickly created server, and client application to test it out, client when it starts it creates 2500 threads, connects to the server and sends/receive JSON payload every 5 sec, there were 4 clients running on different Linux machines located on the Internet
With 5000 connections I have about 17 mln. files open on my Linux VPS (8 cores 16GB RAM) and I fill like this is the limit for stable work. When I try 10.000 simultaneous connections, server drops some of the connections, or I can't connect.
I can't use REST API approach for that because the goal is that server sends some data to all users on specific time intervals.I have been shown that some other company made server with 30.000 simultaneous users connected and sending data to all of them took server 3 seconds. My test was a bit different becuse each testing thread was sending JSON query every 5 sec. but I got the feeling that the problem was with the number of simultaneos connections.
I have made a repo with server example and screens from top command running on Linux
https://github.com/frankipl/load-server.gitDetails of my test:
10.000 user records in PostgresSQL database, server on startup reads them all, so for reading queries I serve data from server memory, only for update or write I update postgres data
Each thread has 88 queries of read data, 10 queries to update data, 1 query add data, 1 query sends small PDF file (315 kb), so there is 100 queries, each testing thread draws a query until it sends them all and repeats again.Question is: How to make it work better ? And especially for a server with 8 cores
Best Regards
Marek@Marek
10.000 threads are honestly horrible ;)
especially if your are aiming for more perfomance. Thread scheduling,/handling is not free and also consumes resoruces!the keyword here is: load balancing
-
@Marek
10.000 threads are honestly horrible ;)
especially if your are aiming for more perfomance. Thread scheduling,/handling is not free and also consumes resoruces!the keyword here is: load balancing
@raven-worx
This was also my idea. But they showed me server running on single VPS with 30.000 users connected, so... maybe I shouldn't create thread for each connection ? Like with select on Linux, just to monitor descriptors ?
I mean surely someone has some experience with that, 10.000 users is not such a big number, probably 10.000 threads is ;)Best,
Marek -
Hi
I have TCP server basically created as from the example of Threaded Fortune Server, I use this scheme for many of my projects, works well and stable.
However someone wanted me to run the server for 10.000 users simultaneosly. So I quickly created server, and client application to test it out, client when it starts it creates 2500 threads, connects to the server and sends/receive JSON payload every 5 sec, there were 4 clients running on different Linux machines located on the Internet
With 5000 connections I have about 17 mln. files open on my Linux VPS (8 cores 16GB RAM) and I fill like this is the limit for stable work. When I try 10.000 simultaneous connections, server drops some of the connections, or I can't connect.
I can't use REST API approach for that because the goal is that server sends some data to all users on specific time intervals.I have been shown that some other company made server with 30.000 simultaneous users connected and sending data to all of them took server 3 seconds. My test was a bit different becuse each testing thread was sending JSON query every 5 sec. but I got the feeling that the problem was with the number of simultaneos connections.
I have made a repo with server example and screens from top command running on Linux
https://github.com/frankipl/load-server.gitDetails of my test:
10.000 user records in PostgresSQL database, server on startup reads them all, so for reading queries I serve data from server memory, only for update or write I update postgres data
Each thread has 88 queries of read data, 10 queries to update data, 1 query add data, 1 query sends small PDF file (315 kb), so there is 100 queries, each testing thread draws a query until it sends them all and repeats again.Question is: How to make it work better ? And especially for a server with 8 cores
Best Regards
Marek@Marek 10k threads are bad in most OS due to overhead. Usage of a thread pool should be the best way for it.
Maybe this tutorial is helpful for a sample implementation. -
@Marek 10k threads are bad in most OS due to overhead. Usage of a thread pool should be the best way for it.
Maybe this tutorial is helpful for a sample implementation.@DerReisende Thanks
Lots of tutorials on this website.
After checking a few of them, I'm not sure I doing my Threaded Server the right way
I don't use QThread::run
I just create new AppNetThread which inherits from QThread and run it keeping pointers to this object in QMap
Can someone wise take a look at the code? :)Best,
Marek -
Hi,
You might want to check the Cutelyst project which may have some of your needs covered.
-
Hi,
You might want to check the Cutelyst project which may have some of your needs covered.
@SGaist Thanks for that link
It does not solve tcp server issue, but I always had problem with web applications in my projects, usually I use qhttpengine to make REST API and then React for frontend. If this Cutelyst can solve issue of proper REST API or I can compile into the server and make simple web panel for administration then it is a huge improvement ;)Best,
Marek -
My understanding is that typical web servers handle multiple connections in a single thread. Just as the others said, 10000 threads is a lot for the operating system to handle. The threaded fortune server example is nice for few connections, but does not scale well. It is okay to have multiple threads, but rather like 8 threads for 8 cores. Each thread can then handle a multitude of connections.
I am not sure if Qt is then the best solution. You might want to have a look at other frameworks for networking, like asio (https://think-async.com/Asio). asio also is part of Boost if you are using this already.
-
My understanding is that typical web servers handle multiple connections in a single thread. Just as the others said, 10000 threads is a lot for the operating system to handle. The threaded fortune server example is nice for few connections, but does not scale well. It is okay to have multiple threads, but rather like 8 threads for 8 cores. Each thread can then handle a multitude of connections.
I am not sure if Qt is then the best solution. You might want to have a look at other frameworks for networking, like asio (https://think-async.com/Asio). asio also is part of Boost if you are using this already.
@SimonSchroeder
Hi,
This is also my idea now, create my own pool of threads related to number of processor cores and handle multiple connections inside one thread, this will utilize cores better. I just don't have time now to rewrite test server, but once I will I do that and report performance. Also my colleague suggested using Boost for that, which we actually mix now with Qt code in mobile apps.
Best Regards
Marek -
@SimonSchroeder
Hi,
This is also my idea now, create my own pool of threads related to number of processor cores and handle multiple connections inside one thread, this will utilize cores better. I just don't have time now to rewrite test server, but once I will I do that and report performance. Also my colleague suggested using Boost for that, which we actually mix now with Qt code in mobile apps.
Best Regards
Marek@Marek said in Tcp server for high volume traffic:
create my own pool of threads
-
@Marek said in Tcp server for high volume traffic:
create my own pool of threads
or Qt Concurrent that does the thread pool management for you.
-
or Qt Concurrent that does the thread pool management for you.
@J-Hilk
Hi
From what I read about it ThreadPool will not be good idea for me.
in QThreadPool you need to provide function to run or Object that will inherit from QRunnable with overwritten run method. If I would like to manage signal/slot for socket then QRunnable should also inherit from QObject, too much complications. It looks like QThreadPool is designed for heavy workload which start and ends after some time then start again with same/different work to do, while in my scenario threads will run constantly, just socket connections will come and go. But thanks for looking into it, I will report what I found out about performance
Best Regards
Marek
