Unsolved Qthread creation error
-
Hello,
I am running into a weird behaviour, here is my specs:
- QT 5.4
- Rasperry pi 3
- application using QT 5.4.
The purpose of my application is to connect to a remote server, make some get requests and feed another thread into my application with what has been fetched.
When I running a stress test on my application to test its robustness and after exactly 150 connection to the server , I got this message:QThread::start: Thread creation error: Resource temporarily unavailable
Of course all the thread that I create for the purpose are cleanup after each connection with the classic QThread cleanup method:
QThread* thread = new QThread; Worker* worker = new Worker(); worker->moveToThread(thread); connect(thread, SIGNAL (started()), worker, SLOT (process())); connect(worker, SIGNAL (finished()), thread, SLOT (quit())); connect(worker, SIGNAL (finished()), worker, SLOT (deleteLater())); connect(thread, SIGNAL (finished()), thread, SLOT (deleteLater())); thread->start();
Of course I verify each period the thread usage of my running application, In a normal usage I am only using 27 Threads on my application and this number keeps balancing between 29 and 27 during the whole stress test period, here is the status of my application after the Qthread creation error shows:
Name: MyApplicationName State: S (sleeping) Tgid: 10593 Ngid: 0 Pid: 10593 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 256 Groups: 0 1001 NStgid: 10593 NSpid: 10593 NSpgid: 10578 NSsid: 7481 VmPeak: 2020820 kB VmSize: 2010988 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 217664 kB VmRSS: 210836 kB VmData: 1946304 kB VmStk: 136 kB VmExe: 1056 kB VmLib: 55664 kB VmPTE: 700 kB VmPMD: 0 kB VmSwap: 0 kB Threads: 28 SigQ: 0/3789 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001004 SigCgt: 0000000180014402 CapInh: 0000000000000000 CapPrm: 0000003fffffffff CapEff: 0000003fffffffff CapBnd: 0000003fffffffff Seccomp: 0 Cpus_allowed: f Cpus_allowed_list: 0-3 Mems_allowed: 1 Mems_allowed_list: 0 voluntary_ctxt_switches: 214357 nonvoluntary_ctxt_switches: 12025
Of course after receiving this message I could not create any thread, the only thing to do is to restart the entire process.
thanks for your help,
-
Hi,
Threads are not a limitless resource, you should also take into account that you are creating also as many event loop which requires also system resources. You should maybe consider QThreadPool.
-
Thanks for your reply, I am aware that thread or process are not a limitless resources, but as I cleanup a thread before creating a new one, I think this will not result in such a problem.
-
Well I found out where the limitation came from, indeed the default thread stack size is set by the system is:
stack size (kbytes, -s) 8192
I have wrote a simple C program to see what is the limit of the thread creation, and even if I create the thread and destroy it, and create the next one , the max is always 251, the system does not decrement the total allowed thread value. to increase the thread number , we need to decrease the default stack size set by the system.
Typically each of the created thread will get this amount of memory (8MB) assigned for it's stack. With a 32bit program and a maximum Virtual mem address space of 2GB, that is a maximum of only 2048MB / 8MB = 256 threads ! Minus program code, minus heap-space will probably lead to the observed max. of 251 threads.I don't understand Why even if I destroy the previous created thread it does not take this into consideration.
-
@rafaelSorel You can simply use QThreadPool as @SGaist suggested
-
@jsulm said in Qthread creation error:
QThreadPool
Okey I will give a try, using the QThreadpool, the thing is that I have to implment my self all the QEventQueu that I am actually using in my Qthread runing obj which is auto handled in QThread classes.
-
Why would you need to do that ?
-
@SGaist Need to do what ?
-
I was referring to that:
@rafaelSorel said in Qthread creation error:
the thing is that I have to implment my self all the QEventQueu that I am actually using in my Qthread runing obj which is auto handled in QThread classes
-
Well I have tried to use QThreadpool and it appears that this class is provided for quick task and in my case it is not the case at all, I have many slots and signals on my worker class, so it is painful to use QRunnable in order to suit all my case.
-
Can you describe what your class does ?
-
My class fetches stream data from a remote server (basically video data) and send this data o our local decoder. so the class itself look like this:
class inputStreamHttp : public QObject { Q_OBJECT public: enum { STREAMING_ACCEPTING, STREAMING_IN_PROGRESS, STREAMING_STOPPING, STREAMING_STOPPED }; typedef struct { int socket; int state; } streaming_server_t; inputStreamHttp(QString); ~inputStreamHttp(); void Start(); void Close(); void execGetRequest(); signals: void httpStreamInterrupted(); void finished(); void httpStreamReady(QString); public slots: void onNewLinkAvailable(); void onGetReqTimeout(); void onServerReady(); private slots: void data_ready(); void finish_get(); void onCurrentSessionError(QNetworkReply::NetworkError); private: void startget(); void cleanUp(); HttpServer * m_http_server; QTimer * m_getReqTimer; QNetworkReply * m_network_reply; QString m_main_url_str; QNetworkAccessManager * m_http_headers_obj; QMutex m_stop_pending_mutex; streaming_server_t m_server; bool m_httpPendingStop; };
I found a way to bypass this limitation, by creating two threads and queue them, then alternete between those two threads each time a new connection came:
#define MAX_THREAD_COUNT 2 for(unsigned int i=0; i<MAX_THREAD_COUNT; i++) { QThread * httpthread = new QThread(); m_thread_queue.enqueue(httpthread); }
After that, I could use the thread queue to alternate between the two threads:
m_httpMediaProc = new inputStreamHttp(*httpStream); m_httpthread = m_thread_queue.dequeue(); m_httpMediaProc->moveToThread(m_httpthread);
and of course enqueue the already used one when session is closed.
-
Out of curiosity, why do you need a QNetworkAccessManager per thread ?
-
@rafaelSorel said in Qthread creation error:
m_http_headers_obj
This is how it was designed at first place, any way the obj itself is deleted at each new connection.
Here is the Qthreadpool like that I have made but using real Qthreads:class OMX_QThreadPool: public QObject { Q_OBJECT public: OMX_QThreadPool(): QObject() , m_active_thread(NULL) , m_reserved_thread(0) {} ~OMX_QThreadPool() {m_thread_queue.clear(); } inline void setMaxThreadCount(int maxThreadCount) { for (int i = 0; i < maxThreadCount; i++) { QThread * httpthread = new QThread(); m_thread_queue.enqueue(httpthread); m_reserved_thread++; } } inline void reserveThread() { QMutexLocker locker(&m_mutex); if (m_thread_queue.isEmpty()) { QThread * fallbackthread = new QThread(); m_reserved_thread++; m_active_thread = fallbackthread; } else { m_active_thread = m_thread_queue.dequeue(); } } inline void start(QObject * obj) { QMutexLocker locker(&m_mutex); obj->moveToThread(m_active_thread); if(!m_active_thread->isRunning()) m_active_thread->start(); } inline void releaseThread() { QMutexLocker locker(&m_mutex); if (!m_thread_queue.contains(m_active_thread)) m_thread_queue.enqueue(m_active_thread); } inline int activeThreadCount() { QMutexLocker locker(&m_mutex); return m_reserved_thread; } private: QQueue<QThread *> m_thread_queue; QThread * m_active_thread; QMutex m_mutex; unsigned int m_reserved_thread; };