Qthread creation error



  • Hello,

    I am running into a weird behaviour, here is my specs:

    • QT 5.4
    • Rasperry pi 3
    • application using QT 5.4.

    The purpose of my application is to connect to a remote server, make some get requests and feed another thread into my application with what has been fetched.
    When I running a stress test on my application to test its robustness and after exactly 150 connection to the server , I got this message:

    QThread::start: Thread creation error: Resource temporarily unavailable
    

    Of course all the thread that I create for the purpose are cleanup after each connection with the classic QThread cleanup method:

    QThread* thread = new QThread;
    Worker* worker = new Worker();
    worker->moveToThread(thread);
    connect(thread, SIGNAL (started()), worker, SLOT (process()));
    connect(worker, SIGNAL (finished()), thread, SLOT (quit()));
    connect(worker, SIGNAL (finished()), worker, SLOT (deleteLater()));
    connect(thread, SIGNAL (finished()), thread, SLOT (deleteLater()));
    thread->start();
    

    Of course I verify each period the thread usage of my running application, In a normal usage I am only using 27 Threads on my application and this number keeps balancing between 29 and 27 during the whole stress test period, here is the status of my application after the Qthread creation error shows:

    Name:	MyApplicationName
    State:	S (sleeping)
    Tgid:	10593
    Ngid:	0
    Pid:	10593
    PPid:	1
    TracerPid:	0
    Uid:	0	0	0	0
    Gid:	0	0	0	0
    FDSize:	256
    Groups:	0 1001 
    NStgid:	10593
    NSpid:	10593
    NSpgid:	10578
    NSsid:	7481
    VmPeak:	 2020820 kB
    VmSize:	 2010988 kB
    VmLck:	       0 kB
    VmPin:	       0 kB
    VmHWM:	  217664 kB
    VmRSS:	  210836 kB
    VmData:	 1946304 kB
    VmStk:	     136 kB
    VmExe:	    1056 kB
    VmLib:	   55664 kB
    VmPTE:	     700 kB
    VmPMD:	       0 kB
    VmSwap:	       0 kB
    Threads:	28
    SigQ:	0/3789
    SigPnd:	0000000000000000
    ShdPnd:	0000000000000000
    SigBlk:	0000000000000000
    SigIgn:	0000000000001004
    SigCgt:	0000000180014402
    CapInh:	0000000000000000
    CapPrm:	0000003fffffffff
    CapEff:	0000003fffffffff
    CapBnd:	0000003fffffffff
    Seccomp:	0
    Cpus_allowed:	f
    Cpus_allowed_list:	0-3
    Mems_allowed:	1
    Mems_allowed_list:	0
    voluntary_ctxt_switches:	214357
    nonvoluntary_ctxt_switches:	12025
    

    Of course after receiving this message I could not create any thread, the only thing to do is to restart the entire process.

    thanks for your help,


  • Lifetime Qt Champion

    Hi,

    Threads are not a limitless resource, you should also take into account that you are creating also as many event loop which requires also system resources. You should maybe consider QThreadPool.



  • Thanks for your reply, I am aware that thread or process are not a limitless resources, but as I cleanup a thread before creating a new one, I think this will not result in such a problem.



  • Well I found out where the limitation came from, indeed the default thread stack size is set by the system is:

    stack size              (kbytes, -s) 8192
    

    I have wrote a simple C program to see what is the limit of the thread creation, and even if I create the thread and destroy it, and create the next one , the max is always 251, the system does not decrement the total allowed thread value. to increase the thread number , we need to decrease the default stack size set by the system.
    Typically each of the created thread will get this amount of memory (8MB) assigned for it's stack. With a 32bit program and a maximum Virtual mem address space of 2GB, that is a maximum of only 2048MB / 8MB = 256 threads ! Minus program code, minus heap-space will probably lead to the observed max. of 251 threads.

    I don't understand Why even if I destroy the previous created thread it does not take this into consideration.


  • Moderators

    @rafaelSorel You can simply use QThreadPool as @SGaist suggested



  • @jsulm said in Qthread creation error:

    QThreadPool

    Okey I will give a try, using the QThreadpool, the thing is that I have to implment my self all the QEventQueu that I am actually using in my Qthread runing obj which is auto handled in QThread classes.


  • Lifetime Qt Champion

    Why would you need to do that ?



  • @SGaist Need to do what ?


  • Lifetime Qt Champion

    I was referring to that:

    @rafaelSorel said in Qthread creation error:

    the thing is that I have to implment my self all the QEventQueu that I am actually using in my Qthread runing obj which is auto handled in QThread classes



  • Well I have tried to use QThreadpool and it appears that this class is provided for quick task and in my case it is not the case at all, I have many slots and signals on my worker class, so it is painful to use QRunnable in order to suit all my case.


  • Lifetime Qt Champion

    Can you describe what your class does ?



  • My class fetches stream data from a remote server (basically video data) and send this data o our local decoder. so the class itself look like this:

    class inputStreamHttp : public QObject
    {
        Q_OBJECT
    public:
        enum
        {
            STREAMING_ACCEPTING,
            STREAMING_IN_PROGRESS,
            STREAMING_STOPPING,
            STREAMING_STOPPED
        };
    
        typedef struct
        {
            int socket;
            int state;
        } streaming_server_t;
    
        inputStreamHttp(QString);
        ~inputStreamHttp();
        void Start();
        void Close();
        void execGetRequest();
    
    signals:
        void httpStreamInterrupted();
        void finished();
        void httpStreamReady(QString);
    
    public slots:
        void onNewLinkAvailable();
        void onGetReqTimeout();
        void onServerReady();
    
    private slots:
        void data_ready();
        void finish_get();
        void onCurrentSessionError(QNetworkReply::NetworkError);
    
    private:
        void startget();
        void cleanUp();
    
        HttpServer * m_http_server;
        QTimer * m_getReqTimer;
        QNetworkReply * m_network_reply;
        QString m_main_url_str;
        QNetworkAccessManager * m_http_headers_obj;
        QMutex m_stop_pending_mutex;
        streaming_server_t m_server;
        bool m_httpPendingStop;
    };
    

    I found a way to bypass this limitation, by creating two threads and queue them, then alternete between those two threads each time a new connection came:

    #define MAX_THREAD_COUNT 2
    
        for(unsigned int i=0; i<MAX_THREAD_COUNT; i++)
        {
            QThread * httpthread = new QThread();
            m_thread_queue.enqueue(httpthread);
        }
    

    After that, I could use the thread queue to alternate between the two threads:

        m_httpMediaProc = new inputStreamHttp(*httpStream);
        m_httpthread = m_thread_queue.dequeue();
        m_httpMediaProc->moveToThread(m_httpthread);
    

    and of course enqueue the already used one when session is closed.


  • Lifetime Qt Champion

    Out of curiosity, why do you need a QNetworkAccessManager per thread ?



  • @rafaelSorel said in Qthread creation error:

    m_http_headers_obj

    This is how it was designed at first place, any way the obj itself is deleted at each new connection.
    Here is the Qthreadpool like that I have made but using real Qthreads:

    class OMX_QThreadPool: public QObject
    {
        Q_OBJECT
    public:
        OMX_QThreadPool():
            QObject()
          , m_active_thread(NULL)
          , m_reserved_thread(0)
        {}
        ~OMX_QThreadPool()
        {m_thread_queue.clear();
        }
    
        inline void setMaxThreadCount(int maxThreadCount)
        {
            for (int i = 0; i < maxThreadCount; i++) {
                QThread * httpthread = new QThread();
                m_thread_queue.enqueue(httpthread);
                m_reserved_thread++;
            }
        }
    
        inline void reserveThread()
        {
            QMutexLocker locker(&m_mutex);
            if (m_thread_queue.isEmpty())
            {
                QThread * fallbackthread = new QThread();
                m_reserved_thread++;
                m_active_thread = fallbackthread;
            }
            else
            {
                m_active_thread = m_thread_queue.dequeue();
            }
        }
    
        inline void start(QObject * obj)
        {
            QMutexLocker locker(&m_mutex);
            obj->moveToThread(m_active_thread);
            if(!m_active_thread->isRunning())
                m_active_thread->start();
        }
    
        inline void releaseThread()
        {
            QMutexLocker locker(&m_mutex);
            if (!m_thread_queue.contains(m_active_thread))
                m_thread_queue.enqueue(m_active_thread);
        }
    
        inline int activeThreadCount()
        {
            QMutexLocker locker(&m_mutex);
            return m_reserved_thread;
        }
    private:
        QQueue<QThread *> m_thread_queue;
        QThread * m_active_thread;
        QMutex m_mutex;
        unsigned int m_reserved_thread;
    };
    

Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.