From 10:00 CET Friday 22nd November we will adjust how the server works to deal with some recently reported problems. Therefore there may be a load problem, if you experience more problems than usual trying to access the forum then please PM AndyS or any of the moderators so they can inform me.


Memory leak in QNetworkProxy (should i post on bugtracker?)



  • I working on a web-spider application that uses lot of proxies. The problem is the memory leaks are enormouse, up to 100 mb in 5 minutes. I wrote a simple test like this, which runs inside QThread:

    worker::process()
    {
    //here i read the proxies from file first
      QString readAll;
      QFile file("proxies.txt");
      if(!file.open(QIODevice::ReadWrite))
      {
         qDebug() << file.errorString();
         emit finished();
         eturn;
      }
      QTextStream in(&file);
    
          while(!in.atEnd())
          {
    
              readAll = in.readAll();
    
          }
          file.close();
    
       
    //split the proxies by new line to the QStringList
      QStringList lst = readAll.split(QRegExp("(\\r\\n)|(\\n\\r)|\\r|\\n"), QString::SkipEmptyParts);
    
      foreach(QString proxy,lst){
    
            QNetworkRequest request;
            request.setUrl(QUrl("https://google.com/"));
    
            request.setRawHeader("User-Agent","Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/2022021 Firefox/46.0");
    
            QStringList parts = proxy.split(":");
            QString s_port = parts[1];
          // here is the thing
            QNetworkProxy webProxy;
                webProxy.setHostName(parts[0]);
                webProxy.setPort(s_port.toUShort());
                webProxy.setType(QNetworkProxy::Socks5Proxy);
    
               m.setProxy(webProxy);
    
    
               QNetworkReply * reply = m.get(request);
               QTimer timer;
               timer.setSingleShot(true);
    
               QEventLoop loop;
               QObject::connect(&timer, SIGNAL(timeout()), &loop, SLOT(quit()));
               QObject::connect(reply, SIGNAL(finished()), &loop, SLOT(quit()));
               timer.start(1*1000);
               loop.exec();
    
               if(reply->isFinished())
                   reply->abort();
    
               reply->close();
               delete reply;
         }
    }
    

    So in this foreach im doing the requests, and see memory goes SUPER crazy, and it never decreases even if i stop the thread. But if i comment out this:

    
     QNetworkProxy webProxy;
           /*     webProxy.setHostName(parts[0]);
                webProxy.setPort(s_port.toUShort());
                webProxy.setType(QNetworkProxy::Socks5Proxy);*/
    
               m.setProxy(webProxy);
    

    So basically no proxy request, i dont see any leakages. All is stable. Is it a possible bug? I don't know if i write post here or report to bugtracker? Also im using a small timeout with QEventLoop here, its just for the sake of example, but in reality, i need to abort the request if the proxy is invalid, otherwise it could hang for hours (maybe even days)


  • Qt Champions 2017

    @Vlad_Savelyev
    Hello,
    Could you prepare a minimal working example that reproduces this behavior. I could test on my machine, and you'd probably be asked for it anyway if you report as a bug.

    Kind regards.



  • sure i will reply with the project code