Qt HTTP request manager

shylock_the_camel

Hi,

I have created a Qt class to ease the management of HTTP requests and their responses. Using this class can produce the following errors at runtime:

- QWinEventNotifier: Cannot have more than 62 enabled at one time
- Qt has caught an exception thrown from an event handler. Throwing exceptions from an event handler is not supported in Qt. You must reimplement QApplication::notify() and catch all exceptions there.

Here's the class:

#ifndef HTTP_REQUEST_MANAGER_HPP
#define HTTP_REQUEST_MANAGER_HPP

#include <qobject.h>
#include <qnetworkreply.h>
#include <qdir.h>
#include <qhash.h>
#include <qmutex.h>
#include <qprocess.h>

class QFile;
class QAuthenticator;
class QTextStream;

class HttpRequestWorker;

class HttpRequestManager : public QObject
{
    Q_OBJECT

public:
    typedef QHash<QByteArray, QByteArray> HeadersHash;

    explicit HttpRequestManager(QObject *parent = nullptr);
    ~HttpRequestManager();


    QUrl url() const;
    void setUrl(const QUrl& val);

    QUrl proxyServerUrl() const;
    void setProxyServerUrl(const QUrl& val);

    bool hasError() const;
    void setHasError(bool has_error);

    HeadersHash additionalHeaders() const;
    void setAdditionalHeaders(const HeadersHash& val);

    QByteArray bodyData() const;
    void setBodyData(const QByteArray& val);

    bool isFinished() const;

    void cancelRequest() const;

public Q_SLOTS:
    void asyncGetData();
    void asyncPostData();
    void asyncPutData();
    void asyncDeleteData();
    QByteArray getData();
    QByteArray postData();
    QByteArray putData();
    QByteArray deleteData();

private Q_SLOTS:
    void onWorkerSuccess(const QByteArray& data);
    void onWorkerError(QNetworkReply::NetworkError e, const QString& text);

private:
    void asyncVerbData(QNetworkAccessManager::Operation op);
    QByteArray verbData(QNetworkAccessManager::Operation op);

    bool hasError_;

    QUrl url_;
    HeadersHash additionalHeaders_;

    HttpRequestWorker* worker_;

Q_SIGNALS:
    void asyncRequestSuccess(const QByteArray& responseData);
    void asyncRequestError(QNetworkReply::NetworkError, const QString&);
};

class HttpRequestWorker : public QObject
{
    Q_OBJECT

public:
    enum Task { FetchData };

    HttpRequestWorker(Task t, const QUrl& url, QNetworkAccessManager::Operation op, bool async, 
        HttpRequestManager::HeadersHash headers = HttpRequestManager::HeadersHash());
    ~HttpRequestWorker();

    QByteArray lastResponseData();
    QNetworkReply::NetworkError lastErrorCode();
    QString lastErrorString();

public Q_SLOTS:
    void start();
    void abort();

    bool stopped();

private:
    void stop();
    void clean();
    void verbRequest(QNetworkAccessManager::Operation op);

private Q_SLOTS:
    void httpFinished();
    void httpReadyRead();
    void slotAuthenticationRequired(QNetworkReply*, QAuthenticator*) const;

private:
    QUrl url_;
    QNetworkAccessManager* qnam_;
    QNetworkReply* reply_;
    HttpRequestManager::HeadersHash additionalHeaders_;

    QString data_;
    QTextStream textStream_;

    bool httpRequestAborted_;

    Task task_;
    QNetworkAccessManager::Operation operation_;
    bool asynchronous_;

    QMutex mutex_;

    bool stopped_;
    QByteArray lastResponseData_;
    QNetworkReply::NetworkError lastNetworkError_;
    QString lastErrorString_;

Q_SIGNALS:
    void finished();
    void success(const QByteArray&);
    void error(QNetworkReply::NetworkError, const QString&);
};

#endif // HTTP_REQUEST_MANAGER_HPP



#include "http_request_manager.hpp"

#include <qnetworkrequest.h>
#include <qauthenticator.h>
#include <qfileinfo.h>
#include <qdatetime.h>
#include <qnetworkconfiguration.h>
#include <qtextstream.h>
#include <qthread.h>

HttpRequestManager::HttpRequestManager(QObject *parent)
    : QObject(parent), hasError_(false), worker_(nullptr)
{
}

HttpRequestManager::~HttpRequestManager()
{
}

void HttpRequestManager::asyncGetData()
{
    asyncVerbData(QNetworkAccessManager::GetOperation);
}

void HttpRequestManager::asyncPostData()
{
    asyncVerbData(QNetworkAccessManager::PostOperation);
}

void HttpRequestManager::asyncPutData()
{
    asyncVerbData(QNetworkAccessManager::PutOperation);
}

void HttpRequestManager::asyncDeleteData()
{
    asyncVerbData(QNetworkAccessManager::DeleteOperation);
}

QByteArray HttpRequestManager::getData()
{
    return verbData(QNetworkAccessManager::GetOperation);
}

QByteArray HttpRequestManager::postData()
{
    return verbData(QNetworkAccessManager::PostOperation);
}

QByteArray HttpRequestManager::putData()
{
    return verbData(QNetworkAccessManager::PutOperation);
}

QByteArray HttpRequestManager::deleteData()
{
    return verbData(QNetworkAccessManager::DeleteOperation);
}

void HttpRequestManager::asyncVerbData(QNetworkAccessManager::Operation op)
{
    hasError_ = false;

    QThread* workerThread = new QThread;

    worker_ = new HttpRequestWorker(HttpRequestWorker::FetchData, url_, op, true, additionalHeaders_, bodyData_);
    worker_->moveToThread(workerThread);

    QObject::connect(workerThread, &QThread::started, worker_, &HttpRequestWorker::start);
    QObject::connect(workerThread, &QThread::finished, workerThread, &QThread::deleteLater);

    QObject::connect(worker_, &HttpRequestWorker::finished, workerThread, &QThread::quit);
    QObject::connect(worker_, &HttpRequestWorker::finished, worker_, &HttpRequestWorker::deleteLater);

    QObject::connect(worker_, &HttpRequestWorker::success, this, &HttpRequestManager::onWorkerSuccess);
    QObject::connect(worker_, &HttpRequestWorker::error, this, &HttpRequestManager::onWorkerError);

    workerThread->start();
}

QByteArray HttpRequestManager::verbData(QNetworkAccessManager::Operation op)
{
    hasError_ = false;

    QThread* workerThread = new QThread;

    worker_ = new HttpRequestWorker(HttpRequestWorker::FetchData, url_, op, false, additionalHeaders_, bodyData_);
    worker_->moveToThread(workerThread);

    QObject::connect(worker_, &HttpRequestWorker::finished, workerThread, &QThread::quit);

    QObject::connect(workerThread, &QThread::started, worker_, &HttpRequestWorker::start);
    QObject::connect(workerThread, &QThread::finished, workerThread, &QThread::deleteLater);

    workerThread->start();

    while (!worker_->stopped())
    {
    }

    worker->deleteLater();

    if (worker_->lastErrorCode() != QNetworkReply::NoError)
    {
        hasError_ = true;

        return QByteArray();
    }

    return worker_->lastResponseData();
}

bool HttpRequestManager::hasError() const
{
    return hasError_;
}

void HttpRequestManager::setHasError(bool has_error)
{
    hasError_ = has_error;
}

void HttpRequestManager::cancelRequest() const
{
    if (!QMetaObject::invokeMethod(worker_, "abort", Qt::QueuedConnection))
    {
        throw std::exception("Failed to invoke abort member of worker.");
    }
}

HttpRequestManager::HeadersHash HttpRequestManager::additionalHeaders() const
{
    return additionalHeaders_;
}

void HttpRequestManager::setAdditionalHeaders(const HeadersHash& val)
{
    additionalHeaders_ = val;
}

QByteArray HttpRequestManager::bodyData() const
{
    return bodyData_;
}

void HttpRequestManager::setBodyData(const QByteArray& val)
{
    bodyData_ = val;
}

bool HttpRequestManager::isFinished() const
{
    return static_cast<bool>(!worker_);
}

void HttpRequestManager::onWorkerSuccess(const QByteArray& data)
{
    worker_ = nullptr;

    emit asyncRequestSuccess(data);
}

void HttpRequestManager::onWorkerError(QNetworkReply::NetworkError e, const QString& text)
{
    worker_ = nullptr;

    emit asyncRequestError(e, text);
}

QUrl HttpRequestManager::url() const
{
    return url_;
}

void HttpRequestManager::setUrl( const QUrl& val )
{
    url_ = val;
}




HttpRequestWorker::HttpRequestWorker(Task t, const QUrl& url, QNetworkAccessManager::Operation op, bool async, HttpRequestManager::HeadersHash headers)
    : QObject(), url_(url), qnam_(nullptr), reply_(nullptr), additionalHeaders_(headers), httpRequestAborted_(false), task_(t), operation_(op), asynchronous_(async), 
    stopped_(false), lastNetworkError_(QNetworkReply::NoError)
{
    qnam_ = new QNetworkAccessManager(this);
    QObject::connect(qnam_, &QNetworkAccessManager::authenticationRequired, this, &HttpRequestWorker::slotAuthenticationRequired, Qt::UniqueConnection);
}

HttpRequestWorker::~HttpRequestWorker()
{
    clean();
}

QByteArray HttpRequestWorker::lastResponseData()
{
    QMutexLocker locker(&mutex_);

    return lastResponseData_;
}

QNetworkReply::NetworkError HttpRequestWorker::lastErrorCode()
{
    QMutexLocker locker(&mutex_);

    return lastNetworkError_;
}

QString HttpRequestWorker::lastErrorString()
{
    QMutexLocker locker(&mutex_);

    return lastErrorString_;
}

void HttpRequestWorker::start()
{
    clean();

    switch (task_)
    {
    case FetchData:
        {
            textStream_.setString(&data_);

            // schedule the request
            httpRequestAborted_ = false;
            verbRequest(operation_);
        }
        break;
    default:
        qFatal("Invalid task");
    }
}

void HttpRequestWorker::abort()
{
    httpRequestAborted_ = true;
    reply_->abort();
}

void HttpRequestWorker::verbRequest(QNetworkAccessManager::Operation op)
{
    lastNetworkError_ = QNetworkReply::NoError;
    lastErrorString_ = QString();

    QNetworkRequest request(url_);
    request.setHeader(QNetworkRequest::UserAgentHeader, "QNetworkAccessManager 5.6");

    for (auto i = additionalHeaders_.cbegin(); i != additionalHeaders_.cend(); ++i)
    {
        request.setRawHeader(i.key(), i.value());
    }

    switch (op)
    {
    case QNetworkAccessManager::GetOperation:
        {
            reply_ = qnam_->get(request);
        }
        break;
    case QNetworkAccessManager::PostOperation:
        {
            request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded");
            reply_ = qnam_->post(request, bodyData_);
        }
        break;
    case QNetworkAccessManager::PutOperation:
        {
            reply_ = qnam_->put(request, bodyData_);
        }
        break;
    case QNetworkAccessManager::DeleteOperation:
        {
            reply_ = qnam_->deleteResource(request);
        }
        break;
    default:
        qFatal("Invalid operation!");
    }

    QObject::connect(reply_, &QNetworkReply::readyRead, this, &HttpRequestWorker::httpReadyRead, Qt::UniqueConnection);
    QObject::connect(reply_, &QNetworkReply::finished, this, &HttpRequestWorker::httpFinished, Qt::UniqueConnection);
}

void HttpRequestWorker::clean()
{
    if (reply_)
    {
        reply_->deleteLater();
        reply_ = nullptr;
    }

    textStream_.resetStatus();
    textStream_.reset();        
    data_ = QString();
}

void HttpRequestWorker::httpFinished()
{
    if (httpRequestAborted_)
    {
        emit finished();

        return;
    }

    if (reply_->error() != QNetworkReply::NoError)
    {
        lastNetworkError_ = reply_->error();
        lastErrorString_ = reply_->errorString();

        if (asynchronous_)
            emit error(lastNetworkError_, lastErrorString_);
    }
    else if (!reply_->attribute(QNetworkRequest::RedirectionTargetAttribute).isNull())
    {
        url_ = url_.resolved(reply_->attribute(QNetworkRequest::RedirectionTargetAttribute).toUrl());

        textStream_.resetStatus();
        textStream_.reset();
        data_ = QString();

        if (reply_)
        {
            reply_->deleteLater();
            reply_ = nullptr;
        }

        verbRequest(reply_->operation());
        return;
    }
    else
    {
        lastResponseData_ = QByteArray::fromStdString(textStream_.readAll().toStdString());

        if (asynchronous_)
            emit success(lastResponseData_);
    }

    stop();
}

void HttpRequestWorker::httpReadyRead()
{
    textStream_ << reply_->readAll();
}

void HttpRequestWorker::slotAuthenticationRequired(QNetworkReply*, QAuthenticator * authenticator) const
{
    authenticator->setUser(url_.userName());
    authenticator->setPassword(url_.password());
}

bool HttpRequestWorker::stopped()
{
    QMutexLocker locker(&mutex_);

    return stopped_;
}

void HttpRequestWorker::stop()
{
    QMutexLocker locker(&mutex_);

    stopped_ = true;

    emit finished();
}

Could anyone give me a hint about what is wrong with my code?

Thanks!

SGaist

Hi,

From a quick look:

Why do you have a while loop right after you started your thread ?
Why create so many QThread ?
Why create so many QNetworkAccessManager ?

Also, you are calling indirectly deleteLater but still use the object afterward to return some data. That doesn't take into account that you could get a context switch, get worker_ deleted and the come back to the return statement.

By the way, out of curiosity, why do you need such a class ?

josephdp

Maybe you want to use QNetworkAccessManager?

SGaist

@josephdp there's already one in the HttpRequestManager class.

shylock_the_camel

Hi and thanks for replying,

I updated the code to take into account your suggestion about the invocation of deleteLater: there was indeed no need to do it that way. However, given that I used the flag Qt::QueuedConnection, I believe the deleteLater method would have been invoked after the program hits the return statement, at some point where the control is given back to the event loop, right? (Here is the original snippet)

if (worker_->lastErrorCode() != QNetworkReply::NoError)
{
    qCritical(core::serverIoLoggingCategory, "HTTP request failed with error: %s", qPrintable(worker_->lastErrorString()));

    hasError_ = true;

    QMetaObject::invokeMethod(worker_, "deleteLater", Qt::QueuedConnection);

    return QByteArray();
}

QMetaObject::invokeMethod(worker_, "deleteLater", Qt::QueuedConnection);
		
return worker_->lastResponseData();

Why do you have a while loop right after you started your thread ?

I created the HttpRequestManager class to be able to make synchronous HTTP requests as well as asynchronous ones. And using this loop to wait for the worker thread to finish was the only I found to implement the synchronous behavior.

Why create so many QThread ?

I am actually using this class inside a loop over a set of items where each item may call it (asynchronously). And I believe that when there are too many of those requests processing simultaneously (from successive items in the loop), then the message "QWinEventNotifier: Cannot have more than 62 enabled at one time" appears.

Why create so many QNetworkAccessManager ?

If I had a unique QNetworkAccessManager instance for all workers, would it be thread-safe to access it from any of these threads simultaneously?

why do you need such a class?

Besides the synchronous requests, I also added to this class, some methods to scrape a web page by running PhantomJS as a process (via QProcess). I got rid of these methods in the version I posted to make it more compact but I do have some problems with QProcess as well.

Thanks for your time!

SGaist

QNetworkAccessManager is reentrant.

You should rather use a QEventLoop if you want to emulate a blocking behavior. Your while loop just blocks the current thread and doesn't allow event processing.