Downloading a file via http

MartynWheeler

Hi,

This may seem a silly question but i am trying to get my head around downloading a file from a webserver. I have the following that I have put together.

    /* connect to server and download data */
    QNetworkAccessManager *mNetworkManager = new QNetworkAccessManager(this);
    connect(mNetworkManager, &QNetworkAccessManager::finished, this, &AtomicData::onNetworkReply);
    mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));

This makes an asynchronous request. How can I check if this is finished so that I can process the data? At the moment I am having to do all of the processing in the onNetworkReply function. I was wondering if I could do the processing after the network request had finished. It just seems to make sense coming from a background where I wait for one thing to happen before moving to the next...

Thank you,

here is my onNetworkReply function:

/* function called if no input data exists to get data from server */
void AtomicData::onNetworkReply(QNetworkReply* reply)
{
    const int RESPONSE_OK = 200;
    const int RESPONSE_ERROR = 404;
    const int RESPONSE_BAD_REQUEST = 400;

    QString replyString;
    if(reply->error() == QNetworkReply::NoError)
    {
        int httpstatuscode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toUInt();
        switch(httpstatuscode)
        {
        case RESPONSE_OK:
            if (reply->isReadable())
            {
                /* replyString now contains the file */
                replyString = QString::fromUtf8(reply->readAll().data());

                /* split the data into rows */
                QStringList l = replyString.split('\n');
                int numberofRows = l.count() - 1;

                /* set up table for displaying data */
                ui->tableWidget->setRowCount(numberofRows - 39);

                /* define a variable to hold substring */
                QString subStringValue;

                for (int row = 39; row < numberofRows ; row++) {
                    /* set up rowConter for table */
                    int rowCounter = row - 39;

                    /* get the number of neutrons */
                    subStringValue = l[row].mid(4,5);
                    int neutrons = subStringValue.toInt();
                    ui->tableWidget->setItem(rowCounter, 0, new QTableWidgetItem(subStringValue));

                    /* get the number of protons */
                    subStringValue = l[row].mid(9,5);
                    int protons = subStringValue.toInt();
                    ui->tableWidget->setItem(rowCounter, 1, new QTableWidgetItem(subStringValue));

                    /* get the number of nucleons */
                    subStringValue = l[row].mid(14,5);
                    int nucleons = subStringValue.toInt();
                    ui->tableWidget->setItem(rowCounter, 2, new QTableWidgetItem(subStringValue));

                    /* get the element name */
                    subStringValue = l[row].mid(20,3);
                    std::string element = subStringValue.toStdString();
                    ui->tableWidget->setItem(rowCounter, 3, new QTableWidgetItem(subStringValue));

                    /* get the binding energy */
                    subStringValue = l[row].mid(53,11);
                    if (subStringValue.contains("#")){
                        subStringValue.replace(subStringValue.indexOf("#"), 2, ".0");
                    }
                    double bindingEnergy = subStringValue.toDouble();
                    ui->tableWidget->setItem(rowCounter, 4, new QTableWidgetItem(subStringValue));

                    /* get the binding energy uncertainty */
                    subStringValue = l[row].mid(64,9);
                    if (subStringValue.contains("#")){
                        subStringValue.replace(subStringValue.indexOf("#"), 2, ".0");
                    }
                    double bindingEnergyUncertainty = subStringValue.toDouble();
                    ui->tableWidget->setItem(rowCounter, 5, new QTableWidgetItem(subStringValue));

                    /* get the atomic mass */
                    subStringValue = l[row].mid(96,3);
                    subStringValue += l[row].mid(100,12);
                    if (subStringValue.contains("#")){
                        subStringValue.replace(subStringValue.indexOf("#"), 2, ".0");
                    }
                    double atomicMass = subStringValue.toDouble();
                    ui->tableWidget->setItem(rowCounter, 6, new QTableWidgetItem(subStringValue));

                    /* get the atomic mass uncertainty */
                    subStringValue = l[row].mid(112,11);
                    if (subStringValue.contains("#")){
                        subStringValue.replace(subStringValue.indexOf("#"), 2, ".0");
                    }
                    double atomicMassUncertainty = subStringValue.toDouble();
                    ui->tableWidget->setItem(rowCounter, 7, new QTableWidgetItem(subStringValue));

                    /* create an atom object array element */
                    this->atoms_[rowCounter] = Atom(neutrons, protons, nucleons, element, bindingEnergy, bindingEnergyUncertainty, atomicMass, atomicMassUncertainty);
                }
                plotNuclearData(ui->customPlot);
            }
            break;
        case RESPONSE_ERROR:
            break;
        case RESPONSE_BAD_REQUEST:
            break;
        default:
            break;
        }
    }
    reply->deleteLater();
}

JKSH

@MartynWheeler said in Downloading a file via http:

I think that I am just misunderstanding things here, because at the moment once I have issued the line
mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));
the program will carry on (I can see a debug statement print out) while the get request is being processed.

Ah, I see.

As the others have mentioned, QNetworkAccessManager is indeed designed to "carry on while the get request is being processed". This allows your GUI to do other things while the request is being processed -- If you force it to wait, your GUI will freeze up.

May I ask why you want to force it to wait? If it is to avoid writing a separate function, you can write your processing code in a lambda expression instead:

QNetworkReply *reply = mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));
connect(reply, &QNetworkReply::finished, [=]
{
    const int RESPONSE_OK = 200;
    const int RESPONSE_ERROR = 404;
    const int RESPONSE_BAD_REQUEST = 400;

    QString replyString;
    if(reply->error() == QNetworkReply::NoError)
    {
        int httpstatuscode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toUInt();
        switch(httpstatuscode)
        {
            // ...
        }
    }
    reply->deleteLater();
});

JKSH

@MartynWheeler said in Downloading a file via http:

How can I check if this is finished so that I can process the data?
...
I was wondering if I could do the processing after the network request had finished.

What do you mean? Your existing code already waits until it's finished before calling onNetworkReply(), right?

When the download finishes, QNetworkAccessManager and QNetworkReply emit their finished() signals.

Note: QNetworkAccessManager::get() returns a QNetworkReply object. You can connect to the QNetworkReply instead of the QNetworkAccessManagere.

If the file is large, then it will be downloaded in chunks. Each time a chunk arrives, QNetworkReply emits a readyRead() and downloadProgress() signals. This allows you to process the file incrementally.

MartynWheeler

Hi,

Thanks for your reply. Sorry if I was not clear. Basically I want to return the reply from the get and THEN process it. Something like:

    //set up 
    QNetworkAccessManager *mNetworkManager = new QNetworkAccessManager(this);
    connect(mNetworkManager, &QNetworkAccessManager::finished, this, &AtomicData::onNetworkReply);


    my reply = mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));

    //now that I have a the full reply I want to process the data in the reply
    process and display...

I think that I am just misunderstanding things here, because at the moment once I have issued the line

mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));

the program will carry on (I can see a debug statement print out) while the get request is being processed. I can't work out how to wait until the processing has finished before carrying on? I tried the following:

    /* connect to server and download data */
    QNetworkAccessManager *mNetworkManager = new QNetworkAccessManager(this);
    connect(mNetworkManager, &QNetworkAccessManager::finished, this, &AtomicData::onNetworkReply);


    if (mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")))->isFinished()){
        qDebug() << "I am here";
    }

with this

/* function called if no input data exists to get data from server */
void AtomicData::onNetworkReply(QNetworkReply* reply)
{
    const int RESPONSE_OK = 200;
    const int RESPONSE_ERROR = 404;
    const int RESPONSE_BAD_REQUEST = 400;

    if(reply->error() == QNetworkReply::NoError)
    {
        int httpstatuscode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toUInt();
        switch(httpstatuscode)
        {
        case RESPONSE_OK:
            if (reply->isReadable()){
                qDebug() << "I am in here";
                /* replyString now contains the file */
//                QString replyString = QString::fromUtf8(reply->readAll().data());

                /* process the data */
//                processData(replyString);
            }
            break;
        case RESPONSE_ERROR:
            break;
        case RESPONSE_BAD_REQUEST:
            break;
        default:
            break;
        }
    }
    qDebug() << "I am now down here";
    reply->finished();
    reply->deleteLater();
}

but it just goes into a loop and crashes? I can't quite work out how to ensure that the reply has finished? Sorry, if I am missing something obvious.

Martyn

jsulm

@MartynWheeler Do you really need to wait until you got whole reply? You will block the caller this way. Qt is an assynchronous framework and you should prefer asynchronous behavior.
You should emit a signal as soon as you got everything, so the caller initiates the download and is informed later that it is finished via a signal.

J.Hilk

@MartynWheeler
alt text

Pablo J. Rogina

@MartynWheeler you may want to look at this example just in case.

JKSH

@MartynWheeler said in Downloading a file via http:

I think that I am just misunderstanding things here, because at the moment once I have issued the line
mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));
the program will carry on (I can see a debug statement print out) while the get request is being processed.

Ah, I see.

As the others have mentioned, QNetworkAccessManager is indeed designed to "carry on while the get request is being processed". This allows your GUI to do other things while the request is being processed -- If you force it to wait, your GUI will freeze up.

May I ask why you want to force it to wait? If it is to avoid writing a separate function, you can write your processing code in a lambda expression instead:

QNetworkReply *reply = mNetworkManager->get(QNetworkRequest(QUrl("https://www-nds.iaea.org/amdc/ame2016/mass16.txt")));
connect(reply, &QNetworkReply::finished, [=]
{
    const int RESPONSE_OK = 200;
    const int RESPONSE_ERROR = 404;
    const int RESPONSE_BAD_REQUEST = 400;

    QString replyString;
    if(reply->error() == QNetworkReply::NoError)
    {
        int httpstatuscode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toUInt();
        switch(httpstatuscode)
        {
            // ...
        }
    }
    reply->deleteLater();
});

MartynWheeler

Thanks for the replies. I guess that i was misunderstanding things. I just do all of the data processing in my onNetworkReply slot which fires off on a network request. Thank you for the explanations.

Martyn