http file download size limit?

SGaist

Hi and welcome to devnet,

Can you show how the file is written ?
Can you show how the request / reply is managed ?

Kent-Dorfman

and being that you are reading a file through a web server, you need to verify that the server or network are not limiting the download size as well. Try multiple servers with 2BG+ files before assuming the code is broken. Even better if you can verify the whole process locally over the loopback on your own machine.

Scalextrix

@Kent-Dorfman said in http file download size limit?:

Try multiple servers with 2BG+ files

Thanks yes another party setup an alternate download URL, I will confirm they have checked their configuration for download size

Scalextrix

@SGaist thanks here is the relevant function:

void Downloader::startRequest(QUrl url)
{
downloadProgress = 0;
downloadFinished = false;

downloadTimer->start(30000);

QNetworkRequest request;
request.setUrl(url);
request.setRawHeader("User-Agent", "Downloader 1.0");
reply = manager->get(request);
reply->ignoreSslErrors();


connect(reply, SIGNAL(readyRead()),
        this, SLOT(httpReadyRead()));


connect(reply, SIGNAL(downloadProgress(qint64,qint64)),
        this, SLOT(updateDownloadProgress(qint64,qint64)));


connect(reply, SIGNAL(finished()),
        this, SLOT(downloaderFinished()));

// Network error
connect(reply, SIGNAL(error(QNetworkReply::NetworkError)),
        this, SLOT(networkError()));

As I said on the larger files it seems to expire on the timer. Thanks.

jsulm

@Scalextrix said in http file download size limit?:

thanks here is the relevant function

This is not relevant code. Please show the code where you actualy write to the file.
Your httpReadyRead() and downloaderFinished().

Pablo J. Rogina

@Scalextrix said in http file download size limit?:

As I said on the larger files it seems to expire on the timer.

As @jsulm mentioned, your code snippet doesn't seem relevant.

It doesn't even show if the timeout() signal for object downloader (QTimer) is connected to some slot

Scalextrix

Thank-you @jsulm

void Downloader::httpReadyRead()
{
if (file)
    file->write(reply->readAll());
}

&

void Downloader::downloaderFinished()
{
// Finished with timer
if (downloadTimer->isActive())
{
    downloadTimer->stop();
}

// when canceled
if (httpRequestAborted)
{
    if (file)
    {
        file->close();
        file->remove();
        delete file;
        file = 0;
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
    ui->quitButton->setEnabled(true);
    return;
}

// when partial
if (processChain && file && file->size() < 1000000)
{
    if (file)
    {
        file->close();
        file->remove();
        delete file;
        file = 0;
    }
    if (!reply->errorString().isEmpty())
    {
        ui->statusLabel->setText(tr("Error: Download ended prematurely.\n\n%1").arg(reply->errorString()));
    }
    else
    {
        ui->statusLabel->setText(tr("Error: Download ended prematurely."));
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
    ui->quitButton->setEnabled(true);
    return;
}

// download finished normally
file->flush();
file->close();

// get redirection url
QVariant redirectionTarget = reply->attribute(QNetworkRequest::RedirectionTargetAttribute);
if (reply->error())
{
    file->remove();
    if (!autoDownload)
    {
        QMessageBox::information(this, tr("Downloader"),
                             tr("Download terminated: %1.").arg(reply->errorString()));
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
}
else
{
    if (!redirectionTarget.isNull())
    {
        QUrl newUrl = url.resolved(redirectionTarget.toUrl());
        if (autoDownload || QMessageBox::question(this, tr("Downloader"),
                              tr("Redirect to %1 ?").arg(newUrl.toString()),
                              QMessageBox::Yes | QMessageBox::No) == QMessageBox::Yes)
        {
            url = newUrl;
            reply->deleteLater();
            file->open(QIODevice::WriteOnly);
            file->resize(0);
            startRequest(url);
            return;
        }
    }
    else
    {
        ui->statusLabel->setText(tr("Download was successful.  Press 'Next' to continue."));
        ui->downloadButton->setEnabled(false);
        ui->continueButton->setEnabled(true);
        ui->continueButton->setDefault(true);
        ui->quitButton->setDefault(false);
    }
}

reply->deleteLater();
reply = 0;
delete file;
file = 0;
manager = 0;
downloadFinished = true;

if (autoDownload)
{
    if (ui->continueButton->isEnabled())
    {
        on_continueButton_clicked();
    }
    else
    {
        on_quitButton_clicked();
    }
}
}

jsulm

@Scalextrix Why is there this timeout and is it possible that it simply takes to long to download bigger files, so the timeout kills the download?

Scalextrix

@jsulm I believe the timeout is for the purpose of catching any download problems and so the application doesn't hang indefinitely. A download of the 1.8GB file that works takes around 4 minutes to complete. With larger file sizes the download never starts, it just hangs for 30 seconds and then the timeout kills the downloader.

As an experiment I initiated the download and then disconnected from my local network, it took a little more than the 30 seconds to recognize the download had failed and exit the downloader.

I assume either as part of the Qt downloader library, or somewhere in the underlying code there must be a signal that asks for the size of the file to be downloaded, and its breaching some kind of threshold, and that is sending a signal that is registering as an error.

Bonnie

@Scalextrix But do you notice that the timeout interval of the downloadTimer is exactly 30 seconds? It won't wait 4 minutes for you to finish the whole file.
If you need the timer to ensure the download doesn't hang infinitely, you should restart it whenever you receive new data:

void Downloader::httpReadyRead()
{
    if (downloadTimer->isActive())
        downloadTimer->start();
    if (file)
        file->write(reply->readAll());
}

Or is the timer updated somewhere else but you haven't showed?
And I think it's also important to show the code where the file is created.