http file download size limit?

Scalextrix

Im trying to understand an issue with a file downloader over http written with Qt by someone else who is no longer available.

The downloader works fine for files of upto ~1.8GB, but over this size the download does not initiate and fails on a timeout. The code appears to use a readyRead() to write chunks to disk.

Code is quite old circa 2013, so I thought it may be a 32bit legacy issue, so I compiled the executable on a 64bit Linux OS and the executable appears to have compiled as 64bit, but the download issue persists.

Is there anything Qt specific that would cause this, or is it likely something in the underlying C++ architecture of the program?

Apologies this is rather general, but Im a volunteer noob trying to learn as I go.

SGaist

Hi and welcome to devnet,

Can you show how the file is written ?
Can you show how the request / reply is managed ?

Kent-Dorfman

and being that you are reading a file through a web server, you need to verify that the server or network are not limiting the download size as well. Try multiple servers with 2BG+ files before assuming the code is broken. Even better if you can verify the whole process locally over the loopback on your own machine.

Scalextrix

@Kent-Dorfman said in http file download size limit?:

Try multiple servers with 2BG+ files

Thanks yes another party setup an alternate download URL, I will confirm they have checked their configuration for download size

Scalextrix

@SGaist thanks here is the relevant function:

void Downloader::startRequest(QUrl url)
{
downloadProgress = 0;
downloadFinished = false;

downloadTimer->start(30000);

QNetworkRequest request;
request.setUrl(url);
request.setRawHeader("User-Agent", "Downloader 1.0");
reply = manager->get(request);
reply->ignoreSslErrors();


connect(reply, SIGNAL(readyRead()),
        this, SLOT(httpReadyRead()));


connect(reply, SIGNAL(downloadProgress(qint64,qint64)),
        this, SLOT(updateDownloadProgress(qint64,qint64)));


connect(reply, SIGNAL(finished()),
        this, SLOT(downloaderFinished()));

// Network error
connect(reply, SIGNAL(error(QNetworkReply::NetworkError)),
        this, SLOT(networkError()));

As I said on the larger files it seems to expire on the timer. Thanks.

jsulm

@Scalextrix said in http file download size limit?:

thanks here is the relevant function

This is not relevant code. Please show the code where you actualy write to the file.
Your httpReadyRead() and downloaderFinished().

Pablo J. Rogina

@Scalextrix said in http file download size limit?:

As I said on the larger files it seems to expire on the timer.

As @jsulm mentioned, your code snippet doesn't seem relevant.

It doesn't even show if the timeout() signal for object downloader (QTimer) is connected to some slot

Scalextrix

Thank-you @jsulm

void Downloader::httpReadyRead()
{
if (file)
    file->write(reply->readAll());
}

&

void Downloader::downloaderFinished()
{
// Finished with timer
if (downloadTimer->isActive())
{
    downloadTimer->stop();
}

// when canceled
if (httpRequestAborted)
{
    if (file)
    {
        file->close();
        file->remove();
        delete file;
        file = 0;
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
    ui->quitButton->setEnabled(true);
    return;
}

// when partial
if (processChain && file && file->size() < 1000000)
{
    if (file)
    {
        file->close();
        file->remove();
        delete file;
        file = 0;
    }
    if (!reply->errorString().isEmpty())
    {
        ui->statusLabel->setText(tr("Error: Download ended prematurely.\n\n%1").arg(reply->errorString()));
    }
    else
    {
        ui->statusLabel->setText(tr("Error: Download ended prematurely."));
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
    ui->quitButton->setEnabled(true);
    return;
}

// download finished normally
file->flush();
file->close();

// get redirection url
QVariant redirectionTarget = reply->attribute(QNetworkRequest::RedirectionTargetAttribute);
if (reply->error())
{
    file->remove();
    if (!autoDownload)
    {
        QMessageBox::information(this, tr("Downloader"),
                             tr("Download terminated: %1.").arg(reply->errorString()));
    }
    ui->downloadButton->setEnabled(true);
    ui->downloadButton->setDefault(true);
    ui->continueButton->setEnabled(false);
}
else
{
    if (!redirectionTarget.isNull())
    {
        QUrl newUrl = url.resolved(redirectionTarget.toUrl());
        if (autoDownload || QMessageBox::question(this, tr("Downloader"),
                              tr("Redirect to %1 ?").arg(newUrl.toString()),
                              QMessageBox::Yes | QMessageBox::No) == QMessageBox::Yes)
        {
            url = newUrl;
            reply->deleteLater();
            file->open(QIODevice::WriteOnly);
            file->resize(0);
            startRequest(url);
            return;
        }
    }
    else
    {
        ui->statusLabel->setText(tr("Download was successful.  Press 'Next' to continue."));
        ui->downloadButton->setEnabled(false);
        ui->continueButton->setEnabled(true);
        ui->continueButton->setDefault(true);
        ui->quitButton->setDefault(false);
    }
}

reply->deleteLater();
reply = 0;
delete file;
file = 0;
manager = 0;
downloadFinished = true;

if (autoDownload)
{
    if (ui->continueButton->isEnabled())
    {
        on_continueButton_clicked();
    }
    else
    {
        on_quitButton_clicked();
    }
}
}

jsulm

@Scalextrix Why is there this timeout and is it possible that it simply takes to long to download bigger files, so the timeout kills the download?

Scalextrix

@jsulm I believe the timeout is for the purpose of catching any download problems and so the application doesn't hang indefinitely. A download of the 1.8GB file that works takes around 4 minutes to complete. With larger file sizes the download never starts, it just hangs for 30 seconds and then the timeout kills the downloader.

As an experiment I initiated the download and then disconnected from my local network, it took a little more than the 30 seconds to recognize the download had failed and exit the downloader.

I assume either as part of the Qt downloader library, or somewhere in the underlying code there must be a signal that asks for the size of the file to be downloaded, and its breaching some kind of threshold, and that is sending a signal that is registering as an error.

Bonnie

@Scalextrix But do you notice that the timeout interval of the downloadTimer is exactly 30 seconds? It won't wait 4 minutes for you to finish the whole file.
If you need the timer to ensure the download doesn't hang infinitely, you should restart it whenever you receive new data:

void Downloader::httpReadyRead()
{
    if (downloadTimer->isActive())
        downloadTimer->start();
    if (file)
        file->write(reply->readAll());
}

Or is the timer updated somewhere else but you haven't showed?
And I think it's also important to show the code where the file is created.