[Solved]QNetworkAccessManager sometimes does not download file

alamahant

Hi Guys,
Noob here.
I just wrote a simple downlod file app.
Everything is working fine progressBar and all.
However although some files even big isos download just fine some other files simply wont start downloading at all.
Either Qt or the remote server closes the connection.
Here is an excerpt from my code:
[code]
void MainWindow::on_downloadButton_clicked()
{

url = QUrl(ui->lineEdit->text());
request.setUrl(url);
reply = manager->get(request);
//reply->ignoreSslErrors();
connect(reply,&QIODevice::readyRead,this,&MainWindow::downloadBegin);
connect(reply, &QNetworkReply::finished, this, &MainWindow::downloadEnd);
connect(reply, &QNetworkReply::downloadProgress, this, &MainWindow::setProgress);
//connect(reply, static_cast<void (QNetworkReply::*)(QNetworkReply::NetworkError)>(&QNetworkReply::error), this, &MainWindow::displayNetworkError);

}
[/code]

[code]
void MainWindow::downloadBegin()
{

mDataBuffer->append(reply->readAll());
qDebug() << "Incoming Data....";

}

[/code]

[code]
void MainWindow::downloadEnd()
{
if(reply->NoError) {
qDebug() << "Error :" + reply->errorString();
}else {
QString filename = MainWindow::savaAs();
QFile file(filename);

    if (file.open(QIODevice::WriteOnly)) {
        qDebug() << "file opened";
        file.write(*mDataBuffer);
        file.close();
        qDebug() << "file closed";
    }else {
        qDebug() << "Download Error!!";
    }
}

}
[/code]

What might the problem be?
Is it an SSL error?
How to deal with it?
For example I can download fine a Void iso but NOT a Debian iso.
I am so confused...
Any help would be god sent.
Thanks for the care and consideration...
:)

JonB

@alamahant
I don't know the answer, but why do you comment out:

connect(reply, static_cast<void (QNetworkReply::*)(QNetworkReply::NetworkError)>(&QNetworkReply::error), this, &MainWindow::displayNetworkError);

Is it not possible that some useful error message is being ignored?

One other thing: how long does your reply variable stay in scope? Is it a MainWindow member variable or a local variable? I haven't used this QNAM stuff, but I assume it must stay till at least QNetworkReply::finished, does it do so for you?

alamahant

@JonB ,
Thanks a lot for the reply.
:)

The manager, reply and data buffer are declared in header file as pointers and initialized in the constructor like this:

manager = new QNetworkAccessManager(this);
reply = nullptr;
mDataBuffer = new QByteArray();

Even if if i uncomment the connect error statement I get no output from qQebug().
This is my displayError method:

void MainWindow::displayNetworkError()
{
qDebug() << reply->errorString();
}

Christian Ehrlicher

I would not download large files this way. I would connect to QNetworkReply::readyRead() to read all currently available data and write it out to the file. Otherwise you may need e.g. 4GB of continuous memory when downloading a 4GB iso which might not work (or only sometimes)

alamahant

@Christian-Ehrlicher
I AM connecting to QNetworkReply::readyRead()
Please see my initial post.
:)

Christian Ehrlicher

@alamahant But then you save the stuff in memory instead directly writing to file so the problem stays the same.

alamahant

@Christian-Ehrlicher
Thanks for the clarity.
How do I do what you are proposing.
In qDebug I see thousands of "incoming data" while the file is being downloaded.
Therefore it seems it is reading and writing in batches....
Please forgive my ignorance but i am not clear what you are proposing that i do......
:)

Christian Ehrlicher

@alamahant said in QNetworkAccessManager sometimes does not download file:

How do I do what you are proposing.

Don't understand - instead writing to a buffer in readyRead() write to you output file.

JonB

@alamahant
During your downloadBegin() (btw, bad name, this is attached to readyRead() so is called over & over) you are presently pushing all the bytes received into *mDataBuffer, and only writing the complete buffer to file at end in downloadEnd(). @Christian-Ehrlicher is pointing out that if the file is, say, 4GB long that will use up 4GB of memory --- bad idea. He wants you to open the output file at the start, and write the bytes received as they arrive in readyRead() to file at that point, so as not to use up memory.

alamahant

@all
Yes I tried the proposed solution but I still cant get certain files to download, while others -- equally large download just fine.
I think its a client/server negotiation issue but i am clueless how to debug it.
Should I maybe set download speed limit or something of the sort?
Or is it an SSL thing.
Does Qt use certificates and where are they located?
Or best how to force it to not close any connection on ssl errors if --indeed the closing happens by Qt.
I dont get any info also from my connect error statement and the pethod associated with it.
Please see above...
Sory for tiring you
Thanks a lot

JonB

@alamahant
I don't know, other than I would have expected some sort of error.

The most obvious question is what are these URLs? If you have one file that downloads and another that does not, are they at the same site/in the same directory, or in different sites, or what? Are these across https? Are you able to visit these URLs in a browser, outside of Qt application?

alamahant

For example This downloads fine
https://alpha.de.repo.voidlinux.org/live/20170220/void-live-x86_64-20170220-xfce.iso

BUT this closes the connection

https://cdimage.debian.org/cdimage/bullseye_di_alpha2/amd64/iso-cd/debian-bullseye-DI-alpha2-amd64-netinst.iso

It seems this is nOT a size amatter as the second is much smaller than the first.
Nor an http/https issue as both are https.
So the logical conclusion is some kind of negotiation tls handshake error.
Both download fine with wget.
BUT I just noticed something in the wget output.

The first one(the one that downloads fine gives this output

wget https://alpha.de.repo.voidlinux.org/live/20170220/void-live-x86_64-20170220-xfce.iso
--2020-09-30 19:19:58-- https://alpha.de.repo.voidlinux.org/live/20170220/void-live-x86_64-20170220-xfce.iso
Resolving alpha.de.repo.voidlinux.org... 95.216.76.97
Connecting to alpha.de.repo.voidlinux.org|95.216.76.97|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 558891008 (533M) [application/octet-stream]
Saving to: ‘void-live-x86_64-20170220-xfce.iso’

void-live-x86_64-20170220-xfc 0%[ ] 2.60M 1.45MB/s

WHEREAS the secon -- the one fails to download gives this:

wget https://cdimage.debian.org/cdimage/bullseye_di_alpha2/amd64/iso-cd/debian-bullseye-DI-alpha2-amd64-netinst.iso
--2020-09-30 19:20:20-- https://cdimage.debian.org/cdimage/bullseye_di_alpha2/amd64/iso-cd/debian-bullseye-DI-alpha2-amd64-netinst.iso
Resolving cdimage.debian.org... 194.71.11.165, 194.71.11.173, 2001:6b0:19::165, ...
Connecting to cdimage.debian.org|194.71.11.165|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://saimei.ftp.acc.umu.se/cdimage/bullseye_di_alpha2/amd64/iso-cd/debian-bullseye-DI-alpha2-amd64-netinst.iso [following]
--2020-09-30 19:20:22-- https://saimei.ftp.acc.umu.se/cdimage/bullseye_di_alpha2/amd64/iso-cd/debian-bullseye-DI-alpha2-amd64-netinst.iso
Resolving saimei.ftp.acc.umu.se... 194.71.11.138, 2001:6b0:19::138
Connecting to saimei.ftp.acc.umu.se|194.71.11.138|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 360710144 (344M) [application/x-iso9660-image]
Saving to: ‘debian-bullseye-DI-alpha2-amd64-netinst.iso’

      debian-bullseye-DI-   0%[                                                ] 792.00K   732KB/s               ^C

IS THERE a redirect of some sort involved in the second?
And if yes how to adjustt my Qt code to cover for this?

Thankssssssss
:)

alamahant

Here is another one that does NOT work

wget https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20200923T214503Z/stage3-amd64-20200923T214503Z.tar.xz
--2020-09-30 19:29:39-- https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds/20200923T214503Z/stage3-amd64-20200923T214503Z.tar.xz
Resolving bouncer.gentoo.org... 140.211.166.178, 2001:470:ea4a:1:225:90ff:fe02:1957
Connecting to bouncer.gentoo.org|140.211.166.178|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://mirror.isoc.org.il/pub/gentoo//releases/amd64/autobuilds/20200923T214503Z/stage3-amd64-20200923T214503Z.tar.xz [following]
--2020-09-30 19:29:41-- https://mirror.isoc.org.il/pub/gentoo//releases/amd64/autobuilds/20200923T214503Z/stage3-amd64-20200923T214503Z.tar.xz
Resolving mirror.isoc.org.il... 192.115.211.70, 2a01:4280:2:211a::70
Connecting to mirror.isoc.org.il|192.115.211.70|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 219663964 (209M) [application/octet-stream]
Saving to: ‘stage3-amd64-20200923T214503Z.tar.xz’

         stage3-amd64-202   0%[                                                ]  39.40K  17.8KB/s               ^C

What all the failed ones have in common is that there is some short of redirect involved....
Or is it Apache vs Nginx?
Where should I look to solve this?
Thanks Guys
:)

Christian Ehrlicher

@alamahant said in QNetworkAccessManager sometimes does not download file:

short of redirect involved....

correct, see https://doc.qt.io/qt-5/qnetworkrequest.html#Attribute-enum / QNetworkRequest::RedirectionTargetAttribute

alamahant

@Christian-Ehrlicher
Yessssssssssssssss.
You are a life saver.
After setting
request.setAttribute(QNetworkRequest::FollowRedirectsAttribute, 21);

It WORKS!!!!!
Thanks a lot....
:)

Christian Ehrlicher

@alamahant Nice, hope you leave the direct write to a file in your code since it saves a lot of memory :)