Download files concurrently

Mr Gisa

I want to create a download manager in Qt, a simple one actually, most of the download managers has a kind of segmented download algorithm where they take advantage of the HTTP protocol and download something from the internet using the range header, get a file using multiple threads and after each piece of the file finished downloading, it will merge all the pieces. It has several advantages, but I don't want to talk about that here.
The real idea here is that I want to create a kind of task management with limitations, like a semaphore so I can execute multiple tasks at the same time (concurrently)(download a file each task) and limit the total of concurrent tasks (semaphore?). The idea is that I need the information of each task, for example, the progress or the status (so I can create a kind of list or table showing each task and its progress).
Is there any idea of how I can create something like?

Maarouf Med Mahdi

Hi
I advise you to watch the tutoriel . i wish that be helpful for you .
https://www.youtube.com/watch?v=PS1cp06wmrk&t=9s
note : take all of the videos that is 6 parts of the project .
good luck .

Mr Gisa

I haven't watched the videos yet but I was wondering in what part of the question these videos will help me? Is it related to the download part? (I'm assuming this cause I saw TCP, etc).

mpergand

I'm using QNetworkAccessManager.
Each request is recorded in a queue.
You need this signals:

NetworkDownload& download=aDownloadQueue[next];
  QNetworkRequest request(download.url);
  QNetworkReply* reply=pNetworkAccesManager->get(request);

  ConnectThis(reply, finished());
  ConnectThis(reply, downloadProgress(qint64, qint64));
  ConnectThis(reply, readyRead());
  ConnectThis(reply, sslErrors(const QList<QSslError>));
  ConnectThis(reply, redirected(const QUrl));
  ConnectThis(reply, error(QNetworkReply::NetworkError));
  ConnectThis(reply, metaDataChanged());

ConnectThis is a macro of mine.

Have a look at the "Network Download Example" in Qt documentation.

Mr Gisa

@mpergand I assume this image is a browser that you made with a download manager right? I was wondering how do you manage to get information of each download and put in each item on the view like yours?
The download part or the queue part I don't think is the complicated part but how to get individual information of each download and update the list view.
btw, if this is a browser that you made, nice ui you made, the dropdown thing and the custom list view to show the download items is dope, I wish I was smart enough to do that. hahaa

mpergand

It's my browser, with a custom GridView in the downloads popup window.

I'm using a protocol to communicate with the gridView.

bool NetworkDownloadManager::startDownload(const QUrl &url, NetworkDownloadManagerClient* client)

The client must respond to the protocol:

struct NetworkDownloadManagerClient
{
    virtual            void  downloadWillStart(NetworkDownload* download) { }
    virtual            void  downloadProgress(NetworkDownload*, int received, float elapsed) {}
    virtual            void  downloadFinished(NetworkDownload*)=0;
    virtual            void  metaData(NetworkDownload* download) {}

for example:

void NetworkDownloadManager::metaDataChanged()
{
    QNetworkReply* reply=qobject_cast<QNetworkReply*>(sender());
    NetworkDownloadManagerClient* client=aDownloads.value(reply);
    NetworkDownload* download=downloadForReply(reply);
// lot of code here
// call the client method
client->metaData(download);
}

// then in the client
void DownloadManager::metaData(NetworkDownload* download)
{
    gridView()->updateCellAtIndex(download->index);
}

NetworkDowload contents all the informations needed by the client to update the grid view (start time, total size, received bytes, etc)

Mr Gisa

@mpergand off-topic: is your browser tab based or a single page only? - the small progress on the button is an actual progress?

Your approach is interesting, so in reality you use signal and slots to communicate when the metadata (information of the reply like the progress and stuff) is changed to update the gridview. Nice.

mpergand

@Mr-Gisa said in Download files concurrently:

the small progress on the button is an actual progress?

Yes, but not implemented yet :)

off-topic: is your browser tab based or a single page only?

Single page, i'm planning to go tab based.

The sad thuth is that i'm going to give up with the QWebEngine, still buggy and incomplete in Qt 5.8 and i'm stuck with this version because of OSX 10.9.

I'm looking to change for Chromium (CEF), but it's a lot of work ...

Mr Gisa

I was planning in implementing a tab based just like other browsers and I found a browser that was made with Qt, that is qupzilla, their implemention is nice (visually), the pinned tabs, the scrollbars in each side of the tab when the tabs overflow on the horizontal, the drag and drop, etc... the real problem is that the code is a pure mess and I couldn't understand a thing so I just gave up (if you want help or join forces in that tell me).
The CEF thing is kinda complicated cause you will need to create the bindings and stuff.
Isn't it possible to update to Qt 5.10 on OSX 10.9?

mpergand

Isn't it possible to update to Qt 5.10 on OSX 10.9?

No, Qt 5.8 is the last version supported for 10.9.

the real problem is that the code is a pure mess

Agree :)