Qt Maintenance Tool Failed to download metadata.

[flashmozzg]

Re: Maintenance Tool Fails to Update

I had the same problem for years. SdkRepoChooser helped for a time before but now it also stopped working. My repo urls are filled with old garbage, like countless qt 5.1/5.2/5.3/5.4/5.5 etc. repos for all platforms previews and such. It's almost impossible to use Qt Maintenance tool on my home ISP, though it works fine when I bring my laptop out.

And each time I have to spent 15+ minutes to manually uncheck hundreds of "temporary" and other garbage repos because there is no bulk unchecking. It's simply infuriating.

[JKSH]

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

Re: Maintenance Tool Fails to Update
My repo urls are filled with old garbage, like countless qt 5.1/5.2/5.3/5.4/5.5 etc. repos for all platforms previews and such.

...

And each time I have to spent 15+ minutes to manually uncheck hundreds of "temporary" and other garbage repos

Hi @flashmozzg, I haven't seen these symptoms before. Could you please share a screenshot that shows these "temporary"/"garbage" repos?

Which mirror do you normally choose in the Qt SDK Repo Chooser?

[flashmozzg]

@JKSH I usually go with some random mirror and I think it always worked before. Now I have to manually "uncheck" half of the repos for obsolete qt version for maintenance tool to work.
It looks like it somehow manages to "overload" my network with huge amount of requests (even regular pages can have trouble loading while qt mt tries to fetch metadata) but it's the only program that does that.

[JKSH]

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

Now I have to manually "uncheck" half of the repos for obsolete qt version for maintenance tool to work.

Thanks for your screenshot. I have the same "temporary repositories" as you.

Does the metadata finish downloading if you wait? Or does it abort halfway?

It looks like it somehow manages to "overload" my network with huge amount of requests (even regular pages can have trouble loading while qt mt tries to fetch metadata) but it's the only program that does that.

I'm very surprised to hear that. I have the opposite problem: I have lots of bandwidth available, but the metadata only gets downloaded at ~10 KiB/s. I wish it would try to overload my network, so that I get the metadata faster!

Try this:

1. Open Resource Monitor.
2. Start downloading the metadata without unchecking the temporary repositories.
3. Look at the "Network" tab of Resource Monitor.

How much network resources gets used by MaintenanceTool.exe?

[flashmozzg]

@JKSH said in Qt Maintenance Tool Failed to download metadata.:

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

Now I have to manually "uncheck" half of the repos for obsolete qt version for maintenance tool to work.

Thanks for your screenshot. I have the same "temporary repositories" as you.

Does the metadata finish downloading if you wait? Or does it abort halfway?

Just like in the thread I was replying to: it goes to around 50-80% and then fails to download some metadata 7z.

It looks like it somehow manages to "overload" my network with huge amount of requests (even regular pages can have trouble loading while qt mt tries to fetch metadata) but it's the only program that does that.

I'm very surprised to hear that. I have the opposite problem: I have lots of bandwidth available, but the metadata only gets downloaded at ~10 KiB/s. I wish it would try to overload my network, so that I get the metadata faster!

I also don't have any problems with bandwidth: the tool itself uses no more than 10 kB/s (judging by procmon, overall network usage is few hundreds kB/s until it drops), dropping to 0 by the end. And I'm on a 60 mB/s wired connection. The tool itself opens almost 1.5k tcp connections to 77.86.229.90 though I don't think that this should be problematic by itself.

Also, I found the related bug https://bugreports.qt.io/browse/QTIFW-835 (but unfortunately no solution except use repo chooser which in my case stopped helping).

[JKSH]

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

I usually go with some random mirror and I think it always worked before.

Some mirrors are better for you than others, so I wonder if you just picked some bad ones recently...

Try the master.qt.io mirror. Does that let you finish downloading metadata without having to uncheck the "temporary repositories"?

[flashmozzg]

@JKSH said in Qt Maintenance Tool Failed to download metadata.:

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

I usually go with some random mirror and I think it always worked before.

Some mirrors are better for you than others, so I wonder if you just picked some bad ones recently...

Try the master.qt.io mirror. Does that let you finish downloading metadata without having to uncheck the "temporary repositories"?

After a few tries and after I've unchecked "use temp repos only" I think It finally worked. Not sure if it will last. Also this option is grayed out now so I'm not sure what to do to enable it back. Definitely not a pleasant experience.

[troste]

I got the same problem. To me it looks like the problem is at the server side.
It don't have the meta data to download. So it times out.
Looking at the network traffic it goes down to zero when for the to time out.

For sure the problem is not on my network or my installation. I reinstalled windows 10 and started the installer with the same result.

If someone could help me I would be real happy.
Or tell me a way to get Qt5.9.4 for windows 10 offline version.

Regards

Trond

[jsulm]

@troste Try this tool to select another repository: https://github.com/JKSH/QtSdkRepoChooser
I hope one day the official installer will provide this functionality (as far as I know there is already a change request for this).

[flashmozzg]

@jsulm said in Qt Maintenance Tool Failed to download metadata.:

@troste Try this tool to select another repository: https://github.com/JKSH/QtSdkRepoChooser
I hope one day the official installer will provide this functionality (as far as I know there is already a change request for this).

Unfortunately it rarely helps in my case (right now I've already spent an hour trying to update QTC to 4.6.0 and it worked once, forced me to updated the isntaller first and restart and now it's not working again), so I assume it's not really a solution.
Also, my repo setting don't look like this:

I have hundreds of stale Qt repos that are added with each new Qt version I install that are not deleted after I remove them from my PC. It literally takes me hours to deselect them all to just try if the new mirror works:

[jsulm]

@JKSH any idea?

[JKSH]

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

I have hundreds of stale Qt repos that are added with each new Qt version I install that are not deleted after I remove them from my PC. It literally takes me hours to deselect them all to just try if the new mirror works

@jsulm said in Qt Maintenance Tool Failed to download metadata.:

@JKSH any idea?

It looks like the MaintenanceTool has 2 distinct stages:

1. Download metadata -- This is always done through download.qt.io.
2. Download actual libraries -- This is normally done through an automatically-selected mirror.

The Qt SDK Repo Chooser lets us override #2, but it doesn't affect #1. Currently, #1 is inefficient because it involves 2391 HTTP GET requests to download tiny metadata files, so that's a lot of room for failure. This problem is felt the worst on Windows because Windows has the most packages available (multiple versions of MSVC, MinGW, Android, UWP). In contrast, Linux has fewer packages (GCC, Android) and only requires 1205 HTTP GET requests. (Numbers obtained via Wireshark today)

I think @flashmozzg's issue is that the connection between his PC and download.qt.io is flaky. This means it's unlikely for him to successfully complete all 2000+ GET requests in one run. Deselecting the repos helps because it reduces the number of GET requests. They are not "stale repos" per se -- they are all the repos for every version of Qt out there, not just the ones you installed before. Unfortunately, all the repos are enabled by default and there's currently no way of mass-deselecting them, so this is painful process.

The metadata download process needs improvement -- all those metadata files really shouldn't be downloaded individually. Things will be much better if the metadata is transmitted as one large file, or if the tool can be configured to only fetch a subset of metadata (e.g. skip Android and UWP packages)

I'll write up something to send to the Qt engineers.

EDIT: @flashmozzg, may I ask which country you currently live in?

[mchinand]

Is it possible to only download the metadata files if they are newer than the most recent request (e.g., using conditional requests). This would be especially useful when doing 'Check for Updates' followed by 'Add or Remove Components'.

[flashmozzg]

@JKSH said in Qt Maintenance Tool Failed to download metadata.:

EDIT: @flashmozzg, may I ask which country you currently live in?

Russia. One thing to note: I think it's not only/just the number of GETs but that they all happen simultaneously that overloads my network somehow. My webpages can even briefly stop loading when Qt MT "does its thing". And it usually works fine from my Uni net (but at home I've only ever experienced this problem with the maintenance tool and it was on different connections and different PCs, so it could be my ISP reacting poorly or something) though I haven't tried in a long time, maybe I was just lucky.

Btw, in any case spawning thousand of GETs per second is no good and I'm pretty sure it's reaching several OS limits (like max tcp connections).

[JKSH]

@flashmozzg said in Qt Maintenance Tool Failed to download metadata.:

Russia. One thing to note: I think it's not only/just the number of GETs but that they all happen simultaneously that overloads my network somehow. My webpages can even briefly stop loading when Qt MT "does its thing"...

Btw, in any case spawning thousand of GETs per second is no good and I'm pretty sure it's reaching several OS limits (like max tcp connections).

Hmm... I'm quite sure that QNetworkAccessManager only allows 6 simultaneous GETS; if more are requested, they get put in a queue until one of the 6 finish.

I can't explain why your webpages stop loading though. I experience slow metadata downloads (it takes a few minutes for me), but I normally just let MaintenanceTool do its thing in the background while I browse other sites.

Anyway, I've written to the Qt devs about this, and also about other issues relating to the online installer. Anyone who's interested can join in at:

• http://lists.qt-project.org/pipermail/development/2018-April/032461.html (If you want to post, you must subscribe to the Development mailing list
• https://bugreports.qt.io/browse/QTIFW-975 (you can post using your Qt Account)