It might be sleeping if for example it's an heroku free tier that hasn't been used for some times. Therefore the first query will have to wait for the server to awaken thus it will be longer.
It might require authentication which takes some time on first query to setup and allow to process.
It might need to query some slow to start resource to answer your question
Oh, I see. Well it is not a my server, it is a thirdy party server, so I can't answer to all that question. All I know is:
It is hosted on akamai.com
Acqtually it require some authentication data that I give with POST data
So I think I can't optimize requests server side, all can I do is caching DNS/TCP/SSL requests but I don't know if Qt 5.11.2 actually implements all that suggestions