Download data only working with some sites

[jcga]

Hi,

I wrote a basic internet data dowloader following the instructions given here:
https://wiki.qt.io/Download_Data_from_URL

The dowloader works for some sites, that is, it returns data identical to what I get if I ask my browser to display the source code on the corresponding page. But for other sites, it returns nothing. In both cases I get the following list of messages:

qt.network.ssl: QSslSocket: cannot resolve TLSv1_1_client_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_2_client_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_1_server_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_2_server_method
qt.network.ssl: QSslSocket: cannot resolve SSL_select_next_proto
qt.network.ssl: QSslSocket: cannot resolve SSL_CTX_set_next_proto_select_cb
qt.network.ssl: QSslSocket: cannot resolve SSL_get0_next_proto_negotiated

Any help is welcome.

jc

[Pablo J. Rogina]

@jcga could you please clarify if for the sites you cannot get data you're using HTTPS? and for the ones that you do received data are just HTTP?
You may want to add error checking to your fileDownloaded method to better understand what's going on; something like this:

void FileDownloader::fileDownloaded(QNetworkReply* pReply) {
if(reply->error())
    {
        qDebug() << "ERROR!";
        qDebug() << reply->errorString();
    }
    else
    {
     ...

In addition you may want to connect to the QNetworkAccessManager::sslErrors(QNetworkReply *reply, const QList<QSslError> &errors) signal to check if something happens during TLS handshake.

[aha_1980]

@jcga said in Download data only working with some sites:

qt.network.ssl: QSslSocket: cannot resolve TLSv1_1_client_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_2_client_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_1_server_method
qt.network.ssl: QSslSocket: cannot resolve TLSv1_2_server_method
qt.network.ssl: QSslSocket: cannot resolve SSL_select_next_proto
qt.network.ssl: QSslSocket: cannot resolve SSL_CTX_set_next_proto_select_cb
qt.network.ssl: QSslSocket: cannot resolve SSL_get0_next_proto_negotiated

To add to @Pablo-J-Rogina: It looks like your Qt has problems with the openssl library (problems resolving functions from that library). So downloading may only work for sites without encryption (HTTP).

[SGaist]

Hi,

What version of OpenSSL are you using ?
With their 1.1 version they broke API/ABI compatibility. Qt 5.10 will have a backend for OpenSSL >= 1.1. Currently you have to use a 1.0* release.

[jcga]

Thanks for your answers.

@Pablo J. Rogina
It seems that it works rather with "simple" pages than with more complex ones (perhaps related to the presence of javascript scripts?)

• it works with e.g. http://phlam.univ-lille1.fr or https://www.lcf.institutoptique.fr
• it does not work with http://www.lkb.upmc.fr/ or with http://www.aps.org

@SGaist
"What version of OpenSSL are you using ?" I don't know. How to find it?

[jcga]

@Pablo-J.-Rogina said in Download data only working with some sites:

You may want to add error checking to your fileDownloaded method to better understand what's going on

There is no error! But the QByteArray is empty.

[SGaist]

What OS are you on ?

[jcga]

@SGaist Windows 7 Pro

[SGaist]

Then where did you get OpenSSL from ?

[jcga]

@SGaist I don't know. As I said, I just wrote the program as indicated on the link (first message), run it and got this list of messages but I don't understand the relationship with SSL.

[SGaist]

If you don't know then its likely that you don't have it at all.

If your downloads at some point get redirected to an encrypted endpoint then you need to have OpenSSL in order to establish the connection.

[jcga]

@SGaist Well, I'm sorry but I'm not very used to network subtleties, but as my browser (chrome) can open these sites, I suppose that the protocol SSL is available on my computers. Is this wrong?

[SGaist]

Yes it is wrong. Chrome comes packed with everything it needs like your application should be when using encryption.

[jcga]

@SGaist OK, I made some progress. I found libraries 'libeay32.dll' and 'ssleay32.dll' in my Qt distribution. I copied them in the working dir of my project and the messages qt.network.ssl... etc disapeared. But now I'm getting an error "SSL handshake failed". Any suggestions ? Thanks a lot for your help.

[SGaist]

You should check the peer certificate: see here.

[Pablo J. Rogina]

@jcga said in Download data only working with some sites:

Any suggestions ?

I already suggested "connect to the QNetworkAccessManager::sslErrors(QNetworkReply *reply, const QList<QSslError> &errors) signal to check if something happens during TLS handshake."

[jcga]

@Pablo-J.-Rogina
Thanks for the suggestion. I have done the connection, but the program never goes into the corresponding slot, so suppose that the signal sslErrors() is never emitted.