Download web pages with QTcpSocket

Herder

Hello.
I want to download web pages with QTcpSocket as in http://www.bogotobogo.com/cplusplus/sockets_server_client_QT.php (QTcpSocket - Signal/Slot)
My code is as follow

// main.cpp

#include <QCoreApplication>
#include "sockettest.h"

int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);

SocketTest mTest;
mTest.Test();

return a.exec();

}

Header file:

// sockettest.h

#ifndef SOCKETTEST_H
#define SOCKETTEST_H

#include <QObject>
#include <QDebug>
#include <QTcpSocket>
#include <QAbstractSocket>

class SocketTest : public QObject
{
Q_OBJECT
public:
explicit SocketTest(QObject *parent = 0);
void Test();

signals:

public slots:

void connected();
void disconnected();
void bytesWritten(qint64 bytes);
void readyRead();

private:
QTcpSocket *socket;

};
#endif // SOCKETTEST_H

Then, the implementation file:

// SocketTest.cpp

#include "sockettest.h"

SocketTest::SocketTest(QObject *parent) :
QObject(parent)
{
}

void SocketTest::Test()
{
socket = new QTcpSocket(this);
connect(socket, SIGNAL(connected()), this, SLOT(connected()));
connect(socket, SIGNAL(disconnected()), this, SLOT(disconnected()));
connect(socket, SIGNAL(readyRead()), this, SLOT(readyRead()));
connect(socket, SIGNAL(bytesWritten(qint64)), this, SLOT(bytesWritten(qint64)));

qDebug() << "Connecting,..";

socket->connectToHost("bogotobogo.com", 80);

if(!socket->waitForDisconnected(1000))
{
    qDebug() << "Error: " << socket->errorString();
}

}

void SocketTest::connected()
{
qDebug() << "Connected!";

socket->write("HEAD / HTTP/1.0\r\n\r\n\r\n\r\n");

}

void SocketTest::disconnected()
{
qDebug() << "Disconnected!";
}

void SocketTest::bytesWritten(qint64 bytes)
{
qDebug() << "We wrote: " << bytes;
}

void SocketTest::readyRead()
{
qDebug() << "Reading...";
qDebug() << socket->readAll();
}

Author of this example has a result "HTTP/1.1 200 OK ...", but I got "HTTP/1.1 400 Bad Request ..." And I can't get HTTP 200 for any address at all. What did I wrong?

Herder

I changed this code as in
http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet

Now I write
QString requestString ="GET /cplusplus/sockets_server_client_QT.php HTTP/1.1\r\nhost: bogotobogo.com\r\n\r\n";
socket->write(requestString.toStdString().c_str());

And it works.

But if i use
socket->connectToHost("blikoon.com", 80);
and
QString requestString ="GET /networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet HTTP/1.1\r\nhost: blikoon.com\r\n\r\n";
socket->write(requestString.toStdString().c_str())

I get
"HTTP/1.1 301 Moved Permanently\r\nServer: nginx/1.4.6 (Ubuntu)\r\nDate: Tue, 08 Nov 2016 09:55:39 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nX-Powered-By: PHP/5.5.9-1ubuntu4.20\r\nX-Pingback: http://www.blikoon.com/xmlrpc.php\r\nLocation: http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet\r\n\r\n0\r\n\r\n"

But http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet is the same i asked. What did I wrong?

Venkatesh V

Hai @Herder

Refer http example provided by Qt itself that may solve your problem.

Thanks

m.sue

Hi,
yes, I would not do this with low-level classes myself but use QNetworkAccessManager and its get() routine to download webpages, see here: http://doc.qt.io/qt-5/qnetworkaccessmanager.html#details
-Michael.

Herder

Thanks.
Yes, it is possible with QNetwoorkAccessManager. But it is need to work with http directly. So, I have to use QTcpSocket

SGaist

Hi and welcome to devnet,

What do you mean by "you need to work with http directly" ?

QNetworkAccessManager will generate such a request for you.

tham

I use QNetworkAccessManager to download more than 100000 web pages(http or https) already, the number keep increasing.it is very easy to use and suit for your task, unless you want to do some experiments by QTcpSocket, else I think QNetworkAccessManager is much easier to use.

A simple example, please tune it by yourself. Especially do not forget error handling.

auto *reply = manager_->get(QNetworkRequest("https://forum.qt.io/topic/73117/download-web-pages-with-qtcpsocket/6"));
        connect(reply, &QNetworkReply::finished, this, &auto_updater::update_local_info_finished);
        connect(reply, &QNetworkReply::downloadProgress,
                [this](qint64 bytesReceived, qint64 bytesTotal)
        {
            qDebug()<<QString("Download update_info : %1/%2").arg(bytesReceived).
                         arg(bytesTotal);
        });

void auto_updater::update_finished()
{
    //someone may say this is not OO, is a bad habit, but I think calling sender
    //in slot could make your codes much easier to read and maintain in this case
    auto *reply = qobject_cast<QNetworkReply*>(sender());
   if(reply){
      //open file, save to disk.

    reply->deleteLater(); 
   }
}

If you worry about exception, you can put reply to a simple resource guard class, or give unique_ptr a customize deleter.

template<typename T>
class delete_later
{
pulibc:
  explicit delete_later(T *ptr) :
    ptr_(ptr) {}

  ~delete_later() { ptr_->deleteLater(); }
private:
   T *ptr_;
}

I love RAII, a lot

Edit :

Remember to monitor your download progress by QTimer , sometimes the downloading progress may stuck forever(no error message at all), you need to restart the request manually if you find out the website cannot download after x seconds.