Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Download web pages with QTcpSocket
Forum Updated to NodeBB v4.3 + New Features

Download web pages with QTcpSocket

Scheduled Pinned Locked Moved Unsolved General and Desktop
7 Posts 5 Posters 2.9k Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • H Offline
    H Offline
    Herder
    wrote on last edited by
    #1

    Hello.
    I want to download web pages with QTcpSocket as in http://www.bogotobogo.com/cplusplus/sockets_server_client_QT.php (QTcpSocket - Signal/Slot)
    My code is as follow

    // main.cpp

    #include <QCoreApplication>
    #include "sockettest.h"

    int main(int argc, char *argv[])
    {
    QCoreApplication a(argc, argv);

    SocketTest mTest;
    mTest.Test();
    
    return a.exec();
    

    }

    Header file:

    // sockettest.h

    #ifndef SOCKETTEST_H
    #define SOCKETTEST_H

    #include <QObject>
    #include <QDebug>
    #include <QTcpSocket>
    #include <QAbstractSocket>

    class SocketTest : public QObject
    {
    Q_OBJECT
    public:
    explicit SocketTest(QObject *parent = 0);
    void Test();

    signals:

    public slots:

    void connected();
    void disconnected();
    void bytesWritten(qint64 bytes);
    void readyRead();
    

    private:
    QTcpSocket *socket;

    };
    #endif // SOCKETTEST_H

    Then, the implementation file:

    // SocketTest.cpp

    #include "sockettest.h"

    SocketTest::SocketTest(QObject *parent) :
    QObject(parent)
    {
    }

    void SocketTest::Test()
    {
    socket = new QTcpSocket(this);
    connect(socket, SIGNAL(connected()), this, SLOT(connected()));
    connect(socket, SIGNAL(disconnected()), this, SLOT(disconnected()));
    connect(socket, SIGNAL(readyRead()), this, SLOT(readyRead()));
    connect(socket, SIGNAL(bytesWritten(qint64)), this, SLOT(bytesWritten(qint64)));

    qDebug() << "Connecting,..";
    
    socket->connectToHost("bogotobogo.com", 80);
    
    if(!socket->waitForDisconnected(1000))
    {
        qDebug() << "Error: " << socket->errorString();
    }
    

    }

    void SocketTest::connected()
    {
    qDebug() << "Connected!";

    socket->write("HEAD / HTTP/1.0\r\n\r\n\r\n\r\n");
    

    }

    void SocketTest::disconnected()
    {
    qDebug() << "Disconnected!";
    }

    void SocketTest::bytesWritten(qint64 bytes)
    {
    qDebug() << "We wrote: " << bytes;
    }

    void SocketTest::readyRead()
    {
    qDebug() << "Reading...";
    qDebug() << socket->readAll();
    }

    Author of this example has a result "HTTP/1.1 200 OK ...", but I got "HTTP/1.1 400 Bad Request ..." And I can't get HTTP 200 for any address at all. What did I wrong?

    1 Reply Last reply
    0
    • H Offline
      H Offline
      Herder
      wrote on last edited by
      #2

      I changed this code as in
      http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet

      Now I write
      QString requestString ="GET /cplusplus/sockets_server_client_QT.php HTTP/1.1\r\nhost: bogotobogo.com\r\n\r\n";
      socket->write(requestString.toStdString().c_str());

      And it works.

      But if i use
      socket->connectToHost("blikoon.com", 80);
      and
      QString requestString ="GET /networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet HTTP/1.1\r\nhost: blikoon.com\r\n\r\n";
      socket->write(requestString.toStdString().c_str())

      I get
      "HTTP/1.1 301 Moved Permanently\r\nServer: nginx/1.4.6 (Ubuntu)\r\nDate: Tue, 08 Nov 2016 09:55:39 GMT\r\nContent-Type: text/html; charset=UTF-8\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nX-Powered-By: PHP/5.5.9-1ubuntu4.20\r\nX-Pingback: http://www.blikoon.com/xmlrpc.php\r\nLocation: http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet\r\n\r\n0\r\n\r\n"

      But http://www.blikoon.com/networking/http-potocol-writting-a-simple-client-using-qt-qtcpsocket-and-troubleshooting-using-telnet is the same i asked. What did I wrong?

      1 Reply Last reply
      0
      • Venkatesh VV Offline
        Venkatesh VV Offline
        Venkatesh V
        wrote on last edited by
        #3

        Hai @Herder

        Refer http example provided by Qt itself that may solve your problem.

        Thanks

        1 Reply Last reply
        1
        • m.sueM Offline
          m.sueM Offline
          m.sue
          wrote on last edited by
          #4

          Hi,
          yes, I would not do this with low-level classes myself but use QNetworkAccessManager and its get() routine to download webpages, see here: http://doc.qt.io/qt-5/qnetworkaccessmanager.html#details
          -Michael.

          1 Reply Last reply
          2
          • H Offline
            H Offline
            Herder
            wrote on last edited by
            #5

            Thanks.
            Yes, it is possible with QNetwoorkAccessManager. But it is need to work with http directly. So, I have to use QTcpSocket

            1 Reply Last reply
            0
            • SGaistS Offline
              SGaistS Offline
              SGaist
              Lifetime Qt Champion
              wrote on last edited by
              #6

              Hi and welcome to devnet,

              What do you mean by "you need to work with http directly" ?

              QNetworkAccessManager will generate such a request for you.

              Interested in AI ? www.idiap.ch
              Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

              1 Reply Last reply
              0
              • thamT Offline
                thamT Offline
                tham
                wrote on last edited by tham
                #7

                I use QNetworkAccessManager to download more than 100000 web pages(http or https) already, the number keep increasing.it is very easy to use and suit for your task, unless you want to do some experiments by QTcpSocket, else I think QNetworkAccessManager is much easier to use.

                A simple example, please tune it by yourself. Especially do not forget error handling.

                auto *reply = manager_->get(QNetworkRequest("https://forum.qt.io/topic/73117/download-web-pages-with-qtcpsocket/6"));
                        connect(reply, &QNetworkReply::finished, this, &auto_updater::update_local_info_finished);
                        connect(reply, &QNetworkReply::downloadProgress,
                                [this](qint64 bytesReceived, qint64 bytesTotal)
                        {
                            qDebug()<<QString("Download update_info : %1/%2").arg(bytesReceived).
                                         arg(bytesTotal);
                        });
                
                void auto_updater::update_finished()
                {
                    //someone may say this is not OO, is a bad habit, but I think calling sender
                    //in slot could make your codes much easier to read and maintain in this case
                    auto *reply = qobject_cast<QNetworkReply*>(sender());
                   if(reply){
                      //open file, save to disk.
                
                    reply->deleteLater(); 
                   }
                }
                

                If you worry about exception, you can put reply to a simple resource guard class, or give unique_ptr a customize deleter.

                template<typename T>
                class delete_later
                {
                pulibc:
                  explicit delete_later(T *ptr) :
                    ptr_(ptr) {}
                
                  ~delete_later() { ptr_->deleteLater(); }
                private:
                   T *ptr_;
                }
                

                I love RAII, a lot

                Edit :

                Remember to monitor your download progress by QTimer , sometimes the downloading progress may stuck forever(no error message at all), you need to restart the request manually if you find out the website cannot download after x seconds.

                1 Reply Last reply
                0

                • Login

                • Login or register to search.
                • First post
                  Last post
                0
                • Categories
                • Recent
                • Tags
                • Popular
                • Users
                • Groups
                • Search
                • Get Qt Extensions
                • Unsolved