Qt web requests

cpper

Hello :)
I decided it's time to learn networking in qt, and am now trying to log into a forum/online game/whatever using qt. I barely know anything about networking in general, so I guess I will have a hard time. From what I've understand, I need to use QNetworkCookieJar and QNetworkRequest for this. My question is, how do I retrieve the request information from a webpage('s login button), in order to write in into the qt code ?
Take as example this forum, do I need to go to the login page, and find some specific data in the page source, like the action that is triggerd when pressing the login buttom, and then use it in the qt program to make the request ? Or how is it done ?

Chris Hennes

When you say you want to log into a forum, what is the end result you want your app to do with that log in? The way a site like this works is that the login page transmits your login information via an HTTP POST request to a specific script on the server. The server validates that information and sets a cookie on your machine with the information it needs to maintain that login. How exactly it works will vary from site to site, but that's the general process.

cpper

Thanks Chris. I actually want to log into an online browser game and get some data from the account, which is displayed in the home page after logging in. What I'm asking is how can I get the information from the website needed to make the post request in qt. I researched a bit more and think I have to take a look into chromes Network tab from Developer Options(the inspect element window)

SGaist

Hi,

Did you check whether that site provides an API for that kind of stuff ?

cpper

Yep, there's no thing like that.

Chris Hennes

You can look at the Network tab in Chrome, but it's actually probably easier to simply examine the source code of the login page and look for the URL that's being submitted to. Then look at the html form elements that are being used to create the POST request and you'll have your data. You are then going to be reduced to writing a screen-scraper, unfortunately, since without an API all you can do is read whatever the server sends you and hope they don't change the page too often.

cpper

I managed to log in by creating a request with data obtained from network tab in chrome.
This is the flow:
Firstly I go to the login page using get function. From the obtained reply's header I get a sessionId(from Set-Cookie) . With that sessionId and other data obtained from studying the log in request in chrome, I make a post request and log in to the game. The operation is successful, because providing an incorrect user name/password does not return a specific Location header. Now again, with data from the previous post reply's cookie header I make a get request(just as chrome shows in the network tab) to download the html page. But here comes a problem, when I'm doing qDebug()<<QString(reply->readAll()); , instead of getting the html source code I get the following line : "\u001F?\b" . The respone header of the get request looks like the one in chrome, I don't know what goes wrong.

Here is my code. I know it looks very bad but I want to keep everything in one single function until I get it to work.:

{
    ui->setupUi(this);

   QNetworkAccessManager* qnam=new QNetworkAccessManager(this);
   QNetworkReply* reply;

   QEventLoop loop;
   connect(qnam,&QNetworkAccessManager::finished,&loop,&QEventLoop::quit);

   reply=qnam->get(QNetworkRequest(QUrl("http://tx3.travian.de")));
   loop.exec();

   QString lognum(reply->readAll());
   lognum=lognum.mid(lognum.indexOf("						<input type=\"hidden\" name=\"login\" value=\"")+47,10);    // i need this value from the log in page source code for the query params, see below
   QByteArray sid(reply->rawHeader("Set-Cookie"));       //sessionid


qDebug()<<"-------------------------Response Header of log in page reply---------------------------------------";
       for(QNetworkReply::RawHeaderPair x: reply->rawHeaderPairs()){
            qDebug()<<QString(x.first)<<"       "<<QString(x.second);
       }


    QUrlQuery params;
    QNetworkRequest req(QUrl("http://tx3.travian.de/dorf1.php"));


    req.setRawHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    req.setRawHeader("Accept-Encoding","gzip, deflate");
    req.setRawHeader("Accept-Language","en-US,en;q=0.8,de;q=0.6,ro;q=0.4,fr;q=0.2");
    req.setRawHeader("Cache-Control","no-cache");
    req.setRawHeader("Cookie",sid);
    req.setRawHeader("Connection","keep-alive");
    req.setRawHeader("Content-Length","74");
    req.setRawHeader("Content-Type","application/x-www-form-urlencoded");
    req.setRawHeader("Host","tx3.travian.de");
    req.setRawHeader("Origin","http://tx3.travian.de");
    req.setRawHeader("Pragma","no-cache");
    req.setRawHeader("Referer","http://tx3.travian.de/");
    req.setRawHeader("Upgrade-Insecure-Requests","1");
    req.setRawHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36");

    params.addQueryItem("name", "username");
    params.addQueryItem("password", "passw");
    params.addQueryItem("s1", "Einloggen");
    params.addQueryItem("w","1366%3A768");
    params.addQueryItem("login", lognum);       //here
    params.addQueryItem("lowRes", "0");

    reply=qnam->post(req, params.toString(QUrl::FullyEncoded).toUtf8());
    loop.exec();


qDebug()<<"-------------------------Response Header of logged in page POST reply ---------------------------------------";
    for(QNetworkReply::RawHeaderPair x: reply->rawHeaderPairs()){
         qDebug()<<QString(x.first)<<"       "<<QString(x.second);
    }


   QNetworkRequest wtr(QUrl("http://tx3.travian.de/dorf1.php"));


   QString getCookie=QString(reply->rawHeader("Set-Cookie"));
   getCookie.remove("path=/; httponly");
   getCookie.remove("\n");
   getCookie.push_front(sid+"; ");


    wtr.setRawHeader("Accept","text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
    wtr.setRawHeader("Accept-Encoding","gzip, deflate, sdch");
    wtr.setRawHeader("Accept-Language","en-US,en;q=0.8,de;q=0.6,ro;q=0.4,fr;q=0.2");
    wtr.setRawHeader("Cache-Control","no-cache");
    wtr.setRawHeader("Connection","keep-alive");
    wtr.setRawHeader("Cookie",getCookie.toUtf8());
    wtr.setRawHeader("Host","tx3.travian.de");
    wtr.setRawHeader("Pragma","no-cache");
    wtr.setRawHeader("Referer","http://tx3.travian.de/");
    wtr.setRawHeader("Upgrade-Insecure-Requests","1");
    wtr.setRawHeader("User-Agent","Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36");


    reply=qnam->get(wtr);
    loop.exec();

qDebug()<<"-------------------------Response Header of logged in page GET reply ---------------------------------------";
        for(QNetworkReply::RawHeaderPair x: reply->rawHeaderPairs()){
             qDebug()<<QString(x.first)<<"       "<<QString(x.second);
        }



        qDebug()<<QString(reply->readAll());   // here , strange result
}

cpper

The status code of the final get request is 200, and the error in NoError. There's no redirection also... I 'm still trying to find out what's wrong with the reply I get.