[solved] Extracting information from a password protected web page.

[ThaRez]

Hello
I'd like to make a small program that does the following:

connects to a password protected site (normal web login, similar to the one found on this forum). The software then provides the password to the server and loads the source of the protected web page (dynamically created with by server php script). The software then extracts information form the page, and according to that submits a form on the page, again to process the reply and then finally log out.

The goal is thereby for the software to extract information and notify the user once such update information is available without the user having to manually log in, search and look for updated information on regular basis.

I've looked into TCP sockets and QHttp but haven't found any information about how to handle the encryption, nor how to submit a form...

Atm I'm stuck, not knowing what to look for, so any clues would be appreciated. Also, in case this can't be done, or there are very hard challenges, I'd also appreciate to know this.

Thank you! Best regards
Richard Salin

[reactive]

Your post is a bit confusing because you say you need to "handle the encryption" but it sounds like you just want to log in to a regular website and then send posts. To log in at most sites you send them the username and password and they respond with a sessionID that you must set in a cookie. You then have to use that cookie for the rest of the session. Look at QNetworkAccessManager and QNetworkRequest I think. (Sorry, I've done this in Java, but not in Qt). Good luck.

[ThaRez]

With a second glance, I fully agree with you, it's a bit confusing...

With encryption I meant pages transmitted over a https connection. I'n my case, I'd like to make this work both over an insecure and a secure connection.

Anyway, your post sounded very helpful. Is it so that most sites keep track of the log in via cookies, or is it sufficient that you log in once and keep the TCP connection open?

Thank
Richard

[ZapB]

You should look at QNAM, QNetworkRequest, and QNetworkReply.

To log in to the site issue a POST request. For example to login to this site I use the following code:

@
#ifndef AUTHENTICATOR_H
#define AUTHENTICATOR_H

#include <QObject>
#include <QNetworkReply>

class QNetworkAccessManager;

class Authenticator : public QObject
{
Q_OBJECT
Q_PROPERTY( bool isLoggedIn READ isLoggedIn NOTIFY isLoggedInChanged )
public:
explicit Authenticator( QNetworkAccessManager* nam, QObject* parent = 0 );

bool isLoggedIn() const { return m_isLoggedIn; }

public slots:
void login( const QString& userName, const QString& password );

signals:
void isLoggedInChanged();
void loggedInAs( const QString& userName );
void loginFailed( const QString& reason );

private slots:
void _q_onLoginRequestFinished();
void _q_onLoginMetaDataChanged();
void _q_onLoginRequestError( QNetworkReply::NetworkError error );

private:
QNetworkAccessManager* m_nam;
bool m_isLoggedIn;
QString m_userName;
};

#endif // AUTHENTICATOR_H
@

and the implementation:

@
#include "authenticator.h"

#include <QNetworkAccessManager>
#include <QNetworkRequest>

#include <QDebug>

Authenticator::Authenticator( QNetworkAccessManager* nam, QObject* parent )
: QObject( parent ),
m_nam( nam ),
m_isLoggedIn( false ),
m_userName()
{
}

void Authenticator::login( const QString& userName, const QString& password )
{
qDebug() << "Attempting to login with Username =" << userName << "and password =" << password;
m_userName = userName;

QNetworkRequest request;
request.setUrl( QUrl( "http://developer.qt.nokia.com/" ) );
request.setRawHeader( "User-Agent", "ZapB's QtDevNet Client 0.1" );
request.setHeader( QNetworkRequest::ContentTypeHeader,"application/x-www-form-urlencoded" );

// Construct a suitable byte array for the form fields and values
QUrl params;
params.addEncodedQueryItem( "ACT", QUrl::toPercentEncoding( "11" ) );
params.addEncodedQueryItem( "RET", QUrl::toPercentEncoding( "/" ) );
params.addEncodedQueryItem( "site_id", QUrl::toPercentEncoding( "1" ) );
params.addEncodedQueryItem( "username", QUrl::toPercentEncoding( userName ) );
params.addEncodedQueryItem( "password", QUrl::toPercentEncoding( password ) );
//params.addEncodedQueryItem( "openid", QUrl::toPercentEncoding( "" ) );
//params.addEncodedQueryItem( "auto_login", QUrl::toPercentEncoding( "1" ) );

// The encoded string contains a ? at the beginning, strip it off
QByteArray paramBytes = params.toString().mid( 1 ).toLatin1();

/** \bug For some reason QUrl::toPercentEncoding() does not encode "/" as "/" */
paramBytes.replace( "/", "/" );
//qDebug() << "Form data =" << paramBytes;

QNetworkReply* reply = m_nam->post( request, paramBytes );
connect( reply, SIGNAL( finished() ),
         SLOT( _q_onLoginRequestFinished() ) );
connect( reply, SIGNAL( metaDataChanged() ),
         SLOT( _q_onLoginMetaDataChanged() ) );
connect( reply, SIGNAL( error( QNetworkReply::NetworkError ) ),
         SLOT( _q_onLoginRequestError( QNetworkReply::NetworkError ) ) );

}

void Authenticator::_q_onLoginRequestFinished()
{
qDebug() << Q_FUNC_INFO;
QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
return;

QByteArray ba = reply->readAll();
if ( ba.isEmpty() )
{
    emit loggedInAs( m_userName );
    m_isLoggedIn = true;
    emit isLoggedInChanged();
}

}

void Authenticator::_q_onLoginMetaDataChanged()
{
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

QVariant statusCode = reply->attribute( QNetworkRequest::HttpStatusCodeAttribute );
if ( !statusCode.isValid() )
    return;

int status = statusCode.toInt();

if ( status != 302 )
{
    QString reason = reply->attribute( QNetworkRequest::HttpReasonPhraseAttribute ).toString();
    qDebug() << tr( "Loging request failed: %1." ).arg( reason );
    emit loginFailed( reason );
}

}

void Authenticator::_q_onLoginRequestError( QNetworkReply::NetworkError error )
{
Q_UNUSED( error );
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

qDebug() << "Error when logging in:" << reply->errorString();

}
@

Then once you are authenticated you can just create a normal GET request using your https based URL.

[ThaRez]

Thank you! These suggestion provides me a good starting point. I'll look in to the information you suggested!
Br
Richard

[Mr.FreakyJo]

Helped me too,but I don't undestand them to well.I'm new to Qt(read only one manual about Qt 4) and I don't know too much of the new versions.If you could explain me how this work it would be a huge help.A manual,tutorial,book would also help.I'm trying to login on http://www.triburile.ro/

[ZapB]

It's all there in the above code sample. It just creates a QNetworkRequest for the url to which you wish to send the request. The request is then sent to the server via the QNetworkAccessManager::post() function along with a QByteArray containing the filled in form data in a percent-encoded format.

We then connect up to various signals to process the reponses we may get back.

[Mr.FreakyJo]

I kind of understood this but that encoding works on all websites?I modified that code with the link of my website ,username and password.In params.addEncodedQueryItem( "username", ai modified username to user as it is in the source code of the page.In the debuger i get the error "Loging request failed: Moved Permanently." .Thank you for your help.

[ZapB]

It depends on what they use for the form. The above example uses the HTTP post method. What does your form look like? What URL is it, the error you are seeing looks like you have the wrong URL in your request?

[Mr.FreakyJo]

I'm trying to login on http://tribalwars.net . The source cod for those forms look like this :
<input id="user" name="user" class="text" type="text" value=""
onkeydown="if((e=window.event||event) && e.keyCode == 13) $('#login_form').trigger('submit');"/>
</span>
</label>
<label for="password">
<strong >Password:</strong>
<span >
<input name="clear" type="hidden" value="true" />
<input id="password" name="password" class="text" type="password"
onkeydown="if((e=window.event||event) && e.keyCode == 13) $('#login_form').trigger('submit');"/>
</span>
</label>

<div id="js_login_button">
<a href="#" class="login_button">
<span class="button_left"></span>
<span class="button_middle">Login</span>
<span class="button_right"></span>
</a>
</div>

[ZapB]

And can you show your C++ code you are trying to use please?

[Mr.FreakyJo]

It's your code with some modifications to the login function:

@void Authenticator::login( const QString& userName, const QString& password )
{
qDebug() << "Attempting to login with Username =" << userName << "and password =" << password;
m_userName = userName;

QNetworkRequest request;
request.setUrl( QUrl( "http://tribalwars.net/" ) );
request.setRawHeader( "User-Agent", " Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.77 Safari/535.7" );
request.setHeader( QNetworkRequest::ContentTypeHeader,"application/x-www-form-urlencoded" );

// Construct a suitable byte array for the form fields and values
QUrl params;
params.addEncodedQueryItem( "ACT", QUrl::toPercentEncoding( "11" ) );
params.addEncodedQueryItem( "RET", QUrl::toPercentEncoding( "/" ) );
params.addEncodedQueryItem( "site_id", QUrl::toPercentEncoding( "1" ) );
params.addEncodedQueryItem( "user", QUrl::toPercentEncoding( userName ) );
params.addEncodedQueryItem( "password", QUrl::toPercentEncoding( password ) );
//params.addEncodedQueryItem( "openid", QUrl::toPercentEncoding( "" ) );
//params.addEncodedQueryItem( "auto_login", QUrl::toPercentEncoding( "1" ) );

// The encoded string contains a ? at the beginning, strip it off
QByteArray paramBytes = params.toString().mid( 1 ).toLatin1();

/** \bug For some reason QUrl::toPercentEncoding() does not encode "/" as "/" */
paramBytes.replace( "/", "/" );
//qDebug() << "Form data =" << paramBytes;

QNetworkReply* reply = m_nam->post( request, paramBytes );
connect( reply, SIGNAL( finished() ),
         SLOT( _q_onLoginRequestFinished() ) );
connect( reply, SIGNAL( metaDataChanged() ),
         SLOT( _q_onLoginMetaDataChanged() ) );
connect( reply, SIGNAL( error( QNetworkReply::NetworkError ) ),
         SLOT( _q_onLoginRequestError( QNetworkReply::NetworkError ) ) );

}@

And in main where I test it i called the function like this
tribalwars->login(QString("mr.freakyjo"),QString("password"));
tribalwars being an "Authenticator *" object.

[Mr.FreakyJo]

I think i can log in with a MD5 hash of a string consisting of the variables “sid”, “username” and “secret” together. e.g. $hash = md5($sid . $username . $secret_password); .How can I do this in QT?Thank you for your help.

[Mr.FreakyJo]

Modified the code and now I can login.A link to a more detailed manual for the webkit and QtNetwork or for a book would be a huge help now.Thank you ZapB for helping me!

[Mr.FreakyJo]

After monitoring http request with the help off google chrome I see that a http post request is made by a script.I see that it’s initiator is game.js:17 (a javascript).Can I make a http request to that script or how can I do it?Is it necessary to use XMLHttpRequest?The scripts takes a user and a password and returns a form and there is some information in that form that I need to login.