Qt DevNet Web Service API

mgran

Good stuff, and a QNAM login. You're on a roll ZapB :)

[quote author="ZapB" date="1302376861"]Oh another api request...would it be possible to have the url to the profile's avatar image returned in the profile call too please?[/quote]

I'll add it to the backlog, no room in the current iteration and filling up fast for the next.

ZapB

It was surprisingly easy - only 453 lines of code plus a little QML so far. If I get something useful going then maybe you'll see a new user-agent string showing up in the analytics soon ;-)

@request.setRawHeader( "User-Agent", "ZapB's QtDevNet Client 0.1" );@

When I finish the next stage I may have some more API requests as that will likely be the much easier option as opposed to scraping fragments from a complete web-page. It will result in a lighter load on the server too.

troubalex

Very nice! I like it. :)

andre

I have so far not been able to compile it yet. QJson is failing to compile at this moment... :-(

ZapB

Thank you. I'll keep adding little bits to it and post back again when it is usable for a little more than looking at your own points ;-)

@Andre, I am using qjson-0.7.1 which I installed via portage. I have not tried building it by hand.

andre

Managed to build qjson (from git), and with some adjustments to your code (in the .pro file and in the include of qjson you use), I even managed to build the app. Trying to run it instantly crashes though, so I'll try for a bit more...

ZapB

Impressive! ;-)

I've not tried at all on Windows yet. I'll see if I can get it workign there tonight. I do not know of any platform-specific things though. Maybe I forgot to initialise a pointer in a ctor or something.

andre

OK, got it working, it was a simple matter of the executable not being able to find the qjson library when running. Now all I have to do is figure out what the app actually does :-)

andre

The changes I had to make to make it compile for me:

In qtdevnet.pro, changed the path handling a bit
@
INCLUDEPATH += $$(QJSONROOT)/src
LIBS += -L$$(QJSONROOT)/lib -lqjson
@

In profile.cpp, I changed the include for qjson's parser.h:
@
#include <parser.h>
@

This way, I can actually make QJSONROOT point to where I downloaded and build QJson.

ZapB

The only thing you can do at present is to go to the "Application->Login..." and enter your QtDevNet credentials (not included support for OpenID yet) and it will login via QNAM (see the Authenticator class). One slight mystery I have not had time to look at yet is why QUrl::toPercentEncoding() will not encode a "/" character. I have to manually replace that with / later. I'll step into the Qt code later and take a look.

If that works it will start a timer inside the Profile class that polls QtDN for your profile stats once every 60 seconds. The response is parsed using qjson and the properties updated accordingly.

The Profile class is exposed to the QML root context and the qml scene just binds to the properties of it (and one property from the autheticator object which determines if the qml items are visible or not).

It's just a starter for 10/proof of concept at this stage. Nothing fancy. Having said that it will be simple to extend to cover other features that could be supported by a web API.

ZapB

Thanks for the mods. I'll patch my source for win support with it.

Edit: I have applied the patch this in svn now.

ZapB

Just a small update. The login dialog is now handled via QML too (the actual network stuff is still in C++ of course). It's in svn if anybody wants to grab the update. Here's a screenie (no bling-tastic animations yet though).

!http://gallery.theharmers.co.uk/upload/2011/04/22/20110422154652-693ce8de.png(QML Login Screen)!

ps I hope you don't mind me grabbing artistic assets from the site for use in this client. ;-)

mgran

Sweet ZapB, I'll try out the updates once I get back from the Easter break and to a proper pc :)

ZapB

OK another hour of playing with QML and the login page now behaves more like you would expect, including things like:

Enabling/disabling login button based on user name and password fields being populated

Visible feedback on login button status (enabled/disabled has focus/no focus)

Tab key navigation between elements

Plus it has some bling now - you'll have to run it to see it. ;-)

My opinion of QML so far is that it is very nice, but it will be a whole let better once we have one or more decent sets of components to use (desktop components would be really cool - and yes I have seen the blog with them in). It is instructional to develop your own from first principals but requires lots of digging in the docs for QML noobs like me.

Next step is to see if I can get this converted into a mobile app and try in on my N8. Now where did I put that SDK...

ZapB

Sweet! I just managed to get this running on my N8. It's very awesome to see the same QML animations running on the phone - smoothly too! :D

It took a little bit of trial and error and I am still not quite sure what made it start to deploy properly and I can still not get qjson to build for the qt-simulator target. I'll write up some instructions on how to do this and commit my mods to the svn repo shortly.

Edit 1: Code to allow this to be built and deployed for Symbian is now in svn.

Edit 2: Figured out what was stopping me from running this on the device originally. I kept getting the "General OS Error" when trying to deploy and run my app. It turns out that it was because of insufficient capabilities in the qjson lib on which my app depends. By changing the TARGET.CAPABILITY line in qjson/src/src.pro to:

@
TARGET.CAPABILITY = NetworkServices ReadDeviceData WriteDeviceData
@

allows my app to load the qjson dll on the device (since my app needs access to the Internet). Read about "capabilities":http://wiki.forum.nokia.com/index.php/Capabilities for more info on this.

I have concerns about how this may scale if several apps want to use the same dll but with different capabilities but that's something to worry about for another day.

ZapB

Marius, I wonder if you could help me with something please? I am trying to add avatar support to my little QtDN client and until the API supports returning the avatar image URL I thought I would take the opportunity to try using QXmlQuery to parse the xhtml of the front page of QtDN to extract the gravatar source url.

This is all going fine until I get this error from QXmlQuery:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 74, column 32: Expected '=', but got '>'.
@

Looking at the page source I see that line 74 is:

@
<div class="wrapper" {set_mode}>
@

which of course is not valid xhtml (or xml) as advertised by the doctype. I can work around this by stripping this out before I pass it onto the QXmlQuery of course but would you like me to report this as a bug? Or even better is there an easy fix you can apply to get rid of the above non-valid entry please?

Many thanks.

ZapB

Another piece of invalid xml is found on line 190:

@
<div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div>
@

The error is:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch.
@

The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag.

Edit: This input tag on line 532 is also missing a closing "/"

@
<input type="text" name="tag_search" id="tag_search_input" value="">
@

ZapB

I found one other issue that prevents the main page from validating as well-formed xml. The javascript at the bottom of the page (starting on line 710):

@
function searchStart()
...
});
@

really needs to be wrapped in a CDATA section like this:

@
<![CDATA[
function searchStart()
...
});
]]>
@

I have put in place some hackish work-arounds to get over these non-conformances. Here is what I have done to scrape the gravatar image from the front page.

In the Profile::update() function (which is where I usually use QNAM to query the profile API) I also do this on the first call after a successful login:

@
static bool firstTime = true;
if ( firstTime )
{
// Load the front page so that we can scrape the avatar
firstTime = false;
QNetworkRequest avatarRequest;
avatarRequest.setUrl( QUrl( "http://developer.qt.nokia.com/" ) );
avatarRequest.setRawHeader( "User-Agent", "ZapBs QtDevNet Client 0.1" );

    QNetworkReply* avatarReply = m_nam->get( avatarRequest );
    connect( avatarReply, SIGNAL( finished() ),
             SLOT( _q_extractAvatar() ) );
    connect( avatarReply, SIGNAL( error( QNetworkReply::NetworkError ) ),
             SLOT( _q_onReplyError( QNetworkReply::NetworkError ) ) );
}

@

This just uses QNAM to fetch the xhtml document for the QtDN front page.

Edit: Had to split post as it was too large.

ZapB

When the request has finished we go into the private slot _q_extractAvatar(). This does a few things:

Checks that we have a real QNetworkReply.

Reads the entire xhtml document from the reply, and schedules the reply for deletion.

Performs a few hacky replacements on the document to work around xml validation errors.

Uses QXmlQuery to find the appropriate img element for the avatar image, extracts the src attribute and stores this as an attribute of a trivial xml document as the output.

Uses some simple manipulations to get the actual avatar source url

Replaces the size from 48px which is used on the front page to 80px which I use in the client.

Emits the avatarUrlChanged() signal to nudge the QML component into action

Here's the code that does it:

@
void Profile::_q_extractAvatar()
{
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

// Hack around broken xhtml on QtDevNet front page
QByteArray input( reply->readAll() );
reply->deleteLater(); // Delete the reply when we go back to the event loop
input.replace( "{set_mode}",
               "" );
input.replace( "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png">",
               "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png" />" );
input.replace( "&lt;input type="text" name="tag_search" id="tag_search_input" value=""&gt;",
               "&lt;input type="text" name="tag_search" id="tag_search_input" value="" /&gt;" );
input.replace( "function searchStart",
               "&lt;![CDATA[function searchStart" );
input.replace( "});n[removed]",
               "});n]]>[removed]" );

QBuffer device;
device.setBuffer( &input );
device.open( QIODevice::ReadOnly );

// Our XQuery string
QString queryString( "declare default element namespace "http://www.w3.org/1999/xhtml";n"
                     "declare variable $inputDocument external;n"
                     "doc($inputDocument)//div[@class="frontpage-profile-image"]/a/img/n"
                     "<p>{@src}</p>" );

// Prepare the xml query object
QXmlQuery query;
query.bindVariable( "inputDocument", &device );
query.setQuery( queryString );
if ( !query.isValid() )
    return;

// We need a buffer to write the output to
QByteArray outArray;
QBuffer buffer( &outArray );
buffer.open( QIODevice::ReadWrite );

// Set up a formatter and execute the query
QXmlFormatter formatter( query, &buffer );
if ( !query.evaluateTo( &formatter ) )
    return;
buffer.close();
//qDebug() << "Query output:" << outArray.constData();

// Now get the actual source url that we need
QString needle( "src="" );
int index = outArray.indexOf( needle );
if ( index != -1 )
{
    QByteArray avatarSource = outArray.mid( index + needle.length() );
    int idx = avatarSource.indexOf( """ );
    if ( idx != -1 )
    {
        avatarSource = avatarSource.mid( 0, idx );
        avatarSource.replace( "size=48", "size=80" );
        m_avatarUrl.setUrl( QString::fromLatin1( avatarSource ) );
        emit avatarUrlChanged();
    }
}

}
@

I know I could have replaced the QXmlQuery stuff with a simple text search but I think this way is more robust. Plus it was a chance for me to use the XmlPatterns module which is still very new to me (which probably shows in my XQuery string).

ZapB

Marius, it might also be nice to have the memberId returned as part of the profile query in the restful API. ie for my account it would include a json attribute of "memberId" : "3246".

So overall it would be nice if the profile json response looked like this:

@
{
"memberId" : "3246",
"gravatarHash" : "d4737278a208398242bdf2535701f8a3",
"current_rank":
{
"points":"494",
"points_last_7_days":"296",
"level":2,
"title":"Ant Farmer",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level2.png"
},
"next_rank":
{
"points_remaining":47,
"level":3,
"title":"Hobby Entomologist",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level3.png"
},
"badges": [
{
"name":"Nokia Certified Qt Developer",
"url":"http://developer.qt.nokia.com/images/qtdn/icon_certified_developer.png"
}]
}
@

By returning the gravatar hash rather than the actual URL, clients could construct their own url for custom sizes and default image fallbacks more easily.