Qt DevNet Web Service API

ZapB

Hi,

Does QtDN provide a webservice API of any sort? I am asking because I was wondering if it would be interesting to you all to have either a plugin for qt-creator and/or a stand-alone app that provides access to QtDevNet.Within qt-creator, this could be integrated into "Welcome" mode or indeed appear as it's own "Community/QtDN" mode.

I do not have any mockups or anything yet. I am just toying with the concept at present.

Thoughts/comments/flames?

dangelog

It would indeed be a good idea. :-)

ZapB

I am thinking of having updates (notifications of new posts, threads etc.) displayed directly in the app (or plugin) rather than having to wait for notification emails to arrive as one nice feature.

giesbert

There is an API, which is used by vass' chrome plugin.
Additionally, you have the rss feeds.

ZapB

The more I think about it, the more I think that this could be a good way to reach more people in the community - especially if done as a plugin to qt-creator. I know several users of qt-creator who do not really use QtDN. Such a plugin would provide them with a way in and provide greater visibility as well as providing a way of quickly asking for or providing help.

ZapB

Gerolf: Ah OK thanks for that. I'll have a look at Vass' plugin.

giesbert

The discussion about it was in "this thread":http://developer.qt.nokia.com/forums/viewthread/2224/
I think, the API was described in "JIRA":http://bugreports.qt.nokia.com/browse/QTWEBSITE-137

ZapB

Thanks Gerolf. I have just read both of those and have tried the API (using browser). Now to learn how to use QJson to parse the responses and get a proof of concept example working...

ZapB

Marius if you happen to read this, is there a way to login to QtDN without scraping the various fields from the login form? Can I just submit a post request using QNAM directly? What fields are required in the post request?

troubalex

[quote author="ZapB" date="1300203182"]Thanks Gerolf. I have just read both of those and have tried the API (using browser). Now to learn how to use QJson to parse the responses and get a proof of concept example working...[/quote]

I like the idea. :) I have been toying with the idea of some integration into the "KDE Social Desktop":http://techbase.kde.org/Projects/Social-Desktop resp. into the "Open Collaboration Services":http://www.freedesktop.org/wiki/Specifications/open-collaboration-services as well. It never really materialized though.

Currently, I am waiting for some proper documentation for the existing API from the developer who wrote it. As soon as I have that I'll make it available on the wiki. Maybe I should nag him a bit...

If you have suggestions for improvements while you play with the API, please let us know. :)

ZapB

Thank you for that. I'll look forward to seeing the docs. I've nto made much progress yet as I was offline last week (son's birthday and decorating home office). I'll post back with API extensions that would be useful as I come across them.

Thanks again!

ZapB

I have had a couple of hours ot play around with this today. Here is the result of my tinkering so far:
!http://gallery.theharmers.co.uk/upload/2011/04/09/20110409200932-876ce30f.png(QtDevNet Profile Details)!

So far my simple test harness just uses QNAM to login to QtDN and then uses the web api described "here":http://bugreports.qt.nokia.com/browse/QTWEBSITE-137 to query my profile details. The main widget is a declarative view in which I have a simple qml scene that binds to properties of a C++ Profile object.

Could I put in a request for an addition to the profile api please? It would be very useful if it also returned an entry for the total number of points in this rank. That is the total number of points required to go from Robot Herder to Dinosaur Breeder for example. This would allow me to calculate a percentage towards next level value and show that in a progress bar without the need to hard-code the rank point boundaries.

Next I will start looking at the Unread Forum Posts api and see if I can make another qml component for that. I will also add in support for profile badges. I'll try to find some time tomorrow to make this little code available to anybody who is interested.

Eventually I would like to release a little QML based QtDevNet app for Symbian etc.

ZapB

Oh another api request...would it be possible to have the url to the profile's avatar image returned in the profile call too please?

ZapB

If anybody is interested, you can now download my little test harness from my subversion repository. Just do:

@svn co https://svn.theharmers.co.uk/svn/codes/public/qtdevnetclient/trunk qtdevnetclient@

Then build as normal. This project requires qjson to be available. If the compiler cannot find it then export QJSONROOT with the path to where qjson is installed and re-run qmake.

The test harness app now polls QtDN every 60 seconds for an update to your profile which is then relfected in the QML scene.

For now it is just functional. I'll add some bling later ;-)

mgran

Good stuff, and a QNAM login. You're on a roll ZapB :)

[quote author="ZapB" date="1302376861"]Oh another api request...would it be possible to have the url to the profile's avatar image returned in the profile call too please?[/quote]

I'll add it to the backlog, no room in the current iteration and filling up fast for the next.

ZapB

It was surprisingly easy - only 453 lines of code plus a little QML so far. If I get something useful going then maybe you'll see a new user-agent string showing up in the analytics soon ;-)

@request.setRawHeader( "User-Agent", "ZapB's QtDevNet Client 0.1" );@

When I finish the next stage I may have some more API requests as that will likely be the much easier option as opposed to scraping fragments from a complete web-page. It will result in a lighter load on the server too.

troubalex

Very nice! I like it. :)

andre

I have so far not been able to compile it yet. QJson is failing to compile at this moment... :-(

ZapB

Thank you. I'll keep adding little bits to it and post back again when it is usable for a little more than looking at your own points ;-)

@Andre, I am using qjson-0.7.1 which I installed via portage. I have not tried building it by hand.

andre

Managed to build qjson (from git), and with some adjustments to your code (in the .pro file and in the include of qjson you use), I even managed to build the app. Trying to run it instantly crashes though, so I'll try for a bit more...

ZapB

Impressive! ;-)

I've not tried at all on Windows yet. I'll see if I can get it workign there tonight. I do not know of any platform-specific things though. Maybe I forgot to initialise a pointer in a ctor or something.

andre

OK, got it working, it was a simple matter of the executable not being able to find the qjson library when running. Now all I have to do is figure out what the app actually does :-)

andre

The changes I had to make to make it compile for me:

In qtdevnet.pro, changed the path handling a bit
@
INCLUDEPATH += $$(QJSONROOT)/src
LIBS += -L$$(QJSONROOT)/lib -lqjson
@

In profile.cpp, I changed the include for qjson's parser.h:
@
#include <parser.h>
@

This way, I can actually make QJSONROOT point to where I downloaded and build QJson.

ZapB

The only thing you can do at present is to go to the "Application->Login..." and enter your QtDevNet credentials (not included support for OpenID yet) and it will login via QNAM (see the Authenticator class). One slight mystery I have not had time to look at yet is why QUrl::toPercentEncoding() will not encode a "/" character. I have to manually replace that with / later. I'll step into the Qt code later and take a look.

If that works it will start a timer inside the Profile class that polls QtDN for your profile stats once every 60 seconds. The response is parsed using qjson and the properties updated accordingly.

The Profile class is exposed to the QML root context and the qml scene just binds to the properties of it (and one property from the autheticator object which determines if the qml items are visible or not).

It's just a starter for 10/proof of concept at this stage. Nothing fancy. Having said that it will be simple to extend to cover other features that could be supported by a web API.

ZapB

Thanks for the mods. I'll patch my source for win support with it.

Edit: I have applied the patch this in svn now.

ZapB

Just a small update. The login dialog is now handled via QML too (the actual network stuff is still in C++ of course). It's in svn if anybody wants to grab the update. Here's a screenie (no bling-tastic animations yet though).

!http://gallery.theharmers.co.uk/upload/2011/04/22/20110422154652-693ce8de.png(QML Login Screen)!

ps I hope you don't mind me grabbing artistic assets from the site for use in this client. ;-)

mgran

Sweet ZapB, I'll try out the updates once I get back from the Easter break and to a proper pc :)

ZapB

OK another hour of playing with QML and the login page now behaves more like you would expect, including things like:

Enabling/disabling login button based on user name and password fields being populated

Visible feedback on login button status (enabled/disabled has focus/no focus)

Tab key navigation between elements

Plus it has some bling now - you'll have to run it to see it. ;-)

My opinion of QML so far is that it is very nice, but it will be a whole let better once we have one or more decent sets of components to use (desktop components would be really cool - and yes I have seen the blog with them in). It is instructional to develop your own from first principals but requires lots of digging in the docs for QML noobs like me.

Next step is to see if I can get this converted into a mobile app and try in on my N8. Now where did I put that SDK...

ZapB

Sweet! I just managed to get this running on my N8. It's very awesome to see the same QML animations running on the phone - smoothly too! :D

It took a little bit of trial and error and I am still not quite sure what made it start to deploy properly and I can still not get qjson to build for the qt-simulator target. I'll write up some instructions on how to do this and commit my mods to the svn repo shortly.

Edit 1: Code to allow this to be built and deployed for Symbian is now in svn.

Edit 2: Figured out what was stopping me from running this on the device originally. I kept getting the "General OS Error" when trying to deploy and run my app. It turns out that it was because of insufficient capabilities in the qjson lib on which my app depends. By changing the TARGET.CAPABILITY line in qjson/src/src.pro to:

@
TARGET.CAPABILITY = NetworkServices ReadDeviceData WriteDeviceData
@

allows my app to load the qjson dll on the device (since my app needs access to the Internet). Read about "capabilities":http://wiki.forum.nokia.com/index.php/Capabilities for more info on this.

I have concerns about how this may scale if several apps want to use the same dll but with different capabilities but that's something to worry about for another day.

ZapB

Marius, I wonder if you could help me with something please? I am trying to add avatar support to my little QtDN client and until the API supports returning the avatar image URL I thought I would take the opportunity to try using QXmlQuery to parse the xhtml of the front page of QtDN to extract the gravatar source url.

This is all going fine until I get this error from QXmlQuery:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 74, column 32: Expected '=', but got '>'.
@

Looking at the page source I see that line 74 is:

@
<div class="wrapper" {set_mode}>
@

which of course is not valid xhtml (or xml) as advertised by the doctype. I can work around this by stripping this out before I pass it onto the QXmlQuery of course but would you like me to report this as a bug? Or even better is there an easy fix you can apply to get rid of the above non-valid entry please?

Many thanks.

ZapB

Another piece of invalid xml is found on line 190:

@
<div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div>
@

The error is:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch.
@

The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag.

Edit: This input tag on line 532 is also missing a closing "/"

@
<input type="text" name="tag_search" id="tag_search_input" value="">
@

ZapB

I found one other issue that prevents the main page from validating as well-formed xml. The javascript at the bottom of the page (starting on line 710):

@
function searchStart()
...
});
@

really needs to be wrapped in a CDATA section like this:

@
<![CDATA[
function searchStart()
...
});
]]>
@

I have put in place some hackish work-arounds to get over these non-conformances. Here is what I have done to scrape the gravatar image from the front page.

In the Profile::update() function (which is where I usually use QNAM to query the profile API) I also do this on the first call after a successful login:

@
static bool firstTime = true;
if ( firstTime )
{
// Load the front page so that we can scrape the avatar
firstTime = false;
QNetworkRequest avatarRequest;
avatarRequest.setUrl( QUrl( "http://developer.qt.nokia.com/" ) );
avatarRequest.setRawHeader( "User-Agent", "ZapBs QtDevNet Client 0.1" );

    QNetworkReply* avatarReply = m_nam->get( avatarRequest );
    connect( avatarReply, SIGNAL( finished() ),
             SLOT( _q_extractAvatar() ) );
    connect( avatarReply, SIGNAL( error( QNetworkReply::NetworkError ) ),
             SLOT( _q_onReplyError( QNetworkReply::NetworkError ) ) );
}

@

This just uses QNAM to fetch the xhtml document for the QtDN front page.

Edit: Had to split post as it was too large.

ZapB

When the request has finished we go into the private slot _q_extractAvatar(). This does a few things:

Checks that we have a real QNetworkReply.

Reads the entire xhtml document from the reply, and schedules the reply for deletion.

Performs a few hacky replacements on the document to work around xml validation errors.

Uses QXmlQuery to find the appropriate img element for the avatar image, extracts the src attribute and stores this as an attribute of a trivial xml document as the output.

Uses some simple manipulations to get the actual avatar source url

Replaces the size from 48px which is used on the front page to 80px which I use in the client.

Emits the avatarUrlChanged() signal to nudge the QML component into action

Here's the code that does it:

@
void Profile::_q_extractAvatar()
{
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

// Hack around broken xhtml on QtDevNet front page
QByteArray input( reply->readAll() );
reply->deleteLater(); // Delete the reply when we go back to the event loop
input.replace( "{set_mode}",
               "" );
input.replace( "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png">",
               "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png" />" );
input.replace( "&lt;input type="text" name="tag_search" id="tag_search_input" value=""&gt;",
               "&lt;input type="text" name="tag_search" id="tag_search_input" value="" /&gt;" );
input.replace( "function searchStart",
               "&lt;![CDATA[function searchStart" );
input.replace( "});n[removed]",
               "});n]]>[removed]" );

QBuffer device;
device.setBuffer( &input );
device.open( QIODevice::ReadOnly );

// Our XQuery string
QString queryString( "declare default element namespace "http://www.w3.org/1999/xhtml";n"
                     "declare variable $inputDocument external;n"
                     "doc($inputDocument)//div[@class="frontpage-profile-image"]/a/img/n"
                     "<p>{@src}</p>" );

// Prepare the xml query object
QXmlQuery query;
query.bindVariable( "inputDocument", &device );
query.setQuery( queryString );
if ( !query.isValid() )
    return;

// We need a buffer to write the output to
QByteArray outArray;
QBuffer buffer( &outArray );
buffer.open( QIODevice::ReadWrite );

// Set up a formatter and execute the query
QXmlFormatter formatter( query, &buffer );
if ( !query.evaluateTo( &formatter ) )
    return;
buffer.close();
//qDebug() << "Query output:" << outArray.constData();

// Now get the actual source url that we need
QString needle( "src="" );
int index = outArray.indexOf( needle );
if ( index != -1 )
{
    QByteArray avatarSource = outArray.mid( index + needle.length() );
    int idx = avatarSource.indexOf( """ );
    if ( idx != -1 )
    {
        avatarSource = avatarSource.mid( 0, idx );
        avatarSource.replace( "size=48", "size=80" );
        m_avatarUrl.setUrl( QString::fromLatin1( avatarSource ) );
        emit avatarUrlChanged();
    }
}

}
@

I know I could have replaced the QXmlQuery stuff with a simple text search but I think this way is more robust. Plus it was a chance for me to use the XmlPatterns module which is still very new to me (which probably shows in my XQuery string).

ZapB

Marius, it might also be nice to have the memberId returned as part of the profile query in the restful API. ie for my account it would include a json attribute of "memberId" : "3246".

So overall it would be nice if the profile json response looked like this:

@
{
"memberId" : "3246",
"gravatarHash" : "d4737278a208398242bdf2535701f8a3",
"current_rank":
{
"points":"494",
"points_last_7_days":"296",
"level":2,
"title":"Ant Farmer",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level2.png"
},
"next_rank":
{
"points_remaining":47,
"level":3,
"title":"Hobby Entomologist",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level3.png"
},
"badges": [
{
"name":"Nokia Certified Qt Developer",
"url":"http://developer.qt.nokia.com/images/qtdn/icon_certified_developer.png"
}]
}
@

By returning the gravatar hash rather than the actual URL, clients could construct their own url for custom sizes and default image fallbacks more easily.

goetz

ZapB, bad news for you: Marius is on holidays for the next two weeks. I think it would be better to file a bug report, so these things are not missing.

ZapB

Hi Volker, thanks for the heads-up. I'll file a bug report.

Edit: Created issues "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 and "QTWEBSITE-221":http://bugreports.qt.nokia.com/browse/QTWEBSITE-221

troubalex

Marius is on holidays. He promised to look at the forums from time to time but not sure when and how much time he will want to spend. ;)

In the meantime I'll poke one of our devs for you.

ZapB

Thanks Alexandra. I've just noticed another problem on the front page (added a comment to "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220). The new forum post summary on the main page does not have reserved characters replaced by their corresponding xml entities. ie & should be "& a m p ;" Edit: had to add spaces to stop forum software from replacing it here too ;-)

I'll pause development of this client there for now whilst I document some of the things used in it as it makes a nice show-case for some C++/QML integration, QNAM usage, QXmlQuery usage, custom QML components etc. Once I get these wiki'fied and the API is reinstated I'll carry on to the next set of features (probably forum related stuff).

Thanks again!

gurudutt

[quote author="ZapB" date="1304333076"]Another piece of invalid xml is found on line 190: @ <div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div> @ The error is: @ Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch. @ The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag. Edit: This input tag on line 532 is also missing a closing "/" @ <input type="text" name="tag_search" id="tag_search_input" value=""> @[/quote]

All possible above stuffs are fixed,
for other fixes it has to wait for deployment in future. I will create a issue here and will try to do all possible fixes.

ZapB

Hi Gurudutt. Thanks for the quick turnaround on those little fixes. Can you mark the task as a sub-task of "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 please so that I can track it?

I appreciate that the other fixes will require changes to some php code and so must be tested and verified. Is the code for QtDevNet open source and on Gitorious? If so, I don't mind taking a poke around to see if I can fix them up.

Thanks again.