Qt DevNet Web Service API

ZapB

Sweet! I just managed to get this running on my N8. It's very awesome to see the same QML animations running on the phone - smoothly too! :D

It took a little bit of trial and error and I am still not quite sure what made it start to deploy properly and I can still not get qjson to build for the qt-simulator target. I'll write up some instructions on how to do this and commit my mods to the svn repo shortly.

Edit 1: Code to allow this to be built and deployed for Symbian is now in svn.

Edit 2: Figured out what was stopping me from running this on the device originally. I kept getting the "General OS Error" when trying to deploy and run my app. It turns out that it was because of insufficient capabilities in the qjson lib on which my app depends. By changing the TARGET.CAPABILITY line in qjson/src/src.pro to:

@
TARGET.CAPABILITY = NetworkServices ReadDeviceData WriteDeviceData
@

allows my app to load the qjson dll on the device (since my app needs access to the Internet). Read about "capabilities":http://wiki.forum.nokia.com/index.php/Capabilities for more info on this.

I have concerns about how this may scale if several apps want to use the same dll but with different capabilities but that's something to worry about for another day.

ZapB

Marius, I wonder if you could help me with something please? I am trying to add avatar support to my little QtDN client and until the API supports returning the avatar image URL I thought I would take the opportunity to try using QXmlQuery to parse the xhtml of the front page of QtDN to extract the gravatar source url.

This is all going fine until I get this error from QXmlQuery:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 74, column 32: Expected '=', but got '>'.
@

Looking at the page source I see that line 74 is:

@
<div class="wrapper" {set_mode}>
@

which of course is not valid xhtml (or xml) as advertised by the doctype. I can work around this by stripping this out before I pass it onto the QXmlQuery of course but would you like me to report this as a bug? Or even better is there an easy fix you can apply to get rid of the above non-valid entry please?

Many thanks.

ZapB

Another piece of invalid xml is found on line 190:

@
<div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div>
@

The error is:

@
Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch.
@

The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag.

Edit: This input tag on line 532 is also missing a closing "/"

@
<input type="text" name="tag_search" id="tag_search_input" value="">
@

ZapB

I found one other issue that prevents the main page from validating as well-formed xml. The javascript at the bottom of the page (starting on line 710):

@
function searchStart()
...
});
@

really needs to be wrapped in a CDATA section like this:

@
<![CDATA[
function searchStart()
...
});
]]>
@

I have put in place some hackish work-arounds to get over these non-conformances. Here is what I have done to scrape the gravatar image from the front page.

In the Profile::update() function (which is where I usually use QNAM to query the profile API) I also do this on the first call after a successful login:

@
static bool firstTime = true;
if ( firstTime )
{
// Load the front page so that we can scrape the avatar
firstTime = false;
QNetworkRequest avatarRequest;
avatarRequest.setUrl( QUrl( "http://developer.qt.nokia.com/" ) );
avatarRequest.setRawHeader( "User-Agent", "ZapBs QtDevNet Client 0.1" );

    QNetworkReply* avatarReply = m_nam->get( avatarRequest );
    connect( avatarReply, SIGNAL( finished() ),
             SLOT( _q_extractAvatar() ) );
    connect( avatarReply, SIGNAL( error( QNetworkReply::NetworkError ) ),
             SLOT( _q_onReplyError( QNetworkReply::NetworkError ) ) );
}

@

This just uses QNAM to fetch the xhtml document for the QtDN front page.

Edit: Had to split post as it was too large.

ZapB

When the request has finished we go into the private slot _q_extractAvatar(). This does a few things:

Checks that we have a real QNetworkReply.

Reads the entire xhtml document from the reply, and schedules the reply for deletion.

Performs a few hacky replacements on the document to work around xml validation errors.

Uses QXmlQuery to find the appropriate img element for the avatar image, extracts the src attribute and stores this as an attribute of a trivial xml document as the output.

Uses some simple manipulations to get the actual avatar source url

Replaces the size from 48px which is used on the front page to 80px which I use in the client.

Emits the avatarUrlChanged() signal to nudge the QML component into action

Here's the code that does it:

@
void Profile::_q_extractAvatar()
{
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

// Hack around broken xhtml on QtDevNet front page
QByteArray input( reply->readAll() );
reply->deleteLater(); // Delete the reply when we go back to the event loop
input.replace( "{set_mode}",
               "" );
input.replace( "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png">",
               "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png" />" );
input.replace( "&lt;input type="text" name="tag_search" id="tag_search_input" value=""&gt;",
               "&lt;input type="text" name="tag_search" id="tag_search_input" value="" /&gt;" );
input.replace( "function searchStart",
               "&lt;![CDATA[function searchStart" );
input.replace( "});n[removed]",
               "});n]]>[removed]" );

QBuffer device;
device.setBuffer( &input );
device.open( QIODevice::ReadOnly );

// Our XQuery string
QString queryString( "declare default element namespace "http://www.w3.org/1999/xhtml";n"
                     "declare variable $inputDocument external;n"
                     "doc($inputDocument)//div[@class="frontpage-profile-image"]/a/img/n"
                     "<p>{@src}</p>" );

// Prepare the xml query object
QXmlQuery query;
query.bindVariable( "inputDocument", &device );
query.setQuery( queryString );
if ( !query.isValid() )
    return;

// We need a buffer to write the output to
QByteArray outArray;
QBuffer buffer( &outArray );
buffer.open( QIODevice::ReadWrite );

// Set up a formatter and execute the query
QXmlFormatter formatter( query, &buffer );
if ( !query.evaluateTo( &formatter ) )
    return;
buffer.close();
//qDebug() << "Query output:" << outArray.constData();

// Now get the actual source url that we need
QString needle( "src="" );
int index = outArray.indexOf( needle );
if ( index != -1 )
{
    QByteArray avatarSource = outArray.mid( index + needle.length() );
    int idx = avatarSource.indexOf( """ );
    if ( idx != -1 )
    {
        avatarSource = avatarSource.mid( 0, idx );
        avatarSource.replace( "size=48", "size=80" );
        m_avatarUrl.setUrl( QString::fromLatin1( avatarSource ) );
        emit avatarUrlChanged();
    }
}

}
@

I know I could have replaced the QXmlQuery stuff with a simple text search but I think this way is more robust. Plus it was a chance for me to use the XmlPatterns module which is still very new to me (which probably shows in my XQuery string).

ZapB

Marius, it might also be nice to have the memberId returned as part of the profile query in the restful API. ie for my account it would include a json attribute of "memberId" : "3246".

So overall it would be nice if the profile json response looked like this:

@
{
"memberId" : "3246",
"gravatarHash" : "d4737278a208398242bdf2535701f8a3",
"current_rank":
{
"points":"494",
"points_last_7_days":"296",
"level":2,
"title":"Ant Farmer",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level2.png"
},
"next_rank":
{
"points_remaining":47,
"level":3,
"title":"Hobby Entomologist",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level3.png"
},
"badges": [
{
"name":"Nokia Certified Qt Developer",
"url":"http://developer.qt.nokia.com/images/qtdn/icon_certified_developer.png"
}]
}
@

By returning the gravatar hash rather than the actual URL, clients could construct their own url for custom sizes and default image fallbacks more easily.

goetz

ZapB, bad news for you: Marius is on holidays for the next two weeks. I think it would be better to file a bug report, so these things are not missing.

ZapB

Hi Volker, thanks for the heads-up. I'll file a bug report.

Edit: Created issues "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 and "QTWEBSITE-221":http://bugreports.qt.nokia.com/browse/QTWEBSITE-221

troubalex

Marius is on holidays. He promised to look at the forums from time to time but not sure when and how much time he will want to spend. ;)

In the meantime I'll poke one of our devs for you.

ZapB

Thanks Alexandra. I've just noticed another problem on the front page (added a comment to "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220). The new forum post summary on the main page does not have reserved characters replaced by their corresponding xml entities. ie & should be "& a m p ;" Edit: had to add spaces to stop forum software from replacing it here too ;-)

I'll pause development of this client there for now whilst I document some of the things used in it as it makes a nice show-case for some C++/QML integration, QNAM usage, QXmlQuery usage, custom QML components etc. Once I get these wiki'fied and the API is reinstated I'll carry on to the next set of features (probably forum related stuff).

Thanks again!

gurudutt

[quote author="ZapB" date="1304333076"]Another piece of invalid xml is found on line 190: @ <div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div> @ The error is: @ Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch. @ The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag. Edit: This input tag on line 532 is also missing a closing "/" @ <input type="text" name="tag_search" id="tag_search_input" value=""> @[/quote]

All possible above stuffs are fixed,
for other fixes it has to wait for deployment in future. I will create a issue here and will try to do all possible fixes.

ZapB

Hi Gurudutt. Thanks for the quick turnaround on those little fixes. Can you mark the task as a sub-task of "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 please so that I can track it?

I appreciate that the other fixes will require changes to some php code and so must be tested and verified. Is the code for QtDevNet open source and on Gitorious? If so, I don't mind taking a poke around to see if I can fix them up.

Thanks again.

mgran

Hey ZapB, I'm on vacation atm but Gurudutt is all over it :)

Looking forward to trying it on my N8 when I'm back!

Edit: Didn't see the last few replies, edited a bit

ZapB

Hi Marius. No worries. Enjoy your holidays! I'm back at work today after having last week off. I've just registered with the Ovi store and once I test the current state of the app (when the API is re-enabled) I'll build it and publish it on Ovi (for free of course).

I've set up the start of a drupal-based "website":http://www.zerotau.co.uk/ so that I could apply for the Qt Ambassador program on the back of this little project. My "plotting library":http://developer.qt.nokia.com/forums/viewthread/4209/ is still a little way off being ready for release yet and I want to keep the source code private until the first public release - just so as to make a nice surprise :-).

ZapB

Gurudutt, another problem today. Onthe front page again, line 226, col 181:

@
<div class="item"><img src="http://developer.qt.nokia.com/uploads/qt_quarterly/qt_ambassador_logo.png" alt="Qt Ambassador" width="150" height="201" alt="image" /></div>
@

the alt attribute is redefined leading to another error when using XQuery on that xhtml. Could you squash that bug too please? It looks like a simple static html one.

Many thanks.

gurudutt

[quote author="ZapB" date="1304592956"]Gurudutt, another problem today. Onthe front page again, line 226, col 181:
the alt attribute is redefined leading to another error when using XQuery on that xhtml. Could you squash that bug too please? It looks like a simple static html one.

Many thanks.[/quote]

It should be fixed now.

ZapB

Yup. Thanks for the swift turnaround.

At the risk of telling you guys how to do your jobs it might be an idea to integrate a call to xmllint to catch these types of errors at source. Could be doen as part of your CMS' add new content function perhaps? Please don't take that as criticism, it's just a polite suggestion. I think you are all doign a great job. :-)

gurudutt

Hi ZapB

The way we have prepared Qt Developer Network,
in that some content's are added/edited by editors and some of them is fetched from mysql database via CMS api.

The things I have fixed for this issue, I am not sure if in future it will not occur again since some of our editors might not be that technical to think about all these standards.

About giving access to admin site, I am not sure if that is ideal, I guess there are some security concerns and many more.

We wish to make all Qt Developer Network source as open source one day but we want to evaluate all security concerns before doing it.

I guess for your concerns qtapi would be best in future, it is disabled for now, but we wish to open it as soon as we can.

In future we wish to have better solution for qtapi for better performance.

troubalex

[quote author="Gurudutt" date="1304609532"]Hi ZapB
About giving access to admin site, I am not sure if that is ideal, I guess there are some security concerns and many more.
[/quote]

Although it might solve some problems, it should be fairly obvious that we can't do that. We're even carefully towards our own colleagues.

Just imagine someone accidentally hitting the "drop all tables" button. :P

ZapB

Hi Gurudutt,

Oh I wasn't suggesting that we all have access to the admin side. I was just saying earlier that if the CMS happened to be open source I could take a look at adding functionality to properly escape xml entities. Since it is not, then no problem. I'll leave it to you guys and gals.

My previous message was just suggesting that to help non-technical content writers, running the submissions (well the page that would result from a submission) through xmllint or some other validator before accepting the submission might help catch these simple errors. Modern browsers will happily process xhtml even with such errors but an author could still potentially break the page with a bad submission.

I realise that long-term using QXmlQUery is not the best way to get info about QtDN and that the REST API is the way to go. But until it is updated or even better, opened up, then we are limited as to what we can do with it.

If the CMS code, and SQL scehema (but not auth details etc) were placed on gitorious we could even help develop the REST API too and make MR's for it. Then you trolls can test the MR's in a sand-boxed environment and merge them only when you are happy with them.

Maybe a topic for discussion at the contributers summit?