Qt DevNet Web Service API

[ZapB]

When the request has finished we go into the private slot _q_extractAvatar(). This does a few things:

Checks that we have a real QNetworkReply.

Reads the entire xhtml document from the reply, and schedules the reply for deletion.

Performs a few hacky replacements on the document to work around xml validation errors.

Uses QXmlQuery to find the appropriate img element for the avatar image, extracts the src attribute and stores this as an attribute of a trivial xml document as the output.

Uses some simple manipulations to get the actual avatar source url

Replaces the size from 48px which is used on the front page to 80px which I use in the client.

Emits the avatarUrlChanged() signal to nudge the QML component into action

Here's the code that does it:

@
void Profile::_q_extractAvatar()
{
qDebug() << Q_FUNC_INFO;

QNetworkReply* reply = qobject_cast<QNetworkReply*>( sender() );
if ( !reply )
    return;

// Hack around broken xhtml on QtDevNet front page
QByteArray input( reply->readAll() );
reply->deleteLater(); // Delete the reply when we go back to the event loop
input.replace( "{set_mode}",
               "" );
input.replace( "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png">",
               "<img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png" />" );
input.replace( "&lt;input type="text" name="tag_search" id="tag_search_input" value=""&gt;",
               "&lt;input type="text" name="tag_search" id="tag_search_input" value="" /&gt;" );
input.replace( "function searchStart",
               "&lt;![CDATA[function searchStart" );
input.replace( "});n[removed]",
               "});n]]>[removed]" );

QBuffer device;
device.setBuffer( &input );
device.open( QIODevice::ReadOnly );

// Our XQuery string
QString queryString( "declare default element namespace "http://www.w3.org/1999/xhtml";n"
                     "declare variable $inputDocument external;n"
                     "doc($inputDocument)//div[@class="frontpage-profile-image"]/a/img/n"
                     "<p>{@src}</p>" );

// Prepare the xml query object
QXmlQuery query;
query.bindVariable( "inputDocument", &device );
query.setQuery( queryString );
if ( !query.isValid() )
    return;

// We need a buffer to write the output to
QByteArray outArray;
QBuffer buffer( &outArray );
buffer.open( QIODevice::ReadWrite );

// Set up a formatter and execute the query
QXmlFormatter formatter( query, &buffer );
if ( !query.evaluateTo( &formatter ) )
    return;
buffer.close();
//qDebug() << "Query output:" << outArray.constData();

// Now get the actual source url that we need
QString needle( "src="" );
int index = outArray.indexOf( needle );
if ( index != -1 )
{
    QByteArray avatarSource = outArray.mid( index + needle.length() );
    int idx = avatarSource.indexOf( """ );
    if ( idx != -1 )
    {
        avatarSource = avatarSource.mid( 0, idx );
        avatarSource.replace( "size=48", "size=80" );
        m_avatarUrl.setUrl( QString::fromLatin1( avatarSource ) );
        emit avatarUrlChanged();
    }
}

}
@

I know I could have replaced the QXmlQuery stuff with a simple text search but I think this way is more robust. Plus it was a chance for me to use the XmlPatterns module which is still very new to me (which probably shows in my XQuery string).

[ZapB]

Marius, it might also be nice to have the memberId returned as part of the profile query in the restful API. ie for my account it would include a json attribute of "memberId" : "3246".

So overall it would be nice if the profile json response looked like this:

@
{
"memberId" : "3246",
"gravatarHash" : "d4737278a208398242bdf2535701f8a3",
"current_rank":
{
"points":"494",
"points_last_7_days":"296",
"level":2,
"title":"Ant Farmer",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level2.png"
},
"next_rank":
{
"points_remaining":47,
"level":3,
"title":"Hobby Entomologist",
"image":"http://developer.qt.nokia.com/images/qtdn/level_images/level3.png"
},
"badges": [
{
"name":"Nokia Certified Qt Developer",
"url":"http://developer.qt.nokia.com/images/qtdn/icon_certified_developer.png"
}]
}
@

By returning the gravatar hash rather than the actual URL, clients could construct their own url for custom sizes and default image fallbacks more easily.

[goetz]

ZapB, bad news for you: Marius is on holidays for the next two weeks. I think it would be better to file a bug report, so these things are not missing.

[ZapB]

Hi Volker, thanks for the heads-up. I'll file a bug report.

Edit: Created issues "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 and "QTWEBSITE-221":http://bugreports.qt.nokia.com/browse/QTWEBSITE-221

[troubalex]

Marius is on holidays. He promised to look at the forums from time to time but not sure when and how much time he will want to spend. ;)

In the meantime I'll poke one of our devs for you.

[ZapB]

Thanks Alexandra. I've just noticed another problem on the front page (added a comment to "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220). The new forum post summary on the main page does not have reserved characters replaced by their corresponding xml entities. ie & should be "& a m p ;" Edit: had to add spaces to stop forum software from replacing it here too ;-)

I'll pause development of this client there for now whilst I document some of the things used in it as it makes a nice show-case for some C++/QML integration, QNAM usage, QXmlQuery usage, custom QML components etc. Once I get these wiki'fied and the API is reinstated I'll carry on to the next set of features (probably forum related stuff).

Thanks again!

[gurudutt]

[quote author="ZapB" date="1304333076"]Another piece of invalid xml is found on line 190: @ <div class="item"><a href="http://info.arcada.fi/sv/qtquick"><img src="http://www.arcada.fi/sites/www.arcada.fi/themes/arcada/media/separate_elements/main_navigation/logo.png"></a></div> @ The error is: @ Error FODC0002 in tag:trolltech.com,2007:QtXmlPatterns:QIODeviceVariable:inputDocument, at line 190, column 180: Opening and ending tag mismatch. @ The img element is missing the empty tag specifier - ie it should have a "/" at the end of the img tag. Edit: This input tag on line 532 is also missing a closing "/" @ <input type="text" name="tag_search" id="tag_search_input" value=""> @[/quote]

All possible above stuffs are fixed,
for other fixes it has to wait for deployment in future. I will create a issue here and will try to do all possible fixes.

[ZapB]

Hi Gurudutt. Thanks for the quick turnaround on those little fixes. Can you mark the task as a sub-task of "QTWEBSITE-220":http://bugreports.qt.nokia.com/browse/QTWEBSITE-220 please so that I can track it?

I appreciate that the other fixes will require changes to some php code and so must be tested and verified. Is the code for QtDevNet open source and on Gitorious? If so, I don't mind taking a poke around to see if I can fix them up.

Thanks again.

[mgran]

Hey ZapB, I'm on vacation atm but Gurudutt is all over it :)

Looking forward to trying it on my N8 when I'm back!

Edit: Didn't see the last few replies, edited a bit

[ZapB]

Hi Marius. No worries. Enjoy your holidays! I'm back at work today after having last week off. I've just registered with the Ovi store and once I test the current state of the app (when the API is re-enabled) I'll build it and publish it on Ovi (for free of course).

I've set up the start of a drupal-based "website":http://www.zerotau.co.uk/ so that I could apply for the Qt Ambassador program on the back of this little project. My "plotting library":http://developer.qt.nokia.com/forums/viewthread/4209/ is still a little way off being ready for release yet and I want to keep the source code private until the first public release - just so as to make a nice surprise :-).

[ZapB]

Gurudutt, another problem today. Onthe front page again, line 226, col 181:

@
<div class="item"><img src="http://developer.qt.nokia.com/uploads/qt_quarterly/qt_ambassador_logo.png" alt="Qt Ambassador" width="150" height="201" alt="image" /></div>
@

the alt attribute is redefined leading to another error when using XQuery on that xhtml. Could you squash that bug too please? It looks like a simple static html one.

Many thanks.

[gurudutt]

[quote author="ZapB" date="1304592956"]Gurudutt, another problem today. Onthe front page again, line 226, col 181:
the alt attribute is redefined leading to another error when using XQuery on that xhtml. Could you squash that bug too please? It looks like a simple static html one.

Many thanks.[/quote]

It should be fixed now.

[ZapB]

Yup. Thanks for the swift turnaround.

At the risk of telling you guys how to do your jobs it might be an idea to integrate a call to xmllint to catch these types of errors at source. Could be doen as part of your CMS' add new content function perhaps? Please don't take that as criticism, it's just a polite suggestion. I think you are all doign a great job. :-)

[gurudutt]

Hi ZapB

The way we have prepared Qt Developer Network,
in that some content's are added/edited by editors and some of them is fetched from mysql database via CMS api.

The things I have fixed for this issue, I am not sure if in future it will not occur again since some of our editors might not be that technical to think about all these standards.

About giving access to admin site, I am not sure if that is ideal, I guess there are some security concerns and many more.

We wish to make all Qt Developer Network source as open source one day but we want to evaluate all security concerns before doing it.

I guess for your concerns qtapi would be best in future, it is disabled for now, but we wish to open it as soon as we can.

In future we wish to have better solution for qtapi for better performance.

[troubalex]

[quote author="Gurudutt" date="1304609532"]Hi ZapB
About giving access to admin site, I am not sure if that is ideal, I guess there are some security concerns and many more.
[/quote]

Although it might solve some problems, it should be fairly obvious that we can't do that. We're even carefully towards our own colleagues.

Just imagine someone accidentally hitting the "drop all tables" button. :P

[ZapB]

Hi Gurudutt,

Oh I wasn't suggesting that we all have access to the admin side. I was just saying earlier that if the CMS happened to be open source I could take a look at adding functionality to properly escape xml entities. Since it is not, then no problem. I'll leave it to you guys and gals.

My previous message was just suggesting that to help non-technical content writers, running the submissions (well the page that would result from a submission) through xmllint or some other validator before accepting the submission might help catch these simple errors. Modern browsers will happily process xhtml even with such errors but an author could still potentially break the page with a bad submission.

I realise that long-term using QXmlQUery is not the best way to get info about QtDN and that the REST API is the way to go. But until it is updated or even better, opened up, then we are limited as to what we can do with it.

If the CMS code, and SQL scehema (but not auth details etc) were placed on gitorious we could even help develop the REST API too and make MR's for it. Then you trolls can test the MR's in a sand-boxed environment and merge them only when you are happy with them.

Maybe a topic for discussion at the contributers summit?

[ZapB]

[quote author="Alexandra" date="1304609933"][quote author="Gurudutt" date="1304609532"]Hi ZapB
About giving access to admin site, I am not sure if that is ideal, I guess there are some security concerns and many more.
[/quote]

Although it might solve some problems, it should be fairly obvious that we can't do that. We're even carefully towards our own colleagues.

Just imagine someone accidentally hitting the "drop all tables" button. :P[/quote]

I never suggested having access to the admin side of the live site - only the source code of the CMS. ;-)

[ZapB]

As for making the API faster, you could use mod_rewite or some other method to redirect the API requests to a dedicated box that runs a Qt/C++ mini-webserver process that uses all those lovely Qt modules to query and process the requests into a suitable json response. All we need to get that off the ground is some details of the database schema. We could open it up as a QtDevNet community project ;-)

[gurudutt]

[quote author="ZapB" date="1304592956"]Gurudutt, another problem today. Onthe front page again, line 226, col 181:

@
<div class="item"><img src="http://developer.qt.nokia.com/uploads/qt_quarterly/qt_ambassador_logo.png" alt="Qt Ambassador" width="150" height="201" alt="image" /></div>
@

the alt attribute is redefined leading to another error when using XQuery on that xhtml. Could you squash that bug too please? It looks like a simple static html one.

Many thanks.[/quote]

Hi ZapB,

Front page of Developer network follows XHTML 1.0 strict standards now.
Please confirm if it works as expected for your plug-in.

[ZapB]

Hi Gurudutt.

Thanks for sorting out those issues. I am able to use QXmlQuery to scrape the gravatar info once again.