Using WebKit on the server

[dialingo]

I am trying to use QWebPage in a server program to retrieve an AJAX remote page. The following program compiles but crashes with :
QWidget: Cannot create a QWidget when no GUI is being used

@#include <QCoreApplication>
#include <QWebPage>
#include <QWebFrame>
#include <QDebug>

int main(int argc, char *argv[])
{
QCoreApplication app(argc, argv);
QWebPage page;
page.mainFrame()->load(QUrl("http://www.yahoo.com"));
app.connect(&page;, SIGNAL(loadFinished(bool)), &app;, SLOT(quit()));
app.exec();
qDebug() << page.mainFrame()->toHtml() ;
return 0;
}@

When I change QCoreApplication to QApplication the program runs perfectly, but then it can not be run on the server because of missing graphics stack. Can this problem be solved with Qt or is it necessary to install dummy graphics hardware?

[mlong]

As far as I know, QWebPage is reliant on the GUI stuff. Would it be sufficient to use QNetworkAccessManager/QNetworkRequest to fetch the web page instead?

[dialingo]

Thanks for the idea mlong, unfortunately this is not possible.

I was using curl initially. But then the web page (it is not yahoo) was changed. Now it consists of a couple of JavaScript lines only. JavaScript pulls in content from different sources to compose the web page. I am interested in the html of the composed page, not the html of the JavaScript program. Therefore I need the JavaScript processing that is available in QWebPage but not in QNetworkAccessManager/QNetworkRequest.

[mlong]

Thanks what I was afraid of. Sorry about that! I don't have any other insight beyond that.

[ZapB]

You may be able to use QNetworkAccessManager together with QtScript. Use QNAM to fetch the intitial page as you said, then use some tricks to get QtScript to process the js. See this "blog post":http://labs.qt.nokia.com/2011/03/04/faking-a-web-browser-environment-in-qtscript/ for ideas on how to do it.

[dangelog]

No, you DO need a QApplication in order to use QtWebkit, which in turn imply installing (at least) the X11 libs, plus something like Xvfb. wkhtmltopdf employs some trickery to avoid to establish an X11 connection, so you could see how it does that.

[dialingo]

Thanks guys, now I have a couple of alternatives. I came across another alternative: HttpFox is a Firefox Plugin that tracks all requests in a Ajax environment. This can help to build a program that pretends to be a Javascript enabled web client.

It will take me some time to evaluate the three solutions:

• QNAM with QtScript
• Installing a fake X11 environment on the server
• using HttpFox to reverse engineer the Ajax traffic

[zester]

Why not use http://code.google.com/p/phantomjs/ uses Qt, QtWebkit and is headless should do exactly what you want to do.

[dialingo]

Thanks, this is what I was looking for. Meanwhile I have reverse engineered the Ajax traffic and the program is working well. I will try phantomjs in the next project.