Qt World Summit: Register Today!

Process QWebPage content in background thread

  • Hi

    The Problem:
    I have QWebPage object with loaded page and after loading process I need to call QMainFrame::eveluateJavaScript() for this object frame. Evaluated script is complex so can takes a long time (even few seconds) so I must move it to background thread (there is not other option here).

    My basic plan:
    Because QWebPage and QWebFrame not inherits from QWidget I was hope it's possible to work with them in background thread, so I wanted to create QWebPage object 'switchable beetwen threads', and call moveToThread() on it, to work with it in main or background thread.
    I see that load page (building DOM) in background thread is impossible, QWebFrame::load() TheeShared makes assertion in debug mode:

    ASSERTION FAILED: isMainThread()
    c:\work\build\qt5_workdir\w\s\qtwebkit\Source\WebCore\platform\TreeShared.h(44) : WebCore::TreeShared<class WebCore::Node>::TreeShared

    in Release nothing crashes, but instead successfull loaded page Im getting some black text as page content, so Document creation fails anyway.

    So I can't load page content in background thread- it's not the big problem, loading page can still be done in main thread, but when page will be loaded in main thread I still must call evaluateJavaScript() in background thread. If after loading page I will call moveToThread() on QWebPage(not binded with QWebView of course) and then if object will work in background thread I will call evaluateJavaScript (to not block main thread) there. Does it fail? Does javascript execution will be safe?
    After that operation I will move QWebPage back to main thread and set it in QWebView.

    I am wondering does it be safe for Qt 5.2 QWebFrame implementation? If not, is it possible to do that without using multiple processes?
    What do you think? Thanks for any tips.

  • Doesn't matter, any evaluateJavaScript out of main frame also crashes in Release

    @void AppPageThreadContext::onAnalyzePageInBackground() {
    QVariant ret = this->mainFrame()->evaluateJavaScript("[removed].href");
    emit analyzeResult(ret.toString());

    in Debug makes assertion:
    ASSERTION FAILED: isMainThread()
    c:\work\build\qt5_workdir\w\s\qtwebkit\source\webcore\bindings\js\DOMWrapperWorld.cpp(72) : WebCore::mainThreadNormalWorld

    I was hope QWebPage isn't so limited to only main thread, I was wrong, rly bad design.

  • All Webkit processing should be done on main GUI thread only
    This is by design!

    otherwise you'll start getting strange error messages like "Widgets must be created in GUI thread" etc.

    see http://qt-project.org/forums/viewthread/6425

  • Thanks for reply.
    Does this design comes from WebKit, that's why Chrome runs in many processes?

    I fixed the problem by changing javascript long function work with many steps evaluated in setInterval(), this is the only way to run something very long in javascript without blocking application.

  • What I would do is the following.

    • Don't work with any threads except the main thread,
    • Create an object and pass it to your javascript using Add addToJavaScriptWindowObject,
    • What will happen is that, you give call your JS code and when your JS code finishes after couple of seconds, it will trigger a slot that you will create,

    I can give an example code for addToJavaScriptWindowObject if you ask.

    Hope this makes sense.

  • I used addToJavaScriptWindowObject many times, but how does this apply to not block main thread?

  • you can run your function asyncronously from within javascript
    e.g. setTimeout( yourFn, 0 );

    Assign result to a global variable - NB Assign status too e.g. MyLongFnCompleted = true; when it finishes

    Later just do few short checks with
    @QVariant ret = this->mainFrame()->evaluateJavaScript("MyLongFnCompleted == true");
    QVariant val = this->mainFrame()->evaluateJavaScript("MyLongFnResult");

  • I wrote that I fixed the problem.
    @ThatDude Your answer isn't quite correct to fix the problem. Why?

    [quote author="ThatDude" date="1418674309"]you can run your function asyncronously from within javascript
    e.g. setTimeout( yourFn, 0 );[/quote]

    single call of setTimeout(veryVeryLongFunc, 0); will change nothing than delay of execution this very very long function later.
    Do you think if you use setTimeout() you will move execution to another magic thread? No. Later execution of this function as Timer Event will still block your main loop for long time, because Javascript code is allways evaluated in main thread.
    As I wrote, to fix this problem, the only way is to divide very very long function to smaller steps and execute them as setInterval(doOneShortStepFromManySteps, 0) many times to complete job in many Javascript Timer Events evaluations.

    [quote author="ThatDude" date="1418674309"]
    Assign result to a global variable - NB Assign status too e.g. MyLongFnCompleted = true; when it finishes

    Later just do few short checks with
    @QVariant ret = this->mainFrame()->evaluateJavaScript("MyLongFnCompleted == true");
    QVariant val = this->mainFrame()->evaluateJavaScript("MyLongFnResult");
    Hmm... To signal end of long work, nicer would be to call slot method or Q_INVOKABLE on object binded with javascript by addToJavaScriptWindowObject() :>
    isn't it?

    If I'm more thinking about this problem I'm starting to understand reasons why Javascript is limited to only GUI thread.
    For example Javascript window object implements actions on some GUI elements, functions like alert(), moveTo(), resizeTo(), etc. operates exactly on GUI elements, If yoo think about that you will see how Javascript objects are connected with GUI objects, so they must work in thread where those GUI objects works, that why Javascript code must be evaluated where works it window object.

  • @crayze
    My approach will work if you don't try to access javascript again and block in the other call.

    1. You have a choice to ether use window.onload event to start long calculations automatically like this:

    Attach a slot to the QWebFrame::javascriptWindowObjectCleared() signal. At this point, call QWebFrame::evaluateJavascript() to add code similar to the following: window.onload = function() { // page has fully loaded }

    or to do this on button click calling
    evaluateJavaScript("setTimeout( yourFn, 0 );")
    this is asynchronous call and will return immediately

    ..then you have to periodically check (on timer) your global variables in java-script if it has finished and grab the result when done ...and don't block main event loop for too long WebKit might need it too

    1. Not all code executed inside WebKit need synchronized access to main GUI thread but once in a while ..based on my experience if you block extensively GUI thread WebKit becomes extremely sluggish sometimes
      so long story short - WebKit executes in its own thread(s) but when it needs access to GUI thread that access must be synchronized access

    so if your long calculation is just that - calculation chances are WebKit won't talk to GUI thread that often but if your longCalc function changes HTML elements/Widgets a lot e.g. delete/move/create HTML elements then yes - that could be a problem but only if you block GUI thread for long periods - otherwise as long as your app is truly asynchronous you don't need to do anything - message pump i.e. main event loop started with app.exec(); is working behind the scenes and does its magic :-)

    1. The only problem if you don't block in another call to evaluateJavascript after the async one "setTimeout( yourFn, 0 );"
      is if you have heavy calculations in your main/GUI thread - do as Webkit, move it to a separate thread and only do/wait for synchronized access with main/GUI thread to exchange data

    2. Of course you can go the "addToJavaScriptWindowObject" route that requires a lot more work i.e. code but that will save only the onTimer checks when the result is ready (vs callback) and you still have to make sure you don't block in another call to evaluateJavascript or e.g. blocking main/GUI thread by heavy calculations in C++ that could block or slow down WebKit significantly


Log in to reply