'evaluateJavaScript()' doesn't return any value. (content position on page wanted)



  • I'm trying to develop a program which will pinpoint specific content on a web page via ID's or Classes and then display only this piece of the page. To achieve this I'm evaluating Javascript on my Web Page to get the position of the wanted content, sadly I just can't find a way to return those javascript values back to my Qt Creator in order to resize the window or change position.

    Here is my Code:

    #include "mainwindow.h"
    #include "ui_mainwindow.h"
    
    #include <string>
    #include <iostream>
    #include <stdlib.h>
    #include <stdio.h>
    
    #include <QMainWindow>
    #include <QApplication>
    #include <Qt>
    #include <QDebug>
    #include <QWebView>
    #include <QWebPage>
    #include <QWebFrame>
    #include <QtWebKit>
    #include <QScrollBar>
    #include <QScrollArea>
    #include <QString>
    
    using namespace std;
    
    MainWindow::MainWindow(QWidget *parent) :
        QMainWindow(parent),
        ui(new Ui::MainWindow)
    {
        ui->setupUi(this);
    
        // Configurating Application-Window
        setWindowFlags(Qt::Window | Qt::WindowMinimizeButtonHint | Qt::WindowCloseButtonHint);
        setFixedSize(width(), height());
    
        // Object-Reference Shortcuts
        QWebView *WV = ui->webView;
        QWebPage *WP = ui->webView->page();
        QWebFrame *MF = ui->webView->page()->mainFrame();
    
        // Elements of interest - Shortcuts
        QVariant user_reg_elem            = MF->evaluateJavaScript("document.getElementById('user_reg')");
        QVariant passwd_elem              = MF->evaluateJavaScript("document.getElementById('passwd_reg')");
        QVariant passwd2_elem             = MF->evaluateJavaScript("document.getElementById('passwd2_reg')");
        QVariant recaptcha_checkbox_elem  = MF->evaluateJavaScript("document.getElementsByClassName('recaptcha-checkbox-checkmark')[0]");
    
        // Deactivating Scrolling-Bars
        MF->setScrollBarPolicy(Qt::Horizontal, Qt::ScrollBarAlwaysOff);
        MF->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);
    
        // Waiting for Browser to load Website & harvesting target position
        QVariant testing = MF->evaluateJavaScript("window.onload = function() {\
                                var bodyRect  = document.body.getBoundingClientRect(),\
                                elemRect  = document.getElementById('passwd2_reg').getBoundingClientRect(),\
                                offsetH   = (elemRect.top - bodyRect.top).toString(),\
                                offsetV   = (elemRect.left - bodyRect.left).toString();\
                                alert([offsetV, offsetH]);\
                                return([offsetV, offsetH])};");
                                      
        //MF->evaluateJavaScript("window.onload = function() {window.scrollTo(0,0)}");
        //MF->addToJavaScriptWindowObject('pos_target', pos_js);
          
        qDebug() << testing;                           
        //qDebug() << "Debug: " << pos_js.toString();
    }
    
    MainWindow::~MainWindow()
    {
        delete ui;
    }
    

    qDebug() returns the following to my console :

    QVariant(QVariantMap, QMap() ) 
    

    I have tried to display the data inside of the QVariant but always receive and empty string (i.e. "" ) as result. I'm not even sure if my return value contains any piece of information. The alert function returns the wanted values normally (for example: 12, 317 )

    Does anybody know how I can slay this dragon?


  • Moderators

    @ShivaHaze What does

    qDebug() << testing.toMap();
    qDebug() << testing.toMap().size();
    

    print out?



  • @jsulm said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    @ShivaHaze What does

    qDebug() << testing.toMap();
    qDebug() << testing.toMap().size();
    

    print out?

    QMap()
    0

    Sadly everything seems to be empty..





  • @JNBarchan Yes I have looked at both sources, but as you can see those people seem to be receiving a return value.
    The first post in the second source states:

    The console window shows:
    Debug: QVariant(QVariantMap, QMap(("length", QVariant(double, 20) ) ) )

    But mine shows:

    QVariant(QVariantMap, QMap() )

    But I don't receive anything at all - that's the problem.
    I'm fighting this problem for a week already, it's driving me so crazy.


    Interesting:

    After using the code snippet from the second source

    QVariant listOfImages = MF->evaluateJavaScript("document.body.getBoundingClientRect()");
    qDebug() << listOfImages;

    I actually receive something

    QVariant(QVariantMap, QMap(("bottom", QVariant(double, 70) ) ( "height" , QVariant(double, 62) ) ( "left" , QVariant(double, 8) ) ( "right" , QVariant(double, 296) ) ( "top" , QVariant(double, 8) ) ( "width" , QVariant(double, 288) ) ) )

    Why doesn't my initial JS-Function work tho?
    Maybe because it's multi-lined?

    What would be the correct way to call a JS-Function in QT?



  • Hold on! I have just actually looked at your JavaScript code :) You don't get any result back because it does not return anything! That's why what you do get back is just an empty QVariant.

    You code is just a window.onload = function { ... return([offsetV, offsetH]); }. Do you understand that your evaluateJavaScript() is simply defining an event handler which will run "at a later date" from from when you call it, i.e. after the window has finished loading? It's not right that your function returns whatever, as that value will actually be returned as the result when window.onload calls it, and window.onload does not expect anything like that as a result.

    If you really need to pick up that [offsetV, offsetH] value, you will not be able to do so via window.onload. You will have to wait till after that has completed, and then issue a brand new evaluateJavaScript() call to calculate and return just what you want....

    I have no idea how/whether you're supposed to achieve that with Qt's WebView (others may know better). To play with, you could try after your evaluateJavaScript() putting in some kind of "Qt Sleep" at the server side (i.e. in your Qt code, not via JavaScript's window.setImeout()), which must allow time for the WebView to complete its client-side's onload) and then issue a new evaluateJavaScript() which does the calculation and returns the value to you. You could execute the code directly, or you could define a JavaScript function to do it and call that --- the point is, there will be no window.onload = function ... for this, you will need to execute the code and receive its result directly.

    (In fact, looking at your code, there is then nothing left to do in your original evaluateJavaScript("window.onload = ..."), you only want to run your function's code after onload has completed, however you achieve that. You could if you wish have the earlier evaluateJavaScript define a function containing your code, for neatness, and call that function from your later evaluateJavaScript.)

    Do you really need to get those dimensions back into your Qt server-side code? It is more usual that you might achieve everything in JavaScript client-side, then you would not have these problems about waiting for the page to finish loading...?



  • I need the variables available to qt because I want to resize my browser window to only display the captcha of a website.
    When the captcha changes in size, the qwebview ( & Application itself) has to adapt to the new size.

    What possibilities do I have to make my qt application wait?
    The best case would be if I could "wait for the website to load".
    Otherwise I need some kind of "general sleep method", a few seconds should be enough in most cases which is good enough for me at the moment.

    I changed my code and now 'nearly' receive the values I need, the evaluateJavascript lines seem to be executed to early which is why I need my program to wait for the website to finish loading. I receive false values which are cleary occuring because the website hasn't finished loading. ( via alert later on I receive the right values, due to window.onload)

    #include "mainwindow.h"
    #include "ui_mainwindow.h"
    
    #include <string>
    #include <iostream>
    #include <stdlib.h>
    #include <stdio.h>
    
    #include <QMainWindow>
    #include <QApplication>
    #include <Qt>
    #include <QDebug>
    #include <QWebView>
    #include <QWebPage>
    #include <QWebFrame>
    #include <QtWebKit>
    #include <QScrollBar>
    #include <QScrollArea>
    #include <QString>
    
    using namespace std;
    
    MainWindow::MainWindow(QWidget *parent) :
        QMainWindow(parent),
        ui(new Ui::MainWindow)
    {
        ui->setupUi(this);
    
        // Configurating Application-Window
        setWindowFlags(Qt::Window | Qt::WindowMinimizeButtonHint | Qt::WindowCloseButtonHint);
        setFixedSize(width(), height());
    
        // Object-Reference Shortcuts
        QWebFrame *MF = ui->webView->page()->mainFrame();
    
        // Deactivating Scrolling-Bars
        //MF->setScrollBarPolicy(Qt::Horizontal, Qt::ScrollBarAlwaysOff);
        //MF->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);
    
        // Waiting for Browser to load Website & harvesting target position
        //QVariant test  = MF->evaluateJavaScript("setTimeout(function(){}, 2000);");
        //    QTimer::singleShot(2000, this, SLOT(yourSlot()));
    
        QVariant bodytop  = MF->evaluateJavaScript("document.body.getBoundingClientRect().top");
        QVariant bodyleft = MF->evaluateJavaScript("document.body.getBoundingClientRect().left");
        QVariant elemtop  = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().top");
        QVariant elemleft = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().left");
    
    
        //MF->evaluateJavaScript("window.onload = function() {window.scrollTo(0,0)}");
    
        qDebug() << "Body Top: " << bodytop.toInt() << " Body Left: " << bodyleft.toInt() << " Elem Top: " << elemtop.toInt() << " Elem Left: " << elemleft.toInt();
    }
    
    MainWindow::~MainWindow()
    {
        delete ui;
    }
    

    As values in my console I receive

    Body Top: 8 Body Left: 8 Elem Top: 0 Elem Left: 0

    but I know for sure that the body top and left numbers have to be alot higher, and elem top and left are def. not zero.

    P.S. I'm not a C++ Dev., the last time I tinkered with C++ is years ago - Please understand. :)



  • How/where are you setting the content of your webview page? Are you using a QWebEngineView?



  • @JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    How/where are you setting the content of your webview page? Are you using a QWebEngineView?

    I am using QWebView only.
    The initial URL is set via the property manager, not via code.
    The Source-Code I posted is literally the only customized part, the other files I have are the typical auto-generated files. :-)

    Question on the side: Can I combine QML and QT? Or include .js files to my Project on QT-Creator?
    I feel incredible bound to C++ which is bugging me, I'm really not that skilled in C++ and creating new Classes with Slots and Signals etc. is something I can't do without reading into it.
    This also makes the QT-Docs hard for me to understand. Often I see code snippets in the docs which are not written in C++ or are not complete, but due to the lack of skill in C++ and QT I never know what to do or how to implement those parts into my project.



  • @ShivaHaze
    I'm afraid this now goes beyond my experience of Qt.

    The point I was trying to make was: the code you want to perform in your evaluateJavaScript() requires that the document/window already be fully loaded to get correct answers, right? So you need to ensure that you only call it once you know that that has happened. (My understanding is that the web view rendering is performed asynchronously, so you don't know when that is.) I believe the results you are showing indicate the page load/render is not yet complete at the time you call evaluateJavaScript(), right?

    Do you also understand that evaluateJavaScript("window.onload = ...) cannot be the expression whose return result you want? That statement simply sets a function which will be evaluated on window load complete, it does not execute the function at the time you call evaluateJavaScript. If anything you would want something like:

    evaluateJavaScript("
    function getOffsets()
    {
      var bodyRect = ...
      ...
      return ([offsetV, offsetH])};
    }
    [ return ] getOffsets();
    ")
    

  • Moderators



  • @ShivaHaze
    I still don't see that you have tried code to verify that asynchronicity is indeed the problem, just in case I'm wrong.

    If it were me, then assuming the web view/JavaScript supports alert(), I would try:

    evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");
    evaluateJavaScript("window.alert('Called directly');");
    

    If you see message Called directly followed by message Called from onload then asynchronicity is indeed the problem. What do you get?



  • @JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");
    evaluateJavaScript("window.alert('Called directly');");

    'Called directly' first, then 'Called from onload' :/



  • @ShivaHaze
    Yikes :( Looks like what I am saying is correct then, do you understand where the problem lies in what I'm trying to explain?



  • @JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    Yikes :) Looks like what I am saying is correct then, do you understand where the problem lies in what I'm trying to explain?

    Yes I believe I understand you, the evaluateJavascript call isn't waiting for the website to load even tho I need it to wait.

    But what can I do to make evaluateJavascript wait?
    I tried to find a simple sleep function within qt but wasn't successful yet.



  • @ShivaHaze
    Yep, you get the principle. We need to execute your code only after page has finished onload, and get the answer back to your application...

    TBH, it doesn't look good to me! (Remember, I am not a Qt professional, I am new to it, so unless someone knows better...) I see about 2 possibilities. Neither one is nice....

    This is done using the addToJavaScriptWindowObject() method. The first argument to this method is the name under which it will be available on the JavaScript side, the second is the actual C++ object you want to embed

    In principle that could be used to allow the JS code to "push" the answer to the C++ code as & when the JS is ready. However, that will mean re-architecting your code as it will only receive the result asynchronously.

    I did think you might not like it. Sorry! I will still think of alternatives....



  • @ShivaHaze
    OK, hold on! :)

    The trouble is, I think you are using QML (right?), and I don't know about that. Somewhere something is setting the HTML of the page --- effectively like QWebView::setHtml(). We need to know when that is "complete" (trusting that includes the full loading of your page, i.e. it would come after onload would execute in the client).

    Now, I think you are saying you can still write a bit more code to add what you have. Take a look at https://forum.qt.io/topic/27773/qwebview-wait-load/6

    QWebView view;
    view.show();
    QObject::connect(&view,SIGNAL(loadFinished(bool)),&loop,SLOT(quit()));
    view.load(QUrl("https://tools.usps.com/"));
    

    This implies you can connect a QWebViews loadFinished signal to a slot. You won't want the slot to execute quit(), of course, but if you can get that principle to work, and it is indeed invoked once the page has fully loaded, we will be able to get your dimensions from JavaScript there. Can you get this code to work with your own function?

    We may also need http://www.qtcentre.org/threads/37122-Detecting-finished-download-of-HTML-content-for-QWebView and http://www.qtforum.org/post/116677/convert-qwebview-to-qimage.html#post116677 and https://stackoverflow.com/a/4979636/489865

    Further: Alternatively we could have the page's JavaScript send us a signal when it is ready, e.g. from onload. It seems this is possible: see https://forum.qt.io/topic/19399/qwebview-how-to-handle-with-js-events in the present forum, QWebView::addToJavaScriptWindowObject, http://doc.qt.io/archives/qt-5.5/qtwebkit-bridge.html .



  • I'm using QT-Creator4, I believe it's version 4.8 :)
    And no, I don't use any QML, I didn't find the time to check it out yet.
    To be honest, I believe a 'loadFinished'-signal which triggers the further events would be a perfect answer, sadly I don't fully understand the signal/slot-system and how I could create my own 'Slot' (a function?) to react on the 'loadFinished'-signal.
    My lack of deep C++ knowledge is not really helping me at this point.

    Have you got the basic knowledge to accomplish this?
    How I manage to add custom entries to the "Signal & Slots Editor" ? :)

    Big thanks for all the help, realy!


    I still have to check some of your links, just so you know. :)



  • @ShivaHaze
    I'm afraid I do my Qt stuff via PyQt in Python. No C++ there. And I don't sit in a Qt environment like you do --- no Qt Creator, I don't know what "Signal & Slots Editor" is. Plus, I'm a beginner at Qt. Other than that... ;-)

    The loadFinished signal would probably be the simplest, but I have a seen a suggestion http://www.qtcentre.org/threads/37122-Detecting-finished-download-of-HTML-content-for-QWebView that it represents when the server considers it has sent all the HTML to the client which is not necessarily the same as when the client has finished its full initialisation, which is it you need for what you have to wait for it to recalculate. It could still be a bit too early. QWebView::addToJavaScriptWindowObject principle should be more robust, but probably even more coding.

    To do what I am saying for loadFinished signal or for QWebView::addToJavaScriptWindowObject you are going to need to do C++ & signals & slots, I think, sorry.

    If you wish to restart your question, either here in the hope of getting any experts to comment (and I will keep quiet) or at stackoverflow, I think the question you need is like "How can I get a JavaScript value out from a QWebView which is only valid after the page has fully finished its rendering?". Or something like that :)

    Or, if someone could tell you if it's possible to have your program "sleep" for a bit while the WebView continues to load, then you could just do the evaluateJavaScript(), that would be really simple and you'd be done! You said somewhere you had tried some QSleep() or something, maybe a different function could do it simply? OIC, I don't think you actually tried getting that QTimer/QSleep working, did you?? I think you're going to need to learn slots etc. to get what you want for this exercise! :)

    OK you need to try either void QThread::sleep(unsigned long secs) if that works, I don't know, before your evaluateJavaScript, or you have to get something like QTimer::singleShot(2000, this, SLOT(yourSlot())); working (where yourSlot() is a function you write and that is where you do the evaluateJavaScript, and the rest of your code, so it actually all runs from an event 2 seconds later). You are attaching a function to a slot, for timer signal. If this "sleep" principle works it's way simpler than alternatives.



  • @ShivaHaze
    To test whether we can apply a really simple fix, take the principle from https://stackoverflow.com/a/20894436/489865 and put in a delay between the 2 evaluateJavaScript()s in the test:

    evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");
    
    QTime dieTime= QTime::currentTime().addSecs(5);
    while (QTime::currentTime() < dieTime)
        QCoreApplication::processEvents(QEventLoop::AllEvents, 100);
    
    evaluateJavaScript("window.alert('Called directly');");
    

    This will result in a 5 second delay. If you then find that you get the Called from onload message before the Called directly we are in business! In that case, we would change the precise delay code, as that posted example is not "right", but it's a quick way to see if the principle works...?



  • @JNBarchan I'm at work right now so I can't test too much, but from what I see the program is def. waiting. :-)
    I guess applying the delay to the right point will do the trick, but I can only verify that later today. :-)
    Really interested to try this out, I'll keep you up-to-date!



  • @ShivaHaze
    The delay principle will only work provided that the web view continues to load/process during the delay. That is what I am hoping the QCoreApplication::processEvents() will allow....



  • @JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    @ShivaHaze
    The delay principle will only work provided that the web view continues to load/process during the delay. That is what I am hoping the QCoreApplication::processEvents() will allow....

    You are a god to me - Thanks so much!
    It works perfectly!
    1 second delay is enough to get the job done. :-)

    Whoever is interested in the full code - here is my full mainwindow.cpp - the only file I really touched until this point. :-)

    #include "mainwindow.h"
    #include "ui_mainwindow.h"
    
    #include <string>
    #include <iostream>
    #include <stdlib.h>
    #include <stdio.h>
    
    #include <QMainWindow>
    #include <QApplication>
    #include <Qt>
    #include <QDebug>
    #include <QWebView>
    #include <QWebPage>
    #include <QWebFrame>
    #include <QtWebKit>
    #include <QScrollBar>
    #include <QScrollArea>
    #include <QString>
    #include <QTimer>
    
    using namespace std;
    
    MainWindow::MainWindow(QWidget *parent) :
      QMainWindow(parent),
      ui(new Ui::MainWindow)
    {
      ui->setupUi(this);
    
      // Configurating Application-Window
      setWindowFlags(Qt::Window | Qt::WindowMinimizeButtonHint | Qt::WindowCloseButtonHint);
      setFixedSize(width(), height());
    
      // Object-Reference Shortcuts
      QWebFrame *MF = ui->webView->page()->mainFrame();
    
      // Deactivating Scrolling-Bars
      MF->setScrollBarPolicy(Qt::Horizontal, Qt::ScrollBarAlwaysOff);
      MF->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);
    
      QTime dieTime= QTime::currentTime().addSecs(1);
      while (QTime::currentTime() < dieTime)
          QCoreApplication::processEvents(QEventLoop::AllEvents, 100);
    
      QVariant bodytop  = MF->evaluateJavaScript("document.body.getBoundingClientRect().top");
      QVariant bodyleft = MF->evaluateJavaScript("document.body.getBoundingClientRect().left");
      QVariant elemtop  = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().top");
      QVariant elemleft = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().left");
    
      qDebug() << "Body Top: " << bodytop.toInt() << " Body Left: " << bodyleft.toInt() << " Elem Top: " << elemtop.toInt() << " Elem Left: " << elemleft.toInt();
    }
    
    MainWindow::~MainWindow()
    {
      delete ui;
    }
    

    Here you can see the console output before and after applying the delay. :-)

    Console output before & after



  • @ShivaHaze
    @ShivaHaze said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

    You are a god to me - Thanks so much!

    I think you might have meant "good" rather than "god" ;-)

    I am very pleased this approach has worked for you. It is potentially useful for me to know for my own work one day. Had I realized you had not tried the "delay" principle earlier, we would have got there quicker.

    The actual implementation of the delay loop is "naughty". It will mean that your application is "running busily" (i.e. using CPU time, potentially blocking other applications) for most of 1 second. If an expert here looks at it, please don't shout at me! Like I said, I got it from elsewhere as "quick & dirty". For your purposes it's probably fine, and we achieved it without you having to do slots & signals which you were not keen on learning at this stage.

    If you feel like you want to improve/experiment, in the same stackoverflow thread have a look at https://stackoverflow.com/a/3756341/489865, leveraging QWaitCondition, whose description sounds like it should have the same effect without the "busy blocking":

    class SleepSimulator{
         QMutex localMutex;
         QWaitCondition sleepSimulator;
    public:
        SleepSimulator::SleepSimulator()
        {
            localMutex.lock();
        }
        void sleep(unsigned long sleepMS)
        {
            sleepSimulator.wait(&localMutex, sleepMS);
        }
        void CancelSleep()
        {
            sleepSimulator.wakeAll();
        }
    };
    

    Certainly I would try it. If you copied that class into your code, you would call it (I assume) via:

    SleepSimulator* sleep = new SleepSimulator();
    sleep->sleep(1000);
    delete sleep;
    

    Now I think you can get down to the real coding you want to do for your application! Best of luck.



Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.