'evaluateJavaScript()' doesn't return any value. (content position on page wanted)

ShivaHaze

I need the variables available to qt because I want to resize my browser window to only display the captcha of a website.
When the captcha changes in size, the qwebview ( & Application itself) has to adapt to the new size.

What possibilities do I have to make my qt application wait?
The best case would be if I could "wait for the website to load".
Otherwise I need some kind of "general sleep method", a few seconds should be enough in most cases which is good enough for me at the moment.

I changed my code and now 'nearly' receive the values I need, the evaluateJavascript lines seem to be executed to early which is why I need my program to wait for the website to finish loading. I receive false values which are cleary occuring because the website hasn't finished loading. ( via alert later on I receive the right values, due to window.onload)

#include "mainwindow.h"
#include "ui_mainwindow.h"

#include <string>
#include <iostream>
#include <stdlib.h>
#include <stdio.h>

#include <QMainWindow>
#include <QApplication>
#include <Qt>
#include <QDebug>
#include <QWebView>
#include <QWebPage>
#include <QWebFrame>
#include <QtWebKit>
#include <QScrollBar>
#include <QScrollArea>
#include <QString>

using namespace std;

MainWindow::MainWindow(QWidget *parent) :
    QMainWindow(parent),
    ui(new Ui::MainWindow)
{
    ui->setupUi(this);

    // Configurating Application-Window
    setWindowFlags(Qt::Window | Qt::WindowMinimizeButtonHint | Qt::WindowCloseButtonHint);
    setFixedSize(width(), height());

    // Object-Reference Shortcuts
    QWebFrame *MF = ui->webView->page()->mainFrame();

    // Deactivating Scrolling-Bars
    //MF->setScrollBarPolicy(Qt::Horizontal, Qt::ScrollBarAlwaysOff);
    //MF->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);

    // Waiting for Browser to load Website & harvesting target position
    //QVariant test  = MF->evaluateJavaScript("setTimeout(function(){}, 2000);");
    //    QTimer::singleShot(2000, this, SLOT(yourSlot()));

    QVariant bodytop  = MF->evaluateJavaScript("document.body.getBoundingClientRect().top");
    QVariant bodyleft = MF->evaluateJavaScript("document.body.getBoundingClientRect().left");
    QVariant elemtop  = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().top");
    QVariant elemleft = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().left");


    //MF->evaluateJavaScript("window.onload = function() {window.scrollTo(0,0)}");

    qDebug() << "Body Top: " << bodytop.toInt() << " Body Left: " << bodyleft.toInt() << " Elem Top: " << elemtop.toInt() << " Elem Left: " << elemleft.toInt();
}

MainWindow::~MainWindow()
{
    delete ui;
}

As values in my console I receive

Body Top: 8 Body Left: 8 Elem Top: 0 Elem Left: 0

but I know for sure that the body top and left numbers have to be alot higher, and elem top and left are def. not zero.

P.S. I'm not a C++ Dev., the last time I tinkered with C++ is years ago - Please understand. :)

JonB

How/where are you setting the content of your webview page? Are you using a QWebEngineView?

ShivaHaze

@JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

How/where are you setting the content of your webview page? Are you using a QWebEngineView?

I am using QWebView only.
The initial URL is set via the property manager, not via code.
The Source-Code I posted is literally the only customized part, the other files I have are the typical auto-generated files. :-)

Question on the side: Can I combine QML and QT? Or include .js files to my Project on QT-Creator?
I feel incredible bound to C++ which is bugging me, I'm really not that skilled in C++ and creating new Classes with Slots and Signals etc. is something I can't do without reading into it.
This also makes the QT-Docs hard for me to understand. Often I see code snippets in the docs which are not written in C++ or are not complete, but due to the lack of skill in C++ and QT I never know what to do or how to implement those parts into my project.

JonB

@ShivaHaze
I'm afraid this now goes beyond my experience of Qt.

The point I was trying to make was: the code you want to perform in your evaluateJavaScript() requires that the document/window already be fully loaded to get correct answers, right? So you need to ensure that you only call it once you know that that has happened. (My understanding is that the web view rendering is performed asynchronously, so you don't know when that is.) I believe the results you are showing indicate the page load/render is not yet complete at the time you call evaluateJavaScript(), right?

Do you also understand that evaluateJavaScript("window.onload = ...) cannot be the expression whose return result you want? That statement simply sets a function which will be evaluated on window load complete, it does not execute the function at the time you call evaluateJavaScript. If anything you would want something like:

evaluateJavaScript("
function getOffsets()
{
  var bodyRect = ...
  ...
  return ([offsetV, offsetH])};
}
[ return ] getOffsets();
")

jsulm

@ShivaHaze said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

Can I combine QML and QT?

Yes we can :-) See http://doc.qt.io/qt-5/qtqml-cppintegration-topic.html

JonB

@ShivaHaze
I still don't see that you have tried code to verify that asynchronicity is indeed the problem, just in case I'm wrong.

If it were me, then assuming the web view/JavaScript supports alert(), I would try:

evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");
evaluateJavaScript("window.alert('Called directly');");

If you see message Called directly followed by message Called from onload then asynchronicity is indeed the problem. What do you get?

ShivaHaze

@JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");
evaluateJavaScript("window.alert('Called directly');");

'Called directly' first, then 'Called from onload' :/

JonB

@ShivaHaze
Yikes :( Looks like what I am saying is correct then, do you understand where the problem lies in what I'm trying to explain?

ShivaHaze

@JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

Yikes :) Looks like what I am saying is correct then, do you understand where the problem lies in what I'm trying to explain?

Yes I believe I understand you, the evaluateJavascript call isn't waiting for the website to load even tho I need it to wait.

But what can I do to make evaluateJavascript wait?
I tried to find a simple sleep function within qt but wasn't successful yet.

JonB

@ShivaHaze
Yep, you get the principle. We need to execute your code only after page has finished onload, and get the answer back to your application...

TBH, it doesn't look good to me! (Remember, I am not a Qt professional, I am new to it, so unless someone knows better...) I see about 2 possibilities. Neither one is nice....

You are using QtWebKit, and QWebFrame etc. What version of Qt are you using? How would you like to change over to QtWebEngine? :) (See http://doc.qt.io/qt-5/qtwebenginewidgets-qtwebkitportingguide.html) The calls work differently there, it might offer a solution, but I couldn't guarantee it wouldn't behave the same...
http://www.purplealienplanet.com/node/24 implies that Qt framework allows JavaScript to access your C++ objects, the opposite way round from evaluateJavaScript. It says:

This is done using the addToJavaScriptWindowObject() method. The first argument to this method is the name under which it will be available on the JavaScript side, the second is the actual C++ object you want to embed

In principle that could be used to allow the JS code to "push" the answer to the C++ code as & when the JS is ready. However, that will mean re-architecting your code as it will only receive the result asynchronously.

I did think you might not like it. Sorry! I will still think of alternatives....

JonB

@ShivaHaze
OK, hold on! :)

The trouble is, I think you are using QML (right?), and I don't know about that. Somewhere something is setting the HTML of the page --- effectively like QWebView::setHtml(). We need to know when that is "complete" (trusting that includes the full loading of your page, i.e. it would come after onload would execute in the client).

Now, I think you are saying you can still write a bit more code to add what you have. Take a look at https://forum.qt.io/topic/27773/qwebview-wait-load/6

QWebView view;
view.show();
QObject::connect(&view,SIGNAL(loadFinished(bool)),&loop,SLOT(quit()));
view.load(QUrl("https://tools.usps.com/"));

This implies you can connect a QWebViews loadFinished signal to a slot. You won't want the slot to execute quit(), of course, but if you can get that principle to work, and it is indeed invoked once the page has fully loaded, we will be able to get your dimensions from JavaScript there. Can you get this code to work with your own function?

We may also need http://www.qtcentre.org/threads/37122-Detecting-finished-download-of-HTML-content-for-QWebView and http://www.qtforum.org/post/116677/convert-qwebview-to-qimage.html#post116677 and https://stackoverflow.com/a/4979636/489865

Further: Alternatively we could have the page's JavaScript send us a signal when it is ready, e.g. from onload. It seems this is possible: see https://forum.qt.io/topic/19399/qwebview-how-to-handle-with-js-events in the present forum, QWebView::addToJavaScriptWindowObject, http://doc.qt.io/archives/qt-5.5/qtwebkit-bridge.html .

ShivaHaze

I'm using QT-Creator4, I believe it's version 4.8 :)
And no, I don't use any QML, I didn't find the time to check it out yet.
To be honest, I believe a 'loadFinished'-signal which triggers the further events would be a perfect answer, sadly I don't fully understand the signal/slot-system and how I could create my own 'Slot' (a function?) to react on the 'loadFinished'-signal.
My lack of deep C++ knowledge is not really helping me at this point.

Have you got the basic knowledge to accomplish this?
How I manage to add custom entries to the "Signal & Slots Editor" ? :)

Big thanks for all the help, realy!

I still have to check some of your links, just so you know. :)

JonB

@ShivaHaze
I'm afraid I do my Qt stuff via PyQt in Python. No C++ there. And I don't sit in a Qt environment like you do --- no Qt Creator, I don't know what "Signal & Slots Editor" is. Plus, I'm a beginner at Qt. Other than that... ;-)

The loadFinished signal would probably be the simplest, but I have a seen a suggestion http://www.qtcentre.org/threads/37122-Detecting-finished-download-of-HTML-content-for-QWebView that it represents when the server considers it has sent all the HTML to the client which is not necessarily the same as when the client has finished its full initialisation, which is it you need for what you have to wait for it to recalculate. It could still be a bit too early. QWebView::addToJavaScriptWindowObject principle should be more robust, but probably even more coding.

To do what I am saying for loadFinished signal or for QWebView::addToJavaScriptWindowObject you are going to need to do C++ & signals & slots, I think, sorry.

If you wish to restart your question, either here in the hope of getting any experts to comment (and I will keep quiet) or at stackoverflow, I think the question you need is like "How can I get a JavaScript value out from a QWebView which is only valid after the page has fully finished its rendering?". Or something like that :)

Or, if someone could tell you if it's possible to have your program "sleep" for a bit while the WebView continues to load, then you could just do the evaluateJavaScript(), that would be really simple and you'd be done! You said somewhere you had tried some QSleep() or something, maybe a different function could do it simply? OIC, I don't think you actually tried getting that QTimer/QSleep working, did you?? I think you're going to need to learn slots etc. to get what you want for this exercise! :)

OK you need to try either void QThread::sleep(unsigned long secs) if that works, I don't know, before your evaluateJavaScript, or you have to get something like QTimer::singleShot(2000, this, SLOT(yourSlot())); working (where yourSlot() is a function you write and that is where you do the evaluateJavaScript, and the rest of your code, so it actually all runs from an event 2 seconds later). You are attaching a function to a slot, for timer signal. If this "sleep" principle works it's way simpler than alternatives.

JonB

@ShivaHaze
To test whether we can apply a really simple fix, take the principle from https://stackoverflow.com/a/20894436/489865 and put in a delay between the 2 evaluateJavaScript()s in the test:

evaluateJavaScript("window.onload = function () { window.alert('Called from onload'); }");

QTime dieTime= QTime::currentTime().addSecs(5);
while (QTime::currentTime() < dieTime)
    QCoreApplication::processEvents(QEventLoop::AllEvents, 100);

evaluateJavaScript("window.alert('Called directly');");

This will result in a 5 second delay. If you then find that you get the Called from onload message before the Called directly we are in business! In that case, we would change the precise delay code, as that posted example is not "right", but it's a quick way to see if the principle works...?

ShivaHaze

@JNBarchan I'm at work right now so I can't test too much, but from what I see the program is def. waiting. :-)
I guess applying the delay to the right point will do the trick, but I can only verify that later today. :-)
Really interested to try this out, I'll keep you up-to-date!

JonB

@ShivaHaze
The delay principle will only work provided that the web view continues to load/process during the delay. That is what I am hoping the QCoreApplication::processEvents() will allow....

ShivaHaze

@JNBarchan said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

@ShivaHaze
The delay principle will only work provided that the web view continues to load/process during the delay. That is what I am hoping the QCoreApplication::processEvents() will allow....

You are a god to me - Thanks so much!
It works perfectly!
1 second delay is enough to get the job done. :-)

Whoever is interested in the full code - here is my full mainwindow.cpp - the only file I really touched until this point. :-)

#include "mainwindow.h"
#include "ui_mainwindow.h"

#include <string>
#include <iostream>
#include <stdlib.h>
#include <stdio.h>

#include <QMainWindow>
#include <QApplication>
#include <Qt>
#include <QDebug>
#include <QWebView>
#include <QWebPage>
#include <QWebFrame>
#include <QtWebKit>
#include <QScrollBar>
#include <QScrollArea>
#include <QString>
#include <QTimer>

using namespace std;

MainWindow::MainWindow(QWidget *parent) :
  QMainWindow(parent),
  ui(new Ui::MainWindow)
{
  ui->setupUi(this);

  // Configurating Application-Window
  setWindowFlags(Qt::Window | Qt::WindowMinimizeButtonHint | Qt::WindowCloseButtonHint);
  setFixedSize(width(), height());

  // Object-Reference Shortcuts
  QWebFrame *MF = ui->webView->page()->mainFrame();

  // Deactivating Scrolling-Bars
  MF->setScrollBarPolicy(Qt::Horizontal, Qt::ScrollBarAlwaysOff);
  MF->setScrollBarPolicy(Qt::Vertical, Qt::ScrollBarAlwaysOff);

  QTime dieTime= QTime::currentTime().addSecs(1);
  while (QTime::currentTime() < dieTime)
      QCoreApplication::processEvents(QEventLoop::AllEvents, 100);

  QVariant bodytop  = MF->evaluateJavaScript("document.body.getBoundingClientRect().top");
  QVariant bodyleft = MF->evaluateJavaScript("document.body.getBoundingClientRect().left");
  QVariant elemtop  = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().top");
  QVariant elemleft = MF->evaluateJavaScript("document.getElementById('user_reg').getBoundingClientRect().left");

  qDebug() << "Body Top: " << bodytop.toInt() << " Body Left: " << bodyleft.toInt() << " Elem Top: " << elemtop.toInt() << " Elem Left: " << elemleft.toInt();
}

MainWindow::~MainWindow()
{
  delete ui;
}

Here you can see the console output before and after applying the delay. :-)

Console output before & after

JonB

@ShivaHaze
@ShivaHaze said in 'evaluateJavaScript()' doesn't return any value. (content position on page wanted):

You are a god to me - Thanks so much!

I think you might have meant "good" rather than "god" ;-)

I am very pleased this approach has worked for you. It is potentially useful for me to know for my own work one day. Had I realized you had not tried the "delay" principle earlier, we would have got there quicker.

The actual implementation of the delay loop is "naughty". It will mean that your application is "running busily" (i.e. using CPU time, potentially blocking other applications) for most of 1 second. If an expert here looks at it, please don't shout at me! Like I said, I got it from elsewhere as "quick & dirty". For your purposes it's probably fine, and we achieved it without you having to do slots & signals which you were not keen on learning at this stage.

If you feel like you want to improve/experiment, in the same stackoverflow thread have a look at https://stackoverflow.com/a/3756341/489865, leveraging QWaitCondition, whose description sounds like it should have the same effect without the "busy blocking":

class SleepSimulator{
     QMutex localMutex;
     QWaitCondition sleepSimulator;
public:
    SleepSimulator::SleepSimulator()
    {
        localMutex.lock();
    }
    void sleep(unsigned long sleepMS)
    {
        sleepSimulator.wait(&localMutex, sleepMS);
    }
    void CancelSleep()
    {
        sleepSimulator.wakeAll();
    }
};

Certainly I would try it. If you copied that class into your code, you would call it (I assume) via:

SleepSimulator* sleep = new SleepSimulator();
sleep->sleep(1000);
delete sleep;

Now I think you can get down to the real coding you want to do for your application! Best of luck.