A desktop app to retrieve values from a site and to perform actions such as button clicking

itsmaxdd

Hello QtForum,
i need a suggestion about the right path to take in order to not overcompilate my project.
My idea is to develop a desktop application with a gui but without a webpage frame, so it means the user is not able to surf the site. The application should provide a graphical interface such as labels and buttons. Labels contain values directly retrieved on the site using GetElementsById for example while buttons allow to simulate what the real site button would normally do.
Of course i could even inject a js script to handle the logic.

Is QtWebEngine the right solution?
The site will require an authentication (user and pw)
As i said i only need to read 2/3 values and press 2 buttons on the site.

Regards,

SGaist

Hi and welcome to devnet,

Do you really need to show that web page ? Depending on the tools you have at disposition, why not just write a client that does the job to grab data from the corresponding API.

itsmaxdd

@SGaist Exactly, i dont really want to show the web page. The point is to have a desktop interface to a site. Which corresponding API are you referring to? Should i use QtNetworkAccessManager to perform GET/POST requests? In this case i wouldnt be able to use the DOM tree, so after issuing a GET request my only solution is to look for words or pattern in the plain text response?

Or can i use this for example

QWebPage page;
page.mainFrame()->setHtml(source); // source is retrieved with GET
QWebElement parse = page.mainFrame()->documentElement();
QWebElement result = parse.findFirst("whatever");

Regards,

SGaist

I thought your data would come from a REST API. Is it the case, or do you really need to grab a page and parse it ?

itsmaxdd

@SGaist How do i recognize if it is based on a REST API?
I thought i would have to parse the entire webpage after a GET request.

Moreover, say you have a button which loads a function when clicked , this function has a text as input (from an html label) and it encodes it in a particular way (branded MD5 and so on).
Can i clone the POST request so that i let the site use its own function without implementing it in my c++ GUI?

SGaist

What page are you trying to access ?

itsmaxdd

Local modem interface :-) Moreover, what if the server has an antiCSRF script? is the API able to handle it as a normal browser?

SGaist

So the admin page of the modem ?

itsmaxdd

@SGaist

Re: A desktop app to retrieve values from a site and to perform actions such as button clicking

Yes, i was pretty much able to achieve a good point. I've used QNetworkManager with QNetworkReply.
Now the doubt i have is about the correct design of a software which implements multiple and nested GET/POST request.

Say you have to run a login() function which requires a GET and a POST and then retrieveXYZ() which requires two GETs (and so on, scalable).

The way i was thinking to do it was like

mainwindow.cpp
    //code
    login();
    retrieveXYZ();
    //code

Mainwindow::login(){ 
    //code
    connect(nam, SIGNAL(finished()), this, SLOT(onGetLoginFinished()));
    nam->get(...);
}

Mainwindow::onLoginFinished(){
    //do stuff
    connect(nam, SIGNAL(finished()), this, SLOT(onPostLoginFinished()));
    nam->post(...);
}    

Mainwindow::onPostLoginFinished(){
    //do stuff
}

Mainwindow::retrieveXYZ(){
    //code
    connect(nam, SIGNAL(finished()), this, SLOT(onGet1RetrieveFinished()));
    nam->get();
    //code
}

Mainwindow::onRetrieveXYZFinished(){
    //do stuff
    connect(nam, SIGNAL(finished()), this, SLOT(onGet2RetrieveFinished()));
    nam->get();
}

or should i use something like QSignalMapper ?
Which are the most correct/efficient way to do so? i've seen people using sender() cast but i didn't understand the point.

Basically i would like to retrieve the particular reply finished() signal rather than the general one (or of the qnam)

This method may work but it is not nice and clean to me

Is this the best we can get?

http://www.johanpaul.com/blog/2011/07/why-qnetworkaccessmanager-should-not-have-the-finishedqnetworkreply-signal/

Moving the connect approach to the reply?

SGaist

We might be overthinking that stuff. What exactly are your users supposed to be doing ?

sendevent

There are two general ways: to use the QtWebKit or the QtWebEngine.
There are memory usage and rendering issues with QtWebKit and it is deprecated since Qt 5.5, but its API is synchronous and simplier to use.
In case it is possible to use the 5.5 or an older version of Qt with precompiled QWebKit or you are able to build the module for target platform(s) manually (which could be a bit tricky), just try to open target web page in the QWebView (see the demo browser in the examples). If the page is rendered properly, interaction with inputs/buttons is adequate and there are no visible crashes/freezes/memory leaking - you can use the QWebKit :)

...
    connect( webView->page(), //-- or webView->page()->mainFrame(), depending on the DOM structure
        SIGNAL(loadFinished(bool), this, SLOT(onWebContentLoaded(bool));
...

void ...::onWebContentLoaded(bool ok)
{
    if(!isLogedIn()) //-- parse HTML to find markers or just check a flag actualized in the doLogIn();
        doLogIn();
    else
        extractTheData();
}

void ...::doLogIn()
{
    if( QWebFrame *frame = webView->page()->mainFrame() )
    {
        QWebElement inputLogin = frame->findFirstElement( "#user" ); //-- Chrome->View Source (Ctrl+Shift+I)->Copy->Copy Selector
        QWebElement inputPass = frame->findFirstElement( "#password" );
        QWebElement buttonSubmit = frame->findFirstElement( "#login" );

        if(!inputLogin.isNull() 
        && !inputPass.isNull()
        && !buttonSubmit.isNull() )
        {
            inputLogin.setAttribute("value", ui->lineEditLogin->text() ); //-- populate HTML's input via API
            inputPass.evaluateJavaScript( QString("this.value='%1'").arg( ui->lineEditPass->text())); //-- or via JS
            buttonSubmit.evaluateJavaScript( "this.click()" ); //-- or you can generate a key/mouse event
        }
    }
}

In case you are forced to use the QWebEngine, things are a bit more complicated: you have to use a QWebChannel to expose your QObject to the JS side, where all interaction/extraction will be done.

In both cases to keep alive all routines/events needed for correct page rendering while widget is hidden, set the webView's (or its parent's) attribute Qt::WA_DontShowOnScreen to true.