A desktop app to retrieve values from a site and to perform actions such as button clicking



  • Hello QtForum,
    i need a suggestion about the right path to take in order to not overcompilate my project.
    My idea is to develop a desktop application with a gui but without a webpage frame, so it means the user is not able to surf the site. The application should provide a graphical interface such as labels and buttons. Labels contain values directly retrieved on the site using GetElementsById for example while buttons allow to simulate what the real site button would normally do.
    Of course i could even inject a js script to handle the logic.

    Is QtWebEngine the right solution?
    The site will require an authentication (user and pw)
    As i said i only need to read 2/3 values and press 2 buttons on the site.

    Regards,


  • Lifetime Qt Champion

    Hi and welcome to devnet,

    Do you really need to show that web page ? Depending on the tools you have at disposition, why not just write a client that does the job to grab data from the corresponding API.



  • @SGaist Exactly, i dont really want to show the web page. The point is to have a desktop interface to a site. Which corresponding API are you referring to? Should i use QtNetworkAccessManager to perform GET/POST requests? In this case i wouldnt be able to use the DOM tree, so after issuing a GET request my only solution is to look for words or pattern in the plain text response?

    Or can i use this for example

    QWebPage page;
    page.mainFrame()->setHtml(source); // source is retrieved with GET
    QWebElement parse = page.mainFrame()->documentElement();
    QWebElement result = parse.findFirst("whatever");

    Regards,


  • Lifetime Qt Champion

    I thought your data would come from a REST API. Is it the case, or do you really need to grab a page and parse it ?



  • @SGaist How do i recognize if it is based on a REST API?
    I thought i would have to parse the entire webpage after a GET request.

    Moreover, say you have a button which loads a function when clicked , this function has a text as input (from an html label) and it encodes it in a particular way (branded MD5 and so on).
    Can i clone the POST request so that i let the site use its own function without implementing it in my c++ GUI?


  • Lifetime Qt Champion

    What page are you trying to access ?



  • Local modem interface :-) Moreover, what if the server has an antiCSRF script? is the API able to handle it as a normal browser?


  • Lifetime Qt Champion

    So the admin page of the modem ?



  • @SGaist

    Re: A desktop app to retrieve values from a site and to perform actions such as button clicking

    Yes, i was pretty much able to achieve a good point. I've used QNetworkManager with QNetworkReply.
    Now the doubt i have is about the correct design of a software which implements multiple and nested GET/POST request.

    Say you have to run a login() function which requires a GET and a POST and then retrieveXYZ() which requires two GETs (and so on, scalable).

    The way i was thinking to do it was like

    mainwindow.cpp
        //code
        login();
        retrieveXYZ();
        //code
    
    Mainwindow::login(){ 
        //code
        connect(nam, SIGNAL(finished()), this, SLOT(onGetLoginFinished()));
        nam->get(...);
    }
    
    Mainwindow::onLoginFinished(){
        //do stuff
        connect(nam, SIGNAL(finished()), this, SLOT(onPostLoginFinished()));
        nam->post(...);
    }    
    
    Mainwindow::onPostLoginFinished(){
        //do stuff
    }
    
    Mainwindow::retrieveXYZ(){
        //code
        connect(nam, SIGNAL(finished()), this, SLOT(onGet1RetrieveFinished()));
        nam->get();
        //code
    }
    
    Mainwindow::onRetrieveXYZFinished(){
        //do stuff
        connect(nam, SIGNAL(finished()), this, SLOT(onGet2RetrieveFinished()));
        nam->get();
    }    
    

    or should i use something like QSignalMapper ?
    Which are the most correct/efficient way to do so? i've seen people using sender() cast but i didn't understand the point.

    Basically i would like to retrieve the particular reply finished() signal rather than the general one (or of the qnam)

    This method may work but it is not nice and clean to me

    Is this the best we can get?

    http://www.johanpaul.com/blog/2011/07/why-qnetworkaccessmanager-should-not-have-the-finishedqnetworkreply-signal/

    Moving the connect approach to the reply?


  • Lifetime Qt Champion

    We might be overthinking that stuff. What exactly are your users supposed to be doing ?



  • There are two general ways: to use the QtWebKit or the QtWebEngine.
    There are memory usage and rendering issues with QtWebKit and it is deprecated since Qt 5.5, but its API is synchronous and simplier to use.
    In case it is possible to use the 5.5 or an older version of Qt with precompiled QWebKit or you are able to build the module for target platform(s) manually (which could be a bit tricky), just try to open target web page in the QWebView (see the demo browser in the examples). If the page is rendered properly, interaction with inputs/buttons is adequate and there are no visible crashes/freezes/memory leaking - you can use the QWebKit :)

    ...
        connect( webView->page(), //-- or webView->page()->mainFrame(), depending on the DOM structure
            SIGNAL(loadFinished(bool), this, SLOT(onWebContentLoaded(bool));
    ...
    
    void ...::onWebContentLoaded(bool ok)
    {
        if(!isLogedIn()) //-- parse HTML to find markers or just check a flag actualized in the doLogIn();
            doLogIn();
        else
            extractTheData();
    }
    
    void ...::doLogIn()
    {
        if( QWebFrame *frame = webView->page()->mainFrame() )
        {
            QWebElement inputLogin = frame->findFirstElement( "#user" ); //-- Chrome->View Source (Ctrl+Shift+I)->Copy->Copy Selector
            QWebElement inputPass = frame->findFirstElement( "#password" );
            QWebElement buttonSubmit = frame->findFirstElement( "#login" );
    
            if(!inputLogin.isNull() 
            && !inputPass.isNull()
            && !buttonSubmit.isNull() )
            {
                inputLogin.setAttribute("value", ui->lineEditLogin->text() ); //-- populate HTML's input via API
                inputPass.evaluateJavaScript( QString("this.value='%1'").arg( ui->lineEditPass->text())); //-- or via JS
                buttonSubmit.evaluateJavaScript( "this.click()" ); //-- or you can generate a key/mouse event
            }
        }
    }
    
    

    In case you are forced to use the QWebEngine, things are a bit more complicated: you have to use a QWebChannel to expose your QObject to the JS side, where all interaction/extraction will be done.

    In both cases to keep alive all routines/events needed for correct page rendering while widget is hidden, set the webView's (or its parent's) attribute Qt::WA_DontShowOnScreen to true.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.