Read data from website


  • Banned

    In my website there is a table. I want to read the data from the 1st row 1st column of the table. Please tell me how can I read ? I have my QWebView. Please help. Thanks

    [code]
    connect(m_WebView, SIGNAL(loadFinished(bool)), this, SLOT(LoadFinishedCB(bool)));

    m_WebView->setUrl(QUrl("http://www.mywebsite.com"));
    

    [/code]

    In LoadFinishedCB I want to read the data. Please help.



  • something like this maybe?
    @
    QWebElementCollection AllFirstColumnEntries =m_WebView->page()->mainFrame()->findAllElements("table.TheTableYoureLookingFor td.FirstColumn"); // this returns all cells in the first column
    @
    this works for a table implemented like this:
    @
    <table class="TheTableYoureLookingFor" >
    <tr>
    <td class="FirstColumn"><p></p></td>
    <td class="SecColumn"><p></p></td>
    </tr>
    <tr>
    <td class="FirstColumn"><p></p></td>
    <td class="SecColumn"><p></p></td>
    </tr>
    <tr>
    <td class="FirstColumn"><p></p></td>
    <td class="SecColumn"><p></p></td>
    </tr>
    </table>
    @


  • Banned

    Please check the code snippet below. I am trying to click on link in the page twice and then a table is loaded i want to read one data from the table from 1st row and 2nd column. Please help.
    [code]
    void MainWindow::LoadFinishedCB(bool ok)
    {
    QFile file("out.txt");
    if (!file.open(QIODevice::WriteOnly))
    return;
    QTextStream out(&file);
    button_click = m_WebView->page()->mainFrame()->documentElement().findAll("a");
    qDebug() << button_click.count() << "\n";

    button_click.at(5).evaluateJavaScript(
                "var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
                );
    
    
    button_click.at(48).evaluateJavaScript(
                "var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
                );
    

    //The above piece loads a new link in the same page.
    //At this stage how can I make the program to wait till the page is //refreshed ???

    out << m_WebView->page()->currentFrame()->toHtml() << "\n";
    
    QWebElementCollection tables = m_WebView->page()->currentFrame()->findAllElements("table [id=layer_showhost]");
    qDebug() << tables.count() << "\n";
    
    QWebElementCollection trs = tables.first().document().findAll("tr");
    qDebug() << trs.count() << "\n";
    
    
    for(int m=0; m<trs.count(); m++)
    {
      QWebElementCollection tds = trs.at(m).document().findAll("td");
      qDebug() << tds.count() << "\n";
    
      for(int n=0; n<tds.count(); n++)
      {
          QWebElementCollection spans = tds.at(n).document().findAll("span");
          qDebug() << spans.count() << "\n";
    
          for(int z=0; z<spans.count(); z++)
          {
              out << spans.at(z).document().toPlainText() << " ";
          }
      }
    
    }
    

    }
    [/code]



  • The code you posted just prints out the number of tables found, the number of table rows found in the first table, the number of table cells for each row in the first table and the text found enclosed by a span tag in each of these cells.

    What exactly do you want from me now? Am I supposed to modify that piece of code you may have pulled somewhere of the internet so that it does what you specified above?

    Why don't you try to do that, its really not that complicated. If you have problems I will gladly help you, after you posted the modified code or the problems you encountered.

    Edit:
    Just read the comment in your code
    [quote]
    The above piece loads a new link in the same page.
    At this stage how can I make the program to wait till the page is refreshed ???
    [/quote]
    Just connect a custom slot (in which you could do all the stuff you do after the comment I quoted above) to the webviews signal loadFinished(bool ok).

    For example:
    @
    button_click.at(48).evaluateJavaScript(
    "var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
    );
    //The above piece loads a new link in the same page.
    //At this stage how can I make the program to wait till the page is //refreshed ???

    //if you have another slot already connected to the signal loadFinished disconnect this slot first
    // you can reconnect it later
    connect(m_WebView, SIGNAL(loadFinished(bool), this, SLOT(doTheTableParsing()));
    }

    void YourClass::doTheTableParsing()
    {
    // here you can reconnect to the loadFinished() signal but first disconnect it from the new custom slot
    disconnect(m_WebView, SIGNAL(loadFinished(bool), this, SLOT(doTheTableParsing()));
    out << m_WebView->page()->currentFrame()->toHtml() << "\n";

    QWebElementCollection tables = m_WebView->page()->currentFrame()->findAllElements("table [id=layer_showhost]");
    qDebug() << tables.count() << "\n";
    
    QWebElementCollection trs = tables.first().document().findAll("tr");
    qDebug() << trs.count() << "\n";
    

    if(trs.size() > 0)
    {
    QWebElementCollection tdsInFirstTr = trs.at(0).document().findAll("td");
    if(tdsInFirstTr.size() > 0)
    {
    QWebElementCollection spanInSecondTDInFirstTR = tdsInFirstTr.at(1).document().findAll("span");
    if(spanInSecondTDInFirstTR .size() > 0)
    {
    qDebug() << spanInSecondTDInFirstTR .at(0).toPlainText() << "\n";
    }
    }
    }
    }
    @


  • Banned

    loadFinished() signal is not being emitted second time. What can be done?



  • hmm. You can try some of the signals provided by QWebFrame (loadFinished(), pageChanged(),...) or QWebPage (contentsChanged(), loadFinished(), ..), if none of them work for your use case the only option I can think of would be a QTimer::singleShot(..).


  • Banned

    ok thanks a lot friend let me try I will post again thanks


  • Banned

    Can u give me your skype id I want to discuss with you. I am facing severe problem. Totally stuck. Thanks Sujan


  • Banned

    I am writing the page as html. where the span values are question mark. no values are displayed.

    [code]
    out << m_WebView->page()->currentFrame()->toHtml() << "\n";
    [/code]

    var msg_proce = "<span style=color:red>??????...</span>";
    var msg_success = "<span style=color:red>????</span>";
    <span style=font-size:15px;color:red>??????????,<a >????</a></span>

    HOW CAN I GET THE SPAN VALUES. PLEASE HELP.


  • Banned

    loadFinished() signal is working friend for currentFrame().. Thanks a lot.

    Please see the above post I am not able to get the span values. Please reply. Thanks Sujan



  • Happy to hear that you got the signal problem solved. About the span issue, I guess we should continue the discussion in "your new post.":http://qt-project.org/forums/viewthread/29334/


  • Banned

    My problem is solved I am using a QTimer with singleshot for updating the site with click. Then it comes ok I am able to fetch data using QWebElement as per your suggestions. Thanks a lot friend.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.