Read data from website

appollosputnik

In my website there is a table. I want to read the data from the 1st row 1st column of the table. Please tell me how can I read ? I have my QWebView. Please help. Thanks

[code]
connect(m_WebView, SIGNAL(loadFinished(bool)), this, SLOT(LoadFinishedCB(bool)));

m_WebView->setUrl(QUrl("http://www.mywebsite.com"));

[/code]

In LoadFinishedCB I want to read the data. Please help.

KA51O

something like this maybe?
@
QWebElementCollection AllFirstColumnEntries =m_WebView->page()->mainFrame()->findAllElements("table.TheTableYoureLookingFor td.FirstColumn"); // this returns all cells in the first column
@
this works for a table implemented like this:
@
<table class="TheTableYoureLookingFor" >
<tr>
<td class="FirstColumn"></td>
<td class="SecColumn"></td>
</tr>
<tr>
<td class="FirstColumn"></td>
<td class="SecColumn"></td>
</tr>
<tr>
<td class="FirstColumn"></td>
<td class="SecColumn"></td>
</tr>
</table>
@

appollosputnik

Please check the code snippet below. I am trying to click on link in the page twice and then a table is loaded i want to read one data from the table from 1st row and 2nd column. Please help.
[code]
void MainWindow::LoadFinishedCB(bool ok)
{
QFile file("out.txt");
if (!file.open(QIODevice::WriteOnly))
return;
QTextStream out(&file);
button_click = m_WebView->page()->mainFrame()->documentElement().findAll("a");
qDebug() << button_click.count() << "\n";

button_click.at(5).evaluateJavaScript(
            "var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
            );


button_click.at(48).evaluateJavaScript(
            "var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
            );

//The above piece loads a new link in the same page.
//At this stage how can I make the program to wait till the page is //refreshed ???

out << m_WebView->page()->currentFrame()->toHtml() << "\n";

QWebElementCollection tables = m_WebView->page()->currentFrame()->findAllElements("table [id=layer_showhost]");
qDebug() << tables.count() << "\n";

QWebElementCollection trs = tables.first().document().findAll("tr");
qDebug() << trs.count() << "\n";


for(int m=0; m<trs.count(); m++)
{
  QWebElementCollection tds = trs.at(m).document().findAll("td");
  qDebug() << tds.count() << "\n";

  for(int n=0; n<tds.count(); n++)
  {
      QWebElementCollection spans = tds.at(n).document().findAll("span");
      qDebug() << spans.count() << "\n";

      for(int z=0; z<spans.count(); z++)
      {
          out << spans.at(z).document().toPlainText() << " ";
      }
  }

}

}
[/code]

KA51O

The code you posted just prints out the number of tables found, the number of table rows found in the first table, the number of table cells for each row in the first table and the text found enclosed by a span tag in each of these cells.

What exactly do you want from me now? Am I supposed to modify that piece of code you may have pulled somewhere of the internet so that it does what you specified above?

Why don't you try to do that, its really not that complicated. If you have problems I will gladly help you, after you posted the modified code or the problems you encountered.

Edit:
Just read the comment in your code
[quote]
The above piece loads a new link in the same page.
At this stage how can I make the program to wait till the page is refreshed ???
[/quote]
Just connect a custom slot (in which you could do all the stuff you do after the comment I quoted above) to the webviews signal loadFinished(bool ok).

For example:
@
button_click.at(48).evaluateJavaScript(
"var evObj = document.createEvent('MouseEvents');evObj.initEvent( 'click', true, true );this.dispatchEvent(evObj);"
);
//The above piece loads a new link in the same page.
//At this stage how can I make the program to wait till the page is //refreshed ???

//if you have another slot already connected to the signal loadFinished disconnect this slot first
// you can reconnect it later
connect(m_WebView, SIGNAL(loadFinished(bool), this, SLOT(doTheTableParsing()));
}

void YourClass::doTheTableParsing()
{
// here you can reconnect to the loadFinished() signal but first disconnect it from the new custom slot
disconnect(m_WebView, SIGNAL(loadFinished(bool), this, SLOT(doTheTableParsing()));
out << m_WebView->page()->currentFrame()->toHtml() << "\n";

QWebElementCollection tables = m_WebView->page()->currentFrame()->findAllElements("table [id=layer_showhost]");
qDebug() << tables.count() << "\n";

QWebElementCollection trs = tables.first().document().findAll("tr");
qDebug() << trs.count() << "\n";

if(trs.size() > 0)
{
QWebElementCollection tdsInFirstTr = trs.at(0).document().findAll("td");
if(tdsInFirstTr.size() > 0)
{
QWebElementCollection spanInSecondTDInFirstTR = tdsInFirstTr.at(1).document().findAll("span");
if(spanInSecondTDInFirstTR .size() > 0)
{
qDebug() << spanInSecondTDInFirstTR .at(0).toPlainText() << "\n";
}
}
}
}
@

appollosputnik

loadFinished() signal is not being emitted second time. What can be done?

KA51O

hmm. You can try some of the signals provided by QWebFrame (loadFinished(), pageChanged(),...) or QWebPage (contentsChanged(), loadFinished(), ..), if none of them work for your use case the only option I can think of would be a QTimer::singleShot(..).

appollosputnik

ok thanks a lot friend let me try I will post again thanks

appollosputnik

Can u give me your skype id I want to discuss with you. I am facing severe problem. Totally stuck. Thanks Sujan

appollosputnik

I am writing the page as html. where the span values are question mark. no values are displayed.

[code]
out << m_WebView->page()->currentFrame()->toHtml() << "\n";
[/code]

var msg_proce = "??????...";
var msg_success = "????";
??????????,<a >????</a>

HOW CAN I GET THE SPAN VALUES. PLEASE HELP.

appollosputnik

loadFinished() signal is working friend for currentFrame().. Thanks a lot.

Please see the above post I am not able to get the span values. Please reply. Thanks Sujan

KA51O

Happy to hear that you got the signal problem solved. About the span issue, I guess we should continue the discussion in "your new post.":http://qt-project.org/forums/viewthread/29334/

appollosputnik

My problem is solved I am using a QTimer with singleshot for updating the site with click. Then it comes ok I am able to fetch data using QWebElement as per your suggestions. Thanks a lot friend.

Update: Forum Guidelines & Code of Conduct

Read data from website