Is there a clear way to parse HTML in Qt 5.7

lmofallis

Hi,

I would like a powerfull HTML parser working with Qt C++ (I'm now with Qt 5.7). I'm really tired of reading a lot of articles, but without finding a clair and recent Parcer.
I found libxml2 v2.9.4 but clear examples are rare. Also, I readed about QtWebKit but it's not supported with Qt 5.7 as I understand.

I'm an amateur programmer with VB.NET in that I can use the good "HTML Agility Pack".

What I want is a parser that:

working in windows and linux OS.
supporting at list HTML4 (HTML5 can be perfect).
don't need a control or a viewer to work.
having simple tutorials or examples.

I found also QXmlQuery, And I want to know if is it a good HTML parser.

Really, I'm tired of looking more.
Thank you.

SGaist

Hi and welcome to devnet,

QtWebEngine is the current module for web related stuff
or, alternatively, @Konstantin-Tokarev QtWebKit reboot if you prefer QtWebKit.

lmofallis

@SGaist Thank you.

I migrated from VB.Net to Qt C++ sinse 3 month. So I still a newbie.
I want to know if QtWebEngine let me to parse directly HTML source (from URL or a file) to extract or remove wanted data without rendering it in control (like webbrowser in .NET).

cochise

Hi.
You can try html-qt, which ps a phraser, not a web engine.
https://github.com/cutelyst/html-qt
Sadly there is nor a good documentation of it. =[.

lmofallis

@cochise Thank you.
Is it a good parser?

I will try it.

cochise

I cant really answer that, as I never used a HTML parser myself. I'm aware of it because it is a spin of of a project I use, Cutelyst, a web framework. I use HTML as output, via grantlee templates, not as input or woring format.

Konstantin Tokarev

@lmofallis said in Is there a clear way to parse HTML in Qt 5.7:

Hi,

I would like a powerfull HTML parser working with Qt C++ (I'm now with Qt 5.7). I'm really tired of reading a lot of articles, but without finding a clair and recent Parcer.
I found libxml2 v2.9.4 but clear examples are rare.

I can confirm that libxml2 can parse HTML but don't have example handy
Also there are other parsers, for example https://github.com/google/gumbo-parser

Also, I readed about QtWebKit but it's not supported with Qt 5.7 as I understand.

Use QtWebKit if you need at least one of these things:

process DOM of web page that uses JavaScript to modify its content
use CSS queries to find interesting elements in DOM
render HTML

I'm an amateur programmer with VB.NET in that I can use the good "HTML Agility Pack".

What I want is a parser that:

working in windows and linux OS.

supporting at list HTML4 (HTML5 can be perfect).

don't need a control or a viewer to work.

having simple tutorials or examples.

I found also QXmlQuery, And I want to know if is it a good HTML parser.

Qt provides implementations of XML parsers (QXmlStreamReader, QDomDocument, QXmlReader) and XQuery (QtXmlPatterns). You can use any XML tools to process XHTML, i.e. HTML that is valid XML document.

Really, I'm tired of looking more.
Thank you.

lmofallis

@cochise said in Is there a clear way to parse HTML in Qt 5.7:

Cutelyst

I heared about Cutelyst, but I don't know about all its features.

lmofallis

@Konstantin-Tokarev Thank you.

So, I don't need QtWebKit, because I don't want to render anything. I only want to download a HTML source or load it from a file, and then parse it.

For implementated XML parsers with Qt, are they also good as libxml2 or gumbo-parser?
Meanwhile for your answer, I will give an other try for libxml2 and gumbo-parser.

Konstantin Tokarev

@lmofallis said in Is there a clear way to parse HTML in Qt 5.7:

@Konstantin-Tokarev Thank you.

So, I don't need QtWebKit, because I don't want to render anything. I only want to download a HTML source or load it from a file, and then parse it.

Don't underestimate value of "CSS queries" point - in case you have complex documents where you need to process only a few deeply nested elements, it can be very handy to have full-blown CSS query engine. With revived QtWebKit complex queries will even be JIT-compiled!

For implementated XML parsers with Qt, are they also good as libxml2 or gumbo-parser?

"Good" is fuzzy term. You may like API of Qt parsers more, but speed may be worse than with others. QDom has abysmal performance, QXmlStreamReader is much faster but still has no way around conversion of all document text to UTF16 internally which hurts performance if your document is e.g. in UTF8

Just in case you want lightning fast XML DOM parser, try pugixml

Meanwhile for your answer, I will give an other try for libxml2 and gumbo-parser.

lmofallis

@Konstantin-Tokarev

I agree with about QtWebKit. But, what about the performances?
Sometimes I have to process with more than a thousand file, and I think it's not a good idea to do this job with QtWebKit as I understand.

99% of case I work with documents encoded with ANSI or UTF-8, therefore for QXmlStreamReader will be good for its speed.

Thanks.

Konstantin Tokarev

@lmofallis HTML parser in WebKit is heavily optimized. If QtWebKit is appropriate for your task (i.e., your task matches one or more reasons listed above), I'd recommend to make a benchmark. Note that QtWebKit will take additional time to initialize and deinitialize, and use a bit more memory (but constant if you disable caches), so benchmark should involve parsing a large number of documents in a cycle.

Konstantin Tokarev

@lmofallis said in Is there a clear way to parse HTML in Qt 5.7:

99% of case I work with documents encoded with ANSI or UTF-8, therefore for QXmlStreamReader will be good for its speed.

It will be good for WebKit, it has fast path for ASCII text. With QXmlStreamReader you will end up converting all text going through parser to UTF16. Also note that QXmlStreamReader won't parse HTML documents that are not valid XML.

lmofallis

@Konstantin-Tokarev

I apologize for my late answer.
Finaly, I used the QGumboParser library, and it works fine until now. But, I don't know how can I remove a entire Tag (OuterHTML) and its children.

For example, I want to delete <div class="content"> from this code:

<html>
   <body>
   	<h3>First header</h3>
   	<p>text text text</p>
   	<div class="content">
   		<h3>Nested header <a href="">My Link</a></h3>
   	</div>
   </body>
</html>

The result:

<html>
   <body>
   	<h3>First header</h3>
   	<p>text text text</p>
   </body>
</html>

Thank you.

dantti

@cochise said in Is there a clear way to parse HTML in Qt 5.7:

https://github.com/cutelyst/html-qt

This is a bit of an old thread but as @cochise said html-qt is an HTML parser, sadly I didn't finish it yet but it follows WHATWG specification on how to implement an HTML parser as HTML is not XML. It's mostly complete but outputting a DOM tree isn't ready yet, so help is welcome.