Content-Encoding: Gzip with Qt Webkit?

ÖlaufmKeks

Hi everyone :),
I try to open a website which only supports "Content-Encoding: gzip". A custom made header with "Accept-Encoding: Identity" is completly ignored by the webserver...
If I try to open this site in a QTWebKit widget... Guess what... Nothing happens :). I already recompiled QtWebKit in its newest version but the site still won't be rendered by WebKit :(...
Is gzipped encoded content not supported in QTWebKit? Or is there another way to show this specific page with WebKit?
Thanks in advance for every answer :)...

fcrochik

Have you tried using firefox with firebug to get all the headers sent and received?
The problem may be with the server filtering the user agent or something else (not the gzip content per se).

baysmith

Chrome also has some great features for web developers, under the Tools menu and off the Inspect element in the context menu.

ÖlaufmKeks

First of all thanks four your answers so far :)...
Secondly, sorry for my late answer, another project took too much time.

So, I'm still stuck with this WebKit problem. I already recompiled WebKit and tried the QtTestBrowser on this page. QtTestBrowser throws a "Frame load interrupted by policy change", even if I change the User-Agent to Opera or any other (every other page works fine)...

Then I tried curl. All I get is scrambled output which looks like gzip encoded data.
If I run "curl --compressed http://specificsite" everything looks good with nice html output.

This page works fine with midori, chrome, opera, firefox. But aurora won't render this page too...

So I really guess this is a content-encoding: gzip problem... I'm thankful for every tip!!!

benjamin.poulain

Any chance to get and example to reproduce that?

shayz

Hi,

I have something of the same issue but I think Qt WebKit supports gzip but takes for ever to end the http load operation. For example, www.weather.com in Arora, takes 10-20 minutes to load.
After some investigations, it seems that only when the underlying TCP connection closes, Qt Network (qhttpnetworkreply.cpp and qhttpnetworkconnectionchannel.cpp) 'understands' that the page has been fully received. Qt Network disregards the content length header if the page is gzipped -
QHttpNetworkReplyPrivate::removeAutoDecompressHeader is called when the content encoding is gzip.

--shay

Thijs

I've run into the same problem and come to the same conclusion:

QtWebKit doesn't support (all?) gzip content
This is because it strips the content length is removed by QtNetwork (as shayz mentioned) or because of missing end-of-stream marker (as mentioned in the bug reports below).
For most websites you can work around this issue by using a custom NetworkAccessManager and using : req.setRawHeader("Accept-Encoding", "none") inside createRequest() function

Unfortunately there are some websites that always return gzipped content. Those are FUBAR, untill QtWebKit gets fixed :(

Thijs

For future visitors, there are some relevant bug reports:

https://bugs.webkit.org/show_bug.cgi?id=58727
http://bugreports.qt.nokia.com/browse/QTBUG-16022

Thijs

I made a small test-case which allowed me to find the cause. From the source code of qhttpnetworkconnection.cpp :

@
value = request.headerField("accept-encoding");
if (value.isEmpty()) {
#ifndef QT_NO_COMPRESS
request.setHeaderField("Accept-Encoding", "gzip");
request.d->autoDecompress = true;
#else
// if zlib is not available set this to false always
request.d->autoDecompress = false;
#endif
}
@

I believe this is a bug: If you manually set the "Accept-Encoding" header, autoDecompress becomes false (even if you set it to "gzip"). So QtWebKit will never decompress the data and it will not be able to render the page obviously.

[EDIT: code formatting, please use @-tags, not code, Volker]