[Solved] downloading gzip file but it's no longer compressed when I try to open the local file

qttrekker

I'm attempting to develop a qt application that will enable users to download specific files from a server. Many of the remote files will be in gzip format. Most are downloaded successfully but occasionally when a gzip file is downloaded it ends up being saved locally as a text file.

For example, I have a download queue of 3 urls that point to gzip files and I want to save them locally using the same file name as on the server. After they've been downloaded I try to open each one using file-roller and 2 of them are fine but the third one doesn't appear to be in gzip format. I then rename the .gz file to .txt and it opens successfully in a text editor. However, if I download that same file using a web browser it remains in gzip format. How can the compression be lost when downloading a .gz file? Any idea what is happening?

Before a QNetworkRequest is created I first remove the local file if it exists.
@
if (netQueueFile.exists()) netQueueFile.remove();
@

When the readyRead signal is called I use the following code to save the file.
@
QByteArray data = netCurrentReply->readAll();
if (!netQueueFile.open(QIODevice::WriteOnly | QIODevice::Append)) {
return;
}

    netQueueFile.write(data);
    netQueueFile.close();

@

qttrekker

Here's an example of the code I'm using.
@
void Example::startDownload()
{
currentReply = 0;
connect(&manager, SIGNAL(finished(QNetworkReply*)),
this, SLOT(finished(QNetworkReply*)));
QFile netQueueFile("/tmp/file.gz");
if (netQueueFile.exists()) netQueueFile.remove();
QUrl url("http://example.com/file.gz");
get(url);
}

void Example::get(const QUrl &url)
{
QNetworkRequest request(url);
if (currentReply) {
currentReply->disconnect(this);
currentReply->deleteLater();
}
currentReply = manager.get(request);
connect(currentReply, SIGNAL(readyRead()), this, SLOT(readyRead()));
connect(currentReply, SIGNAL(metaDataChanged()), this, SLOT(metaDataChanged()));
connect(currentReply, SIGNAL(error(QNetworkReply::NetworkError)), this,
SLOT(error(QNetworkReply::NetworkError)));
}

void Example::metaDataChanged()
{
QUrl redirectionTarget = currentReply->attribute(QNetworkRequest::RedirectionTargetAttribute).toUrl();
if (redirectionTarget.isValid()) {
get(redirectionTarget);
}
}

void Example::readyRead()
{
int statusCode = currentReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
if (statusCode >= 200 && statusCode < 300) {
if (!netQueueFile.open(QIODevice::WriteOnly)) {
return;
}
QByteArray data = currentReply->readAll();
netQueueFile.write(data);
netQueueFile.close();
}
}

void Example::finished(QNetworkReply *reply)
{
Q_UNUSED(reply);
}

void Example::error(QNetworkReply::NetworkError)
{
qWarning("error retrieving file");
currentReply->disconnect(this);
currentReply->deleteLater();
currentReply = 0;
}
@

The gzip files that I need to download are simply compressed text files. I've compared the http headers between a .gz file that is successfully downloaded and one that loses its compression during the download and the only real difference appears to be the mime type. Both requests are redirected to new urls but I don't think the problem is with the redirection. The working .gz file is sent from the server with the "application/octet-stream" mime type. The .gz file that loses compression is sent with the "application/x-gzip" mime type. I've tried using the setHeader() method on the QNetworkRequest to force it to be downloaded as "application/octet-stream" but it doesn't seem to make any difference.

I've also tried to use qUncompress on the QByteArray received from the QNetworkReply and it apparently doesn't support gzip compression which makes this situation even more puzzling. How is it possible to download the same .gz file using Firefox successfully and yet when I download it with the code above in my qt application it somehow decompresses during the download? I've tried writing the file from the finished() method instead of the readyRead() method but the same problem occurs. It's saved as a text file without any compression. Any suggestions would be greatly appreciated.

goetz

Your readyRead function is faulty.

The data may come in in multiple chunks. Your file only contains the last received chunk because you reopen the file every time and do not append the data.

qttrekker

I've tried opening with the append flag but it doesn't make any difference. My original code contained the append flag (see my first post above) but I still lose the gzip compression when I save the QByteArray. I've also tried writing to the local file from the finished() method instead of readyRead() but the same problem occurs. Why would this problem occur with one gzip file but not another?
@
readyRead() {
int statusCode = netCurrentReply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt();
if (statusCode >= 200 && statusCode < 300) {
if (!netQueueFile->open(QIODevice::WriteOnly | QIODevice::Append)) {
return;
}
QByteArray data = currentReply->readAll();
netQueueFile->write(data);
netQueueFile->close();
}
}

finished(QNetworkReply *reply)
{
if (!netQueueFile->open(QIODevice::WriteOnly | QIODevice::Append)) {
return;
}
QByteArray data = reply->readAll();
netQueueFile->write(data);
netQueueFile->close();
}
@

goetz

From a quick look at the Qt sources, it seems that QNetworkAccessmanager and friends automatically decodes (uncompresses) gzipped data, if the header "content-encoding" is set to "gzip".

It seems it is a bug in your server implementation to return such encoding values if you do not expect them.

You might force things by playing around with setting a value for "accept-encoding" header.

qttrekker

The accept-encoding trick fixed it. After the QNetworkRequest is created I set the accept-encoding header to gzip and it successfully downloads and saves the compressed data. Many thanks for the suggestion.
@
QNetworkRequest request(url);
request.setRawHeader("Accept-Encoding", "gzip");
@