Zip directory recursion

[Cobra91151]

Hello!

I am working on creating zip archive using old Qt - ZipWriter class. The problem is when I want to add the directory. The default Qt code for addDirectory method - d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray());. It does not add any content, only the empty directory. So, I have improved it to add the directories and content as well.

My code:

QList<QString> dirs;
int recursion = 0;

void ZipWriter::addDirectory(const QString &dirPath)
{
    QDir archDir(dirPath);
    archDir.setFilter(QDir::AllEntries | QDir::NoDotAndDotDot);
    dirs << addDirSeparator(archDir.dirName());

    if (archDir.exists()) {
        QString archDirName = "";

        if (recursion > 0) {
            for (int i = recursion; i < dirs.count(); i++) {
                 archDirName = dirs.first().append(dirs.at(i));
            }
        } else {
            archDirName = dirs.at(recursion);
        }

        if (!archDir.isEmpty()) {
            const QStringList archFileList = archDir.entryList();

            if (archFileList.count() > 0) {
                for (QString archFile : archFileList) {
                     QFileInfo archFileInfo(QDir::toNativeSeparators(QString("%1/%2").arg(archDir.absolutePath(), archFile)));

                     if (archFileInfo.isDir()) {
                         recursion++;
                         addDirectory(archFileInfo.absoluteFilePath());
                     } else {
                         QFile zipFile(archFileInfo.absoluteFilePath());
                         zipFile.open(QIODevice::ReadOnly);
                         addFile(QString("%1%2").arg(archDirName, archFile), zipFile.readAll());
                         zipFile.close();
                     }
                }
            }
        } else {
            d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray());
        }
    }
}

Now, it adds the directory and content recursively but it has issue when directory is on the same level, it appends it to the end. I think, I must use the STL container to keep track of the directory for example QMap but the question is how to get the current directory level? Any ideas? Thank you.

[Cobra91151]

I will try to use the QDirIterator and Qt containers to solve this issue.

[Cobra91151]

So, I have created something like this:

Code:

void ZipWriter::addDirectory(const QString &dirPath)
{
    QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
    while (dirIt.hasNext()) {
        QString archDirPath = dirIt.next();
        QFile zipFile(archDirPath);
        zipFile.open(QIODevice::ReadOnly);

        if (!dirIt.fileInfo().isDir()) {
            addFile(archDirPath, zipFile.readAll());
        }

        zipFile.close();
    }
}

Now, it adds everything recursively and in the correct order but I have another issue. It adds the full path to the archive. For example, I want to add this folder and it's content to the archive: 22610.1_amd64_en-us_professional_00fb7ba0_convert. In the archive I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert. Any ideas how to make this relative path or trim it? Thank you.

[JonB]

@Cobra91151
So don't add the full path (absoluteFilePath()?) wherever you do that (in ZipWriter::addDirectory()?), if you're using QFileInfo there is QString QFileInfo::fileName() const for the plain filename, or maybe you want to make it relative to something inside what you are zipping (there is also QString QFileInfo::filePath() const which is the path you are passing to QFileInfo which may be relatve rather than absolute).

[Cobra91151]

@JonB

QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well. Also, I have tried QString QFileInfo::filePath() const but it's still the same as with archDirPath path. In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

addFile(dirIt.fileInfo().filePath(), zipFile.readAll());

I need somehow to omit the full path in the archive and get the following:

22610.1_amd64_en-us_professional_00fb7ba0_convert => then directories/files.

[JonB]

@Cobra91151 said in Zip directory recursion:

QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well

I really don't know what you mean. QFileInfo::-anything does not "add" anything at all, it's just a method to call on a filepath to find about it.

Somewhere in your code (no, I'm not going through it) there is presumably a filepath you use to open a file to read it in and store it in the zip and there is also presumably a filepath you use to store its name/path. Why do they have to be the same?

Purely at a guess this is done in your addFile() code, but I don't see that.

I am working on creating zip archive using old Qt - ZipWriter class.

I have only just noticed this. So since I know nothing about it wouldn't you have to look at its code/documentation if it's not doing whatever it is you want?

In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

If you cded to C:\Users\userProfile\Downloads and then specified relative paths, like 22610.1_amd64_en-us_professional_00fb7ba0_convert --- and did not store QFileInfo::absolutePath(), rather QFileInfo::filePath() --- would that achieve what you want?

Note that this issue of relative versus absolute paths to store in archives is one that archivers like 7Zip offer you a command-line option to choose what you want.

Anyway, either the line which does your issue is in yourn own code and can be changed or it's in Qt - ZipWriter and would need changing there.

[Cobra91151]

@JonB

First of all, addFile() code is not mine, it's old Qt code. I can share it:

void ZipWriter::addFile(const QString &fileName, QIODevice *device)
{
    Q_ASSERT(device);
    QIODevice::OpenMode mode = device->openMode();
    bool opened = false;
    if ((mode & QIODevice::ReadOnly) == 0) {
        opened = true;
        if (! device->open(QIODevice::ReadOnly)) {
            d->status = FileOpenError;
            return;
        }
    }
    d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());
    if (opened) {
        device->close();
    }
}

void ZipWriterPrivate::addEntry(EntryType type, const QString &fileName, const QByteArray &contents/*, QFile::Permissions permissions, QZip::Method m*/)
{
    static const char *entryTypes[] = {
        "directory",
        "file     ",
        "symlink  " };
    qDebug() << "adding" << entryTypes[type] <<":" << fileName.toUtf8().data() << (type == 2 ? QByteArray(" -> " + contents).constData() : "");

    if (!device->isOpen()) {
        bool isOpen = device->open(QIODevice::WriteOnly);

        if (!isOpen) {
            status = ZipWriter::FileOpenError;
            return;
        }
    }

    device->seek(start_of_directory);

    // don't compress small files
    ZipWriter::CompressionPolicy compression = compressionPolicy;
    if (compressionPolicy == ZipWriter::AutoCompress) {
        if (contents.length() < 64) {
            compression = ZipWriter::NeverCompress;
        } else {
            compression = ZipWriter::AlwaysCompress;
        }
    }

    FileHeader header;
    memset(&header.h, 0, sizeof(CentralFileHeader));
    writeUInt(header.h.signature, 0x02014b50);
    writeUShort(header.h.version_needed, 0x14);
    writeUInt(header.h.uncompressed_size, contents.length());
    writeMSDosDate(header.h.last_mod_file, QDateTime::currentDateTime());

    qDebug() << contents.size();
    QByteArray data = contents;

    if (compression == ZipWriter::AlwaysCompress) {
        writeUShort(header.h.compression_method, 8);
        ulong len = contents.length();
        // shamelessly copied form zlib
        len += (len >> 12) + (len >> 14) + 11;
        int res;
        do {
            data.resize(len);
            res = deflate((uchar*)data.data(), &len, (const uchar*)contents.constData(), contents.length());

            switch (res) {
                case Z_OK:
                    data.resize(len);
                    break;
                case Z_MEM_ERROR:
                    qWarning("QZip: Z_MEM_ERROR: Not enough memory to compress file, skipping");
                    data.resize(0);
                    break;
                case Z_BUF_ERROR:
                    len *= 2;
                    break;
            }
        } while (res == Z_BUF_ERROR);
    }
    // TODO add a check if data.length() > contents.length().  Then try to store the original and revert the compression method to be uncompressed
    writeUInt(header.h.compressed_size, data.length());
    uint crc_32 = ::crc32(0, 0, 0);
    crc_32 = ::crc32(crc_32, (const uchar *)contents.constData(), contents.length());
    writeUInt(header.h.crc_32, crc_32);
    header.file_name = fileName.toLocal8Bit();

    if (header.file_name.size() > 0xffff) {
        qWarning("QZip: Filename too long, chopping it to 65535 characters");
        header.file_name = header.file_name.left(0xffff);
    }

    writeUShort(header.h.file_name_length, header.file_name.length());
    //h.extra_field_length[2];

    writeUShort(header.h.version_made, 3 << 8);
    //uchar internal_file_attributes[2];
    //uchar external_file_attributes[4];
    quint32 mode = permissionsToMode(permissions);
    switch (type) {
        case File: mode |= S_IFREG; break;
        case Directory: mode |= S_IFDIR; break;
        case Symlink: mode |= S_IFLNK; break;
    }
    writeUInt(header.h.external_file_attributes, mode << 16);
    writeUInt(header.h.offset_local_header, start_of_directory);
    fileHeaders.append(header);
    LocalFileHeader h = header.h.toLocalHeader();
    device->write((const char *)&h, sizeof(LocalFileHeader));
    device->write(header.file_name);
    device->write(data);
    start_of_directory = device->pos();
    dirtyFileTree = true;
}

As I wrote in my main post it only stores the files, not directories with content. For directories, it contains only this one line: d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray()); which stores the empty directory. No docs are available for it. I do not care what 7Zip or other archives offers, I have tasks which I must complete.
Secondly, if you don't know what I mean or do not have a working idea than feel free to omit my question :)

[JonB]

@Cobra91151 said in Zip directory recursion:

Secondly, if you don't know what I mean or do not have a working idea than feel free to omit my question :)

:) Just let me know if this answer is not helping.

d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());

I would guess this is the line which adds the filepath string into the archive? And you say " It adds the full path to the archive.".

So that implies to me the parameter const QString &fileName to ZipWriter::addFile()?

So you would need what you pass there to be the "relative" path or file name you want it to store. Right?

In your ZipWriter::addDirectory() I see lines like

addFile(QString("%1%2").arg(archDirName, archFile), zipFile.readAll());

d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray());

As I said earlier, why are you/do you have to pass a full path you do not want stored there? Have you tried passing what you want (some relative path) there?

[Cobra91151]

@JonB
You are looking at absolute (old) code. I have already removed it and change to this below:

void ZipWriter::addDirectory(const QString &dirPath)
{
    QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
    while (dirIt.hasNext()) {
        QString archDirPath = dirIt.next();
        QFile zipFile(archDirPath);
        zipFile.open(QIODevice::ReadOnly);

        if (!dirIt.fileInfo().isDir()) {
            addFile(archDirPath, zipFile.readAll());
        }

        zipFile.close();
    }
}

Here is your test case: d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());

void ZipWriter::addDirectory(const QString &dirPath)
{
    QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
    while (dirIt.hasNext()) {
        QString archDirPath = dirIt.next();
        QFile zipFile(archDirPath);
        zipFile.open(QIODevice::ReadOnly);
        d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(dirIt.fileInfo().fileName()), zipFile.readAll());
        zipFile.close();
    }
}

Returns:

So, as you can see, it does not work. No dirs, only files with 0 size :)
I think, the first directory must be only directory, for example: dirIt.fileInfo().dir().dirName();. It will be the starting point. Then, everything should be the absolute path.

[Cobra91151]

So, I have figured it out. It requires only the name of the directories and to construct the path by myself.

Code:

void ZipWriter::addDirectory(const QString &dirPath)
{
    int i = 0;
    int k = 0;
    QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
    while (dirIt.hasNext()) {
        QString archDirPath = dirIt.next();
        qDebug() << archDirPath;
        QFile zipFile(archDirPath);
        zipFile.open(QIODevice::ReadOnly);

        if (i > 0) {
            if (dirIt.fileInfo().isDir()) {
                k++;
            } else {
                addFile(QString("%1%2").arg(addDirSeparator(dirIt.fileInfo().dir().dirName()), dirIt.fileInfo().fileName()), zipFile.readAll());
            }
        } else {
            addFile(QString("%1%2").arg(addDirSeparator(dirIt.fileInfo().dir().dirName()), dirIt.fileInfo().fileName()), zipFile.readAll());
        }

        i++;
        zipFile.close();
    }

    qDebug() << "All iterations: " << i << " | All dirs: " << k;
}

Result:

The one issue is still present. It does not add the subdirectories inside of the main directory but instead it adds all directories as main. I need to handle it as well.

[Cobra91151]

Finally, I have fixed it.

Code:

void ZipWriter::addDirectory(const QString &dirPath)
{
    QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
    while (dirIt.hasNext()) {
        QString archDirPath = dirIt.next();
        QFile zipFile(archDirPath);
        zipFile.open(QIODevice::ReadOnly);

        if (!dirIt.fileInfo().isDir()) {
            addFile(QDir(dirPath).relativeFilePath(archDirPath), zipFile.readAll());
        }

        zipFile.close();
    }
}

Screenshot:

Now, it works as expected. The issue is resolved.