Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. General and Desktop
  4. Zip directory recursion
Forum Updated to NodeBB v4.3 + New Features

Zip directory recursion

Scheduled Pinned Locked Moved Solved General and Desktop
11 Posts 2 Posters 954 Views
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • Cobra91151C Offline
    Cobra91151C Offline
    Cobra91151
    wrote on last edited by Cobra91151
    #2

    I will try to use the QDirIterator and Qt containers to solve this issue.

    1 Reply Last reply
    0
    • Cobra91151C Offline
      Cobra91151C Offline
      Cobra91151
      wrote on last edited by
      #3

      So, I have created something like this:

      Code:

      void ZipWriter::addDirectory(const QString &dirPath)
      {
          QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
          while (dirIt.hasNext()) {
              QString archDirPath = dirIt.next();
              QFile zipFile(archDirPath);
              zipFile.open(QIODevice::ReadOnly);
      
              if (!dirIt.fileInfo().isDir()) {
                  addFile(archDirPath, zipFile.readAll());
              }
      
              zipFile.close();
          }
      }
      

      Now, it adds everything recursively and in the correct order but I have another issue. It adds the full path to the archive. For example, I want to add this folder and it's content to the archive: 22610.1_amd64_en-us_professional_00fb7ba0_convert. In the archive I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert. Any ideas how to make this relative path or trim it? Thank you.

      JonBJ 1 Reply Last reply
      0
      • Cobra91151C Cobra91151

        So, I have created something like this:

        Code:

        void ZipWriter::addDirectory(const QString &dirPath)
        {
            QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
            while (dirIt.hasNext()) {
                QString archDirPath = dirIt.next();
                QFile zipFile(archDirPath);
                zipFile.open(QIODevice::ReadOnly);
        
                if (!dirIt.fileInfo().isDir()) {
                    addFile(archDirPath, zipFile.readAll());
                }
        
                zipFile.close();
            }
        }
        

        Now, it adds everything recursively and in the correct order but I have another issue. It adds the full path to the archive. For example, I want to add this folder and it's content to the archive: 22610.1_amd64_en-us_professional_00fb7ba0_convert. In the archive I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert. Any ideas how to make this relative path or trim it? Thank you.

        JonBJ Offline
        JonBJ Offline
        JonB
        wrote on last edited by JonB
        #4

        @Cobra91151
        So don't add the full path (absoluteFilePath()?) wherever you do that (in ZipWriter::addDirectory()?), if you're using QFileInfo there is QString QFileInfo::fileName() const for the plain filename, or maybe you want to make it relative to something inside what you are zipping (there is also QString QFileInfo::filePath() const which is the path you are passing to QFileInfo which may be relatve rather than absolute).

        Cobra91151C 1 Reply Last reply
        1
        • JonBJ JonB

          @Cobra91151
          So don't add the full path (absoluteFilePath()?) wherever you do that (in ZipWriter::addDirectory()?), if you're using QFileInfo there is QString QFileInfo::fileName() const for the plain filename, or maybe you want to make it relative to something inside what you are zipping (there is also QString QFileInfo::filePath() const which is the path you are passing to QFileInfo which may be relatve rather than absolute).

          Cobra91151C Offline
          Cobra91151C Offline
          Cobra91151
          wrote on last edited by Cobra91151
          #5

          @JonB

          QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well. Also, I have tried QString QFileInfo::filePath() const but it's still the same as with archDirPath path. In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

          addFile(dirIt.fileInfo().filePath(), zipFile.readAll());
          

          I need somehow to omit the full path in the archive and get the following:

          22610.1_amd64_en-us_professional_00fb7ba0_convert => then directories/files.

          JonBJ 1 Reply Last reply
          0
          • Cobra91151C Cobra91151

            @JonB

            QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well. Also, I have tried QString QFileInfo::filePath() const but it's still the same as with archDirPath path. In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

            addFile(dirIt.fileInfo().filePath(), zipFile.readAll());
            

            I need somehow to omit the full path in the archive and get the following:

            22610.1_amd64_en-us_professional_00fb7ba0_convert => then directories/files.

            JonBJ Offline
            JonBJ Offline
            JonB
            wrote on last edited by JonB
            #6

            @Cobra91151 said in Zip directory recursion:

            QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well

            I really don't know what you mean. QFileInfo::-anything does not "add" anything at all, it's just a method to call on a filepath to find about it.

            Somewhere in your code (no, I'm not going through it) there is presumably a filepath you use to open a file to read it in and store it in the zip and there is also presumably a filepath you use to store its name/path. Why do they have to be the same?

            Purely at a guess this is done in your addFile() code, but I don't see that.

            I am working on creating zip archive using old Qt - ZipWriter class.

            I have only just noticed this. So since I know nothing about it wouldn't you have to look at its code/documentation if it's not doing whatever it is you want?

            In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

            If you cded to C:\Users\userProfile\Downloads and then specified relative paths, like 22610.1_amd64_en-us_professional_00fb7ba0_convert --- and did not store QFileInfo::absolutePath(), rather QFileInfo::filePath() --- would that achieve what you want?

            Note that this issue of relative versus absolute paths to store in archives is one that archivers like 7Zip offer you a command-line option to choose what you want.

            Anyway, either the line which does your issue is in yourn own code and can be changed or it's in Qt - ZipWriter and would need changing there.

            Cobra91151C 1 Reply Last reply
            0
            • JonBJ JonB

              @Cobra91151 said in Zip directory recursion:

              QString QFileInfo::fileName() const adds only files without directories but I must have the directories as well

              I really don't know what you mean. QFileInfo::-anything does not "add" anything at all, it's just a method to call on a filepath to find about it.

              Somewhere in your code (no, I'm not going through it) there is presumably a filepath you use to open a file to read it in and store it in the zip and there is also presumably a filepath you use to store its name/path. Why do they have to be the same?

              Purely at a guess this is done in your addFile() code, but I don't see that.

              I am working on creating zip archive using old Qt - ZipWriter class.

              I have only just noticed this. So since I know nothing about it wouldn't you have to look at its code/documentation if it's not doing whatever it is you want?

              In the archive, I get: C:\Users\userProfile\Downloads\22610.1_amd64_en-us_professional_00fb7ba0_convert

              If you cded to C:\Users\userProfile\Downloads and then specified relative paths, like 22610.1_amd64_en-us_professional_00fb7ba0_convert --- and did not store QFileInfo::absolutePath(), rather QFileInfo::filePath() --- would that achieve what you want?

              Note that this issue of relative versus absolute paths to store in archives is one that archivers like 7Zip offer you a command-line option to choose what you want.

              Anyway, either the line which does your issue is in yourn own code and can be changed or it's in Qt - ZipWriter and would need changing there.

              Cobra91151C Offline
              Cobra91151C Offline
              Cobra91151
              wrote on last edited by
              #7

              @JonB

              First of all, addFile() code is not mine, it's old Qt code. I can share it:

              void ZipWriter::addFile(const QString &fileName, QIODevice *device)
              {
                  Q_ASSERT(device);
                  QIODevice::OpenMode mode = device->openMode();
                  bool opened = false;
                  if ((mode & QIODevice::ReadOnly) == 0) {
                      opened = true;
                      if (! device->open(QIODevice::ReadOnly)) {
                          d->status = FileOpenError;
                          return;
                      }
                  }
                  d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());
                  if (opened) {
                      device->close();
                  }
              }
              
              void ZipWriterPrivate::addEntry(EntryType type, const QString &fileName, const QByteArray &contents/*, QFile::Permissions permissions, QZip::Method m*/)
              {
                  static const char *entryTypes[] = {
                      "directory",
                      "file     ",
                      "symlink  " };
                  qDebug() << "adding" << entryTypes[type] <<":" << fileName.toUtf8().data() << (type == 2 ? QByteArray(" -> " + contents).constData() : "");
              
                  if (!device->isOpen()) {
                      bool isOpen = device->open(QIODevice::WriteOnly);
              
                      if (!isOpen) {
                          status = ZipWriter::FileOpenError;
                          return;
                      }
                  }
              
                  device->seek(start_of_directory);
              
                  // don't compress small files
                  ZipWriter::CompressionPolicy compression = compressionPolicy;
                  if (compressionPolicy == ZipWriter::AutoCompress) {
                      if (contents.length() < 64) {
                          compression = ZipWriter::NeverCompress;
                      } else {
                          compression = ZipWriter::AlwaysCompress;
                      }
                  }
              
                  FileHeader header;
                  memset(&header.h, 0, sizeof(CentralFileHeader));
                  writeUInt(header.h.signature, 0x02014b50);
                  writeUShort(header.h.version_needed, 0x14);
                  writeUInt(header.h.uncompressed_size, contents.length());
                  writeMSDosDate(header.h.last_mod_file, QDateTime::currentDateTime());
              
                  qDebug() << contents.size();
                  QByteArray data = contents;
              
                  if (compression == ZipWriter::AlwaysCompress) {
                      writeUShort(header.h.compression_method, 8);
                      ulong len = contents.length();
                      // shamelessly copied form zlib
                      len += (len >> 12) + (len >> 14) + 11;
                      int res;
                      do {
                          data.resize(len);
                          res = deflate((uchar*)data.data(), &len, (const uchar*)contents.constData(), contents.length());
              
                          switch (res) {
                              case Z_OK:
                                  data.resize(len);
                                  break;
                              case Z_MEM_ERROR:
                                  qWarning("QZip: Z_MEM_ERROR: Not enough memory to compress file, skipping");
                                  data.resize(0);
                                  break;
                              case Z_BUF_ERROR:
                                  len *= 2;
                                  break;
                          }
                      } while (res == Z_BUF_ERROR);
                  }
                  // TODO add a check if data.length() > contents.length().  Then try to store the original and revert the compression method to be uncompressed
                  writeUInt(header.h.compressed_size, data.length());
                  uint crc_32 = ::crc32(0, 0, 0);
                  crc_32 = ::crc32(crc_32, (const uchar *)contents.constData(), contents.length());
                  writeUInt(header.h.crc_32, crc_32);
                  header.file_name = fileName.toLocal8Bit();
              
                  if (header.file_name.size() > 0xffff) {
                      qWarning("QZip: Filename too long, chopping it to 65535 characters");
                      header.file_name = header.file_name.left(0xffff);
                  }
              
                  writeUShort(header.h.file_name_length, header.file_name.length());
                  //h.extra_field_length[2];
              
                  writeUShort(header.h.version_made, 3 << 8);
                  //uchar internal_file_attributes[2];
                  //uchar external_file_attributes[4];
                  quint32 mode = permissionsToMode(permissions);
                  switch (type) {
                      case File: mode |= S_IFREG; break;
                      case Directory: mode |= S_IFDIR; break;
                      case Symlink: mode |= S_IFLNK; break;
                  }
                  writeUInt(header.h.external_file_attributes, mode << 16);
                  writeUInt(header.h.offset_local_header, start_of_directory);
                  fileHeaders.append(header);
                  LocalFileHeader h = header.h.toLocalHeader();
                  device->write((const char *)&h, sizeof(LocalFileHeader));
                  device->write(header.file_name);
                  device->write(data);
                  start_of_directory = device->pos();
                  dirtyFileTree = true;
              }
              

              As I wrote in my main post it only stores the files, not directories with content. For directories, it contains only this one line: d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray()); which stores the empty directory. No docs are available for it. I do not care what 7Zip or other archives offers, I have tasks which I must complete.
              Secondly, if you don't know what I mean or do not have a working idea than feel free to omit my question :)

              JonBJ 1 Reply Last reply
              0
              • Cobra91151C Cobra91151

                @JonB

                First of all, addFile() code is not mine, it's old Qt code. I can share it:

                void ZipWriter::addFile(const QString &fileName, QIODevice *device)
                {
                    Q_ASSERT(device);
                    QIODevice::OpenMode mode = device->openMode();
                    bool opened = false;
                    if ((mode & QIODevice::ReadOnly) == 0) {
                        opened = true;
                        if (! device->open(QIODevice::ReadOnly)) {
                            d->status = FileOpenError;
                            return;
                        }
                    }
                    d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());
                    if (opened) {
                        device->close();
                    }
                }
                
                void ZipWriterPrivate::addEntry(EntryType type, const QString &fileName, const QByteArray &contents/*, QFile::Permissions permissions, QZip::Method m*/)
                {
                    static const char *entryTypes[] = {
                        "directory",
                        "file     ",
                        "symlink  " };
                    qDebug() << "adding" << entryTypes[type] <<":" << fileName.toUtf8().data() << (type == 2 ? QByteArray(" -> " + contents).constData() : "");
                
                    if (!device->isOpen()) {
                        bool isOpen = device->open(QIODevice::WriteOnly);
                
                        if (!isOpen) {
                            status = ZipWriter::FileOpenError;
                            return;
                        }
                    }
                
                    device->seek(start_of_directory);
                
                    // don't compress small files
                    ZipWriter::CompressionPolicy compression = compressionPolicy;
                    if (compressionPolicy == ZipWriter::AutoCompress) {
                        if (contents.length() < 64) {
                            compression = ZipWriter::NeverCompress;
                        } else {
                            compression = ZipWriter::AlwaysCompress;
                        }
                    }
                
                    FileHeader header;
                    memset(&header.h, 0, sizeof(CentralFileHeader));
                    writeUInt(header.h.signature, 0x02014b50);
                    writeUShort(header.h.version_needed, 0x14);
                    writeUInt(header.h.uncompressed_size, contents.length());
                    writeMSDosDate(header.h.last_mod_file, QDateTime::currentDateTime());
                
                    qDebug() << contents.size();
                    QByteArray data = contents;
                
                    if (compression == ZipWriter::AlwaysCompress) {
                        writeUShort(header.h.compression_method, 8);
                        ulong len = contents.length();
                        // shamelessly copied form zlib
                        len += (len >> 12) + (len >> 14) + 11;
                        int res;
                        do {
                            data.resize(len);
                            res = deflate((uchar*)data.data(), &len, (const uchar*)contents.constData(), contents.length());
                
                            switch (res) {
                                case Z_OK:
                                    data.resize(len);
                                    break;
                                case Z_MEM_ERROR:
                                    qWarning("QZip: Z_MEM_ERROR: Not enough memory to compress file, skipping");
                                    data.resize(0);
                                    break;
                                case Z_BUF_ERROR:
                                    len *= 2;
                                    break;
                            }
                        } while (res == Z_BUF_ERROR);
                    }
                    // TODO add a check if data.length() > contents.length().  Then try to store the original and revert the compression method to be uncompressed
                    writeUInt(header.h.compressed_size, data.length());
                    uint crc_32 = ::crc32(0, 0, 0);
                    crc_32 = ::crc32(crc_32, (const uchar *)contents.constData(), contents.length());
                    writeUInt(header.h.crc_32, crc_32);
                    header.file_name = fileName.toLocal8Bit();
                
                    if (header.file_name.size() > 0xffff) {
                        qWarning("QZip: Filename too long, chopping it to 65535 characters");
                        header.file_name = header.file_name.left(0xffff);
                    }
                
                    writeUShort(header.h.file_name_length, header.file_name.length());
                    //h.extra_field_length[2];
                
                    writeUShort(header.h.version_made, 3 << 8);
                    //uchar internal_file_attributes[2];
                    //uchar external_file_attributes[4];
                    quint32 mode = permissionsToMode(permissions);
                    switch (type) {
                        case File: mode |= S_IFREG; break;
                        case Directory: mode |= S_IFDIR; break;
                        case Symlink: mode |= S_IFLNK; break;
                    }
                    writeUInt(header.h.external_file_attributes, mode << 16);
                    writeUInt(header.h.offset_local_header, start_of_directory);
                    fileHeaders.append(header);
                    LocalFileHeader h = header.h.toLocalHeader();
                    device->write((const char *)&h, sizeof(LocalFileHeader));
                    device->write(header.file_name);
                    device->write(data);
                    start_of_directory = device->pos();
                    dirtyFileTree = true;
                }
                

                As I wrote in my main post it only stores the files, not directories with content. For directories, it contains only this one line: d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray()); which stores the empty directory. No docs are available for it. I do not care what 7Zip or other archives offers, I have tasks which I must complete.
                Secondly, if you don't know what I mean or do not have a working idea than feel free to omit my question :)

                JonBJ Offline
                JonBJ Offline
                JonB
                wrote on last edited by JonB
                #8

                @Cobra91151 said in Zip directory recursion:

                Secondly, if you don't know what I mean or do not have a working idea than feel free to omit my question :)

                :) Just let me know if this answer is not helping.

                d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());

                I would guess this is the line which adds the filepath string into the archive? And you say " It adds the full path to the archive.".

                So that implies to me the parameter const QString &fileName to ZipWriter::addFile()?

                So you would need what you pass there to be the "relative" path or file name you want it to store. Right?

                In your ZipWriter::addDirectory() I see lines like

                addFile(QString("%1%2").arg(archDirName, archFile), zipFile.readAll());
                
                d->addEntry(ZipWriterPrivate::Directory, archDirName, QByteArray());
                

                As I said earlier, why are you/do you have to pass a full path you do not want stored there? Have you tried passing what you want (some relative path) there?

                1 Reply Last reply
                0
                • Cobra91151C Offline
                  Cobra91151C Offline
                  Cobra91151
                  wrote on last edited by Cobra91151
                  #9

                  @JonB
                  You are looking at absolute (old) code. I have already removed it and change to this below:

                  void ZipWriter::addDirectory(const QString &dirPath)
                  {
                      QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
                      while (dirIt.hasNext()) {
                          QString archDirPath = dirIt.next();
                          QFile zipFile(archDirPath);
                          zipFile.open(QIODevice::ReadOnly);
                  
                          if (!dirIt.fileInfo().isDir()) {
                              addFile(archDirPath, zipFile.readAll());
                          }
                  
                          zipFile.close();
                      }
                  }
                  

                  Here is your test case: d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(fileName), device->readAll());

                  void ZipWriter::addDirectory(const QString &dirPath)
                  {
                      QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
                      while (dirIt.hasNext()) {
                          QString archDirPath = dirIt.next();
                          QFile zipFile(archDirPath);
                          zipFile.open(QIODevice::ReadOnly);
                          d->addEntry(ZipWriterPrivate::File, QDir::fromNativeSeparators(dirIt.fileInfo().fileName()), zipFile.readAll());
                          zipFile.close();
                      }
                  }
                  

                  Returns:

                  2022-05-01_174854.png

                  So, as you can see, it does not work. No dirs, only files with 0 size :)
                  I think, the first directory must be only directory, for example: dirIt.fileInfo().dir().dirName();. It will be the starting point. Then, everything should be the absolute path.

                  1 Reply Last reply
                  0
                  • Cobra91151C Offline
                    Cobra91151C Offline
                    Cobra91151
                    wrote on last edited by Cobra91151
                    #10

                    So, I have figured it out. It requires only the name of the directories and to construct the path by myself.

                    Code:

                    void ZipWriter::addDirectory(const QString &dirPath)
                    {
                        int i = 0;
                        int k = 0;
                        QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
                        while (dirIt.hasNext()) {
                            QString archDirPath = dirIt.next();
                            qDebug() << archDirPath;
                            QFile zipFile(archDirPath);
                            zipFile.open(QIODevice::ReadOnly);
                    
                            if (i > 0) {
                                if (dirIt.fileInfo().isDir()) {
                                    k++;
                                } else {
                                    addFile(QString("%1%2").arg(addDirSeparator(dirIt.fileInfo().dir().dirName()), dirIt.fileInfo().fileName()), zipFile.readAll());
                                }
                            } else {
                                addFile(QString("%1%2").arg(addDirSeparator(dirIt.fileInfo().dir().dirName()), dirIt.fileInfo().fileName()), zipFile.readAll());
                            }
                    
                            i++;
                            zipFile.close();
                        }
                    
                        qDebug() << "All iterations: " << i << " | All dirs: " << k;
                    }
                    

                    Result:
                    2022-05-01_220219.png

                    The one issue is still present. It does not add the subdirectories inside of the main directory but instead it adds all directories as main. I need to handle it as well.

                    1 Reply Last reply
                    0
                    • Cobra91151C Offline
                      Cobra91151C Offline
                      Cobra91151
                      wrote on last edited by
                      #11

                      Finally, I have fixed it.

                      Code:

                      void ZipWriter::addDirectory(const QString &dirPath)
                      {
                          QDirIterator dirIt(dirPath, QDir::AllEntries | QDir::NoDotAndDotDot, QDirIterator::Subdirectories);
                          while (dirIt.hasNext()) {
                              QString archDirPath = dirIt.next();
                              QFile zipFile(archDirPath);
                              zipFile.open(QIODevice::ReadOnly);
                      
                              if (!dirIt.fileInfo().isDir()) {
                                  addFile(QDir(dirPath).relativeFilePath(archDirPath), zipFile.readAll());
                              }
                      
                              zipFile.close();
                          }
                      }
                      

                      Screenshot:

                      2022-05-02_012558.png

                      Now, it works as expected. The issue is resolved.

                      1 Reply Last reply
                      0

                      • Login

                      • Login or register to search.
                      • First post
                        Last post
                      0
                      • Categories
                      • Recent
                      • Tags
                      • Popular
                      • Users
                      • Groups
                      • Search
                      • Get Qt Extensions
                      • Unsolved