Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Get Qt Extensions
  • Unsolved
Collapse
Brand Logo
  1. Home
  2. Qt Development
  3. Mobile and Embedded
  4. QTcpSocket question
Forum Updated to NodeBB v4.3 + New Features

QTcpSocket question

Scheduled Pinned Locked Moved Mobile and Embedded
23 Posts 7 Posters 13.0k Views 1 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • T Offline
    T Offline
    tobias.hunger
    wrote on last edited by
    #10

    What happens when a system runs out of memory? It will try to swap out memory areas to disk, try to reclaim memory from all running applications, and as a very last measure it will select one application and kill it (that is the Out-Of-Memory (OOM) killer I mentioned earlier). The heuristics of the OOM killer are usually good enough to pick the right app in a "stupid developer tries to read a blueray rip into memory in one go" use case:-)

    Why do you want to use a QDataStream here? Streams are great to handle typesafe data but are not really necessary when pushing raw bytes. Why don't you just do file->write(bytes, blockSize) directly? The code makes little sense anyway (where do bytes come from, etc.).

    1 Reply Last reply
    0
    • L Offline
      L Offline
      lamprosg
      wrote on last edited by
      #11

      Well, to understand things better, here's my effort for the code (only code that matters), if you're in the mood of reading it :)

      Server sending a list of files:
      @
      for (int j=0;j<filelist.size();j++)
      {
      QFileInfo fileinfo;
      fileinfo.setFile(filelist.value(j));
      int filesize = fileinfo.size(); //Individual file size in Bytes

      QFile *file= new QFile(filelist.value(j));
      file->open(QIODevice::ReadOnly);

      //Sending the files in packets of 5MB (short of)

      if (filesize < 5000000) //If file size is smaller than 5MB
      {
      char *data = new char[filesize];

         file->read(data, filesize);                             //Read it all and put it in *data
      
         clientConnection->write(data);
         delete data;                                            //Free memory
      

      }
      else
      {
      qint64 position=0;
      for (int x=0; x<(int) filesize/5000000 ; x++)
      {
      char *data = new char[5000000]; //5MB block to send

               file->seek(position);                               //Go the previous position we were
               file->read(data, 5000000);                          //Read 5MB of data
               position = file->pos();                             //Get the position we're in
      
               clientConnection->write(data);                      //Send the packet
               delete data;                                        //Free the memory for the next 5MB
          }
      

      int leftovers=0;
      if ( filesize % 5000000 != 0 ) //If we have left-overs
      {
      leftovers = filesize % 5000000; //Left-over bytes
      char *data = new char[leftovers];
      file->seek(position);
      file->read(data, leftovers);
      clientConnection->write(data); //Send those too
      delete data;
      }
      }
      file->close();

      delete file;
      @

      Client getting those files, (doesn't work, gets stuck in line 2 'return'):
      @
      if (tcpSocket->bytesAvailable() < totalfilesize) //Wait until everything is available
      return;

          ui->fileslabel->setText("Saving..");
      
          for (int i=0;i<numberofitems;i++)                       //For each file
          {
              if (filesizes[i] < 5000000)                         //If file is smaller than 5MB (or so)
              {
                  char *data = new char[filesizes[i]];
      
                  QFile *file = new QFile&#40;filenames[i]&#41;;          //Create the file
                  file->open(QIODevice::WriteOnly);
      
                  tcpSocket->read(data,filesizes[i]);             //Read the data from the socket
      
                  file->write(data,filesizes[i]);                 //Write it all at once
                  file->close();
      
                  delete file;
                  delete data;
              }
              else
              {
                  QFile *file = new QFile&#40;filenames[i]&#41;;
                  file->open(QIODevice::WriteOnly);
      
                  qint64 position=0;
                  for (int x=0; x<(int) filesizes[i]/5000000 ; x++)
                  {
                      char *data = new char[5000000];
      
                      tcpSocket->seek(position);                  //Go the position we were
                      file->seek(position);
      
                      tcpSocket->read(data, 5000000);             //Read 5 megs from the socket
                      file->write(data, 5000000);                 //Write 5 megs in the file
      
                      position = file->pos();                     //Get the position we are
      
                      delete data;                                //Free the memory for the next 5 megs
                  }
      
                  if (filesizes[i] % 5000000 !=0)
                  {
                      int leftovers = filesizes[i] % 5000000;
                      char *data = new char[leftovers];
      
                      tcpSocket->seek(position);
                      file->seek(position);                       //Go the position we were
      
                      tcpSocket->read(data, leftovers);
                      file->write(data, leftovers);               //Write the leftovers
      
                      delete data;
                  }
      
                  file->close();
                  delete file;
      

      delete [] filenames; //Finally setting the memory free
      delete [] filesizes;
      @

      1 Reply Last reply
      0
      • G Offline
        G Offline
        goetz
        wrote on last edited by
        #12

        For reading the file and sending the data you could use this:

        @
        int bufSize = 5000000;
        QVector<char> readBuffer(bufSize);
        foreach(const QString &fileName, filelist) {
        QFile file(fileName);
        if(!file.open(QIODevice::ReadOnly)) {
        qDebug() << "Error on opening" << filename;
        continue;
        }

        while( qint64 readLen = file&#40;readBuffer.data(&#41;, bufSize))
            clientConnection->write(readBuffer, readLen); // assuming clientConnection is a QIODevice
        
        file.close();
        

        }
        @

        See the "Blocking Fortune Client Example":http://doc.qt.nokia.com/qt-maemo-4.7/network-blockingfortuneclient.html for a prototype for your read function. Use QFile (no need for a heap allocation with new here, btw) and the write(data, len) functions for saving the received data.

        http://www.catb.org/~esr/faqs/smart-questions.html

        1 Reply Last reply
        0
        • T Offline
          T Offline
          tobias.hunger
          wrote on last edited by
          #13

          codestein: Read up on network programming before continuing with this project of yours! Your receiver is just waiting for someone to bring down the whole machine. Network programming is tricky as it is easy to exploit any mistakes made remotely. So you need to know what you are doing or you will allow somebody to launch denial of service attacks against your machine.

          The blocksize is much too high for most networks. I was thinking of a block size of maybe 64k or something! And you could get the same effect with much less code.

          @
          char * data[BLOCKSIZE];
          quint64 filepos = 0;

          while (filepos < file.size()) {
          quint64 length = file.read(data, BLOCKSIZE);
          filepos += length;
          socket.write(data, length);
          }
          @

          should do pretty much the same thing your code does.

          There is no reason to seek() all the time. Reading etc. implicitly moves the position in the stream, so you are in the right position to read/write already.

          @
          if (tcpSocket->bytesAvailable() < totalfilesize) //Wait until everything is available
          return;
          @

          So you send those 20 blueray disk rips in blocks, but at the receiver side you buffer all 20 of those in the network buffers before you start to write them out to disk? After making sure all the data of all files is already buffered in the receiver you then proceed to read the data in chunks. Why don't you just grab the whole thing in one go.

          You should get the data from the OS as soon as possible. Read as much as possible whenever you get a readReady signal and save it out to files asap. As an optimization you should make sure that you write decently sized chunks of data to the file. It is painfully slow to append BLOCKSIZE times one byte to a file, and much faster to add 1 times BLOCKSIZE bytes.

          1 Reply Last reply
          0
          • L Offline
            L Offline
            lamprosg
            wrote on last edited by
            #14

            I'm doing this small program for the sake of learning! Didn't know where to start so I went straight for the code, examples etc

            bq. There is no reason to seek() all the time. Reading etc. implicitly moves the position in the stream, so you are in the right position to read/write already.

            In the case of bytesAvailable, if I read 64k
            @if (tcpSocket->bytesAvailable() < 64000)
            return;
            @

            And then I want the other 64, I'll do it with the same code again and not like this, right?
            @if (tcpSocket->bytesAvailable() < 128000)
            return;
            @

            That was a major question I had.

            Thanks for the replies!

            1 Reply Last reply
            0
            • T Offline
              T Offline
              tobias.hunger
              wrote on last edited by
              #15

              bytesAvailable returns the number of bytes not yet read.

              Actually I do not see any need to check tcpSocket->bytesAvailable(): In a slot triggered by readReady() you know there are bytes available. Just do something like this:

              @
              // const int BUFFERSIZE = 1024*1024;
              // char * buffer[BUFFERSIZE];
              // quint64 offset = 0;
              // quint64 fileSize = 0;

              void dataAvailable()
              {
              quint64 length = tcpSocket->read(buffer[offset], BUFFERSIZE - offset);
              if (length + offset == BUFFERSIZE || length + fileSize == sizeOfFile[i]) {
              file.write(buffer, BUFFERSIZE);
              offset = 0;
              fileSize += length;
              // switch to next file if necessary!
              } else {
              offset += length;
              }
              }
              @

              1 Reply Last reply
              0
              • G Offline
                G Offline
                goetz
                wrote on last edited by
                #16

                There is no need to do the work in chunks of, say 64k. The operating system and Qt libs do some reasonable buffering for you, so there is no need to add another one and increase complexity of your code. Make it work correctly in the very first place, and, only if it is too slow, optimize it afterwards!

                http://www.catb.org/~esr/faqs/smart-questions.html

                1 Reply Last reply
                0
                • L Offline
                  L Offline
                  lamprosg
                  wrote on last edited by
                  #17

                  Greetings again!

                  Indeed after a number of debuggings I saw some descent buffering without coding it, smaller or arround 1 MB (many times much less). Probably best to put it in your code.

                  Still, I tried this code below, and I can't find any errors (downloading multiple files).
                  Seems to work for 1 file.. When the 1st file ends, it crashes where I put the asterisk. :O

                  @
                  qint64 savepos=0; //Position of bytes we're in
                  qint64 length=0; //Length of bytes read
                  int counter=0;
                  (bool) newfile=true;
                  qint64 buffer=1048576; // 1 MB
                  QFile *file;

                          if (newfile == true)
                          {
                              file = new QFile&#40;filenames[counter]&#41;;         //Create the file
                              file->open(QIODevice::WriteOnly);            //Open it
                              newfile = false;
                          }
                  
                          char *data = new char[buffer];
                          length = tcpSocket->read(data, buffer);        //Read at max: 1MB
                          file->write(data,length);
                          savepos += length;
                          ui->getprogressbar->setValue( (int) (savepos/filesizes[counter])*100);
                          delete data;
                  
                  •       if (savepos == filesizes[counter])       //File size reached, saved and closing
                          {
                              file->close();
                              delete file;
                              counter++;
                              savepos=0;                           //Reseting
                              length=0;
                              newfile = true;
                    
                              if (counter == numberofitems)
                              {
                                  delete [] filenames;                                    //Setting the memory free
                                  delete [] filesizes;
                                  tcpSocket->disconnectFromHost();
                              }
                          }
                    

                  @

                  1 Reply Last reply
                  0
                  • G Offline
                    G Offline
                    goetz
                    wrote on last edited by
                    #18

                    Could you explain "crash" a bit more? Any kind of message on the console?

                    BTW: If you do data = new char[] you must call delete[] data afterwards, otherwise you have memory leak.

                    http://www.catb.org/~esr/faqs/smart-questions.html

                    1 Reply Last reply
                    0
                    • L Offline
                      L Offline
                      lamprosg
                      wrote on last edited by
                      #19

                      bq. If you do data = new char[] you must call delete[] data afterwards, otherwise you have memory leak.

                      You're totally right, how did I miss that?

                      Crash:
                      Signal name: SIGSEGV
                      Signal meaning: Segmentation fault

                      Points me to QAbstractSocketPrivate::resetSocketLayer

                      Now I'm thinking about it, it's possible with multiple files that savepos goes higher than the filesize after last reading of the first file (reads part of the 2d too) and savepos is never equal to filesize.. And goes all wrong from there

                      1 Reply Last reply
                      0
                      • G Offline
                        G Offline
                        goetz
                        wrote on last edited by
                        #20

                        That sounds reasonable. We don't know what your "protocol" for the file transfer is. Maybe you should do it one by one, only sending a new file when the preceding has arrived completely.

                        On the other hand, if the equality never becomes true, you will write only one file that grows bigger and bigger. The segfault should not happen on the comparison line you indicated, but on the socket->read().

                        http://www.catb.org/~esr/faqs/smart-questions.html

                        1 Reply Last reply
                        0
                        • L Offline
                          L Offline
                          lamprosg
                          wrote on last edited by
                          #21

                          bq. The segfault should not happen on the comparison line you indicated, but on the socket->read().

                          That's where it crashed. The asterisk was the last line read, then went on the tcpSocket->read..

                          Anyway, I did it one by one. Seems it's working on the Simulator.. All I have to do is test it for real!
                          I hope I won't have any more surprises heh.

                          You've all been very helpful, and I learned few things.

                          More questions to come in the future :D

                          1 Reply Last reply
                          0
                          • F Offline
                            F Offline
                            fuhaijin
                            wrote on last edited by
                            #22

                            I have a question to ask:how to use QTcpSocket in ThreadPool?

                            1 Reply Last reply
                            0
                            • SGaistS Offline
                              SGaistS Offline
                              SGaist
                              Lifetime Qt Champion
                              wrote on last edited by
                              #23

                              Hi,

                              please open your own thread rather than reviving a three years old topic

                              Interested in AI ? www.idiap.ch
                              Please read the Qt Code of Conduct - https://forum.qt.io/topic/113070/qt-code-of-conduct

                              1 Reply Last reply
                              0

                              • Login

                              • Login or register to search.
                              • First post
                                Last post
                              0
                              • Categories
                              • Recent
                              • Tags
                              • Popular
                              • Users
                              • Groups
                              • Search
                              • Get Qt Extensions
                              • Unsolved