QFile save for usb drive



  • Hi,
    I used the following snippnet to save myData to a file. It works well when I save it to my hard disk c:. but when I try to save it to usb drive, it is extremly slow. Is there any way to improve this performance?

            QFile file( myfile );
            if( !file.open( QIODevice::ReadWrite ) )
                return false;
            QDataStream stream( &file );
            stream.setVersion( QDataStream::Qt_4_2 );
            qDebug()<<" takes long for usb drive";
            stream >>myData;
            file.close();


  • @samdol said in QFile save for usb drive:

    stream.setVersion( QDataStream::Qt_4_2 );

    Do you really need that old version?

    Just as an experiment, remove that line altogether and see if it improves (I really doubt it will but worth a try)



  • @VRonin
    Thanks ,I have replaced it with Qt_5_6. But it is still the same. It takes long time to save on usb drive.


  • Lifetime Qt Champion

    Hi,

    What USB device is it ?
    What generation of USB do you have available ?
    What is the size of the data that you are trying to write ?



  • @SGaist

    The USB device is a normal standard usb2.0 stick drive and I am trying to save 200KB on it. Compare to internal hard drive, it is tool slow.

    I also found that QDir::entryList() is very slow for the directory in usb drive. Is there any workaround at least for windows?
    QStringList dir_list = qdir.entryList( QDir::NoDotAndDotDot | QDir::NoSymLinks | QDir::Hidden |QDir::AllDirs);
    QStringList fn_list = qdir.entryList( QDir::NoDotAndDotDot | QDir::NoSymLinks | QDir::Hidden | QDir::Files);


  • Qt Champions 2016

    Hi
    Usb 2.0 is really slow compared to sata III (and II) that is typically used for hard drives.
    (hence usb 3 was invented)

    https://superuser.com/questions/317217/whats-the-maximum-typical-speed-possible-with-a-usb2-0-drive

    So have you tried to copy a large file to the usb stick and watch the MB/s ?
    If you land between 30-40 MB/s that it not the stick itself.

    When you say "too slow" , does it takes seconds or how slow ?

    Also make sure there is no scanner examining the file in the background.

    You can also play around with this

    #include <stdio.h>
    #include <chrono>  // for high_resolution_clock
    
    void MainWindow::on_pushButton_released() {
    
      const int FILE_SIZE = 200; //size in KB
      const int BUFFER_SIZE = 1024;
      char buffer [BUFFER_SIZE + 1];
      int i;
      auto start = std::chrono::high_resolution_clock::now();
      for(i = 0; i < BUFFER_SIZE; i++)
      { buffer[i] = (char)(i % 8 + 'a'); }
      buffer[BUFFER_SIZE] = '\0';
    
      FILE* pFile = fopen ("g:\\somefile.txt", "w");
      for (i = 0; i < FILE_SIZE; i++)
      { fprintf(pFile, buffer); }
    
      fclose(pFile);
      auto finish = std::chrono::high_resolution_clock::now();
      std::chrono::duration<double> elapsed = finish - start;
      qDebug() << "Elapsed time: " << elapsed.count();
    }
    
    

    I get around 0.2 - 0.3 for saving 200KB file on my usb 2 port with usb 2 device.
    Note that timing file writes using chrono is imprecise but could give hints.



  • @mrjj

    I have no problem when I copy file from usb to internal hard disk or internal hard disk to usb stick. Compare to the speed for copying file by windows explorer, copying file by QFile::copy() is not slow. It is slow when I save file to usb stick by QFile/QDataStream. Also when I browse directories, qdir.entryList() shows poor performance compare to explorer or other windows programs. For example if I enter a directory with 10000 files, it takes more than one minute to finish qdir.entryList().


  • Qt Champions 2016

    @samdol
    ok. so we assume device is ok.
    Did you try test program and compared elapse times?
    0.3 is not slow as such.
    Could be interesting to see where you land.

    What is myData ?
    One big array ?

    update:
    tried

    For /L %i in (1,1,10000) do fsutil file createnew B%i.tmp 65536
    

    gives me 10.000 files on usb stick.
    (unmount / mount before test to avoid cashing)

    #include <QDir>
    void MainWindow::on_pushButton_2_released()
    {
        auto start = std::chrono::high_resolution_clock::now();
        QDir qdir("g:/");
        QStringList fn_list = qdir.entryList( QDir::NoDotAndDotDot | QDir::NoSymLinks | QDir::Hidden | QDir::Files);
        auto finish = std::chrono::high_resolution_clock::now();
        std::chrono::duration<double> elapsed = finish - start;
        qDebug() << "Elapsed time: " << elapsed.count();
    }
    

    Elapsed time: 1.54938
    So under 2 secs.
    This is win 10. Qt 5.7.



  • @mrjj
    Thank you for your example.
    When I run on_pushButton_released() first time, I got
    Elapsed time: 16.3639
    but when I run it second time and later
    Elapsed time: 0.280801
    When I run on_pushButton_2_released() for directory with 10000 files on usb, the result is
    Elapsed time: 53.6064
    quite slow compare to your result. I used windows7 and Qt 5.6.2
    Does it mean qdir.entryList() does not like my usb drive
    or should I upgrade to Qt 5.7?


  • Qt Champions 2016

    Hi
    Could you try with clean / new usb stick ?

    Those numbers are pretty high for entryList.

    Are those files in sub folders?

    Its hard to say if it is device or driver issue or something with windows.
    or if it simply is that slow in some cases.

    If time permits simple install 5.7 along side 5.6 and test if it changes anything.

    I will also test on a win 7 tomorrow just to be sure.



  • @mrjj
    As you suggested, it seems that could be a problem of Qt 5.6.2.
    After long wait, finally I could try to compile Qt 5.7.1 in my machine today.
    I have downloaded qt-everywhere 5.7.1 and tried to compile in windows 7 platform.
    I did
    configure -prefix C:\Qt\5.7.1 -static -release -platform win32-g++ -no-compile-examples -opengl desktop
    and then
    mingw32-make
    After several hours(Slow machine), I met the following error message
    In file included from release\statemachine.cpp:7:0:
    release\statemachine.h:10:30: fatal error: QScxmlStateMachine: No such file or directory
    #include <QScxmlStateMachine>
    ^
    compilation terminated.
    Makefile.Release:135: recipe for target '.obj/release/statemachine.o' failed.

    Do I have to add some extra options to configure to solve this problem?


  • Lifetime Qt Champion

    Hi,

    Since you are upgrading, why not 5.9.1 ?



  • @SGaist
    Because my license is expired and I could not download that version with my account. I used Mingw 4.9.1 to compile it. Do I have to use 5.3 version to compile Qt 5.7.1?


  • Moderators

    @samdol Do you have a commercial Qt license?



  • @jsulm
    It is paid for developement with Qt. before it is expired, I could download Qt 5.9 alpha version. I am not sure whether I could use this as commercial version because when I download alpha version, my license was still valid.


  • Qt Champions 2016

    @samdol

    Hi
    Can't you just grap it directly then ?
    https://download.qt.io/official_releases/qt/5.9/5.9.1/

    Or something Im missing?



  • @mrjj
    I could download enterprise version before 5.9 but since my license expired, I could not gain it. By the way, I could compile Qt 5.6.2 with that configure options but Qt 5.7.1 always spews error when I tried to compile with the same options. I could not understand the method to compile for Qt 5.6.2 did not apply to Qt 5.7.1.


  • Moderators

    @samdol Is there a reason why you compile Qt by yourself?



  • @jsulm
    I need to develop a stand alone application which requires static build of Qt. I could do with Qt 5.6.2 but I could not do it anymore for 5.7.1. I don't know whether there is big change between them.


  • Moderators

    @samdol Why don't you build Qt 5.9.1? This is the latest version. If you say you have errors while compiling you should say which errors and which compiler you're using.



  • @jsulm
    I could not use Qt 5.9.1 as a commercial version because my license has expired.
    I downloaded Qt 5.7.1 and configured by
    configure -prefix C:\Qt\5.7.1 -static -release -platform win32-g++ -no-compile-examples -opengl desktop
    and then
    mingw32-make
    After several hours(Slow machine), I met the following error message
    In file included from release\statemachine.cpp:7:0:
    release\statemachine.h:10:30: fatal error: QScxmlStateMachine: No such file or directory
    #include <QScxmlStateMachine>
    ^
    compilation terminated.
    Makefile.Release:135: recipe for target '.obj/release/statemachine.o' failed.

    This is quite strange, because when I compile Qt 5.6.2 with the same options, It could compile well. I used mingw4.9.3 and mingw7.1 but the results are the same.


  • Lifetime Qt Champion

    If you have a slot machine then you should avoid building modules you don't use. That will be a gain of time and space. You can always build additional modules on demand.



  • @mrjj

    Following the instruction by
    http://dimitris.apeiro.gr/2015/06/24/build-a-static-qt5-for-windows-by-compiling/
    I could compile Qt 5.7.1. It seems some of configure options were not correct. I'll check it out more closely.
    So I prepared a Qt 5.7.1 in windows 10 and when I run on_pushButton_2_released() by mrjj, it still shows poor performance. It takes 24 seconds for 10000 files on usb 2.0 drive. Now I am getting confused. Before I thought because Qt 5.6.2 is slow for entryList of files in usb. But changing Qt 5.7.1 did not improve much. If it is the problem of usb stick, accessing file should be slow with other programs. But other programs could access files on this usb very quickly.


  • Qt Champions 2016

    @samdol
    Hi
    Can you possible try the USB stick + program on other computer?
    Win 10 if possible.

    You are right to think there is something fishy. If it was the device, explorer should be slow too.

    So i do wonder.



  • @mrjj

    When I run on_pushButton_2_released() to access the directory with 10000 files on hard disk it took less than a second. But accessing the same directory on usb 2.0 stick takes 53 seconds. I also tried QDirIterator but the results are the same. So I run the code in another machine with windows 10 to the same usb stick, no difference. Did it take less than a second when you access the directory with 10000 files on usb 2.0 stick?


  • Qt Champions 2016

    @samdol
    Hi
    I tried Qt5.7 static and its maximum 1.2 sec on the oldest usb stick i could find.
    win 10
    alt text

    But inside virtual win 7.
    alt text
    Pretty slow. Explorer show it instantly.

    I wish i had an other win 7 to try on.



  • @mrjj
    Thank you for pictures. That is interesting. Does it mean Qt does not support well usb access for windows 7 and support well for windows 10? But my windows10 also shows poor performance. Do you think I have to try with 5.7.0 instead of 5.7.1?



  • @samdol Maybe you could try to make a minimal program, nothing but the slow operation inside main(), and do profiling for it. It could reveal where the time is spent.

    Edit: I mean real profiling, not just timing output.


  • Qt Champions 2016

    @samdol

    Im not sure what it means.
    The exact same program runs much faster in win 10.

    I would need a non virtual win 7 and test with to make sure its not related to being virtual.

    But qdir.entryList does seems very slow (on win 7) and its not repeatable with other utils or explorer.

    I will install Qt in the win 7 and see directly from creator.

    Update:
    To install 5.9 i installed service pack 1 for win 7
    The old static compiled test program no longer takes a long time.

    The pc you are testing on, what service pack is that ?

    I think this is related to win 7 non service pack so make sure you have a fully patched system.



  • @mrjj
    Maybe you are correct. It could be related to windows service pack update. I did not apply update to my windows because the machine is not connected to internet. I will figure it out how to download from other machine and copy it to that. Thank you for details.


  • Qt Champions 2016

    @samdol
    I hope you get same result :)
    You can download sp 1 as a file and move to other machine.


Log in to reply
 

Looks like your connection to Qt Forum was lost, please wait while we try to reconnect.