About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation

lwei2

When a multithreaded call to CopyFileEx reaches 100, why does it take 30-60 seconds to return the result of this function?
The specific code is as follows:

//Thread start
    m_cWorker = new cWorker();
    m_cThread = new subThread();
    qWarning()<<"Id1="<<m_cThread->currentThreadId();
    m_cWorker->moveToThread(m_cThread);
    connect(m_cThread, SIGNAL(finished()), m_cWorker, SLOT(slotSetWorkerNULL()));
    connect(this, SIGNAL(sigCopyWork()), m_cWorker, SLOT(slotCopyWork()), Qt::BlockingQueuedConnection);
    connect(m_cWorker, &cWorker::sigCopyValue, this, &subThread::slotCopyValue, Qt::QueuedConnection);
    connect(m_cWorker, &cWorker::sigExitCopyThread, this, &subThread::slotExitCopyThread, Qt::QueuedConnection);
    connect(m_cWorker, &cWorker::sigCopyFinished, this, &subThread::slotCopyFinished, Qt::QueuedConnection);
    m_cThread->start();
    emit sigCopyWork();

void cWorker::slotCopyWork()
{
                  //....
         bool bResult = copyFileProgress(srcPath, dstPath, true);  
         if(!bResult)
         {
                   QFile fileTmp(dstPath);
                    fileTmp.remove();
                    emit sigCopyValue(-5, 0);
          }
        emit sigExitCopyThread(0);
        emit sigCopyFinished();
       
}

void cWorker::copyFileProgress(QString &srcPath, QString &dstPath, bool isFileExist)
{
    dstPath.replace("\\", "/");
    if(dstPath == srcPath)
    {
        return true;
    }
    if(!QFile::exists(srcPath))
    {
        return false;
    }

    QDir *pDir = new QDir;
    bool bExist = pDir->exists(dstPath);
    qWarning()<<"bExist="<<bExist;
    if(bExist)
    {
        if(isFileExist)
            pDir->remove(dstPath);
    }

    const wchar_t *src = reinterpret_cast<const wchar_t *>(srcPath.utf16());
    const wchar_t *dst = reinterpret_cast<const wchar_t *>(dstPath.utf16());
    m_nCopyValue = 0;
    emit sigCopyValue(0, m_nCopyValue);
    qWarning()<<"m_nCopyValue="<<m_nCopyValue;
    BOOL m_bCancel = FALSE;
    qWarning()<<"startTime="<<QTime::currentTime().toString("hh:mm:ss");
    m_nRetCode = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
    qWarning()<<"endTime="<<QTime::currentTime().toString("hh:mm:ss");
    qWarning()<<"ret="<<m_nRetCode;
    if(!m_nRetCode)//When the callback progress value is 100%, CopyFileEx waits 30 to 60 seconds before returning CopyFileEx results, causing the main thread to stall
    {
        QFile fileTmp(dstPath);
        fileTmp.remove();
        qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
        return false;
    }
    else
    {
        qWarning()<<QString("CopyFile Success!");
    }
    return true;
}

//Copy the callback function
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize,  LARGE_INTEGER totalTransferred, LARGE_INTEGER streamSize,
                                        LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
    Q_UNUSED(streamSize) Q_UNUSED(streamTransferred)
            Q_UNUSED(streamNo) Q_UNUSED(callbackReason)
            Q_UNUSED(src) Q_UNUSED(dst)

            static int nRecord = 0;
    nRecord++;

    auto pObject =static_cast<cWorker *>(data);
    if(totalSize.QuadPart != 0)
    {
        pObject->m_nCopyValue = totalTransferred.QuadPart*100/totalSize.QuadPart;
        if(pObject->m_nCopyValue > 0)
        {            
            qWarning()<<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(0)).arg(QString::number(pObject->m_nCopyValue));
            emit pObject->sigCopyValue(0, pObject->m_nCopyValue);
            qApp->processEvents();
            if(pObject->m_nCopyValue == 100)
            {
                emit pObject->sigExitCopyThread(0);
                return PROGRESS_QUIET;
            }
            qWarning()<<"waiting....\n";
        }
    }

    return PROGRESS_CONTINUE;
}


//

Because the thread does not return the return value of the function immediately when the call CopyFileEx reaches 100%, it waits 30 to 60 seconds before returning its return value, so the main thread is waiting for the child thread, which appears to be suspended animation. Is there any way to solve this problem?

jsulm

@lwei2 Probably because the file system caches the data. So, an operation can already be reported as finished even while the OS is still writing data.

JonB

@jsulm
Absolutely! Ever seen a "Microsoft progress bar/percentage" reach 100% and stick there for ages (e.g. copying files or doing Windows updates)? One of my MS$ bug-bears!

lwei2

@jsulm Excuse me, do you have any way to solve this problem? Because CopyFileEx has no return value and waits 30-60 seconds to return its result, the main thread has been waiting for the child thread, causing the phenomenon of suspended animation of the main interface.

JonB

@lwei2
Since you are using a Windows function of CopyFileEx, and you say that is how it behaves, what should a Qt program do about it?

the main thread has been waiting for the child thread, causing the phenomenon of suspended animation of the main interface.

So if you don't want the main thread to be waiting on it, don't wait on it! Instead of a synchronous function returning a result, invent your own signal which you emit on conclusion of the CopyFileEx call (or conclusion of the separate thread running it, if that's what you do), and slot onto it. That's the Qt way of doing asynchronous things.

lwei2

@JonB I have sent a signal to the main thread in the callback function of CopyFileEx until the progress value reaches 100%, which is supposed to be copied because I see that the copied source and destination files are the same size. However, the CopyFileEx function waits between 30 and 60 seconds before returning a result before continuing with the child thread. The CopyFileEx function waits between 30 and 60 seconds to return a result, and I don't know where to send the signal so that CopyFileEx ends and the main interface doesn't feign death?
The code for the CopyFileEx callback function is as follows:
static DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalTransferred, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED(streamSize) Q_UNUSED(streamTransferred)
Q_UNUSED(streamNo) Q_UNUSED(callbackReason)
Q_UNUSED(src) Q_UNUSED(dst)

static int nRecord = 0;
nRecord++;

auto pObject =static_cast<subThread *>(data);
if(totalSize.QuadPart != 0)
{
    pObject->m_nCopyValue = totalTransferred.QuadPart*100/totalSize.QuadPart;
    if(pObject->m_nCopyValue != 0 && pObject->m_nCopyValue%50 == 0)
        qWarning()<<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject->m_nCopyValue));
    if(pObject->m_nCopyValue > 0)
    {
        emit pObject->sigCopyValue(pObject->m_nCopyValue);
        qApp->processEvents();
        if(pObject->m_nCopyValue == 100)
        {
            pObject->m_bCancel = TRUE;
        }
    }
}

return PROGRESS_CONTINUE;

}

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

do you have any way to solve this problem?

You should check the CopyFileEx documentation.

lwei2

@jsulm I've looked at the CopyFileEx documentation and tried to find another way, but it hasn't worked. The CopyFileEx callback function is as follows:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
Static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
Emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/ //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
Emit pObject - >sigExitCopyThread(0);
Return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 I actually mean to check whether there is a way to force CopyFileEx to only report 100% if everething was really copied. Or to search for other calls which maybe behave like this.

lwei2

@jsulm I've tried to call CopyFileEx to copy a file greater than 1GB and the time it takes to manually copy a file greater than 1GB is about the same. However, when CopyFileEx's callback function copies to 100%, it may not actually copy everything, so CopyFileEx waits between 30 and 60 seconds before returning its results. During this time, I didn't know what data to send to the main thread, because the CopyFileEx callback was copying at 100%. In this case, what can you do to signal the main thread to prevent the child thread from suspended animation while waiting for CopyFileEx's return value?

jsulm

@lwei2 You could simply emit a signal just after CopyFileEx finishes.

JonB

@jsulm
...which is what I answered earlier on...

lwei2

@JonB @jsulm
I tried to send a signal to exit the CopyFileEx callback when the copy progress reached 100%. Even so, it stays between 30 and 60 seconds before responding to the signal.
The following is my CopyFileEx callback code:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/* //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 @JonB is not talking about the callback. What he means is:

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
return true;

lwei2

@jsulm

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.
return true;

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.

I don't understand what you are expecting. You have said CopyFileEx() takes an extra 30 seconds, after it "reports" 100%, to complete. So that's what it takes to wait for it to complete.

If you really want to accept the signal for completion at the moment it reports 100%: You cannot make CopyFileEx somehow "return" at the point, because it doesn't offer it. As I said to you earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire.

lwei2

@JonB I hope the main thread doesn't get stuck during the copy process because the child thread is waiting for the CopyFIleEx copy. As you said to earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire. Finally, the main thread is still stuck for 30 to 60 seconds.

DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
pObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/*the result of the CopyFileEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);//send signals to the main thread
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

the main thread is still stuck for 30 to 60 seconds

Then please show what you are doing in the main thread! Do you wait there?

lwei2

@jsulm I'm just showing the progress value on the main thread. I didn't wait, the child thread itself waited 30 to 60 seconds while calling CopyFIleEx before continuing with the rest of the code. Because the child thread is running on the main thread, if the child thread is waiting without sending a signal to the main thread, the main thread will also be stuck.

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

Because the child thread is running on the main thread

This makes no sense! A child thread cannot run on main thread. Threads are independent from each other. I'm already quite confused from what you're writing...