About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation

JonB

@lwei2
Since you are using a Windows function of CopyFileEx, and you say that is how it behaves, what should a Qt program do about it?

the main thread has been waiting for the child thread, causing the phenomenon of suspended animation of the main interface.

So if you don't want the main thread to be waiting on it, don't wait on it! Instead of a synchronous function returning a result, invent your own signal which you emit on conclusion of the CopyFileEx call (or conclusion of the separate thread running it, if that's what you do), and slot onto it. That's the Qt way of doing asynchronous things.

lwei2

@JonB I have sent a signal to the main thread in the callback function of CopyFileEx until the progress value reaches 100%, which is supposed to be copied because I see that the copied source and destination files are the same size. However, the CopyFileEx function waits between 30 and 60 seconds before returning a result before continuing with the child thread. The CopyFileEx function waits between 30 and 60 seconds to return a result, and I don't know where to send the signal so that CopyFileEx ends and the main interface doesn't feign death?
The code for the CopyFileEx callback function is as follows:
static DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalTransferred, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED(streamSize) Q_UNUSED(streamTransferred)
Q_UNUSED(streamNo) Q_UNUSED(callbackReason)
Q_UNUSED(src) Q_UNUSED(dst)

static int nRecord = 0;
nRecord++;

auto pObject =static_cast<subThread *>(data);
if(totalSize.QuadPart != 0)
{
    pObject->m_nCopyValue = totalTransferred.QuadPart*100/totalSize.QuadPart;
    if(pObject->m_nCopyValue != 0 && pObject->m_nCopyValue%50 == 0)
        qWarning()<<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject->m_nCopyValue));
    if(pObject->m_nCopyValue > 0)
    {
        emit pObject->sigCopyValue(pObject->m_nCopyValue);
        qApp->processEvents();
        if(pObject->m_nCopyValue == 100)
        {
            pObject->m_bCancel = TRUE;
        }
    }
}

return PROGRESS_CONTINUE;

}

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

do you have any way to solve this problem?

You should check the CopyFileEx documentation.

lwei2

@jsulm I've looked at the CopyFileEx documentation and tried to find another way, but it hasn't worked. The CopyFileEx callback function is as follows:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
Static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
Emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/ //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
Emit pObject - >sigExitCopyThread(0);
Return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 I actually mean to check whether there is a way to force CopyFileEx to only report 100% if everething was really copied. Or to search for other calls which maybe behave like this.

lwei2

@jsulm I've tried to call CopyFileEx to copy a file greater than 1GB and the time it takes to manually copy a file greater than 1GB is about the same. However, when CopyFileEx's callback function copies to 100%, it may not actually copy everything, so CopyFileEx waits between 30 and 60 seconds before returning its results. During this time, I didn't know what data to send to the main thread, because the CopyFileEx callback was copying at 100%. In this case, what can you do to signal the main thread to prevent the child thread from suspended animation while waiting for CopyFileEx's return value?

jsulm

@lwei2 You could simply emit a signal just after CopyFileEx finishes.

JonB

@jsulm
...which is what I answered earlier on...

lwei2

@JonB @jsulm
I tried to send a signal to exit the CopyFileEx callback when the copy progress reached 100%. Even so, it stays between 30 and 60 seconds before responding to the signal.
The following is my CopyFileEx callback code:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/* //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 @JonB is not talking about the callback. What he means is:

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
return true;

lwei2

@jsulm

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.
return true;

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.

I don't understand what you are expecting. You have said CopyFileEx() takes an extra 30 seconds, after it "reports" 100%, to complete. So that's what it takes to wait for it to complete.

If you really want to accept the signal for completion at the moment it reports 100%: You cannot make CopyFileEx somehow "return" at the point, because it doesn't offer it. As I said to you earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire.

lwei2

@JonB I hope the main thread doesn't get stuck during the copy process because the child thread is waiting for the CopyFIleEx copy. As you said to earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire. Finally, the main thread is still stuck for 30 to 60 seconds.

DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
pObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/*the result of the CopyFileEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);//send signals to the main thread
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

the main thread is still stuck for 30 to 60 seconds

Then please show what you are doing in the main thread! Do you wait there?

lwei2

@jsulm I'm just showing the progress value on the main thread. I didn't wait, the child thread itself waited 30 to 60 seconds while calling CopyFIleEx before continuing with the rest of the code. Because the child thread is running on the main thread, if the child thread is waiting without sending a signal to the main thread, the main thread will also be stuck.

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

Because the child thread is running on the main thread

This makes no sense! A child thread cannot run on main thread. Threads are independent from each other. I'm already quite confused from what you're writing...

lwei2

@jsulm I didn't say that the child thread cannot run on the main thread.What I want to say is that the main thread is stalled because the child thread is waiting for the return value of CopyFileEx without running the code behind CopyFileEx.Such as:

void run()
{
    const wchar_t *src = reinterpret_cast<const wchar_t *>(srcPath.utf16());
    const wchar_t *dst = reinterpret_cast<const wchar_t *>(dstPath.utf16());
    m_nCopyValue = 0;
    emit sigCopyValue(0, m_nCopyValue);
    qWarning()<<"m_nCopyValue="<<m_nCopyValue;
    BOOL m_bCancel = FALSE;
    qWarning()<<"startTime="<<QTime::currentTime().toString("hh:mm:ss");
    CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);  //wait 30 to 60 seconds before executing the following code

//there is other code here

}

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

the main thread is stalled because the child thread is waiting for the return value of CopyFileEx

Your main thread cannot afford to wait on the child thread! If the child runs CopyFileEx it's going not going to exit until that has had its extra 30 seconds at the end, which is the whole point of what you are wanting to avoid.

That's why if it waits at all the main thread should only wait until the child signals the 100% completion.

If you are using the wait on the child thread for some other purpose, then you will need a third thread for the CopyFileEx. Whatever, the only way you will be able to avoid waiting for the CopyFileEx's extra 30 seconds is if the CopyFileEx runs in its own thread, and sends a signal when the 100% is reached.

JoeCFD

@lwei2 It is flushing time. Therefore, progress display is sort of a lie. But on Windows you may try some MS code for file copy and do not use Qt file copy. I recently coded file copy on Linux with Qt and can not do anything about flushing.
If the file size is too big, chop the file into small pieces and copy them one after another since you may need cancel feature. The optimal size is 131072.

lwei2

@JonB Could you please tell me how I can wait for it to copy to 100% before sending the signal?