About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation

lwei2

@jsulm I've looked at the CopyFileEx documentation and tried to find another way, but it hasn't worked. The CopyFileEx callback function is as follows:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
Static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
Emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/ //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
Emit pObject - >sigExitCopyThread(0);
Return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 I actually mean to check whether there is a way to force CopyFileEx to only report 100% if everething was really copied. Or to search for other calls which maybe behave like this.

lwei2

@jsulm I've tried to call CopyFileEx to copy a file greater than 1GB and the time it takes to manually copy a file greater than 1GB is about the same. However, when CopyFileEx's callback function copies to 100%, it may not actually copy everything, so CopyFileEx waits between 30 and 60 seconds before returning its results. During this time, I didn't know what data to send to the main thread, because the CopyFileEx callback was copying at 100%. In this case, what can you do to signal the main thread to prevent the child thread from suspended animation while waiting for CopyFileEx's return value?

jsulm

@lwei2 You could simply emit a signal just after CopyFileEx finishes.

JonB

@jsulm
...which is what I answered earlier on...

lwei2

@JonB @jsulm
I tried to send a signal to exit the CopyFileEx callback when the copy progress reached 100%. Even so, it stays between 30 and 60 seconds before responding to the signal.
The following is my CopyFileEx callback code:
DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
PObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/* //With or without this code, CopyFIleEx will wait between 30 and 60 seconds when the callback reaches 100% before returning the result of the CopyFIleEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 @JonB is not talking about the callback. What he means is:

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
return true;

lwei2

@jsulm

BOOL m_bCancel = FALSE;
BOOL bRet = CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);
qWarning()<<"ret="<<bRet;
if(!bRet)
{
    QFile fileTmp(dstPath);
    fileTmp.remove();
    qWarning()<<QString("CopyFile Failed, ErrorCode=%1!").arg(QString::number(GetLastError()));
    return false;
}
qWarning()<<QString("CopyFile Success!");
// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.
return true;

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

// EMIT A SIGNAL HERE
//If you send a signal here, you will still wait for CopyFileEx to return the result before sending successfully. By doing so, the main thread is still suspended animation by waiting 30 to 60 seconds for the child thread.

I don't understand what you are expecting. You have said CopyFileEx() takes an extra 30 seconds, after it "reports" 100%, to complete. So that's what it takes to wait for it to complete.

If you really want to accept the signal for completion at the moment it reports 100%: You cannot make CopyFileEx somehow "return" at the point, because it doesn't offer it. As I said to you earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire.

lwei2

@JonB I hope the main thread doesn't get stuck during the copy process because the child thread is waiting for the CopyFIleEx copy. As you said to earlier, put the call to CopyFileEx in its own thread. Then you should be able to emit your own completion signal when the callback receives the 100% notification you desire. Finally, the main thread is still stuck for 30 to 60 seconds.

DWORD CALLBACK CallBackCopyProgressFunc(LARGE_INTEGER totalSize, LARGE_INTEGER totalSize, LARGE_INTEGER Totalextension, LARGE_INTEGER streamSize,
LARGE_INTEGER streamTransferred, DWORD streamNo,DWORD callbackReason,HANDLE src,HANDLE dst,LPVOID data)
{
Q_UNUSED (streamSize) Q_UNUSED (streamTransferred)
Q_UNUSED (streamNo) Q_UNUSED (callbackReason)
Q_UNUSED (SRC) Q_UNUSED (DST)
static int nRecord = 0;
nRecord ++;
Auto pObject = static_cast<eWorker*> (data);
If (totalSize QuadPart! = 0)
{
pObject - >m_nCopyValue = totalTransferred.QuadPart100/totalSize.QuadPart;
QWarning () <<QString("emit Number of callbacks:%1,The copy progress of value:%2").arg(QString::number(nRecord)).arg(QString::number(pObject - >m_nCopyValue));
If (pObject - >m_nCopyValue > 0)
{
emit pObject - >sigCopyValue(0, pObject - >m_nCopyValue);
QApp - >processEvents();
/*the result of the CopyFileEx function.
If (pObject->m_nCopyValue == 100)
{
emit pObject - >sigExitCopyThread(0);//send signals to the main thread
return PROGRESS_QUIET;
}
*/
}
}
Return PROGRESS_CONTINUE;
}

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

the main thread is still stuck for 30 to 60 seconds

Then please show what you are doing in the main thread! Do you wait there?

lwei2

@jsulm I'm just showing the progress value on the main thread. I didn't wait, the child thread itself waited 30 to 60 seconds while calling CopyFIleEx before continuing with the rest of the code. Because the child thread is running on the main thread, if the child thread is waiting without sending a signal to the main thread, the main thread will also be stuck.

jsulm

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

Because the child thread is running on the main thread

This makes no sense! A child thread cannot run on main thread. Threads are independent from each other. I'm already quite confused from what you're writing...

lwei2

@jsulm I didn't say that the child thread cannot run on the main thread.What I want to say is that the main thread is stalled because the child thread is waiting for the return value of CopyFileEx without running the code behind CopyFileEx.Such as:

void run()
{
    const wchar_t *src = reinterpret_cast<const wchar_t *>(srcPath.utf16());
    const wchar_t *dst = reinterpret_cast<const wchar_t *>(dstPath.utf16());
    m_nCopyValue = 0;
    emit sigCopyValue(0, m_nCopyValue);
    qWarning()<<"m_nCopyValue="<<m_nCopyValue;
    BOOL m_bCancel = FALSE;
    qWarning()<<"startTime="<<QTime::currentTime().toString("hh:mm:ss");
    CopyFileEx((LPCWSTR)src,(LPCWSTR)dst, CallBackCopyProgressFunc, this, (LPBOOL)m_bCancel, COPY_FILE_FAIL_IF_EXISTS);  //wait 30 to 60 seconds before executing the following code

//there is other code here

}

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

the main thread is stalled because the child thread is waiting for the return value of CopyFileEx

Your main thread cannot afford to wait on the child thread! If the child runs CopyFileEx it's going not going to exit until that has had its extra 30 seconds at the end, which is the whole point of what you are wanting to avoid.

That's why if it waits at all the main thread should only wait until the child signals the 100% completion.

If you are using the wait on the child thread for some other purpose, then you will need a third thread for the CopyFileEx. Whatever, the only way you will be able to avoid waiting for the CopyFileEx's extra 30 seconds is if the CopyFileEx runs in its own thread, and sends a signal when the 100% is reached.

JoeCFD

@lwei2 It is flushing time. Therefore, progress display is sort of a lie. But on Windows you may try some MS code for file copy and do not use Qt file copy. I recently coded file copy on Linux with Qt and can not do anything about flushing.
If the file size is too big, chop the file into small pieces and copy them one after another since you may need cancel feature. The optimal size is 131072.

lwei2

@JonB Could you please tell me how I can wait for it to copy to 100% before sending the signal?

SGaist

Hi,

Did you consider using the COPY_FILE_NO_BUFFERING flag as recommended in the documentation for large files ?

JonB

@lwei2 said in About multithreaded call CopyFileEx copy large files (more than 1GB) when the phenomenon of suspended animation:

@JonB Could you please tell me how I can wait for it to copy to 100% before sending the signal?

I really don't understand. You said that presently CopyFileEx does not return till after the extra 30 seconds of "flushing" at the end, so if that is what you mean don't send the signal till CopyFileEx is done, which is what you have now. If you mean when the progress reaches 100%, before the extra 30 seconds, then you detect that in your CallBackCopyProgressFunc, and send the signal from there. This latter approach will require the CopyFileEx be run in its own thread.

In any case, first try @SGaist excellent spot of COPY_FILE_NO_BUFFERING flag, it might change the behaviour over to what you want by eliminating a large "flush" at the end.

lwei2

@SGaist I tried to use COPY_FILE_NO_BUFFERING, but I had the same problem.