How to monitor files on an external device?

mrjj

Hi
How many files are we talking about ?
If you need to handle files that are move to other folders or renamed, then
you need to check all files before copying the truly new ones.
This gets very heavy for many files.
You need to keep record of last name, and some sort of hash / checksum for each file.
Then you can detect if one is moved but not really altered.
There is backup tools that can do this.
Its not trivial to program.
Can I ask what the overall goal is ?
and why you need your own app to do and not some backup tool?

Blaster

@mrjj The number of files is not defined, what I need is to know the space occupied by the new files.
I need to make an application of this type because I do not find any that do what I need.
It is an entity intended to control information and the owner needs to keep track of the amount of information that leaves the company. You just need to know the volume ... not what information came out.

mrjj

@Blaster
Hi
you mean you just need to calculate the amount of MB on the drive ?
With the added requirement that files already on drive, even if moved around
to new folder or renamed, do not count to that total?

Blaster

@mrjj Yes, that's it. Initially it had occurred to me to see the size when putting the device and then verify it when extracting it, but then the calculation fails when files are deleted.

mrjj

@Blaster
If think you need to use CRC for each file and store in a database.
This way you can discover if file has been moved or renamed and/or changed.
https://stackoverflow.com/questions/27939882/fast-crc-algorithm

jayden982

This post is deleted!

Blaster

@mrjj but how do I get the amount of megas that are created from the new files? In addition, the process of checking each file is too expensive in time

JonB

@Blaster
When I first saw this question days ago I would have suggested QFileSystemWatcher. I'm sure someone did but there's no mention. So I wonder why @VRonin has deleted a reply, which I would have guessed suggested that?

Anyway, let's restart. You're going to have to think out just what you want/can live with, because nothing is going to do exactly as you ask, because there's so many things you can do with files it's hard to define just what you will want here.

I agree you won't want to have examine file contents for CRC, too slow.

Most importantly: when do you need to monitor/examine this information?

If you need to in real-time, as the user is doing things, can you leverage QFileSystemWatcher for this device? I don't know whether it will support it.
If you only need to discover this when your program is run, in some shape or form you want to save a "database" of what was there before and compare it to what is there now.

Which approach?

BTW, just maybe, if all you want to know about is the "new" ones, you could simply look at the files' "creation time" attribute, if that's acceptable for your purposes?

VRonin

@JonB said in How to monitor files on an external device?:

So I wonder why @VRonin has deleted a reply

Because QFileSystemWatcher can't differentiate about what is happening to the file so the solution is not as easy as it seams

JonB

@VRonin
Then, my friend, an edit to the post explaining why not might have better than a delete :)

I still think that's where you'd start from if you needed it in real-time. And I'm asking the OP to think about just what he wants to know. Maybe all he needs to know about is new file creation, and he'll ignore rename/move? He needs to clarify whether real-time or one-off run program is what he wants.

VRonin

actually, maybe there's a easy way:

Untested Code

// QFileSystemWatcher* fwatcher = new QFileSystemWatcher;
QDir targetDir("path/to/folder/");
fwatcher->addPath(targetDir.canonicalPath());
// private: uint oldCount;
oldCount = targetDir.count();
QObject::connect(fwatcher,&QFileSystemWatcher::directoryChanged,[targetDir,&oldCount](const QString& path)->void{
if(path!=targetDir.canonicalPath()) return;
if(oldCount<targetDir.count()) qDebug() << "New File Added";
oldCount=targetDir.count();
});

Blaster

@VRonin What you propose only returns me the amount of new files in the directory and what I need is the amount of space it occupies, but that's the point ... I think I can go from there

Blaster

@JonB The truth, I had not thought about the date of creation. I think the most viable option is to use the QFileSystemWatcher and verify the date. I think that could work ... Thanks friend. I will check for this

JonB

@Blaster said in How to monitor files on an external device?:

@VRonin What you propose only returns me the amount of new files in the directory and what I need is the amount of space it occupies, but that's the point ... I think I can go from there

QFileSystemWatcher returns path of file changed. The important thing is that only fires when a change is being made. (Remember, you still need to test whatever your "device" is to make sure it fires change events.) You would use a QFileInfo on that to get the size information you are interested in. @VRonin's count() is only an example. This combination is as efficient as possible.

EDIT
Wait, I see, I get it now. QFileSystemWatcher won't tell you which file(s). OK, that's problematic.

You'd have to do something like: iterate all children by creation time to spot which ones are new. At least you'll only do it when something gets added or deleted, but not great.

Go back to question: do you need to know in real-time when user does something, or just once at some specified point when they run an app of yours?

P.S. What OS? If Windows, and you find you really need this and have to roll your own, maybe some of the suggestions in https://stackoverflow.com/questions/2107275/does-anyone-have-a-filesystemwatcher-like-class-in-c-winapi would help? Better: https://developersarea.wordpress.com/2014/09/26/win32-file-watcher-api-to-monitor-directory-changes/ And the WinAPI is around https://docs.microsoft.com/en-gb/windows/desktop/api/winbase/nf-winbase-readdirectorychangesw , that leads to event "FILE_ACTION_ADDED 0x00000001 The file was added to the directory. ". You may need the Win level to get exactly what you want specifically rather than QFileSystemWatcher.

Otherwise, this might sound like a journaling requirement? It's all getting a bit hairy...!

Blaster

@JonB Exact. It is a bit tangled.
I need to know the volume of information, by flash drive inserted in the pc. It is not necessary for the user to know that the new file was registered in the application, only those who have the administration password of the application should know it, when they open it. The OS is Windows.