Suggestion for new QFileSystemWatcher.

[stefbon]

Hi,

I've been working on a filesystem event watcher for some time now. As user of KDE I contacted first the developers of kDirWatch. They send me to developers of Qt, because the KDirWatch is being deprecated, as the same is done already in Qt.

Then contacted Thiago Macieira for suggestions about howto test notifyfs (that's the name of it) as backend of QFileSystemWatcher. He suggested me posting a message here, since QFileSystemWatcher is "flawed by design" (his words) and a replacement is required.

So here I present notifyfs. Here a copy from the project website:

Notifyfs is a FUSE overlay fs, which tries to stay uptodate with the underlying filesystems.

It monitors:

a. mounts via watching /proc/self/mountinfo

it can forward added and removed mounts via messages to connected clients.

(it handles mounts which are managed by autofs different, here a unmount is treated as set to "sleep", instead of remove, cause it can get back ("wakeup") any moment)

b. filesystem changes via the default backend, on linux that is inotify

c. filesystem changes on filesystems like cifs, nfs, FUSE based fs's and other network filesystems by forwarding the watch to the filesystem. With inotify the target fs is not aware a watch has been set. By forwarding the watch to the target fs, (which should be connected to notifyfs somehow) this fs can setup a backend specific watch.

Connecting notifyfs with a fs in userspace (FUSE!) is easy, with cifs this is difficult, but not impossible. I guess this is possible with netlink.

I'm experimenting with making notifyfs connect to a notifyfs process on another host, and test it with networkfilesystems like sshfs. This is just an experiment, this is not the way I want it to work: filesystems themselves should setup a watch on their backend, and forward any gsevent back to notifyfs.

Futher:

• notifyfs uses the inotify format for describing a fsevent, and setting a watch.

• notifyfs tries to minimize the number of events send to clients

• notifyfs can send very detailed information, but also basic signals to the client. I've read that the implementation of KDE (kdirwatch) and QT internally the status created, deleted and dirty is used. It's very possible to make notifyfs only send that information.

• notifyfs sends with every fsevent the id the client uses to identify the watch.

Recently I;ve had contact with a developer of cifs (CIFS on Linux) and aggreed that it's possible and worth trying to make this work for cifs. It's possible for cifs to set a watch on the backend using "SMB2_CHANGE_NOTIFY" request. Futher I'm waiting for the notify request to become available in libsmbclient, to test with another FUSE filesystem fusesmb.

Projectpage:

http://code.google.com/p/notifyfs/

Are you interested??

Stef Bon

[koahnig]

welcome to devnet

I am certainly not deeply into the details of system file watching. However, it seems that your approach is concentrating on linux. So, the questions of what is happening with the other systems windows and mac os?

Nevertheless, it could be better to place "your suggestion on JIRA":https://bugreports.qt-project.org/secure/Dashboard.jspa

[stefbon]

Well yes I'm developing this on Linux. My intention is to create a successor to gamin. I do not intend to create soemthing on Apple, since it has already something:

https://developer.apple.com/library/mac/#documentation/Darwin/Conceptual/FSEvents_ProgGuide/Introduction/Introduction.html

and as far as i know Windows platforms have also something.

My intention is not to replace that. Why reinventing the wheel?

The reason for me to develop notifyfs is that gamin lacks some important features (like forwarding a watch), and Linux has not something simluar yet.

And the reason I post here is that I would like to know - Thiago Macieira noted the current implementation is to replaced - what are the requirments of the new fsevent watcher is and notifyfs can offer. (my plan is to make notifyfs gui toolkit indendent)

Stef

[koahnig]

It depends on the kind of implementation under Qt's hood. If there have to be separate implementations for the different OS anyhow, than it would be something to pursue. There are different implementations for sure because different stuff deep in the file systems. However, as already noted I am for sure not in those details at all. Maybe someone of the Trolls, who is, is picking it from here. Nevertheless, placing the idea on JIRA, it will require follow-up in one way or the other. So, it is IMO also a good idea to place there.

[koahnig]

I discussed with the moderators. You could send it also to the developers on "this mailing list":http://lists.qt-project.org/mailman/listinfo Hopefully, someone of there will pick it up.

[stefbon]

Aha, thanks a lot!

Stef Bon

[dangelog]

Hi stefbon. As koahnig already said: discussion about development of Qt itself happens on the official development mailing list and/or on the #qt-labs channel on Freenode. So please repost your message there. :-)

JIRA is used to track bug reports, feature requests, etc; in the particular case of QFSW, there is already a ton of reports on JIRA, so maybe adding a new one isn't very useful.

You should also coordinate with Robin Burchell's efforts of somehow fixing QFSW implementation in Qt 5.1, see

http://blog.rburchell.com/2012/03/qt-51-aka-when-qfilesystemwatcher-might.html
https://codereview.qt-project.org/#change,19274

for more info.

[andre]

It's not that the message isn't welcome here, it's more that you'll find a more focussed audience at that mailinglist. Thank you for your efforts!

I was wondering what your suggestions mean for QFileSystemWatcher. Do you propose to change the implementation of it on Linux to use your notifyfs instead of the current situation, and leave the backend implementation as is for Windows and Mac?