File system watcher sometimes does not pick up files properly even if the window service is running properly

I am sorry that I only saw this question today. I have a several years' experience with FileSystemWatcher (and with the Win32 API functions when using C++ before C#).

These are the main areas to look out for:

1) As you said, make sure the code that handles notifications is as quick as possible, so as not to potentially miss notifications. The best solution, again as you have indicated, is to add the path to a "queue" which is then processed on another thread. (You need to make sure your queue(s) are managed in a thread-safe way. Use the handy Synchronized feature).

2) Because you may get a notification about a file as soon as it is created, but it still may be being written to (you may even get several more "change" notifications) watch out that you don't actually try to process the file before it has finished being written.

4) Error handling: FileSystemWatcher can fall over for all sorts of reasons, which may depend on where the folder is that you are watching. Especially since you are using it in a service you will want to ensure that it will at least report the error to alert someone, and/or be able to recover from problems itself where possible.

3) Existing files: FileSystemWatcher will not tell you about files that exist when you create it, so when your service starts you most likely want to look for files that already exist in your watched folders, because your service may have been stopped before finishing the queue (if it's written properly this SHOULD be possible!) or files may have arrived while it was stopped.


In order to overcome and handle these issues and limitations, where I have used FileSystemWatcher I have ended up with no less than 3 (of my own) threads, 2 queues and a dictionary!
I'm afraid I can't post the code itself, so this is my description of it:

Threads:
1) The main application thread is where the FileSystemWatcher is created, as well as the other 3 threads.
2) A thread which processes the queues.
3) A thread which does nothing but wait and frequently check (every minute in my case) that the FileSystemWatcher is still operating.
4) A temporary thread used at start-up (or re-initialisation) for adding existing files to the "changed" queue.

Queues:
1) A queue of strings for "changed" (or created) file paths.
2) A queue of strings for "removed" file paths.

Dictionary with file path as the key, holding information (such as last modified time and file size) for each file.


Respond to the FileSystemWatcher events as follows:

Changed / Created: Add "FullPath" to the "changed" queue.
Deleted: Add "FullPath" to the "removed" queue.
Renamed: Add "OldFullPath" to "removed" queue, and "FullPath" to "changed" queue.
OnError: Log exception and ensure "EnableRaisingEvents" is set to false.

My second thread is then running in a loop, to do some work every configurable number of milliseconds, to process the queues as follows:
1) Go through the "removed" paths queue, for each path dequeue it and remove the entry for the path in the dictionary if it exists.
2) Go through the "changed" paths queue in a similar way, and add (or update) information about the file in the dictionary.
3) Go through each path in the dictionary:
a) Check if the last modified time stored in the information is greater than a configurable number of milliseconds. If it is, then
b) Check that the file size has not changed since the information was last checked (and update it). If it has not changed, then
c) Check that the file is not locked, by attempting to temporarily open it read-write. If this succeeds
d) Process the file (do whatever you need to do with it) and
e) Finally, add the file path to the "removed" queue.

My third thread is always initialised immediately after the FileSystemWatcher. It uses the same filtering options to go through the folder being watched adding any files it finds to the "changed" queue. This thread exits when it has finished.

My fourth and final thread runs for the lifetime of the service like the first two. It is another loop with a waiting time (e.g. 1 minute). Each time it enters the loop, it checks that the FileSystemWatcher is active (EnableRaisingEvents is true - hence why I ensure it is forced back to false in OnError). If this is false, it checks that the folder to watch exists. If the folder does not exist it will log a warning, and then go back round the loop, i.e. wait for a minute and then check again. If/when the folder does exist, it will call the initialisation method, which recreates the FileSystemWatcher and runs the third thread (to check for existing files), and then log an information message to say that the file system watcher was successfully reset. (If reset fails, an error is logged, and this thread will exit because there's no point continuing).


Well, that's a rather long winded description, but I hope it is helpful.

Regards,
Ian.

Hi All,

After couple of days i was able to find a solution by replicating the same scenario in my development enviroment just the way it was on my test environment. I requested the team member of external system to FTP sample file i post from my archive test files to my polling folder.

I have previously read in msdn where it states that files that are copied or moved may sometime only will be recognised as RENAMED event. Felt this might be applicable and so apart from CHANGED event, i also included RENAMED event to process files. This solved my issue and no files are getting clogged now in polling folder.

Replicating issue is one thing helped me. And another way to crack your issue while dealing with filesystemwatcher concepts in your coding is to use small project to test your case by monitoring the folder you watch for changes and note them down. So you can construct code accordingly.

Thanks

Hi!
I had the same problem and reading your solution I realized why. In our code we where listening to CREATED event filtering on XML files as you. This event is always triggered, but since most SFTP software supports continuation on errors the files are created using a temp extension like .PART or .TMP or similar for then to be renamed when it's fully transferred. This is the reason your RENAMED event with filter on XML is the one hitting your code as the file isn't named XML until renamed.
Your,
Lasse S. Jensen