Monitoring File System Changes (Watchdog)

  flowchart TD
  A["you save a file"] --> B["the OS emits events"]
  B --> C["created"]
  B --> D["modified -- often more than once"]
  B --> E["moved  -- editors write a temp file then move it"]
  C --> F["your handler fires"]
  D --> F
  E --> F
  F --> G{"is the file finished being written?"}
  G -->|"maybe not"| H["reading it now can give a partial file"]
  G --> I["debounce: wait for quiet, then act once"]
  H --> I
  I --> J["retry on PermissionError -- the writer may still hold it"]

What watchdog does

watchdog lets you react to:

• file created
• file modified
• file deleted

This is useful for:

• auto-processing new files
• syncing folders

Example watcher

from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler
 
 
class Handler(FileSystemEventHandler):
    def on_created(self, event):
        if event.is_directory:
            return
        print("created:", event.src_path)
 
 
if __name__ == "__main__":
    import time
 
    path = "."
    observer = Observer()
    observer.schedule(Handler(), path=path, recursive=True)
    observer.start()
 
    print("watching", path)
    try:
        while True:
            time.sleep(1)
    except KeyboardInterrupt:
        observer.stop()
 
    observer.join()

How watchdog works

The operating system can tell a program when something in a folder changes. On Linux this is inotify, on macOS FSEvents, on Windows ReadDirectoryChangesW. watchdog wraps all three behind one interface, so the same script runs on each system.

There are two objects to know:

• An observer runs in a background thread. You tell it which folder to watch with schedule(handler, path, recursive=True).
• A handler is your class. The observer calls its methods (on_created, on_modified, on_deleted, on_moved) with an event object. The event has src_path and is_directory.

The main thread only needs to stay alive, which is why the example loops on time.sleep(1). Press Ctrl+C, catch KeyboardInterrupt, and call observer.stop() followed by observer.join() to shut down cleanly.

Waiting until the file is finished

The diagram above shows the main trap. on_created fires when the file first appears, not when it has been fully written. If you open a large download immediately, you may read half of it. A simple fix is to wait until the size stops changing.

import time
from pathlib import Path
from watchdog.events import FileSystemEventHandler
 
 
def wait_until_stable(path: Path, checks: int = 3, delay: float = 0.5) -> bool:
    last = -1
    stable = 0
    while stable < checks:
        try:
            size = path.stat().st_size
        except FileNotFoundError:
            return False  # removed while we waited
        stable = stable + 1 if size == last else 0
        last = size
        time.sleep(delay)
    return True
 
 
class NewFileHandler(FileSystemEventHandler):
    def on_created(self, event):
        if event.is_directory:
            return
        path = Path(event.src_path)
        if wait_until_stable(path):
            print("ready:", path.name, path.stat().st_size, "bytes")

Use NewFileHandler in place of Handler in the example above. Because it sleeps, it blocks the observer thread for that file. For heavy work, hand the path to a queue or a worker thread.

Common mistakes

• Writing into the folder you are watching. If your handler creates files in the watched folder, each one triggers another event and you get an endless loop. Write results somewhere else.
• Acting on every on_modified. One save can raise several. Debounce, or track which files you already handled.
• Ignoring directories. Directory events arrive too. Check event.is_directory.
• Not stopping the observer. Always call stop() and join(), or the thread keeps the process alive.

Adapting it

Point the observer at a Downloads folder and move each finished PDF into a PDFs folder. Or watch a drop folder and run a CSV cleanup on every new file. Add a dry-run flag that only prints the move, as you would for any script that touches files.

Check your understanding

Notes

• Install dependency: watchdog
• Keep handlers fast (offload heavy work)