fix(watchers/watcher.node): Improve Node watcher robustness: inode tracking, ENOSPC detection, enhanced health checks and temp-file handling

changelog.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## 2025-12-08 - 6.1.1 - fix(watchers/watcher.node)
+Improve Node watcher robustness: inode tracking, ENOSPC detection, enhanced health checks and temp-file handling
+
+- Track directory inodes (watchedInodes) and restart watchers if inode changes are detected (addresses stale watchers when directories are replaced).
+- Health check now validates inode stability and explicitly detects ENOSPC (inotify max_user_watches) errors, emitting errors and logging a recommended fix command.
+- Detect ENOSPC in FSWatcher error events and log guidance to increase inotify limits.
+- Clear inode tracking state on watcher stop to avoid stale state across restarts.
+- Improve temporary file handling and logging to avoid dropping events for atomic writes (only skip pure temp files and log skipped temp events).
+- Documentation (readme.hints.md) updated with robustness notes, known fs.watch limitations, and example logs.
+
 ## 2025-12-08 - 6.1.0 - feat(watcher.node)
 Add automatic restart, periodic health checks, and safe event emission to Node watcher; improve logging and stat handling
 

readme.hints.md

@@ -73,17 +73,28 @@ The `WriteStabilizer` class replaces chokidar's built-in write stabilization:
 
 ### Robustness Features (v6.1.0+)
 
-The Node.js watcher includes automatic recovery mechanisms:
+The Node.js watcher includes automatic recovery mechanisms based on learnings from [chokidar](https://github.com/paulmillr/chokidar) and known [fs.watch issues](https://github.com/nodejs/node/issues/47058):
 
 **Auto-restart on failure:**
 - Watchers automatically restart when errors occur
 - Exponential backoff (1s → 30s max)
 - Maximum 3 retry attempts before giving up
 
+**Inode tracking (critical for long-running watchers):**
+- `fs.watch()` watches the **inode**, not the path!
+- When directories are replaced (git checkout, atomic saves), the inode changes
+- Health check detects inode changes and restarts the watcher
+- This is the most common cause of "watcher stops working after some time"
+
 **Health check monitoring:**
 - 30-second periodic health checks
 - Detects when watched paths disappear
-- Triggers automatic restart when issues detected
+- Detects inode changes (directory replacement)
+- Detects ENOSPC errors (inotify limit exceeded)
+
+**ENOSPC detection (Linux inotify limit):**
+- Detects when `/proc/sys/fs/inotify/max_user_watches` is exceeded
+- Logs fix command: `echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p`
 
 **Error isolation:**
 - Subscriber errors don't crash the watcher
@@ -100,8 +111,17 @@ Example log output:
 [smartwatch] Starting health check (every 30s)
 [smartwatch] Watcher started with 1 active watcher(s)
 [smartwatch] Health check: 1 watchers active
+[smartwatch] Inode changed for ./src: 12345 -> 67890
+[smartwatch] fs.watch watches inode, not path - restarting watcher
 ```
 
+### Known fs.watch Limitations
+
+1. **Watches inode, not path** - If a directory is replaced, watcher goes stale
+2. **inotify limits on Linux** - Default `max_user_watches` (8192) may be too low
+3. **No events for some atomic writes** - Some editors' save patterns may not trigger events
+4. **Platform differences** - Linux uses inotify, macOS uses FSEvents/kqueue
+
 ### Testing
 
 ```bash

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartwatch',
-  version: '6.1.0',
+  version: '6.1.1',
   description: 'A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.'
 }

ts/watchers/watcher.node.ts

@@ -19,6 +19,10 @@ export class NodeWatcher implements IWatcher {
   private restartAttempts: Map<string, number> = new Map();
   private healthCheckInterval: NodeJS.Timeout | null = null;
 
+  // Inode tracking - detect when directories are replaced (atomic saves, etc.)
+  // fs.watch watches the inode, not the path. If inode changes, we need to restart.
+  private watchedInodes: Map<string, bigint> = new Map();
+
   // Configuration constants
   private static readonly MAX_RETRIES = 3;
   private static readonly INITIAL_RESTART_DELAY = 1000;
@@ -91,14 +95,21 @@ export class NodeWatcher implements IWatcher {
 
   /**
    * Start periodic health checks to detect silent failures
+   * Checks for:
+   * 1. Path no longer exists
+   * 2. Inode changed (directory was replaced - fs.watch watches inode, not path!)
    */
   private startHealthCheck(): void {
     console.log('[smartwatch] Starting health check (every 30s)');
     this.healthCheckInterval = setInterval(async () => {
       console.log(`[smartwatch] Health check: ${this.watchers.size} watchers active`);
       for (const [basePath] of this.watchers) {
-        const stats = await this.statSafe(basePath);
-        if (!stats && this._isWatching) {
+        try {
+          const stats = await fs.promises.stat(basePath);
+          const currentInode = stats.ino;
+          const previousInode = this.watchedInodes.get(basePath);
+
+          if (!stats) {
             console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
             this.safeEmit({
               type: 'error',
@@ -106,6 +117,25 @@ export class NodeWatcher implements IWatcher {
               error: new Error('Watched path no longer exists')
             });
             this.restartWatcher(basePath, new Error('Watched path disappeared'));
+          } else if (previousInode !== undefined && BigInt(currentInode) !== previousInode) {
+            // CRITICAL: Inode changed! fs.watch is now watching a stale inode.
+            // This happens when the directory is replaced (atomic operations, git checkout, etc.)
+            console.warn(`[smartwatch] Inode changed for ${basePath}: ${previousInode} -> ${currentInode}`);
+            console.warn('[smartwatch] fs.watch watches inode, not path - restarting watcher');
+            this.restartWatcher(basePath, new Error('Inode changed - directory was replaced'));
+          }
+        } catch (error: any) {
+          if (error.code === 'ENOENT') {
+            console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
+            this.restartWatcher(basePath, new Error('Watched path disappeared'));
+          } else if (error.code === 'ENOSPC') {
+            // inotify watch limit exceeded - critical system issue
+            console.error(`[smartwatch] ENOSPC: inotify watch limit exceeded!`);
+            console.error('[smartwatch] Fix: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
+            this.safeEmit({ type: 'error', path: basePath, error });
+          } else {
+            console.error(`[smartwatch] Health check error for ${basePath}:`, error);
+          }
         }
       }
     }, NodeWatcher.HEALTH_CHECK_INTERVAL);
@@ -196,6 +226,7 @@ export class NodeWatcher implements IWatcher {
     // Clear restart tracking state
     this.restartDelays.clear();
     this.restartAttempts.clear();
+    this.watchedInodes.clear();
 
     console.log('[smartwatch] Watcher stopped');
   }
@@ -215,6 +246,10 @@ export class NodeWatcher implements IWatcher {
       }
 
       if (stats.isDirectory()) {
+        // Store inode for health check - fs.watch watches inode, not path!
+        // If inode changes (directory replaced), watcher becomes stale
+        this.watchedInodes.set(watchPath, BigInt(stats.ino));
+
         // Watch the directory with recursive option (Node.js 20+ supports this on all platforms)
         const watcher = fs.watch(
           watchPath,
@@ -226,8 +261,15 @@ export class NodeWatcher implements IWatcher {
           }
         );
 
-        watcher.on('error', (error) => {
+        watcher.on('error', (error: NodeJS.ErrnoException) => {
           console.error(`[smartwatch] FSWatcher error event on ${watchPath}:`, error);
+
+          // Detect inotify watch limit exceeded - common cause of "stops working"
+          if (error.code === 'ENOSPC') {
+            console.error('[smartwatch] CRITICAL: inotify watch limit exceeded!');
+            console.error('[smartwatch] Fix with: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
+          }
+
           this.safeEmit({ type: 'error', path: watchPath, error });
           if (this._isWatching) {
             this.restartWatcher(watchPath, error);
@@ -261,8 +303,14 @@ export class NodeWatcher implements IWatcher {
   ): void {
     const fullPath = path.join(basePath, filename);
 
-    // Skip temporary files created by editors (atomic saves)
+    // Skip temporary files - but ONLY pure temp files, not the target of atomic writes
+    // Atomic writes: editor writes to file.tmp.xxx then renames to file
+    // We need to detect the final file, so only skip files that ARE temp files
+    // and haven't been renamed to the real file yet
     if (this.isTemporaryFile(fullPath)) {
+      // For temp files, we still want to track if they get renamed TO a real file
+      // The 'rename' event fires for both source and target, so we'll catch the real file
+      console.log(`[smartwatch] Skipping temp file event: ${filename}`);
       return;
     }