v6.2.0

changelog.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## 2025-12-10 - 6.2.0 - feat(watchers)
+Improve Node watcher robustness: file-level inode tracking, abortable restarts, restart race guards, and untracked-file handling
+
+- Add file-level inode tracking to detect delete+recreate (editor atomic saves) and emit correct 'change'/'add' events
+- Make restart delays abortable via AbortController so stop() cancels pending restarts and prevents orphan watchers
+- Guard against concurrent/dual restarts with restartingPaths to avoid race conditions between health checks and error handlers
+- Emit 'unlink' for deletions of previously untracked files (files created after initial scan) and clean up inode state
+- Track file inodes during initial scan and update/cleanup inode state on events
+- Improve logging for restart/inode/delete+recreate scenarios and update documentation/readme hints to v6.2.0+
+
+## 2025-12-08 - 6.1.1 - fix(watchers/watcher.node)
+Improve Node watcher robustness: inode tracking, ENOSPC detection, enhanced health checks and temp-file handling
+
+- Track directory inodes (watchedInodes) and restart watchers if inode changes are detected (addresses stale watchers when directories are replaced).
+- Health check now validates inode stability and explicitly detects ENOSPC (inotify max_user_watches) errors, emitting errors and logging a recommended fix command.
+- Detect ENOSPC in FSWatcher error events and log guidance to increase inotify limits.
+- Clear inode tracking state on watcher stop to avoid stale state across restarts.
+- Improve temporary file handling and logging to avoid dropping events for atomic writes (only skip pure temp files and log skipped temp events).
+- Documentation (readme.hints.md) updated with robustness notes, known fs.watch limitations, and example logs.
+
 ## 2025-12-08 - 6.1.0 - feat(watcher.node)
 Add automatic restart, periodic health checks, and safe event emission to Node watcher; improve logging and stat handling
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartwatch",
-  "version": "6.1.0",
+  "version": "6.2.0",
   "private": false,
   "description": "A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.",
   "main": "dist_ts/index.js",

readme.hints.md

@@ -73,22 +73,40 @@ The `WriteStabilizer` class replaces chokidar's built-in write stabilization:
 
 ### Robustness Features (v6.1.0+)
 
-The Node.js watcher includes automatic recovery mechanisms:
+The Node.js watcher includes automatic recovery mechanisms based on learnings from [chokidar](https://github.com/paulmillr/chokidar) and known [fs.watch issues](https://github.com/nodejs/node/issues/47058):
 
 **Auto-restart on failure:**
 - Watchers automatically restart when errors occur
 - Exponential backoff (1s → 30s max)
 - Maximum 3 retry attempts before giving up
+- **v6.2.0+**: Race condition guards prevent orphan watchers when `stop()` is called during restart
+
+**Inode tracking (critical for long-running watchers):**
+- `fs.watch()` watches the **inode**, not the path!
+- When directories are replaced (git checkout, atomic saves), the inode changes
+- Health check detects inode changes and restarts the watcher
+- **v6.2.0+**: File-level inode tracking detects delete+recreate (common editor save pattern)
+- This is the most common cause of "watcher stops working after some time"
 
 **Health check monitoring:**
 - 30-second periodic health checks
 - Detects when watched paths disappear
-- Triggers automatic restart when issues detected
+- Detects inode changes (directory replacement)
+- Detects ENOSPC errors (inotify limit exceeded)
+- **v6.2.0+**: Protected against dual-restart race conditions (health check + error handler)
+
+**ENOSPC detection (Linux inotify limit):**
+- Detects when `/proc/sys/fs/inotify/max_user_watches` is exceeded
+- Logs fix command: `echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p`
 
 **Error isolation:**
 - Subscriber errors don't crash the watcher
 - All events emitted via `safeEmit()` with try-catch
 
+**Untracked file handling (v6.2.0+):**
+- Files created after initial scan are properly detected
+- Untracked file deletions emit `unlink` events instead of being silently dropped
+
 **Verbose logging:**
 - All lifecycle events logged with `[smartwatch]` prefix
 - Helps debug watcher issues in production
@@ -100,8 +118,18 @@ Example log output:
 [smartwatch] Starting health check (every 30s)
 [smartwatch] Watcher started with 1 active watcher(s)
 [smartwatch] Health check: 1 watchers active
+[smartwatch] Inode changed for ./src: 12345 -> 67890
+[smartwatch] fs.watch watches inode, not path - restarting watcher
+[smartwatch] File inode changed (delete+recreate): ./src/file.ts
 ```
 
+### Known fs.watch Limitations
+
+1. **Watches inode, not path** - If a directory is replaced, watcher goes stale
+2. **inotify limits on Linux** - Default `max_user_watches` (8192) may be too low
+3. **No events for some atomic writes** - Some editors' save patterns may not trigger events
+4. **Platform differences** - Linux uses inotify, macOS uses FSEvents/kqueue
+
 ### Testing
 
 ```bash

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartwatch',
-  version: '6.1.0',
+  version: '6.2.0',
   description: 'A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.'
 }

ts/watchers/watcher.node.ts

@@ -19,6 +19,21 @@ export class NodeWatcher implements IWatcher {
   private restartAttempts: Map<string, number> = new Map();
   private healthCheckInterval: NodeJS.Timeout | null = null;
 
+  // Inode tracking - detect when directories are replaced (atomic saves, etc.)
+  // fs.watch watches the inode, not the path. If inode changes, we need to restart.
+  private watchedInodes: Map<string, bigint> = new Map();
+
+  // File inode tracking - detect when individual files are deleted and recreated
+  // This is critical: editors delete+recreate files, fs.watch watches OLD inode!
+  // See: https://github.com/paulmillr/chokidar/issues/972
+  private fileInodes: Map<string, bigint> = new Map();
+
+  // Abort controllers for pending restart delays - prevents orphan watchers on stop()
+  private restartAbortControllers: Map<string, AbortController> = new Map();
+
+  // Prevent concurrent restarts for the same path (health check + error can race)
+  private restartingPaths: Set<string> = new Set();
+
   // Configuration constants
   private static readonly MAX_RETRIES = 3;
   private static readonly INITIAL_RESTART_DELAY = 1000;
@@ -43,8 +58,19 @@ export class NodeWatcher implements IWatcher {
 
   /**
    * Restart a watcher after an error with exponential backoff
+   * Includes guards against:
+   * - Dual restart race condition (health check + error handler calling simultaneously)
+   * - Orphan watchers when stop() is called during restart delay
    */
   private async restartWatcher(basePath: string, error: Error): Promise<void> {
+    // Guard: Prevent concurrent restarts for the same path
+    if (this.restartingPaths.has(basePath)) {
+      console.log(`[smartwatch] Restart already in progress for ${basePath}, skipping`);
+      return;
+    }
+    this.restartingPaths.add(basePath);
+
+    try {
       const attempts = (this.restartAttempts.get(basePath) || 0) + 1;
       this.restartAttempts.set(basePath, attempts);
 
@@ -72,10 +98,34 @@ export class NodeWatcher implements IWatcher {
         this.watchers.delete(basePath);
       }
 
-    // Exponential backoff
+      // Exponential backoff with AbortController (so stop() can cancel)
       const delay = this.restartDelays.get(basePath) || NodeWatcher.INITIAL_RESTART_DELAY;
       console.log(`[smartwatch] Waiting ${delay}ms before restart...`);
-    await new Promise((resolve) => setTimeout(resolve, delay));
+
+      const abortController = new AbortController();
+      this.restartAbortControllers.set(basePath, abortController);
+
+      try {
+        await new Promise<void>((resolve, reject) => {
+          const timeout = setTimeout(resolve, delay);
+          abortController.signal.addEventListener('abort', () => {
+            clearTimeout(timeout);
+            reject(new Error('Restart aborted by stop()'));
+          });
+        });
+      } catch (abortError) {
+        console.log(`[smartwatch] Restart aborted for ${basePath}`);
+        return; // stop() was called, don't continue
+      } finally {
+        this.restartAbortControllers.delete(basePath);
+      }
+
+      // Double-check we're still watching after the delay
+      if (!this._isWatching) {
+        console.log(`[smartwatch] Watcher stopped during restart delay, aborting`);
+        return;
+      }
+
       this.restartDelays.set(basePath, Math.min(delay * 2, NodeWatcher.MAX_RESTART_DELAY));
 
       try {
@@ -85,20 +135,33 @@ export class NodeWatcher implements IWatcher {
         this.restartAttempts.set(basePath, 0);
       } catch (restartError) {
         console.error(`[smartwatch] Restart failed for ${basePath}:`, restartError);
+        // Clear restartingPaths before recursive call
+        this.restartingPaths.delete(basePath);
         this.restartWatcher(basePath, restartError as Error); // Recursive retry
+        return; // Don't delete from restartingPaths again in finally
+      }
+    } finally {
+      this.restartingPaths.delete(basePath);
     }
   }
 
   /**
    * Start periodic health checks to detect silent failures
+   * Checks for:
+   * 1. Path no longer exists
+   * 2. Inode changed (directory was replaced - fs.watch watches inode, not path!)
    */
   private startHealthCheck(): void {
     console.log('[smartwatch] Starting health check (every 30s)');
     this.healthCheckInterval = setInterval(async () => {
       console.log(`[smartwatch] Health check: ${this.watchers.size} watchers active`);
       for (const [basePath] of this.watchers) {
-        const stats = await this.statSafe(basePath);
+        try {
-        if (!stats && this._isWatching) {
+          const stats = await fs.promises.stat(basePath);
+          const currentInode = stats.ino;
+          const previousInode = this.watchedInodes.get(basePath);
+
+          if (!stats) {
             console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
             this.safeEmit({
               type: 'error',
@@ -106,6 +169,25 @@ export class NodeWatcher implements IWatcher {
               error: new Error('Watched path no longer exists')
             });
             this.restartWatcher(basePath, new Error('Watched path disappeared'));
+          } else if (previousInode !== undefined && BigInt(currentInode) !== previousInode) {
+            // CRITICAL: Inode changed! fs.watch is now watching a stale inode.
+            // This happens when the directory is replaced (atomic operations, git checkout, etc.)
+            console.warn(`[smartwatch] Inode changed for ${basePath}: ${previousInode} -> ${currentInode}`);
+            console.warn('[smartwatch] fs.watch watches inode, not path - restarting watcher');
+            this.restartWatcher(basePath, new Error('Inode changed - directory was replaced'));
+          }
+        } catch (error: any) {
+          if (error.code === 'ENOENT') {
+            console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
+            this.restartWatcher(basePath, new Error('Watched path disappeared'));
+          } else if (error.code === 'ENOSPC') {
+            // inotify watch limit exceeded - critical system issue
+            console.error(`[smartwatch] ENOSPC: inotify watch limit exceeded!`);
+            console.error('[smartwatch] Fix: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
+            this.safeEmit({ type: 'error', path: basePath, error });
+          } else {
+            console.error(`[smartwatch] Health check error for ${basePath}:`, error);
+          }
         }
       }
     }, NodeWatcher.HEALTH_CHECK_INTERVAL);
@@ -179,6 +261,13 @@ export class NodeWatcher implements IWatcher {
     // Stop health check monitoring
     this.stopHealthCheck();
 
+    // Abort all pending restart delays (prevents orphan watchers)
+    for (const [path, controller] of this.restartAbortControllers) {
+      console.log(`[smartwatch] Aborting pending restart for: ${path}`);
+      controller.abort();
+    }
+    this.restartAbortControllers.clear();
+
     // Cancel all pending debounced emits
     for (const timeout of this.pendingEmits.values()) {
       clearTimeout(timeout);
@@ -193,9 +282,12 @@ export class NodeWatcher implements IWatcher {
     this.watchers.clear();
     this.watchedFiles.clear();
 
-    // Clear restart tracking state
+    // Clear all tracking state
     this.restartDelays.clear();
     this.restartAttempts.clear();
+    this.watchedInodes.clear();
+    this.fileInodes.clear();
+    this.restartingPaths.clear();
 
     console.log('[smartwatch] Watcher stopped');
   }
@@ -215,6 +307,10 @@ export class NodeWatcher implements IWatcher {
       }
 
       if (stats.isDirectory()) {
+        // Store inode for health check - fs.watch watches inode, not path!
+        // If inode changes (directory replaced), watcher becomes stale
+        this.watchedInodes.set(watchPath, BigInt(stats.ino));
+
         // Watch the directory with recursive option (Node.js 20+ supports this on all platforms)
         const watcher = fs.watch(
           watchPath,
@@ -226,8 +322,15 @@ export class NodeWatcher implements IWatcher {
           }
         );
 
-        watcher.on('error', (error) => {
+        watcher.on('error', (error: NodeJS.ErrnoException) => {
           console.error(`[smartwatch] FSWatcher error event on ${watchPath}:`, error);
+
+          // Detect inotify watch limit exceeded - common cause of "stops working"
+          if (error.code === 'ENOSPC') {
+            console.error('[smartwatch] CRITICAL: inotify watch limit exceeded!');
+            console.error('[smartwatch] Fix with: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
+          }
+
           this.safeEmit({ type: 'error', path: watchPath, error });
           if (this._isWatching) {
             this.restartWatcher(watchPath, error);
@@ -261,8 +364,14 @@ export class NodeWatcher implements IWatcher {
   ): void {
     const fullPath = path.join(basePath, filename);
 
-    // Skip temporary files created by editors (atomic saves)
+    // Skip temporary files - but ONLY pure temp files, not the target of atomic writes
+    // Atomic writes: editor writes to file.tmp.xxx then renames to file
+    // We need to detect the final file, so only skip files that ARE temp files
+    // and haven't been renamed to the real file yet
     if (this.isTemporaryFile(fullPath)) {
+      // For temp files, we still want to track if they get renamed TO a real file
+      // The 'rename' event fires for both source and target, so we'll catch the real file
+      console.log(`[smartwatch] Skipping temp file event: ${filename}`);
       return;
     }
 
@@ -283,6 +392,12 @@ export class NodeWatcher implements IWatcher {
 
   /**
    * Emit the actual file event after debounce
+   *
+   * Handles file inode tracking to detect delete+recreate scenarios:
+   * - fs.watch watches the inode, not the path
+   * - When editors delete+recreate files, the inode changes
+   * - Without inode tracking, events for the new file would be missed
+   * - See: https://github.com/paulmillr/chokidar/issues/972
    */
   private async emitFileEvent(
     fullPath: string,
@@ -302,38 +417,75 @@ export class NodeWatcher implements IWatcher {
             }
           } else {
             const wasWatched = this.watchedFiles.has(fullPath);
+            const currentInode = BigInt(stats.ino);
+            const previousInode = this.fileInodes.get(fullPath);
+
+            // Track file inode for delete+recreate detection
+            this.fileInodes.set(fullPath, currentInode);
             this.watchedFiles.add(fullPath);
+
+            // Check if file was recreated with different inode (delete+recreate scenario)
+            if (wasWatched && previousInode !== undefined && previousInode !== currentInode) {
+              console.log(`[smartwatch] File inode changed (delete+recreate): ${fullPath}`);
+              console.log(`[smartwatch]   Previous inode: ${previousInode}, current: ${currentInode}`);
+              // Emit as 'change' since the file content likely changed
+              this.safeEmit({ type: 'change', path: fullPath, stats });
+            } else {
               this.safeEmit({
                 type: wasWatched ? 'change' : 'add',
                 path: fullPath,
                 stats
               });
             }
+          }
         } else {
           // File doesn't exist - it was deleted
           if (this.watchedFiles.has(fullPath)) {
             const wasDir = this.isKnownDirectory(fullPath);
             this.watchedFiles.delete(fullPath);
+            this.fileInodes.delete(fullPath); // Clean up inode tracking
             this.safeEmit({
               type: wasDir ? 'unlinkDir' : 'unlink',
               path: fullPath
             });
+          } else {
+            // Fix #4: File wasn't tracked but was deleted - still emit event
+            // This handles files created after initial scan that we may have missed
+            console.log(`[smartwatch] Untracked file deleted: ${fullPath}`);
+            this.fileInodes.delete(fullPath);
+            this.safeEmit({ type: 'unlink', path: fullPath });
           }
         }
       } else if (eventType === 'change') {
         // File was modified
         if (stats && !stats.isDirectory()) {
           const wasWatched = this.watchedFiles.has(fullPath);
+          const currentInode = BigInt(stats.ino);
+          const previousInode = this.fileInodes.get(fullPath);
+
+          // Track file inode
+          this.fileInodes.set(fullPath, currentInode);
+
           if (!wasWatched) {
             // This is actually an 'add' - file wasn't being watched before
             this.watchedFiles.add(fullPath);
             this.safeEmit({ type: 'add', path: fullPath, stats });
+          } else if (previousInode !== undefined && previousInode !== currentInode) {
+            // Inode changed during 'change' event - file was replaced
+            console.log(`[smartwatch] File replaced (inode change on modify): ${fullPath}`);
+            this.safeEmit({ type: 'change', path: fullPath, stats });
           } else {
             this.safeEmit({ type: 'change', path: fullPath, stats });
           }
         } else if (!stats && this.watchedFiles.has(fullPath)) {
           // File was deleted
           this.watchedFiles.delete(fullPath);
+          this.fileInodes.delete(fullPath);
+          this.safeEmit({ type: 'unlink', path: fullPath });
+        } else if (!stats && !this.watchedFiles.has(fullPath)) {
+          // Fix #4: Untracked file deleted during 'change' event
+          console.log(`[smartwatch] Untracked file deleted (change event): ${fullPath}`);
+          this.fileInodes.delete(fullPath);
           this.safeEmit({ type: 'unlink', path: fullPath });
         }
       }
@@ -373,6 +525,8 @@ export class NodeWatcher implements IWatcher {
           await this.scanDirectory(fullPath, depth + 1);
         } else if (entry.isFile()) {
           this.watchedFiles.add(fullPath);
+          // Track file inode for delete+recreate detection
+          this.fileInodes.set(fullPath, BigInt(stats.ino));
           this.safeEmit({ type: 'add', path: fullPath, stats });
         }
       }