Compare commits

..

10 Commits

Author SHA1 Message Date
8677f61da1 v6.1.1
Some checks failed
Default (tags) / security (push) Failing after 17s
Default (tags) / test (push) Failing after 19s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-12-08 19:31:48 +00:00
0c236d44d3 fix(watchers/watcher.node): Improve Node watcher robustness: inode tracking, ENOSPC detection, enhanced health checks and temp-file handling 2025-12-08 19:31:48 +00:00
913c14bfcf v6.1.0
Some checks failed
Default (tags) / security (push) Failing after 16s
Default (tags) / test (push) Failing after 14s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-12-08 17:48:50 +00:00
2ce056f401 feat(watcher.node): Add automatic restart, periodic health checks, and safe event emission to Node watcher; improve logging and stat handling 2025-12-08 17:48:50 +00:00
1c77a94b96 v6.0.0
Some checks failed
Default (tags) / security (push) Failing after 15s
Default (tags) / test (push) Failing after 17s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-12-08 16:06:18 +00:00
097ea96e99 BREAKING CHANGE(watchers): Replace polling-based write stabilization with debounce-based event coalescing and simplify watcher options 2025-12-08 16:06:18 +00:00
5a6d9a2575 v5.1.0
Some checks failed
Default (tags) / security (push) Failing after 16s
Default (tags) / test (push) Failing after 17s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-12-08 15:09:16 +00:00
7a7ee041a3 feat(watchers): Improve write stabilization and ignore temporary editor files 2025-12-08 15:09:16 +00:00
0dc4eaba52 v5.0.0
Some checks failed
Default (tags) / security (push) Failing after 17s
Default (tags) / test (push) Failing after 17s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-30 18:01:01 +00:00
31ab561859 BREAKING CHANGE(@push.rocks/smartwatch): Rename package and update branding/docs: switch from @push.rocks/smartchok to @push.rocks/smartwatch, update repository/homepage/bugs URLs and author, and refresh README examples and install instructions. 2025-11-30 18:01:01 +00:00
11 changed files with 460 additions and 314 deletions

View File

@@ -1,5 +1,56 @@
# Changelog # Changelog
## 2025-12-08 - 6.1.1 - fix(watchers/watcher.node)
Improve Node watcher robustness: inode tracking, ENOSPC detection, enhanced health checks and temp-file handling
- Track directory inodes (watchedInodes) and restart watchers if inode changes are detected (addresses stale watchers when directories are replaced).
- Health check now validates inode stability and explicitly detects ENOSPC (inotify max_user_watches) errors, emitting errors and logging a recommended fix command.
- Detect ENOSPC in FSWatcher error events and log guidance to increase inotify limits.
- Clear inode tracking state on watcher stop to avoid stale state across restarts.
- Improve temporary file handling and logging to avoid dropping events for atomic writes (only skip pure temp files and log skipped temp events).
- Documentation (readme.hints.md) updated with robustness notes, known fs.watch limitations, and example logs.
## 2025-12-08 - 6.1.0 - feat(watcher.node)
Add automatic restart, periodic health checks, and safe event emission to Node watcher; improve logging and stat handling
- NodeWatcher: introduced safeEmit() to isolate subscriber errors and prevent watcher crashes
- Auto-restart on failure with exponential backoff (1s → 30s) and up to 3 retry attempts per watched base path
- Periodic health checks (every 30s) to detect missing watched paths and trigger automatic restarts
- Handle unexpected FSWatcher 'close' events and restart watchers when they close silently
- Verbose lifecycle logging with `[smartwatch]` prefix for start/stop/health/restart events
- Clear restart tracking and stop health checks on watcher.stop() to ensure clean shutdown
- Improved statSafe() to normalize followSymlinks logic and log non-ENO errors as warnings
- Updated readme.hints.md documenting the new robustness features (v6.1.0+)
## 2025-12-08 - 6.0.0 - BREAKING CHANGE(watchers)
Replace polling-based write stabilization with debounce-based event coalescing and simplify watcher options
- Remove polling-based WriteStabilizer (ts/utils/write-stabilizer.ts) and related waitForWriteFinish logic
- Introduce debounce-based coalescing (debounceMs) for Node and Deno watchers (ts/watchers/watcher.node.ts, ts/watchers/watcher.deno.ts)
- Change IWatcherOptions: remove stabilityThreshold/pollInterval/maxWaitTime and add debounceMs
- Default watcher options updated to use debounceMs = 100
- Node/Deno watchers now skip editor temp files earlier, debounce duplicate events, and emit events directly (no size polling)
- Updated default watcher creation in Smartwatch to pass debounceMs
- Update package.json build script to run 'tsbuild tsfolders'
## 2025-12-08 - 5.1.0 - feat(watchers)
Improve write stabilization and ignore temporary editor files
- Add maxWaitTime option (ms) to IWatcherOptions and WriteStabilizer to cap how long stabilization will wait (default: 1000ms).
- WriteStabilizer: reduce default stabilityThreshold from 300ms to 100ms and track write start time to enforce maxWaitTime and avoid indefinite polling.
- Node and Deno watchers: detect and ignore common temporary/editor files (e.g. *.tmp.*, *.swp, *.swx, trailing ~, .#*) to prevent spurious events from atomic saves.
- Node watcher: treat rename (atomic-save) events as already-complete files and emit add/change immediately without stabilization.
- Deno watcher: use the configured maxWaitTime and polling-based stabilization for modify events to ensure consistent behavior across runtimes.
## 2025-11-30 - 5.0.0 - BREAKING CHANGE(@push.rocks/smartwatch)
Rename package and update branding/docs: switch from @push.rocks/smartchok to @push.rocks/smartwatch, update repository/homepage/bugs URLs and author, and refresh README examples and install instructions.
- Package name changed from @push.rocks/smartchok to @push.rocks/smartwatch in package.json
- Repository, homepage and issue URLs updated to point to the new smartwatch repository
- Author changed to Task Venture Capital GmbH in package metadata
- README updated: install commands, import examples and references now use @push.rocks/smartwatch
- Documentation text and branding updated throughout README (project name, internal references)
## 2025-11-30 - 4.0.1 - fix(readme) ## 2025-11-30 - 4.0.1 - fix(readme)
Update README: refine description and clarify trademark/legal information Update README: refine description and clarify trademark/legal information

View File

@@ -1,6 +1,6 @@
{ {
"name": "@push.rocks/smartchok", "name": "@push.rocks/smartwatch",
"version": "4.0.1", "version": "6.1.1",
"private": false, "private": false,
"description": "A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.", "description": "A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.",
"main": "dist_ts/index.js", "main": "dist_ts/index.js",
@@ -8,19 +8,19 @@
"scripts": { "scripts": {
"test": "(npm run prepareTest && tstest test/)", "test": "(npm run prepareTest && tstest test/)",
"prepareTest": "(rm -f ./test/assets/hi.txt)", "prepareTest": "(rm -f ./test/assets/hi.txt)",
"build": "tsbuild", "build": "tsbuild tsfolders",
"buildDocs": "tsdoc" "buildDocs": "tsdoc"
}, },
"repository": { "repository": {
"type": "git", "type": "git",
"url": "https://code.foss.global/push.rocks/smartchok.git" "url": "https://code.foss.global/push.rocks/smartwatch.git"
}, },
"author": "Lossless GmbH", "author": "Task Venture Capital GmbH",
"license": "MIT", "license": "MIT",
"bugs": { "bugs": {
"url": "https://gitlab.com/push.rocks/smartchok/issues" "url": "https://gitlab.com/push.rocks/smartwatch/issues"
}, },
"homepage": "https://code.foss.global/push.rocks/smartchok", "homepage": "https://code.foss.global/push.rocks/smartwatch",
"engines": { "engines": {
"node": ">=20.0.0" "node": ">=20.0.0"
}, },

View File

@@ -71,6 +71,57 @@ The `WriteStabilizer` class replaces chokidar's built-in write stabilization:
- **Deno**: Works on all versions with `Deno.watchFs()` - **Deno**: Works on all versions with `Deno.watchFs()`
- **Bun**: Uses Node.js compatibility layer - **Bun**: Uses Node.js compatibility layer
### Robustness Features (v6.1.0+)
The Node.js watcher includes automatic recovery mechanisms based on learnings from [chokidar](https://github.com/paulmillr/chokidar) and known [fs.watch issues](https://github.com/nodejs/node/issues/47058):
**Auto-restart on failure:**
- Watchers automatically restart when errors occur
- Exponential backoff (1s → 30s max)
- Maximum 3 retry attempts before giving up
**Inode tracking (critical for long-running watchers):**
- `fs.watch()` watches the **inode**, not the path!
- When directories are replaced (git checkout, atomic saves), the inode changes
- Health check detects inode changes and restarts the watcher
- This is the most common cause of "watcher stops working after some time"
**Health check monitoring:**
- 30-second periodic health checks
- Detects when watched paths disappear
- Detects inode changes (directory replacement)
- Detects ENOSPC errors (inotify limit exceeded)
**ENOSPC detection (Linux inotify limit):**
- Detects when `/proc/sys/fs/inotify/max_user_watches` is exceeded
- Logs fix command: `echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p`
**Error isolation:**
- Subscriber errors don't crash the watcher
- All events emitted via `safeEmit()` with try-catch
**Verbose logging:**
- All lifecycle events logged with `[smartwatch]` prefix
- Helps debug watcher issues in production
Example log output:
```
[smartwatch] Starting watcher for 1 base path(s)...
[smartwatch] Started watching: ./test/assets/
[smartwatch] Starting health check (every 30s)
[smartwatch] Watcher started with 1 active watcher(s)
[smartwatch] Health check: 1 watchers active
[smartwatch] Inode changed for ./src: 12345 -> 67890
[smartwatch] fs.watch watches inode, not path - restarting watcher
```
### Known fs.watch Limitations
1. **Watches inode, not path** - If a directory is replaced, watcher goes stale
2. **inotify limits on Linux** - Default `max_user_watches` (8192) may be too low
3. **No events for some atomic writes** - Some editors' save patterns may not trigger events
4. **Platform differences** - Linux uses inotify, macOS uses FSEvents/kqueue
### Testing ### Testing
```bash ```bash

View File

@@ -1,4 +1,4 @@
# @push.rocks/smartchok # @push.rocks/smartwatch
A lightweight, cross-runtime file watcher with glob pattern support for **Node.js**, **Deno**, and **Bun**. Zero heavyweight dependencies — just native file watching APIs for maximum performance. 🚀 A lightweight, cross-runtime file watcher with glob pattern support for **Node.js**, **Deno**, and **Bun**. Zero heavyweight dependencies — just native file watching APIs for maximum performance. 🚀
@@ -9,9 +9,9 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
## Install ## Install
```sh ```sh
npm install @push.rocks/smartchok npm install @push.rocks/smartwatch
# or # or
pnpm add @push.rocks/smartchok pnpm add @push.rocks/smartwatch
``` ```
## Features ## Features
@@ -30,7 +30,7 @@ pnpm add @push.rocks/smartchok
### Basic Setup ### Basic Setup
```typescript ```typescript
import { Smartwatch } from '@push.rocks/smartchok'; import { Smartwatch } from '@push.rocks/smartwatch';
// Create a watcher with glob patterns // Create a watcher with glob patterns
const watcher = new Smartwatch([ const watcher = new Smartwatch([
@@ -110,7 +110,7 @@ await watcher.stop();
### Complete Example ### Complete Example
```typescript ```typescript
import { Smartwatch } from '@push.rocks/smartchok'; import { Smartwatch } from '@push.rocks/smartwatch';
async function watchProject() { async function watchProject() {
// Initialize with patterns // Initialize with patterns
@@ -155,7 +155,7 @@ watchProject();
## How It Works ## How It Works
smartchok uses native file watching APIs for each runtime: smartwatch uses native file watching APIs for each runtime:
| Runtime | API Used | | Runtime | API Used |
|-----------------|----------------------------------| |-----------------|----------------------------------|
@@ -165,7 +165,7 @@ smartchok uses native file watching APIs for each runtime:
### Under the Hood ### Under the Hood
Native file watching APIs don't support glob patterns directly, so smartchok handles pattern matching internally: Native file watching APIs don't support glob patterns directly, so smartwatch handles pattern matching internally:
1. **Base path extraction** — Extracts the static portion from each glob pattern (e.g., `./src/` from `./src/**/*.ts`) 1. **Base path extraction** — Extracts the static portion from each glob pattern (e.g., `./src/` from `./src/**/*.ts`)
2. **Efficient watching** — Native watchers monitor only the base directories 2. **Efficient watching** — Native watchers monitor only the base directories
@@ -174,7 +174,7 @@ Native file watching APIs don't support glob patterns directly, so smartchok han
### Write Stabilization ### Write Stabilization
smartchok includes built-in write stabilization (similar to chokidar's `awaitWriteFinish`). When a file is being written, events are held until the file size stabilizes, preventing multiple events for a single write operation. smartwatch includes built-in write stabilization (similar to chokidar's `awaitWriteFinish`). When a file is being written, events are held until the file size stabilizes, preventing multiple events for a single write operation.
Default settings: Default settings:
- **Stability threshold**: 300ms - **Stability threshold**: 300ms
@@ -226,9 +226,9 @@ type TFsEvent = 'add' | 'addDir' | 'change' | 'error' | 'unlink' | 'unlinkDir' |
type TSmartwatchStatus = 'idle' | 'starting' | 'watching'; type TSmartwatchStatus = 'idle' | 'starting' | 'watching';
``` ```
## Why smartchok? ## Why smartwatch?
| Feature | smartchok | chokidar | | Feature | smartwatch | chokidar |
|-------------------------|----------------------|--------------------| |-------------------------|----------------------|--------------------|
| Native API | ✅ Direct `fs.watch` | ❌ FSEvents bindings | | Native API | ✅ Direct `fs.watch` | ❌ FSEvents bindings |
| Cross-runtime | ✅ Node, Deno, Bun | ❌ Node only | | Cross-runtime | ✅ Node, Deno, Bun | ❌ Node only |
@@ -237,7 +237,7 @@ type TSmartwatchStatus = 'idle' | 'starting' | 'watching';
| Glob support | ✅ picomatch | ✅ anymatch | | Glob support | ✅ picomatch | ✅ anymatch |
| Bundle size | ~15KB | ~200KB+ | | Bundle size | ~15KB | ~200KB+ |
If you need a lightweight file watcher without native compilation headaches, smartchok has you covered. If you need a lightweight file watcher without native compilation headaches, smartwatch has you covered.
## License and Legal Information ## License and Legal Information

View File

@@ -2,7 +2,7 @@
* autocreated commitinfo by @push.rocks/commitinfo * autocreated commitinfo by @push.rocks/commitinfo
*/ */
export const commitinfo = { export const commitinfo = {
name: '@push.rocks/smartchok', name: '@push.rocks/smartwatch',
version: '4.0.1', version: '6.1.1',
description: 'A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.' description: 'A cross-runtime file watcher with glob pattern support for Node.js, Deno, and Bun.'
} }

View File

@@ -132,8 +132,7 @@ export class Smartwatch {
basePaths: watchPaths, basePaths: watchPaths,
depth: 4, depth: 4,
followSymlinks: false, followSymlinks: false,
stabilityThreshold: 300, debounceMs: 100
pollInterval: 100
}); });
// Subscribe to watcher events and dispatch to appropriate subjects // Subscribe to watcher events and dispatch to appropriate subjects

View File

@@ -1,97 +0,0 @@
import * as fs from 'fs';
interface IPendingWrite {
lastSize: number;
lastChange: number;
timeoutId: ReturnType<typeof setTimeout>;
resolve: (stats: fs.Stats) => void;
reject: (error: Error) => void;
}
/**
* Implements awaitWriteFinish functionality by polling file size until stable.
* This replaces chokidar's built-in write stabilization.
*/
export class WriteStabilizer {
private pendingWrites = new Map<string, IPendingWrite>();
constructor(
private stabilityThreshold: number = 300,
private pollInterval: number = 100
) {}
/**
* Wait for a file write to complete by polling until size is stable
*/
async waitForWriteFinish(filePath: string): Promise<fs.Stats> {
// Cancel any existing pending check for this file
this.cancel(filePath);
return new Promise((resolve, reject) => {
const poll = async () => {
try {
const stats = await fs.promises.stat(filePath);
const pending = this.pendingWrites.get(filePath);
if (!pending) {
// Was cancelled
return;
}
const now = Date.now();
if (stats.size !== pending.lastSize) {
// Size changed - file is still being written, reset timer
pending.lastSize = stats.size;
pending.lastChange = now;
pending.timeoutId = setTimeout(poll, this.pollInterval);
} else if (now - pending.lastChange >= this.stabilityThreshold) {
// Size has been stable for the threshold duration
this.pendingWrites.delete(filePath);
resolve(stats);
} else {
// Size is the same but not yet past threshold
pending.timeoutId = setTimeout(poll, this.pollInterval);
}
} catch (error: any) {
this.pendingWrites.delete(filePath);
if (error.code === 'ENOENT') {
// File was deleted during polling
reject(new Error(`File was deleted: ${filePath}`));
} else {
reject(error);
}
}
};
this.pendingWrites.set(filePath, {
lastSize: -1,
lastChange: Date.now(),
timeoutId: setTimeout(poll, this.pollInterval),
resolve,
reject
});
});
}
/**
* Cancel any pending write stabilization for a file
*/
cancel(filePath: string): void {
const pending = this.pendingWrites.get(filePath);
if (pending) {
clearTimeout(pending.timeoutId);
this.pendingWrites.delete(filePath);
}
}
/**
* Cancel all pending write stabilizations
*/
cancelAll(): void {
for (const [filePath, pending] of this.pendingWrites) {
clearTimeout(pending.timeoutId);
}
this.pendingWrites.clear();
}
}

View File

@@ -28,6 +28,5 @@ export const defaultWatcherOptions: IWatcherOptions = {
basePaths: [], basePaths: [],
depth: 4, depth: 4,
followSymlinks: false, followSymlinks: false,
stabilityThreshold: 300, debounceMs: 100
pollInterval: 100
}; };

View File

@@ -26,10 +26,8 @@ export interface IWatcherOptions {
depth: number; depth: number;
/** Whether to follow symbolic links */ /** Whether to follow symbolic links */
followSymlinks: boolean; followSymlinks: boolean;
/** Stability threshold for write detection (ms) */ /** Debounce time in ms - events for the same file within this window are coalesced */
stabilityThreshold: number; debounceMs: number;
/** Poll interval for write detection (ms) */
pollInterval: number;
} }
/** /**

View File

@@ -65,15 +65,27 @@ export class DenoWatcher implements IWatcher {
private watcher: ReturnType<typeof Deno.watchFs> | null = null; private watcher: ReturnType<typeof Deno.watchFs> | null = null;
private watchedFiles: Set<string> = new Set(); private watchedFiles: Set<string> = new Set();
private _isWatching = false; private _isWatching = false;
private abortController: AbortController | null = null;
private recentEvents: Map<string, number> = new Map(); // Debounce: pending emits per file path
private throttleMs = 50; private pendingEmits: Map<string, ReturnType<typeof setTimeout>> = new Map();
private pendingWrites: Map<string, ReturnType<typeof setTimeout>> = new Map();
public readonly events$ = new smartrx.rxjs.Subject<IWatchEvent>(); public readonly events$ = new smartrx.rxjs.Subject<IWatchEvent>();
constructor(private options: IWatcherOptions) {} constructor(private options: IWatcherOptions) {}
/**
* Check if a file is a temporary file created by editors
*/
private isTemporaryFile(filePath: string): boolean {
const basename = filePath.split('/').pop() || '';
// Editor temp files: *.tmp.*, *.swp, *.swx, *~, .#*
if (basename.includes('.tmp.')) return true;
if (basename.endsWith('.swp') || basename.endsWith('.swx')) return true;
if (basename.endsWith('~')) return true;
if (basename.startsWith('.#')) return true;
return false;
}
get isWatching(): boolean { get isWatching(): boolean {
return this._isWatching; return this._isWatching;
} }
@@ -84,8 +96,6 @@ export class DenoWatcher implements IWatcher {
} }
try { try {
this.abortController = new AbortController();
// Start watching all base paths // Start watching all base paths
this.watcher = Deno.watchFs(this.options.basePaths, { recursive: true }); this.watcher = Deno.watchFs(this.options.basePaths, { recursive: true });
this._isWatching = true; this._isWatching = true;
@@ -109,11 +119,11 @@ export class DenoWatcher implements IWatcher {
async stop(): Promise<void> { async stop(): Promise<void> {
this._isWatching = false; this._isWatching = false;
// Cancel all pending write stabilizations // Cancel all pending debounced emits
for (const timeout of this.pendingWrites.values()) { for (const timeout of this.pendingEmits.values()) {
clearTimeout(timeout); clearTimeout(timeout);
} }
this.pendingWrites.clear(); this.pendingEmits.clear();
// Close the watcher // Close the watcher
if (this.watcher) { if (this.watcher) {
@@ -122,7 +132,6 @@ export class DenoWatcher implements IWatcher {
} }
this.watchedFiles.clear(); this.watchedFiles.clear();
this.recentEvents.clear();
} }
/** /**
@@ -140,7 +149,7 @@ export class DenoWatcher implements IWatcher {
} }
for (const filePath of event.paths) { for (const filePath of event.paths) {
await this.handleDenoEvent(event.kind, filePath); this.handleDenoEvent(event.kind, filePath);
} }
} }
} catch (error: any) { } catch (error: any) {
@@ -151,46 +160,56 @@ export class DenoWatcher implements IWatcher {
} }
/** /**
* Handle a Deno file system event * Handle a Deno file system event - debounce and normalize
*/ */
private async handleDenoEvent( private handleDenoEvent(
kind: 'create' | 'modify' | 'remove' | 'access' | 'any' | 'other', kind: 'create' | 'modify' | 'remove' | 'access' | 'any' | 'other',
filePath: string filePath: string
): Promise<void> { ): void {
// Ignore 'access' events (just reading the file) // Ignore 'access' events (just reading the file)
if (kind === 'access') { if (kind === 'access') {
return; return;
} }
// Throttle duplicate events // Skip temporary files created by editors (atomic saves)
if (!this.shouldEmit(filePath, kind)) { if (this.isTemporaryFile(filePath)) {
return; return;
} }
// Debounce: cancel any pending emit for this file
const existing = this.pendingEmits.get(filePath);
if (existing) {
clearTimeout(existing);
}
// Schedule debounced emit
const timeout = setTimeout(() => {
this.pendingEmits.delete(filePath);
this.emitFileEvent(filePath, kind);
}, this.options.debounceMs);
this.pendingEmits.set(filePath, timeout);
}
/**
* Emit the actual file event after debounce
*/
private async emitFileEvent(
filePath: string,
kind: 'create' | 'modify' | 'remove' | 'access' | 'any' | 'other'
): Promise<void> {
try { try {
if (kind === 'create') { if (kind === 'create') {
const stats = await this.statSafe(filePath); const stats = await this.statSafe(filePath);
if (stats) { if (stats) {
// Wait for write to stabilize
await this.waitForWriteFinish(filePath);
const finalStats = await this.statSafe(filePath);
if (finalStats) {
this.watchedFiles.add(filePath); this.watchedFiles.add(filePath);
const eventType: TWatchEventType = finalStats.isDirectory() ? 'addDir' : 'add'; const eventType: TWatchEventType = stats.isDirectory() ? 'addDir' : 'add';
this.events$.next({ type: eventType, path: filePath, stats: finalStats }); this.events$.next({ type: eventType, path: filePath, stats });
}
} }
} else if (kind === 'modify') { } else if (kind === 'modify') {
const stats = await this.statSafe(filePath); const stats = await this.statSafe(filePath);
if (stats && !stats.isDirectory()) { if (stats && !stats.isDirectory()) {
// Wait for write to stabilize this.events$.next({ type: 'change', path: filePath, stats });
await this.waitForWriteFinish(filePath);
const finalStats = await this.statSafe(filePath);
if (finalStats) {
this.events$.next({ type: 'change', path: filePath, stats: finalStats });
}
} }
} else if (kind === 'remove') { } else if (kind === 'remove') {
const wasDirectory = this.isKnownDirectory(filePath); const wasDirectory = this.isKnownDirectory(filePath);
@@ -205,43 +224,6 @@ export class DenoWatcher implements IWatcher {
} }
} }
/**
* Wait for file write to complete (polling-based)
*/
private async waitForWriteFinish(filePath: string): Promise<void> {
return new Promise((resolve) => {
let lastSize = -1;
let lastChange = Date.now();
const poll = async () => {
try {
const stats = await this.statSafe(filePath);
if (!stats) {
resolve();
return;
}
const now = Date.now();
if (stats.size !== lastSize) {
lastSize = stats.size;
lastChange = now;
this.pendingWrites.set(filePath, setTimeout(poll, this.options.pollInterval));
} else if (now - lastChange >= this.options.stabilityThreshold) {
this.pendingWrites.delete(filePath);
resolve();
} else {
this.pendingWrites.set(filePath, setTimeout(poll, this.options.pollInterval));
}
} catch {
this.pendingWrites.delete(filePath);
resolve();
}
};
this.pendingWrites.set(filePath, setTimeout(poll, this.options.pollInterval));
});
}
/** /**
* Scan directory and emit 'add' events for existing files * Scan directory and emit 'add' events for existing files
*/ */
@@ -253,6 +235,12 @@ export class DenoWatcher implements IWatcher {
try { try {
for await (const entry of Deno.readDir(dirPath)) { for await (const entry of Deno.readDir(dirPath)) {
const fullPath = `${dirPath}/${entry.name}`; const fullPath = `${dirPath}/${entry.name}`;
// Skip temp files during initial scan too
if (this.isTemporaryFile(fullPath)) {
continue;
}
const stats = await this.statSafe(fullPath); const stats = await this.statSafe(fullPath);
if (!stats) { if (!stats) {
@@ -299,31 +287,4 @@ export class DenoWatcher implements IWatcher {
} }
return false; return false;
} }
/**
* Throttle duplicate events
*/
private shouldEmit(filePath: string, eventType: string): boolean {
const key = `${filePath}:${eventType}`;
const now = Date.now();
const lastEmit = this.recentEvents.get(key);
if (lastEmit && now - lastEmit < this.throttleMs) {
return false;
}
this.recentEvents.set(key, now);
// Clean up old entries periodically
if (this.recentEvents.size > 1000) {
const cutoff = now - this.throttleMs * 2;
for (const [k, time] of this.recentEvents) {
if (time < cutoff) {
this.recentEvents.delete(k);
}
}
}
return true;
}
} }

View File

@@ -2,7 +2,6 @@ import * as fs from 'fs';
import * as path from 'path'; import * as path from 'path';
import * as smartrx from '@push.rocks/smartrx'; import * as smartrx from '@push.rocks/smartrx';
import type { IWatcher, IWatcherOptions, IWatchEvent, TWatchEventType } from './interfaces.js'; import type { IWatcher, IWatcherOptions, IWatchEvent, TWatchEventType } from './interfaces.js';
import { WriteStabilizer } from '../utils/write-stabilizer.js';
/** /**
* Node.js/Bun file watcher using native fs.watch API * Node.js/Bun file watcher using native fs.watch API
@@ -11,17 +10,159 @@ export class NodeWatcher implements IWatcher {
private watchers: Map<string, fs.FSWatcher> = new Map(); private watchers: Map<string, fs.FSWatcher> = new Map();
private watchedFiles: Set<string> = new Set(); private watchedFiles: Set<string> = new Set();
private _isWatching = false; private _isWatching = false;
private writeStabilizer: WriteStabilizer;
private recentEvents: Map<string, number> = new Map(); // Debounce: pending emits per file path
private throttleMs = 50; private pendingEmits: Map<string, NodeJS.Timeout> = new Map();
// Restart tracking
private restartDelays: Map<string, number> = new Map();
private restartAttempts: Map<string, number> = new Map();
private healthCheckInterval: NodeJS.Timeout | null = null;
// Inode tracking - detect when directories are replaced (atomic saves, etc.)
// fs.watch watches the inode, not the path. If inode changes, we need to restart.
private watchedInodes: Map<string, bigint> = new Map();
// Configuration constants
private static readonly MAX_RETRIES = 3;
private static readonly INITIAL_RESTART_DELAY = 1000;
private static readonly MAX_RESTART_DELAY = 30000;
private static readonly HEALTH_CHECK_INTERVAL = 30000;
public readonly events$ = new smartrx.rxjs.Subject<IWatchEvent>(); public readonly events$ = new smartrx.rxjs.Subject<IWatchEvent>();
constructor(private options: IWatcherOptions) { constructor(private options: IWatcherOptions) {}
this.writeStabilizer = new WriteStabilizer(
options.stabilityThreshold, /**
options.pollInterval * Safely emit an event, catching any subscriber errors
); */
private safeEmit(event: IWatchEvent): void {
try {
this.events$.next(event);
} catch (error) {
console.error('[smartwatch] Subscriber threw error (isolated):', error);
// Don't let subscriber errors kill the watcher
}
}
/**
* Restart a watcher after an error with exponential backoff
*/
private async restartWatcher(basePath: string, error: Error): Promise<void> {
const attempts = (this.restartAttempts.get(basePath) || 0) + 1;
this.restartAttempts.set(basePath, attempts);
console.log(`[smartwatch] Watcher error for ${basePath}: ${error.message}`);
console.log(`[smartwatch] Restart attempt ${attempts}/${NodeWatcher.MAX_RETRIES}`);
if (attempts > NodeWatcher.MAX_RETRIES) {
console.error(`[smartwatch] Max retries exceeded for ${basePath}, giving up`);
this.safeEmit({
type: 'error',
path: basePath,
error: new Error(`Max restart retries (${NodeWatcher.MAX_RETRIES}) exceeded`)
});
return;
}
// Close failed watcher
const oldWatcher = this.watchers.get(basePath);
if (oldWatcher) {
try {
oldWatcher.close();
} catch {
// Ignore close errors
}
this.watchers.delete(basePath);
}
// Exponential backoff
const delay = this.restartDelays.get(basePath) || NodeWatcher.INITIAL_RESTART_DELAY;
console.log(`[smartwatch] Waiting ${delay}ms before restart...`);
await new Promise((resolve) => setTimeout(resolve, delay));
this.restartDelays.set(basePath, Math.min(delay * 2, NodeWatcher.MAX_RESTART_DELAY));
try {
await this.watchPath(basePath, 0);
console.log(`[smartwatch] Successfully restarted watcher for ${basePath}`);
this.restartDelays.set(basePath, NodeWatcher.INITIAL_RESTART_DELAY);
this.restartAttempts.set(basePath, 0);
} catch (restartError) {
console.error(`[smartwatch] Restart failed for ${basePath}:`, restartError);
this.restartWatcher(basePath, restartError as Error); // Recursive retry
}
}
/**
* Start periodic health checks to detect silent failures
* Checks for:
* 1. Path no longer exists
* 2. Inode changed (directory was replaced - fs.watch watches inode, not path!)
*/
private startHealthCheck(): void {
console.log('[smartwatch] Starting health check (every 30s)');
this.healthCheckInterval = setInterval(async () => {
console.log(`[smartwatch] Health check: ${this.watchers.size} watchers active`);
for (const [basePath] of this.watchers) {
try {
const stats = await fs.promises.stat(basePath);
const currentInode = stats.ino;
const previousInode = this.watchedInodes.get(basePath);
if (!stats) {
console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
this.safeEmit({
type: 'error',
path: basePath,
error: new Error('Watched path no longer exists')
});
this.restartWatcher(basePath, new Error('Watched path disappeared'));
} else if (previousInode !== undefined && BigInt(currentInode) !== previousInode) {
// CRITICAL: Inode changed! fs.watch is now watching a stale inode.
// This happens when the directory is replaced (atomic operations, git checkout, etc.)
console.warn(`[smartwatch] Inode changed for ${basePath}: ${previousInode} -> ${currentInode}`);
console.warn('[smartwatch] fs.watch watches inode, not path - restarting watcher');
this.restartWatcher(basePath, new Error('Inode changed - directory was replaced'));
}
} catch (error: any) {
if (error.code === 'ENOENT') {
console.error(`[smartwatch] Health check failed: ${basePath} no longer exists`);
this.restartWatcher(basePath, new Error('Watched path disappeared'));
} else if (error.code === 'ENOSPC') {
// inotify watch limit exceeded - critical system issue
console.error(`[smartwatch] ENOSPC: inotify watch limit exceeded!`);
console.error('[smartwatch] Fix: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
this.safeEmit({ type: 'error', path: basePath, error });
} else {
console.error(`[smartwatch] Health check error for ${basePath}:`, error);
}
}
}
}, NodeWatcher.HEALTH_CHECK_INTERVAL);
}
/**
* Stop health check interval
*/
private stopHealthCheck(): void {
if (this.healthCheckInterval) {
clearInterval(this.healthCheckInterval);
this.healthCheckInterval = null;
console.log('[smartwatch] Stopped health check');
}
}
/**
* Check if a file is a temporary file created by editors
*/
private isTemporaryFile(filePath: string): boolean {
const basename = path.basename(filePath);
// Editor temp files: *.tmp.*, *.swp, *.swx, *~, .#*
if (basename.includes('.tmp.')) return true;
if (basename.endsWith('.swp') || basename.endsWith('.swx')) return true;
if (basename.endsWith('~')) return true;
if (basename.startsWith('.#')) return true;
return false;
} }
get isWatching(): boolean { get isWatching(): boolean {
@@ -33,6 +174,8 @@ export class NodeWatcher implements IWatcher {
return; return;
} }
console.log(`[smartwatch] Starting watcher for ${this.options.basePaths.length} base path(s)...`);
try { try {
// Start watching each base path // Start watching each base path
for (const basePath of this.options.basePaths) { for (const basePath of this.options.basePaths) {
@@ -41,30 +184,51 @@ export class NodeWatcher implements IWatcher {
this._isWatching = true; this._isWatching = true;
// Start health check monitoring
this.startHealthCheck();
// Perform initial scan to emit 'add' events for existing files // Perform initial scan to emit 'add' events for existing files
for (const basePath of this.options.basePaths) { for (const basePath of this.options.basePaths) {
await this.scanDirectory(basePath, 0); await this.scanDirectory(basePath, 0);
} }
// Emit ready event // Emit ready event
this.events$.next({ type: 'ready', path: '' }); this.safeEmit({ type: 'ready', path: '' });
console.log(`[smartwatch] Watcher started with ${this.watchers.size} active watcher(s)`);
} catch (error: any) { } catch (error: any) {
this.events$.next({ type: 'error', path: '', error }); console.error('[smartwatch] Failed to start watcher:', error);
this.safeEmit({ type: 'error', path: '', error });
throw error; throw error;
} }
} }
async stop(): Promise<void> { async stop(): Promise<void> {
console.log('[smartwatch] Stopping watcher...');
this._isWatching = false; this._isWatching = false;
this.writeStabilizer.cancelAll();
// Stop health check monitoring
this.stopHealthCheck();
// Cancel all pending debounced emits
for (const timeout of this.pendingEmits.values()) {
clearTimeout(timeout);
}
this.pendingEmits.clear();
// Close all watchers // Close all watchers
for (const [watchPath, watcher] of this.watchers) { for (const [watchPath, watcher] of this.watchers) {
console.log(`[smartwatch] Closing watcher for: ${watchPath}`);
watcher.close(); watcher.close();
} }
this.watchers.clear(); this.watchers.clear();
this.watchedFiles.clear(); this.watchedFiles.clear();
this.recentEvents.clear();
// Clear restart tracking state
this.restartDelays.clear();
this.restartAttempts.clear();
this.watchedInodes.clear();
console.log('[smartwatch] Watcher stopped');
} }
/** /**
@@ -82,6 +246,10 @@ export class NodeWatcher implements IWatcher {
} }
if (stats.isDirectory()) { if (stats.isDirectory()) {
// Store inode for health check - fs.watch watches inode, not path!
// If inode changes (directory replaced), watcher becomes stale
this.watchedInodes.set(watchPath, BigInt(stats.ino));
// Watch the directory with recursive option (Node.js 20+ supports this on all platforms) // Watch the directory with recursive option (Node.js 20+ supports this on all platforms)
const watcher = fs.watch( const watcher = fs.watch(
watchPath, watchPath,
@@ -93,32 +261,81 @@ export class NodeWatcher implements IWatcher {
} }
); );
watcher.on('error', (error) => { watcher.on('error', (error: NodeJS.ErrnoException) => {
this.events$.next({ type: 'error', path: watchPath, error }); console.error(`[smartwatch] FSWatcher error event on ${watchPath}:`, error);
// Detect inotify watch limit exceeded - common cause of "stops working"
if (error.code === 'ENOSPC') {
console.error('[smartwatch] CRITICAL: inotify watch limit exceeded!');
console.error('[smartwatch] Fix with: echo fs.inotify.max_user_watches=524288 | sudo tee -a /etc/sysctl.conf && sudo sysctl -p');
}
this.safeEmit({ type: 'error', path: watchPath, error });
if (this._isWatching) {
this.restartWatcher(watchPath, error);
}
});
// Handle 'close' event - fs.watch can close without error
watcher.on('close', () => {
console.warn(`[smartwatch] FSWatcher closed unexpectedly for ${watchPath}`);
if (this._isWatching) {
this.restartWatcher(watchPath, new Error('Watcher closed unexpectedly'));
}
}); });
this.watchers.set(watchPath, watcher); this.watchers.set(watchPath, watcher);
console.log(`[smartwatch] Started watching: ${watchPath}`);
} }
} catch (error: any) { } catch (error: any) {
this.events$.next({ type: 'error', path: watchPath, error }); console.error(`[smartwatch] Failed to watch path ${watchPath}:`, error);
this.safeEmit({ type: 'error', path: watchPath, error });
} }
} }
/** /**
* Handle raw fs.watch events and normalize them * Handle raw fs.watch events - debounce and normalize them
*/ */
private async handleFsEvent( private handleFsEvent(
basePath: string, basePath: string,
filename: string, filename: string,
eventType: 'rename' | 'change' | string eventType: 'rename' | 'change' | string
): Promise<void> { ): void {
const fullPath = path.join(basePath, filename); const fullPath = path.join(basePath, filename);
// Throttle duplicate events // Skip temporary files - but ONLY pure temp files, not the target of atomic writes
if (!this.shouldEmit(fullPath, eventType)) { // Atomic writes: editor writes to file.tmp.xxx then renames to file
// We need to detect the final file, so only skip files that ARE temp files
// and haven't been renamed to the real file yet
if (this.isTemporaryFile(fullPath)) {
// For temp files, we still want to track if they get renamed TO a real file
// The 'rename' event fires for both source and target, so we'll catch the real file
console.log(`[smartwatch] Skipping temp file event: ${filename}`);
return; return;
} }
// Debounce: cancel any pending emit for this file
const existing = this.pendingEmits.get(fullPath);
if (existing) {
clearTimeout(existing);
}
// Schedule debounced emit
const timeout = setTimeout(() => {
this.pendingEmits.delete(fullPath);
this.emitFileEvent(fullPath, eventType);
}, this.options.debounceMs);
this.pendingEmits.set(fullPath, timeout);
}
/**
* Emit the actual file event after debounce
*/
private async emitFileEvent(
fullPath: string,
eventType: 'rename' | 'change' | string
): Promise<void> {
try { try {
const stats = await this.statSafe(fullPath); const stats = await this.statSafe(fullPath);
@@ -129,33 +346,23 @@ export class NodeWatcher implements IWatcher {
if (stats.isDirectory()) { if (stats.isDirectory()) {
if (!this.watchedFiles.has(fullPath)) { if (!this.watchedFiles.has(fullPath)) {
this.watchedFiles.add(fullPath); this.watchedFiles.add(fullPath);
this.events$.next({ type: 'addDir', path: fullPath, stats }); this.safeEmit({ type: 'addDir', path: fullPath, stats });
} }
} else { } else {
// Wait for write to stabilize before emitting
try {
const stableStats = await this.writeStabilizer.waitForWriteFinish(fullPath);
const wasWatched = this.watchedFiles.has(fullPath); const wasWatched = this.watchedFiles.has(fullPath);
this.watchedFiles.add(fullPath); this.watchedFiles.add(fullPath);
this.events$.next({ this.safeEmit({
type: wasWatched ? 'change' : 'add', type: wasWatched ? 'change' : 'add',
path: fullPath, path: fullPath,
stats: stableStats stats
}); });
} catch {
// File was deleted during stabilization
if (this.watchedFiles.has(fullPath)) {
this.watchedFiles.delete(fullPath);
this.events$.next({ type: 'unlink', path: fullPath });
}
}
} }
} else { } else {
// File doesn't exist - it was deleted // File doesn't exist - it was deleted
if (this.watchedFiles.has(fullPath)) { if (this.watchedFiles.has(fullPath)) {
const wasDir = this.isKnownDirectory(fullPath); const wasDir = this.isKnownDirectory(fullPath);
this.watchedFiles.delete(fullPath); this.watchedFiles.delete(fullPath);
this.events$.next({ this.safeEmit({
type: wasDir ? 'unlinkDir' : 'unlink', type: wasDir ? 'unlinkDir' : 'unlink',
path: fullPath path: fullPath
}); });
@@ -164,28 +371,22 @@ export class NodeWatcher implements IWatcher {
} else if (eventType === 'change') { } else if (eventType === 'change') {
// File was modified // File was modified
if (stats && !stats.isDirectory()) { if (stats && !stats.isDirectory()) {
try {
const stableStats = await this.writeStabilizer.waitForWriteFinish(fullPath);
// Check if this is a new file (not yet in watchedFiles)
const wasWatched = this.watchedFiles.has(fullPath); const wasWatched = this.watchedFiles.has(fullPath);
if (!wasWatched) { if (!wasWatched) {
// This is actually an 'add' - file wasn't being watched before // This is actually an 'add' - file wasn't being watched before
this.watchedFiles.add(fullPath); this.watchedFiles.add(fullPath);
this.events$.next({ type: 'add', path: fullPath, stats: stableStats }); this.safeEmit({ type: 'add', path: fullPath, stats });
} else { } else {
this.events$.next({ type: 'change', path: fullPath, stats: stableStats }); this.safeEmit({ type: 'change', path: fullPath, stats });
} }
} catch { } else if (!stats && this.watchedFiles.has(fullPath)) {
// File was deleted during write // File was deleted
if (this.watchedFiles.has(fullPath)) {
this.watchedFiles.delete(fullPath); this.watchedFiles.delete(fullPath);
this.events$.next({ type: 'unlink', path: fullPath }); this.safeEmit({ type: 'unlink', path: fullPath });
}
}
} }
} }
} catch (error: any) { } catch (error: any) {
this.events$.next({ type: 'error', path: fullPath, error }); this.safeEmit({ type: 'error', path: fullPath, error });
} }
} }
@@ -202,6 +403,12 @@ export class NodeWatcher implements IWatcher {
for (const entry of entries) { for (const entry of entries) {
const fullPath = path.join(dirPath, entry.name); const fullPath = path.join(dirPath, entry.name);
// Skip temp files during initial scan too
if (this.isTemporaryFile(fullPath)) {
continue;
}
const stats = await this.statSafe(fullPath); const stats = await this.statSafe(fullPath);
if (!stats) { if (!stats) {
@@ -210,16 +417,16 @@ export class NodeWatcher implements IWatcher {
if (entry.isDirectory()) { if (entry.isDirectory()) {
this.watchedFiles.add(fullPath); this.watchedFiles.add(fullPath);
this.events$.next({ type: 'addDir', path: fullPath, stats }); this.safeEmit({ type: 'addDir', path: fullPath, stats });
await this.scanDirectory(fullPath, depth + 1); await this.scanDirectory(fullPath, depth + 1);
} else if (entry.isFile()) { } else if (entry.isFile()) {
this.watchedFiles.add(fullPath); this.watchedFiles.add(fullPath);
this.events$.next({ type: 'add', path: fullPath, stats }); this.safeEmit({ type: 'add', path: fullPath, stats });
} }
} }
} catch (error: any) { } catch (error: any) {
if (error.code !== 'ENOENT' && error.code !== 'EACCES') { if (error.code !== 'ENOENT' && error.code !== 'EACCES') {
this.events$.next({ type: 'error', path: dirPath, error }); this.safeEmit({ type: 'error', path: dirPath, error });
} }
} }
} }
@@ -229,12 +436,16 @@ export class NodeWatcher implements IWatcher {
*/ */
private async statSafe(filePath: string): Promise<fs.Stats | null> { private async statSafe(filePath: string): Promise<fs.Stats | null> {
try { try {
if (this.options.followSymlinks) { return await (this.options.followSymlinks
return await fs.promises.stat(filePath); ? fs.promises.stat(filePath)
} else { : fs.promises.lstat(filePath));
return await fs.promises.lstat(filePath); } catch (error: any) {
// Only silently return null for expected "file doesn't exist" errors
if (error.code === 'ENOENT' || error.code === 'ENOTDIR') {
return null;
} }
} catch { // Log other errors (permission, I/O) but still return null
console.warn(`[smartwatch] statSafe warning for ${filePath}: ${error.code} - ${error.message}`);
return null; return null;
} }
} }
@@ -251,31 +462,4 @@ export class NodeWatcher implements IWatcher {
} }
return false; return false;
} }
/**
* Throttle duplicate events
*/
private shouldEmit(filePath: string, eventType: string): boolean {
const key = `${filePath}:${eventType}`;
const now = Date.now();
const lastEmit = this.recentEvents.get(key);
if (lastEmit && now - lastEmit < this.throttleMs) {
return false;
}
this.recentEvents.set(key, now);
// Clean up old entries periodically
if (this.recentEvents.size > 1000) {
const cutoff = now - this.throttleMs * 2;
for (const [k, time] of this.recentEvents) {
if (time < cutoff) {
this.recentEvents.delete(k);
}
}
}
return true;
}
} }