smartfs/readme.hints.md

SmartFS Architecture Hints

Overview

SmartFS is a modern, pluggable filesystem module built with TypeScript. It provides a fluent API for filesystem operations with support for multiple storage backends, transactions, streaming, and file watching.

Core Design Principles

1. Fluent API with Action-Last Pattern

The API uses a builder pattern where configuration methods return this for chaining, and action methods return Promise for execution:

await fs.file('/path')
  .encoding('utf8')    // configuration
  .atomic()            // configuration
  .write('content');   // action (returns Promise)

Reasoning:

• Configuration is explicit and discoverable
• Action methods clearly indicate execution points
• Natural reading order: "configure what you want, then do it"
• Type-safe chaining prevents invalid operations

2. Provider Architecture

All filesystem operations go through a provider interface (ISmartFsProvider). This allows:

• Pluggable backends: Node.js fs, memory, S3, etc.
• Testing: Use memory provider for fast, isolated tests
• Abstraction: Hide platform-specific details
• Extensibility: Easy to add new storage backends

Provider Responsibilities:

• Implement all filesystem operations
• Handle path normalization
• Provide capability flags
• Implement transactions (or fall back to sequential)

3. Async-Only Design

No synchronous operations are exposed. All methods return Promises.

Reasoning:

• Modern async/await patterns
• Better performance (non-blocking)
• Consistent API surface
• Simplifies implementation

4. Web Streams API

Uses Web Streams (ReadableStream, WritableStream) instead of Node.js streams.

Reasoning:

• Standard API across platforms (Node.js, browser, Deno)
• Better composability with .pipeTo()
• Backpressure handling built-in
• Future-proof (web standard)

5. Transaction System

Transactions provide atomic multi-file operations with automatic rollback.

Implementation:

1. prepareTransaction() - Create backups of existing files
2. executeTransaction() - Execute all operations
3. rollbackTransaction() - Restore from backups if any operation fails

Trade-offs:

• Not truly atomic at OS level (no fsync barriers)
• Best-effort rollback (can fail if disk full, etc.)
• Sufficient for most use cases

6. File Watching

Event-based file watching with debouncing and filtering.

Features:

• Recursive watching
• Pattern filtering (glob, RegExp, function)
• Debouncing to reduce event spam
• Multiple event handlers per watcher

Implementation Notes:

• Node provider uses fs.watch
• Memory provider triggers events synchronously
• Events include stats when available

Directory Structure

ts/
├── classes/
│   ├── smartfs.ts              # Main entry point
│   ├── smartfs.file.ts         # File builder
│   ├── smartfs.directory.ts    # Directory builder
│   ├── smartfs.transaction.ts  # Transaction builder
│   └── smartfs.watcher.ts      # Watcher builder
├── interfaces/
│   ├── mod.provider.ts         # Provider interface
│   └── mod.types.ts            # Type definitions
├── providers/
│   ├── smartfs.provider.node.ts    # Node.js implementation
│   └── smartfs.provider.memory.ts  # Memory implementation
└── index.ts                    # Public exports

Key Architectural Decisions

Why Action-Last?

Considered alternatives:

• Action-first: fs.file('/path').read().asText() - Less intuitive
• Mixed: fs.read('/path', { encoding: 'utf8' }) - Less fluent

Chosen: Action-last pattern

• Clear execution point
• Natural configuration flow
• Better IDE autocomplete

Why Separate Builders?

Each builder (SmartFsFile, SmartFsDirectory, etc.) is a separate class.

Benefits:

• Type safety (can't call .list() on a file)
• Clear separation of concerns
• Better code organization
• Easier to extend

Transaction Implementation

Design choice: Prepare → Execute → Rollback pattern

Alternative considered: Copy-on-write filesystem

• Would require provider-specific implementations
• More complex
• Not worth the complexity for most use cases

Chosen approach:

1. Read existing files before operations
2. Store backup data in operation objects
3. Rollback by restoring from backups

Limitations:

• Not truly atomic (no cross-file locks)
• Rollback can fail (rare)
• Memory overhead for large files

Future improvements:

• Providers could override with native transactions
• Add transaction isolation levels
• Support for distributed transactions

Path Handling

All paths are normalized by the provider.

Reasoning:

• Providers know their path conventions
• Allows Windows vs. Unix path handling
• S3 provider can handle virtual paths
• Memory provider uses consistent format

Error Handling

Errors bubble up from providers.

Design:

• No custom error wrapping
• Provider errors are descriptive
• Use standard Node.js error codes (ENOENT, etc.)

Reasoning:

• Simpler implementation
• Familiar error messages
• Less abstraction overhead

Performance Considerations

Streaming

Use streams for files > 1MB to avoid loading entire file into memory.

Chunk size defaults:

• Read: 64KB (configurable via .chunkSize())
• Write: 16KB (Node.js default)

Memory Provider

All data stored in Map<string, IMemoryEntry>.

Trade-offs:

• Fast (no I/O)
• Limited by available memory
• No persistence
• Perfect for testing

Node Provider

Direct use of Node.js fs/promises API.

Optimizations:

• Atomic writes use temp files + rename (atomic at OS level)
• Move operations try rename first, fallback to copy+delete
• Stat caching not implemented (can add if needed)

Testing Strategy

Test Organization

• test/test.memory.provider.ts - Memory provider tests
• test/test.node.provider.ts - Node.js provider tests
• test/test.ts - Main test entry

Testing Approach

1. Memory provider tests - Fast, no I/O, comprehensive
2. Node provider tests - Real filesystem, integration
3. Both test suites share similar test cases

Reasoning:

• Memory tests are fast and isolated
• Node tests verify real filesystem behavior
• Easy to add new provider tests

Future Enhancements

Potential Features

1. S3 Provider - Cloud storage backend
2. FTP Provider - Remote filesystem access
3. Virtual Provider - Union of multiple providers
4. Caching Layer - In-memory cache for frequently accessed files
5. Compression - Transparent compression/decompression
6. Encryption - Transparent encryption at rest
7. Versioning - Automatic file versioning

API Extensions

1. Shortcut methods - fs.read('/path') as alternative to fs.file('/path').read()
2. Batch operations - fs.batch().file(...).file(...).execute()
3. Query API - SQL-like queries for file listings
4. Hooks - Before/after operation hooks

Dependencies

Core Dependencies

• @push.rocks/smartpath - Path utilities
• @types/node - TypeScript types for Node.js

Why Minimal Dependencies?

Philosophy:

• Keep the core light
• Avoid dependency hell
• Easier to maintain
• Faster installation

Trade-off:

• More code to maintain
• Can't leverage external libraries

Decision:

• Worth it for a foundational library
• Providers can have their own dependencies

Compatibility

Node.js Versions

Requires Node.js 18+ for:

• Native Web Streams API
• fs.rm() (replaces deprecated fs.rmdir())
• fs/promises API

Browser Compatibility

Core architecture supports browser, but:

• No browser provider implemented yet
• Would need IndexedDB or similar backend
• Stream handling already compatible

TypeScript

Uses ES2022 target, NodeNext modules.

Reasoning:

• Modern JavaScript features
• ESM-first approach
• Better tree-shaking
• Future-proof

Common Patterns

Provider Implementation Checklist

When implementing a new provider:

1. ✅ Implement ISmartFsProvider interface
2. ✅ Set capability flags correctly
3. ✅ Normalize paths in constructor/methods
4. ✅ Handle errors with descriptive messages
5. ✅ Implement transaction support (or fallback)
6. ✅ Add comprehensive tests
7. ✅ Document provider-specific limitations

Builder Pattern Example

class SmartFsFile {
  private options = {};

  // Configuration (returns this)
  encoding(enc: string): this {
    this.options.encoding = enc;
    return this;
  }

  // Action (returns Promise)
  async read(): Promise<string | Buffer> {
    return this.provider.readFile(this.path, this.options);
  }
}

Known Limitations

Transaction Atomicity

• Not truly atomic across files
• Rollback can fail in edge cases
• No distributed transaction support

Mitigation:

• Document limitations clearly
• Best-effort rollback is sufficient for most cases
• Providers can override with native transactions

File Watching

• Node.js fs.watch has platform-specific behavior
• May miss rapid changes
• No guarantee of event order

Mitigation:

• Debouncing helps with rapid changes
• Document platform differences
• Memory provider has predictable behavior (testing)

Path Handling

• No cross-provider path compatibility
• Provider-specific path formats

Mitigation:

• Document path format per provider
• Use provider's normalizePath() method
• Consider adding path conversion utilities

Maintenance Notes

When to Update

• Breaking changes: Avoid unless absolutely necessary
• New features: Add as fluent methods or new builders
• Bug fixes: Prioritize data integrity
• Performance: Profile before optimizing

Code Style

• Use TypeScript strict mode
• Prefer composition over inheritance
• Keep classes focused (SRP)
• Document public APIs with JSDoc
• Use meaningful variable names

Testing Requirements

All changes must:

• Pass existing tests
• Add new tests for new features
• Maintain >90% code coverage
• Test both memory and Node providers

Resources

• Web Streams API
• Node.js fs/promises
• Builder Pattern
• Provider Pattern