smartfs/readme.hints.md

# SmartFS Architecture Hints

## Overview

SmartFS is a modern, pluggable filesystem module built with TypeScript. It provides a fluent API for filesystem operations with support for multiple storage backends, transactions, streaming, and file watching.

## Core Design Principles

### 1. Fluent API with Action-Last Pattern

The API uses a **builder pattern** where configuration methods return `this` for chaining, and action methods return `Promise` for execution:

```typescript
await fs.file('/path')
  .encoding('utf8')    // configuration
  .atomic()            // configuration
  .write('content');   // action (returns Promise)
```

**Reasoning:**
- Configuration is explicit and discoverable
- Action methods clearly indicate execution points
- Natural reading order: "configure what you want, then do it"
- Type-safe chaining prevents invalid operations

### 2. Provider Architecture

All filesystem operations go through a provider interface (`ISmartFsProvider`). This allows:

- **Pluggable backends**: Node.js fs, memory, S3, etc.
- **Testing**: Use memory provider for fast, isolated tests
- **Abstraction**: Hide platform-specific details
- **Extensibility**: Easy to add new storage backends

**Provider Responsibilities:**
- Implement all filesystem operations
- Handle path normalization
- Provide capability flags
- Implement transactions (or fall back to sequential)

### 3. Async-Only Design

No synchronous operations are exposed. All methods return Promises.

**Reasoning:**
- Modern async/await patterns
- Better performance (non-blocking)
- Consistent API surface
- Simplifies implementation

### 4. Web Streams API

Uses Web Streams (`ReadableStream`, `WritableStream`) instead of Node.js streams.

**Reasoning:**
- Standard API across platforms (Node.js, browser, Deno)
- Better composability with `.pipeTo()`
- Backpressure handling built-in
- Future-proof (web standard)

### 5. Transaction System

Transactions provide atomic multi-file operations with automatic rollback.

**Implementation:**
1. `prepareTransaction()` - Create backups of existing files
2. `executeTransaction()` - Execute all operations
3. `rollbackTransaction()` - Restore from backups if any operation fails

**Trade-offs:**
- Not truly atomic at OS level (no fsync barriers)
- Best-effort rollback (can fail if disk full, etc.)
- Sufficient for most use cases

### 6. File Watching

Event-based file watching with debouncing and filtering.

**Features:**
- Recursive watching
- Pattern filtering (glob, RegExp, function)
- Debouncing to reduce event spam
- Multiple event handlers per watcher

**Implementation Notes:**
- Node provider uses `fs.watch`
- Memory provider triggers events synchronously
- Events include stats when available

## Directory Structure

```
ts/
├── classes/
│   ├── smartfs.ts              # Main entry point
│   ├── smartfs.file.ts         # File builder
│   ├── smartfs.directory.ts    # Directory builder
│   ├── smartfs.transaction.ts  # Transaction builder
│   └── smartfs.watcher.ts      # Watcher builder
├── interfaces/
│   ├── mod.provider.ts         # Provider interface
│   └── mod.types.ts            # Type definitions
├── providers/
│   ├── smartfs.provider.node.ts    # Node.js implementation
│   └── smartfs.provider.memory.ts  # Memory implementation
└── index.ts                    # Public exports
```

## Key Architectural Decisions

### Why Action-Last?

**Considered alternatives:**
- Action-first: `fs.file('/path').read().asText()` - Less intuitive
- Mixed: `fs.read('/path', { encoding: 'utf8' })` - Less fluent

**Chosen:** Action-last pattern
- Clear execution point
- Natural configuration flow
- Better IDE autocomplete

### Why Separate Builders?

Each builder (`SmartFsFile`, `SmartFsDirectory`, etc.) is a separate class.

**Benefits:**
- Type safety (can't call `.list()` on a file)
- Clear separation of concerns
- Better code organization
- Easier to extend

### Transaction Implementation

**Design choice:** Prepare → Execute → Rollback pattern

**Alternative considered:** Copy-on-write filesystem
- Would require provider-specific implementations
- More complex
- Not worth the complexity for most use cases

**Chosen approach:**
1. Read existing files before operations
2. Store backup data in operation objects
3. Rollback by restoring from backups

**Limitations:**
- Not truly atomic (no cross-file locks)
- Rollback can fail (rare)
- Memory overhead for large files

**Future improvements:**
- Providers could override with native transactions
- Add transaction isolation levels
- Support for distributed transactions

### Path Handling

All paths are normalized by the provider.

**Reasoning:**
- Providers know their path conventions
- Allows Windows vs. Unix path handling
- S3 provider can handle virtual paths
- Memory provider uses consistent format

### Error Handling

Errors bubble up from providers.

**Design:**
- No custom error wrapping
- Provider errors are descriptive
- Use standard Node.js error codes (ENOENT, etc.)

**Reasoning:**
- Simpler implementation
- Familiar error messages
- Less abstraction overhead

## Performance Considerations

### Streaming

Use streams for files > 1MB to avoid loading entire file into memory.

**Chunk size defaults:**
- Read: 64KB (configurable via `.chunkSize()`)
- Write: 16KB (Node.js default)

### Memory Provider

All data stored in Map<string, IMemoryEntry>.

**Trade-offs:**
- Fast (no I/O)
- Limited by available memory
- No persistence
- Perfect for testing

### Node Provider

Direct use of Node.js `fs/promises` API.

**Optimizations:**
- Atomic writes use temp files + rename (atomic at OS level)
- Move operations try rename first, fallback to copy+delete
- Stat caching not implemented (can add if needed)

## Testing Strategy

### Test Organization

- `test/test.memory.provider.ts` - Memory provider tests
- `test/test.node.provider.ts` - Node.js provider tests
- `test/test.ts` - Main test entry

### Testing Approach

1. **Memory provider tests** - Fast, no I/O, comprehensive
2. **Node provider tests** - Real filesystem, integration
3. **Both test suites** share similar test cases

**Reasoning:**
- Memory tests are fast and isolated
- Node tests verify real filesystem behavior
- Easy to add new provider tests

## Future Enhancements

### Potential Features

1. **S3 Provider** - Cloud storage backend
2. **FTP Provider** - Remote filesystem access
3. **Virtual Provider** - Union of multiple providers
4. **Caching Layer** - In-memory cache for frequently accessed files
5. **Compression** - Transparent compression/decompression
6. **Encryption** - Transparent encryption at rest
7. **Versioning** - Automatic file versioning

### API Extensions

1. **Shortcut methods** - `fs.read('/path')` as alternative to `fs.file('/path').read()`
2. **Batch operations** - `fs.batch().file(...).file(...).execute()`
3. **Query API** - SQL-like queries for file listings
4. **Hooks** - Before/after operation hooks

## Dependencies

### Core Dependencies

- `@push.rocks/smartpath` - Path utilities
- `@types/node` - TypeScript types for Node.js

### Why Minimal Dependencies?

**Philosophy:**
- Keep the core light
- Avoid dependency hell
- Easier to maintain
- Faster installation

**Trade-off:**
- More code to maintain
- Can't leverage external libraries

**Decision:**
- Worth it for a foundational library
- Providers can have their own dependencies

## Compatibility

### Node.js Versions

Requires Node.js 18+ for:
- Native Web Streams API
- `fs.rm()` (replaces deprecated `fs.rmdir()`)
- `fs/promises` API

### Browser Compatibility

Core architecture supports browser, but:
- No browser provider implemented yet
- Would need IndexedDB or similar backend
- Stream handling already compatible

### TypeScript

Uses ES2022 target, NodeNext modules.

**Reasoning:**
- Modern JavaScript features
- ESM-first approach
- Better tree-shaking
- Future-proof

## Common Patterns

### Provider Implementation Checklist

When implementing a new provider:

1. ✅ Implement `ISmartFsProvider` interface
2. ✅ Set capability flags correctly
3. ✅ Normalize paths in constructor/methods
4. ✅ Handle errors with descriptive messages
5. ✅ Implement transaction support (or fallback)
6. ✅ Add comprehensive tests
7. ✅ Document provider-specific limitations

### Builder Pattern Example

```typescript
class SmartFsFile {
  private options = {};

  // Configuration (returns this)
  encoding(enc: string): this {
    this.options.encoding = enc;
    return this;
  }

  // Action (returns Promise)
  async read(): Promise<string | Buffer> {
    return this.provider.readFile(this.path, this.options);
  }
}
```

## Known Limitations

### Transaction Atomicity

- Not truly atomic across files
- Rollback can fail in edge cases
- No distributed transaction support

**Mitigation:**
- Document limitations clearly
- Best-effort rollback is sufficient for most cases
- Providers can override with native transactions

### File Watching

- Node.js `fs.watch` has platform-specific behavior
- May miss rapid changes
- No guarantee of event order

**Mitigation:**
- Debouncing helps with rapid changes
- Document platform differences
- Memory provider has predictable behavior (testing)

### Path Handling

- No cross-provider path compatibility
- Provider-specific path formats

**Mitigation:**
- Document path format per provider
- Use provider's `normalizePath()` method
- Consider adding path conversion utilities

## Maintenance Notes

### When to Update

- **Breaking changes:** Avoid unless absolutely necessary
- **New features:** Add as fluent methods or new builders
- **Bug fixes:** Prioritize data integrity
- **Performance:** Profile before optimizing

### Code Style

- Use TypeScript strict mode
- Prefer composition over inheritance
- Keep classes focused (SRP)
- Document public APIs with JSDoc
- Use meaningful variable names

### Testing Requirements

All changes must:
- Pass existing tests
- Add new tests for new features
- Maintain >90% code coverage
- Test both memory and Node providers

## Resources

- [Web Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Streams_API)
- [Node.js fs/promises](https://nodejs.org/api/fs.html#promises-api)
- [Builder Pattern](https://refactoring.guru/design-patterns/builder)
- [Provider Pattern](https://en.wikipedia.org/wiki/Provider_model)