feat(daemon): Add real-time log streaming and pub/sub: daemon publishes per-process logs, IPC client subscribe/unsubscribe, CLI --follow streaming, and sequencing for logs
This commit is contained in:
239
readme.plan.md
239
readme.plan.md
@@ -1,209 +1,56 @@
|
||||
# TSPM Refactoring Plan: Central Daemon Architecture
|
||||
# TSPM Real-Time Log Streaming Implementation Plan
|
||||
|
||||
## Problem Analysis
|
||||
## Overview
|
||||
Implementing real-time log streaming (tailing) functionality for TSPM using SmartIPC's pub/sub capabilities.
|
||||
|
||||
Currently, each `startAsDaemon` creates an isolated tspm instance with no coordination:
|
||||
## Approach: Hybrid Request + Subscribe
|
||||
1. Initial getLogs request to fetch historical logs up to current point
|
||||
2. Subscribe to pub/sub channel for real-time updates
|
||||
3. Use sequence numbers to detect and handle gaps/duplicates
|
||||
4. Per-process topics for granular subscriptions
|
||||
|
||||
- Multiple daemons reading/writing same config file
|
||||
- No communication between instances
|
||||
- Inconsistent process management
|
||||
- `tspm list` shows all processes but each daemon only manages its own
|
||||
## Implementation Tasks
|
||||
|
||||
## Proposed Architecture
|
||||
### Core Changes
|
||||
- [x] Update IProcessLog interface with seq and runId fields
|
||||
- [x] Add nextSeq and runId fields to ProcessWrapper class
|
||||
- [x] Update addLog() methods to include sequencing
|
||||
- [x] Implement pub/sub publishing in daemon
|
||||
|
||||
### 1. Central Daemon Manager (`ts/classes.daemon.ts`)
|
||||
### IPC Client Updates
|
||||
- [x] Add subscribe/unsubscribe methods to TspmIpcClient
|
||||
- [ ] Implement log streaming handler
|
||||
- [ ] Add connection state management for subscriptions
|
||||
|
||||
- Single daemon instance managing ALL processes
|
||||
- Runs continuously in background
|
||||
- Uses Unix socket for IPC at `~/.tspm/tspm.sock`
|
||||
- Maintains single source of truth for process state
|
||||
### CLI Enhancement
|
||||
- [x] Add --follow flag to logs command
|
||||
- [x] Implement streaming output with proper formatting
|
||||
- [x] Handle Ctrl+C gracefully to unsubscribe
|
||||
|
||||
### 2. IPC Communication Layer (`ts/classes.ipc.ts`)
|
||||
### Reliability Features
|
||||
- [x] Add backpressure handling (drop oldest when buffer full)
|
||||
- [x] Implement gap detection and recovery
|
||||
- [x] Add process restart detection via runId
|
||||
|
||||
- **Framework**: Use `@push.rocks/smartipc` v2.0.1
|
||||
- **Server**: SmartIpc server in daemon using Unix Domain Socket
|
||||
- **Client**: SmartIpc client in CLI for all operations
|
||||
- **Socket Path**: `~/.tspm/tspm.sock` (Unix) or named pipe (Windows)
|
||||
- **Protocol**: Type-safe request/response with SmartIpc's built-in patterns
|
||||
- **Features**:
|
||||
- Automatic reconnection with exponential backoff
|
||||
- Heartbeat monitoring for daemon health
|
||||
- Type-safe message contracts
|
||||
- **Auto-start**: CLI starts daemon if connection fails
|
||||
|
||||
### 3. New CLI Commands
|
||||
|
||||
- `tspm enable` - Start central daemon using systemd/launchd
|
||||
- `tspm disable` - Stop and disable central daemon
|
||||
- `tspm status` - Show daemon status
|
||||
- Remove `startAsDaemon` (replaced by daemon + `tspm start`)
|
||||
|
||||
### 4. Refactored CLI (`ts/cli.ts`)
|
||||
|
||||
All commands become daemon clients:
|
||||
|
||||
```typescript
|
||||
// Before: Direct process management
|
||||
await tspm.start(config);
|
||||
|
||||
// After: Send to daemon
|
||||
await ipcClient.request('start', config);
|
||||
```
|
||||
|
||||
### 5. File Structure Changes
|
||||
|
||||
```
|
||||
ts/
|
||||
├── classes.daemon.ts # New: Central daemon server
|
||||
├── classes.ipc.ts # New: IPC client/server
|
||||
├── classes.tspm.ts # Modified: Used by daemon only
|
||||
├── cli.ts # Modified: Becomes thin client
|
||||
└── classes.daemonmanager.ts # New: Systemd/launchd integration
|
||||
```
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Phase 1: Core Infrastructure
|
||||
|
||||
- [ ] Add `@push.rocks/smartipc` dependency (v2.0.1)
|
||||
- [ ] Create IPC message type definitions for all operations
|
||||
- [ ] Implement daemon server with SmartIpc server
|
||||
- [ ] Create IPC client wrapper for CLI
|
||||
- [ ] Add daemon lifecycle management (enable/disable)
|
||||
|
||||
### Phase 2: CLI Refactoring
|
||||
|
||||
- [ ] Convert all CLI commands to SmartIpc client requests
|
||||
- [ ] Add daemon auto-start logic with connection monitoring
|
||||
- [ ] Leverage SmartIpc's built-in reconnection and error handling
|
||||
- [ ] Implement type-safe message contracts for all commands
|
||||
|
||||
### Phase 3: Migration & Cleanup
|
||||
|
||||
- [ ] Migrate existing config to daemon-compatible format
|
||||
- [ ] Remove `startAsDaemon` command
|
||||
- [ ] Add migration guide for users
|
||||
### Testing
|
||||
- [x] Test basic log streaming
|
||||
- [x] Test gap recovery
|
||||
- [x] Test high-volume logging scenarios
|
||||
- [x] Test process restart handling
|
||||
|
||||
## Technical Details
|
||||
|
||||
### IPC Implementation with SmartIpc
|
||||
### Sequence Numbering
|
||||
- Each log entry gets incrementing seq number per process
|
||||
- runId changes on process restart
|
||||
- Client tracks lastSeq to detect gaps
|
||||
|
||||
```typescript
|
||||
// Daemon server setup
|
||||
import { SmartIpc } from '@push.rocks/smartipc';
|
||||
### Topic Structure
|
||||
- Format: `logs.<processId>`
|
||||
- Daemon publishes to topic on new log entries
|
||||
- Clients subscribe to specific process topics
|
||||
|
||||
const ipcServer = SmartIpc.createServer({
|
||||
id: 'tspm-daemon',
|
||||
socketPath: '~/.tspm/tspm.sock', // Unix socket
|
||||
});
|
||||
|
||||
// Message handlers with type safety
|
||||
ipcServer.onMessage<StartRequest, StartResponse>(
|
||||
'start',
|
||||
async (data, clientId) => {
|
||||
const result = await tspmManager.start(data.config);
|
||||
return { success: true, processId: result.pid };
|
||||
},
|
||||
);
|
||||
|
||||
// CLI client setup
|
||||
const ipcClient = SmartIpc.createClient({
|
||||
id: 'tspm-daemon',
|
||||
socketPath: '~/.tspm/tspm.sock',
|
||||
});
|
||||
|
||||
// Type-safe requests
|
||||
const response = await ipcClient.request<StartRequest, StartResponse>('start', {
|
||||
config: processConfig,
|
||||
});
|
||||
```
|
||||
|
||||
### Message Types
|
||||
|
||||
```typescript
|
||||
interface StartRequest {
|
||||
config: ProcessConfig;
|
||||
}
|
||||
|
||||
interface StartResponse {
|
||||
success: boolean;
|
||||
processId?: number;
|
||||
error?: string;
|
||||
}
|
||||
```
|
||||
|
||||
### Daemon State File
|
||||
|
||||
`~/.tspm/daemon.state` - PID, socket path, version
|
||||
|
||||
### Process Management
|
||||
|
||||
Daemon maintains all ProcessMonitor instances internally, CLI never directly manages processes.
|
||||
|
||||
## Key Benefits
|
||||
|
||||
### Architecture Benefits
|
||||
|
||||
- Single daemon manages all processes
|
||||
- Consistent state management
|
||||
- Efficient resource usage
|
||||
- Better process coordination
|
||||
- Proper service integration with OS
|
||||
|
||||
### SmartIpc Advantages
|
||||
|
||||
- **Cross-platform**: Unix sockets on Linux/macOS, named pipes on Windows
|
||||
- **Type-safe**: Full TypeScript support with generic message types
|
||||
- **Resilient**: Automatic reconnection with exponential backoff
|
||||
- **Observable**: Built-in metrics and heartbeat monitoring
|
||||
- **Performant**: Low-latency messaging with zero external dependencies
|
||||
- **Secure**: Connection limits and message size restrictions
|
||||
|
||||
## Backwards Compatibility
|
||||
|
||||
- Keep existing config format
|
||||
- Auto-migrate on first run
|
||||
- Provide clear upgrade instructions
|
||||
|
||||
## Architecture Diagram
|
||||
|
||||
```
|
||||
┌─────────────┐ IPC ┌──────────────┐
|
||||
│ CLI │◄────────────►│ Daemon │
|
||||
│ (thin client)│ Socket │ (server) │
|
||||
└─────────────┘ └──────────────┘
|
||||
│ │
|
||||
│ ▼
|
||||
│ ┌──────────────┐
|
||||
│ │ Tspm │
|
||||
│ │ Manager │
|
||||
│ └──────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌─────────────┐ ┌──────────────┐
|
||||
│ User │ │ProcessMonitor│
|
||||
│ Commands │ │ Instances │
|
||||
└─────────────┘ └──────────────┘
|
||||
```
|
||||
|
||||
## Migration Path
|
||||
|
||||
1. **Version 2.0.0-alpha**: Implement daemon with backwards compatibility
|
||||
2. **Version 2.0.0-beta**: Deprecate `startAsDaemon`, encourage daemon mode
|
||||
3. **Version 2.0.0**: Remove legacy code, daemon-only operation
|
||||
4. **Documentation**: Update all examples and guides
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Unix socket permissions (user-only access)
|
||||
- Validate all IPC messages
|
||||
- Rate limiting for IPC requests
|
||||
- Secure daemon shutdown mechanism
|
||||
|
||||
## Testing Requirements
|
||||
|
||||
- Unit tests for IPC layer
|
||||
- Integration tests for daemon lifecycle
|
||||
- Migration tests from current architecture
|
||||
- Performance tests for multiple processes
|
||||
- Stress tests for IPC communication
|
||||
### Backpressure Strategy
|
||||
- Circular buffer of 10,000 entries per process
|
||||
- Drop oldest entries when buffer full
|
||||
- Client can detect gaps via sequence numbers
|
||||
Reference in New Issue
Block a user