platformservice/readme.plan.md

# DcRouter SMTP Store-and-Forward Implementation Plan

## Overview
This plan outlines the implementation of a store-and-forward SMTP proxy within DcRouter that receives emails, processes them, and forwards them to the appropriate destinations. This capability expands DcRouter beyond simple connection proxying to provide full control over email flow, including content inspection, transformation, and reliable delivery.

## 0. Configuration Approaches

### 0.1 Core SmartProxy Direct Configuration
DcRouter should leverage SmartProxy's configuration directly, exposing SmartProxy's full domain configuration options to give users maximum flexibility for all HTTP/HTTPS and TCP/SNI traffic:

```typescript
interface IDcRouterOptions {
  // Direct SmartProxy configuration for general HTTP/HTTPS and TCP/SNI traffic
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // SMTP-specific configurations - can be used alongside smartProxyConfig
  // SMTP Store-and-forward configuration for advanced email processing
  smtpConfig?: ISmtpConfig;
  
  // Other DcRouter options...
}
```

This approach allows direct configuration of SmartProxy's powerful domain-based routing, giving full control over HTTP/HTTPS and SNI-based traffic:

```typescript
const dcRouter = new DcRouter({
  // Direct SmartProxy configuration for HTTP/HTTPS traffic
  smartProxyConfig: {
    fromPort: 443,
    toPort: 8080,
    targetIP: '10.0.0.10',
    sniEnabled: true,
    acme: {
      port: 80,
      enabled: true,
      autoRenew: true,
      useProduction: true,
      renewThresholdDays: 30,
      accountEmail: 'admin@example.com'
    },
    globalPortRanges: [
      { from: 80, to: 80 },
      { from: 443, to: 443 }
    ],
    
    // SmartProxy's full domain configuration flexibility
    domainConfigs: [
      {
        domains: ['example.com', 'www.example.com'],
        allowedIPs: ['0.0.0.0/0'],
        blockedIPs: ['1.2.3.4/32'],
        targetIPs: ['10.0.0.10', '10.0.0.11'],
        portRanges: [
          { from: 80, to: 80 },
          { from: 443, to: 443 }
        ],
        connectionTimeout: 60000,
        useNetworkProxy: true
      },
      // Additional domain configurations...
    ],
    
    // Additional SmartProxy options...
  },
  
  // Email-specific configuration (complementary to smartProxyConfig)
  smtpConfig: {
    // Email handling configuration...
  },
  
  // Other DcRouter configuration...
}
```

### 0.2 Store-and-Forward SMTP Implementation
For advanced email handling, we'll build a complete store-and-forward SMTP system to work alongside the direct SmartProxy configuration. This provides full control over email processing while maintaining SmartProxy's flexibility for HTTP/HTTPS traffic:

## 1. Core Architecture

### 1.1 SMTP Server Implementation
- [ ] Integrate an SMTP server library (like `smtp-server`) to accept incoming mail
  - Create a wrapper class that initializes and manages the SMTP server instance
  - Configure to listen on standard ports (25, 587, 465)
  - Implement TLS support (STARTTLS and implicit TLS)
  - Support authentication methods (PLAIN, LOGIN, OAUTH2)
  - Set up size limits and connection timeouts

### 1.2 Email Processing Pipeline
- [ ] Create a modular processing pipeline for emails
  - Build the core pipeline executor that manages the processing workflow
  - Implement plugin architecture for extensible processing steps
  - Create interfaces for each processing stage
  - Add metrics and logging points throughout the pipeline

### 1.3 Queue Management
- [ ] Develop a persistent queue system for email delivery
  - Implement in-memory queue for immediate delivery attempts
  - Create persistent storage for delivery retry queue
  - Build queue manager with priority, sorting, and scheduling capabilities
  - Add transaction support to prevent message loss during crashes

### 1.4 Email Delivery System
- [ ] Create a robust delivery system for outbound email
  - Implement connection pool for outbound SMTP connections
  - Add retry logic with configurable exponential backoff
  - Create delivery status tracking and notification
  - Set up bounce handling and processing

## 2. Email Processing Features

### 2.1 Routing and Forwarding
- [ ] Implement flexible email routing based on various criteria
  - Create domain-based routing rules
  - Support pattern matching for domains (exact match, wildcard)
  - Implement sender and recipient-based routing
  - Add load balancing across multiple target servers
  - Create failover support for high availability

### 2.2 Content Inspection
- [ ] Develop content inspection capabilities
  - Add MIME parsing and content extraction
  - Implement attachment scanning and filtering
  - Create text content analysis framework
  - Add integration points for external scanners (spam, virus)
  - Implement policy enforcement based on content

### 2.3 Email Transformation
- [ ] Create tools for modifying emails during transit
  - Implement header addition, modification, and removal
  - Add DKIM signing capability
  - Support email rewriting (address, content)
  - Create tools for attachment handling (remove, sanitize)
  - Support for adding tracking or compliance information

### 2.4 Rate Limiting and Traffic Control
- [ ] Build advanced rate limiting controls
  - Implement per-IP, per-sender, and per-domain rate limits
  - Create tiered rate limiting with different thresholds
  - Add traffic shaping capabilities to prevent spikes
  - Implement quota enforcement with customizable time windows
  - Create alert system for exceeding thresholds

## 3. Integration with DcRouter

### 3.1 Configuration Interface
- [ ] Extend DcRouter's configuration schema
  - Create comprehensive SMTP configuration section in IDcRouterOptions
  - Define interfaces for each SMTP feature set
  - Add validation rules for configuration values
  - Implement defaults for all configuration options
  - Document configuration parameters thoroughly

### 3.2 Management API
- [ ] Develop management APIs for runtime control
  - Create methods to update routing rules without restart
  - Implement queue management functions (pause, resume, inspect)
  - Add real-time monitoring endpoints
  - Create rate limit adjustment capabilities
  - Implement logging level controls

### 3.3 Metrics and Logging
- [ ] Implement comprehensive metrics gathering
  - Create counters for messages processed, delivered, and failed
  - Add timing metrics for processing stages
  - Implement detailed logging with configurable levels
  - Create structured log output for easier parsing
  - Add correlation IDs for tracking messages through the system

## 4. Detailed Component Specifications

### 4.0 DcRouter Configuration Extension

```typescript
export interface IDcRouterOptions {
  // Core configuration options
  
  // Direct SmartProxy configuration - gives full control over all TCP/SNI traffic
  // including HTTP, HTTPS, and any other TCP-based protocol
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // For backward compatibility and simplified HTTP configuration
  httpDomainRoutes?: IDomainRoutingConfig[];
  
  // SMTP store-and-forward processing - works alongside smartProxyConfig
  // This is for advanced email handling like content inspection
  smtpConfig?: ISmtpConfig;
  
  // Shared configurations
  tls?: {
    contactEmail: string;
    domain?: string;
    certPath?: string;
    keyPath?: string;
  };
  
  // Other DcRouter options
  dnsServerConfig?: plugins.smartdns.IDnsServerOptions;
  mtaConfig?: IMtaConfig; 
  mtaServiceInstance?: MtaService;
}
```

### 4.1 SmtpServer Class

```typescript
interface ISmtpServerOptions {
  // Base server options
  ports: number[];
  hostname: string;
  banner?: string;
  
  // Authentication options
  authMethods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
  requireAuth?: boolean;
  
  // TLS options
  tls?: {
    key?: string | Buffer;
    cert?: string | Buffer;
    ca?: string | Buffer | Array<string | Buffer>;
    ciphers?: string;
    minVersion?: string;
  };
  
  // Limits
  maxMessageSize?: number;
  maxClients?: number;
  maxConnections?: number;
  
  // Connection options
  connectionTimeout?: number;
  socketTimeout?: number;
}

/**
 * Manages the SMTP server for receiving emails
 */
class SmtpServer {
  constructor(options: ISmtpServerOptions);
  
  // Start and stop the server
  start(): Promise<void>;
  stop(): Promise<void>;
  
  // Event handlers
  onConnect(handler: (session: Session, callback: (err?: Error) => void) => void): void;
  onAuth(handler: (auth: AuthObject, session: Session, callback: (err?: Error, user?: UserInfo) => void) => void): void;
  onMailFrom(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onRcptTo(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onData(handler: (stream: Readable, session: Session, callback: (err?: Error) => void) => void): void;
  
  // Check email size before accepting data
  checkMessageSize(size: number): boolean;
  
  // Configuration updates
  updateOptions(options: Partial<ISmtpServerOptions>): void;
  
  // Server stats
  getStats(): IServerStats;
}
```

### 4.2 EmailProcessor Class

```typescript
interface IEmailProcessorOptions {
  // Processing options
  maxParallelProcessing?: number;
  processingTimeout?: number;
  
  // Feature flags
  contentScanning?: boolean;
  headerProcessing?: boolean;
  dkimSigning?: boolean;
  
  // Processing rules
  scanners?: IScannerConfig[];
  transformations?: ITransformConfig[];
  
  // Routing rules
  routingRules?: IRoutingRule[];
  defaultServer?: string;
  defaultPort?: number;
}

/**
 * Handles all email processing steps
 */
class EmailProcessor {
  constructor(options: IEmailProcessorOptions);
  
  // Main processing method
  async processEmail(message: ParsedMail, session: Session): Promise<ProcessingResult>;
  
  // Individual processing steps
  async extractMetadata(message: ParsedMail): Promise<EmailMetadata>;
  async determineRouting(metadata: EmailMetadata): Promise<RoutingDecision>;
  async scanContent(message: ParsedMail): Promise<ScanResult>;
  async applyTransformations(message: ParsedMail): Promise<ParsedMail>;
  
  // Update processor configuration
  updateOptions(options: Partial<IEmailProcessorOptions>): void;
  
  // Manage processing plugins
  addScanner(scanner: IScanner): void;
  addTransformation(transformation: ITransformation): void;
  addRoutingRule(rule: IRoutingRule): void;
}
```

### 4.3 DeliveryQueue Class

```typescript
interface IQueueOptions {
  // Storage options
  storageType: 'memory' | 'disk' | 'redis';
  storagePath?: string;
  redisUrl?: string;
  
  // Queue behavior
  checkInterval?: number;
  maxQueueSize?: number;
  maxPerDestination?: number;
  
  // Delivery attempts
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

/**
 * Manages the queue of messages waiting for delivery
 */
class DeliveryQueue {
  constructor(options: IQueueOptions);
  
  // Queue operations
  async enqueue(item: QueueItem): Promise<string>;
  async dequeue(id: string): Promise<QueueItem | null>;
  async update(id: string, updates: Partial<QueueItem>): Promise<boolean>;
  async getNext(count?: number): Promise<QueueItem[]>;
  
  // Query methods
  async getByStatus(status: QueueItemStatus): Promise<QueueItem[]>;
  async getByDestination(server: string): Promise<QueueItem[]>;
  async getItemCount(): Promise<number>;
  
  // Queue maintenance
  async purgeExpired(): Promise<number>;
  async purgeAll(): Promise<number>;
  
  // Persistence
  async load(): Promise<void>;
  async save(): Promise<void>;
  
  // Processing control
  pause(): void;
  resume(): void;
  isProcessing(): boolean;
}
```

### 4.4 DeliveryManager Class

```typescript
interface IDeliveryOptions {
  // Connection options
  connectionPoolSize?: number;
  socketTimeout?: number;
  
  // Delivery behavior
  concurrentDeliveries?: number;
  sendTimeout?: number;
  
  // TLS options
  verifyCertificates?: boolean;
  tlsMinVersion?: string;
  
  // Rate limiting
  globalRateLimit?: number;
  perServerRateLimit?: number;
  perDomainRateLimit?: Record<string, number>;
}

/**
 * Handles delivery of emails to destination servers
 */
class DeliveryManager {
  constructor(queue: DeliveryQueue, options: IDeliveryOptions);
  
  // Core delivery methods
  async start(): Promise<void>;
  async stop(): Promise<void>;
  async deliverMessage(item: QueueItem): Promise<DeliveryResult>;
  
  // Delivery management
  pauseDeliveries(): void;
  resumeDeliveries(): void;
  getDeliveryStats(): DeliveryStats;
  
  // Configure delivery behavior
  updateOptions(options: Partial<IDeliveryOptions>): void;
  setRateLimit(domain: string, limit: number): void;
  clearRateLimit(domain: string): void;
}
```

### 4.5 DcRouter SMTP Integration

```typescript
interface ISmtpConfig {
  // Server configuration
  ports: number[];
  hostname: string;
  banner?: string;
  maxMessageSize?: number;
  
  // TLS configuration
  tls?: {
    certPath?: string;
    keyPath?: string;
    caPath?: string;
    minVersion?: string;
  };
  
  // Authentication
  auth?: {
    required?: boolean;
    methods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
    users?: Array<{username: string, password: string}>;
    ldapUrl?: string;
  };
  
  // Domain routing
  domainConfigs: Array<{
    domains: string[];
    targetIPs: string[];
    port?: number;
    useTls?: boolean;
    authentication?: {
      user?: string;
      pass?: string;
    };
    allowedIPs?: string[];
    rateLimits?: {
      maxMessagesPerMinute?: number;
      maxRecipientsPerMessage?: number;
    };
    addHeaders?: boolean;
    headerInfo?: Array<{
      name: string;
      value: string;
    }>;
    signDkim?: boolean;
    dkimOptions?: {
      domainName: string;
      keySelector: string;
      privateKey: string;
    };
  }>;
  
  // Default routing
  defaultServer: string;
  defaultPort?: number;
  useTls?: boolean;
  
  // Content scanning
  contentScanning?: boolean;
  scanners?: Array<{
    type: 'spam' | 'virus' | 'attachment';
    threshold?: number;
    action: 'tag' | 'reject';
    blockedExtensions?: string[];
  }>;
  
  // Message transformations
  transformations?: Array<{
    type: string;
    [key: string]: any;
  }>;
  
  // Queue settings
  queueStorage?: 'memory' | 'disk';
  persistentPath?: string;
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

// Extended IDcRouterOptions
interface IDcRouterOptions {
  // Existing options...
  
  // New SMTP configuration
  smtpConfig?: ISmtpConfig;
}
```

## 5. Implementation Phases

### Phase 1: Core SMTP Server Setup
- [ ] Implement the SmtpServer class
- [ ] Set up TLS handling for both STARTTLS and implicit TLS
- [ ] Create the basic connection validation logic
- [ ] Implement authentication support
- [ ] Build email receiving pipeline to accept complete messages
- [ ] Create initial email parsing and storage

### Phase 2: Mail Processing and Routing
- [ ] Implement the EmailProcessor class
- [ ] Create domain-based routing rules
- [ ] Build email metadata extraction
- [ ] Implement MIME parsing and handling
- [ ] Create the transformation pipeline
- [ ] Build header manipulation capabilities

### Phase 3: Queue and Delivery System
- [ ] Implement the DeliveryQueue class
- [ ] Create persistent storage for queued messages
- [ ] Build the retry and scheduling logic
- [ ] Implement DeliveryManager with connection pooling
- [ ] Create the delivery status tracking and reporting
- [ ] Implement bounce handling and notification

### Phase 4: Advanced Features and Integration
- [ ] Integrate content scanning capabilities
- [ ] Implement DKIM signing
- [ ] Add rate limiting and traffic shaping
- [ ] Create comprehensive metrics and logging
- [ ] Build management APIs for runtime control
- [ ] Implement full integration with DcRouter

### Phase 5: Testing and Optimization
- [ ] Create unit tests for all components
- [ ] Implement integration tests for end-to-end verification
- [ ] Perform load testing and optimize performance
- [ ] Conduct security testing and hardening
- [ ] Build documentation and examples

## 6. Technical Requirements

### 6.1 Dependencies
- SMTP server library (smtp-server or similar)
- Email parsing library (mailparser or similar)
- MIME handling library
- DKIM signing library
- Queue management system (optional Redis support)
- Cryptographic libraries for TLS and authentication

### 6.2 Performance Targets
- Handle 1000+ concurrent SMTP connections
- Process 100+ messages per second on standard hardware
- Support message sizes up to 50MB
- Maintain delivery queue of 100,000+ messages
- Sub-second processing time for standard emails

### 6.3 Security Requirements
- Full TLS support with modern cipher configurations
- Authentication verification and rate limiting
- Input validation for all SMTP commands
- Secure storage of queued emails
- Proper error handling to prevent information leakage
- Access controls based on IP addresses and authentication

## 7. API Examples

### 7.1 Basic DcRouter SMTP Configuration

```typescript
const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    maxMessageSize: 50 * 1024 * 1024, // 50MB
    
    // TLS configuration
    tls: {
      certPath: '/path/to/cert.pem',
      keyPath: '/path/to/key.pem'
    },
    
    // Domain routing
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true
      }
    ],
    
    // Default routing
    defaultServer: 'fallback-mail.example.com',
    defaultPort: 25,
    useTls: true,
    
    // Queue settings
    queueStorage: 'disk',
    persistentPath: '/var/mail/queue',
    maxRetries: 5
  }
});
```

### 7.2 Advanced Configuration with Processing

```typescript
const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    // Basic settings
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    
    // Domain routing with advanced features
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true,
        // Add custom headers
        addHeaders: true,
        headerInfo: [
          { name: 'X-Processed-By', value: 'gateway' },
          { name: 'X-Scanned', value: 'true' }
        ],
        // Sign with DKIM
        signDkim: true,
        dkimOptions: {
          domainName: 'example.com',
          keySelector: 'mail',
          privateKey: '...'
        },
        // Rate limiting
        rateLimits: {
          maxMessagesPerMinute: 100,
          maxRecipientsPerMessage: 50
        }
      }
    ],
    
    // Content scanning
    contentScanning: true,
    scanners: [
      {
        type: 'spam',
        threshold: 5.0,
        action: 'tag'
      },
      {
        type: 'virus',
        action: 'reject'
      },
      {
        type: 'attachment',
        blockedExtensions: ['.exe', '.bat', '.vbs'],
        action: 'reject'
      }
    ],
    
    // Transformations
    transformations: [
      {
        type: 'addHeader',
        header: 'X-Gateway',
        value: 'DcRouter 1.0'
      },
      {
        type: 'dkimSign',
        domains: ['example.com']
      }
    ]
  }
});
```

## 8. Extensibility Points

### 8.1 Plugin Architecture
- Custom content scanners
- Custom transformation handlers
- Routing rule extensions
- Authentication providers
- Queue storage backends

### 8.2 Event System
- Connection events (connect, disconnect, error)
- Message events (received, processed, queued, delivered)
- Error events (delivery failure, processing error)
- Performance events (queue size, processing time)
- Security events (authentication failure, policy violation)

## 9. Migration Plan

### 9.1 From Simple Proxy to Store-and-Forward
- [ ] Create compatibility layer for existing configurations
- [ ] Implement graceful transition from connection proxy to full processing
- [ ] Add configuration validation to ensure smooth migration
- [ ] Create feature flags to enable advanced features incrementally
- [ ] Provide documentation for migrating existing deployments

### 9.2 Backward Compatibility
- [ ] Maintain support for basic proxy functionality
- [ ] Provide simple configuration options for common use cases
- [ ] Create migration utilities to update configuration formats
- [ ] Support running in hybrid mode during transition

## 10. SmartProxy Integration

### 10.1 SmartProxy Configuration Handling
- [ ] Implement comprehensive support for SmartProxy configuration
  - Pass through all SmartProxy options directly
  - Support all SmartProxy domain configuration features
  - Ensure proper handling of SmartProxy events and callbacks
- [ ] Create clear documentation on SmartProxy configuration:
  - Explain how all SmartProxy features are available through DcRouter
  - Document common usage patterns and examples
  - Provide guidance on advanced SmartProxy configurations

### 10.2 SMTP Integration with SmartProxy
- [ ] Ensure store-and-forward SMTP works alongside SmartProxy
  - Document how SMTP ports are properly handled by SMTP processing
  - Ensure no port conflicts between SmartProxy and SMTP server
  - Create examples showing SmartProxy and SMTP working together
- [ ] Document combined deployment models:
  - HTTP/HTTPS traffic using SmartProxy configuration
  - SMTP traffic using store-and-forward for advanced processing
  - Examples for multi-service environments

## 11. Documentation Requirements

### 11.1 Code Documentation
- [ ] Comprehensive JSDoc comments for all classes and methods
- [ ] Interface definitions with detailed parameter descriptions
- [ ] Example code snippets for common operations
- [ ] Architecture documentation with component diagrams
- [ ] Decision logs for key design choices

### 11.2 User Documentation
- [ ] Getting started guide with configuration approach selection guidance
- [ ] Complete configuration reference for both approaches
- [ ] Deployment scenarios and examples
- [ ] Troubleshooting guide
- [ ] Performance tuning recommendations
- [ ] Security best practices

### 11.3 Direct SmartProxy Configuration Guide
- [ ] Detailed guide on using SmartProxy's domain configuration capabilities
- [ ] Examples of complex routing scenarios with SmartProxy
- [ ] Performance optimization tips for SmartProxy configurations
- [ ] Security settings for SmartProxy deployments