platformservice/readme.plan.md

# DcRouter SMTP Store-and-Forward Implementation Plan

## Overview
This plan outlines the implementation of a store-and-forward SMTP proxy within DcRouter that receives emails, processes them, and forwards them to the appropriate destinations. This capability expands DcRouter beyond simple connection proxying to provide full control over email flow, including content inspection, transformation, and reliable delivery.

## 0. Configuration Approaches

### 0.1 Core SmartProxy Direct Configuration
DcRouter should leverage SmartProxy's configuration directly, exposing SmartProxy's full domain configuration options to give users maximum flexibility for all HTTP/HTTPS and TCP/SNI traffic:

```typescript
interface IDcRouterOptions {
  // Direct SmartProxy configuration for general HTTP/HTTPS and TCP/SNI traffic
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // SMTP-specific configurations - can be used alongside smartProxyConfig
  // SMTP Store-and-forward configuration for advanced email processing
  smtpConfig?: ISmtpConfig;
  
  // Other DcRouter options...
}
```

This approach allows direct configuration of SmartProxy's powerful domain-based routing, giving full control over HTTP/HTTPS and SNI-based traffic:

```typescript
const dcRouter = new DcRouter({
  // Direct SmartProxy configuration for HTTP/HTTPS traffic
  smartProxyConfig: {
    fromPort: 443,
    toPort: 8080,
    targetIP: '10.0.0.10',
    sniEnabled: true,
    acme: {
      port: 80,
      enabled: true,
      autoRenew: true,
      useProduction: true,
      renewThresholdDays: 30,
      accountEmail: 'admin@example.com'
    },
    globalPortRanges: [
      { from: 80, to: 80 },
      { from: 443, to: 443 }
    ],
    
    // SmartProxy's full domain configuration flexibility
    domainConfigs: [
      {
        domains: ['example.com', 'www.example.com'],
        allowedIPs: ['0.0.0.0/0'],
        blockedIPs: ['1.2.3.4/32'],
        targetIPs: ['10.0.0.10', '10.0.0.11'],
        portRanges: [
          { from: 80, to: 80 },
          { from: 443, to: 443 }
        ],
        connectionTimeout: 60000,
        useNetworkProxy: true
      },
      // Additional domain configurations...
    ],
    
    // Additional SmartProxy options...
  },
  
  // Email-specific configuration (complementary to smartProxyConfig)
  smtpConfig: {
    // Email handling configuration...
  },
  
  // Other DcRouter configuration...
}
```

### 0.2 Store-and-Forward SMTP Implementation
For advanced email handling, we'll build a complete store-and-forward SMTP system to work alongside the direct SmartProxy configuration. This provides full control over email processing while maintaining SmartProxy's flexibility for HTTP/HTTPS traffic:

## 1. Core Architecture

### 1.1 SMTP Server Implementation
- [x] Integrate an SMTP server library (like `smtp-server`) to accept incoming mail
  - Created the SmtpServer class that initializes and manages the SMTP server instance
  - Configured to listen on standard ports (25, 587, 465)
  - Implemented TLS support (STARTTLS and implicit TLS)
  - Added support for authentication methods (PLAIN, LOGIN, OAUTH2)
  - Set up size limits and connection timeouts

### 1.2 Email Processing Pipeline
- [x] Create a modular processing pipeline for emails
  - Built the EmailProcessor class that manages the processing workflow
  - Implemented event-based architecture for extensible processing steps
  - Created interfaces for each processing stage (metadata extraction, content scanning, routing, transformation)
  - Added metrics and logging points throughout the pipeline

### 1.3 Queue Management
- [x] Develop a persistent queue system for email delivery
  - Implemented DeliveryQueue class with in-memory queue for immediate delivery attempts
  - Created persistent storage for delivery retry queue with file-based storage
  - Built queue manager with scheduling capabilities
  - Added transaction support to prevent message loss during crashes

### 1.4 Email Delivery System
- [x] Create a robust delivery system for outbound email
  - Implemented DeliverySystem class for outbound SMTP connections
  - Added retry logic with configurable exponential backoff
  - Created delivery status tracking and notifications via events
  - Set up initial bounce handling and processing

## 2. Email Processing Features

### 2.1 Routing and Forwarding
- [x] Implement flexible email routing based on various criteria
  - Created domain-based routing rules in EmailProcessor
  - Added support for pattern matching for domains (exact match, wildcard)
  - Implemented recipient-based routing
  - Added support for routing across multiple target servers
  - Added initial failover support for high availability

### 2.2 Content Inspection
- [x] Develop content inspection capabilities
  - Added MIME parsing and content extraction using mailparser
  - Implemented attachment scanning and filtering based on extensions
  - Created plugin architecture for content analysis
  - Added integration points for external scanners (spam, virus)
  - Implemented policy enforcement based on content scan results

### 2.3 Email Transformation
- [x] Create tools for modifying emails during transit
  - Implemented header addition capabilities
  - Added DKIM signing capability placeholder
  - Created framework for email transformations
  - Added attachment handling capability
  - Implemented support for adding compliance information

### 2.4 Rate Limiting and Traffic Control
- [x] Build rate limiting controls
  - Implemented per-domain rate limits
  - Added support for configurable rate limiting thresholds
  - Created quota enforcement with domain-based configuration
  - Added event system for rate limit notifications

## 3. Integration with DcRouter

### 3.1 Configuration Interface
- [x] Extend DcRouter's configuration schema
  - Created comprehensive SMTP configuration section in IDcRouterOptions
  - Defined interfaces for each SMTP feature set
  - Added validation with defaults for configuration values
  - Implemented sensible defaults for all configuration options
  - Added detailed documentation in code comments

### 3.2 Management API
- [x] Develop management APIs for runtime control
  - Created methods to update configuration without restart
  - Implemented queue management functions (pause, resume, inspect)
  - Added status reporting through events
  - Created configuration update methods
  - Implemented graceful shutdown capabilities

### 3.3 Metrics and Logging
- [x] Implement metrics gathering
  - Created counters for messages processed, delivered, and failed
  - Added tracking for processing stages
  - Implemented detailed logging
  - Added message IDs for tracking through the system

## 4. Detailed Component Specifications

### 4.0 DcRouter Configuration Extension

```typescript
export interface IDcRouterOptions {
  // Core configuration options
  
  // Direct SmartProxy configuration - gives full control over all TCP/SNI traffic
  // including HTTP, HTTPS, and any other TCP-based protocol
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // For backward compatibility and simplified HTTP configuration
  httpDomainRoutes?: IDomainRoutingConfig[];
  
  // SMTP store-and-forward processing - works alongside smartProxyConfig
  // This is for advanced email handling like content inspection
  smtpConfig?: ISmtpConfig;
  
  // Shared configurations
  tls?: {
    contactEmail: string;
    domain?: string;
    certPath?: string;
    keyPath?: string;
  };
  
  // Other DcRouter options
  dnsServerConfig?: plugins.smartdns.IDnsServerOptions;
  mtaConfig?: IMtaConfig; 
  mtaServiceInstance?: MtaService;
}
```

### 4.1 SmtpServer Class

```typescript
interface ISmtpServerOptions {
  // Base server options
  ports: number[];
  hostname: string;
  banner?: string;
  
  // Authentication options
  authMethods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
  requireAuth?: boolean;
  
  // TLS options
  tls?: {
    key?: string | Buffer;
    cert?: string | Buffer;
    ca?: string | Buffer | Array<string | Buffer>;
    ciphers?: string;
    minVersion?: string;
  };
  
  // Limits
  maxMessageSize?: number;
  maxClients?: number;
  maxConnections?: number;
  
  // Connection options
  connectionTimeout?: number;
  socketTimeout?: number;
}

/**
 * Manages the SMTP server for receiving emails
 */
class SmtpServer {
  constructor(options: ISmtpServerOptions);
  
  // Start and stop the server
  start(): Promise<void>;
  stop(): Promise<void>;
  
  // Event handlers
  onConnect(handler: (session: Session, callback: (err?: Error) => void) => void): void;
  onAuth(handler: (auth: AuthObject, session: Session, callback: (err?: Error, user?: UserInfo) => void) => void): void;
  onMailFrom(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onRcptTo(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onData(handler: (stream: Readable, session: Session, callback: (err?: Error) => void) => void): void;
  
  // Check email size before accepting data
  checkMessageSize(size: number): boolean;
  
  // Configuration updates
  updateOptions(options: Partial<ISmtpServerOptions>): void;
  
  // Server stats
  getStats(): IServerStats;
}
```

### 4.2 EmailProcessor Class

```typescript
interface IEmailProcessorOptions {
  // Processing options
  maxParallelProcessing?: number;
  processingTimeout?: number;
  
  // Feature flags
  contentScanning?: boolean;
  headerProcessing?: boolean;
  dkimSigning?: boolean;
  
  // Processing rules
  scanners?: IScannerConfig[];
  transformations?: ITransformConfig[];
  
  // Routing rules
  routingRules?: IRoutingRule[];
  defaultServer?: string;
  defaultPort?: number;
}

/**
 * Handles all email processing steps
 */
class EmailProcessor {
  constructor(options: IEmailProcessorOptions);
  
  // Main processing method
  async processEmail(message: ParsedMail, session: Session): Promise<ProcessingResult>;
  
  // Individual processing steps
  async extractMetadata(message: ParsedMail): Promise<EmailMetadata>;
  async determineRouting(metadata: EmailMetadata): Promise<RoutingDecision>;
  async scanContent(message: ParsedMail): Promise<ScanResult>;
  async applyTransformations(message: ParsedMail): Promise<ParsedMail>;
  
  // Update processor configuration
  updateOptions(options: Partial<IEmailProcessorOptions>): void;
  
  // Manage processing plugins
  addScanner(scanner: IScanner): void;
  addTransformation(transformation: ITransformation): void;
  addRoutingRule(rule: IRoutingRule): void;
}
```

### 4.3 DeliveryQueue Class

```typescript
interface IQueueOptions {
  // Storage options
  storageType: 'memory' | 'disk' | 'redis';
  storagePath?: string;
  redisUrl?: string;
  
  // Queue behavior
  checkInterval?: number;
  maxQueueSize?: number;
  maxPerDestination?: number;
  
  // Delivery attempts
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

/**
 * Manages the queue of messages waiting for delivery
 */
class DeliveryQueue {
  constructor(options: IQueueOptions);
  
  // Queue operations
  async enqueue(item: QueueItem): Promise<string>;
  async dequeue(id: string): Promise<QueueItem | null>;
  async update(id: string, updates: Partial<QueueItem>): Promise<boolean>;
  async getNext(count?: number): Promise<QueueItem[]>;
  
  // Query methods
  async getByStatus(status: QueueItemStatus): Promise<QueueItem[]>;
  async getByDestination(server: string): Promise<QueueItem[]>;
  async getItemCount(): Promise<number>;
  
  // Queue maintenance
  async purgeExpired(): Promise<number>;
  async purgeAll(): Promise<number>;
  
  // Persistence
  async load(): Promise<void>;
  async save(): Promise<void>;
  
  // Processing control
  pause(): void;
  resume(): void;
  isProcessing(): boolean;
}
```

### 4.4 DeliveryManager Class

```typescript
interface IDeliveryOptions {
  // Connection options
  connectionPoolSize?: number;
  socketTimeout?: number;
  
  // Delivery behavior
  concurrentDeliveries?: number;
  sendTimeout?: number;
  
  // TLS options
  verifyCertificates?: boolean;
  tlsMinVersion?: string;
  
  // Rate limiting
  globalRateLimit?: number;
  perServerRateLimit?: number;
  perDomainRateLimit?: Record<string, number>;
}

/**
 * Handles delivery of emails to destination servers
 */
class DeliveryManager {
  constructor(queue: DeliveryQueue, options: IDeliveryOptions);
  
  // Core delivery methods
  async start(): Promise<void>;
  async stop(): Promise<void>;
  async deliverMessage(item: QueueItem): Promise<DeliveryResult>;
  
  // Delivery management
  pauseDeliveries(): void;
  resumeDeliveries(): void;
  getDeliveryStats(): DeliveryStats;
  
  // Configure delivery behavior
  updateOptions(options: Partial<IDeliveryOptions>): void;
  setRateLimit(domain: string, limit: number): void;
  clearRateLimit(domain: string): void;
}
```

### 4.5 DcRouter SMTP Integration

```typescript
interface ISmtpConfig {
  // Server configuration
  ports: number[];
  hostname: string;
  banner?: string;
  maxMessageSize?: number;
  
  // TLS configuration
  tls?: {
    certPath?: string;
    keyPath?: string;
    caPath?: string;
    minVersion?: string;
  };
  
  // Authentication
  auth?: {
    required?: boolean;
    methods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
    users?: Array<{username: string, password: string}>;
    ldapUrl?: string;
  };
  
  // Domain routing
  domainConfigs: Array<{
    domains: string[];
    targetIPs: string[];
    port?: number;
    useTls?: boolean;
    authentication?: {
      user?: string;
      pass?: string;
    };
    allowedIPs?: string[];
    rateLimits?: {
      maxMessagesPerMinute?: number;
      maxRecipientsPerMessage?: number;
    };
    addHeaders?: boolean;
    headerInfo?: Array<{
      name: string;
      value: string;
    }>;
    signDkim?: boolean;
    dkimOptions?: {
      domainName: string;
      keySelector: string;
      privateKey: string;
    };
  }>;
  
  // Default routing
  defaultServer: string;
  defaultPort?: number;
  useTls?: boolean;
  
  // Content scanning
  contentScanning?: boolean;
  scanners?: Array<{
    type: 'spam' | 'virus' | 'attachment';
    threshold?: number;
    action: 'tag' | 'reject';
    blockedExtensions?: string[];
  }>;
  
  // Message transformations
  transformations?: Array<{
    type: string;
    [key: string]: any;
  }>;
  
  // Queue settings
  queueStorage?: 'memory' | 'disk';
  persistentPath?: string;
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

// Extended IDcRouterOptions
interface IDcRouterOptions {
  // Existing options...
  
  // New SMTP configuration
  smtpConfig?: ISmtpConfig;
}
```

## 5. Implementation Phases

### Phase 1: Core SMTP Server Setup
- [ ] Implement the SmtpServer class
- [ ] Set up TLS handling for both STARTTLS and implicit TLS
- [ ] Create the basic connection validation logic
- [ ] Implement authentication support
- [ ] Build email receiving pipeline to accept complete messages
- [ ] Create initial email parsing and storage

### Phase 2: Mail Processing and Routing
- [ ] Implement the EmailProcessor class
- [ ] Create domain-based routing rules
- [ ] Build email metadata extraction
- [ ] Implement MIME parsing and handling
- [ ] Create the transformation pipeline
- [ ] Build header manipulation capabilities

### Phase 3: Queue and Delivery System
- [ ] Implement the DeliveryQueue class
- [ ] Create persistent storage for queued messages
- [ ] Build the retry and scheduling logic
- [ ] Implement DeliveryManager with connection pooling
- [ ] Create the delivery status tracking and reporting
- [ ] Implement bounce handling and notification

### Phase 4: Advanced Features and Integration
- [ ] Integrate content scanning capabilities
- [ ] Implement DKIM signing
- [ ] Add rate limiting and traffic shaping
- [ ] Create comprehensive metrics and logging
- [ ] Build management APIs for runtime control
- [ ] Implement full integration with DcRouter

### Phase 5: Testing and Optimization
- [ ] Create unit tests for all components
- [ ] Implement integration tests for end-to-end verification
- [ ] Perform load testing and optimize performance
- [ ] Conduct security testing and hardening
- [ ] Build documentation and examples

## 6. Technical Requirements

### 6.1 Dependencies
- SMTP server library (smtp-server or similar)
- Email parsing library (mailparser or similar)
- MIME handling library
- DKIM signing library
- Queue management system (optional Redis support)
- Cryptographic libraries for TLS and authentication

### 6.2 Performance Targets
- Handle 1000+ concurrent SMTP connections
- Process 100+ messages per second on standard hardware
- Support message sizes up to 50MB
- Maintain delivery queue of 100,000+ messages
- Sub-second processing time for standard emails

### 6.3 Security Requirements
- Full TLS support with modern cipher configurations
- Authentication verification and rate limiting
- Input validation for all SMTP commands
- Secure storage of queued emails
- Proper error handling to prevent information leakage
- Access controls based on IP addresses and authentication

## 7. API Examples

### 7.1 Basic DcRouter SMTP Configuration

```typescript
const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    maxMessageSize: 50 * 1024 * 1024, // 50MB
    
    // TLS configuration
    tls: {
      certPath: '/path/to/cert.pem',
      keyPath: '/path/to/key.pem'
    },
    
    // Domain routing
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true
      }
    ],
    
    // Default routing
    defaultServer: 'fallback-mail.example.com',
    defaultPort: 25,
    useTls: true,
    
    // Queue settings
    queueStorage: 'disk',
    persistentPath: '/var/mail/queue',
    maxRetries: 5
  }
});
```

### 7.2 Advanced Configuration with Processing

```typescript
const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    // Basic settings
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    
    // Domain routing with advanced features
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true,
        // Add custom headers
        addHeaders: true,
        headerInfo: [
          { name: 'X-Processed-By', value: 'gateway' },
          { name: 'X-Scanned', value: 'true' }
        ],
        // Sign with DKIM
        signDkim: true,
        dkimOptions: {
          domainName: 'example.com',
          keySelector: 'mail',
          privateKey: '...'
        },
        // Rate limiting
        rateLimits: {
          maxMessagesPerMinute: 100,
          maxRecipientsPerMessage: 50
        }
      }
    ],
    
    // Content scanning
    contentScanning: true,
    scanners: [
      {
        type: 'spam',
        threshold: 5.0,
        action: 'tag'
      },
      {
        type: 'virus',
        action: 'reject'
      },
      {
        type: 'attachment',
        blockedExtensions: ['.exe', '.bat', '.vbs'],
        action: 'reject'
      }
    ],
    
    // Transformations
    transformations: [
      {
        type: 'addHeader',
        header: 'X-Gateway',
        value: 'DcRouter 1.0'
      },
      {
        type: 'dkimSign',
        domains: ['example.com']
      }
    ]
  }
});
```

## 8. Extensibility Points

### 8.1 Plugin Architecture
- Custom content scanners
- Custom transformation handlers
- Routing rule extensions
- Authentication providers
- Queue storage backends

### 8.2 Event System
- Connection events (connect, disconnect, error)
- Message events (received, processed, queued, delivered)
- Error events (delivery failure, processing error)
- Performance events (queue size, processing time)
- Security events (authentication failure, policy violation)

## 9. Migration Plan

### 9.1 From Simple Proxy to Store-and-Forward
- [ ] Create compatibility layer for existing configurations
- [ ] Implement graceful transition from connection proxy to full processing
- [ ] Add configuration validation to ensure smooth migration
- [ ] Create feature flags to enable advanced features incrementally
- [ ] Provide documentation for migrating existing deployments

### 9.2 Backward Compatibility
- [ ] Maintain support for basic proxy functionality
- [ ] Provide simple configuration options for common use cases
- [ ] Create migration utilities to update configuration formats
- [ ] Support running in hybrid mode during transition

## 10. SmartProxy Integration

### 10.1 SmartProxy Configuration Handling
- [x] Implement comprehensive support for SmartProxy configuration
  - Passed through all SmartProxy options directly in DcRouter's configuration
  - Added support for all SmartProxy domain configuration features
  - Implemented proper handling of SmartProxy events and callbacks
- [x] Added documentation on SmartProxy configuration:
  - Documented how all SmartProxy features are available through DcRouter
  - Added examples of different configuration approaches
  - Provided guidance in code comments

### 10.2 SMTP Integration with SmartProxy
- [x] Ensured store-and-forward SMTP works alongside SmartProxy
  - Handled SMTP ports separately from HTTP/HTTPS ports
  - Prevented port conflicts between SmartProxy and SMTP server
  - Created code structure showing SmartProxy and SMTP working together
- [x] Implemented combined usage model:
  - HTTP/HTTPS traffic using SmartProxy configuration
  - SMTP traffic using store-and-forward for advanced processing
  - Added support for multi-service environments

## 11. Documentation Requirements

### 11.1 Code Documentation
- [ ] Comprehensive JSDoc comments for all classes and methods
- [ ] Interface definitions with detailed parameter descriptions
- [ ] Example code snippets for common operations
- [ ] Architecture documentation with component diagrams
- [ ] Decision logs for key design choices

### 11.2 User Documentation
- [ ] Getting started guide with configuration approach selection guidance
- [ ] Complete configuration reference for both approaches
- [ ] Deployment scenarios and examples
- [ ] Troubleshooting guide
- [ ] Performance tuning recommendations
- [ ] Security best practices

### 11.3 Direct SmartProxy Configuration Guide
- [ ] Detailed guide on using SmartProxy's domain configuration capabilities
- [ ] Examples of complex routing scenarios with SmartProxy
- [ ] Performance optimization tips for SmartProxy configurations
- [ ] Security settings for SmartProxy deployments