2025-05-07 23:30:04 +00:00

22 KiB

Raw Blame History

DcRouter SMTP Store-and-Forward Implementation Plan

Overview

This plan outlines the implementation of a store-and-forward SMTP proxy within DcRouter that receives emails, processes them, and forwards them to the appropriate destinations. This capability expands DcRouter beyond simple connection proxying to provide full control over email flow, including content inspection, transformation, and reliable delivery.

0. Configuration Approaches

0.1 Core SmartProxy Direct Configuration

DcRouter should leverage SmartProxy's configuration directly, exposing SmartProxy's full domain configuration options to give users maximum flexibility for all HTTP/HTTPS and TCP/SNI traffic:

interface IDcRouterOptions {
  // Direct SmartProxy configuration for general HTTP/HTTPS and TCP/SNI traffic
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // SMTP-specific configurations - can be used alongside smartProxyConfig
  // SMTP Store-and-forward configuration for advanced email processing
  smtpConfig?: ISmtpConfig;
  
  // Other DcRouter options...
}

This approach allows direct configuration of SmartProxy's powerful domain-based routing, giving full control over HTTP/HTTPS and SNI-based traffic:

const dcRouter = new DcRouter({
  // Direct SmartProxy configuration for HTTP/HTTPS traffic
  smartProxyConfig: {
    fromPort: 443,
    toPort: 8080,
    targetIP: '10.0.0.10',
    sniEnabled: true,
    acme: {
      port: 80,
      enabled: true,
      autoRenew: true,
      useProduction: true,
      renewThresholdDays: 30,
      accountEmail: 'admin@example.com'
    },
    globalPortRanges: [
      { from: 80, to: 80 },
      { from: 443, to: 443 }
    ],
    
    // SmartProxy's full domain configuration flexibility
    domainConfigs: [
      {
        domains: ['example.com', 'www.example.com'],
        allowedIPs: ['0.0.0.0/0'],
        blockedIPs: ['1.2.3.4/32'],
        targetIPs: ['10.0.0.10', '10.0.0.11'],
        portRanges: [
          { from: 80, to: 80 },
          { from: 443, to: 443 }
        ],
        connectionTimeout: 60000,
        useNetworkProxy: true
      },
      // Additional domain configurations...
    ],
    
    // Additional SmartProxy options...
  },
  
  // Email-specific configuration (complementary to smartProxyConfig)
  smtpConfig: {
    // Email handling configuration...
  },
  
  // Other DcRouter configuration...
}

0.2 Store-and-Forward SMTP Implementation

For advanced email handling, we'll build a complete store-and-forward SMTP system to work alongside the direct SmartProxy configuration. This provides full control over email processing while maintaining SmartProxy's flexibility for HTTP/HTTPS traffic:

1. Core Architecture

1.1 SMTP Server Implementation

Integrate an SMTP server library (like smtp-server) to accept incoming mail
- Create a wrapper class that initializes and manages the SMTP server instance
- Configure to listen on standard ports (25, 587, 465)
- Implement TLS support (STARTTLS and implicit TLS)
- Support authentication methods (PLAIN, LOGIN, OAUTH2)
- Set up size limits and connection timeouts

1.2 Email Processing Pipeline

Create a modular processing pipeline for emails
- Build the core pipeline executor that manages the processing workflow
- Implement plugin architecture for extensible processing steps
- Create interfaces for each processing stage
- Add metrics and logging points throughout the pipeline

1.3 Queue Management

Develop a persistent queue system for email delivery
- Implement in-memory queue for immediate delivery attempts
- Create persistent storage for delivery retry queue
- Build queue manager with priority, sorting, and scheduling capabilities
- Add transaction support to prevent message loss during crashes

1.4 Email Delivery System

Create a robust delivery system for outbound email
- Implement connection pool for outbound SMTP connections
- Add retry logic with configurable exponential backoff
- Create delivery status tracking and notification
- Set up bounce handling and processing

2. Email Processing Features

2.1 Routing and Forwarding

Implement flexible email routing based on various criteria
- Create domain-based routing rules
- Support pattern matching for domains (exact match, wildcard)
- Implement sender and recipient-based routing
- Add load balancing across multiple target servers
- Create failover support for high availability

2.2 Content Inspection

Develop content inspection capabilities
- Add MIME parsing and content extraction
- Implement attachment scanning and filtering
- Create text content analysis framework
- Add integration points for external scanners (spam, virus)
- Implement policy enforcement based on content

2.3 Email Transformation

Create tools for modifying emails during transit
- Implement header addition, modification, and removal
- Add DKIM signing capability
- Support email rewriting (address, content)
- Create tools for attachment handling (remove, sanitize)
- Support for adding tracking or compliance information

2.4 Rate Limiting and Traffic Control

Build advanced rate limiting controls
- Implement per-IP, per-sender, and per-domain rate limits
- Create tiered rate limiting with different thresholds
- Add traffic shaping capabilities to prevent spikes
- Implement quota enforcement with customizable time windows
- Create alert system for exceeding thresholds

3. Integration with DcRouter

3.1 Configuration Interface

Extend DcRouter's configuration schema
- Create comprehensive SMTP configuration section in IDcRouterOptions
- Define interfaces for each SMTP feature set
- Add validation rules for configuration values
- Implement defaults for all configuration options
- Document configuration parameters thoroughly

3.2 Management API

Develop management APIs for runtime control
- Create methods to update routing rules without restart
- Implement queue management functions (pause, resume, inspect)
- Add real-time monitoring endpoints
- Create rate limit adjustment capabilities
- Implement logging level controls

3.3 Metrics and Logging

Implement comprehensive metrics gathering
- Create counters for messages processed, delivered, and failed
- Add timing metrics for processing stages
- Implement detailed logging with configurable levels
- Create structured log output for easier parsing
- Add correlation IDs for tracking messages through the system

4. Detailed Component Specifications

4.0 DcRouter Configuration Extension

export interface IDcRouterOptions {
  // Core configuration options
  
  // Direct SmartProxy configuration - gives full control over all TCP/SNI traffic
  // including HTTP, HTTPS, and any other TCP-based protocol
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // For backward compatibility and simplified HTTP configuration
  httpDomainRoutes?: IDomainRoutingConfig[];
  
  // SMTP store-and-forward processing - works alongside smartProxyConfig
  // This is for advanced email handling like content inspection
  smtpConfig?: ISmtpConfig;
  
  // Shared configurations
  tls?: {
    contactEmail: string;
    domain?: string;
    certPath?: string;
    keyPath?: string;
  };
  
  // Other DcRouter options
  dnsServerConfig?: plugins.smartdns.IDnsServerOptions;
  mtaConfig?: IMtaConfig; 
  mtaServiceInstance?: MtaService;
}

4.1 SmtpServer Class

interface ISmtpServerOptions {
  // Base server options
  ports: number[];
  hostname: string;
  banner?: string;
  
  // Authentication options
  authMethods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
  requireAuth?: boolean;
  
  // TLS options
  tls?: {
    key?: string | Buffer;
    cert?: string | Buffer;
    ca?: string | Buffer | Array<string | Buffer>;
    ciphers?: string;
    minVersion?: string;
  };
  
  // Limits
  maxMessageSize?: number;
  maxClients?: number;
  maxConnections?: number;
  
  // Connection options
  connectionTimeout?: number;
  socketTimeout?: number;
}

/**
 * Manages the SMTP server for receiving emails
 */
class SmtpServer {
  constructor(options: ISmtpServerOptions);
  
  // Start and stop the server
  start(): Promise<void>;
  stop(): Promise<void>;
  
  // Event handlers
  onConnect(handler: (session: Session, callback: (err?: Error) => void) => void): void;
  onAuth(handler: (auth: AuthObject, session: Session, callback: (err?: Error, user?: UserInfo) => void) => void): void;
  onMailFrom(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onRcptTo(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onData(handler: (stream: Readable, session: Session, callback: (err?: Error) => void) => void): void;
  
  // Check email size before accepting data
  checkMessageSize(size: number): boolean;
  
  // Configuration updates
  updateOptions(options: Partial<ISmtpServerOptions>): void;
  
  // Server stats
  getStats(): IServerStats;
}

4.2 EmailProcessor Class

interface IEmailProcessorOptions {
  // Processing options
  maxParallelProcessing?: number;
  processingTimeout?: number;
  
  // Feature flags
  contentScanning?: boolean;
  headerProcessing?: boolean;
  dkimSigning?: boolean;
  
  // Processing rules
  scanners?: IScannerConfig[];
  transformations?: ITransformConfig[];
  
  // Routing rules
  routingRules?: IRoutingRule[];
  defaultServer?: string;
  defaultPort?: number;
}

/**
 * Handles all email processing steps
 */
class EmailProcessor {
  constructor(options: IEmailProcessorOptions);
  
  // Main processing method
  async processEmail(message: ParsedMail, session: Session): Promise<ProcessingResult>;
  
  // Individual processing steps
  async extractMetadata(message: ParsedMail): Promise<EmailMetadata>;
  async determineRouting(metadata: EmailMetadata): Promise<RoutingDecision>;
  async scanContent(message: ParsedMail): Promise<ScanResult>;
  async applyTransformations(message: ParsedMail): Promise<ParsedMail>;
  
  // Update processor configuration
  updateOptions(options: Partial<IEmailProcessorOptions>): void;
  
  // Manage processing plugins
  addScanner(scanner: IScanner): void;
  addTransformation(transformation: ITransformation): void;
  addRoutingRule(rule: IRoutingRule): void;
}

4.3 DeliveryQueue Class

interface IQueueOptions {
  // Storage options
  storageType: 'memory' | 'disk' | 'redis';
  storagePath?: string;
  redisUrl?: string;
  
  // Queue behavior
  checkInterval?: number;
  maxQueueSize?: number;
  maxPerDestination?: number;
  
  // Delivery attempts
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

/**
 * Manages the queue of messages waiting for delivery
 */
class DeliveryQueue {
  constructor(options: IQueueOptions);
  
  // Queue operations
  async enqueue(item: QueueItem): Promise<string>;
  async dequeue(id: string): Promise<QueueItem | null>;
  async update(id: string, updates: Partial<QueueItem>): Promise<boolean>;
  async getNext(count?: number): Promise<QueueItem[]>;
  
  // Query methods
  async getByStatus(status: QueueItemStatus): Promise<QueueItem[]>;
  async getByDestination(server: string): Promise<QueueItem[]>;
  async getItemCount(): Promise<number>;
  
  // Queue maintenance
  async purgeExpired(): Promise<number>;
  async purgeAll(): Promise<number>;
  
  // Persistence
  async load(): Promise<void>;
  async save(): Promise<void>;
  
  // Processing control
  pause(): void;
  resume(): void;
  isProcessing(): boolean;
}

4.4 DeliveryManager Class

interface IDeliveryOptions {
  // Connection options
  connectionPoolSize?: number;
  socketTimeout?: number;
  
  // Delivery behavior
  concurrentDeliveries?: number;
  sendTimeout?: number;
  
  // TLS options
  verifyCertificates?: boolean;
  tlsMinVersion?: string;
  
  // Rate limiting
  globalRateLimit?: number;
  perServerRateLimit?: number;
  perDomainRateLimit?: Record<string, number>;
}

/**
 * Handles delivery of emails to destination servers
 */
class DeliveryManager {
  constructor(queue: DeliveryQueue, options: IDeliveryOptions);
  
  // Core delivery methods
  async start(): Promise<void>;
  async stop(): Promise<void>;
  async deliverMessage(item: QueueItem): Promise<DeliveryResult>;
  
  // Delivery management
  pauseDeliveries(): void;
  resumeDeliveries(): void;
  getDeliveryStats(): DeliveryStats;
  
  // Configure delivery behavior
  updateOptions(options: Partial<IDeliveryOptions>): void;
  setRateLimit(domain: string, limit: number): void;
  clearRateLimit(domain: string): void;
}

4.5 DcRouter SMTP Integration

interface ISmtpConfig {
  // Server configuration
  ports: number[];
  hostname: string;
  banner?: string;
  maxMessageSize?: number;
  
  // TLS configuration
  tls?: {
    certPath?: string;
    keyPath?: string;
    caPath?: string;
    minVersion?: string;
  };
  
  // Authentication
  auth?: {
    required?: boolean;
    methods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
    users?: Array<{username: string, password: string}>;
    ldapUrl?: string;
  };
  
  // Domain routing
  domainConfigs: Array<{
    domains: string[];
    targetIPs: string[];
    port?: number;
    useTls?: boolean;
    authentication?: {
      user?: string;
      pass?: string;
    };
    allowedIPs?: string[];
    rateLimits?: {
      maxMessagesPerMinute?: number;
      maxRecipientsPerMessage?: number;
    };
    addHeaders?: boolean;
    headerInfo?: Array<{
      name: string;
      value: string;
    }>;
    signDkim?: boolean;
    dkimOptions?: {
      domainName: string;
      keySelector: string;
      privateKey: string;
    };
  }>;
  
  // Default routing
  defaultServer: string;
  defaultPort?: number;
  useTls?: boolean;
  
  // Content scanning
  contentScanning?: boolean;
  scanners?: Array<{
    type: 'spam' | 'virus' | 'attachment';
    threshold?: number;
    action: 'tag' | 'reject';
    blockedExtensions?: string[];
  }>;
  
  // Message transformations
  transformations?: Array<{
    type: string;
    [key: string]: any;
  }>;
  
  // Queue settings
  queueStorage?: 'memory' | 'disk';
  persistentPath?: string;
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

// Extended IDcRouterOptions
interface IDcRouterOptions {
  // Existing options...
  
  // New SMTP configuration
  smtpConfig?: ISmtpConfig;
}

5. Implementation Phases

Phase 1: Core SMTP Server Setup

Implement the SmtpServer class
Set up TLS handling for both STARTTLS and implicit TLS
Create the basic connection validation logic
Implement authentication support
Build email receiving pipeline to accept complete messages
Create initial email parsing and storage

Phase 2: Mail Processing and Routing

Implement the EmailProcessor class
Create domain-based routing rules
Build email metadata extraction
Implement MIME parsing and handling
Create the transformation pipeline
Build header manipulation capabilities

Phase 3: Queue and Delivery System

Implement the DeliveryQueue class
Create persistent storage for queued messages
Build the retry and scheduling logic
Implement DeliveryManager with connection pooling
Create the delivery status tracking and reporting
Implement bounce handling and notification

Phase 4: Advanced Features and Integration

Integrate content scanning capabilities
Implement DKIM signing
Add rate limiting and traffic shaping
Create comprehensive metrics and logging
Build management APIs for runtime control
Implement full integration with DcRouter

Phase 5: Testing and Optimization

Create unit tests for all components
Implement integration tests for end-to-end verification
Perform load testing and optimize performance
Conduct security testing and hardening
Build documentation and examples

6. Technical Requirements

6.1 Dependencies

SMTP server library (smtp-server or similar)
Email parsing library (mailparser or similar)
MIME handling library
DKIM signing library
Queue management system (optional Redis support)
Cryptographic libraries for TLS and authentication

6.2 Performance Targets

Handle 1000+ concurrent SMTP connections
Process 100+ messages per second on standard hardware
Support message sizes up to 50MB
Maintain delivery queue of 100,000+ messages
Sub-second processing time for standard emails

6.3 Security Requirements

Full TLS support with modern cipher configurations
Authentication verification and rate limiting
Input validation for all SMTP commands
Secure storage of queued emails
Proper error handling to prevent information leakage
Access controls based on IP addresses and authentication

7. API Examples

7.1 Basic DcRouter SMTP Configuration

const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    maxMessageSize: 50 * 1024 * 1024, // 50MB
    
    // TLS configuration
    tls: {
      certPath: '/path/to/cert.pem',
      keyPath: '/path/to/key.pem'
    },
    
    // Domain routing
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true
      }
    ],
    
    // Default routing
    defaultServer: 'fallback-mail.example.com',
    defaultPort: 25,
    useTls: true,
    
    // Queue settings
    queueStorage: 'disk',
    persistentPath: '/var/mail/queue',
    maxRetries: 5
  }
});

7.2 Advanced Configuration with Processing

const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    // Basic settings
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    
    // Domain routing with advanced features
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true,
        // Add custom headers
        addHeaders: true,
        headerInfo: [
          { name: 'X-Processed-By', value: 'gateway' },
          { name: 'X-Scanned', value: 'true' }
        ],
        // Sign with DKIM
        signDkim: true,
        dkimOptions: {
          domainName: 'example.com',
          keySelector: 'mail',
          privateKey: '...'
        },
        // Rate limiting
        rateLimits: {
          maxMessagesPerMinute: 100,
          maxRecipientsPerMessage: 50
        }
      }
    ],
    
    // Content scanning
    contentScanning: true,
    scanners: [
      {
        type: 'spam',
        threshold: 5.0,
        action: 'tag'
      },
      {
        type: 'virus',
        action: 'reject'
      },
      {
        type: 'attachment',
        blockedExtensions: ['.exe', '.bat', '.vbs'],
        action: 'reject'
      }
    ],
    
    // Transformations
    transformations: [
      {
        type: 'addHeader',
        header: 'X-Gateway',
        value: 'DcRouter 1.0'
      },
      {
        type: 'dkimSign',
        domains: ['example.com']
      }
    ]
  }
});

8. Extensibility Points

8.1 Plugin Architecture

Custom content scanners
Custom transformation handlers
Routing rule extensions
Authentication providers
Queue storage backends

8.2 Event System

Connection events (connect, disconnect, error)
Message events (received, processed, queued, delivered)
Error events (delivery failure, processing error)
Performance events (queue size, processing time)
Security events (authentication failure, policy violation)

9. Migration Plan

9.1 From Simple Proxy to Store-and-Forward

Create compatibility layer for existing configurations
Implement graceful transition from connection proxy to full processing
Add configuration validation to ensure smooth migration
Create feature flags to enable advanced features incrementally
Provide documentation for migrating existing deployments

9.2 Backward Compatibility

Maintain support for basic proxy functionality
Provide simple configuration options for common use cases
Create migration utilities to update configuration formats
Support running in hybrid mode during transition

10. SmartProxy Integration

10.1 SmartProxy Configuration Handling

Implement comprehensive support for SmartProxy configuration
- Pass through all SmartProxy options directly
- Support all SmartProxy domain configuration features
- Ensure proper handling of SmartProxy events and callbacks
Create clear documentation on SmartProxy configuration:
- Explain how all SmartProxy features are available through DcRouter
- Document common usage patterns and examples
- Provide guidance on advanced SmartProxy configurations

10.2 SMTP Integration with SmartProxy

Ensure store-and-forward SMTP works alongside SmartProxy
- Document how SMTP ports are properly handled by SMTP processing
- Ensure no port conflicts between SmartProxy and SMTP server
- Create examples showing SmartProxy and SMTP working together
Document combined deployment models:
- HTTP/HTTPS traffic using SmartProxy configuration
- SMTP traffic using store-and-forward for advanced processing
- Examples for multi-service environments

11. Documentation Requirements

11.1 Code Documentation

Comprehensive JSDoc comments for all classes and methods
Interface definitions with detailed parameter descriptions
Example code snippets for common operations
Architecture documentation with component diagrams
Decision logs for key design choices

11.2 User Documentation

Getting started guide with configuration approach selection guidance
Complete configuration reference for both approaches
Deployment scenarios and examples
Troubleshooting guide
Performance tuning recommendations
Security best practices

11.3 Direct SmartProxy Configuration Guide

Detailed guide on using SmartProxy's domain configuration capabilities
Examples of complex routing scenarios with SmartProxy
Performance optimization tips for SmartProxy configurations
Security settings for SmartProxy deployments

22 KiB Raw Blame History