Philipp Kunz 5cf4c0f150 feat(dcrouter): Implement integrated DcRouter with comprehensive SmartProxy configuration, enhanced SMTP processing, and robust store‐and‐forward email routing

2025-05-07 23:45:19 +00:00

22 KiB

Raw Blame History

DcRouter SMTP Store-and-Forward Implementation Plan

Overview

This plan outlines the implementation of a store-and-forward SMTP proxy within DcRouter that receives emails, processes them, and forwards them to the appropriate destinations. This capability expands DcRouter beyond simple connection proxying to provide full control over email flow, including content inspection, transformation, and reliable delivery.

0. Configuration Approaches

0.1 Core SmartProxy Direct Configuration

DcRouter should leverage SmartProxy's configuration directly, exposing SmartProxy's full domain configuration options to give users maximum flexibility for all HTTP/HTTPS and TCP/SNI traffic:

interface IDcRouterOptions {
  // Direct SmartProxy configuration for general HTTP/HTTPS and TCP/SNI traffic
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // SMTP-specific configurations - can be used alongside smartProxyConfig
  // SMTP Store-and-forward configuration for advanced email processing
  smtpConfig?: ISmtpConfig;
  
  // Other DcRouter options...
}

This approach allows direct configuration of SmartProxy's powerful domain-based routing, giving full control over HTTP/HTTPS and SNI-based traffic:

const dcRouter = new DcRouter({
  // Direct SmartProxy configuration for HTTP/HTTPS traffic
  smartProxyConfig: {
    fromPort: 443,
    toPort: 8080,
    targetIP: '10.0.0.10',
    sniEnabled: true,
    acme: {
      port: 80,
      enabled: true,
      autoRenew: true,
      useProduction: true,
      renewThresholdDays: 30,
      accountEmail: 'admin@example.com'
    },
    globalPortRanges: [
      { from: 80, to: 80 },
      { from: 443, to: 443 }
    ],
    
    // SmartProxy's full domain configuration flexibility
    domainConfigs: [
      {
        domains: ['example.com', 'www.example.com'],
        allowedIPs: ['0.0.0.0/0'],
        blockedIPs: ['1.2.3.4/32'],
        targetIPs: ['10.0.0.10', '10.0.0.11'],
        portRanges: [
          { from: 80, to: 80 },
          { from: 443, to: 443 }
        ],
        connectionTimeout: 60000,
        useNetworkProxy: true
      },
      // Additional domain configurations...
    ],
    
    // Additional SmartProxy options...
  },
  
  // Email-specific configuration (complementary to smartProxyConfig)
  smtpConfig: {
    // Email handling configuration...
  },
  
  // Other DcRouter configuration...
}

0.2 Store-and-Forward SMTP Implementation

For advanced email handling, we'll build a complete store-and-forward SMTP system to work alongside the direct SmartProxy configuration. This provides full control over email processing while maintaining SmartProxy's flexibility for HTTP/HTTPS traffic:

1. Core Architecture

1.1 SMTP Server Implementation

Integrate an SMTP server library (like smtp-server) to accept incoming mail
- Created the SmtpServer class that initializes and manages the SMTP server instance
- Configured to listen on standard ports (25, 587, 465)
- Implemented TLS support (STARTTLS and implicit TLS)
- Added support for authentication methods (PLAIN, LOGIN, OAUTH2)
- Set up size limits and connection timeouts

1.2 Email Processing Pipeline

Create a modular processing pipeline for emails
- Built the EmailProcessor class that manages the processing workflow
- Implemented event-based architecture for extensible processing steps
- Created interfaces for each processing stage (metadata extraction, content scanning, routing, transformation)
- Added metrics and logging points throughout the pipeline

1.3 Queue Management

Develop a persistent queue system for email delivery
- Implemented DeliveryQueue class with in-memory queue for immediate delivery attempts
- Created persistent storage for delivery retry queue with file-based storage
- Built queue manager with scheduling capabilities
- Added transaction support to prevent message loss during crashes

1.4 Email Delivery System

Create a robust delivery system for outbound email
- Implemented DeliverySystem class for outbound SMTP connections
- Added retry logic with configurable exponential backoff
- Created delivery status tracking and notifications via events
- Set up initial bounce handling and processing

2. Email Processing Features

2.1 Routing and Forwarding

Implement flexible email routing based on various criteria
- Created domain-based routing rules in EmailProcessor
- Added support for pattern matching for domains (exact match, wildcard)
- Implemented recipient-based routing
- Added support for routing across multiple target servers
- Added initial failover support for high availability

2.2 Content Inspection

Develop content inspection capabilities
- Added MIME parsing and content extraction using mailparser
- Implemented attachment scanning and filtering based on extensions
- Created plugin architecture for content analysis
- Added integration points for external scanners (spam, virus)
- Implemented policy enforcement based on content scan results

2.3 Email Transformation

Create tools for modifying emails during transit
- Implemented header addition capabilities
- Added DKIM signing capability placeholder
- Created framework for email transformations
- Added attachment handling capability
- Implemented support for adding compliance information

2.4 Rate Limiting and Traffic Control

Build rate limiting controls
- Implemented per-domain rate limits
- Added support for configurable rate limiting thresholds
- Created quota enforcement with domain-based configuration
- Added event system for rate limit notifications

3. Integration with DcRouter

3.1 Configuration Interface

Extend DcRouter's configuration schema
- Created comprehensive SMTP configuration section in IDcRouterOptions
- Defined interfaces for each SMTP feature set
- Added validation with defaults for configuration values
- Implemented sensible defaults for all configuration options
- Added detailed documentation in code comments

3.2 Management API

Develop management APIs for runtime control
- Created methods to update configuration without restart
- Implemented queue management functions (pause, resume, inspect)
- Added status reporting through events
- Created configuration update methods
- Implemented graceful shutdown capabilities

3.3 Metrics and Logging

Implement metrics gathering
- Created counters for messages processed, delivered, and failed
- Added tracking for processing stages
- Implemented detailed logging
- Added message IDs for tracking through the system

4. Detailed Component Specifications

4.0 DcRouter Configuration Extension

export interface IDcRouterOptions {
  // Core configuration options
  
  // Direct SmartProxy configuration - gives full control over all TCP/SNI traffic
  // including HTTP, HTTPS, and any other TCP-based protocol
  smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
  
  // For backward compatibility and simplified HTTP configuration
  httpDomainRoutes?: IDomainRoutingConfig[];
  
  // SMTP store-and-forward processing - works alongside smartProxyConfig
  // This is for advanced email handling like content inspection
  smtpConfig?: ISmtpConfig;
  
  // Shared configurations
  tls?: {
    contactEmail: string;
    domain?: string;
    certPath?: string;
    keyPath?: string;
  };
  
  // Other DcRouter options
  dnsServerConfig?: plugins.smartdns.IDnsServerOptions;
  mtaConfig?: IMtaConfig; 
  mtaServiceInstance?: MtaService;
}

4.1 SmtpServer Class

interface ISmtpServerOptions {
  // Base server options
  ports: number[];
  hostname: string;
  banner?: string;
  
  // Authentication options
  authMethods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
  requireAuth?: boolean;
  
  // TLS options
  tls?: {
    key?: string | Buffer;
    cert?: string | Buffer;
    ca?: string | Buffer | Array<string | Buffer>;
    ciphers?: string;
    minVersion?: string;
  };
  
  // Limits
  maxMessageSize?: number;
  maxClients?: number;
  maxConnections?: number;
  
  // Connection options
  connectionTimeout?: number;
  socketTimeout?: number;
}

/**
 * Manages the SMTP server for receiving emails
 */
class SmtpServer {
  constructor(options: ISmtpServerOptions);
  
  // Start and stop the server
  start(): Promise<void>;
  stop(): Promise<void>;
  
  // Event handlers
  onConnect(handler: (session: Session, callback: (err?: Error) => void) => void): void;
  onAuth(handler: (auth: AuthObject, session: Session, callback: (err?: Error, user?: UserInfo) => void) => void): void;
  onMailFrom(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onRcptTo(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
  onData(handler: (stream: Readable, session: Session, callback: (err?: Error) => void) => void): void;
  
  // Check email size before accepting data
  checkMessageSize(size: number): boolean;
  
  // Configuration updates
  updateOptions(options: Partial<ISmtpServerOptions>): void;
  
  // Server stats
  getStats(): IServerStats;
}

4.2 EmailProcessor Class

interface IEmailProcessorOptions {
  // Processing options
  maxParallelProcessing?: number;
  processingTimeout?: number;
  
  // Feature flags
  contentScanning?: boolean;
  headerProcessing?: boolean;
  dkimSigning?: boolean;
  
  // Processing rules
  scanners?: IScannerConfig[];
  transformations?: ITransformConfig[];
  
  // Routing rules
  routingRules?: IRoutingRule[];
  defaultServer?: string;
  defaultPort?: number;
}

/**
 * Handles all email processing steps
 */
class EmailProcessor {
  constructor(options: IEmailProcessorOptions);
  
  // Main processing method
  async processEmail(message: ParsedMail, session: Session): Promise<ProcessingResult>;
  
  // Individual processing steps
  async extractMetadata(message: ParsedMail): Promise<EmailMetadata>;
  async determineRouting(metadata: EmailMetadata): Promise<RoutingDecision>;
  async scanContent(message: ParsedMail): Promise<ScanResult>;
  async applyTransformations(message: ParsedMail): Promise<ParsedMail>;
  
  // Update processor configuration
  updateOptions(options: Partial<IEmailProcessorOptions>): void;
  
  // Manage processing plugins
  addScanner(scanner: IScanner): void;
  addTransformation(transformation: ITransformation): void;
  addRoutingRule(rule: IRoutingRule): void;
}

4.3 DeliveryQueue Class

interface IQueueOptions {
  // Storage options
  storageType: 'memory' | 'disk' | 'redis';
  storagePath?: string;
  redisUrl?: string;
  
  // Queue behavior
  checkInterval?: number;
  maxQueueSize?: number;
  maxPerDestination?: number;
  
  // Delivery attempts
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

/**
 * Manages the queue of messages waiting for delivery
 */
class DeliveryQueue {
  constructor(options: IQueueOptions);
  
  // Queue operations
  async enqueue(item: QueueItem): Promise<string>;
  async dequeue(id: string): Promise<QueueItem | null>;
  async update(id: string, updates: Partial<QueueItem>): Promise<boolean>;
  async getNext(count?: number): Promise<QueueItem[]>;
  
  // Query methods
  async getByStatus(status: QueueItemStatus): Promise<QueueItem[]>;
  async getByDestination(server: string): Promise<QueueItem[]>;
  async getItemCount(): Promise<number>;
  
  // Queue maintenance
  async purgeExpired(): Promise<number>;
  async purgeAll(): Promise<number>;
  
  // Persistence
  async load(): Promise<void>;
  async save(): Promise<void>;
  
  // Processing control
  pause(): void;
  resume(): void;
  isProcessing(): boolean;
}

4.4 DeliveryManager Class

interface IDeliveryOptions {
  // Connection options
  connectionPoolSize?: number;
  socketTimeout?: number;
  
  // Delivery behavior
  concurrentDeliveries?: number;
  sendTimeout?: number;
  
  // TLS options
  verifyCertificates?: boolean;
  tlsMinVersion?: string;
  
  // Rate limiting
  globalRateLimit?: number;
  perServerRateLimit?: number;
  perDomainRateLimit?: Record<string, number>;
}

/**
 * Handles delivery of emails to destination servers
 */
class DeliveryManager {
  constructor(queue: DeliveryQueue, options: IDeliveryOptions);
  
  // Core delivery methods
  async start(): Promise<void>;
  async stop(): Promise<void>;
  async deliverMessage(item: QueueItem): Promise<DeliveryResult>;
  
  // Delivery management
  pauseDeliveries(): void;
  resumeDeliveries(): void;
  getDeliveryStats(): DeliveryStats;
  
  // Configure delivery behavior
  updateOptions(options: Partial<IDeliveryOptions>): void;
  setRateLimit(domain: string, limit: number): void;
  clearRateLimit(domain: string): void;
}

4.5 DcRouter SMTP Integration

interface ISmtpConfig {
  // Server configuration
  ports: number[];
  hostname: string;
  banner?: string;
  maxMessageSize?: number;
  
  // TLS configuration
  tls?: {
    certPath?: string;
    keyPath?: string;
    caPath?: string;
    minVersion?: string;
  };
  
  // Authentication
  auth?: {
    required?: boolean;
    methods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
    users?: Array<{username: string, password: string}>;
    ldapUrl?: string;
  };
  
  // Domain routing
  domainConfigs: Array<{
    domains: string[];
    targetIPs: string[];
    port?: number;
    useTls?: boolean;
    authentication?: {
      user?: string;
      pass?: string;
    };
    allowedIPs?: string[];
    rateLimits?: {
      maxMessagesPerMinute?: number;
      maxRecipientsPerMessage?: number;
    };
    addHeaders?: boolean;
    headerInfo?: Array<{
      name: string;
      value: string;
    }>;
    signDkim?: boolean;
    dkimOptions?: {
      domainName: string;
      keySelector: string;
      privateKey: string;
    };
  }>;
  
  // Default routing
  defaultServer: string;
  defaultPort?: number;
  useTls?: boolean;
  
  // Content scanning
  contentScanning?: boolean;
  scanners?: Array<{
    type: 'spam' | 'virus' | 'attachment';
    threshold?: number;
    action: 'tag' | 'reject';
    blockedExtensions?: string[];
  }>;
  
  // Message transformations
  transformations?: Array<{
    type: string;
    [key: string]: any;
  }>;
  
  // Queue settings
  queueStorage?: 'memory' | 'disk';
  persistentPath?: string;
  maxRetries?: number;
  baseRetryDelay?: number;
  maxRetryDelay?: number;
}

// Extended IDcRouterOptions
interface IDcRouterOptions {
  // Existing options...
  
  // New SMTP configuration
  smtpConfig?: ISmtpConfig;
}

5. Implementation Phases

Phase 1: Core SMTP Server Setup

Implement the SmtpServer class
Set up TLS handling for both STARTTLS and implicit TLS
Create the basic connection validation logic
Implement authentication support
Build email receiving pipeline to accept complete messages
Create initial email parsing and storage

Phase 2: Mail Processing and Routing

Implement the EmailProcessor class
Create domain-based routing rules
Build email metadata extraction
Implement MIME parsing and handling
Create the transformation pipeline
Build header manipulation capabilities

Phase 3: Queue and Delivery System

Implement the DeliveryQueue class
Create persistent storage for queued messages
Build the retry and scheduling logic
Implement DeliveryManager with connection pooling
Create the delivery status tracking and reporting
Implement bounce handling and notification

Phase 4: Advanced Features and Integration

Integrate content scanning capabilities
Implement DKIM signing
Add rate limiting and traffic shaping
Create comprehensive metrics and logging
Build management APIs for runtime control
Implement full integration with DcRouter

Phase 5: Testing and Optimization

Create unit tests for all components
Implement integration tests for end-to-end verification
Perform load testing and optimize performance
Conduct security testing and hardening
Build documentation and examples

6. Technical Requirements

6.1 Dependencies

SMTP server library (smtp-server or similar)
Email parsing library (mailparser or similar)
MIME handling library
DKIM signing library
Queue management system (optional Redis support)
Cryptographic libraries for TLS and authentication

6.2 Performance Targets

Handle 1000+ concurrent SMTP connections
Process 100+ messages per second on standard hardware
Support message sizes up to 50MB
Maintain delivery queue of 100,000+ messages
Sub-second processing time for standard emails

6.3 Security Requirements

Full TLS support with modern cipher configurations
Authentication verification and rate limiting
Input validation for all SMTP commands
Secure storage of queued emails
Proper error handling to prevent information leakage
Access controls based on IP addresses and authentication

7. API Examples

7.1 Basic DcRouter SMTP Configuration

const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    maxMessageSize: 50 * 1024 * 1024, // 50MB
    
    // TLS configuration
    tls: {
      certPath: '/path/to/cert.pem',
      keyPath: '/path/to/key.pem'
    },
    
    // Domain routing
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true
      }
    ],
    
    // Default routing
    defaultServer: 'fallback-mail.example.com',
    defaultPort: 25,
    useTls: true,
    
    // Queue settings
    queueStorage: 'disk',
    persistentPath: '/var/mail/queue',
    maxRetries: 5
  }
});

7.2 Advanced Configuration with Processing

const dcRouter = new DcRouter({
  // HTTP configuration...
  
  smtpConfig: {
    // Basic settings
    ports: [25, 587, 465],
    hostname: 'mail.example.com',
    
    // Domain routing with advanced features
    domainConfigs: [
      {
        domains: ['example.com', '*.example.com'],
        targetIPs: ['mail1.example.com', 'mail2.example.com'],
        port: 25,
        useTls: true,
        // Add custom headers
        addHeaders: true,
        headerInfo: [
          { name: 'X-Processed-By', value: 'gateway' },
          { name: 'X-Scanned', value: 'true' }
        ],
        // Sign with DKIM
        signDkim: true,
        dkimOptions: {
          domainName: 'example.com',
          keySelector: 'mail',
          privateKey: '...'
        },
        // Rate limiting
        rateLimits: {
          maxMessagesPerMinute: 100,
          maxRecipientsPerMessage: 50
        }
      }
    ],
    
    // Content scanning
    contentScanning: true,
    scanners: [
      {
        type: 'spam',
        threshold: 5.0,
        action: 'tag'
      },
      {
        type: 'virus',
        action: 'reject'
      },
      {
        type: 'attachment',
        blockedExtensions: ['.exe', '.bat', '.vbs'],
        action: 'reject'
      }
    ],
    
    // Transformations
    transformations: [
      {
        type: 'addHeader',
        header: 'X-Gateway',
        value: 'DcRouter 1.0'
      },
      {
        type: 'dkimSign',
        domains: ['example.com']
      }
    ]
  }
});

8. Extensibility Points

8.1 Plugin Architecture

Custom content scanners
Custom transformation handlers
Routing rule extensions
Authentication providers
Queue storage backends

8.2 Event System

Connection events (connect, disconnect, error)
Message events (received, processed, queued, delivered)
Error events (delivery failure, processing error)
Performance events (queue size, processing time)
Security events (authentication failure, policy violation)

9. Migration Plan

9.1 From Simple Proxy to Store-and-Forward

Create compatibility layer for existing configurations
Implement graceful transition from connection proxy to full processing
Add configuration validation to ensure smooth migration
Create feature flags to enable advanced features incrementally
Provide documentation for migrating existing deployments

9.2 Backward Compatibility

Maintain support for basic proxy functionality
Provide simple configuration options for common use cases
Create migration utilities to update configuration formats
Support running in hybrid mode during transition

10. SmartProxy Integration

10.1 SmartProxy Configuration Handling

Implement comprehensive support for SmartProxy configuration
- Passed through all SmartProxy options directly in DcRouter's configuration
- Added support for all SmartProxy domain configuration features
- Implemented proper handling of SmartProxy events and callbacks
Added documentation on SmartProxy configuration:
- Documented how all SmartProxy features are available through DcRouter
- Added examples of different configuration approaches
- Provided guidance in code comments

10.2 SMTP Integration with SmartProxy

Ensured store-and-forward SMTP works alongside SmartProxy
- Handled SMTP ports separately from HTTP/HTTPS ports
- Prevented port conflicts between SmartProxy and SMTP server
- Created code structure showing SmartProxy and SMTP working together
Implemented combined usage model:
- HTTP/HTTPS traffic using SmartProxy configuration
- SMTP traffic using store-and-forward for advanced processing
- Added support for multi-service environments

11. Documentation Requirements

11.1 Code Documentation

Comprehensive JSDoc comments for all classes and methods
Interface definitions with detailed parameter descriptions
Example code snippets for common operations
Architecture documentation with component diagrams
Decision logs for key design choices

11.2 User Documentation

Getting started guide with configuration approach selection guidance
Complete configuration reference for both approaches
Deployment scenarios and examples
Troubleshooting guide
Performance tuning recommendations
Security best practices

11.3 Direct SmartProxy Configuration Guide

Detailed guide on using SmartProxy's domain configuration capabilities
Examples of complex routing scenarios with SmartProxy
Performance optimization tips for SmartProxy configurations
Security settings for SmartProxy deployments

22 KiB Raw Blame History