update plan
This commit is contained in:
parent
1528d29b0d
commit
04b7552b34
886
readme.plan.md
886
readme.plan.md
@ -1,172 +1,756 @@
|
|||||||
# DcRouter Improvement Plan
|
# DcRouter SMTP Store-and-Forward Implementation Plan
|
||||||
|
|
||||||
## Objective
|
## Overview
|
||||||
Create a new version of DcRouter that doesn't rely on SzPlatformService but instead uses SmartProxy and the complete email stack directly. This will make DcRouter more modular, lightweight, and capable of SMTP forwarding with the email stack.
|
This plan outlines the implementation of a store-and-forward SMTP proxy within DcRouter that receives emails, processes them, and forwards them to the appropriate destinations. This capability expands DcRouter beyond simple connection proxying to provide full control over email flow, including content inspection, transformation, and reliable delivery.
|
||||||
|
|
||||||
## 1. Core Architecture Changes
|
## 0. Configuration Approaches
|
||||||
|
|
||||||
### 1.1 Remove SzPlatformService Dependency
|
### 0.1 Core SmartProxy Direct Configuration
|
||||||
- [x] Remove the `platformServiceInstance` option from `IDcRouterOptions`
|
DcRouter should leverage SmartProxy's configuration directly, exposing SmartProxy's full domain configuration options to give users maximum flexibility for all HTTP/HTTPS and TCP/SNI traffic:
|
||||||
- Update `classes.dcrouter.ts` to remove platformServiceInstance from the options interface
|
|
||||||
- Create a new utility class `DcRouterEnvironment` to handle environment variables directly using qenv
|
|
||||||
- Replace all uses of SzDcRouterConnector with direct calls to DcRouterEnvironment
|
|
||||||
|
|
||||||
### 1.2 Direct Integration with Email Stack
|
```typescript
|
||||||
- [x] Add options for direct MtaService integration
|
interface IDcRouterOptions {
|
||||||
- Enhance the existing mtaConfig/mtaServiceInstance options to include all necessary MTA configuration
|
// Direct SmartProxy configuration for general HTTP/HTTPS and TCP/SNI traffic
|
||||||
- Add new options for email forwarding capabilities that will be passed to the MTA service
|
smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
|
||||||
- Implement initializers for setting up MTA with or without existing instances
|
|
||||||
- [x] Create a new SMTP forwarding configuration interface
|
|
||||||
- Create `ISmtpForwardingConfig` interface with fields for destination domains, routing rules, and authentication
|
|
||||||
- Implement domain-to-server mapping for routing emails to appropriate SMTP servers
|
|
||||||
- Add options for SMTP authentication methods (PLAIN, LOGIN, OAUTH2)
|
|
||||||
- [x] Implement proper connection between SmartProxy and MTA service
|
|
||||||
- Update `configureSmtpProxy()` method to create bidirectional communication with MTA service
|
|
||||||
- Implement proxy protocol support for preserving client IP addresses during SMTP forwarding
|
|
||||||
- Create listener for MTA status changes to adjust proxy settings dynamically
|
|
||||||
|
|
||||||
## 2. SmartProxy Configuration
|
// SMTP-specific configurations - can be used alongside smartProxyConfig
|
||||||
|
// SMTP Store-and-forward configuration for advanced email processing
|
||||||
|
smtpConfig?: ISmtpConfig;
|
||||||
|
|
||||||
### 2.1 Enhanced SmartProxy Integration
|
// Other DcRouter options...
|
||||||
- [x] Update `SmartProxy` configuration for better TCP/SNI handling
|
}
|
||||||
- Modify initialization to support more advanced SNI-based routing decisions
|
```
|
||||||
- [x] Add specific configuration for SMTP ports (25, 465, 587)
|
|
||||||
- Create a `SmtpPortConfig` class to manage SMTP-specific port settings
|
|
||||||
- Add TLS termination options specific to SMTP protocols (STARTTLS vs. implicit TLS)
|
|
||||||
- Implement connection rate limiting and concurrent connection management for SMTP ports
|
|
||||||
|
|
||||||
### 2.2 Routing Configuration
|
This approach allows direct configuration of SmartProxy's powerful domain-based routing, giving full control over HTTP/HTTPS and SNI-based traffic:
|
||||||
- [x] Allow domain-based routing for email traffic
|
|
||||||
- Add domain matching patterns with wildcard support for inbound email routing
|
|
||||||
- Implement sender domain-based routing for outbound emails
|
|
||||||
- Create domain groups for applying consistent rules across related domains
|
|
||||||
- [x] Implement IP-based allow/block lists for advanced filtering
|
|
||||||
- Develop IP range and CIDR notation support for filtering
|
|
||||||
- Create separate lists for inbound and outbound connections
|
|
||||||
- Implement geo-based filtering using IP geolocation
|
|
||||||
|
|
||||||
## 3. SMTP Forwarding Functionality
|
```typescript
|
||||||
|
const dcRouter = new DcRouter({
|
||||||
|
// Direct SmartProxy configuration for HTTP/HTTPS traffic
|
||||||
|
smartProxyConfig: {
|
||||||
|
fromPort: 443,
|
||||||
|
toPort: 8080,
|
||||||
|
targetIP: '10.0.0.10',
|
||||||
|
sniEnabled: true,
|
||||||
|
acme: {
|
||||||
|
port: 80,
|
||||||
|
enabled: true,
|
||||||
|
autoRenew: true,
|
||||||
|
useProduction: true,
|
||||||
|
renewThresholdDays: 30,
|
||||||
|
accountEmail: 'admin@example.com'
|
||||||
|
},
|
||||||
|
globalPortRanges: [
|
||||||
|
{ from: 80, to: 80 },
|
||||||
|
{ from: 443, to: 443 }
|
||||||
|
],
|
||||||
|
|
||||||
### 3.1 SMTP Routing Engine
|
// SmartProxy's full domain configuration flexibility
|
||||||
- [x] Enhance the SMTP rule engine to support advanced forwarding scenarios
|
domainConfigs: [
|
||||||
- Extend email routing capabilities with SmartProxy configuration
|
{
|
||||||
- Add context information to routing for making informed routing decisions
|
domains: ['example.com', 'www.example.com'],
|
||||||
- Implement domain-based routing for traffic management
|
allowedIPs: ['0.0.0.0/0'],
|
||||||
- [x] Create efficient routing for common email patterns
|
blockedIPs: ['1.2.3.4/32'],
|
||||||
- Develop email forwarding configuration for common use cases
|
targetIPs: ['10.0.0.10', '10.0.0.11'],
|
||||||
- Implement domain-to-server mapping for email routing
|
portRanges: [
|
||||||
- Create simple but effective routing mechanisms
|
{ from: 80, to: 80 },
|
||||||
- [x] Implement per-domain routing configuration
|
{ from: 443, to: 443 }
|
||||||
- Create domain configuration support in SmtpForwardingConfig
|
],
|
||||||
- Implement dynamic updating of domain routes
|
connectionTimeout: 60000,
|
||||||
- Add domain-level connection handling
|
useNetworkProxy: true
|
||||||
|
},
|
||||||
|
// Additional domain configurations...
|
||||||
|
],
|
||||||
|
|
||||||
### 3.2 MTA Integration
|
// Additional SmartProxy options...
|
||||||
- [x] Configure MTA service for use with DcRouter
|
},
|
||||||
- Extend DcRouter to work with existing MTA configuration
|
|
||||||
- Implement proper MTA service initialization and startup
|
|
||||||
- Create clean integration between DcRouter and MTA
|
|
||||||
- [x] Implement SMTP forwarding as alternative to MTA
|
|
||||||
- Add SMTP forwarding configuration for simpler deployments
|
|
||||||
- Implement SmartProxy configuration for SMTP traffic
|
|
||||||
- Create clean separation between MTA and forwarding modes
|
|
||||||
- [x] Maintain email traffic integrity
|
|
||||||
- Ensure proper handling of connections between services
|
|
||||||
- Implement source IP preservation for proper tracking
|
|
||||||
- Create configuration options for security settings
|
|
||||||
|
|
||||||
## 4. Implementation Tasks
|
// Email-specific configuration (complementary to smartProxyConfig)
|
||||||
|
smtpConfig: {
|
||||||
|
// Email handling configuration...
|
||||||
|
},
|
||||||
|
|
||||||
### 4.1 Create New Classes
|
// Other DcRouter configuration...
|
||||||
- [x] Create utility classes to handle configuration
|
}
|
||||||
- Implement `DcRouterEnvironment` for environment variable access
|
```
|
||||||
- Create `SmtpPortConfig` for managing SMTP port settings
|
|
||||||
- Implement `EmailDomainRouter` for email domain routing
|
|
||||||
- [x] Develop SMTP-specific functionality
|
|
||||||
- Create specialized configuration for SMTP traffic
|
|
||||||
- Implement port configuration for different SMTP protocols
|
|
||||||
- Add TLS options handling for SMTP ports
|
|
||||||
- [x] Implement certificate management
|
|
||||||
- Utilize SmartProxy's Port80Handler for ACME certificate management
|
|
||||||
- Add certificate application to NetworkProxy
|
|
||||||
- Create certificate event handling
|
|
||||||
|
|
||||||
### 4.2 Update Existing Components
|
### 0.2 Store-and-Forward SMTP Implementation
|
||||||
- [x] Refactor `DcRouter` class to remove SzPlatformService dependencies
|
For advanced email handling, we'll build a complete store-and-forward SMTP system to work alongside the direct SmartProxy configuration. This provides full control over email processing while maintaining SmartProxy's flexibility for HTTP/HTTPS traffic:
|
||||||
- Remove all direct references to SzPlatformService
|
|
||||||
- Update constructor to use new configuration system
|
|
||||||
- Refactor initialization logic to work independently
|
|
||||||
- [x] Update certificate handling to use SmartACME directly
|
|
||||||
- Implement Port80Handler for ACME certificate management
|
|
||||||
- Add automatic certificate renewal with event handling
|
|
||||||
- Apply certificates to appropriate services
|
|
||||||
- [x] Enhance SmartProxy configuration for better SMTP support
|
|
||||||
- Implement separate SmartProxy instances for different protocols
|
|
||||||
- Add TLS settings for different SMTP ports (STARTTLS vs. implicit TLS)
|
|
||||||
- Create clean service lifecycle management
|
|
||||||
|
|
||||||
### 4.3 Configuration Interface
|
## 1. Core Architecture
|
||||||
- [x] Create a clean, declarative configuration interface
|
|
||||||
- Design structured TypeScript interfaces for all configuration options
|
|
||||||
- Implement simple, focused configuration objects
|
|
||||||
- Create clean separation between different component configurations
|
|
||||||
- [x] Support environment variables and programmatic configuration
|
|
||||||
- Create DcRouterEnvironment for environment variable access
|
|
||||||
- Implement environment variable caching for better performance
|
|
||||||
- Allow programmatic configuration updates
|
|
||||||
- [x] Implement well-defined configuration APIs
|
|
||||||
- Add typed interfaces for all configuration options
|
|
||||||
- Create clear documentation in interface comments
|
|
||||||
- Implement runtime configuration updating
|
|
||||||
|
|
||||||
## 5. Testing and Documentation
|
### 1.1 SMTP Server Implementation
|
||||||
|
- [ ] Integrate an SMTP server library (like `smtp-server`) to accept incoming mail
|
||||||
|
- Create a wrapper class that initializes and manages the SMTP server instance
|
||||||
|
- Configure to listen on standard ports (25, 587, 465)
|
||||||
|
- Implement TLS support (STARTTLS and implicit TLS)
|
||||||
|
- Support authentication methods (PLAIN, LOGIN, OAUTH2)
|
||||||
|
- Set up size limits and connection timeouts
|
||||||
|
|
||||||
### 5.1 Code Implementation
|
### 1.2 Email Processing Pipeline
|
||||||
- [x] Implement core components
|
- [ ] Create a modular processing pipeline for emails
|
||||||
- Create new classes for configuration and domain routing
|
- Build the core pipeline executor that manages the processing workflow
|
||||||
- Update existing DcRouter with new functionality
|
- Implement plugin architecture for extensible processing steps
|
||||||
- Implement environment variable handling
|
- Create interfaces for each processing stage
|
||||||
- [x] Implement SMTP functionality
|
- Add metrics and logging points throughout the pipeline
|
||||||
- Add SMTP forwarding configuration
|
|
||||||
- Implement port-specific settings
|
|
||||||
- Create domain-based email routing
|
|
||||||
- [x] Implement HTTP/HTTPS functionality
|
|
||||||
- Add NetworkProxy integration
|
|
||||||
- Implement certificate management
|
|
||||||
- Create domain-based HTTP routing
|
|
||||||
|
|
||||||
### 5.2 Quality and Performance
|
### 1.3 Queue Management
|
||||||
- [x] Ensure code quality
|
- [ ] Develop a persistent queue system for email delivery
|
||||||
- Fix all TypeScript errors
|
- Implement in-memory queue for immediate delivery attempts
|
||||||
- Implement clean interfaces
|
- Create persistent storage for delivery retry queue
|
||||||
- Create well-documented code
|
- Build queue manager with priority, sorting, and scheduling capabilities
|
||||||
- [x] Optimize for performance
|
- Add transaction support to prevent message loss during crashes
|
||||||
- Implement parallel service shutdown
|
|
||||||
- Use environment variable caching
|
|
||||||
- Create efficient routing lookups
|
|
||||||
- [x] Maintain compatibility
|
|
||||||
- Ensure backward compatibility where possible
|
|
||||||
- Create clean extension points
|
|
||||||
- Maintain consistent APIs
|
|
||||||
|
|
||||||
## 6. Future Enhancements (Pending)
|
### 1.4 Email Delivery System
|
||||||
|
- [ ] Create a robust delivery system for outbound email
|
||||||
|
- Implement connection pool for outbound SMTP connections
|
||||||
|
- Add retry logic with configurable exponential backoff
|
||||||
|
- Create delivery status tracking and notification
|
||||||
|
- Set up bounce handling and processing
|
||||||
|
|
||||||
### 6.1 Testing
|
## 2. Email Processing Features
|
||||||
|
|
||||||
|
### 2.1 Routing and Forwarding
|
||||||
|
- [ ] Implement flexible email routing based on various criteria
|
||||||
|
- Create domain-based routing rules
|
||||||
|
- Support pattern matching for domains (exact match, wildcard)
|
||||||
|
- Implement sender and recipient-based routing
|
||||||
|
- Add load balancing across multiple target servers
|
||||||
|
- Create failover support for high availability
|
||||||
|
|
||||||
|
### 2.2 Content Inspection
|
||||||
|
- [ ] Develop content inspection capabilities
|
||||||
|
- Add MIME parsing and content extraction
|
||||||
|
- Implement attachment scanning and filtering
|
||||||
|
- Create text content analysis framework
|
||||||
|
- Add integration points for external scanners (spam, virus)
|
||||||
|
- Implement policy enforcement based on content
|
||||||
|
|
||||||
|
### 2.3 Email Transformation
|
||||||
|
- [ ] Create tools for modifying emails during transit
|
||||||
|
- Implement header addition, modification, and removal
|
||||||
|
- Add DKIM signing capability
|
||||||
|
- Support email rewriting (address, content)
|
||||||
|
- Create tools for attachment handling (remove, sanitize)
|
||||||
|
- Support for adding tracking or compliance information
|
||||||
|
|
||||||
|
### 2.4 Rate Limiting and Traffic Control
|
||||||
|
- [ ] Build advanced rate limiting controls
|
||||||
|
- Implement per-IP, per-sender, and per-domain rate limits
|
||||||
|
- Create tiered rate limiting with different thresholds
|
||||||
|
- Add traffic shaping capabilities to prevent spikes
|
||||||
|
- Implement quota enforcement with customizable time windows
|
||||||
|
- Create alert system for exceeding thresholds
|
||||||
|
|
||||||
|
## 3. Integration with DcRouter
|
||||||
|
|
||||||
|
### 3.1 Configuration Interface
|
||||||
|
- [ ] Extend DcRouter's configuration schema
|
||||||
|
- Create comprehensive SMTP configuration section in IDcRouterOptions
|
||||||
|
- Define interfaces for each SMTP feature set
|
||||||
|
- Add validation rules for configuration values
|
||||||
|
- Implement defaults for all configuration options
|
||||||
|
- Document configuration parameters thoroughly
|
||||||
|
|
||||||
|
### 3.2 Management API
|
||||||
|
- [ ] Develop management APIs for runtime control
|
||||||
|
- Create methods to update routing rules without restart
|
||||||
|
- Implement queue management functions (pause, resume, inspect)
|
||||||
|
- Add real-time monitoring endpoints
|
||||||
|
- Create rate limit adjustment capabilities
|
||||||
|
- Implement logging level controls
|
||||||
|
|
||||||
|
### 3.3 Metrics and Logging
|
||||||
|
- [ ] Implement comprehensive metrics gathering
|
||||||
|
- Create counters for messages processed, delivered, and failed
|
||||||
|
- Add timing metrics for processing stages
|
||||||
|
- Implement detailed logging with configurable levels
|
||||||
|
- Create structured log output for easier parsing
|
||||||
|
- Add correlation IDs for tracking messages through the system
|
||||||
|
|
||||||
|
## 4. Detailed Component Specifications
|
||||||
|
|
||||||
|
### 4.0 DcRouter Configuration Extension
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
export interface IDcRouterOptions {
|
||||||
|
// Core configuration options
|
||||||
|
|
||||||
|
// Direct SmartProxy configuration - gives full control over all TCP/SNI traffic
|
||||||
|
// including HTTP, HTTPS, and any other TCP-based protocol
|
||||||
|
smartProxyConfig?: plugins.smartproxy.ISmartProxyOptions;
|
||||||
|
|
||||||
|
// For backward compatibility and simplified HTTP configuration
|
||||||
|
httpDomainRoutes?: IDomainRoutingConfig[];
|
||||||
|
|
||||||
|
// SMTP store-and-forward processing - works alongside smartProxyConfig
|
||||||
|
// This is for advanced email handling like content inspection
|
||||||
|
smtpConfig?: ISmtpConfig;
|
||||||
|
|
||||||
|
// Shared configurations
|
||||||
|
tls?: {
|
||||||
|
contactEmail: string;
|
||||||
|
domain?: string;
|
||||||
|
certPath?: string;
|
||||||
|
keyPath?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Other DcRouter options
|
||||||
|
dnsServerConfig?: plugins.smartdns.IDnsServerOptions;
|
||||||
|
mtaConfig?: IMtaConfig;
|
||||||
|
mtaServiceInstance?: MtaService;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.1 SmtpServer Class
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ISmtpServerOptions {
|
||||||
|
// Base server options
|
||||||
|
ports: number[];
|
||||||
|
hostname: string;
|
||||||
|
banner?: string;
|
||||||
|
|
||||||
|
// Authentication options
|
||||||
|
authMethods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
|
||||||
|
requireAuth?: boolean;
|
||||||
|
|
||||||
|
// TLS options
|
||||||
|
tls?: {
|
||||||
|
key?: string | Buffer;
|
||||||
|
cert?: string | Buffer;
|
||||||
|
ca?: string | Buffer | Array<string | Buffer>;
|
||||||
|
ciphers?: string;
|
||||||
|
minVersion?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Limits
|
||||||
|
maxMessageSize?: number;
|
||||||
|
maxClients?: number;
|
||||||
|
maxConnections?: number;
|
||||||
|
|
||||||
|
// Connection options
|
||||||
|
connectionTimeout?: number;
|
||||||
|
socketTimeout?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Manages the SMTP server for receiving emails
|
||||||
|
*/
|
||||||
|
class SmtpServer {
|
||||||
|
constructor(options: ISmtpServerOptions);
|
||||||
|
|
||||||
|
// Start and stop the server
|
||||||
|
start(): Promise<void>;
|
||||||
|
stop(): Promise<void>;
|
||||||
|
|
||||||
|
// Event handlers
|
||||||
|
onConnect(handler: (session: Session, callback: (err?: Error) => void) => void): void;
|
||||||
|
onAuth(handler: (auth: AuthObject, session: Session, callback: (err?: Error, user?: UserInfo) => void) => void): void;
|
||||||
|
onMailFrom(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
|
||||||
|
onRcptTo(handler: (address: Address, session: Session, callback: (err?: Error) => void) => void): void;
|
||||||
|
onData(handler: (stream: Readable, session: Session, callback: (err?: Error) => void) => void): void;
|
||||||
|
|
||||||
|
// Check email size before accepting data
|
||||||
|
checkMessageSize(size: number): boolean;
|
||||||
|
|
||||||
|
// Configuration updates
|
||||||
|
updateOptions(options: Partial<ISmtpServerOptions>): void;
|
||||||
|
|
||||||
|
// Server stats
|
||||||
|
getStats(): IServerStats;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 EmailProcessor Class
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface IEmailProcessorOptions {
|
||||||
|
// Processing options
|
||||||
|
maxParallelProcessing?: number;
|
||||||
|
processingTimeout?: number;
|
||||||
|
|
||||||
|
// Feature flags
|
||||||
|
contentScanning?: boolean;
|
||||||
|
headerProcessing?: boolean;
|
||||||
|
dkimSigning?: boolean;
|
||||||
|
|
||||||
|
// Processing rules
|
||||||
|
scanners?: IScannerConfig[];
|
||||||
|
transformations?: ITransformConfig[];
|
||||||
|
|
||||||
|
// Routing rules
|
||||||
|
routingRules?: IRoutingRule[];
|
||||||
|
defaultServer?: string;
|
||||||
|
defaultPort?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Handles all email processing steps
|
||||||
|
*/
|
||||||
|
class EmailProcessor {
|
||||||
|
constructor(options: IEmailProcessorOptions);
|
||||||
|
|
||||||
|
// Main processing method
|
||||||
|
async processEmail(message: ParsedMail, session: Session): Promise<ProcessingResult>;
|
||||||
|
|
||||||
|
// Individual processing steps
|
||||||
|
async extractMetadata(message: ParsedMail): Promise<EmailMetadata>;
|
||||||
|
async determineRouting(metadata: EmailMetadata): Promise<RoutingDecision>;
|
||||||
|
async scanContent(message: ParsedMail): Promise<ScanResult>;
|
||||||
|
async applyTransformations(message: ParsedMail): Promise<ParsedMail>;
|
||||||
|
|
||||||
|
// Update processor configuration
|
||||||
|
updateOptions(options: Partial<IEmailProcessorOptions>): void;
|
||||||
|
|
||||||
|
// Manage processing plugins
|
||||||
|
addScanner(scanner: IScanner): void;
|
||||||
|
addTransformation(transformation: ITransformation): void;
|
||||||
|
addRoutingRule(rule: IRoutingRule): void;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 DeliveryQueue Class
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface IQueueOptions {
|
||||||
|
// Storage options
|
||||||
|
storageType: 'memory' | 'disk' | 'redis';
|
||||||
|
storagePath?: string;
|
||||||
|
redisUrl?: string;
|
||||||
|
|
||||||
|
// Queue behavior
|
||||||
|
checkInterval?: number;
|
||||||
|
maxQueueSize?: number;
|
||||||
|
maxPerDestination?: number;
|
||||||
|
|
||||||
|
// Delivery attempts
|
||||||
|
maxRetries?: number;
|
||||||
|
baseRetryDelay?: number;
|
||||||
|
maxRetryDelay?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Manages the queue of messages waiting for delivery
|
||||||
|
*/
|
||||||
|
class DeliveryQueue {
|
||||||
|
constructor(options: IQueueOptions);
|
||||||
|
|
||||||
|
// Queue operations
|
||||||
|
async enqueue(item: QueueItem): Promise<string>;
|
||||||
|
async dequeue(id: string): Promise<QueueItem | null>;
|
||||||
|
async update(id: string, updates: Partial<QueueItem>): Promise<boolean>;
|
||||||
|
async getNext(count?: number): Promise<QueueItem[]>;
|
||||||
|
|
||||||
|
// Query methods
|
||||||
|
async getByStatus(status: QueueItemStatus): Promise<QueueItem[]>;
|
||||||
|
async getByDestination(server: string): Promise<QueueItem[]>;
|
||||||
|
async getItemCount(): Promise<number>;
|
||||||
|
|
||||||
|
// Queue maintenance
|
||||||
|
async purgeExpired(): Promise<number>;
|
||||||
|
async purgeAll(): Promise<number>;
|
||||||
|
|
||||||
|
// Persistence
|
||||||
|
async load(): Promise<void>;
|
||||||
|
async save(): Promise<void>;
|
||||||
|
|
||||||
|
// Processing control
|
||||||
|
pause(): void;
|
||||||
|
resume(): void;
|
||||||
|
isProcessing(): boolean;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.4 DeliveryManager Class
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface IDeliveryOptions {
|
||||||
|
// Connection options
|
||||||
|
connectionPoolSize?: number;
|
||||||
|
socketTimeout?: number;
|
||||||
|
|
||||||
|
// Delivery behavior
|
||||||
|
concurrentDeliveries?: number;
|
||||||
|
sendTimeout?: number;
|
||||||
|
|
||||||
|
// TLS options
|
||||||
|
verifyCertificates?: boolean;
|
||||||
|
tlsMinVersion?: string;
|
||||||
|
|
||||||
|
// Rate limiting
|
||||||
|
globalRateLimit?: number;
|
||||||
|
perServerRateLimit?: number;
|
||||||
|
perDomainRateLimit?: Record<string, number>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Handles delivery of emails to destination servers
|
||||||
|
*/
|
||||||
|
class DeliveryManager {
|
||||||
|
constructor(queue: DeliveryQueue, options: IDeliveryOptions);
|
||||||
|
|
||||||
|
// Core delivery methods
|
||||||
|
async start(): Promise<void>;
|
||||||
|
async stop(): Promise<void>;
|
||||||
|
async deliverMessage(item: QueueItem): Promise<DeliveryResult>;
|
||||||
|
|
||||||
|
// Delivery management
|
||||||
|
pauseDeliveries(): void;
|
||||||
|
resumeDeliveries(): void;
|
||||||
|
getDeliveryStats(): DeliveryStats;
|
||||||
|
|
||||||
|
// Configure delivery behavior
|
||||||
|
updateOptions(options: Partial<IDeliveryOptions>): void;
|
||||||
|
setRateLimit(domain: string, limit: number): void;
|
||||||
|
clearRateLimit(domain: string): void;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.5 DcRouter SMTP Integration
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
interface ISmtpConfig {
|
||||||
|
// Server configuration
|
||||||
|
ports: number[];
|
||||||
|
hostname: string;
|
||||||
|
banner?: string;
|
||||||
|
maxMessageSize?: number;
|
||||||
|
|
||||||
|
// TLS configuration
|
||||||
|
tls?: {
|
||||||
|
certPath?: string;
|
||||||
|
keyPath?: string;
|
||||||
|
caPath?: string;
|
||||||
|
minVersion?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Authentication
|
||||||
|
auth?: {
|
||||||
|
required?: boolean;
|
||||||
|
methods?: ('PLAIN' | 'LOGIN' | 'OAUTH2')[];
|
||||||
|
users?: Array<{username: string, password: string}>;
|
||||||
|
ldapUrl?: string;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Domain routing
|
||||||
|
domainConfigs: Array<{
|
||||||
|
domains: string[];
|
||||||
|
targetIPs: string[];
|
||||||
|
port?: number;
|
||||||
|
useTls?: boolean;
|
||||||
|
authentication?: {
|
||||||
|
user?: string;
|
||||||
|
pass?: string;
|
||||||
|
};
|
||||||
|
allowedIPs?: string[];
|
||||||
|
rateLimits?: {
|
||||||
|
maxMessagesPerMinute?: number;
|
||||||
|
maxRecipientsPerMessage?: number;
|
||||||
|
};
|
||||||
|
addHeaders?: boolean;
|
||||||
|
headerInfo?: Array<{
|
||||||
|
name: string;
|
||||||
|
value: string;
|
||||||
|
}>;
|
||||||
|
signDkim?: boolean;
|
||||||
|
dkimOptions?: {
|
||||||
|
domainName: string;
|
||||||
|
keySelector: string;
|
||||||
|
privateKey: string;
|
||||||
|
};
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Default routing
|
||||||
|
defaultServer: string;
|
||||||
|
defaultPort?: number;
|
||||||
|
useTls?: boolean;
|
||||||
|
|
||||||
|
// Content scanning
|
||||||
|
contentScanning?: boolean;
|
||||||
|
scanners?: Array<{
|
||||||
|
type: 'spam' | 'virus' | 'attachment';
|
||||||
|
threshold?: number;
|
||||||
|
action: 'tag' | 'reject';
|
||||||
|
blockedExtensions?: string[];
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Message transformations
|
||||||
|
transformations?: Array<{
|
||||||
|
type: string;
|
||||||
|
[key: string]: any;
|
||||||
|
}>;
|
||||||
|
|
||||||
|
// Queue settings
|
||||||
|
queueStorage?: 'memory' | 'disk';
|
||||||
|
persistentPath?: string;
|
||||||
|
maxRetries?: number;
|
||||||
|
baseRetryDelay?: number;
|
||||||
|
maxRetryDelay?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extended IDcRouterOptions
|
||||||
|
interface IDcRouterOptions {
|
||||||
|
// Existing options...
|
||||||
|
|
||||||
|
// New SMTP configuration
|
||||||
|
smtpConfig?: ISmtpConfig;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Implementation Phases
|
||||||
|
|
||||||
|
### Phase 1: Core SMTP Server Setup
|
||||||
|
- [ ] Implement the SmtpServer class
|
||||||
|
- [ ] Set up TLS handling for both STARTTLS and implicit TLS
|
||||||
|
- [ ] Create the basic connection validation logic
|
||||||
|
- [ ] Implement authentication support
|
||||||
|
- [ ] Build email receiving pipeline to accept complete messages
|
||||||
|
- [ ] Create initial email parsing and storage
|
||||||
|
|
||||||
|
### Phase 2: Mail Processing and Routing
|
||||||
|
- [ ] Implement the EmailProcessor class
|
||||||
|
- [ ] Create domain-based routing rules
|
||||||
|
- [ ] Build email metadata extraction
|
||||||
|
- [ ] Implement MIME parsing and handling
|
||||||
|
- [ ] Create the transformation pipeline
|
||||||
|
- [ ] Build header manipulation capabilities
|
||||||
|
|
||||||
|
### Phase 3: Queue and Delivery System
|
||||||
|
- [ ] Implement the DeliveryQueue class
|
||||||
|
- [ ] Create persistent storage for queued messages
|
||||||
|
- [ ] Build the retry and scheduling logic
|
||||||
|
- [ ] Implement DeliveryManager with connection pooling
|
||||||
|
- [ ] Create the delivery status tracking and reporting
|
||||||
|
- [ ] Implement bounce handling and notification
|
||||||
|
|
||||||
|
### Phase 4: Advanced Features and Integration
|
||||||
|
- [ ] Integrate content scanning capabilities
|
||||||
|
- [ ] Implement DKIM signing
|
||||||
|
- [ ] Add rate limiting and traffic shaping
|
||||||
|
- [ ] Create comprehensive metrics and logging
|
||||||
|
- [ ] Build management APIs for runtime control
|
||||||
|
- [ ] Implement full integration with DcRouter
|
||||||
|
|
||||||
|
### Phase 5: Testing and Optimization
|
||||||
- [ ] Create unit tests for all components
|
- [ ] Create unit tests for all components
|
||||||
- Test environment variable handling
|
- [ ] Implement integration tests for end-to-end verification
|
||||||
- Test domain routing logic
|
- [ ] Perform load testing and optimize performance
|
||||||
- Test certificate management
|
- [ ] Conduct security testing and hardening
|
||||||
- [ ] Create integration tests
|
- [ ] Build documentation and examples
|
||||||
- Test email forwarding between domains
|
|
||||||
- Test HTTP/HTTPS routing
|
|
||||||
- Test TLS connections
|
|
||||||
|
|
||||||
### 6.2 Documentation
|
## 6. Technical Requirements
|
||||||
- [ ] Create comprehensive user documentation
|
|
||||||
- Add setup guide for common scenarios
|
### 6.1 Dependencies
|
||||||
- Document all configuration options
|
- SMTP server library (smtp-server or similar)
|
||||||
- Provide example configurations
|
- Email parsing library (mailparser or similar)
|
||||||
- [ ] Create API documentation
|
- MIME handling library
|
||||||
- Document all public interfaces
|
- DKIM signing library
|
||||||
- Add usage examples
|
- Queue management system (optional Redis support)
|
||||||
- Create integration examples
|
- Cryptographic libraries for TLS and authentication
|
||||||
|
|
||||||
|
### 6.2 Performance Targets
|
||||||
|
- Handle 1000+ concurrent SMTP connections
|
||||||
|
- Process 100+ messages per second on standard hardware
|
||||||
|
- Support message sizes up to 50MB
|
||||||
|
- Maintain delivery queue of 100,000+ messages
|
||||||
|
- Sub-second processing time for standard emails
|
||||||
|
|
||||||
|
### 6.3 Security Requirements
|
||||||
|
- Full TLS support with modern cipher configurations
|
||||||
|
- Authentication verification and rate limiting
|
||||||
|
- Input validation for all SMTP commands
|
||||||
|
- Secure storage of queued emails
|
||||||
|
- Proper error handling to prevent information leakage
|
||||||
|
- Access controls based on IP addresses and authentication
|
||||||
|
|
||||||
|
## 7. API Examples
|
||||||
|
|
||||||
|
### 7.1 Basic DcRouter SMTP Configuration
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const dcRouter = new DcRouter({
|
||||||
|
// HTTP configuration...
|
||||||
|
|
||||||
|
smtpConfig: {
|
||||||
|
ports: [25, 587, 465],
|
||||||
|
hostname: 'mail.example.com',
|
||||||
|
maxMessageSize: 50 * 1024 * 1024, // 50MB
|
||||||
|
|
||||||
|
// TLS configuration
|
||||||
|
tls: {
|
||||||
|
certPath: '/path/to/cert.pem',
|
||||||
|
keyPath: '/path/to/key.pem'
|
||||||
|
},
|
||||||
|
|
||||||
|
// Domain routing
|
||||||
|
domainConfigs: [
|
||||||
|
{
|
||||||
|
domains: ['example.com', '*.example.com'],
|
||||||
|
targetIPs: ['mail1.example.com', 'mail2.example.com'],
|
||||||
|
port: 25,
|
||||||
|
useTls: true
|
||||||
|
}
|
||||||
|
],
|
||||||
|
|
||||||
|
// Default routing
|
||||||
|
defaultServer: 'fallback-mail.example.com',
|
||||||
|
defaultPort: 25,
|
||||||
|
useTls: true,
|
||||||
|
|
||||||
|
// Queue settings
|
||||||
|
queueStorage: 'disk',
|
||||||
|
persistentPath: '/var/mail/queue',
|
||||||
|
maxRetries: 5
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7.2 Advanced Configuration with Processing
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
const dcRouter = new DcRouter({
|
||||||
|
// HTTP configuration...
|
||||||
|
|
||||||
|
smtpConfig: {
|
||||||
|
// Basic settings
|
||||||
|
ports: [25, 587, 465],
|
||||||
|
hostname: 'mail.example.com',
|
||||||
|
|
||||||
|
// Domain routing with advanced features
|
||||||
|
domainConfigs: [
|
||||||
|
{
|
||||||
|
domains: ['example.com', '*.example.com'],
|
||||||
|
targetIPs: ['mail1.example.com', 'mail2.example.com'],
|
||||||
|
port: 25,
|
||||||
|
useTls: true,
|
||||||
|
// Add custom headers
|
||||||
|
addHeaders: true,
|
||||||
|
headerInfo: [
|
||||||
|
{ name: 'X-Processed-By', value: 'gateway' },
|
||||||
|
{ name: 'X-Scanned', value: 'true' }
|
||||||
|
],
|
||||||
|
// Sign with DKIM
|
||||||
|
signDkim: true,
|
||||||
|
dkimOptions: {
|
||||||
|
domainName: 'example.com',
|
||||||
|
keySelector: 'mail',
|
||||||
|
privateKey: '...'
|
||||||
|
},
|
||||||
|
// Rate limiting
|
||||||
|
rateLimits: {
|
||||||
|
maxMessagesPerMinute: 100,
|
||||||
|
maxRecipientsPerMessage: 50
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
|
||||||
|
// Content scanning
|
||||||
|
contentScanning: true,
|
||||||
|
scanners: [
|
||||||
|
{
|
||||||
|
type: 'spam',
|
||||||
|
threshold: 5.0,
|
||||||
|
action: 'tag'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'virus',
|
||||||
|
action: 'reject'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'attachment',
|
||||||
|
blockedExtensions: ['.exe', '.bat', '.vbs'],
|
||||||
|
action: 'reject'
|
||||||
|
}
|
||||||
|
],
|
||||||
|
|
||||||
|
// Transformations
|
||||||
|
transformations: [
|
||||||
|
{
|
||||||
|
type: 'addHeader',
|
||||||
|
header: 'X-Gateway',
|
||||||
|
value: 'DcRouter 1.0'
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: 'dkimSign',
|
||||||
|
domains: ['example.com']
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## 8. Extensibility Points
|
||||||
|
|
||||||
|
### 8.1 Plugin Architecture
|
||||||
|
- Custom content scanners
|
||||||
|
- Custom transformation handlers
|
||||||
|
- Routing rule extensions
|
||||||
|
- Authentication providers
|
||||||
|
- Queue storage backends
|
||||||
|
|
||||||
|
### 8.2 Event System
|
||||||
|
- Connection events (connect, disconnect, error)
|
||||||
|
- Message events (received, processed, queued, delivered)
|
||||||
|
- Error events (delivery failure, processing error)
|
||||||
|
- Performance events (queue size, processing time)
|
||||||
|
- Security events (authentication failure, policy violation)
|
||||||
|
|
||||||
|
## 9. Migration Plan
|
||||||
|
|
||||||
|
### 9.1 From Simple Proxy to Store-and-Forward
|
||||||
|
- [ ] Create compatibility layer for existing configurations
|
||||||
|
- [ ] Implement graceful transition from connection proxy to full processing
|
||||||
|
- [ ] Add configuration validation to ensure smooth migration
|
||||||
|
- [ ] Create feature flags to enable advanced features incrementally
|
||||||
|
- [ ] Provide documentation for migrating existing deployments
|
||||||
|
|
||||||
|
### 9.2 Backward Compatibility
|
||||||
|
- [ ] Maintain support for basic proxy functionality
|
||||||
|
- [ ] Provide simple configuration options for common use cases
|
||||||
|
- [ ] Create migration utilities to update configuration formats
|
||||||
|
- [ ] Support running in hybrid mode during transition
|
||||||
|
|
||||||
|
## 10. SmartProxy Integration
|
||||||
|
|
||||||
|
### 10.1 SmartProxy Configuration Handling
|
||||||
|
- [ ] Implement comprehensive support for SmartProxy configuration
|
||||||
|
- Pass through all SmartProxy options directly
|
||||||
|
- Support all SmartProxy domain configuration features
|
||||||
|
- Ensure proper handling of SmartProxy events and callbacks
|
||||||
|
- [ ] Create clear documentation on SmartProxy configuration:
|
||||||
|
- Explain how all SmartProxy features are available through DcRouter
|
||||||
|
- Document common usage patterns and examples
|
||||||
|
- Provide guidance on advanced SmartProxy configurations
|
||||||
|
|
||||||
|
### 10.2 SMTP Integration with SmartProxy
|
||||||
|
- [ ] Ensure store-and-forward SMTP works alongside SmartProxy
|
||||||
|
- Document how SMTP ports are properly handled by SMTP processing
|
||||||
|
- Ensure no port conflicts between SmartProxy and SMTP server
|
||||||
|
- Create examples showing SmartProxy and SMTP working together
|
||||||
|
- [ ] Document combined deployment models:
|
||||||
|
- HTTP/HTTPS traffic using SmartProxy configuration
|
||||||
|
- SMTP traffic using store-and-forward for advanced processing
|
||||||
|
- Examples for multi-service environments
|
||||||
|
|
||||||
|
## 11. Documentation Requirements
|
||||||
|
|
||||||
|
### 11.1 Code Documentation
|
||||||
|
- [ ] Comprehensive JSDoc comments for all classes and methods
|
||||||
|
- [ ] Interface definitions with detailed parameter descriptions
|
||||||
|
- [ ] Example code snippets for common operations
|
||||||
|
- [ ] Architecture documentation with component diagrams
|
||||||
|
- [ ] Decision logs for key design choices
|
||||||
|
|
||||||
|
### 11.2 User Documentation
|
||||||
|
- [ ] Getting started guide with configuration approach selection guidance
|
||||||
|
- [ ] Complete configuration reference for both approaches
|
||||||
|
- [ ] Deployment scenarios and examples
|
||||||
|
- [ ] Troubleshooting guide
|
||||||
|
- [ ] Performance tuning recommendations
|
||||||
|
- [ ] Security best practices
|
||||||
|
|
||||||
|
### 11.3 Direct SmartProxy Configuration Guide
|
||||||
|
- [ ] Detailed guide on using SmartProxy's domain configuration capabilities
|
||||||
|
- [ ] Examples of complex routing scenarios with SmartProxy
|
||||||
|
- [ ] Performance optimization tips for SmartProxy configurations
|
||||||
|
- [ ] Security settings for SmartProxy deployments
|
Binary file not shown.
Loading…
x
Reference in New Issue
Block a user