feat(socket-handler): implement direct socket passing for DNS and email services

- Add socket-handler mode eliminating internal port binding for improved performance - Add `dnsDomain` config option for automatic DNS-over-HTTPS (DoH) setup - Add `useSocketHandler` flag to email config for direct socket processing - Update SmartProxy route generation to support socket-handler actions - Integrate smartdns with manual HTTPS mode for DoH without port binding - Add automatic route creation for DNS paths when dnsDomain is configured - Update documentation with socket-handler configuration and benefits - Improve resource efficiency by eliminating internal port forwarding
2025-05-29 16:26:19 +00:00
parent 6c8458f63c
commit b11fea7334
9 changed files with 687 additions and 540 deletions
--- a/readme.plan.md
+++ b/readme.plan.md
@@ -1,298 +1,293 @@
-# Simplified Email Routing Plan
+# DcRouter Socket-Handler Integration Plan

-## Core Principle
-Following SmartProxy's elegant pattern: **One array of routes with match/action pairs**. No complex nested configurations, no scattered features - just simple, powerful routing.
+First line: Remember to reread CLAUDE.md file for guidelines.

-## Architecture Overview
+## Overview
+Integrate socket-handler support for both DNS and Mail services in DcRouter, allowing smartproxy to pass sockets directly instead of servers listening on ports.

-### Single Router Class
- **EmailRouter** - The ONLY routing class
- Handles ALL routing logic: domains, IPs, content, relay, etc.
- No separate RelayManager, no RecipientExpander - all built-in
+## Core Logic
+- **DNS**: If dnsDomain is set, DnsServer is instantiated with socket-handler for HTTPS/DoH
+- **Mail**: If useSocketHandler is enabled in emailConfig, mail servers accept sockets from smartproxy
+- **Automatic Routing**: Setting these options automatically configures smartproxy routes

-### Core Interfaces (Just 3!)
+## Architecture

+### DNS Architecture
+- **HTTPS/DoH Traffic**: smartproxy → socket-handler → smartdns (no port listening)
+- **UDP Traffic**: Direct to smartdns DnsServer on VM IP:53 (bypasses smartproxy)
+- **Automatic Setup**: Setting dnsDomain triggers all DNS infrastructure
+
+### Mail Architecture  
+- **Socket-Handler Mode**: smartproxy → socket-handler → mail server (no port listening)
+- **Traditional Mode**: smartproxy → forward → mail server (listening on ports)
+- **Configurable**: useSocketHandler flag determines the mode
+
+## Implementation Steps
+
+### 1. Analyze Current Architecture
+- [x] Examine how DcRouter currently integrates services
+- [x] Study email server socket handling in SmtpServer
+- [x] Check existing smartproxy socket-handler patterns
+- [x] Review UnifiedEmailServer connection handling
+
+### 2. DNS Socket-Handler Implementation
+
+#### 2.1 Create DNS Socket Handler
+- [x] Create a custom socket handler that processes DNS-over-HTTPS requests
+- [x] Handler should use smartdns without port binding
+- [x] Use built-in handleHttpsSocket method from smartdns
+- [x] Set manualHttpsMode: true in DnsServer options
+
+#### 2.2 Configure smartdns DnsServer
+- [x] Create DnsServer instance with UDP only on VM IP interface
+- [x] Set manualHttpsMode: true to disable HTTPS port binding
+- [x] Configure to listen on port 53 for UDP traffic only
+- [x] Use handleHttpsSocket method for DoH sockets
+
+### 3. Mail Socket-Handler Implementation
+
+#### 3.1 Modify SmtpServer
+- [x] ConnectionManager already has handleConnection method
+- [x] Connection handling already works with provided sockets
+- [x] Session management works with socket-handler mode
+- [x] Backward compatibility maintained
+
+#### 3.2 Update UnifiedEmailServer
+- [x] Add useSocketHandler flag to IUnifiedEmailServerOptions
+- [x] Modify start() to skip server creation when useSocketHandler is true
+- [x] Add handleSocket method to accept sockets from smartproxy
+- [x] All email processing works in socket-handler mode
+
+#### 3.3 Create Mail Socket Handler
+- [x] Create socket handler for SMTP/submission/SMTPS
+- [x] Handle TLS for port 465 (immediate TLS wrapping)
+- [x] Pass sockets to UnifiedEmailServer.handleSocket
+
+### 4. DcRouter Integration
+
+#### 4.1 Configuration Updates
+- [x] Add `dnsDomain` property to DcRouter config
+- [x] Add `useSocketHandler` to email config options
+- [x] Update interface definitions
+
+#### 4.2 Route Generation Updates
+- [x] Modify generateEmailRoutes to create socket-handler actions when enabled
+- [x] Create DNS routes with socket-handler actions
+- [x] Ensure proper domain and path matching
+
+#### 4.3 Service Lifecycle
+- [x] Update setupSmartProxy to handle socket-handler routes
+- [x] Services start correctly without port binding
+- [x] Socket cleanup handled by connection managers
+
+### 5. SmartProxy Route Configuration
+- [x] DNS routes: match dnsDomain with paths /dns-query and /resolve
+- [x] Mail routes: match ports 25, 587, 465 with socket-handler actions
+- [x] Configure TLS handling for each protocol appropriately
+- [x] Automatic Let's Encrypt via smartproxy certificate: 'auto'
+
+### 6. Testing (To be done)
+
+#### 6.1 DNS Testing
+- [ ] Test that DNS server is NOT instantiated when dnsDomain is not set
+- [ ] Test that DNS server IS instantiated when dnsDomain is set
+- [ ] Test UDP DNS queries on port 53
+- [ ] Test DoH queries through smartproxy socket-handler
+- [ ] Verify HTTP/2 support for DoH
+
+#### 6.2 Mail Testing
+- [ ] Test traditional port-based mail delivery
+- [ ] Test socket-handler mail delivery
+- [ ] Verify STARTTLS works in socket-handler mode
+- [ ] Test SMTPS (port 465) with socket-handler
+- [ ] Ensure connection pooling works correctly
+
+#### 6.3 Integration Testing
+- [ ] Test both DNS and Mail with socket-handlers simultaneously
+- [ ] Verify no port conflicts
+- [ ] Test automatic startup/shutdown
+- [ ] Load test socket-handler performance
+
+## Technical Details
+
+### DcRouter Configuration
 ```typescript
-interface IEmailRoute {
-  name: string;                    // Route identifier
-  priority?: number;               // Order of evaluation (default: 0)
-  match: IEmailMatch;              // What to match
-  action: IEmailAction;            // What to do
+// DcRouter config with socket-handler support
+const dcRouterConfig = {
+  // ... other config
+  dnsDomain: 'example.com', // Optional - if set, DNS server is enabled
+  emailConfig: {
+    ports: [25, 587, 465],
+    domains: ['mail.example.com'],
+    useSocketHandler: true, // Enable socket-handler mode for mail
+    routes: [/* email routing rules */]
+  }
 }
+```

-interface IEmailMatch {
-  // Simple pattern matching
-  recipients?: string | string[];    // "*@example.com", "admin@*"
-  senders?: string | string[];       
-  clientIp?: string | string[];      // IPs or CIDR ranges
-  authenticated?: boolean;
-  
-  // Optional advanced matching
-  headers?: Record<string, string | RegExp>;
-  sizeRange?: { min?: number; max?: number };
-}
+### DNS Implementation

-interface IEmailAction {
-  type: 'forward' | 'deliver' | 'reject' | 'process';
+#### Conditional DNS Instantiation
+```typescript
+// In DcRouter startup logic
+if (config.dnsDomain) {
+  // Create DNS server instance
+  this.dnsServer = new DnsServer({
+    udpPort: 53,
+    udpBindAddress: vmIpAddress,
+    httpsPort: undefined, // No HTTPS listening
+    dnssecZone: config.dnsDomain
+  });
  
-  // Action-specific options
-  forward?: { host: string; port?: number; };
-  reject?: { code: number; message: string; };
-  process?: { 
-    scan?: boolean;
-    dkim?: boolean;
-    queue?: 'normal' | 'priority';
+  // Create smartproxy route for DoH
+  const dnsRoute = {
+    match: {
+      domain: config.dnsDomain,
+      path: ['/dns-query', '/resolve']
+    },
+    action: {
+      type: 'socket-handler',
+      handler: this.createDnsSocketHandler()
+    }
  };
 }
 ```

-## Implementation Plan
-
-### Phase 1: Core Functionality (1-2 weeks)
-
-1. **Create new EmailRouter**
-   - Fresh implementation with route-based design
-   - Implement `evaluateRoutes(context)` method
-   - Clean, modern API only
-
-2. **Update UnifiedEmailServer**
-   - Replace domainRules with routes array
-   - Simplify `processEmailByMode()` to use actions
-   - Implement forward mode (currently missing)
-
-3. **Essential Features Only**
-   - Forward emails (using existing SmtpClient)
-   - Process emails (existing flow)
-   - Reject emails (simple SMTP response)
-   - IP-based relay (just another match condition)
-
-### Phase 2: Progressive Enhancement (2-3 weeks)
-
-Only add features that are actually needed:
- Multi-target forwarding (array of hosts)
- Basic transformations (headers only)
- Rate limiting per route
-
-## Configuration Example
-
+#### DNS Socket Handler
 ```typescript
-const emailServer = new UnifiedEmailServer({
-  ports: [25, 587, 465],
-  hostname: 'mail.example.com',
+const createDnsSocketHandler = () => {
+  return async (socket: net.Socket) => {
+    // Handle HTTP/2 for DoH
+    const session = http2.createSession(socket);
+    
+    session.on('stream', async (stream, headers) => {
+      if (headers[':path'] === '/dns-query') {
+        const dnsQuery = await parseDnsQuery(stream);
+        const response = await dnsServer.resolveQuery(dnsQuery);
+        stream.respond({ ':status': 200, 'content-type': 'application/dns-message' });
+        stream.end(response);
+      }
+    });
+  };
+}
+```
+
+### Mail Implementation
+
+#### Email Route Generation with Socket-Handler
+```typescript
+private generateEmailRoutes(emailConfig: IUnifiedEmailServerOptions): IRouteConfig[] {
+  const routes: IRouteConfig[] = [];
  
-  // ALL routing in one simple array
-  routes: [
-    // Relay from office
-    {
-      name: 'office-relay',
-      priority: 100,
-      match: { clientIp: '192.168.0.0/16' },
-      action: { type: 'forward', forward: { host: 'internal.mail' } }
-    },
+  for (const port of emailConfig.ports) {
+    const action = emailConfig.useSocketHandler
+      ? {
+          type: 'socket-handler',
+          handler: this.createMailSocketHandler(port)
+        }
+      : {
+          type: 'forward',
+          target: { host: 'localhost', port: mapPort(port) }
+        };
    
-    // Process local mail
-    {
-      name: 'local-mail',
-      priority: 50,
-      match: { recipients: '*@mycompany.com' },
-      action: { 
-        type: 'process',
-        process: { scan: true, dkim: true, queue: 'normal' }
-      }
-    },
-    
-    // Reject everything else
-    {
-      name: 'default',
-      match: { recipients: '*' },
-      action: { 
-        type: 'reject',
-        reject: { code: 550, message: 'Relay denied' }
-      }
-    }
-  ]
-});
+    routes.push({
+      match: { ports: [port] },
+      action
+    });
+  }
+  
+  return routes;
+}
 ```

-## Key Simplifications
-
-### What We're NOT Building
- ❌ Separate RelayManager class - relay is just a route match
- ❌ RecipientExpander class - alias expansion is a delivery concern
- ❌ ContentAnalyzer with ML - overkill for email routing
- ❌ 20+ new classes - keep it simple
- ❌ Complex analytics - use existing logging
- ❌ API-driven routing - config file is enough
- ❌ Compliance routing - not a router concern
-
-### What We ARE Building
- ✅ One router class that does routing well
- ✅ Simple match/action pattern
- ✅ IP-based relay as a native feature
- ✅ Email forwarding (currently missing)
- ✅ Clean, modern implementation
-
-## Clean Implementation
-
+#### Mail Socket Handler
 ```typescript
-// Simple, single way to configure:
-const router = new EmailRouter(routes);
-
-// No legacy formats, no auto-detection
-// Just clean route-based configuration
+const createMailSocketHandler = (port: number) => {
+  return async (socket: net.Socket) => {
+    // Determine protocol based on port
+    const isSecure = port === 465;
+    
+    if (isSecure) {
+      // For SMTPS, handle TLS immediately
+      const tlsSocket = new tls.TLSSocket(socket, {
+        isServer: true,
+        cert: tlsCert,
+        key: tlsKey
+      });
+      await this.emailServer.handleSocket(tlsSocket, port);
+    } else {
+      // For SMTP/Submission, pass raw socket
+      await this.emailServer.handleSocket(socket, port);
+    }
+  };
+}
 ```

-## Benefits
+#### UnifiedEmailServer Socket Handling
+```typescript
+// In UnifiedEmailServer class
+public async handleSocket(socket: net.Socket, port: number): Promise<void> {
+  // Create session for this socket
+  const session = this.sessionManager.createSession(socket);
+  
+  // Handle connection based on port
+  switch (port) {
+    case 25:  // SMTP
+    case 587: // Submission (STARTTLS)
+      await this.connectionManager.handleConnection(socket, session);
+      break;
+    case 465: // SMTPS (already TLS)
+      await this.connectionManager.handleSecureConnection(socket, session);
+      break;
+  }
+}

-1. **Simplicity** - One router, one config array, clear patterns
-2. **Flexibility** - Any match criteria, any action
-3. **Performance** - Simple priority-based evaluation
-4. **Maintainability** - Less code, fewer classes, clearer intent
-5. **Modern** - Clean slate implementation
+## Dependencies
+- @push.rocks/smartproxy (already integrated)
+- @push.rocks/smartdns (to be added)

-## Detailed Implementation Steps
+## Implementation Phases

-### Step 1: Create Core Interfaces (30 minutes)
- [x] Create `ts/mail/routing/interfaces.ts`
- [x] Add `IEmailRoute` interface
- [x] Add `IEmailMatch` interface
- [x] Add `IEmailAction` interface
- [x] Add `IEmailContext` interface: `{ email: Email; session: IExtendedSmtpSession }`
- [x] Export all interfaces
+### Phase 1: DNS Socket-Handler (Priority)
+1. Add smartdns dependency
+2. Implement DNS socket-handler
+3. Add dnsDomain configuration
+4. Test DoH functionality

-### Step 2: Create EmailRouter Class (2 hours)
- [x] Create `ts/mail/routing/classes.email.router.ts`
- [x] Import necessary dependencies
- [x] Define class with routes property
- [x] Implement constructor: `constructor(routes: IEmailRoute[])`
- [x] Add `sortRoutesByPriority()` method
- [x] Call sort in constructor
+### Phase 2: Mail Socket-Handler  
+1. Refactor SmtpServer for socket handling
+2. Update UnifiedEmailServer
+3. Implement mail socket-handlers
+4. Add useSocketHandler configuration
+5. Test all mail protocols

-### Step 3: Implement Route Matching (3 hours)
- [x] Add `evaluateRoutes(context: IEmailContext): Promise<IEmailRoute | null>` method
- [x] Add `matchesRoute(route: IEmailRoute, context: IEmailContext): boolean` method
- [x] Implement `matchesRecipients()` with glob support
- [x] Implement `matchesSenders()` with glob support
- [x] Implement `matchesClientIp()` with CIDR support
- [x] Implement `matchesAuthenticated()` check
- [x] Add pattern cache for performance
+### Phase 3: Integration & Testing
+1. Full integration testing
+2. Performance benchmarking
+3. Documentation updates
+4. Migration guide for existing users

-### Step 4: Update UnifiedEmailServer Configuration (2 hours)
- [x] Update `IUnifiedEmailServerOptions` interface
- [x] Replace `domainRules` with `routes: IEmailRoute[]`
- [x] Remove `defaultMode`, `defaultServer`, etc.
- [x] Update constructor to create EmailRouter with routes
- [x] Replace DomainRouter usage with EmailRouter
+## Notes

-### Step 5: Implement Action Execution (4 hours)
- [x] Add `executeAction(action: IEmailAction, email: Email, context: IEmailContext): Promise<void>` to UnifiedEmailServer
- [x] Implement 'forward' action:
-  - [x] Get SMTP client via `getSmtpClient()`
-  - [x] Add forwarding headers (X-Forwarded-For, etc.)
-  - [x] Send email using pooled client
-  - [x] Handle errors with bounce manager
- [x] Implement 'process' action:
-  - [x] Use existing `handleProcessMode()` logic
-  - [x] Apply scan/dkim options from action
- [x] Implement 'deliver' action:
-  - [x] Queue for local delivery
- [x] Implement 'reject' action:
-  - [x] Return SMTP rejection with code/message
- [x] **IMPROVEMENT**: Moved DKIM signing to delivery system (right before sending) to ensure signature validity
+### DNS Notes
+- UDP traffic bypasses proxy entirely (kernel-level routing)
+- DoH provides encrypted DNS over HTTPS through proxy
+- Socket handler allows DNS processing without port binding
+- DNS functionality is entirely optional - only enabled when dnsDomain is configured
+- Setting dnsDomain triggers automatic setup of all DNS infrastructure
+- Automatic Let's Encrypt certificate provisioning for configured dnsDomain

-### Step 6: Refactor processEmailByMode (2 hours)
- [x] Update `processEmailByMode()` to use EmailRouter
- [x] Call `evaluateRoutes()` for each recipient
- [x] Call `executeAction()` based on matched route
- [x] Handle no-match case (default reject)
- [x] Remove old mode-based logic
+### Mail Notes
+- Socket-handler mode eliminates internal port binding for mail services
+- Traditional port forwarding mode remains available for compatibility
+- STARTTLS negotiation handled within socket-handler for ports 25/587
+- Port 465 (SMTPS) requires immediate TLS handshake in socket-handler
+- Connection pooling and session management work in both modes
+- Socket-handler reduces latency by eliminating internal forwarding

-### Step 7: Testing (4 hours)
- [x] Create `test/test.email.router.ts`
- [x] Test route priority sorting
- [x] Test recipient matching (exact, glob, multiple)
- [x] Test IP matching (single, CIDR, arrays)
- [x] Test authentication matching
- [x] Test action execution (basic)
- [x] Test no-match scenarios
-
-### Step 8: Integration Testing (2 hours)
- [x] Create comprehensive integration test suite (`test/test.email.integration.ts`)
- [x] Test relay scenario (IP-based forward) with priority routing
- [x] Test CIDR IP matching with multiple ranges
- [x] Test authentication-based routing scenarios
- [x] Test pattern caching performance
- [x] Test dynamic route updates
- [x] Verify all match/action patterns work end-to-end
-
-### Step 9: Update Examples and Docs (1 hour)
- [x] Update configuration examples in readme
- [x] Add route examples for common scenarios
- [x] Document glob pattern syntax
- [x] Document CIDR notation for IPs
- [x] Add troubleshooting section
-
-### Step 10: Cleanup (1 hour)
- [x] Remove DomainRouter imports
- [x] Remove old interfaces (IDomainRule, etc.)
- [x] Remove legacy configuration code
- [x] Update any remaining references
- [x] Run full test suite
- [x] Commit changes
-
-## Total Time Estimate
- **Day 1**: Steps 1-3 (5.5 hours)
- **Day 2**: Steps 4-6 (8 hours)
- **Day 3**: Steps 7-8 (6 hours)
- **Day 4**: Steps 9-10 (2 hours)
-
-**Total**: ~21 hours of focused work (2-3 days)
-
-## Implementation Status
-
-### ✅ Completed (January 2025)
- Created routing interfaces and EmailRouter class
- Implemented comprehensive route matching (recipients, senders, IPs, authentication)
- Updated UnifiedEmailServer to use new routing system
- Implemented all four action types (forward, process, deliver, reject)
- Moved DKIM signing to delivery system for proper signature validity
- Fixed all compilation errors and updated dependencies
- Created basic routing tests with full coverage
- **NEW**: Created comprehensive integration test suite with real scenarios
- **NEW**: Verified all match/action patterns work end-to-end
-
-### Key Improvements Made
-1. **DKIM Signing**: Moved to delivery system right before sending to ensure signatures remain valid
-2. **Error Handling**: Integrated with BounceManager for proper failure handling
-3. **Connection Pooling**: Leveraged existing SmtpClient pooling for efficient forwarding
-4. **Pattern Caching**: Added caching for glob patterns to improve performance
-5. **Integration Testing**: Comprehensive tests covering IP-based relay, authentication routing, CIDR matching, and dynamic route updates
-
-### Integration Test Results ✅
- Route-based forwarding with priority: 5/5 scenarios passed
- CIDR IP matching with multiple ranges: 4/4 IP tests passed
- Authentication-based routing: 3/3 auth scenarios passed
- Pattern caching performance: Working correctly
- Dynamic route updates: Working correctly
-
-### ✅ Final Completion (January 2025)
- **DOCUMENTATION**: Updated readme.md with comprehensive route-based configuration examples
- **EXAMPLES**: Added common routing patterns (IP relay, domain routing, auth-based, content filtering)
- **PATTERN SYNTAX**: Documented glob patterns and CIDR notation with examples
- **TROUBLESHOOTING**: Added routing-specific troubleshooting guide
- **CLEANUP**: Removed all legacy DomainRouter and IDomainRule code
- **INTERFACES**: Cleaned up interfaces to only include implemented features
- **TESTING**: Verified build and core functionality tests pass
- **COMMIT**: All changes committed with detailed documentation
-
-**The match/action pattern implementation is COMPLETE, DOCUMENTED, and PRODUCTION-READY!** 🎉
-
-### Summary of Achievement
-✅ **Simple Configuration**: One routes array with match/action pairs  
-✅ **Flexible Matching**: Recipients, senders, IPs (CIDR), authentication, headers, size, subject  
-✅ **Four Actions**: forward, process, deliver, reject  
-✅ **Priority Routing**: Higher priority routes evaluated first  
-✅ **Pattern Caching**: Performance optimization for glob matching  
-✅ **IP Relay Support**: CIDR notation for network-based routing  
-✅ **Complete Documentation**: Examples, patterns, troubleshooting  
-✅ **Clean Implementation**: Legacy code removed, modern interfaces only
+### General Benefits
+- Reduced resource usage (no internal port binding)
+- Better performance (direct socket passing)
+- Simplified configuration (automatic route creation)
+- Enhanced security (no exposed internal ports)
+- Unified approach for all services through smartproxy