smartproxy/readme.plan.md

# SmartProxy Fully Unified Configuration Plan

## Project Goal
Redesign SmartProxy's configuration for a more elegant, unified, and comprehensible approach by:
1. Creating a single, unified configuration model that covers both "where to listen" and "how to forward"
2. Eliminating the confusion between domain configs and port forwarding
3. Providing a clear, declarative API that makes the intent obvious
4. Enhancing maintainability and extensibility for future features

## Current Issues

The current approach has several issues:

1. **Dual Configuration Systems**:
   - Port configuration (`fromPort`, `toPort`, `globalPortRanges`) for "where to listen"
   - Domain configuration (`domainConfigs`) for "how to forward"
   - Unclear relationship between these two systems

2. **Mixed Concerns**:
   - Port management is mixed with forwarding logic
   - Domain routing is separated from port listening
   - Security settings defined in multiple places

3. **Complex Logic**:
   - PortRangeManager must coordinate with domain configuration
   - Implicit rules for handling connections based on port and domain

4. **Difficult to Understand and Configure**:
   - Two separate configuration hierarchies that must work together
   - Unclear which settings take precedence

## Proposed Solution: Fully Unified Routing Configuration

Replace both port and domain configuration with a single, unified configuration:

```typescript
// The core unified configuration interface
interface IRouteConfig {
  // What to match
  match: {
    // Listen on these ports (required)
    ports: number | number[] | Array<{ from: number, to: number }>;
    
    // Optional domain patterns to match (default: all domains)
    domains?: string | string[];
    
    // Advanced matching criteria
    path?: string;           // Match specific paths
    clientIp?: string[];     // Match specific client IPs
    tlsVersion?: string[];   // Match specific TLS versions
  };
  
  // What to do with matched traffic
  action: {
    // Basic routing
    type: 'forward' | 'redirect' | 'block';
    
    // Target for forwarding
    target?: {
      host: string | string[];  // Support single host or round-robin
      port: number;
      preservePort?: boolean;   // Use incoming port as target port
    };
    
    // TLS handling
    tls?: {
      mode: 'passthrough' | 'terminate' | 'terminate-and-reencrypt';
      certificate?: 'auto' | {   // Auto = use ACME
        key: string;
        cert: string;
      };
    };
    
    // For redirects
    redirect?: {
      to: string;            // URL or template with {domain}, {port}, etc.
      status: 301 | 302 | 307 | 308;
    };
    
    // Security options
    security?: {
      allowedIps?: string[];
      blockedIps?: string[];
      maxConnections?: number;
      authentication?: {
        type: 'basic' | 'digest' | 'oauth';
        // Auth-specific options
      };
    };
    
    // Advanced options
    advanced?: {
      timeout?: number;
      headers?: Record<string, string>;
      keepAlive?: boolean;
      // etc.
    };
  };
  
  // Optional metadata
  name?: string;             // Human-readable name for this route
  description?: string;      // Description of the route's purpose
  priority?: number;         // Controls matching order (higher = matched first)
  tags?: string[];           // Arbitrary tags for categorization
}

// Main SmartProxy options
interface ISmartProxyOptions {
  // The unified configuration array (required)
  routes: IRouteConfig[];
  
  // Global/default settings
  defaults?: {
    target?: {
      host: string;
      port: number;
    };
    security?: {
      // Global security defaults
    };
    tls?: {
      // Global TLS defaults
    };
    // ...other defaults
  };
  
  // Other global settings remain (acme, etc.)
  acme?: IAcmeOptions;
  
  // Advanced settings remain as well
  // ...
}
```

### Example Configuration

```typescript
const proxy = new SmartProxy({
  // All routing is configured in a single array
  routes: [
    // Basic HTTPS server with automatic certificates
    {
      match: {
        ports: 443,
        domains: ['example.com', '*.example.com']
      },
      action: {
        type: 'forward',
        target: {
          host: 'localhost',
          port: 8080
        },
        tls: {
          mode: 'terminate',
          certificate: 'auto' // Use ACME
        }
      },
      name: 'Main HTTPS Server'
    },
    
    // HTTP to HTTPS redirect
    {
      match: {
        ports: 80,
        domains: ['example.com', '*.example.com']
      },
      action: {
        type: 'redirect',
        redirect: {
          to: 'https://{domain}{path}',
          status: 301
        }
      },
      name: 'HTTP to HTTPS Redirect'
    },
    
    // Admin portal with IP restriction
    {
      match: {
        ports: 8443,
        domains: 'admin.example.com'
      },
      action: {
        type: 'forward',
        target: {
          host: 'admin-backend',
          port: 3000
        },
        tls: {
          mode: 'terminate',
          certificate: 'auto'
        },
        security: {
          allowedIps: ['192.168.1.*', '10.0.0.*'],
          maxConnections: 10
        }
      },
      priority: 100, // Higher priority than default rules
      name: 'Admin Portal'
    },
    
    // Port range for direct forwarding
    {
      match: {
        ports: [{ from: 10000, to: 10010 }],
        // No domains = all domains or direct IP
      },
      action: {
        type: 'forward',
        target: {
          host: 'backend-server',
          port: 10000,
          preservePort: true // Use same port number as incoming
        },
        tls: {
          mode: 'passthrough' // Direct TCP forwarding
        }
      },
      name: 'Dynamic Port Range'
    },
    
    // Path-based routing
    {
      match: {
        ports: 443,
        domains: 'api.example.com',
        path: '/v1/*'
      },
      action: {
        type: 'forward',
        target: {
          host: 'api-v1-service',
          port: 8001
        },
        tls: {
          mode: 'terminate'
        }
      },
      name: 'API v1 Endpoints'
    },
    
    // Load balanced backend
    {
      match: {
        ports: 443,
        domains: 'app.example.com'
      },
      action: {
        type: 'forward',
        target: {
          // Round-robin load balancing
          host: [
            'app-server-1',
            'app-server-2',
            'app-server-3'
          ],
          port: 8080
        },
        tls: {
          mode: 'terminate'
        },
        advanced: {
          headers: {
            'X-Served-By': '{server}',
            'X-Client-IP': '{clientIp}'
          }
        }
      },
      name: 'Load Balanced App Servers'
    }
  ],
  
  // Global defaults
  defaults: {
    target: {
      host: 'localhost',
      port: 8080
    },
    security: {
      maxConnections: 1000,
      // Global security defaults
    }
  },
  
  // ACME configuration for auto certificates
  acme: {
    enabled: true,
    email: 'admin@example.com',
    production: true,
    renewThresholdDays: 30
  },
  
  // Other global settings
  // ...
});
```

## Implementation Plan

### Phase 1: Core Design & Interface Definition

1. **Define New Core Interfaces**:
   - Create `IRouteConfig` interface with `match` and `action` branches
   - Define all sub-interfaces for matching and actions
   - Update `ISmartProxyOptions` to use `routes` array
   - Define template variable system for dynamic values

2. **Create Helper Functions**:
   - `createRoute()` - Basic route creation with reasonable defaults
   - `createHttpRoute()`, `createHttpsRoute()`, `createRedirect()` - Common scenarios
   - `createLoadBalancer()` - For multi-target setups
   - `mergeSecurity()`, `mergeDefaults()` - For combining configs

3. **Design Router**:
   - Decision tree for route matching algorithm
   - Priority system for route ordering
   - Optimized lookup strategy for fast routing

### Phase 2: Core Implementation

1. **Create RouteManager**:
   - Replaces both PortRangeManager and DomainConfigManager
   - Handles all routing decisions in one place
   - Provides fast lookup from port+domain+path to action
   - Manages server instances for different ports

2. **Update ConnectionHandler**:
   - Simplify to work with the unified route system
   - Implement templating system for dynamic values
   - Support more sophisticated routing rules

3. **Implement New SmartProxy Core**:
   - Rewrite initialization to use route-based configuration
   - Create network servers based on port specifications
   - Manage TLS contexts and certificates

### Phase 3: Feature Implementation

1. **Path-Based Routing**:
   - Implement HTTP path matching
   - Add wildcard and regex support for paths
   - Create route differentiation based on HTTP method

2. **Enhanced Security**:
   - Implement per-route security rules
   - Add authentication methods (basic, digest, etc.)
   - Support for IP-based access control

3. **TLS Management**:
   - Support multiple certificate types (auto, manual, wildcard)
   - Implement certificate selection based on SNI
   - Support different TLS modes per route

4. **Metrics & Monitoring**:
   - Per-route statistics
   - Named route tracking for better visibility
   - Tag-based grouping of metrics

### Phase 4: Backward Compatibility

1. **Legacy Adapter Layer**:
   - Convert old configuration format to route-based format
   - Map fromPort/toPort/domainConfigs to unified routes
   - Preserve all functionality during migration

2. **API Compatibility**:
   - Support both old and new configuration methods
   - Add deprecation warnings when using legacy properties
   - Provide migration utilities

3. **Documentation**:
   - Clear migration guide for existing users
   - Examples mapping old config to new config
   - Timeline for deprecation

### Phase 5: Documentation & Examples

1. **Update Core Documentation**:
   - Rewrite README.md with a focus on route-based configuration
   - Create interface reference documentation
   - Document all template variables

2. **Create Example Library**:
   - Common configuration patterns
   - Complex use cases for advanced features
   - Infrastructure-as-code examples

3. **Add Validation Tooling**:
   - Configuration validator to check for issues
   - Schema definitions for IDE autocomplete
   - Runtime validation helpers

### Phase 6: Testing

1. **Unit Tests**:
   - Test route matching logic
   - Validate priority handling
   - Test template processing

2. **Integration Tests**:
   - Verify full proxy flows
   - Test complex routing scenarios
   - Check backward compatibility

3. **Performance Testing**:
   - Benchmark routing performance
   - Evaluate memory usage
   - Test with large numbers of routes

## Benefits of the New Approach

1. **Truly Unified Configuration**:
   - One "source of truth" for all routing
   - Entire routing flow visible in a single configuration
   - No overlapping or conflicting configuration systems

2. **Declarative Intent**:
   - Configuration clearly states what to match and what action to take
   - Metadata provides context and documentation inline
   - Easy to understand the purpose of each route

3. **Advanced Routing Capabilities**:
   - Path-based routing with pattern matching
   - Client IP-based conditional routing
   - Fine-grained security controls

4. **Composable and Extensible**:
   - Each route is a self-contained unit of configuration
   - Routes can be grouped by tags or priority
   - New match criteria or actions can be added without breaking changes

5. **Better Developer Experience**:
   - Clear, consistent configuration pattern
   - Helper functions for common scenarios
   - Better error messages and validation

## Example Use Cases

### 1. Complete Reverse Proxy with Auto SSL

```typescript
const proxy = new SmartProxy({
  routes: [
    // HTTPS server for all domains
    {
      match: { ports: 443 },
      action: {
        type: 'forward',
        target: { host: 'localhost', port: 8080 },
        tls: { mode: 'terminate', certificate: 'auto' }
      }
    },
    // HTTP to HTTPS redirect
    {
      match: { ports: 80 },
      action: {
        type: 'redirect',
        redirect: { to: 'https://{domain}{path}', status: 301 }
      }
    }
  ],
  acme: {
    enabled: true,
    email: 'admin@example.com',
    production: true
  }
});
```

### 2. Microservices API Gateway

```typescript
const apiGateway = new SmartProxy({
  routes: [
    // Users API
    {
      match: { 
        ports: 443,
        domains: 'api.example.com',
        path: '/users/*'
      },
      action: {
        type: 'forward',
        target: { host: 'users-service', port: 8001 },
        tls: { mode: 'terminate', certificate: 'auto' }
      },
      name: 'Users Service'
    },
    // Products API
    {
      match: { 
        ports: 443,
        domains: 'api.example.com',
        path: '/products/*'
      },
      action: {
        type: 'forward',
        target: { host: 'products-service', port: 8002 },
        tls: { mode: 'terminate', certificate: 'auto' }
      },
      name: 'Products Service'
    },
    // Orders API with authentication
    {
      match: { 
        ports: 443,
        domains: 'api.example.com',
        path: '/orders/*'
      },
      action: {
        type: 'forward',
        target: { host: 'orders-service', port: 8003 },
        tls: { mode: 'terminate', certificate: 'auto' },
        security: {
          authentication: {
            type: 'basic',
            users: { 'admin': 'password' }
          }
        }
      },
      name: 'Orders Service (Protected)'
    }
  ],
  acme: {
    enabled: true,
    email: 'admin@example.com'
  }
});
```

### 3. Multi-Environment Setup

```typescript
const environments = {
  production: {
    target: { host: 'prod-backend', port: 8080 },
    security: { maxConnections: 1000 }
  },
  staging: {
    target: { host: 'staging-backend', port: 8080 },
    security: { 
      allowedIps: ['10.0.0.*', '192.168.1.*'],
      maxConnections: 100
    }
  },
  development: {
    target: { host: 'localhost', port: 3000 },
    security: { 
      allowedIps: ['127.0.0.1'],
      maxConnections: 10
    }
  }
};

// Select environment based on hostname
const proxy = new SmartProxy({
  routes: [
    // Production environment
    {
      match: { 
        ports: [80, 443],
        domains: ['app.example.com', 'www.example.com']
      },
      action: {
        type: 'forward',
        target: environments.production.target,
        tls: { mode: 'terminate', certificate: 'auto' },
        security: environments.production.security
      },
      name: 'Production Environment'
    },
    // Staging environment
    {
      match: { 
        ports: [80, 443],
        domains: 'staging.example.com'
      },
      action: {
        type: 'forward',
        target: environments.staging.target,
        tls: { mode: 'terminate', certificate: 'auto' },
        security: environments.staging.security
      },
      name: 'Staging Environment'
    },
    // Development environment
    {
      match: { 
        ports: [80, 443],
        domains: 'dev.example.com'
      },
      action: {
        type: 'forward',
        target: environments.development.target,
        tls: { mode: 'terminate', certificate: 'auto' },
        security: environments.development.security
      },
      name: 'Development Environment'
    }
  ],
  acme: { enabled: true, email: 'admin@example.com' }
});
```

## Implementation Strategy

### Code Organization

1. **New Files**:
   - `route-manager.ts` (core routing engine)
   - `route-types.ts` (interface definitions)
   - `route-helpers.ts` (helper functions)
   - `route-matcher.ts` (matching logic)
   - `template-engine.ts` (for variable substitution)

2. **Modified Files**:
   - `smart-proxy.ts` (update to use route-based configuration)
   - `connection-handler.ts` (simplify using route-based approach)
   - Replace `port-range-manager.ts` and `domain-config-manager.ts`

### Backward Compatibility

The backward compatibility layer will convert the legacy configuration to the new format:

```typescript
function convertLegacyConfig(legacy: ILegacySmartProxyOptions): ISmartProxyOptions {
  const routes: IRouteConfig[] = [];
  
  // Convert main port configuration
  if (legacy.fromPort) {
    // Add main listener for fromPort
    routes.push({
      match: { ports: legacy.fromPort },
      action: {
        type: 'forward',
        target: { host: legacy.targetIP || 'localhost', port: legacy.toPort },
        tls: { mode: legacy.sniEnabled ? 'passthrough' : 'terminate' }
      },
      name: 'Main Listener (Legacy)'
    });
    
    // If ACME is enabled, add HTTP listener for challenges
    if (legacy.acme?.enabled) {
      routes.push({
        match: { ports: 80 },
        action: {
          type: 'forward',
          target: { host: 'localhost', port: 80 },
          // Special flag for ACME handler
          acmeEnabled: true
        },
        name: 'ACME Challenge Handler (Legacy)'
      });
    }
  }
  
  // Convert domain configs
  if (legacy.domainConfigs) {
    for (const domainConfig of legacy.domainConfigs) {
      const { domains, forwarding } = domainConfig;
      
      // Determine action based on forwarding type
      let action: Partial<IRouteAction> = {
        type: 'forward',
        target: {
          host: forwarding.target.host,
          port: forwarding.target.port
        }
      };
      
      // Set TLS mode based on forwarding type
      switch (forwarding.type) {
        case 'http-only':
          // No TLS
          break;
        case 'https-passthrough':
          action.tls = { mode: 'passthrough' };
          break;
        case 'https-terminate-to-http':
          action.tls = { 
            mode: 'terminate',
            certificate: forwarding.https?.customCert || 'auto'
          };
          break;
        case 'https-terminate-to-https':
          action.tls = { 
            mode: 'terminate-and-reencrypt',
            certificate: forwarding.https?.customCert || 'auto'
          };
          break;
      }
      
      // Security settings
      if (forwarding.security) {
        action.security = forwarding.security;
      }
      
      // Add HTTP redirect if needed
      if (forwarding.http?.redirectToHttps) {
        routes.push({
          match: { ports: 80, domains },
          action: {
            type: 'redirect',
            redirect: { to: 'https://{domain}{path}', status: 301 }
          },
          name: `HTTP Redirect for ${domains.join(', ')} (Legacy)`
        });
      }
      
      // Add main route
      routes.push({
        match: { 
          ports: forwarding.type.startsWith('https') ? 443 : 80,
          domains
        },
        action: action as IRouteAction,
        name: `Route for ${domains.join(', ')} (Legacy)`
      });
      
      // Add port ranges if specified
      if (forwarding.advanced?.portRanges) {
        for (const range of forwarding.advanced.portRanges) {
          routes.push({
            match: {
              ports: { from: range.from, to: range.to },
              domains
            },
            action: action as IRouteAction,
            name: `Port Range ${range.from}-${range.to} for ${domains.join(', ')} (Legacy)`
          });
        }
      }
    }
  }
  
  // Global port ranges
  if (legacy.globalPortRanges) {
    for (const range of legacy.globalPortRanges) {
      routes.push({
        match: { ports: { from: range.from, to: range.to } },
        action: {
          type: 'forward',
          target: { 
            host: legacy.targetIP || 'localhost',
            port: legacy.forwardAllGlobalRanges ? 0 : legacy.toPort,
            preservePort: !!legacy.forwardAllGlobalRanges
          },
          tls: { mode: 'passthrough' }
        },
        name: `Global Port Range ${range.from}-${range.to} (Legacy)`
      });
    }
  }
  
  return {
    routes,
    defaults: {
      target: {
        host: legacy.targetIP || 'localhost',
        port: legacy.toPort
      }
    },
    acme: legacy.acme
  };
}
```

## Success Criteria

- All existing functionality works with the new route-based configuration
- Performance is equal or better than the current implementation
- Configuration is more intuitive and easier to understand
- New features can be added without breaking existing code
- Code is more maintainable with clear separation of concerns
- Migration from old configuration to new is straightforward