smartproxy/readme.plan.md

# NFTables-SmartProxy Integration Plan

## Overview

This document outlines a comprehensive plan to integrate the existing NFTables functionality with the SmartProxy core to provide advanced network-level routing capabilities. The NFTables proxy already exists in the codebase but is not fully integrated with the SmartProxy routing system. This integration will allow SmartProxy to leverage the power of Linux's NFTables firewall system for high-performance port forwarding, load balancing, and security filtering.

## Current State

1. **NFTablesProxy**: A standalone implementation exists in `ts/proxies/nftables-proxy/` with its own configuration and API.
2. **SmartProxy**: The main routing system with route-based configuration.
3. **No Integration**: Currently, these systems operate independently with no shared configuration or coordination.

## Goals

1. Create a unified configuration system where SmartProxy routes can specify NFTables-based forwarding.
2. Allow SmartProxy to dynamically provision and manage NFTables rules based on route configuration.
3. Support advanced filtering and security rules through NFTables for better performance.
4. Ensure backward compatibility with existing setups.
5. Provide metrics integration between the systems.

## Implementation Plan

### Phase 1: Route Configuration Schema Extension

1. **Extend Route Configuration Schema**:
   - Add new `forwardingEngine` option to IRouteAction to specify the forwarding implementation.
   - Support values: 'node' (current NodeJS implementation) and 'nftables' (Linux NFTables).
   - Add NFTables-specific configuration options to IRouteAction.

2. **Update Type Definitions**:
   ```typescript
   // In route-types.ts
   export interface IRouteAction {
     type: 'forward' | 'redirect' | 'block';
     target?: IRouteTarget;
     security?: IRouteSecurity;
     options?: IRouteOptions;
     tls?: IRouteTlsOptions;
     forwardingEngine?: 'node' | 'nftables';  // New field
     nftables?: INfTablesOptions;             // New field
   }

   export interface INfTablesOptions {
     preserveSourceIP?: boolean;
     protocol?: 'tcp' | 'udp' | 'all';
     maxRate?: string;                // QoS rate limiting
     priority?: number;               // QoS priority
     tableName?: string;              // Optional custom table name
     useIPSets?: boolean;             // Use IP sets for performance
     useAdvancedNAT?: boolean;        // Use connection tracking
   }
   ```

### Phase 2: NFTablesManager Implementation

1. **Create NFTablesManager Class**:
   - Create a new class to manage NFTables rules based on SmartProxy routes.
   - Add methods to create, update, and remove NFTables rules.
   - Design a rule naming scheme to track which rules correspond to which routes.

2. **Implementation**:
   ```typescript
   // In ts/proxies/smart-proxy/nftables-manager.ts
   export class NFTablesManager {
     private rulesMap: Map<string, NfTablesProxy> = new Map();
     
     constructor(private options: ISmartProxyOptions) {}
     
     /**
      * Provision NFTables rules for a route
      */
     public async provisionRoute(route: IRouteConfig): Promise<boolean> {
       // Generate a unique ID for this route
       const routeId = this.generateRouteId(route);
       
       // Skip if route doesn't use NFTables
       if (route.action.forwardingEngine !== 'nftables') {
         return true;
       }
       
       // Create NFTables options from route configuration
       const nftOptions = this.createNfTablesOptions(route);
       
       // Create and start an NFTablesProxy instance
       const proxy = new NfTablesProxy(nftOptions);
       
       try {
         await proxy.start();
         this.rulesMap.set(routeId, proxy);
         return true;
       } catch (err) {
         console.error(`Failed to provision NFTables rules for route ${route.name}: ${err.message}`);
         return false;
       }
     }
     
     /**
      * Remove NFTables rules for a route
      */
     public async deprovisionRoute(route: IRouteConfig): Promise<boolean> {
       const routeId = this.generateRouteId(route);
       
       const proxy = this.rulesMap.get(routeId);
       if (!proxy) {
         return true; // Nothing to remove
       }
       
       try {
         await proxy.stop();
         this.rulesMap.delete(routeId);
         return true;
       } catch (err) {
         console.error(`Failed to deprovision NFTables rules for route ${route.name}: ${err.message}`);
         return false;
       }
     }
     
     /**
      * Update NFTables rules when route changes
      */
     public async updateRoute(oldRoute: IRouteConfig, newRoute: IRouteConfig): Promise<boolean> {
       // Remove old rules and add new ones
       await this.deprovisionRoute(oldRoute);
       return this.provisionRoute(newRoute);
     }
     
     /**
      * Generate a unique ID for a route
      */
     private generateRouteId(route: IRouteConfig): string {
       // Generate a unique ID based on route properties
       return `${route.name || 'unnamed'}-${JSON.stringify(route.match)}-${Date.now()}`;
     }
     
     /**
      * Create NFTablesProxy options from a route configuration
      */
     private createNfTablesOptions(route: IRouteConfig): NfTableProxyOptions {
       const { action } = route;
       
       // Ensure we have a target
       if (!action.target) {
         throw new Error('Route must have a target to use NFTables forwarding');
       }
       
       // Convert port specifications
       const fromPorts = this.expandPortRange(route.match.ports);
       
       // Determine target port
       let toPorts;
       if (action.target.port === 'preserve') {
         // 'preserve' means use the same ports as the source
         toPorts = fromPorts;
       } else if (typeof action.target.port === 'function') {
         // For function-based ports, we can't determine at setup time
         // Use the "preserve" approach and let NFTables handle it
         toPorts = fromPorts;
       } else {
         toPorts = action.target.port;
       }
       
       // Create options
       const options: NfTableProxyOptions = {
         fromPort: fromPorts,
         toPort: toPorts,
         toHost: typeof action.target.host === 'function' 
           ? 'localhost' // Can't determine at setup time, use localhost
           : (Array.isArray(action.target.host) 
              ? action.target.host[0] // Use first host for now
              : action.target.host),
         protocol: action.nftables?.protocol || 'tcp',
         preserveSourceIP: action.nftables?.preserveSourceIP,
         useIPSets: action.nftables?.useIPSets !== false,
         useAdvancedNAT: action.nftables?.useAdvancedNAT,
         enableLogging: this.options.enableDetailedLogging,
         deleteOnExit: true,
         tableName: action.nftables?.tableName || 'smartproxy'
       };
       
       // Add security-related options
       if (action.security?.ipAllowList?.length) {
         options.allowedSourceIPs = action.security.ipAllowList;
       }
       
       if (action.security?.ipBlockList?.length) {
         options.bannedSourceIPs = action.security.ipBlockList;
       }
       
       // Add QoS options
       if (action.nftables?.maxRate || action.nftables?.priority) {
         options.qos = {
           enabled: true,
           maxRate: action.nftables.maxRate,
           priority: action.nftables.priority
         };
       }
       
       return options;
     }
     
     /**
      * Expand port range specifications
      */
     private expandPortRange(ports: TPortRange): number | PortRange | Array<number | PortRange> {
       // Use RouteManager's expandPortRange to convert to actual port numbers
       const routeManager = new RouteManager(this.options);
       
       // Process different port specifications
       if (typeof ports === 'number') {
         return ports;
       } else if (Array.isArray(ports)) {
         const result: Array<number | PortRange> = [];
         
         for (const item of ports) {
           if (typeof item === 'number') {
             result.push(item);
           } else if ('from' in item && 'to' in item) {
             result.push({ from: item.from, to: item.to });
           }
         }
         
         return result;
       } else if ('from' in ports && 'to' in ports) {
         return { from: ports.from, to: ports.to };
       }
       
       // Fallback
       return 80;
     }
     
     /**
      * Get status of all managed rules
      */
     public async getStatus(): Promise<Record<string, NfTablesStatus>> {
       const result: Record<string, NfTablesStatus> = {};
       
       for (const [routeId, proxy] of this.rulesMap.entries()) {
         result[routeId] = await proxy.getStatus();
       }
       
       return result;
     }
     
     /**
      * Stop all NFTables rules
      */
     public async stop(): Promise<void> {
       // Stop all NFTables proxies
       const stopPromises = Array.from(this.rulesMap.values()).map(proxy => proxy.stop());
       await Promise.all(stopPromises);
       
       this.rulesMap.clear();
     }
   }
   ```

### Phase 3: SmartProxy Integration

1. **Extend SmartProxy Class**:
   - Add NFTablesManager as a property of SmartProxy.
   - Hook into route configuration to provision NFTables rules.
   - Add methods to manage NFTables functionality.

2. **Implementation**:
   ```typescript
   // In ts/proxies/smart-proxy/smart-proxy.ts
   import { NFTablesManager } from './nftables-manager.js';

   export class SmartProxy {
     // Existing properties
     private nftablesManager: NFTablesManager;
     
     constructor(options: ISmartProxyOptions) {
       // Existing initialization
       
       // Initialize NFTablesManager
       this.nftablesManager = new NFTablesManager(options);
     }
     
     /**
      * Start the SmartProxy server
      */
     public async start(): Promise<void> {
       // Existing initialization
       
       // If we have routes, provision NFTables rules for them
       for (const route of this.settings.routes) {
         if (route.action.forwardingEngine === 'nftables') {
           await this.nftablesManager.provisionRoute(route);
         }
       }
       
       // Rest of existing start method
     }
     
     /**
      * Stop the SmartProxy server
      */
     public async stop(): Promise<void> {
       // Stop NFTablesManager first
       await this.nftablesManager.stop();
       
       // Rest of existing stop method
     }
     
     /**
      * Update routes
      */
     public async updateRoutes(routes: IRouteConfig[]): Promise<void> {
       // Get existing routes that use NFTables
       const oldNfTablesRoutes = this.settings.routes.filter(
         r => r.action.forwardingEngine === 'nftables'
       );
       
       // Get new routes that use NFTables
       const newNfTablesRoutes = routes.filter(
         r => r.action.forwardingEngine === 'nftables'
       );
       
       // Find routes to remove, update, or add
       for (const oldRoute of oldNfTablesRoutes) {
         const newRoute = newNfTablesRoutes.find(r => r.name === oldRoute.name);
         
         if (!newRoute) {
           // Route was removed
           await this.nftablesManager.deprovisionRoute(oldRoute);
         } else {
           // Route was updated
           await this.nftablesManager.updateRoute(oldRoute, newRoute);
         }
       }
       
       // Find new routes to add
       for (const newRoute of newNfTablesRoutes) {
         const oldRoute = oldNfTablesRoutes.find(r => r.name === newRoute.name);
         
         if (!oldRoute) {
           // New route
           await this.nftablesManager.provisionRoute(newRoute);
         }
       }
       
       // Update settings with the new routes
       this.settings.routes = routes;
       
       // Update route manager with new routes
       this.routeManager.updateRoutes(routes);
     }
     
     /**
      * Get NFTables status
      */
     public async getNfTablesStatus(): Promise<Record<string, NfTablesStatus>> {
       return this.nftablesManager.getStatus();
     }
   }
   ```

### Phase 4: Routing System Integration

1. **Extend the Route-Connection-Handler**:
   - Modify to check if a route uses NFTables.
   - Skip Node.js-based connection handling for NFTables routes.

2. **Implementation**:
   ```typescript
   // In ts/proxies/smart-proxy/route-connection-handler.ts
   export class RouteConnectionHandler {
     // Existing methods
     
     /**
      * Route the connection based on match criteria
      */
     private routeConnection(
       socket: plugins.net.Socket, 
       record: IConnectionRecord,
       serverName: string,
       initialChunk?: Buffer
     ): void {
       // Find matching route
       const routeMatch = this.routeManager.findMatchingRoute({
         port: record.localPort,
         domain: serverName,
         clientIp: record.remoteIP,
         path: undefined,
         tlsVersion: undefined
       });
       
       if (!routeMatch) {
         // Existing code for no matching route
         return;
       }
       
       const route = routeMatch.route;
       
       // Check if this route uses NFTables for forwarding
       if (route.action.forwardingEngine === 'nftables') {
         // For NFTables routes, we don't need to do anything at the application level
         // The packet is forwarded at the kernel level
         
         // Log the connection
         console.log(
           `[${record.id}] Connection forwarded by NFTables: ${record.remoteIP} -> port ${record.localPort}`
         );
         
         // Just close the socket in our application since it's handled at kernel level
         socket.end();
         this.connectionManager.initiateCleanupOnce(record, 'nftables_handled');
         return;
       }
       
       // Existing code for handling the route
     }
   }
   ```

### Phase 5: CLI and Configuration Helpers

1. **Add Helper Functions**:
   - Create helper functions for easy route creation with NFTables.
   - Update the route-helpers.ts utility file.

2. **Implementation**:
   ```typescript
   // In ts/proxies/smart-proxy/utils/route-helpers.ts
   
   /**
    * Create an NFTables-based route
    */
   export function createNfTablesRoute(
     nameOrDomains: string | string[],
     target: { host: string; port: number | 'preserve' },
     options: {
       ports?: TPortRange;
       protocol?: 'tcp' | 'udp' | 'all';
       preserveSourceIP?: boolean;
       allowedIps?: string[];
       maxRate?: string;
       priority?: number;
       useTls?: boolean;
     } = {}
   ): IRouteConfig {
     // Determine if this is a name or domain
     let name: string;
     let domains: string | string[];
     
     if (Array.isArray(nameOrDomains) || nameOrDomains.includes('.')) {
       domains = nameOrDomains;
       name = Array.isArray(nameOrDomains) ? nameOrDomains[0] : nameOrDomains;
     } else {
       name = nameOrDomains;
       domains = []; // No domains
     }
     
     const route: IRouteConfig = {
       name,
       match: {
         domains,
         ports: options.ports || 80
       },
       action: {
         type: 'forward',
         target: {
           host: target.host,
           port: target.port
         },
         forwardingEngine: 'nftables',
         nftables: {
           protocol: options.protocol || 'tcp',
           preserveSourceIP: options.preserveSourceIP,
           maxRate: options.maxRate,
           priority: options.priority
         }
       }
     };
     
     // Add security if allowed IPs are specified
     if (options.allowedIps?.length) {
       route.action.security = {
         ipAllowList: options.allowedIps
       };
     }
     
     // Add TLS options if needed
     if (options.useTls) {
       route.action.tls = {
         mode: 'passthrough'
       };
     }
     
     return route;
   }
   
   /**
    * Create an NFTables-based TLS termination route
    */
   export function createNfTablesTerminateRoute(
     nameOrDomains: string | string[],
     target: { host: string; port: number | 'preserve' },
     options: {
       ports?: TPortRange;
       protocol?: 'tcp' | 'udp' | 'all';
       preserveSourceIP?: boolean;
       allowedIps?: string[];
       maxRate?: string;
       priority?: number;
       certificate?: string | { cert: string; key: string };
     } = {}
   ): IRouteConfig {
     const route = createNfTablesRoute(
       nameOrDomains,
       target,
       {
         ...options,
         ports: options.ports || 443,
         useTls: false
       }
     );
     
     // Set TLS termination
     route.action.tls = {
       mode: 'terminate',
       certificate: options.certificate || 'auto'
     };
     
     return route;
   }
   ```

### Phase 6: Documentation and Testing

1. **Update Documentation**:
   - Add NFTables integration documentation to README and API docs.
   - Document the implementation and use cases.

2. **Test Cases**:
   - Create test cases for NFTables-based routing.
   - Test performance comparison with Node.js-based forwarding.
   - Test security features with IP allowlists/blocklists.

```typescript
// In test/test.nftables-integration.ts
import { SmartProxy } from '../ts/proxies/smart-proxy/index.js';
import { createNfTablesRoute } from '../ts/proxies/smart-proxy/utils/route-helpers.js';
import { expect, tap } from '@push.rocks/tapbundle';
import * as net from 'net';

// Test server and client utilities
let testServer: net.Server;
let smartProxy: SmartProxy;

const TEST_PORT = 4000;
const PROXY_PORT = 5000;
const TEST_DATA = 'Hello through NFTables!';

tap.test('setup NFTables integration test environment', async () => {
  // Create a test TCP server
  testServer = net.createServer((socket) => {
    socket.on('data', (data) => {
      socket.write(`Server says: ${data.toString()}`);
    });
  });
  
  await new Promise<void>((resolve) => {
    testServer.listen(TEST_PORT, () => {
      console.log(`Test server listening on port ${TEST_PORT}`);
      resolve();
    });
  });
  
  // Create SmartProxy with NFTables route
  smartProxy = new SmartProxy({
    routes: [
      createNfTablesRoute('test-nftables', {
        host: 'localhost',
        port: TEST_PORT
      }, {
        ports: PROXY_PORT,
        protocol: 'tcp'
      })
    ]
  });
  
  // Start the proxy
  await smartProxy.start();
});

tap.test('should forward TCP connections through NFTables', async () => {
  // Connect to the proxy port
  const client = new net.Socket();
  
  const response = await new Promise<string>((resolve, reject) => {
    let responseData = '';
    
    client.connect(PROXY_PORT, 'localhost', () => {
      client.write(TEST_DATA);
    });
    
    client.on('data', (data) => {
      responseData += data.toString();
      client.end();
    });
    
    client.on('end', () => {
      resolve(responseData);
    });
    
    client.on('error', (err) => {
      reject(err);
    });
  });
  
  expect(response).toEqual(`Server says: ${TEST_DATA}`);
});

tap.test('cleanup NFTables integration test environment', async () => {
  // Stop the proxy and test server
  await smartProxy.stop();
  
  await new Promise<void>((resolve) => {
    testServer.close(() => {
      resolve();
    });
  });
});

export default tap.start();
```

## Expected Benefits

1. **Performance**: NFTables operates at the kernel level, offering much higher performance than Node.js-based routing.
2. **Scalability**: Handle more connections with less CPU and memory usage.
3. **Security**: Leverage kernel-level security features for better protection.
4. **Integration**: Unified configuration model between application and network layers.
5. **Advanced Features**: Support for QoS, rate limiting, and other advanced networking features.

## Implementation Notes

- This integration requires root/sudo access to configure NFTables rules.
- Consider adding a capability check to gracefully fall back to Node.js routing if NFTables is not available.
- The NFTables integration should be optional and SmartProxy should continue to work without it.
- The integration provides a path for future extensions to other kernel-level networking features.

## Timeline

- Phase 1 (Route Configuration Schema): 1-2 days
- Phase 2 (NFTablesManager): 2-3 days
- Phase 3 (SmartProxy Integration): 1-2 days
- Phase 4 (Routing System Integration): 1 day
- Phase 5 (CLI and Helpers): 1 day
- Phase 6 (Documentation and Testing): 2 days

**Total Estimated Time: 8-11 days**