19.6.12

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartproxy",
-  "version": "19.6.8",
+  "version": "19.6.12",
   "private": false,
   "description": "A powerful proxy package with unified route-based configuration for high traffic management. Features include SSL/TLS support, flexible routing patterns, WebSocket handling, advanced security options, and automatic ACME certificate management.",
   "main": "dist_ts/index.js",

readme.byte-counting-audit.md

@@ -0,0 +1,169 @@
+# SmartProxy Byte Counting Audit Report
+
+## Executive Summary
+
+After a comprehensive audit of the SmartProxy codebase, I can confirm that **byte counting is implemented correctly** with no instances of double counting. Each byte transferred through the proxy is counted exactly once in each direction.
+
+## Byte Counting Implementation
+
+### 1. Core Tracking Mechanisms
+
+SmartProxy uses two complementary tracking systems:
+
+1. **Connection Records** (`IConnectionRecord`):
+   - `bytesReceived`: Total bytes received from client
+   - `bytesSent`: Total bytes sent to client
+
+2. **MetricsCollector**:
+   - Global throughput tracking via `ThroughputTracker`
+   - Per-connection byte tracking for route/IP metrics
+   - Called via `recordBytes(connectionId, bytesIn, bytesOut)`
+
+### 2. Where Bytes Are Counted
+
+Bytes are counted in only two files:
+
+#### a) `route-connection-handler.ts`
+- **Line 351**: TLS alert bytes when no SNI is provided
+- **Lines 1286-1301**: Data forwarding callbacks in `setupBidirectionalForwarding()`
+
+#### b) `http-proxy-bridge.ts`  
+- **Line 127**: Initial TLS chunk for HttpProxy connections
+- **Lines 142-154**: Data forwarding callbacks in `setupBidirectionalForwarding()`
+
+## Connection Flow Analysis
+
+### 1. Direct TCP Connection (No TLS)
+
+```
+Client → SmartProxy → Target Server
+```
+
+1. Connection arrives at `RouteConnectionHandler.handleConnection()`
+2. For non-TLS ports, immediately routes via `routeConnection()`
+3. `setupDirectConnection()` creates target connection
+4. `setupBidirectionalForwarding()` handles all data transfer:
+   - `onClientData`: `bytesReceived += chunk.length` + `recordBytes(chunk.length, 0)`
+   - `onServerData`: `bytesSent += chunk.length` + `recordBytes(0, chunk.length)`
+
+**Result**: ✅ Each byte counted exactly once
+
+### 2. TLS Passthrough Connection
+
+```
+Client (TLS) → SmartProxy → Target Server (TLS)
+```
+
+1. Connection waits for initial data to detect TLS
+2. TLS handshake detected, SNI extracted
+3. Route matched, `setupDirectConnection()` called
+4. Initial chunk stored in `pendingData` (NOT counted yet)
+5. On target connect, `pendingData` written to target (still not counted)
+6. `setupBidirectionalForwarding()` counts ALL bytes including initial chunk
+
+**Result**: ✅ Each byte counted exactly once
+
+### 3. TLS Termination via HttpProxy
+
+```
+Client (TLS) → SmartProxy → HttpProxy (localhost) → Target Server
+```
+
+1. TLS connection detected with `tls.mode = "terminate"`
+2. `forwardToHttpProxy()` called:
+   - Initial chunk: `bytesReceived += chunk.length` + `recordBytes(chunk.length, 0)`
+3. Proxy connection created to HttpProxy on localhost
+4. `setupBidirectionalForwarding()` handles subsequent data
+
+**Result**: ✅ Each byte counted exactly once
+
+### 4. HTTP Connection via HttpProxy
+
+```
+Client (HTTP) → SmartProxy → HttpProxy (localhost) → Target Server
+```
+
+1. Connection on configured HTTP port (`useHttpProxy` ports)
+2. Same flow as TLS termination
+3. All byte counting identical to TLS termination
+
+**Result**: ✅ Each byte counted exactly once
+
+### 5. NFTables Forwarding
+
+```
+Client → [Kernel NFTables] → Target Server
+```
+
+1. Connection detected, route matched with `forwardingEngine: 'nftables'`
+2. Connection marked as `usingNetworkProxy = true`
+3. NO application-level forwarding (kernel handles packet routing)
+4. NO byte counting in application layer
+
+**Result**: ✅ No counting (correct - kernel handles everything)
+
+## Special Cases
+
+### PROXY Protocol
+- PROXY protocol headers sent to backend servers are NOT counted in client metrics
+- Only actual client data is counted
+- **Correct behavior**: Protocol overhead is not client data
+
+### TLS Alerts
+- TLS alerts (e.g., for missing SNI) are counted as sent bytes
+- **Correct behavior**: Alerts are actual data sent to the client
+
+### Initial Chunks
+- **Direct connections**: Stored in `pendingData`, counted when forwarded
+- **HttpProxy connections**: Counted immediately upon receipt
+- **Both approaches**: Count each byte exactly once
+
+## Verification Methodology
+
+1. **Code Analysis**: Searched for all instances of:
+   - `bytesReceived +=` and `bytesSent +=`
+   - `recordBytes()` calls
+   - Data forwarding implementations
+
+2. **Flow Tracing**: Followed data path for each connection type from entry to exit
+
+3. **Handler Review**: Examined all forwarding handlers to ensure no additional counting
+
+## Findings
+
+### ✅ No Double Counting Detected
+
+- Each byte is counted exactly once in the direction it flows
+- Connection records and metrics are updated consistently
+- No overlapping or duplicate counting logic found
+
+### Areas of Excellence
+
+1. **Centralized Counting**: All byte counting happens in just two files
+2. **Consistent Pattern**: Uses `setupBidirectionalForwarding()` with callbacks
+3. **Clear Separation**: Forwarding handlers don't interfere with proxy metrics
+
+## Recommendations
+
+1. **Debug Logging**: Add optional debug logging to verify byte counts in production:
+   ```typescript
+   if (settings.debugByteCount) {
+     logger.log('debug', `Bytes counted: ${connectionId} +${bytes} (total: ${record.bytesReceived})`);
+   }
+   ```
+
+2. **Unit Tests**: Create specific tests to ensure byte counting accuracy:
+   - Test initial chunk handling
+   - Test PROXY protocol overhead exclusion
+   - Test HttpProxy forwarding accuracy
+
+3. **Protocol Overhead Tracking**: Consider separately tracking:
+   - PROXY protocol headers
+   - TLS handshake bytes
+   - HTTP headers vs body
+
+4. **NFTables Documentation**: Clearly document that NFTables-forwarded connections are not included in application metrics
+
+## Conclusion
+
+SmartProxy's byte counting implementation is **robust and accurate**. The design ensures that each byte is counted exactly once, with clear separation between connection tracking and metrics collection. No remediation is required.

readme.hints.md

@@ -0,0 +1,145 @@
+# SmartProxy Development Hints
+
+## Byte Tracking and Metrics
+
+### Throughput Drift Issue (Fixed)
+
+**Problem**: Throughput numbers were gradually increasing over time for long-lived connections.
+
+**Root Cause**: The `byRoute()` and `byIP()` methods were dividing cumulative total bytes (since connection start) by the window duration, causing rates to appear higher as connections aged:
+- Hour 1: 1GB total / 60s = 17 MB/s ✓
+- Hour 2: 2GB total / 60s = 34 MB/s ✗ (appears doubled!)
+- Hour 3: 3GB total / 60s = 50 MB/s ✗ (keeps rising!)
+
+**Solution**: Implemented dedicated ThroughputTracker instances for each route and IP address:
+- Each route and IP gets its own throughput tracker with per-second sampling
+- Samples are taken every second and stored in a circular buffer
+- Rate calculations use actual samples within the requested window
+- Default window is now 1 second for real-time accuracy
+
+### What Gets Counted (Network Interface Throughput)
+
+The byte tracking is designed to match network interface throughput (what Unifi/network monitoring tools show):
+
+**Counted bytes include:**
+- All application data
+- TLS handshakes and protocol overhead
+- TLS record headers and encryption padding
+- HTTP headers and protocol data
+- WebSocket frames and protocol overhead
+- TLS alerts sent to clients
+
+**NOT counted:**
+- PROXY protocol headers (sent to backend, not client)
+- TCP/IP headers (handled by OS, not visible at application layer)
+
+**Byte direction:**
+- `bytesReceived`: All bytes received FROM the client on the incoming connection
+- `bytesSent`: All bytes sent TO the client on the incoming connection
+- Backend connections are separate and not mixed with client metrics
+
+### Double Counting Issue (Fixed)
+
+**Problem**: Initial data chunks were being counted twice in the byte tracking:
+1. Once when stored in `pendingData` in `setupDirectConnection()`
+2. Again when the data flowed through bidirectional forwarding
+
+**Solution**: Removed the byte counting when storing initial chunks. Bytes are now only counted when they actually flow through the `setupBidirectionalForwarding()` callbacks.
+
+### HttpProxy Metrics (Fixed)
+
+**Problem**: HttpProxy forwarding was updating connection record byte counts but not calling `metricsCollector.recordBytes()`, resulting in missing throughput data.
+
+**Solution**: Added `metricsCollector.recordBytes()` calls to the HttpProxy bidirectional forwarding callbacks.
+
+### Metrics Architecture
+
+The metrics system has multiple layers:
+1. **Connection Records** (`record.bytesReceived/bytesSent`): Track total bytes per connection
+2. **Global ThroughputTracker**: Accumulates bytes between samples for overall rate calculations
+3. **Per-Route ThroughputTrackers**: Dedicated tracker for each route with per-second sampling
+4. **Per-IP ThroughputTrackers**: Dedicated tracker for each IP with per-second sampling
+5. **connectionByteTrackers**: Track cumulative bytes and metadata for active connections
+
+Key features:
+- All throughput trackers sample every second (1Hz)
+- Each tracker maintains a circular buffer of samples (default: 1 hour retention)
+- Rate calculations are accurate for any requested window (default: 1 second)
+- All byte counting happens exactly once at the data flow point
+- Unused route/IP trackers are automatically cleaned up when connections close
+
+### Understanding "High" Byte Counts
+
+If byte counts seem high compared to actual application data, remember:
+- TLS handshakes can be 1-5KB depending on cipher suites and certificates
+- Each TLS record has 5 bytes of header overhead
+- TLS encryption adds 16-48 bytes of padding/MAC per record
+- HTTP/2 has additional framing overhead
+- WebSocket has frame headers (2-14 bytes per message)
+
+This overhead is real network traffic and should be counted for accurate throughput metrics.
+
+### Byte Counting Paths
+
+There are two mutually exclusive paths for connections:
+
+1. **Direct forwarding** (route-connection-handler.ts):
+   - Used for TCP passthrough, TLS passthrough, and direct connections
+   - Bytes counted in `setupBidirectionalForwarding` callbacks
+   - Initial chunk NOT counted separately (flows through bidirectional forwarding)
+
+2. **HttpProxy forwarding** (http-proxy-bridge.ts):
+   - Used for TLS termination (terminate, terminate-and-reencrypt)
+   - Initial chunk counted when written to proxy
+   - All subsequent bytes counted in `setupBidirectionalForwarding` callbacks
+   - This is the ONLY counting point for these connections
+
+### Byte Counting Audit (2025-01-06)
+
+A comprehensive audit was performed to verify byte counting accuracy:
+
+**Audit Results:**
+- ✅ No double counting detected in any connection flow
+- ✅ Each byte counted exactly once in each direction
+- ✅ Connection records and metrics updated consistently
+- ✅ PROXY protocol headers correctly excluded from client metrics
+- ✅ NFTables forwarded connections correctly not counted (kernel handles)
+
+**Key Implementation Points:**
+- All byte counting happens in only 2 files: `route-connection-handler.ts` and `http-proxy-bridge.ts`
+- Both use the same pattern: increment `record.bytesReceived/Sent` AND call `metricsCollector.recordBytes()`
+- Initial chunks handled correctly: stored but not counted until forwarded
+- TLS alerts counted as sent bytes (correct - they are sent to client)
+
+For full audit details, see `readme.byte-counting-audit.md`
+
+## Connection Cleanup
+
+### Zombie Connection Detection
+
+The connection manager performs comprehensive zombie detection every 10 seconds:
+- **Full zombies**: Both incoming and outgoing sockets destroyed but connection not cleaned up
+- **Half zombies**: One socket destroyed, grace period expired (5 minutes for TLS, 30 seconds for non-TLS)
+- **Stuck connections**: Data received but none sent back after threshold (5 minutes for TLS, 60 seconds for non-TLS)
+
+### Cleanup Queue
+
+Connections are cleaned up through a batched queue system:
+- Batch size: 100 connections
+- Processing triggered immediately when batch size reached
+- Otherwise processed after 100ms delay
+- Prevents overwhelming the system during mass disconnections
+
+## Keep-Alive Handling
+
+Keep-alive connections receive special treatment based on `keepAliveTreatment` setting:
+- **standard**: Normal timeout applies
+- **extended**: Timeout multiplied by `keepAliveInactivityMultiplier` (default 6x)
+- **immortal**: No timeout, connections persist indefinitely
+
+## PROXY Protocol
+
+The system supports both receiving and sending PROXY protocol:
+- **Receiving**: Automatically detected from trusted proxy IPs (configured in `proxyIPs`)
+- **Sending**: Enabled per-route or globally via `sendProxyProtocol` setting
+- Real client IP is preserved and used for all connection tracking and security checks

ts/proxies/smart-proxy/http-proxy-bridge.ts

@@ -123,6 +123,11 @@ export class HttpProxyBridge {
     
     // Send initial chunk if present
     if (initialChunk) {
+      // Count the initial chunk bytes
+      record.bytesReceived += initialChunk.length;
+      if (this.smartProxy.metricsCollector) {
+        this.smartProxy.metricsCollector.recordBytes(record.id, initialChunk.length, 0);
+      }
       proxySocket.write(initialChunk);
     }
     
@@ -132,15 +137,21 @@ export class HttpProxyBridge {
     
     setupBidirectionalForwarding(underlyingSocket, proxySocket, {
       onClientData: (chunk) => {
-        // Update stats if needed
+        // Update stats - this is the ONLY place bytes are counted for HttpProxy connections
         if (record) {
           record.bytesReceived += chunk.length;
+          if (this.smartProxy.metricsCollector) {
+            this.smartProxy.metricsCollector.recordBytes(record.id, chunk.length, 0);
+          }
         }
       },
       onServerData: (chunk) => {
-        // Update stats if needed
+        // Update stats - this is the ONLY place bytes are counted for HttpProxy connections
         if (record) {
           record.bytesSent += chunk.length;
+          if (this.smartProxy.metricsCollector) {
+            this.smartProxy.metricsCollector.recordBytes(record.id, 0, chunk.length);
+          }
         }
       },
       onCleanup: (reason) => {

ts/proxies/smart-proxy/metrics-collector.ts

@@ -15,6 +15,8 @@ import { logger } from '../../core/utils/logger.js';
 export class MetricsCollector implements IMetrics {
   // Throughput tracking
   private throughputTracker: ThroughputTracker;
+  private routeThroughputTrackers = new Map<string, ThroughputTracker>();
+  private ipThroughputTrackers = new Map<string, ThroughputTracker>();
   
   // Request tracking
   private requestTimestamps: number[] = [];
@@ -119,78 +121,28 @@ export class MetricsCollector implements IMetrics {
       return this.throughputTracker.getHistory(seconds);
     },
     
-    byRoute: (windowSeconds: number = 60): Map<string, IThroughputData> => {
+    byRoute: (windowSeconds: number = 1): Map<string, IThroughputData> => {
       const routeThroughput = new Map<string, IThroughputData>();
-      const now = Date.now();
-      const windowStart = now - (windowSeconds * 1000);
       
-      // Aggregate bytes by route with proper time calculation
-      const routeData = new Map<string, { bytesIn: number; bytesOut: number; totalDuration: number }>();
-      
-      for (const [_, tracker] of this.connectionByteTrackers) {
-        // Only include connections that were active within the window
-        if (tracker.lastUpdate > windowStart || tracker.startTime > windowStart) {
-          // Calculate the actual duration this connection was active within the window
-          const connectionStart = Math.max(tracker.startTime, windowStart);
-          const connectionEnd = tracker.lastUpdate;
-          const durationInWindow = (connectionEnd - connectionStart) / 1000; // Convert to seconds
-          
-          if (durationInWindow > 0) {
-            const current = routeData.get(tracker.routeName) || { bytesIn: 0, bytesOut: 0, totalDuration: 0 };
-            current.bytesIn += tracker.bytesIn;
-            current.bytesOut += tracker.bytesOut;
-            current.totalDuration += durationInWindow;
-            routeData.set(tracker.routeName, current);
-          }
-        }
-      }
-      
-      // Convert to rates (bytes per second)
-      for (const [route, data] of routeData) {
-        if (data.totalDuration > 0) {
-          routeThroughput.set(route, {
-            in: Math.round(data.bytesIn / data.totalDuration),
-            out: Math.round(data.bytesOut / data.totalDuration)
-          });
+      // Get throughput from each route's dedicated tracker
+      for (const [route, tracker] of this.routeThroughputTrackers) {
+        const rate = tracker.getRate(windowSeconds);
+        if (rate.in > 0 || rate.out > 0) {
+          routeThroughput.set(route, rate);
         }
       }
       
       return routeThroughput;
     },
     
-    byIP: (windowSeconds: number = 60): Map<string, IThroughputData> => {
+    byIP: (windowSeconds: number = 1): Map<string, IThroughputData> => {
       const ipThroughput = new Map<string, IThroughputData>();
-      const now = Date.now();
-      const windowStart = now - (windowSeconds * 1000);
       
-      // Aggregate bytes by IP with proper time calculation
-      const ipData = new Map<string, { bytesIn: number; bytesOut: number; totalDuration: number }>();
-      
-      for (const [_, tracker] of this.connectionByteTrackers) {
-        // Only include connections that were active within the window
-        if (tracker.lastUpdate > windowStart || tracker.startTime > windowStart) {
-          // Calculate the actual duration this connection was active within the window
-          const connectionStart = Math.max(tracker.startTime, windowStart);
-          const connectionEnd = tracker.lastUpdate;
-          const durationInWindow = (connectionEnd - connectionStart) / 1000; // Convert to seconds
-          
-          if (durationInWindow > 0) {
-            const current = ipData.get(tracker.remoteIP) || { bytesIn: 0, bytesOut: 0, totalDuration: 0 };
-            current.bytesIn += tracker.bytesIn;
-            current.bytesOut += tracker.bytesOut;
-            current.totalDuration += durationInWindow;
-            ipData.set(tracker.remoteIP, current);
-          }
-        }
-      }
-      
-      // Convert to rates (bytes per second)
-      for (const [ip, data] of ipData) {
-        if (data.totalDuration > 0) {
-          ipThroughput.set(ip, {
-            in: Math.round(data.bytesIn / data.totalDuration),
-            out: Math.round(data.bytesOut / data.totalDuration)
-          });
+      // Get throughput from each IP's dedicated tracker
+      for (const [ip, tracker] of this.ipThroughputTrackers) {
+        const rate = tracker.getRate(windowSeconds);
+        if (rate.in > 0 || rate.out > 0) {
+          ipThroughput.set(ip, rate);
         }
       }
       
@@ -323,6 +275,22 @@ export class MetricsCollector implements IMetrics {
       tracker.bytesIn += bytesIn;
       tracker.bytesOut += bytesOut;
       tracker.lastUpdate = Date.now();
+      
+      // Update per-route throughput tracker
+      let routeTracker = this.routeThroughputTrackers.get(tracker.routeName);
+      if (!routeTracker) {
+        routeTracker = new ThroughputTracker(this.retentionSeconds);
+        this.routeThroughputTrackers.set(tracker.routeName, routeTracker);
+      }
+      routeTracker.recordBytes(bytesIn, bytesOut);
+      
+      // Update per-IP throughput tracker
+      let ipTracker = this.ipThroughputTrackers.get(tracker.remoteIP);
+      if (!ipTracker) {
+        ipTracker = new ThroughputTracker(this.retentionSeconds);
+        this.ipThroughputTrackers.set(tracker.remoteIP, ipTracker);
+      }
+      ipTracker.recordBytes(bytesIn, bytesOut);
     }
   }
   
@@ -343,8 +311,19 @@ export class MetricsCollector implements IMetrics {
     
     // Start periodic sampling
     this.samplingInterval = setInterval(() => {
+      // Sample global throughput
       this.throughputTracker.takeSample();
       
+      // Sample per-route throughput
+      for (const [_, tracker] of this.routeThroughputTrackers) {
+        tracker.takeSample();
+      }
+      
+      // Sample per-IP throughput
+      for (const [_, tracker] of this.ipThroughputTrackers) {
+        tracker.takeSample();
+      }
+      
       // Clean up old connection trackers (connections closed more than 5 minutes ago)
       const cutoff = Date.now() - 300000;
       for (const [id, tracker] of this.connectionByteTrackers) {
@@ -352,6 +331,22 @@ export class MetricsCollector implements IMetrics {
           this.connectionByteTrackers.delete(id);
         }
       }
+      
+      // Clean up unused route trackers
+      const activeRoutes = new Set(Array.from(this.connectionByteTrackers.values()).map(t => t.routeName));
+      for (const [route, _] of this.routeThroughputTrackers) {
+        if (!activeRoutes.has(route)) {
+          this.routeThroughputTrackers.delete(route);
+        }
+      }
+      
+      // Clean up unused IP trackers
+      const activeIPs = new Set(Array.from(this.connectionByteTrackers.values()).map(t => t.remoteIP));
+      for (const [ip, _] of this.ipThroughputTrackers) {
+        if (!activeIPs.has(ip)) {
+          this.ipThroughputTrackers.delete(ip);
+        }
+      }
     }, this.sampleIntervalMs);
     
     // Subscribe to new connections

ts/proxies/smart-proxy/models/metrics-types.ts

@@ -49,8 +49,8 @@ export interface IMetrics {
     average(): IThroughputData;      // Last 60 seconds
     custom(seconds: number): IThroughputData;
     history(seconds: number): Array<IThroughputHistoryPoint>;
-    byRoute(windowSeconds?: number): Map<string, IThroughputData>;
-    byIP(windowSeconds?: number): Map<string, IThroughputData>;
+    byRoute(windowSeconds?: number): Map<string, IThroughputData>;  // Default: 1 second
+    byIP(windowSeconds?: number): Map<string, IThroughputData>;     // Default: 1 second
   };
   
   // Request metrics

ts/proxies/smart-proxy/route-connection-handler.ts

@@ -347,6 +347,12 @@ export class RouteConnectionHandler {
             }
             const alert = Buffer.from([0x15, 0x03, 0x03, 0x00, 0x02, 0x01, 0x70]);
             try {
+              // Count the alert bytes being sent
+              record.bytesSent += alert.length;
+              if (this.smartProxy.metricsCollector) {
+                this.smartProxy.metricsCollector.recordBytes(record.id, 0, alert.length);
+              }
+              
               socket.cork();
               socket.write(alert);
               socket.uncork();
@@ -1114,14 +1120,9 @@ export class RouteConnectionHandler {
 
     // Store initial data if provided
     if (initialChunk) {
-      record.bytesReceived += initialChunk.length;
+      // Don't count bytes here - they will be counted when actually forwarded through bidirectional forwarding
       record.pendingData.push(Buffer.from(initialChunk));
       record.pendingDataSize = initialChunk.length;
-      
-      // Record bytes for metrics
-      if (this.smartProxy.metricsCollector) {
-        this.smartProxy.metricsCollector.recordBytes(record.id, initialChunk.length, 0);
-      }
     }
 
     // Create the target socket with immediate error handling
@@ -1213,6 +1214,9 @@ export class RouteConnectionHandler {
             
             const proxyHeader = ProxyProtocolParser.generate(proxyInfo);
             
+            // Note: PROXY protocol headers are sent to the backend, not to the client
+            // They are internal protocol overhead and shouldn't be counted in client-facing metrics
+            
             // Send PROXY protocol header first
             await new Promise<void>((resolve, reject) => {
               targetSocket.write(proxyHeader, (err) => {