smartproxy/Connection-Termination-Issues.md

# Connection Termination Issues and Solutions in SmartProxy/NetworkProxy

## Common Connection Termination Scenarios

### 1. Normal Connection Closure

**Flow**:
- Client or server initiates graceful close
- 'close' event triggers cleanup
- Connection removed from tracking
- Resources freed

**Code Path**:
```typescript
// In ConnectionManager
handleClose(side: 'incoming' | 'outgoing', record: IConnectionRecord) {
  record.incomingTerminationReason = 'normal';
  this.initiateCleanupOnce(record, 'closed_' + side);
}
```

### 2. Error-Based Termination

**Common Errors**:
- ECONNRESET: Connection reset by peer
- ETIMEDOUT: Connection timed out
- ECONNREFUSED: Connection refused
- EHOSTUNREACH: Host unreachable

**Handling**:
```typescript
handleError(side: 'incoming' | 'outgoing', record: IConnectionRecord) {
  return (err: Error) => {
    const code = (err as any).code;
    let reason = 'error';
    
    if (code === 'ECONNRESET') {
      reason = 'econnreset';
    } else if (code === 'ETIMEDOUT') {
      reason = 'etimedout';
    }
    
    this.initiateCleanupOnce(record, reason);
  };
}
```

### 3. Inactivity Timeout

**Detection**:
```typescript
performInactivityCheck(): void {
  const now = Date.now();
  for (const record of this.connectionRecords.values()) {
    const inactivityTime = now - record.lastActivity;
    if (inactivityTime > effectiveTimeout) {
      this.cleanupConnection(record, 'inactivity');
    }
  }
}
```

**Special Cases**:
- Keep-alive connections get extended timeouts
- "Immortal" connections bypass inactivity checks
- Warning issued before closure for keep-alive connections

### 4. NFTables-Handled Connections

**Special Handling**:
```typescript
if (route.action.forwardingEngine === 'nftables') {
  socket.end();
  record.nftablesHandled = true;
  this.connectionManager.initiateCleanupOnce(record, 'nftables_handled');
  return;
}
```

These connections are:
- Handled at kernel level
- Closed immediately at application level
- Tracked for metrics only

### 5. NetworkProxy Bridge Termination

**Bridge Cleanup**:
```typescript
const cleanup = (reason: string) => {
  socket.unpipe(proxySocket);
  proxySocket.unpipe(socket);
  proxySocket.destroy();
  cleanupCallback(reason);
};

socket.on('end', () => cleanup('socket_end'));
socket.on('error', () => cleanup('socket_error'));
proxySocket.on('end', () => cleanup('proxy_end'));
proxySocket.on('error', () => cleanup('proxy_error'));
```

## Preventing Connection Leaks

### 1. Always Remove Event Listeners

```typescript
cleanupConnection(record: IConnectionRecord, reason: string): void {
  if (record.incoming) {
    record.incoming.removeAllListeners('data');
    record.renegotiationHandler = undefined;
  }
}
```

### 2. Clear Timers

```typescript
if (record.cleanupTimer) {
  clearTimeout(record.cleanupTimer);
  record.cleanupTimer = undefined;
}
```

### 3. Proper Socket Cleanup

```typescript
private cleanupSocket(record: IConnectionRecord, side: string, socket: net.Socket): void {
  try {
    if (!socket.destroyed) {
      socket.end(); // Graceful
      setTimeout(() => {
        if (!socket.destroyed) {
          socket.destroy(); // Forced
        }
      }, 1000);
    }
  } catch (err) {
    console.log(`Error closing ${side} socket: ${err}`);
  }
}
```

### 4. Connection Record Cleanup

```typescript
// Clear pending data to prevent memory leaks
record.pendingData = [];
record.pendingDataSize = 0;

// Remove from tracking map
this.connectionRecords.delete(record.id);
```

## Monitoring and Debugging

### 1. Termination Statistics

```typescript
private terminationStats: {
  incoming: Record<string, number>;
  outgoing: Record<string, number>;
} = { incoming: {}, outgoing: {} };

incrementTerminationStat(side: 'incoming' | 'outgoing', reason: string): void {
  this.terminationStats[side][reason] = (this.terminationStats[side][reason] || 0) + 1;
}
```

### 2. Connection Logging

**Detailed Logging**:
```typescript
console.log(
  `[${record.id}] Connection from ${record.remoteIP} terminated (${reason}).` +
  ` Duration: ${prettyMs(duration)}, Bytes IN: ${bytesReceived}, OUT: ${bytesSent}`
);
```

### 3. Active Connection Tracking

```typescript
getConnectionCount(): number {
  return this.connectionRecords.size;
}

// In NetworkProxy
metrics = {
  activeConnections: this.connectedClients,
  portProxyConnections: this.portProxyConnections,
  tlsTerminatedConnections: this.tlsTerminatedConnections
};
```

## Best Practices for Connection Termination

1. **Always Use initiateCleanupOnce()**:
   - Prevents duplicate cleanup operations
   - Ensures proper termination reason tracking

2. **Handle All Socket Events**:
   - 'error', 'close', 'end' events
   - Both incoming and outgoing sockets

3. **Implement Proper Timeouts**:
   - Initial data timeout
   - Inactivity timeout  
   - Maximum connection lifetime

4. **Track Resources**:
   - Connection records
   - Socket maps
   - Timer references

5. **Log Termination Reasons**:
   - Helps debug connection issues
   - Provides metrics for monitoring

6. **Graceful Shutdown**:
   - Try socket.end() before socket.destroy()
   - Allow time for graceful closure

7. **Memory Management**:
   - Clear pending data buffers
   - Remove event listeners
   - Delete connection records

## Common Issues and Solutions

### Issue: Memory Leaks from Event Listeners
**Solution**: Always call removeAllListeners() during cleanup

### Issue: Orphaned Connections
**Solution**: Implement multiple cleanup triggers (timeout, error, close)

### Issue: Duplicate Cleanup Operations
**Solution**: Use connectionClosed flag and initiateCleanupOnce()

### Issue: Hanging Connections
**Solution**: Implement inactivity checks and maximum lifetime limits

### Issue: Resource Exhaustion
**Solution**: Track connection counts and implement limits

### Issue: Lost Data During Cleanup
**Solution**: Use proper unpipe operations and graceful shutdown

### Issue: Debugging Connection Issues
**Solution**: Track termination reasons and maintain detailed logs