smartproxy/Connection-Termination-Issues.md

Connection Termination Issues and Solutions in SmartProxy/NetworkProxy

Common Connection Termination Scenarios

1. Normal Connection Closure

Flow:

• Client or server initiates graceful close
• 'close' event triggers cleanup
• Connection removed from tracking
• Resources freed

Code Path:

// In ConnectionManager
handleClose(side: 'incoming' | 'outgoing', record: IConnectionRecord) {
  record.incomingTerminationReason = 'normal';
  this.initiateCleanupOnce(record, 'closed_' + side);
}

2. Error-Based Termination

Common Errors:

• ECONNRESET: Connection reset by peer
• ETIMEDOUT: Connection timed out
• ECONNREFUSED: Connection refused
• EHOSTUNREACH: Host unreachable

Handling:

handleError(side: 'incoming' | 'outgoing', record: IConnectionRecord) {
  return (err: Error) => {
    const code = (err as any).code;
    let reason = 'error';
    
    if (code === 'ECONNRESET') {
      reason = 'econnreset';
    } else if (code === 'ETIMEDOUT') {
      reason = 'etimedout';
    }
    
    this.initiateCleanupOnce(record, reason);
  };
}

3. Inactivity Timeout

Detection:

performInactivityCheck(): void {
  const now = Date.now();
  for (const record of this.connectionRecords.values()) {
    const inactivityTime = now - record.lastActivity;
    if (inactivityTime > effectiveTimeout) {
      this.cleanupConnection(record, 'inactivity');
    }
  }
}

Special Cases:

• Keep-alive connections get extended timeouts
• "Immortal" connections bypass inactivity checks
• Warning issued before closure for keep-alive connections

4. NFTables-Handled Connections

Special Handling:

if (route.action.forwardingEngine === 'nftables') {
  socket.end();
  record.nftablesHandled = true;
  this.connectionManager.initiateCleanupOnce(record, 'nftables_handled');
  return;
}

These connections are:

• Handled at kernel level
• Closed immediately at application level
• Tracked for metrics only

5. NetworkProxy Bridge Termination

Bridge Cleanup:

const cleanup = (reason: string) => {
  socket.unpipe(proxySocket);
  proxySocket.unpipe(socket);
  proxySocket.destroy();
  cleanupCallback(reason);
};

socket.on('end', () => cleanup('socket_end'));
socket.on('error', () => cleanup('socket_error'));
proxySocket.on('end', () => cleanup('proxy_end'));
proxySocket.on('error', () => cleanup('proxy_error'));

Preventing Connection Leaks

1. Always Remove Event Listeners

cleanupConnection(record: IConnectionRecord, reason: string): void {
  if (record.incoming) {
    record.incoming.removeAllListeners('data');
    record.renegotiationHandler = undefined;
  }
}

2. Clear Timers

if (record.cleanupTimer) {
  clearTimeout(record.cleanupTimer);
  record.cleanupTimer = undefined;
}

3. Proper Socket Cleanup

private cleanupSocket(record: IConnectionRecord, side: string, socket: net.Socket): void {
  try {
    if (!socket.destroyed) {
      socket.end(); // Graceful
      setTimeout(() => {
        if (!socket.destroyed) {
          socket.destroy(); // Forced
        }
      }, 1000);
    }
  } catch (err) {
    console.log(`Error closing ${side} socket: ${err}`);
  }
}

4. Connection Record Cleanup

// Clear pending data to prevent memory leaks
record.pendingData = [];
record.pendingDataSize = 0;

// Remove from tracking map
this.connectionRecords.delete(record.id);

Monitoring and Debugging

1. Termination Statistics

private terminationStats: {
  incoming: Record<string, number>;
  outgoing: Record<string, number>;
} = { incoming: {}, outgoing: {} };

incrementTerminationStat(side: 'incoming' | 'outgoing', reason: string): void {
  this.terminationStats[side][reason] = (this.terminationStats[side][reason] || 0) + 1;
}

2. Connection Logging

Detailed Logging:

console.log(
  `[${record.id}] Connection from ${record.remoteIP} terminated (${reason}).` +
  ` Duration: ${prettyMs(duration)}, Bytes IN: ${bytesReceived}, OUT: ${bytesSent}`
);

3. Active Connection Tracking

getConnectionCount(): number {
  return this.connectionRecords.size;
}

// In NetworkProxy
metrics = {
  activeConnections: this.connectedClients,
  portProxyConnections: this.portProxyConnections,
  tlsTerminatedConnections: this.tlsTerminatedConnections
};

Best Practices for Connection Termination

1. Always Use initiateCleanupOnce():

   • Prevents duplicate cleanup operations
   • Ensures proper termination reason tracking

2. Handle All Socket Events:

   • 'error', 'close', 'end' events
   • Both incoming and outgoing sockets

3. Implement Proper Timeouts:

   • Initial data timeout
   • Inactivity timeout
   • Maximum connection lifetime

4. Track Resources:

   • Connection records
   • Socket maps
   • Timer references

5. Log Termination Reasons:

   • Helps debug connection issues
   • Provides metrics for monitoring

6. Graceful Shutdown:

   • Try socket.end() before socket.destroy()
   • Allow time for graceful closure

7. Memory Management:

   • Clear pending data buffers
   • Remove event listeners
   • Delete connection records

Common Issues and Solutions

Issue: Memory Leaks from Event Listeners

Solution: Always call removeAllListeners() during cleanup

Issue: Orphaned Connections

Solution: Implement multiple cleanup triggers (timeout, error, close)

Issue: Duplicate Cleanup Operations

Solution: Use connectionClosed flag and initiateCleanupOnce()

Issue: Hanging Connections

Solution: Implement inactivity checks and maximum lifetime limits

Issue: Resource Exhaustion

Solution: Track connection counts and implement limits

Issue: Lost Data During Cleanup

Solution: Use proper unpipe operations and graceful shutdown

Issue: Debugging Connection Issues

Solution: Track termination reasons and maintain detailed logs