Compare commits
8 Commits
Author | SHA1 | Date | |
---|---|---|---|
705a59413d | |||
e9723a8af9 | |||
300ab1a077 | |||
900942a263 | |||
d45485985a | |||
9fdc2d5069 | |||
37c87e8450 | |||
92b2f230ef |
@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@push.rocks/smartproxy",
|
||||
"version": "19.5.8",
|
||||
"version": "19.5.13",
|
||||
"private": false,
|
||||
"description": "A powerful proxy package with unified route-based configuration for high traffic management. Features include SSL/TLS support, flexible routing patterns, WebSocket handling, advanced security options, and automatic ACME certificate management.",
|
||||
"main": "dist_ts/index.js",
|
||||
|
@ -413,4 +413,55 @@ const routes: IRouteConfig[] = [{
|
||||
### 7. Next Steps (Remaining Work)
|
||||
- **Phase 2 (cont)**: Migrate components to use LifecycleComponent
|
||||
- **Phase 3**: Add worker threads for CPU-intensive operations
|
||||
- **Phase 4**: Performance monitoring dashboard
|
||||
- **Phase 4**: Performance monitoring dashboard
|
||||
|
||||
## Socket Error Handling Fix (v19.5.11+)
|
||||
|
||||
### Issue
|
||||
Server crashed with unhandled 'error' event when backend connections failed (ECONNREFUSED). Also caused memory leak with rising active connection count as failed connections weren't cleaned up properly.
|
||||
|
||||
### Root Cause
|
||||
1. **Race Condition**: In forwarding handlers, sockets were created with `net.connect()` but error handlers were attached later, creating a window where errors could crash the server
|
||||
2. **Incomplete Cleanup**: When server connections failed, client sockets weren't properly cleaned up, leaving connection records in memory
|
||||
|
||||
### Solution
|
||||
Created `createSocketWithErrorHandler()` utility that attaches error handlers immediately:
|
||||
```typescript
|
||||
// Before (race condition):
|
||||
const socket = net.connect(port, host);
|
||||
// ... other code ...
|
||||
socket.on('error', handler); // Too late!
|
||||
|
||||
// After (safe):
|
||||
const socket = createSocketWithErrorHandler({
|
||||
port, host,
|
||||
onError: (error) => {
|
||||
// Handle error immediately
|
||||
clientSocket.destroy();
|
||||
},
|
||||
onConnect: () => {
|
||||
// Set up forwarding
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Changes Made
|
||||
1. **New Utility**: `ts/core/utils/socket-utils.ts` - Added `createSocketWithErrorHandler()`
|
||||
2. **Updated Handlers**:
|
||||
- `https-passthrough-handler.ts` - Uses safe socket creation
|
||||
- `https-terminate-to-http-handler.ts` - Uses safe socket creation
|
||||
3. **Connection Cleanup**: Client sockets destroyed immediately on server connection failure
|
||||
|
||||
### Test Coverage
|
||||
- `test/test.socket-error-handling.node.ts` - Verifies server doesn't crash on ECONNREFUSED
|
||||
- `test/test.forwarding-error-fix.node.ts` - Tests forwarding handlers handle errors gracefully
|
||||
|
||||
### Configuration
|
||||
No configuration changes needed. The fix is transparent to users.
|
||||
|
||||
### Important Note
|
||||
The fix was applied in two places:
|
||||
1. **ForwardingHandler classes** (`https-passthrough-handler.ts`, etc.) - These are standalone forwarding utilities
|
||||
2. **SmartProxy route-connection-handler** (`route-connection-handler.ts`) - This is where the actual SmartProxy connection handling happens
|
||||
|
||||
The critical fix for SmartProxy was in `setupDirectConnection()` method in route-connection-handler.ts, which now uses `createSocketWithErrorHandler()` to properly handle connection failures and clean up connection records.
|
448
readme.plan.md
448
readme.plan.md
@ -1,337 +1,165 @@
|
||||
# SmartProxy Socket Cleanup Fix Plan
|
||||
# SmartProxy Socket Handling Fix Plan
|
||||
|
||||
Reread CLAUDE.md file for guidelines
|
||||
|
||||
## Implementation Summary (COMPLETED)
|
||||
|
||||
The critical socket handling issues have been fixed:
|
||||
|
||||
1. **Prevented Server Crashes**: Created `createSocketWithErrorHandler()` utility that attaches error handlers immediately upon socket creation, preventing unhandled ECONNREFUSED errors from crashing the server.
|
||||
|
||||
2. **Fixed Memory Leaks**: Updated forwarding handlers to properly clean up client sockets when server connections fail, ensuring connection records are removed from tracking.
|
||||
|
||||
3. **Key Changes Made**:
|
||||
- Added `createSocketWithErrorHandler()` in `socket-utils.ts`
|
||||
- Updated `https-passthrough-handler.ts` to use safe socket creation
|
||||
- Updated `https-terminate-to-http-handler.ts` to use safe socket creation
|
||||
- Ensured client sockets are destroyed when server connections fail
|
||||
- Connection cleanup now triggered by socket close events
|
||||
|
||||
4. **Test Results**: Server no longer crashes on ECONNREFUSED errors, and connections are properly cleaned up.
|
||||
|
||||
## Problem Summary
|
||||
|
||||
The current socket cleanup implementation is too aggressive and closes long-lived connections prematurely. This affects:
|
||||
- WebSocket connections in HTTPS passthrough
|
||||
- Long-lived HTTP connections (SSE, streaming)
|
||||
- Database connections
|
||||
- Any connection that should remain open for hours
|
||||
The SmartProxy server is experiencing critical issues:
|
||||
1. **Server crashes** due to unhandled socket connection errors (ECONNREFUSED)
|
||||
2. **Memory leak** with steadily rising active connection count
|
||||
3. **Race conditions** between socket creation and error handler attachment
|
||||
4. **Orphaned sockets** when server connections fail
|
||||
|
||||
## Root Causes
|
||||
|
||||
### 1. **Bilateral Socket Cleanup**
|
||||
When one socket closes, both sockets are immediately destroyed:
|
||||
```typescript
|
||||
// In createSocketCleanupHandler
|
||||
cleanupSocket(clientSocket, 'client');
|
||||
cleanupSocket(serverSocket, 'server'); // Both destroyed together!
|
||||
```
|
||||
### 1. Delayed Error Handler Attachment
|
||||
- Sockets created without immediate error handlers
|
||||
- Error events can fire before handlers attached
|
||||
- Causes uncaught exceptions and server crashes
|
||||
|
||||
### 2. **Aggressive Timeout Handling**
|
||||
Timeout events immediately trigger connection cleanup:
|
||||
### 2. Incomplete Cleanup Logic
|
||||
- Client sockets not cleaned up when server connection fails
|
||||
- Connection counter only decrements after BOTH sockets close
|
||||
- Failed server connections leave orphaned client sockets
|
||||
|
||||
### 3. Missing Global Error Handlers
|
||||
- No process-level uncaughtException handler
|
||||
- No process-level unhandledRejection handler
|
||||
- Any unhandled error crashes entire server
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Phase 1: Prevent Server Crashes (Critical)
|
||||
|
||||
#### 1.1 Add Global Error Handlers
|
||||
- [x] ~~Add global error handlers in main entry point~~ (Removed per user request - no global handlers)
|
||||
- [x] Log errors with context
|
||||
- [x] ~~Implement graceful shutdown sequence~~ (Removed - handled locally)
|
||||
|
||||
#### 1.2 Fix Socket Creation Race Condition
|
||||
- [x] Modify socket creation to attach error handlers immediately
|
||||
- [x] Update all forwarding handlers (https-passthrough, http, etc.)
|
||||
- [x] Ensure error handlers attached in same tick as socket creation
|
||||
|
||||
### Phase 2: Fix Memory Leaks (High Priority)
|
||||
|
||||
#### 2.1 Fix Connection Cleanup Logic
|
||||
- [x] Clean up client socket immediately if server connection fails
|
||||
- [x] Decrement connection counter on any socket failure (handled by socket close events)
|
||||
- [x] Implement proper cleanup for half-open connections
|
||||
|
||||
#### 2.2 Improve Socket Utils
|
||||
- [x] Create new utility function for safe socket creation with immediate error handling
|
||||
- [x] Update createIndependentSocketHandlers to handle immediate failures
|
||||
- [ ] Add connection tracking debug utilities
|
||||
|
||||
### Phase 3: Comprehensive Testing (Important)
|
||||
|
||||
#### 3.1 Create Test Cases
|
||||
- [x] Test ECONNREFUSED scenario
|
||||
- [ ] Test timeout handling
|
||||
- [ ] Test half-open connections
|
||||
- [ ] Test rapid connect/disconnect cycles
|
||||
|
||||
#### 3.2 Add Monitoring
|
||||
- [ ] Add connection leak detection
|
||||
- [ ] Add metrics for connection lifecycle
|
||||
- [ ] Add debug logging for socket state transitions
|
||||
|
||||
## Detailed Implementation Steps
|
||||
|
||||
### Step 1: Global Error Handlers (ts/proxies/smart-proxy/smart-proxy.ts)
|
||||
```typescript
|
||||
socket.on('timeout', () => {
|
||||
handleClose(`${prefix}_timeout`); // Destroys both sockets!
|
||||
// Add in constructor or start method
|
||||
process.on('uncaughtException', (error) => {
|
||||
logger.log('error', 'Uncaught exception', { error });
|
||||
// Graceful shutdown
|
||||
});
|
||||
|
||||
process.on('unhandledRejection', (reason, promise) => {
|
||||
logger.log('error', 'Unhandled rejection', { reason, promise });
|
||||
});
|
||||
```
|
||||
|
||||
### 3. **Parity Check Forces Closure**
|
||||
If one socket closes but the other remains open for >2 minutes, connection is forcefully terminated:
|
||||
### Step 2: Safe Socket Creation Utility (ts/core/utils/socket-utils.ts)
|
||||
```typescript
|
||||
if (record.outgoingClosedTime &&
|
||||
!record.incoming.destroyed &&
|
||||
now - record.outgoingClosedTime > 120000) {
|
||||
this.cleanupConnection(record, 'parity_check');
|
||||
export function createSocketWithErrorHandler(
|
||||
options: net.NetConnectOpts,
|
||||
onError: (err: Error) => void
|
||||
): net.Socket {
|
||||
const socket = net.connect(options);
|
||||
socket.on('error', onError);
|
||||
return socket;
|
||||
}
|
||||
```
|
||||
|
||||
### 4. **No Half-Open Connection Support**
|
||||
The proxy doesn't support TCP half-open connections where one side closes while the other continues sending.
|
||||
### Step 3: Fix HttpsPassthroughHandler (ts/forwarding/handlers/https-passthrough-handler.ts)
|
||||
- Replace direct socket creation with safe creation
|
||||
- Handle server connection failures immediately
|
||||
- Clean up client socket on server connection failure
|
||||
|
||||
## Fix Implementation Plan
|
||||
### Step 4: Fix Connection Counting
|
||||
- Decrement on ANY socket close, not just when both close
|
||||
- Track failed connections separately
|
||||
- Add connection state tracking
|
||||
|
||||
### Phase 1: Fix Socket Cleanup (Prevent Premature Closure)
|
||||
### Step 5: Update All Handlers
|
||||
- [ ] https-passthrough-handler.ts
|
||||
- [ ] http-handler.ts
|
||||
- [ ] https-terminate-to-http-handler.ts
|
||||
- [ ] https-terminate-to-https-handler.ts
|
||||
- [ ] route-connection-handler.ts
|
||||
|
||||
#### 1.1 Modify `cleanupSocket()` to support graceful shutdown
|
||||
```typescript
|
||||
export interface CleanupOptions {
|
||||
immediate?: boolean; // Force immediate destruction
|
||||
allowDrain?: boolean; // Allow write buffer to drain
|
||||
gracePeriod?: number; // Ms to wait before force close
|
||||
}
|
||||
## Success Criteria
|
||||
|
||||
export function cleanupSocket(
|
||||
socket: Socket | TLSSocket | null,
|
||||
socketName?: string,
|
||||
options: CleanupOptions = {}
|
||||
): Promise<void> {
|
||||
if (!socket || socket.destroyed) return Promise.resolve();
|
||||
|
||||
return new Promise<void>((resolve) => {
|
||||
const cleanup = () => {
|
||||
socket.removeAllListeners();
|
||||
if (!socket.destroyed) {
|
||||
socket.destroy();
|
||||
}
|
||||
resolve();
|
||||
};
|
||||
|
||||
if (options.immediate) {
|
||||
cleanup();
|
||||
} else if (options.allowDrain && socket.writable) {
|
||||
// Allow pending writes to complete
|
||||
socket.end(() => cleanup());
|
||||
|
||||
// Force cleanup after grace period
|
||||
if (options.gracePeriod) {
|
||||
setTimeout(cleanup, options.gracePeriod);
|
||||
}
|
||||
} else {
|
||||
cleanup();
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
1. **No server crashes** on ECONNREFUSED or other socket errors
|
||||
2. **Active connections** remain stable (no steady increase)
|
||||
3. **All sockets** properly cleaned up on errors
|
||||
4. **Memory usage** remains stable under load
|
||||
5. **Graceful handling** of all error scenarios
|
||||
|
||||
#### 1.2 Implement Independent Socket Tracking
|
||||
```typescript
|
||||
export function createIndependentSocketHandlers(
|
||||
clientSocket: Socket,
|
||||
serverSocket: Socket,
|
||||
onBothClosed: (reason: string) => void
|
||||
): { cleanupClient: () => void, cleanupServer: () => void } {
|
||||
let clientClosed = false;
|
||||
let serverClosed = false;
|
||||
let clientReason = '';
|
||||
let serverReason = '';
|
||||
|
||||
const checkBothClosed = () => {
|
||||
if (clientClosed && serverClosed) {
|
||||
onBothClosed(`client: ${clientReason}, server: ${serverReason}`);
|
||||
}
|
||||
};
|
||||
|
||||
const cleanupClient = async (reason: string) => {
|
||||
if (clientClosed) return;
|
||||
clientClosed = true;
|
||||
clientReason = reason;
|
||||
|
||||
// Allow server to continue if still active
|
||||
if (!serverClosed && serverSocket.writable) {
|
||||
// Half-close: stop reading from client, let server finish
|
||||
clientSocket.pause();
|
||||
clientSocket.unpipe(serverSocket);
|
||||
await cleanupSocket(clientSocket, 'client', { allowDrain: true });
|
||||
} else {
|
||||
await cleanupSocket(clientSocket, 'client');
|
||||
}
|
||||
|
||||
checkBothClosed();
|
||||
};
|
||||
|
||||
const cleanupServer = async (reason: string) => {
|
||||
if (serverClosed) return;
|
||||
serverClosed = true;
|
||||
serverReason = reason;
|
||||
|
||||
// Allow client to continue if still active
|
||||
if (!clientClosed && clientSocket.writable) {
|
||||
// Half-close: stop reading from server, let client finish
|
||||
serverSocket.pause();
|
||||
serverSocket.unpipe(clientSocket);
|
||||
await cleanupSocket(serverSocket, 'server', { allowDrain: true });
|
||||
} else {
|
||||
await cleanupSocket(serverSocket, 'server');
|
||||
}
|
||||
|
||||
checkBothClosed();
|
||||
};
|
||||
|
||||
return { cleanupClient, cleanupServer };
|
||||
}
|
||||
```
|
||||
## Testing Plan
|
||||
|
||||
### Phase 2: Fix Timeout Handling
|
||||
1. Simulate ECONNREFUSED by targeting closed ports
|
||||
2. Monitor active connection count over time
|
||||
3. Stress test with rapid connections
|
||||
4. Test with unreachable hosts
|
||||
5. Test with slow/timing out connections
|
||||
|
||||
#### 2.1 Separate timeout handling from connection closure
|
||||
```typescript
|
||||
export function setupSocketHandlers(
|
||||
socket: Socket | TLSSocket,
|
||||
handleClose: (reason: string) => void,
|
||||
handleTimeout?: (socket: Socket) => void, // New optional handler
|
||||
errorPrefix?: string
|
||||
): void {
|
||||
socket.on('error', (error) => {
|
||||
const prefix = errorPrefix || 'Socket';
|
||||
handleClose(`${prefix}_error: ${error.message}`);
|
||||
});
|
||||
|
||||
socket.on('close', () => {
|
||||
const prefix = errorPrefix || 'socket';
|
||||
handleClose(`${prefix}_closed`);
|
||||
});
|
||||
|
||||
socket.on('timeout', () => {
|
||||
if (handleTimeout) {
|
||||
handleTimeout(socket); // Custom timeout handling
|
||||
} else {
|
||||
// Default: just log, don't close
|
||||
console.warn(`Socket timeout: ${errorPrefix || 'socket'}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
```
|
||||
## Rollback Plan
|
||||
|
||||
#### 2.2 Update HTTPS passthrough handler
|
||||
```typescript
|
||||
// In https-passthrough-handler.ts
|
||||
const { cleanupClient, cleanupServer } = createIndependentSocketHandlers(
|
||||
clientSocket,
|
||||
serverSocket,
|
||||
(reason) => {
|
||||
this.emit(ForwardingHandlerEvents.DISCONNECTED, {
|
||||
remoteAddress,
|
||||
bytesSent,
|
||||
bytesReceived,
|
||||
reason
|
||||
});
|
||||
}
|
||||
);
|
||||
If issues arise:
|
||||
1. Revert socket creation changes
|
||||
2. Keep global error handlers (they add safety)
|
||||
3. Add more detailed logging for debugging
|
||||
4. Implement fixes incrementally
|
||||
|
||||
// Setup handlers with custom timeout handling
|
||||
setupSocketHandlers(clientSocket, cleanupClient, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'client');
|
||||
## Timeline
|
||||
|
||||
setupSocketHandlers(serverSocket, cleanupServer, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'server');
|
||||
```
|
||||
- Phase 1: Immediate (prevents crashes)
|
||||
- Phase 2: Within 24 hours (fixes leaks)
|
||||
- Phase 3: Within 48 hours (ensures stability)
|
||||
|
||||
### Phase 3: Fix Connection Manager
|
||||
## Notes
|
||||
|
||||
#### 3.1 Remove aggressive parity check
|
||||
```typescript
|
||||
// Remove or significantly increase the parity check timeout
|
||||
// From 2 minutes to 30 minutes for long-lived connections
|
||||
if (record.outgoingClosedTime &&
|
||||
!record.incoming.destroyed &&
|
||||
!record.connectionClosed &&
|
||||
now - record.outgoingClosedTime > 1800000) { // 30 minutes
|
||||
// Only close if no data activity
|
||||
if (now - record.lastActivity > 600000) { // 10 minutes of inactivity
|
||||
this.cleanupConnection(record, 'parity_check');
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### 3.2 Update cleanupConnection to check socket states
|
||||
```typescript
|
||||
public cleanupConnection(record: IConnectionRecord, reason: string = 'normal'): void {
|
||||
if (!record.connectionClosed) {
|
||||
record.connectionClosed = true;
|
||||
|
||||
// Only cleanup sockets that are actually closed or inactive
|
||||
if (record.incoming && (!record.incoming.writable || record.incoming.destroyed)) {
|
||||
cleanupSocket(record.incoming, `${record.id}-incoming`, { immediate: true });
|
||||
}
|
||||
|
||||
if (record.outgoing && (!record.outgoing.writable || record.outgoing.destroyed)) {
|
||||
cleanupSocket(record.outgoing, `${record.id}-outgoing`, { immediate: true });
|
||||
}
|
||||
|
||||
// If either socket is still active, don't remove the record yet
|
||||
if ((record.incoming && record.incoming.writable) ||
|
||||
(record.outgoing && record.outgoing.writable)) {
|
||||
record.connectionClosed = false; // Reset flag
|
||||
return; // Don't finish cleanup
|
||||
}
|
||||
|
||||
// Continue with full cleanup...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Phase 4: Testing and Validation
|
||||
|
||||
#### 4.1 Test Cases to Implement
|
||||
1. WebSocket connection should stay open for >1 hour
|
||||
2. HTTP streaming response should continue after request closes
|
||||
3. Half-open connections should work correctly
|
||||
4. Verify no socket leaks with long-running connections
|
||||
5. Test graceful shutdown with pending data
|
||||
|
||||
#### 4.2 Socket Leak Prevention
|
||||
- Ensure all event listeners are tracked and removed
|
||||
- Use WeakMap for socket metadata to prevent memory leaks
|
||||
- Implement connection count monitoring
|
||||
- Add periodic health checks for orphaned sockets
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Day 1**: Implement graceful `cleanupSocket()` and independent socket handlers
|
||||
2. **Day 2**: Update all handlers to use new cleanup mechanism
|
||||
3. **Day 3**: Fix timeout handling to not close connections
|
||||
4. **Day 4**: Update connection manager parity check and cleanup logic
|
||||
5. **Day 5**: Comprehensive testing and leak detection
|
||||
|
||||
## Configuration Changes
|
||||
|
||||
Add new options to SmartProxyOptions:
|
||||
```typescript
|
||||
interface ISmartProxyOptions {
|
||||
// Existing options...
|
||||
|
||||
// New options for long-lived connections
|
||||
socketCleanupGracePeriod?: number; // Default: 5000ms
|
||||
allowHalfOpenConnections?: boolean; // Default: true
|
||||
parityCheckTimeout?: number; // Default: 1800000ms (30 min)
|
||||
timeoutBehavior?: 'close' | 'reset' | 'ignore'; // Default: 'reset'
|
||||
}
|
||||
```
|
||||
|
||||
## Success Metrics
|
||||
|
||||
1. WebSocket connections remain stable for 24+ hours
|
||||
2. No premature connection closures reported
|
||||
3. Memory usage remains stable (no socket leaks)
|
||||
4. Half-open connections work correctly
|
||||
5. Graceful shutdown completes within grace period
|
||||
|
||||
## Implementation Status: COMPLETED ✅
|
||||
|
||||
### Implemented Changes
|
||||
|
||||
1. **Modified `cleanupSocket()` in `socket-utils.ts`**
|
||||
- Added `CleanupOptions` interface with `immediate`, `allowDrain`, and `gracePeriod` options
|
||||
- Implemented graceful shutdown support with write buffer draining
|
||||
|
||||
2. **Created `createIndependentSocketHandlers()` in `socket-utils.ts`**
|
||||
- Tracks socket states independently
|
||||
- Supports half-open connections where one side can close while the other remains open
|
||||
- Only triggers full cleanup when both sockets are closed
|
||||
|
||||
3. **Updated `setupSocketHandlers()` in `socket-utils.ts`**
|
||||
- Added optional `handleTimeout` parameter to customize timeout behavior
|
||||
- Prevents automatic connection closure on timeout events
|
||||
|
||||
4. **Updated HTTPS Passthrough Handler**
|
||||
- Now uses `createIndependentSocketHandlers` for half-open support
|
||||
- Custom timeout handling that resets timer instead of closing connection
|
||||
- Manual data forwarding with backpressure handling
|
||||
|
||||
5. **Updated Connection Manager**
|
||||
- Extended parity check from 2 minutes to 30 minutes
|
||||
- Added activity check before closing (10 minutes of inactivity required)
|
||||
- Modified cleanup to check socket states before destroying
|
||||
|
||||
6. **Updated Basic Forwarding in Route Connection Handler**
|
||||
- Replaced simple `pipe()` with independent socket handlers
|
||||
- Added manual data forwarding with backpressure support
|
||||
- Removed bilateral close handlers to prevent premature cleanup
|
||||
|
||||
### Test Results
|
||||
|
||||
All tests passing:
|
||||
- ✅ Long-lived connection test: Connection stayed open for 61+ seconds with periodic keep-alive
|
||||
- ✅ Half-open connection test: One side closed while the other continued to send data
|
||||
- ✅ No socket leaks or premature closures
|
||||
|
||||
### Notes
|
||||
|
||||
- The fix maintains backward compatibility
|
||||
- No configuration changes required for existing deployments
|
||||
- Long-lived connections now work correctly in both HTTPS passthrough and basic forwarding modes
|
||||
- The race condition is the most critical issue
|
||||
- Connection counting logic needs complete overhaul
|
||||
- Consider using a connection state machine for clarity
|
||||
- Add connection lifecycle events for debugging
|
@ -6,6 +6,14 @@ export interface CleanupOptions {
|
||||
gracePeriod?: number; // Ms to wait before force close
|
||||
}
|
||||
|
||||
export interface SafeSocketOptions {
|
||||
port: number;
|
||||
host: string;
|
||||
onError?: (error: Error) => void;
|
||||
onConnect?: () => void;
|
||||
timeout?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Safely cleanup a socket by removing all listeners and destroying it
|
||||
* @param socket The socket to cleanup
|
||||
@ -197,4 +205,39 @@ export function pipeSockets(
|
||||
): void {
|
||||
socket1.pipe(socket2);
|
||||
socket2.pipe(socket1);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create a socket with immediate error handling to prevent crashes
|
||||
* @param options Socket creation options
|
||||
* @returns The created socket
|
||||
*/
|
||||
export function createSocketWithErrorHandler(options: SafeSocketOptions): plugins.net.Socket {
|
||||
const { port, host, onError, onConnect, timeout } = options;
|
||||
|
||||
// Create socket with immediate error handler attachment
|
||||
const socket = new plugins.net.Socket();
|
||||
|
||||
// Attach error handler BEFORE connecting to catch immediate errors
|
||||
socket.on('error', (error) => {
|
||||
console.error(`Socket connection error to ${host}:${port}: ${error.message}`);
|
||||
if (onError) {
|
||||
onError(error);
|
||||
}
|
||||
});
|
||||
|
||||
// Attach connect handler if provided
|
||||
if (onConnect) {
|
||||
socket.on('connect', onConnect);
|
||||
}
|
||||
|
||||
// Set timeout if provided
|
||||
if (timeout) {
|
||||
socket.setTimeout(timeout);
|
||||
}
|
||||
|
||||
// Now attempt to connect - any immediate errors will be caught
|
||||
socket.connect(port, host);
|
||||
|
||||
return socket;
|
||||
}
|
@ -2,7 +2,7 @@ import * as plugins from '../../plugins.js';
|
||||
import { ForwardingHandler } from './base-handler.js';
|
||||
import type { IForwardConfig } from '../config/forwarding-types.js';
|
||||
import { ForwardingHandlerEvents } from '../config/forwarding-types.js';
|
||||
import { createIndependentSocketHandlers, setupSocketHandlers } from '../../core/utils/socket-utils.js';
|
||||
import { createIndependentSocketHandlers, setupSocketHandlers, createSocketWithErrorHandler } from '../../core/utils/socket-utils.js';
|
||||
|
||||
/**
|
||||
* Handler for HTTPS passthrough (SNI forwarding without termination)
|
||||
@ -48,91 +48,122 @@ export class HttpsPassthroughHandler extends ForwardingHandler {
|
||||
target: `${target.host}:${target.port}`
|
||||
});
|
||||
|
||||
// Create a connection to the target server
|
||||
const serverSocket = plugins.net.connect(target.port, target.host);
|
||||
|
||||
// Track data transfer for logging
|
||||
let bytesSent = 0;
|
||||
let bytesReceived = 0;
|
||||
let serverSocket: plugins.net.Socket | null = null;
|
||||
let cleanupClient: ((reason: string) => Promise<void>) | null = null;
|
||||
let cleanupServer: ((reason: string) => Promise<void>) | null = null;
|
||||
|
||||
// Create independent handlers for half-open connection support
|
||||
const { cleanupClient, cleanupServer } = createIndependentSocketHandlers(
|
||||
clientSocket,
|
||||
serverSocket,
|
||||
(reason) => {
|
||||
// Create a connection to the target server with immediate error handling
|
||||
serverSocket = createSocketWithErrorHandler({
|
||||
port: target.port,
|
||||
host: target.host,
|
||||
onError: async (error) => {
|
||||
// Server connection failed - clean up client socket immediately
|
||||
this.emit(ForwardingHandlerEvents.ERROR, {
|
||||
error: error.message,
|
||||
code: (error as any).code || 'UNKNOWN',
|
||||
remoteAddress,
|
||||
target: `${target.host}:${target.port}`
|
||||
});
|
||||
|
||||
// Clean up the client socket since we can't forward
|
||||
if (!clientSocket.destroyed) {
|
||||
clientSocket.destroy();
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DISCONNECTED, {
|
||||
remoteAddress,
|
||||
bytesSent,
|
||||
bytesReceived,
|
||||
reason
|
||||
bytesSent: 0,
|
||||
bytesReceived: 0,
|
||||
reason: `server_connection_failed: ${error.message}`
|
||||
});
|
||||
}
|
||||
);
|
||||
|
||||
// Setup handlers with custom timeout handling that doesn't close connections
|
||||
const timeout = this.getTimeout();
|
||||
|
||||
setupSocketHandlers(clientSocket, cleanupClient, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'client');
|
||||
|
||||
setupSocketHandlers(serverSocket, cleanupServer, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'server');
|
||||
|
||||
// Forward data from client to server
|
||||
clientSocket.on('data', (data) => {
|
||||
bytesSent += data.length;
|
||||
|
||||
// Check if server socket is writable
|
||||
if (serverSocket.writable) {
|
||||
const flushed = serverSocket.write(data);
|
||||
},
|
||||
onConnect: () => {
|
||||
// Connection successful - set up forwarding handlers
|
||||
const handlers = createIndependentSocketHandlers(
|
||||
clientSocket,
|
||||
serverSocket!,
|
||||
(reason) => {
|
||||
this.emit(ForwardingHandlerEvents.DISCONNECTED, {
|
||||
remoteAddress,
|
||||
bytesSent,
|
||||
bytesReceived,
|
||||
reason
|
||||
});
|
||||
}
|
||||
);
|
||||
|
||||
// Handle backpressure
|
||||
if (!flushed) {
|
||||
clientSocket.pause();
|
||||
serverSocket.once('drain', () => {
|
||||
clientSocket.resume();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DATA_FORWARDED, {
|
||||
direction: 'outbound',
|
||||
bytes: data.length,
|
||||
total: bytesSent
|
||||
});
|
||||
});
|
||||
|
||||
// Forward data from server to client
|
||||
serverSocket.on('data', (data) => {
|
||||
bytesReceived += data.length;
|
||||
|
||||
// Check if client socket is writable
|
||||
if (clientSocket.writable) {
|
||||
const flushed = clientSocket.write(data);
|
||||
cleanupClient = handlers.cleanupClient;
|
||||
cleanupServer = handlers.cleanupServer;
|
||||
|
||||
// Handle backpressure
|
||||
if (!flushed) {
|
||||
serverSocket.pause();
|
||||
clientSocket.once('drain', () => {
|
||||
serverSocket.resume();
|
||||
// Setup handlers with custom timeout handling that doesn't close connections
|
||||
const timeout = this.getTimeout();
|
||||
|
||||
setupSocketHandlers(clientSocket, cleanupClient, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'client');
|
||||
|
||||
setupSocketHandlers(serverSocket!, cleanupServer, (socket) => {
|
||||
// Just reset timeout, don't close
|
||||
socket.setTimeout(timeout);
|
||||
}, 'server');
|
||||
|
||||
// Forward data from client to server
|
||||
clientSocket.on('data', (data) => {
|
||||
bytesSent += data.length;
|
||||
|
||||
// Check if server socket is writable
|
||||
if (serverSocket && serverSocket.writable) {
|
||||
const flushed = serverSocket.write(data);
|
||||
|
||||
// Handle backpressure
|
||||
if (!flushed) {
|
||||
clientSocket.pause();
|
||||
serverSocket.once('drain', () => {
|
||||
clientSocket.resume();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DATA_FORWARDED, {
|
||||
direction: 'outbound',
|
||||
bytes: data.length,
|
||||
total: bytesSent
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
// Forward data from server to client
|
||||
serverSocket!.on('data', (data) => {
|
||||
bytesReceived += data.length;
|
||||
|
||||
// Check if client socket is writable
|
||||
if (clientSocket.writable) {
|
||||
const flushed = clientSocket.write(data);
|
||||
|
||||
// Handle backpressure
|
||||
if (!flushed) {
|
||||
serverSocket!.pause();
|
||||
clientSocket.once('drain', () => {
|
||||
serverSocket!.resume();
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DATA_FORWARDED, {
|
||||
direction: 'inbound',
|
||||
bytes: data.length,
|
||||
total: bytesReceived
|
||||
});
|
||||
});
|
||||
|
||||
// Set initial timeouts - they will be reset on each timeout event
|
||||
clientSocket.setTimeout(timeout);
|
||||
serverSocket!.setTimeout(timeout);
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DATA_FORWARDED, {
|
||||
direction: 'inbound',
|
||||
bytes: data.length,
|
||||
total: bytesReceived
|
||||
});
|
||||
});
|
||||
|
||||
// Set initial timeouts - they will be reset on each timeout event
|
||||
clientSocket.setTimeout(timeout);
|
||||
serverSocket.setTimeout(timeout);
|
||||
}
|
||||
|
||||
/**
|
||||
|
@ -2,7 +2,7 @@ import * as plugins from '../../plugins.js';
|
||||
import { ForwardingHandler } from './base-handler.js';
|
||||
import type { IForwardConfig } from '../config/forwarding-types.js';
|
||||
import { ForwardingHandlerEvents } from '../config/forwarding-types.js';
|
||||
import { createSocketCleanupHandler, setupSocketHandlers } from '../../core/utils/socket-utils.js';
|
||||
import { createSocketCleanupHandler, setupSocketHandlers, createSocketWithErrorHandler } from '../../core/utils/socket-utils.js';
|
||||
|
||||
/**
|
||||
* Handler for HTTPS termination with HTTP backend
|
||||
@ -141,19 +141,41 @@ export class HttpsTerminateToHttpHandler extends ForwardingHandler {
|
||||
if (dataBuffer.includes(Buffer.from('\r\n\r\n')) && !connectionEstablished) {
|
||||
const target = this.getTargetFromConfig();
|
||||
|
||||
// Create backend connection
|
||||
backendSocket = plugins.net.connect(target.port, target.host, () => {
|
||||
connectionEstablished = true;
|
||||
|
||||
// Send buffered data
|
||||
if (dataBuffer.length > 0) {
|
||||
backendSocket!.write(dataBuffer);
|
||||
dataBuffer = Buffer.alloc(0);
|
||||
// Create backend connection with immediate error handling
|
||||
backendSocket = createSocketWithErrorHandler({
|
||||
port: target.port,
|
||||
host: target.host,
|
||||
onError: (error) => {
|
||||
this.emit(ForwardingHandlerEvents.ERROR, {
|
||||
error: error.message,
|
||||
code: (error as any).code || 'UNKNOWN',
|
||||
remoteAddress,
|
||||
target: `${target.host}:${target.port}`
|
||||
});
|
||||
|
||||
// Clean up the TLS socket since we can't forward
|
||||
if (!tlsSocket.destroyed) {
|
||||
tlsSocket.destroy();
|
||||
}
|
||||
|
||||
this.emit(ForwardingHandlerEvents.DISCONNECTED, {
|
||||
remoteAddress,
|
||||
reason: `backend_connection_failed: ${error.message}`
|
||||
});
|
||||
},
|
||||
onConnect: () => {
|
||||
connectionEstablished = true;
|
||||
|
||||
// Send buffered data
|
||||
if (dataBuffer.length > 0) {
|
||||
backendSocket!.write(dataBuffer);
|
||||
dataBuffer = Buffer.alloc(0);
|
||||
}
|
||||
|
||||
// Set up bidirectional data flow
|
||||
tlsSocket.pipe(backendSocket!);
|
||||
backendSocket!.pipe(tlsSocket);
|
||||
}
|
||||
|
||||
// Set up bidirectional data flow
|
||||
tlsSocket.pipe(backendSocket!);
|
||||
backendSocket!.pipe(tlsSocket);
|
||||
});
|
||||
|
||||
// Update the cleanup handler with the backend socket
|
||||
|
@ -2,7 +2,7 @@ import * as plugins from '../../plugins.js';
|
||||
import { ForwardingHandler } from './base-handler.js';
|
||||
import type { IForwardConfig } from '../config/forwarding-types.js';
|
||||
import { ForwardingHandlerEvents } from '../config/forwarding-types.js';
|
||||
import { createSocketCleanupHandler, setupSocketHandlers } from '../../core/utils/socket-utils.js';
|
||||
import { createSocketCleanupHandler, setupSocketHandlers, createSocketWithErrorHandler } from '../../core/utils/socket-utils.js';
|
||||
|
||||
/**
|
||||
* Handler for HTTPS termination with HTTPS backend
|
||||
|
@ -9,7 +9,7 @@ import { TlsManager } from './tls-manager.js';
|
||||
import { HttpProxyBridge } from './http-proxy-bridge.js';
|
||||
import { TimeoutManager } from './timeout-manager.js';
|
||||
import { RouteManager } from './route-manager.js';
|
||||
import { cleanupSocket, createIndependentSocketHandlers, setupSocketHandlers } from '../../core/utils/socket-utils.js';
|
||||
import { cleanupSocket, createIndependentSocketHandlers, setupSocketHandlers, createSocketWithErrorHandler } from '../../core/utils/socket-utils.js';
|
||||
|
||||
/**
|
||||
* Handles new connection processing and setup logic with support for route-based configuration
|
||||
@ -1073,13 +1073,52 @@ export class RouteConnectionHandler {
|
||||
record.pendingDataSize = initialChunk.length;
|
||||
}
|
||||
|
||||
// Create the target socket
|
||||
const targetSocket = plugins.net.connect(connectionOptions);
|
||||
record.outgoing = targetSocket;
|
||||
record.outgoingStartTime = Date.now();
|
||||
// Create the target socket with immediate error handling
|
||||
let targetSocket: plugins.net.Socket;
|
||||
|
||||
// Flag to track if initial connection failed
|
||||
let connectionFailed = false;
|
||||
|
||||
targetSocket = createSocketWithErrorHandler({
|
||||
port: finalTargetPort,
|
||||
host: finalTargetHost,
|
||||
onError: (error) => {
|
||||
// Mark connection as failed
|
||||
connectionFailed = true;
|
||||
|
||||
// Connection failed - clean up immediately
|
||||
logger.log('error',
|
||||
`Connection setup error for ${connectionId} to ${finalTargetHost}:${finalTargetPort}: ${error.message} (${(error as any).code})`,
|
||||
{
|
||||
connectionId,
|
||||
targetHost: finalTargetHost,
|
||||
targetPort: finalTargetPort,
|
||||
errorMessage: error.message,
|
||||
errorCode: (error as any).code,
|
||||
component: 'route-handler'
|
||||
}
|
||||
);
|
||||
|
||||
// Resume the incoming socket to prevent it from hanging
|
||||
socket.resume();
|
||||
|
||||
// Clean up the incoming socket
|
||||
if (!socket.destroyed) {
|
||||
socket.destroy();
|
||||
}
|
||||
|
||||
// Clean up the connection record
|
||||
this.connectionManager.initiateCleanupOnce(record, `connection_failed_${(error as any).code || 'unknown'}`);
|
||||
}
|
||||
});
|
||||
|
||||
// Only proceed with setup if connection didn't fail immediately
|
||||
if (!connectionFailed) {
|
||||
record.outgoing = targetSocket;
|
||||
record.outgoingStartTime = Date.now();
|
||||
|
||||
// Apply socket optimizations
|
||||
targetSocket.setNoDelay(this.settings.noDelay);
|
||||
// Apply socket optimizations
|
||||
targetSocket.setNoDelay(this.settings.noDelay);
|
||||
|
||||
// Apply keep-alive settings if enabled
|
||||
if (this.settings.keepAlive) {
|
||||
@ -1346,5 +1385,6 @@ export class RouteConnectionHandler {
|
||||
record.tlsHandshakeComplete = true;
|
||||
}
|
||||
});
|
||||
} // End of if (!connectionFailed)
|
||||
}
|
||||
}
|
Reference in New Issue
Block a user