Implement zombie connection detection and cleanup in ConnectionManager; enhance tests for edge cases

This commit is contained in:
Juergen Kunz
2025-06-07 10:55:59 +00:00
parent 9094b76b1b
commit 47735adbf2
4 changed files with 564 additions and 2 deletions

View File

@ -856,4 +856,42 @@ The WrappedSocket class has been implemented as the foundation for PROXY protoco
For detailed information about proxy protocol implementation and proxy chaining:
- **[Proxy Protocol Guide](./readme.proxy-protocol.md)** - Complete implementation details and configuration
- **[Proxy Protocol Examples](./readme.proxy-protocol-example.md)** - Code examples and conceptual implementation
- **[Proxy Chain Summary](./readme.proxy-chain-summary.md)** - Quick reference for proxy chaining setup
- **[Proxy Chain Summary](./readme.proxy-chain-summary.md)** - Quick reference for proxy chaining setup
## Connection Cleanup Edge Cases Investigation (v19.5.20+)
### Issue Discovered
"Zombie connections" can occur when both sockets are destroyed but the connection record hasn't been cleaned up. This happens when sockets are destroyed without triggering their close/error event handlers.
### Root Cause
1. **Event Handler Bypass**: In edge cases (network failures, proxy chain failures, forced socket destruction), sockets can be destroyed without their event handlers being called
2. **Cleanup Queue Delay**: The `initiateCleanupOnce` method adds connections to a cleanup queue (batch of 100 every 100ms), which may not process fast enough
3. **Inactivity Check Limitation**: The periodic inactivity check only examines `lastActivity` timestamps, not actual socket states
### Test Results
Debug script (`connection-manager-direct-test.ts`) revealed:
- **Normal cleanup works**: When socket events fire normally, cleanup is reliable
- **Zombies ARE created**: Direct socket destruction creates zombies (destroyed sockets, connectionClosed=false)
- **Manual cleanup works**: Calling `initiateCleanupOnce` on a zombie does clean it up
- **Inactivity check misses zombies**: The check doesn't detect connections with destroyed sockets
### Potential Solutions
1. **Periodic Zombie Detection**: Add zombie detection to the inactivity check:
```typescript
// In performOptimizedInactivityCheck
if (record.incoming?.destroyed && record.outgoing?.destroyed && !record.connectionClosed) {
this.cleanupConnection(record, 'zombie_detected');
}
```
2. **Socket State Monitoring**: Check socket states during connection operations
3. **Defensive Socket Handling**: Always attach cleanup handlers before any operation that might destroy sockets
4. **Immediate Cleanup Option**: For critical paths, use `cleanupConnection` instead of `initiateCleanupOnce`
### Impact
- Memory leaks in edge cases (network failures, proxy chain issues)
- Connection count inaccuracy
- Potential resource exhaustion over time
### Test Files
- `.nogit/debug/connection-manager-direct-test.ts` - Direct ConnectionManager testing showing zombie creation