fix(metrics): fix metrics

This commit is contained in:
Juergen Kunz
2025-06-23 09:35:37 +00:00
parent 22418cd65e
commit 8df0333dc3
4 changed files with 251 additions and 4 deletions

View File

@ -2,6 +2,27 @@
## Byte Tracking and Metrics
### What Gets Counted (Network Interface Throughput)
The byte tracking is designed to match network interface throughput (what Unifi/network monitoring tools show):
**Counted bytes include:**
- All application data
- TLS handshakes and protocol overhead
- TLS record headers and encryption padding
- HTTP headers and protocol data
- WebSocket frames and protocol overhead
- TLS alerts sent to clients
**NOT counted:**
- PROXY protocol headers (sent to backend, not client)
- TCP/IP headers (handled by OS, not visible at application layer)
**Byte direction:**
- `bytesReceived`: All bytes received FROM the client on the incoming connection
- `bytesSent`: All bytes sent TO the client on the incoming connection
- Backend connections are separate and not mixed with client metrics
### Double Counting Issue (Fixed)
**Problem**: Initial data chunks were being counted twice in the byte tracking:
@ -25,6 +46,51 @@ The metrics system has three layers:
Total byte counts come from connection records only, preventing double counting.
### Understanding "High" Byte Counts
If byte counts seem high compared to actual application data, remember:
- TLS handshakes can be 1-5KB depending on cipher suites and certificates
- Each TLS record has 5 bytes of header overhead
- TLS encryption adds 16-48 bytes of padding/MAC per record
- HTTP/2 has additional framing overhead
- WebSocket has frame headers (2-14 bytes per message)
This overhead is real network traffic and should be counted for accurate throughput metrics.
### Byte Counting Paths
There are two mutually exclusive paths for connections:
1. **Direct forwarding** (route-connection-handler.ts):
- Used for TCP passthrough, TLS passthrough, and direct connections
- Bytes counted in `setupBidirectionalForwarding` callbacks
- Initial chunk NOT counted separately (flows through bidirectional forwarding)
2. **HttpProxy forwarding** (http-proxy-bridge.ts):
- Used for TLS termination (terminate, terminate-and-reencrypt)
- Initial chunk counted when written to proxy
- All subsequent bytes counted in `setupBidirectionalForwarding` callbacks
- This is the ONLY counting point for these connections
### Byte Counting Audit (2025-01-06)
A comprehensive audit was performed to verify byte counting accuracy:
**Audit Results:**
- ✅ No double counting detected in any connection flow
- ✅ Each byte counted exactly once in each direction
- ✅ Connection records and metrics updated consistently
- ✅ PROXY protocol headers correctly excluded from client metrics
- ✅ NFTables forwarded connections correctly not counted (kernel handles)
**Key Implementation Points:**
- All byte counting happens in only 2 files: `route-connection-handler.ts` and `http-proxy-bridge.ts`
- Both use the same pattern: increment `record.bytesReceived/Sent` AND call `metricsCollector.recordBytes()`
- Initial chunks handled correctly: stored but not counted until forwarded
- TLS alerts counted as sent bytes (correct - they are sent to client)
For full audit details, see `readme.byte-counting-audit.md`
## Connection Cleanup
### Zombie Connection Detection