diff --git a/readme.hints.md b/readme.hints.md index 11aee5b..bd100a8 100644 --- a/readme.hints.md +++ b/readme.hints.md @@ -1,5 +1,30 @@ # Implementation Hints and Learnings +## Network Metrics Implementation (2025-06-23) + +### SmartProxy Metrics API Integration +- Updated to use new SmartProxy metrics API (v19.6.7) +- Use `getMetrics()` for detailed metrics with grouped methods: + ```typescript + const metrics = smartProxy.getMetrics(); + metrics.connections.active() // Current active connections + metrics.throughput.instant() // Real-time throughput {in, out} + metrics.connections.topIPs(10) // Top 10 IPs by connection count + ``` +- Use `getStatistics()` for basic stats + +### Network Traffic Display +- All throughput values shown in bits per second (kbit/s, Mbit/s, Gbit/s) +- Conversion: `bytesPerSecond * 8 / 1000000` for Mbps +- Network graph shows separate lines for inbound (green) and outbound (purple) +- Throughput tiles and graph use same data source for consistency + +### Requests/sec vs Connections +- Requests/sec shows HTTP request counts (derived from connections) +- Single connection can handle multiple requests +- Current implementation tracks connections, not individual requests +- Trend line shows historical request counts, not throughput + ## DKIM Implementation Status (2025-05-30) ### Current Implementation diff --git a/readme.plan2.md b/readme.plan2.md index 1011620..6aac189 100644 --- a/readme.plan2.md +++ b/readme.plan2.md @@ -1,125 +1,71 @@ -# Network Traffic Graph Fix Plan +# Network Metrics Integration Status -## Command: `pnpm run reread` +## Command: `pnpm run build && curl https://code.foss.global/push.rocks/smartproxy/raw/branch/master/readme.md` -## Issue Summary -The network traffic graph in ops-view-network.ts is not displaying data due to three critical issues: -1. Timestamp format mismatch - chart expects ISO strings but receives numeric timestamps -2. Empty data when no active connections exist -3. Potential bucket alignment issues +## Completed Tasks (2025-06-23) -## Root Causes +### ✅ SmartProxy Metrics API Integration +- Updated MetricsManager to use new SmartProxy v19.6.7 metrics API +- Replaced deprecated `getStats()` with `getMetrics()` and `getStatistics()` +- Fixed method calls to use grouped API structure: + - `metrics.connections.active()` for active connections + - `metrics.throughput.instant()` for real-time throughput + - `metrics.connections.topIPs()` for top connected IPs -### 1. Timestamp Format Issue -- **Current**: `x: time` (numeric timestamp like 1703123456789) -- **Expected**: `x: new Date(time).toISOString()` (ISO string like "2023-12-20T12:34:56.789Z") -- **Impact**: ApexCharts cannot parse the x-axis values, resulting in no visible data +### ✅ Removed Mock Data +- Removed hardcoded `0.0.0.0` IPs in security.handler.ts +- Removed `Math.random()` trend data in ops-view-network.ts +- Now using real IP data from SmartProxy metrics -### 2. Empty Data Handling -- When no active connections exist, `networkRequests` array is empty -- Empty array leads to no buckets being created -- Chart shows flat line at 0 +### ✅ Enhanced Metrics Functionality +- Email metrics: delivery time tracking, top recipients, activity log +- DNS metrics: query rate calculations, response time tracking +- Security metrics: incident logging with severity levels -### 3. Data Bucketing Logic -- Current logic creates buckets but uses numeric timestamps as Map keys -- This works for calculation but fails when looking up values for chart display +### ✅ Fixed Network Traffic Display +- All throughput now shown in bits per second (kbit/s, Mbit/s, Gbit/s) +- Network graph shows separate lines for inbound (green) and outbound (purple) +- Fixed throughput calculation to use same data source as tiles +- Added tooltips showing both timestamp and value -## Implementation Plan +### ✅ Fixed Requests/sec Tile +- Shows actual request counts (derived from connections) +- Trend line now shows request history, not throughput +- Consistent data between number and trend visualization -### Step 1: Fix Timestamp Format in updateTrafficData() -```typescript -// In ops-view-network.ts, line 548-554 -this.trafficData = Array.from({ length: 60 }, (_, i) => { - const time = now - (i * bucketSize); - return { - x: new Date(time).toISOString(), // Convert to ISO string - y: buckets.get(time) || 0, - }; -}).reverse(); +## Current Architecture + +### Data Flow +1. SmartProxy collects metrics via its internal MetricsCollector +2. MetricsManager retrieves data using `smartProxy.getMetrics()` +3. Handlers transform metrics for UI consumption +4. UI components display real-time data with auto-refresh + +### Key Components +- **MetricsManager**: Central metrics aggregation and tracking +- **SmartProxy Integration**: Uses grouped metrics API +- **UI Components**: ops-view-network shows real-time traffic graphs +- **State Management**: Uses appstate for reactive updates + +## Known Limitations +- Request counting is derived from connection data (not true HTTP request counts) +- Some metrics still need backend implementation (e.g., per-connection bytes) +- Historical data limited to current session + +## Testing +```bash +# Build and run +pnpm build +pnpm start + +# Check metrics endpoints +curl http://localhost:4000/api/stats/server +curl http://localhost:4000/api/stats/network ``` -### Step 2: Add Data Generation for Empty States -Create synthetic data points when no connections exist to show the chart grid: -```typescript -private updateTrafficData() { - // ... existing code ... - - // If no data, create zero-value points to show grid - if (this.networkRequests.length === 0) { - this.trafficData = Array.from({ length: 60 }, (_, i) => { - const time = now - (i * bucketSize); - return { - x: new Date(time).toISOString(), - y: 0, - }; - }).reverse(); - return; - } - - // ... rest of existing bucketing logic ... -} -``` - -### Step 3: Improve Bucket Alignment (Optional Enhancement) -Align buckets to start of time periods for cleaner data: -```typescript -// Calculate bucket start time -const bucketStartTime = Math.floor(req.timestamp / bucketSize) * bucketSize; -buckets.set(bucketStartTime, (buckets.get(bucketStartTime) || 0) + 1); -``` - -### Step 4: Add Debug Logging (Temporary) -Add console logs to verify data flow: -```typescript -console.log('Traffic data generated:', this.trafficData); -console.log('Network requests count:', this.networkRequests.length); -``` - -### Step 5: Update Chart Configuration -Ensure chart component has proper configuration: -```typescript - - -``` - -## Testing Plan - -1. **Test with no connections**: Verify chart shows grid with zero line -2. **Test with active connections**: Verify chart shows actual traffic data -3. **Test time range changes**: Verify chart updates when selecting different time ranges -4. **Test auto-refresh**: Verify chart updates every second with new data - -## Expected Outcome - -- Network traffic chart displays properly with time on x-axis -- Chart shows grid and zero line even when no data exists -- Real-time updates work correctly -- Time ranges (1m, 5m, 15m, 1h, 24h) all function properly - -## Implementation Order - -1. Fix timestamp format (critical fix) -2. Add empty state handling -3. Test basic functionality -4. Add debug logging if issues persist -5. Implement bucket alignment improvement if needed - -## Success Criteria - -- [ ] Chart displays time labels on x-axis -- [ ] Chart shows data points when connections exist -- [ ] Chart shows zero line when no connections exist -- [ ] Chart updates in real-time as new connections arrive -- [ ] All time range selections work correctly - -## Estimated Effort - -- Implementation: 30 minutes -- Testing: 15 minutes -- Total: 45 minutes \ No newline at end of file +## Success Metrics +- [x] Real-time throughput data displayed correctly +- [x] No mock data in production UI +- [x] Consistent units across all displays +- [x] Separate in/out traffic visualization +- [x] Working trend lines in stat tiles \ No newline at end of file diff --git a/ts_web/elements/ops-view-network.ts b/ts_web/elements/ops-view-network.ts index 46c80d7..ae3d876 100644 --- a/ts_web/elements/ops-view-network.ts +++ b/ts_web/elements/ops-view-network.ts @@ -43,7 +43,10 @@ export class OpsViewNetwork extends DeesElement { private networkRequests: INetworkRequest[] = []; @state() - private trafficData: Array<{ x: string | number; y: number }> = []; + private trafficDataIn: Array<{ x: string | number; y: number }> = []; + + @state() + private trafficDataOut: Array<{ x: string | number; y: number }> = []; @state() private isLoading = false; @@ -52,11 +55,9 @@ export class OpsViewNetwork extends DeesElement { private trafficUpdateInterval = 1000; // Update every 1 second private requestCountHistory = new Map(); // Track requests per time bucket private trafficUpdateTimer: any = null; + private requestsPerSecHistory: number[] = []; // Track requests/sec over time for trend - // Track bytes for calculating true per-second throughput - private lastBytesIn = 0; - private lastBytesOut = 0; - private lastBytesSampleTime = 0; + // Removed byte tracking - now using real-time data from SmartProxy constructor() { super(); @@ -96,8 +97,8 @@ export class OpsViewNetwork extends DeesElement { const range = timeRanges[this.selectedTimeRange]; const bucketSize = range / 60; - // Initialize with empty data points - this.trafficData = Array.from({ length: 60 }, (_, i) => { + // Initialize with empty data points for both in and out + const emptyData = Array.from({ length: 60 }, (_, i) => { const time = now - ((59 - i) * bucketSize); return { x: new Date(time).toISOString(), @@ -105,6 +106,9 @@ export class OpsViewNetwork extends DeesElement { }; }); + this.trafficDataIn = [...emptyData]; + this.trafficDataOut = emptyData.map(point => ({ ...point })); + this.lastTrafficUpdateTime = now; } @@ -265,11 +269,29 @@ export class OpsViewNetwork extends DeesElement { .label=${'Network Traffic'} .series=${[ { - name: 'Throughput (Mbps)', - data: this.trafficData, + name: 'Inbound', + data: this.trafficDataIn, + color: '#22c55e', // Green for download + }, + { + name: 'Outbound', + data: this.trafficDataOut, + color: '#8b5cf6', // Purple for upload } ]} - .yAxisFormatter=${(val: number) => `${val} Mbps`} + .stacked=${false} + .yAxisFormatter=${(val: number) => `${val} Mbit/s`} + .tooltipFormatter=${(point: any) => { + const mbps = point.y || 0; + const seriesName = point.series?.name || 'Throughput'; + const timestamp = new Date(point.x).toLocaleTimeString(); + return ` +
+
${timestamp}
+
${seriesName}: ${mbps.toFixed(2)} Mbit/s
+
+ `; + }} > @@ -416,12 +438,34 @@ export class OpsViewNetwork extends DeesElement { return `${size.toFixed(1)} ${units[unitIndex]}`; } + + private formatBitsPerSecond(bytesPerSecond: number): string { + const bitsPerSecond = bytesPerSecond * 8; // Convert bytes to bits + const units = ['bit/s', 'kbit/s', 'Mbit/s', 'Gbit/s']; + let size = bitsPerSecond; + let unitIndex = 0; + + while (size >= 1000 && unitIndex < units.length - 1) { + size /= 1000; // Use 1000 for bits (not 1024) + unitIndex++; + } + + return `${size.toFixed(1)} ${units[unitIndex]}`; + } private calculateRequestsPerSecond(): number { // Calculate from actual request data in the last minute const oneMinuteAgo = Date.now() - 60000; const recentRequests = this.networkRequests.filter(req => req.timestamp >= oneMinuteAgo); - return Math.round(recentRequests.length / 60); + const reqPerSec = Math.round(recentRequests.length / 60); + + // Track history for trend (keep last 20 values) + this.requestsPerSecHistory.push(reqPerSec); + if (this.requestsPerSecHistory.length > 20) { + this.requestsPerSecHistory.shift(); + } + + return reqPerSec; } private calculateThroughput(): { in: number; out: number } { @@ -437,12 +481,12 @@ export class OpsViewNetwork extends DeesElement { const throughput = this.calculateThroughput(); const activeConnections = this.statsState.serverStats?.activeConnections || 0; - // Use actual traffic data for trends (last 20 points) - const trendData = this.trafficData.slice(-20).map(point => point.y); + // Use request count history for the requests/sec trend + const trendData = [...this.requestsPerSecHistory]; - // If we don't have enough data, pad with the current value + // If we don't have enough data, pad with zeros while (trendData.length < 20) { - trendData.unshift(reqPerSec); + trendData.unshift(0); } const tiles: IStatsTile[] = [ @@ -471,13 +515,13 @@ export class OpsViewNetwork extends DeesElement { icon: 'chartLine', color: '#3b82f6', trendData: trendData, - description: `${this.formatNumber(reqPerSec)} req/s`, + description: `Average over last minute`, }, { id: 'throughputIn', title: 'Throughput In', - value: this.formatBytes(throughput.in), - unit: '/s', + value: this.formatBitsPerSecond(throughput.in), + unit: '', type: 'number', icon: 'download', color: '#22c55e', @@ -485,8 +529,8 @@ export class OpsViewNetwork extends DeesElement { { id: 'throughputOut', title: 'Throughput Out', - value: this.formatBytes(throughput.out), - unit: '/s', + value: this.formatBitsPerSecond(throughput.out), + unit: '', type: 'number', icon: 'upload', color: '#8b5cf6', @@ -586,77 +630,65 @@ export class OpsViewNetwork extends DeesElement { networkRequestsCount: this.networkRequests.length, timeSinceLastUpdate, shouldAddNewPoint, - currentDataPoints: this.trafficData.length + currentDataPoints: this.trafficDataIn.length }); - if (!shouldAddNewPoint && this.trafficData.length > 0) { + if (!shouldAddNewPoint && this.trafficDataIn.length > 0) { // Not enough time has passed, don't update return; } - // Calculate actual per-second throughput by tracking deltas - let throughputMbps = 0; + // Use real-time throughput data from SmartProxy (same as throughput tiles) + const throughput = this.calculateThroughput(); - // Get total bytes from all active connections - let currentBytesIn = 0; - let currentBytesOut = 0; + // Convert to Mbps (bytes * 8 / 1,000,000) + const throughputInMbps = (throughput.in * 8) / 1000000; + const throughputOutMbps = (throughput.out * 8) / 1000000; - this.networkRequests.forEach(req => { - currentBytesIn += req.bytesIn; - currentBytesOut += req.bytesOut; + console.log('Throughput calculation:', { + bytesInPerSecond: throughput.in, + bytesOutPerSecond: throughput.out, + throughputInMbps, + throughputOutMbps, + throughputTileValue: `${this.formatBitsPerSecond(throughput.in)} IN, ${this.formatBitsPerSecond(throughput.out)} OUT` }); - // If we have a previous sample, calculate the delta - if (this.lastBytesSampleTime > 0) { - const timeDelta = (now - this.lastBytesSampleTime) / 1000; // Convert to seconds - const bytesInDelta = Math.max(0, currentBytesIn - this.lastBytesIn); - const bytesOutDelta = Math.max(0, currentBytesOut - this.lastBytesOut); - - // Calculate bytes per second for this interval - const bytesPerSecond = (bytesInDelta + bytesOutDelta) / timeDelta; - - // Convert to Mbps (1 Mbps = 125000 bytes/second) - throughputMbps = bytesPerSecond / 125000; - - console.log('Throughput calculation:', { - timeDelta, - bytesInDelta, - bytesOutDelta, - bytesPerSecond, - throughputMbps - }); - } - - // Update last sample values - this.lastBytesIn = currentBytesIn; - this.lastBytesOut = currentBytesOut; - this.lastBytesSampleTime = now; - - if (this.trafficData.length === 0) { + if (this.trafficDataIn.length === 0) { // Initialize if empty this.initializeTrafficData(); } else { - // Add new data point and remove oldest if we have 60 points - const newDataPoint = { - x: new Date(now).toISOString(), - y: Math.round(throughputMbps * 10) / 10 // Round to 1 decimal place + // Add new data points for both in and out + const timestamp = new Date(now).toISOString(); + + const newDataPointIn = { + x: timestamp, + y: Math.round(throughputInMbps * 10) / 10 // Round to 1 decimal place }; - // Create new array with existing data plus new point - const newTrafficData = [...this.trafficData, newDataPoint]; + const newDataPointOut = { + x: timestamp, + y: Math.round(throughputOutMbps * 10) / 10 // Round to 1 decimal place + }; + + // Create new arrays with existing data plus new points + const newTrafficDataIn = [...this.trafficDataIn, newDataPointIn]; + const newTrafficDataOut = [...this.trafficDataOut, newDataPointOut]; // Keep only the last 60 points - if (newTrafficData.length > 60) { - newTrafficData.shift(); // Remove oldest point + if (newTrafficDataIn.length > 60) { + newTrafficDataIn.shift(); // Remove oldest point + newTrafficDataOut.shift(); } - this.trafficData = newTrafficData; + this.trafficDataIn = newTrafficDataIn; + this.trafficDataOut = newTrafficDataOut; this.lastTrafficUpdateTime = now; - console.log('Added new traffic data point:', { - timestamp: newDataPoint.x, - throughputMbps: newDataPoint.y, - totalPoints: this.trafficData.length + console.log('Added new traffic data points:', { + timestamp: timestamp, + throughputInMbps: newDataPointIn.y, + throughputOutMbps: newDataPointOut.y, + totalPoints: this.trafficDataIn.length }); } }