serve.zone/dcrouter
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(metrics): fix metrics

Browse Source
This commit is contained in:
Juergen Kunz
2025-06-23 00:19:47 +00:00
parent d24e51117d
commit 28cbf84f97
3 changed files with 188 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
readme.hints.md
View File

@ -1,5 +1,30 @@
# Implementation Hints and Learnings # Implementation Hints and Learnings
## Network Metrics Implementation (2025-06-23)
### SmartProxy Metrics API Integration
- Updated to use new SmartProxy metrics API (v19.6.7)
- Use `getMetrics()` for detailed metrics with grouped methods:
```typescript
const metrics = smartProxy.getMetrics();
metrics.connections.active() // Current active connections
metrics.throughput.instant() // Real-time throughput {in, out}
metrics.connections.topIPs(10) // Top 10 IPs by connection count
```
- Use `getStatistics()` for basic stats
### Network Traffic Display
- All throughput values shown in bits per second (kbit/s, Mbit/s, Gbit/s)
- Conversion: `bytesPerSecond * 8 / 1000000` for Mbps
- Network graph shows separate lines for inbound (green) and outbound (purple)
- Throughput tiles and graph use same data source for consistency
### Requests/sec vs Connections
- Requests/sec shows HTTP request counts (derived from connections)
- Single connection can handle multiple requests
- Current implementation tracks connections, not individual requests
- Trend line shows historical request counts, not throughput
## DKIM Implementation Status (2025-05-30) ## DKIM Implementation Status (2025-05-30)
### Current Implementation ### Current Implementation

176
readme.plan2.md
View File

@ -1,125 +1,71 @@
# Network Traffic Graph Fix Plan # Network Metrics Integration Status
## Command: `pnpm run reread` ## Command: `pnpm run build && curl https://code.foss.global/push.rocks/smartproxy/raw/branch/master/readme.md`
## Issue Summary ## Completed Tasks (2025-06-23)
The network traffic graph in ops-view-network.ts is not displaying data due to three critical issues:
1. Timestamp format mismatch - chart expects ISO strings but receives numeric timestamps
2. Empty data when no active connections exist
3. Potential bucket alignment issues
## Root Causes ### ✅ SmartProxy Metrics API Integration
- Updated MetricsManager to use new SmartProxy v19.6.7 metrics API
- Replaced deprecated `getStats()` with `getMetrics()` and `getStatistics()`
- Fixed method calls to use grouped API structure:
- `metrics.connections.active()` for active connections
- `metrics.throughput.instant()` for real-time throughput
- `metrics.connections.topIPs()` for top connected IPs
### 1. Timestamp Format Issue ### ✅ Removed Mock Data
- **Current**: `x: time` (numeric timestamp like 1703123456789) - Removed hardcoded `0.0.0.0` IPs in security.handler.ts
- **Expected**: `x: new Date(time).toISOString()` (ISO string like "2023-12-20T12:34:56.789Z") - Removed `Math.random()` trend data in ops-view-network.ts
- **Impact**: ApexCharts cannot parse the x-axis values, resulting in no visible data - Now using real IP data from SmartProxy metrics
### 2. Empty Data Handling ### ✅ Enhanced Metrics Functionality
- When no active connections exist, `networkRequests` array is empty - Email metrics: delivery time tracking, top recipients, activity log
- Empty array leads to no buckets being created - DNS metrics: query rate calculations, response time tracking
- Chart shows flat line at 0 - Security metrics: incident logging with severity levels
### 3. Data Bucketing Logic ### ✅ Fixed Network Traffic Display
- Current logic creates buckets but uses numeric timestamps as Map keys - All throughput now shown in bits per second (kbit/s, Mbit/s, Gbit/s)
- This works for calculation but fails when looking up values for chart display - Network graph shows separate lines for inbound (green) and outbound (purple)
- Fixed throughput calculation to use same data source as tiles
- Added tooltips showing both timestamp and value
## Implementation Plan ### ✅ Fixed Requests/sec Tile
- Shows actual request counts (derived from connections)
- Trend line now shows request history, not throughput
- Consistent data between number and trend visualization
### Step 1: Fix Timestamp Format in updateTrafficData() ## Current Architecture
```typescript
// In ops-view-network.ts, line 548-554 ### Data Flow
this.trafficData = Array.from({ length: 60 }, (_, i) => { 1. SmartProxy collects metrics via its internal MetricsCollector
const time = now - (i * bucketSize); 2. MetricsManager retrieves data using `smartProxy.getMetrics()`
return { 3. Handlers transform metrics for UI consumption
x: new Date(time).toISOString(), // Convert to ISO string 4. UI components display real-time data with auto-refresh
y: buckets.get(time) || 0,
}; ### Key Components
}).reverse(); - **MetricsManager**: Central metrics aggregation and tracking
- **SmartProxy Integration**: Uses grouped metrics API
- **UI Components**: ops-view-network shows real-time traffic graphs
- **State Management**: Uses appstate for reactive updates
## Known Limitations
- Request counting is derived from connection data (not true HTTP request counts)
- Some metrics still need backend implementation (e.g., per-connection bytes)
- Historical data limited to current session
## Testing
```bash
# Build and run
pnpm build
pnpm start
# Check metrics endpoints
curl http://localhost:4000/api/stats/server
curl http://localhost:4000/api/stats/network
``` ```
### Step 2: Add Data Generation for Empty States ## Success Metrics
Create synthetic data points when no connections exist to show the chart grid: - [x] Real-time throughput data displayed correctly
```typescript - [x] No mock data in production UI
private updateTrafficData() { - [x] Consistent units across all displays
// ... existing code ... - [x] Separate in/out traffic visualization
- [x] Working trend lines in stat tiles
// If no data, create zero-value points to show grid
if (this.networkRequests.length === 0) {
this.trafficData = Array.from({ length: 60 }, (_, i) => {
const time = now - (i * bucketSize);
return {
x: new Date(time).toISOString(),
y: 0,
};
}).reverse();
return;
}
// ... rest of existing bucketing logic ...
}
```
### Step 3: Improve Bucket Alignment (Optional Enhancement)
Align buckets to start of time periods for cleaner data:
```typescript
// Calculate bucket start time
const bucketStartTime = Math.floor(req.timestamp / bucketSize) * bucketSize;
buckets.set(bucketStartTime, (buckets.get(bucketStartTime) || 0) + 1);
```
### Step 4: Add Debug Logging (Temporary)
Add console logs to verify data flow:
```typescript
console.log('Traffic data generated:', this.trafficData);
console.log('Network requests count:', this.networkRequests.length);
```
### Step 5: Update Chart Configuration
Ensure chart component has proper configuration:
```typescript
<!-- Already correct in the template -->
<dees-chart-area
.label=${'Network Traffic'}
.series=${[{
name: 'Requests/min',
data: this.trafficData,
}]}
></dees-chart-area>
```
## Testing Plan
1. **Test with no connections**: Verify chart shows grid with zero line
2. **Test with active connections**: Verify chart shows actual traffic data
3. **Test time range changes**: Verify chart updates when selecting different time ranges
4. **Test auto-refresh**: Verify chart updates every second with new data
## Expected Outcome
- Network traffic chart displays properly with time on x-axis
- Chart shows grid and zero line even when no data exists
- Real-time updates work correctly
- Time ranges (1m, 5m, 15m, 1h, 24h) all function properly
## Implementation Order
1. Fix timestamp format (critical fix)
2. Add empty state handling
3. Test basic functionality
4. Add debug logging if issues persist
5. Implement bucket alignment improvement if needed
## Success Criteria
- [ ] Chart displays time labels on x-axis
- [ ] Chart shows data points when connections exist
- [ ] Chart shows zero line when no connections exist
- [ ] Chart updates in real-time as new connections arrive
- [ ] All time range selections work correctly
## Estimated Effort
- Implementation: 30 minutes
- Testing: 15 minutes
- Total: 45 minutes

172
ts_web/elements/ops-view-network.ts
View File

@ -43,7 +43,10 @@ export class OpsViewNetwork extends DeesElement {
private networkRequests: INetworkRequest[] = []; private networkRequests: INetworkRequest[] = [];
@state() @state()
private trafficData: Array<{ x: string | number; y: number }> = []; private trafficDataIn: Array<{ x: string | number; y: number }> = [];
@state()
private trafficDataOut: Array<{ x: string | number; y: number }> = [];
@state() @state()
private isLoading = false; private isLoading = false;
@ -52,11 +55,9 @@ export class OpsViewNetwork extends DeesElement {
private trafficUpdateInterval = 1000; // Update every 1 second private trafficUpdateInterval = 1000; // Update every 1 second
private requestCountHistory = new Map<number, number>(); // Track requests per time bucket private requestCountHistory = new Map<number, number>(); // Track requests per time bucket
private trafficUpdateTimer: any = null; private trafficUpdateTimer: any = null;
private requestsPerSecHistory: number[] = []; // Track requests/sec over time for trend
// Track bytes for calculating true per-second throughput // Removed byte tracking - now using real-time data from SmartProxy
private lastBytesIn = 0;
private lastBytesOut = 0;
private lastBytesSampleTime = 0;
constructor() { constructor() {
super(); super();
@ -96,8 +97,8 @@ export class OpsViewNetwork extends DeesElement {
const range = timeRanges[this.selectedTimeRange]; const range = timeRanges[this.selectedTimeRange];
const bucketSize = range / 60; const bucketSize = range / 60;
// Initialize with empty data points // Initialize with empty data points for both in and out
this.trafficData = Array.from({ length: 60 }, (_, i) => { const emptyData = Array.from({ length: 60 }, (_, i) => {
const time = now - ((59 - i) * bucketSize); const time = now - ((59 - i) * bucketSize);
return { return {
x: new Date(time).toISOString(), x: new Date(time).toISOString(),
@ -105,6 +106,9 @@ export class OpsViewNetwork extends DeesElement {
}; };
}); });
this.trafficDataIn = [...emptyData];
this.trafficDataOut = emptyData.map(point => ({ ...point }));
this.lastTrafficUpdateTime = now; this.lastTrafficUpdateTime = now;
} }
@ -265,11 +269,29 @@ export class OpsViewNetwork extends DeesElement {
.label=${'Network Traffic'} .label=${'Network Traffic'}
.series=${[ .series=${[
{ {
name: 'Throughput (Mbps)', name: 'Inbound',
data: this.trafficData, data: this.trafficDataIn,
color: '#22c55e', // Green for download
},
{
name: 'Outbound',
data: this.trafficDataOut,
color: '#8b5cf6', // Purple for upload
} }
]} ]}
.yAxisFormatter=${(val: number) => `${val} Mbps`} .stacked=${false}
.yAxisFormatter=${(val: number) => `${val} Mbit/s`}
.tooltipFormatter=${(point: any) => {
const mbps = point.y || 0;
const seriesName = point.series?.name || 'Throughput';
const timestamp = new Date(point.x).toLocaleTimeString();
return `
<div style="padding: 8px;">
<div style="font-weight: bold; margin-bottom: 4px;">${timestamp}</div>
<div>${seriesName}: ${mbps.toFixed(2)} Mbit/s</div>
</div>
`;
}}
></dees-chart-area> ></dees-chart-area>
<!-- Top IPs Section --> <!-- Top IPs Section -->
@ -416,12 +438,34 @@ export class OpsViewNetwork extends DeesElement {
return `${size.toFixed(1)} ${units[unitIndex]}`; return `${size.toFixed(1)} ${units[unitIndex]}`;
} }
private formatBitsPerSecond(bytesPerSecond: number): string {
const bitsPerSecond = bytesPerSecond * 8; // Convert bytes to bits
const units = ['bit/s', 'kbit/s', 'Mbit/s', 'Gbit/s'];
let size = bitsPerSecond;
let unitIndex = 0;
while (size >= 1000 && unitIndex < units.length - 1) {
size /= 1000; // Use 1000 for bits (not 1024)
unitIndex++;
}
return `${size.toFixed(1)} ${units[unitIndex]}`;
}
private calculateRequestsPerSecond(): number { private calculateRequestsPerSecond(): number {
// Calculate from actual request data in the last minute // Calculate from actual request data in the last minute
const oneMinuteAgo = Date.now() - 60000; const oneMinuteAgo = Date.now() - 60000;
const recentRequests = this.networkRequests.filter(req => req.timestamp >= oneMinuteAgo); const recentRequests = this.networkRequests.filter(req => req.timestamp >= oneMinuteAgo);
return Math.round(recentRequests.length / 60); const reqPerSec = Math.round(recentRequests.length / 60);
// Track history for trend (keep last 20 values)
this.requestsPerSecHistory.push(reqPerSec);
if (this.requestsPerSecHistory.length > 20) {
this.requestsPerSecHistory.shift();
}
return reqPerSec;
} }
private calculateThroughput(): { in: number; out: number } { private calculateThroughput(): { in: number; out: number } {
@ -437,12 +481,12 @@ export class OpsViewNetwork extends DeesElement {
const throughput = this.calculateThroughput(); const throughput = this.calculateThroughput();
const activeConnections = this.statsState.serverStats?.activeConnections || 0; const activeConnections = this.statsState.serverStats?.activeConnections || 0;
// Use actual traffic data for trends (last 20 points) // Use request count history for the requests/sec trend
const trendData = this.trafficData.slice(-20).map(point => point.y); const trendData = [...this.requestsPerSecHistory];
// If we don't have enough data, pad with the current value // If we don't have enough data, pad with zeros
while (trendData.length < 20) { while (trendData.length < 20) {
trendData.unshift(reqPerSec); trendData.unshift(0);
} }
const tiles: IStatsTile[] = [ const tiles: IStatsTile[] = [
@ -471,13 +515,13 @@ export class OpsViewNetwork extends DeesElement {
icon: 'chartLine', icon: 'chartLine',
color: '#3b82f6', color: '#3b82f6',
trendData: trendData, trendData: trendData,
description: `${this.formatNumber(reqPerSec)} req/s`, description: `Average over last minute`,
}, },
{ {
id: 'throughputIn', id: 'throughputIn',
title: 'Throughput In', title: 'Throughput In',
value: this.formatBytes(throughput.in), value: this.formatBitsPerSecond(throughput.in),
unit: '/s', unit: '',
type: 'number', type: 'number',
icon: 'download', icon: 'download',
color: '#22c55e', color: '#22c55e',
@ -485,8 +529,8 @@ export class OpsViewNetwork extends DeesElement {
{ {
id: 'throughputOut', id: 'throughputOut',
title: 'Throughput Out', title: 'Throughput Out',
value: this.formatBytes(throughput.out), value: this.formatBitsPerSecond(throughput.out),
unit: '/s', unit: '',
type: 'number', type: 'number',
icon: 'upload', icon: 'upload',
color: '#8b5cf6', color: '#8b5cf6',
@ -586,77 +630,65 @@ export class OpsViewNetwork extends DeesElement {
networkRequestsCount: this.networkRequests.length, networkRequestsCount: this.networkRequests.length,
timeSinceLastUpdate, timeSinceLastUpdate,
shouldAddNewPoint, shouldAddNewPoint,
currentDataPoints: this.trafficData.length currentDataPoints: this.trafficDataIn.length
}); });
if (!shouldAddNewPoint && this.trafficData.length > 0) { if (!shouldAddNewPoint && this.trafficDataIn.length > 0) {
// Not enough time has passed, don't update // Not enough time has passed, don't update
return; return;
} }
// Calculate actual per-second throughput by tracking deltas // Use real-time throughput data from SmartProxy (same as throughput tiles)
let throughputMbps = 0; const throughput = this.calculateThroughput();
// Get total bytes from all active connections // Convert to Mbps (bytes * 8 / 1,000,000)
let currentBytesIn = 0; const throughputInMbps = (throughput.in * 8) / 1000000;
let currentBytesOut = 0; const throughputOutMbps = (throughput.out * 8) / 1000000;
this.networkRequests.forEach(req => { console.log('Throughput calculation:', {
currentBytesIn += req.bytesIn; bytesInPerSecond: throughput.in,
currentBytesOut += req.bytesOut; bytesOutPerSecond: throughput.out,
throughputInMbps,
throughputOutMbps,
throughputTileValue: `${this.formatBitsPerSecond(throughput.in)} IN, ${this.formatBitsPerSecond(throughput.out)} OUT`
}); });
// If we have a previous sample, calculate the delta if (this.trafficDataIn.length === 0) {
if (this.lastBytesSampleTime > 0) {
const timeDelta = (now - this.lastBytesSampleTime) / 1000; // Convert to seconds
const bytesInDelta = Math.max(0, currentBytesIn - this.lastBytesIn);
const bytesOutDelta = Math.max(0, currentBytesOut - this.lastBytesOut);
// Calculate bytes per second for this interval
const bytesPerSecond = (bytesInDelta + bytesOutDelta) / timeDelta;
// Convert to Mbps (1 Mbps = 125000 bytes/second)
throughputMbps = bytesPerSecond / 125000;
console.log('Throughput calculation:', {
timeDelta,
bytesInDelta,
bytesOutDelta,
bytesPerSecond,
throughputMbps
});
}
// Update last sample values
this.lastBytesIn = currentBytesIn;
this.lastBytesOut = currentBytesOut;
this.lastBytesSampleTime = now;
if (this.trafficData.length === 0) {
// Initialize if empty // Initialize if empty
this.initializeTrafficData(); this.initializeTrafficData();
} else { } else {
// Add new data point and remove oldest if we have 60 points // Add new data points for both in and out
const newDataPoint = { const timestamp = new Date(now).toISOString();
x: new Date(now).toISOString(),
y: Math.round(throughputMbps * 10) / 10 // Round to 1 decimal place const newDataPointIn = {
x: timestamp,
y: Math.round(throughputInMbps * 10) / 10 // Round to 1 decimal place
}; };
// Create new array with existing data plus new point const newDataPointOut = {
const newTrafficData = [...this.trafficData, newDataPoint]; x: timestamp,
y: Math.round(throughputOutMbps * 10) / 10 // Round to 1 decimal place
};
// Create new arrays with existing data plus new points
const newTrafficDataIn = [...this.trafficDataIn, newDataPointIn];
const newTrafficDataOut = [...this.trafficDataOut, newDataPointOut];
// Keep only the last 60 points // Keep only the last 60 points
if (newTrafficData.length > 60) { if (newTrafficDataIn.length > 60) {
newTrafficData.shift(); // Remove oldest point newTrafficDataIn.shift(); // Remove oldest point
newTrafficDataOut.shift();
} }
this.trafficData = newTrafficData; this.trafficDataIn = newTrafficDataIn;
this.trafficDataOut = newTrafficDataOut;
this.lastTrafficUpdateTime = now; this.lastTrafficUpdateTime = now;
console.log('Added new traffic data point:', { console.log('Added new traffic data points:', {
timestamp: newDataPoint.x, timestamp: timestamp,
throughputMbps: newDataPoint.y, throughputInMbps: newDataPointIn.y,
totalPoints: this.trafficData.length throughputOutMbps: newDataPointOut.y,
totalPoints: this.trafficDataIn.length
}); });
} }
} }