serve.zone/dcrouter
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(metrics): fix metrics

Browse Source
This commit is contained in:
Juergen Kunz
2025-06-23 00:19:47 +00:00
parent d24e51117d
commit 28cbf84f97
3 changed files with 188 additions and 185 deletions
Download Patch File Download Diff File Expand all files Collapse all files

25
readme.hints.md
View File

@ -1,5 +1,30 @@
# Implementation Hints and Learnings
## Network Metrics Implementation (2025-06-23)
### SmartProxy Metrics API Integration
- Updated to use new SmartProxy metrics API (v19.6.7)
- Use `getMetrics()` for detailed metrics with grouped methods:
```typescript
const metrics = smartProxy.getMetrics();
metrics.connections.active() // Current active connections
metrics.throughput.instant() // Real-time throughput {in, out}
metrics.connections.topIPs(10) // Top 10 IPs by connection count
```
- Use `getStatistics()` for basic stats
### Network Traffic Display
- All throughput values shown in bits per second (kbit/s, Mbit/s, Gbit/s)
- Conversion: `bytesPerSecond * 8 / 1000000` for Mbps
- Network graph shows separate lines for inbound (green) and outbound (purple)
- Throughput tiles and graph use same data source for consistency
### Requests/sec vs Connections
- Requests/sec shows HTTP request counts (derived from connections)
- Single connection can handle multiple requests
- Current implementation tracks connections, not individual requests
- Trend line shows historical request counts, not throughput
## DKIM Implementation Status (2025-05-30)
### Current Implementation

176
readme.plan2.md
View File

@ -1,125 +1,71 @@
# Network Traffic Graph Fix Plan
# Network Metrics Integration Status
## Command: `pnpm run reread`
## Command: `pnpm run build && curl https://code.foss.global/push.rocks/smartproxy/raw/branch/master/readme.md`
## Issue Summary
The network traffic graph in ops-view-network.ts is not displaying data due to three critical issues:
1. Timestamp format mismatch - chart expects ISO strings but receives numeric timestamps
2. Empty data when no active connections exist
3. Potential bucket alignment issues
## Completed Tasks (2025-06-23)
## Root Causes
### ✅ SmartProxy Metrics API Integration
- Updated MetricsManager to use new SmartProxy v19.6.7 metrics API
- Replaced deprecated `getStats()` with `getMetrics()` and `getStatistics()`
- Fixed method calls to use grouped API structure:
- `metrics.connections.active()` for active connections
- `metrics.throughput.instant()` for real-time throughput
- `metrics.connections.topIPs()` for top connected IPs
### 1. Timestamp Format Issue
- **Current**: `x: time` (numeric timestamp like 1703123456789)
- **Expected**: `x: new Date(time).toISOString()` (ISO string like "2023-12-20T12:34:56.789Z")
- **Impact**: ApexCharts cannot parse the x-axis values, resulting in no visible data
### ✅ Removed Mock Data
- Removed hardcoded `0.0.0.0` IPs in security.handler.ts
- Removed `Math.random()` trend data in ops-view-network.ts
- Now using real IP data from SmartProxy metrics
### 2. Empty Data Handling
- When no active connections exist, `networkRequests` array is empty
- Empty array leads to no buckets being created
- Chart shows flat line at 0
### ✅ Enhanced Metrics Functionality
- Email metrics: delivery time tracking, top recipients, activity log
- DNS metrics: query rate calculations, response time tracking
- Security metrics: incident logging with severity levels
### 3. Data Bucketing Logic
- Current logic creates buckets but uses numeric timestamps as Map keys
- This works for calculation but fails when looking up values for chart display
### ✅ Fixed Network Traffic Display
- All throughput now shown in bits per second (kbit/s, Mbit/s, Gbit/s)
- Network graph shows separate lines for inbound (green) and outbound (purple)
- Fixed throughput calculation to use same data source as tiles
- Added tooltips showing both timestamp and value
## Implementation Plan
### ✅ Fixed Requests/sec Tile
- Shows actual request counts (derived from connections)
- Trend line now shows request history, not throughput
- Consistent data between number and trend visualization
### Step 1: Fix Timestamp Format in updateTrafficData()
```typescript
// In ops-view-network.ts, line 548-554
this.trafficData = Array.from({ length: 60 }, (_, i) => {
const time = now - (i * bucketSize);
return {
x: new Date(time).toISOString(), // Convert to ISO string
y: buckets.get(time) || 0,
};
}).reverse();
## Current Architecture
### Data Flow
1. SmartProxy collects metrics via its internal MetricsCollector
2. MetricsManager retrieves data using `smartProxy.getMetrics()`
3. Handlers transform metrics for UI consumption
4. UI components display real-time data with auto-refresh
### Key Components
- **MetricsManager**: Central metrics aggregation and tracking
- **SmartProxy Integration**: Uses grouped metrics API
- **UI Components**: ops-view-network shows real-time traffic graphs
- **State Management**: Uses appstate for reactive updates
## Known Limitations
- Request counting is derived from connection data (not true HTTP request counts)
- Some metrics still need backend implementation (e.g., per-connection bytes)
- Historical data limited to current session
## Testing
```bash
# Build and run
pnpm build
pnpm start
# Check metrics endpoints
curl http://localhost:4000/api/stats/server
curl http://localhost:4000/api/stats/network
```
### Step 2: Add Data Generation for Empty States
Create synthetic data points when no connections exist to show the chart grid:
```typescript
private updateTrafficData() {
// ... existing code ...
// If no data, create zero-value points to show grid
if (this.networkRequests.length === 0) {
this.trafficData = Array.from({ length: 60 }, (_, i) => {
const time = now - (i * bucketSize);
return {
x: new Date(time).toISOString(),
y: 0,
};
}).reverse();
return;
}
// ... rest of existing bucketing logic ...
}
```
### Step 3: Improve Bucket Alignment (Optional Enhancement)
Align buckets to start of time periods for cleaner data:
```typescript
// Calculate bucket start time
const bucketStartTime = Math.floor(req.timestamp / bucketSize) * bucketSize;
buckets.set(bucketStartTime, (buckets.get(bucketStartTime) || 0) + 1);
```
### Step 4: Add Debug Logging (Temporary)
Add console logs to verify data flow:
```typescript
console.log('Traffic data generated:', this.trafficData);
console.log('Network requests count:', this.networkRequests.length);
```
### Step 5: Update Chart Configuration
Ensure chart component has proper configuration:
```typescript
<!-- Already correct in the template -->
<dees-chart-area
.label=${'Network Traffic'}
.series=${[{
name: 'Requests/min',
data: this.trafficData,
}]}
></dees-chart-area>
```
## Testing Plan
1. **Test with no connections**: Verify chart shows grid with zero line
2. **Test with active connections**: Verify chart shows actual traffic data
3. **Test time range changes**: Verify chart updates when selecting different time ranges
4. **Test auto-refresh**: Verify chart updates every second with new data
## Expected Outcome
- Network traffic chart displays properly with time on x-axis
- Chart shows grid and zero line even when no data exists
- Real-time updates work correctly
- Time ranges (1m, 5m, 15m, 1h, 24h) all function properly
## Implementation Order
1. Fix timestamp format (critical fix)
2. Add empty state handling
3. Test basic functionality
4. Add debug logging if issues persist
5. Implement bucket alignment improvement if needed
## Success Criteria
- [ ] Chart displays time labels on x-axis
- [ ] Chart shows data points when connections exist
- [ ] Chart shows zero line when no connections exist
- [ ] Chart updates in real-time as new connections arrive
- [ ] All time range selections work correctly
## Estimated Effort
- Implementation: 30 minutes
- Testing: 15 minutes
- Total: 45 minutes
## Success Metrics
- [x] Real-time throughput data displayed correctly
- [x] No mock data in production UI
- [x] Consistent units across all displays
- [x] Separate in/out traffic visualization
- [x] Working trend lines in stat tiles

172
ts_web/elements/ops-view-network.ts
View File

@ -43,7 +43,10 @@ export class OpsViewNetwork extends DeesElement {
private networkRequests: INetworkRequest[] = [];
@state()
private trafficData: Array<{ x: string | number; y: number }> = [];
private trafficDataIn: Array<{ x: string | number; y: number }> = [];
@state()
private trafficDataOut: Array<{ x: string | number; y: number }> = [];
@state()
private isLoading = false;
@ -52,11 +55,9 @@ export class OpsViewNetwork extends DeesElement {
private trafficUpdateInterval = 1000; // Update every 1 second
private requestCountHistory = new Map<number, number>(); // Track requests per time bucket
private trafficUpdateTimer: any = null;
private requestsPerSecHistory: number[] = []; // Track requests/sec over time for trend
// Track bytes for calculating true per-second throughput
private lastBytesIn = 0;
private lastBytesOut = 0;
private lastBytesSampleTime = 0;
// Removed byte tracking - now using real-time data from SmartProxy
constructor() {
super();
@ -96,8 +97,8 @@ export class OpsViewNetwork extends DeesElement {
const range = timeRanges[this.selectedTimeRange];
const bucketSize = range / 60;
// Initialize with empty data points
this.trafficData = Array.from({ length: 60 }, (_, i) => {
// Initialize with empty data points for both in and out
const emptyData = Array.from({ length: 60 }, (_, i) => {
const time = now - ((59 - i) * bucketSize);
return {
x: new Date(time).toISOString(),
@ -105,6 +106,9 @@ export class OpsViewNetwork extends DeesElement {
};
});
this.trafficDataIn = [...emptyData];
this.trafficDataOut = emptyData.map(point => ({ ...point }));
this.lastTrafficUpdateTime = now;
}
@ -265,11 +269,29 @@ export class OpsViewNetwork extends DeesElement {
.label=${'Network Traffic'}
.series=${[
{
name: 'Throughput (Mbps)',
data: this.trafficData,
name: 'Inbound',
data: this.trafficDataIn,
color: '#22c55e', // Green for download
},
{
name: 'Outbound',
data: this.trafficDataOut,
color: '#8b5cf6', // Purple for upload
}
]}
.yAxisFormatter=${(val: number) => `${val} Mbps`}
.stacked=${false}
.yAxisFormatter=${(val: number) => `${val} Mbit/s`}
.tooltipFormatter=${(point: any) => {
const mbps = point.y || 0;
const seriesName = point.series?.name || 'Throughput';
const timestamp = new Date(point.x).toLocaleTimeString();
return `
<div style="padding: 8px;">
<div style="font-weight: bold; margin-bottom: 4px;">${timestamp}</div>
<div>${seriesName}: ${mbps.toFixed(2)} Mbit/s</div>
</div>
`;
}}
></dees-chart-area>
<!-- Top IPs Section -->
@ -417,11 +439,33 @@ export class OpsViewNetwork extends DeesElement {
return `${size.toFixed(1)} ${units[unitIndex]}`;
}
private formatBitsPerSecond(bytesPerSecond: number): string {
const bitsPerSecond = bytesPerSecond * 8; // Convert bytes to bits
const units = ['bit/s', 'kbit/s', 'Mbit/s', 'Gbit/s'];
let size = bitsPerSecond;
let unitIndex = 0;
while (size >= 1000 && unitIndex < units.length - 1) {
size /= 1000; // Use 1000 for bits (not 1024)
unitIndex++;
}
return `${size.toFixed(1)} ${units[unitIndex]}`;
}
private calculateRequestsPerSecond(): number {
// Calculate from actual request data in the last minute
const oneMinuteAgo = Date.now() - 60000;
const recentRequests = this.networkRequests.filter(req => req.timestamp >= oneMinuteAgo);
return Math.round(recentRequests.length / 60);
const reqPerSec = Math.round(recentRequests.length / 60);
// Track history for trend (keep last 20 values)
this.requestsPerSecHistory.push(reqPerSec);
if (this.requestsPerSecHistory.length > 20) {
this.requestsPerSecHistory.shift();
}
return reqPerSec;
}
private calculateThroughput(): { in: number; out: number } {
@ -437,12 +481,12 @@ export class OpsViewNetwork extends DeesElement {
const throughput = this.calculateThroughput();
const activeConnections = this.statsState.serverStats?.activeConnections || 0;
// Use actual traffic data for trends (last 20 points)
const trendData = this.trafficData.slice(-20).map(point => point.y);
// Use request count history for the requests/sec trend
const trendData = [...this.requestsPerSecHistory];
// If we don't have enough data, pad with the current value
// If we don't have enough data, pad with zeros
while (trendData.length < 20) {
trendData.unshift(reqPerSec);
trendData.unshift(0);
}
const tiles: IStatsTile[] = [
@ -471,13 +515,13 @@ export class OpsViewNetwork extends DeesElement {
icon: 'chartLine',
color: '#3b82f6',
trendData: trendData,
description: `${this.formatNumber(reqPerSec)} req/s`,
description: `Average over last minute`,
},
{
id: 'throughputIn',
title: 'Throughput In',
value: this.formatBytes(throughput.in),
unit: '/s',
value: this.formatBitsPerSecond(throughput.in),
unit: '',
type: 'number',
icon: 'download',
color: '#22c55e',
@ -485,8 +529,8 @@ export class OpsViewNetwork extends DeesElement {
{
id: 'throughputOut',
title: 'Throughput Out',
value: this.formatBytes(throughput.out),
unit: '/s',
value: this.formatBitsPerSecond(throughput.out),
unit: '',
type: 'number',
icon: 'upload',
color: '#8b5cf6',
@ -586,77 +630,65 @@ export class OpsViewNetwork extends DeesElement {
networkRequestsCount: this.networkRequests.length,
timeSinceLastUpdate,
shouldAddNewPoint,
currentDataPoints: this.trafficData.length
currentDataPoints: this.trafficDataIn.length
});
if (!shouldAddNewPoint && this.trafficData.length > 0) {
if (!shouldAddNewPoint && this.trafficDataIn.length > 0) {
// Not enough time has passed, don't update
return;
}
// Calculate actual per-second throughput by tracking deltas
let throughputMbps = 0;
// Use real-time throughput data from SmartProxy (same as throughput tiles)
const throughput = this.calculateThroughput();
// Get total bytes from all active connections
let currentBytesIn = 0;
let currentBytesOut = 0;
// Convert to Mbps (bytes * 8 / 1,000,000)
const throughputInMbps = (throughput.in * 8) / 1000000;
const throughputOutMbps = (throughput.out * 8) / 1000000;
this.networkRequests.forEach(req => {
currentBytesIn += req.bytesIn;
currentBytesOut += req.bytesOut;
console.log('Throughput calculation:', {
bytesInPerSecond: throughput.in,
bytesOutPerSecond: throughput.out,
throughputInMbps,
throughputOutMbps,
throughputTileValue: `${this.formatBitsPerSecond(throughput.in)} IN, ${this.formatBitsPerSecond(throughput.out)} OUT`
});
// If we have a previous sample, calculate the delta
if (this.lastBytesSampleTime > 0) {
const timeDelta = (now - this.lastBytesSampleTime) / 1000; // Convert to seconds
const bytesInDelta = Math.max(0, currentBytesIn - this.lastBytesIn);
const bytesOutDelta = Math.max(0, currentBytesOut - this.lastBytesOut);
// Calculate bytes per second for this interval
const bytesPerSecond = (bytesInDelta + bytesOutDelta) / timeDelta;
// Convert to Mbps (1 Mbps = 125000 bytes/second)
throughputMbps = bytesPerSecond / 125000;
console.log('Throughput calculation:', {
timeDelta,
bytesInDelta,
bytesOutDelta,
bytesPerSecond,
throughputMbps
});
}
// Update last sample values
this.lastBytesIn = currentBytesIn;
this.lastBytesOut = currentBytesOut;
this.lastBytesSampleTime = now;
if (this.trafficData.length === 0) {
if (this.trafficDataIn.length === 0) {
// Initialize if empty
this.initializeTrafficData();
} else {
// Add new data point and remove oldest if we have 60 points
const newDataPoint = {
x: new Date(now).toISOString(),
y: Math.round(throughputMbps * 10) / 10 // Round to 1 decimal place
// Add new data points for both in and out
const timestamp = new Date(now).toISOString();
const newDataPointIn = {
x: timestamp,
y: Math.round(throughputInMbps * 10) / 10 // Round to 1 decimal place
};
// Create new array with existing data plus new point
const newTrafficData = [...this.trafficData, newDataPoint];
const newDataPointOut = {
x: timestamp,
y: Math.round(throughputOutMbps * 10) / 10 // Round to 1 decimal place
};
// Create new arrays with existing data plus new points
const newTrafficDataIn = [...this.trafficDataIn, newDataPointIn];
const newTrafficDataOut = [...this.trafficDataOut, newDataPointOut];
// Keep only the last 60 points
if (newTrafficData.length > 60) {
newTrafficData.shift(); // Remove oldest point
if (newTrafficDataIn.length > 60) {
newTrafficDataIn.shift(); // Remove oldest point
newTrafficDataOut.shift();
}
this.trafficData = newTrafficData;
this.trafficDataIn = newTrafficDataIn;
this.trafficDataOut = newTrafficDataOut;
this.lastTrafficUpdateTime = now;
console.log('Added new traffic data point:', {
timestamp: newDataPoint.x,
throughputMbps: newDataPoint.y,
totalPoints: this.trafficData.length
console.log('Added new traffic data points:', {
timestamp: timestamp,
throughputInMbps: newDataPointIn.y,
throughputOutMbps: newDataPointOut.y,
totalPoints: this.trafficDataIn.length
});
}
}