fix(metrics): fix metrics
This commit is contained in:
Juergen Kunz
176
readme.plan2.md
176
readme.plan2.md
@ -1,125 +1,71 @@
|
||||
# Network Traffic Graph Fix Plan
|
||||
# Network Metrics Integration Status
|
||||
|
||||
## Command: `pnpm run reread`
|
||||
## Command: `pnpm run build && curl https://code.foss.global/push.rocks/smartproxy/raw/branch/master/readme.md`
|
||||
|
||||
## Issue Summary
|
||||
The network traffic graph in ops-view-network.ts is not displaying data due to three critical issues:
|
||||
1. Timestamp format mismatch - chart expects ISO strings but receives numeric timestamps
|
||||
2. Empty data when no active connections exist
|
||||
3. Potential bucket alignment issues
|
||||
## Completed Tasks (2025-06-23)
|
||||
|
||||
## Root Causes
|
||||
### ✅ SmartProxy Metrics API Integration
|
||||
- Updated MetricsManager to use new SmartProxy v19.6.7 metrics API
|
||||
- Replaced deprecated `getStats()` with `getMetrics()` and `getStatistics()`
|
||||
- Fixed method calls to use grouped API structure:
|
||||
- `metrics.connections.active()` for active connections
|
||||
- `metrics.throughput.instant()` for real-time throughput
|
||||
- `metrics.connections.topIPs()` for top connected IPs
|
||||
|
||||
### 1. Timestamp Format Issue
|
||||
- **Current**: `x: time` (numeric timestamp like 1703123456789)
|
||||
- **Expected**: `x: new Date(time).toISOString()` (ISO string like "2023-12-20T12:34:56.789Z")
|
||||
- **Impact**: ApexCharts cannot parse the x-axis values, resulting in no visible data
|
||||
### ✅ Removed Mock Data
|
||||
- Removed hardcoded `0.0.0.0` IPs in security.handler.ts
|
||||
- Removed `Math.random()` trend data in ops-view-network.ts
|
||||
- Now using real IP data from SmartProxy metrics
|
||||
|
||||
### 2. Empty Data Handling
|
||||
- When no active connections exist, `networkRequests` array is empty
|
||||
- Empty array leads to no buckets being created
|
||||
- Chart shows flat line at 0
|
||||
### ✅ Enhanced Metrics Functionality
|
||||
- Email metrics: delivery time tracking, top recipients, activity log
|
||||
- DNS metrics: query rate calculations, response time tracking
|
||||
- Security metrics: incident logging with severity levels
|
||||
|
||||
### 3. Data Bucketing Logic
|
||||
- Current logic creates buckets but uses numeric timestamps as Map keys
|
||||
- This works for calculation but fails when looking up values for chart display
|
||||
### ✅ Fixed Network Traffic Display
|
||||
- All throughput now shown in bits per second (kbit/s, Mbit/s, Gbit/s)
|
||||
- Network graph shows separate lines for inbound (green) and outbound (purple)
|
||||
- Fixed throughput calculation to use same data source as tiles
|
||||
- Added tooltips showing both timestamp and value
|
||||
|
||||
## Implementation Plan
|
||||
### ✅ Fixed Requests/sec Tile
|
||||
- Shows actual request counts (derived from connections)
|
||||
- Trend line now shows request history, not throughput
|
||||
- Consistent data between number and trend visualization
|
||||
|
||||
### Step 1: Fix Timestamp Format in updateTrafficData()
|
||||
```typescript
|
||||
// In ops-view-network.ts, line 548-554
|
||||
this.trafficData = Array.from({ length: 60 }, (_, i) => {
|
||||
const time = now - (i * bucketSize);
|
||||
return {
|
||||
x: new Date(time).toISOString(), // Convert to ISO string
|
||||
y: buckets.get(time) || 0,
|
||||
};
|
||||
}).reverse();
|
||||
## Current Architecture
|
||||
|
||||
### Data Flow
|
||||
1. SmartProxy collects metrics via its internal MetricsCollector
|
||||
2. MetricsManager retrieves data using `smartProxy.getMetrics()`
|
||||
3. Handlers transform metrics for UI consumption
|
||||
4. UI components display real-time data with auto-refresh
|
||||
|
||||
### Key Components
|
||||
- **MetricsManager**: Central metrics aggregation and tracking
|
||||
- **SmartProxy Integration**: Uses grouped metrics API
|
||||
- **UI Components**: ops-view-network shows real-time traffic graphs
|
||||
- **State Management**: Uses appstate for reactive updates
|
||||
|
||||
## Known Limitations
|
||||
- Request counting is derived from connection data (not true HTTP request counts)
|
||||
- Some metrics still need backend implementation (e.g., per-connection bytes)
|
||||
- Historical data limited to current session
|
||||
|
||||
## Testing
|
||||
```bash
|
||||
# Build and run
|
||||
pnpm build
|
||||
pnpm start
|
||||
|
||||
# Check metrics endpoints
|
||||
curl http://localhost:4000/api/stats/server
|
||||
curl http://localhost:4000/api/stats/network
|
||||
```
|
||||
|
||||
### Step 2: Add Data Generation for Empty States
|
||||
Create synthetic data points when no connections exist to show the chart grid:
|
||||
```typescript
|
||||
private updateTrafficData() {
|
||||
// ... existing code ...
|
||||
|
||||
// If no data, create zero-value points to show grid
|
||||
if (this.networkRequests.length === 0) {
|
||||
this.trafficData = Array.from({ length: 60 }, (_, i) => {
|
||||
const time = now - (i * bucketSize);
|
||||
return {
|
||||
x: new Date(time).toISOString(),
|
||||
y: 0,
|
||||
};
|
||||
}).reverse();
|
||||
return;
|
||||
}
|
||||
|
||||
// ... rest of existing bucketing logic ...
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Improve Bucket Alignment (Optional Enhancement)
|
||||
Align buckets to start of time periods for cleaner data:
|
||||
```typescript
|
||||
// Calculate bucket start time
|
||||
const bucketStartTime = Math.floor(req.timestamp / bucketSize) * bucketSize;
|
||||
buckets.set(bucketStartTime, (buckets.get(bucketStartTime) || 0) + 1);
|
||||
```
|
||||
|
||||
### Step 4: Add Debug Logging (Temporary)
|
||||
Add console logs to verify data flow:
|
||||
```typescript
|
||||
console.log('Traffic data generated:', this.trafficData);
|
||||
console.log('Network requests count:', this.networkRequests.length);
|
||||
```
|
||||
|
||||
### Step 5: Update Chart Configuration
|
||||
Ensure chart component has proper configuration:
|
||||
```typescript
|
||||
<!-- Already correct in the template -->
|
||||
<dees-chart-area
|
||||
.label=${'Network Traffic'}
|
||||
.series=${[{
|
||||
name: 'Requests/min',
|
||||
data: this.trafficData,
|
||||
}]}
|
||||
></dees-chart-area>
|
||||
```
|
||||
|
||||
## Testing Plan
|
||||
|
||||
1. **Test with no connections**: Verify chart shows grid with zero line
|
||||
2. **Test with active connections**: Verify chart shows actual traffic data
|
||||
3. **Test time range changes**: Verify chart updates when selecting different time ranges
|
||||
4. **Test auto-refresh**: Verify chart updates every second with new data
|
||||
|
||||
## Expected Outcome
|
||||
|
||||
- Network traffic chart displays properly with time on x-axis
|
||||
- Chart shows grid and zero line even when no data exists
|
||||
- Real-time updates work correctly
|
||||
- Time ranges (1m, 5m, 15m, 1h, 24h) all function properly
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. Fix timestamp format (critical fix)
|
||||
2. Add empty state handling
|
||||
3. Test basic functionality
|
||||
4. Add debug logging if issues persist
|
||||
5. Implement bucket alignment improvement if needed
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Chart displays time labels on x-axis
|
||||
- [ ] Chart shows data points when connections exist
|
||||
- [ ] Chart shows zero line when no connections exist
|
||||
- [ ] Chart updates in real-time as new connections arrive
|
||||
- [ ] All time range selections work correctly
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
- Implementation: 30 minutes
|
||||
- Testing: 15 minutes
|
||||
- Total: 45 minutes
|
||||
## Success Metrics
|
||||
- [x] Real-time throughput data displayed correctly
|
||||
- [x] No mock data in production UI
|
||||
- [x] Consistent units across all displays
|
||||
- [x] Separate in/out traffic visualization
|
||||
- [x] Working trend lines in stat tiles
|
||||
Reference in New Issue
Block a user