773 lines
27 KiB
Markdown
773 lines
27 KiB
Markdown
# Implementation Hints and Learnings
|
|
|
|
## smartmta Migration (2026-02-11)
|
|
|
|
### Overview
|
|
dcrouter's custom MTA code (~27,149 lines / 68 files in `ts/mail/` + `ts/deliverability/`) has been replaced with `@push.rocks/smartmta` v5.2.1, a TypeScript+Rust hybrid MTA. dcrouter is now an orchestrator that wires together SmartProxy, smartmta, smartdns, smartradius, and OpsServer.
|
|
|
|
### Architecture
|
|
- **No socket-handler mode** — smartmta's Rust SMTP server binds its own ports directly
|
|
- **SmartProxy forward mode only** — external email ports forwarded to internal ports where smartmta listens
|
|
- Email traffic flow: External Port → SmartProxy → Internal Port → smartmta UnifiedEmailServer
|
|
|
|
### Key API Differences (smartmta vs old custom MTA)
|
|
- `updateEmailRoutes()` instead of `updateRoutes()`
|
|
- `dkimCreator` is public (no need for `(this.emailServer as any).dkimCreator`)
|
|
- `bounceManager` is private, but exposed via public methods:
|
|
- `emailServer.getSuppressionList()`
|
|
- `emailServer.getHardBouncedAddresses()`
|
|
- `emailServer.getBounceHistory(email)`
|
|
- `emailServer.removeFromSuppressionList(email)`
|
|
- `Email` class imported from `@push.rocks/smartmta`
|
|
- `IAttachment` type accessed via `Core` namespace: `import { type Core } from '@push.rocks/smartmta'; type IAttachment = Core.IAttachment;`
|
|
|
|
### Deleted Directories
|
|
- `ts/mail/` (60 files) — replaced by smartmta
|
|
- `ts/deliverability/` (3 files) — IPWarmupManager/SenderReputationMonitor will move to smartmta
|
|
- `ts/errors/email.errors.ts`, `ts/errors/mta.errors.ts` — smartmta has its own errors
|
|
- `ts/cache/documents/classes.cached.bounce.ts`, `classes.cached.suppression.ts`, `classes.cached.dkim.ts` — smartmta handles its own persistence
|
|
|
|
### Remaining Cache Documents
|
|
- `CachedEmail` — kept (dcrouter-level queue persistence)
|
|
- `CachedIPReputation` — kept (dcrouter-level IP reputation caching)
|
|
|
|
### Dependencies Removed
|
|
mailauth, mailparser, @types/mailparser, ip, @push.rocks/smartmail, @push.rocks/smartrule, node-forge
|
|
|
|
### Pre-existing Test Failures (not caused by migration)
|
|
- `test/test.jwt-auth.ts` — `response.text is not a function` (webrequest compatibility issue)
|
|
- `test/test.opsserver-api.ts` — same webrequest issue, timeouts
|
|
|
|
### smartmta Location
|
|
Source at `../../push.rocks/smartmta`, release with `gitzone commit -ypbrt`
|
|
|
|
## Dependency Upgrade (2026-02-11)
|
|
|
|
### SmartProxy v23.1.2 Route Validation
|
|
- SmartProxy 23.1.2 enforces stricter route validation
|
|
- Forward actions MUST use `targets` (array) instead of `target` (singular)
|
|
- Test configurations that call `DcRouter.start()` need `cacheConfig: { enabled: false }` to avoid starting a real MongoDB process in tests
|
|
|
|
```typescript
|
|
// WRONG - will fail validation
|
|
action: { type: 'forward', target: { host: 'localhost', port: 10025 } }
|
|
|
|
// CORRECT
|
|
action: { type: 'forward', targets: [{ host: 'localhost', port: 10025 }] }
|
|
```
|
|
|
|
**Files Fixed:**
|
|
- `ts/classes.dcrouter.ts` - `generateEmailRoutes()` method
|
|
- `test/test.dcrouter.email.ts` - Updated assertions and added `cacheConfig: { enabled: false }`
|
|
|
|
## Dependency Upgrade (2026-02-10)
|
|
|
|
### SmartProxy v23.1.0 Upgrade
|
|
- `@push.rocks/smartproxy`: 22.4.2 → 23.1.0
|
|
|
|
**Key Changes:**
|
|
- Rust-based proxy components for improved performance
|
|
- Rust binary runs as separate process via IPC
|
|
- `getStatistics()` now returns `Promise<any>` (was synchronous)
|
|
- nftables-proxy removed (not used by dcrouter)
|
|
|
|
**Code Changes Required:**
|
|
```typescript
|
|
// Old (synchronous)
|
|
const proxyStats = this.dcRouter.smartProxy.getStatistics();
|
|
|
|
// New (async)
|
|
const proxyStats = await this.dcRouter.smartProxy.getStatistics();
|
|
```
|
|
|
|
**Files Modified:**
|
|
- `ts/monitoring/classes.metricsmanager.ts` - Added `await` to `getStatistics()` call
|
|
|
|
## Dependency Upgrade (2026-02-01)
|
|
|
|
### Major Upgrades Completed
|
|
- `@api.global/typedserver`: 3.0.80 → 8.3.0
|
|
- `@api.global/typedsocket`: 3.1.1 → 4.1.0
|
|
- `@apiclient.xyz/cloudflare`: 6.4.3 → 7.1.0
|
|
- `@design.estate/dees-catalog`: 1.12.4 → 3.41.4
|
|
- `@push.rocks/smartpath`: 5.1.0 → 6.0.0
|
|
- `@push.rocks/smartproxy`: 19.6.17 → 22.4.2
|
|
- `@push.rocks/smartrequest`: 2.1.0 → 5.0.1
|
|
- `uuid`: 11.1.0 → 13.0.0
|
|
|
|
### Breaking Changes Fixed
|
|
|
|
1. **SmartProxy v22**: `target` → `targets` (array)
|
|
```typescript
|
|
// Old
|
|
action: { type: 'forward', target: { host: 'x', port: 25 } }
|
|
// New
|
|
action: { type: 'forward', targets: [{ host: 'x', port: 25 }] }
|
|
```
|
|
|
|
2. **SmartRequest v5**: `SmartRequestClient` → `SmartRequest`, `.body` → `.json()`
|
|
```typescript
|
|
// Old
|
|
const resp = await plugins.smartrequest.SmartRequestClient.create()...post();
|
|
const json = resp.body;
|
|
// New
|
|
const resp = await plugins.smartrequest.SmartRequest.create()...post();
|
|
const json = await resp.json();
|
|
```
|
|
|
|
3. **dees-catalog v3**: Icon naming changed to library-prefixed format
|
|
```typescript
|
|
// Old (deprecated but supported)
|
|
<dees-icon iconFA="check"></dees-icon>
|
|
// New
|
|
<dees-icon icon="fa:check"></dees-icon>
|
|
<dees-icon icon="lucide:menu"></dees-icon>
|
|
```
|
|
|
|
### TC39 Decorators
|
|
- ts_web components updated to use `accessor` keyword for `@state()` decorators
|
|
- Required for TC39 standard decorator support
|
|
|
|
### tswatch Configuration
|
|
The project now uses tswatch for development:
|
|
```bash
|
|
pnpm run watch
|
|
```
|
|
Configuration in `npmextra.json`:
|
|
```json
|
|
{
|
|
"@git.zone/tswatch": {
|
|
"watchers": [{
|
|
"name": "dcrouter-dev",
|
|
"watch": ["ts/**/*.ts", "ts_*/**/*.ts", "test_watch/devserver.ts"],
|
|
"command": "pnpm run build && tsrun test_watch/devserver.ts",
|
|
"restart": true,
|
|
"debounce": 500,
|
|
"runOnStart": true
|
|
}]
|
|
}
|
|
}
|
|
```
|
|
|
|
## RADIUS Server Integration (2026-02-01)
|
|
|
|
### Overview
|
|
DcRouter now supports RADIUS server functionality for network authentication via `@push.rocks/smartradius`.
|
|
|
|
### Key Features
|
|
- **MAC Authentication Bypass (MAB)** - Authenticate network devices based on MAC address
|
|
- **VLAN Assignment** - Assign VLANs based on MAC address or OUI patterns
|
|
- **RADIUS Accounting** - Track sessions, data usage, and billing
|
|
|
|
### Configuration Example
|
|
```typescript
|
|
const dcRouter = new DcRouter({
|
|
radiusConfig: {
|
|
authPort: 1812, // Authentication port (default)
|
|
acctPort: 1813, // Accounting port (default)
|
|
clients: [
|
|
{
|
|
name: 'switch-1',
|
|
ipRange: '192.168.1.0/24',
|
|
secret: 'shared-secret',
|
|
enabled: true
|
|
}
|
|
],
|
|
vlanAssignment: {
|
|
defaultVlan: 100, // VLAN for unknown MACs
|
|
allowUnknownMacs: true,
|
|
mappings: [
|
|
{ mac: '00:11:22:33:44:55', vlan: 10, enabled: true },
|
|
{ mac: '00:11:22', vlan: 20, enabled: true } // OUI pattern
|
|
]
|
|
},
|
|
accounting: {
|
|
enabled: true,
|
|
retentionDays: 30
|
|
}
|
|
}
|
|
});
|
|
```
|
|
|
|
### Components
|
|
- `RadiusServer` - Main server wrapping smartradius
|
|
- `VlanManager` - MAC-to-VLAN mapping with OUI pattern support
|
|
- `AccountingManager` - Session tracking and billing data
|
|
|
|
### OpsServer API Endpoints
|
|
- `getRadiusClients` / `setRadiusClient` / `removeRadiusClient` - Client management
|
|
- `getVlanMappings` / `setVlanMapping` / `removeVlanMapping` - VLAN mappings
|
|
- `testVlanAssignment` - Test what VLAN a MAC would get
|
|
- `getRadiusSessions` / `disconnectRadiusSession` - Session management
|
|
- `getRadiusStatistics` / `getRadiusAccountingSummary` - Statistics
|
|
|
|
### Files
|
|
- `ts/radius/` - RADIUS module
|
|
- `ts/opsserver/handlers/radius.handler.ts` - OpsServer handler
|
|
- `ts_interfaces/requests/radius.ts` - TypedRequest interfaces
|
|
|
|
## Test Fix: test.dcrouter.email.ts (2026-02-01)
|
|
|
|
### Issue
|
|
The test `DcRouter class - Custom email storage path` was failing with "domainConfigs is not iterable".
|
|
|
|
### Root Cause
|
|
The test was using outdated email config properties:
|
|
- Used `domainRules: []` (non-existent property)
|
|
- Used `defaultMode` (non-existent property)
|
|
- Missing required `domains: []` property
|
|
- Missing required `routes: []` property
|
|
- Referenced `router.unifiedEmailServer` instead of `router.emailServer`
|
|
|
|
### Fix
|
|
Updated the test to use the correct `IUnifiedEmailServerOptions` interface properties:
|
|
```typescript
|
|
const emailConfig: IEmailConfig = {
|
|
ports: [2525],
|
|
hostname: 'mail.example.com',
|
|
domains: [], // Required: domain configurations
|
|
routes: [] // Required: email routing rules
|
|
};
|
|
```
|
|
|
|
And fixed the property name:
|
|
```typescript
|
|
expect(router.emailServer).toBeTruthy(); // Not unifiedEmailServer
|
|
```
|
|
|
|
### Key Learning
|
|
When using `IUnifiedEmailServerOptions` (aliased as `IEmailConfig` in some tests):
|
|
- `domains: IEmailDomainConfig[]` is required (array of domain configs)
|
|
- `routes: IEmailRoute[]` is required (email routing rules)
|
|
- Access the email server via `dcRouter.emailServer` not `dcRouter.unifiedEmailServer`
|
|
|
|
## Network Metrics Implementation (2025-06-23)
|
|
|
|
### SmartProxy Metrics API Integration
|
|
- Updated to use new SmartProxy metrics API (v19.6.7)
|
|
- Use `getMetrics()` for detailed metrics with grouped methods:
|
|
```typescript
|
|
const metrics = smartProxy.getMetrics();
|
|
metrics.connections.active() // Current active connections
|
|
metrics.throughput.instant() // Real-time throughput {in, out}
|
|
metrics.connections.topIPs(10) // Top 10 IPs by connection count
|
|
```
|
|
- Use `getStatistics()` for basic stats
|
|
|
|
### Network Traffic Display
|
|
- All throughput values shown in bits per second (kbit/s, Mbit/s, Gbit/s)
|
|
- Conversion: `bytesPerSecond * 8 / 1000000` for Mbps
|
|
- Network graph shows separate lines for inbound (green) and outbound (purple)
|
|
- Throughput tiles and graph use same data source for consistency
|
|
|
|
### Requests/sec vs Connections
|
|
- Requests/sec shows HTTP request counts (derived from connections)
|
|
- Single connection can handle multiple requests
|
|
- Current implementation tracks connections, not individual requests
|
|
- Trend line shows historical request counts, not throughput
|
|
|
|
## DKIM Implementation Status (2025-05-30)
|
|
|
|
**Note:** DKIM is now handled by `@push.rocks/smartmta`. The `dkimCreator` is a public property on `UnifiedEmailServer`.
|
|
|
|
## SmartProxy Usage
|
|
|
|
### New Route-Based Architecture (v18+)
|
|
- SmartProxy now uses a route-based configuration system
|
|
- Routes define match criteria and actions instead of simple port-to-port forwarding
|
|
- All traffic types (HTTP, HTTPS, TCP, WebSocket) are configured through routes
|
|
|
|
```typescript
|
|
// NEW: Route-based SmartProxy configuration
|
|
const smartProxy = new plugins.smartproxy.SmartProxy({
|
|
routes: [
|
|
{
|
|
name: 'https-traffic',
|
|
match: {
|
|
ports: 443,
|
|
domains: ['example.com', '*.example.com']
|
|
},
|
|
action: {
|
|
type: 'forward',
|
|
target: {
|
|
host: 'backend.server.com',
|
|
port: 8080
|
|
}
|
|
},
|
|
tls: {
|
|
mode: 'terminate',
|
|
certificate: 'auto'
|
|
}
|
|
}
|
|
],
|
|
defaults: {
|
|
target: {
|
|
host: 'fallback.server.com',
|
|
port: 8080
|
|
}
|
|
},
|
|
acme: {
|
|
accountEmail: 'admin@example.com',
|
|
enabled: true,
|
|
useProduction: true
|
|
}
|
|
});
|
|
```
|
|
|
|
### Migration from Old to New
|
|
```typescript
|
|
// OLD configuration style (deprecated)
|
|
{
|
|
fromPort: 443,
|
|
toPort: 8080,
|
|
targetIP: 'backend.server.com',
|
|
domainConfigs: [...]
|
|
}
|
|
|
|
// NEW route-based style
|
|
{
|
|
routes: [{
|
|
name: 'main-route',
|
|
match: { ports: 443 },
|
|
action: {
|
|
type: 'forward',
|
|
target: { host: 'backend.server.com', port: 8080 }
|
|
}
|
|
}]
|
|
}
|
|
```
|
|
|
|
### Direct Component Usage
|
|
- Use SmartProxy components directly instead of creating your own wrappers
|
|
- SmartProxy already includes Port80Handler and NetworkProxy functionality
|
|
- When using SmartProxy, configure it directly rather than instantiating Port80Handler or NetworkProxy separately
|
|
|
|
### Certificate Management
|
|
- SmartProxy has built-in ACME certificate management
|
|
- Configure it in the `acme` property of SmartProxy options
|
|
- Use `accountEmail` (not `email`) for the ACME contact email
|
|
- SmartProxy handles both HTTP-01 challenges and certificate application automatically
|
|
|
|
## qenv Usage
|
|
|
|
### Direct Usage
|
|
- Use qenv directly instead of creating environment variable wrappers
|
|
- Instantiate qenv with appropriate basePath and nogitPath:
|
|
|
|
```typescript
|
|
const qenv = new plugins.qenv.Qenv('./', '.nogit/');
|
|
const value = await qenv.getEnvVarOnDemand('ENV_VAR_NAME');
|
|
```
|
|
|
|
## TypeScript Interfaces
|
|
|
|
### SmartProxy Interfaces
|
|
- Always check the interfaces from the node_modules to ensure correct property names
|
|
- Important interfaces for the new architecture:
|
|
- `ISmartProxyOptions`: Main configuration with `routes` array
|
|
- `IRouteConfig`: Individual route configuration
|
|
- `IRouteMatch`: Match criteria for routes
|
|
- `IRouteTarget`: Target configuration for forwarding
|
|
- `IAcmeOptions`: ACME certificate configuration
|
|
- `TTlsMode`: TLS handling modes ('passthrough' | 'terminate' | 'terminate-and-reencrypt')
|
|
|
|
### New Route Configuration
|
|
```typescript
|
|
interface IRouteConfig {
|
|
name: string;
|
|
match: {
|
|
ports: number | number[];
|
|
domains?: string | string[];
|
|
path?: string;
|
|
headers?: Record<string, string | RegExp>;
|
|
};
|
|
action: {
|
|
type: 'forward' | 'redirect' | 'block' | 'static';
|
|
target?: {
|
|
host: string | string[] | ((context) => string);
|
|
port: number | 'preserve' | ((context) => number);
|
|
};
|
|
};
|
|
tls?: {
|
|
mode: TTlsMode;
|
|
certificate?: 'auto' | { key: string; cert: string; };
|
|
};
|
|
security?: {
|
|
authentication?: IRouteAuthentication;
|
|
rateLimit?: IRouteRateLimit;
|
|
ipAllowList?: string[];
|
|
ipBlockList?: string[];
|
|
};
|
|
}
|
|
```
|
|
|
|
### Required Properties
|
|
- For `ISmartProxyOptions`, `routes` array is the main configuration
|
|
- For `IAcmeOptions`, use `accountEmail` for the contact email
|
|
- Routes must have `name`, `match`, and `action` properties
|
|
|
|
## Testing
|
|
|
|
### Test Structure
|
|
- Follow the project's test structure, using `@push.rocks/tapbundle`
|
|
- Use `expect(value).toEqual(expected)` for equality checks
|
|
- Use `expect(value).toBeTruthy()` for boolean assertions
|
|
|
|
```typescript
|
|
tap.test('test description', async () => {
|
|
const result = someFunction();
|
|
expect(result.property).toEqual('expected value');
|
|
expect(result.valid).toBeTruthy();
|
|
});
|
|
```
|
|
|
|
### Cleanup
|
|
- Include a cleanup test to ensure proper test resource handling
|
|
- Add a `stop` test to forcefully end the test when needed:
|
|
|
|
```typescript
|
|
tap.test('stop', async () => {
|
|
await tap.stopForcefully();
|
|
});
|
|
```
|
|
|
|
## Architecture Principles
|
|
|
|
### Simplicity
|
|
- Prefer direct usage of libraries instead of creating wrappers
|
|
- Don't reinvent functionality that already exists in dependencies
|
|
- Keep interfaces clean and focused, avoiding unnecessary abstraction layers
|
|
|
|
### Component Integration
|
|
- Leverage built-in integrations between components (like SmartProxy's ACME handling)
|
|
- Use parallel operations for performance (like in the `stop()` method)
|
|
- Separate concerns clearly (HTTP handling vs. SMTP handling)
|
|
|
|
## Email Integration with SmartProxy
|
|
|
|
### Architecture (Post-Migration)
|
|
- Email traffic is routed through SmartProxy using automatic route generation
|
|
- smartmta's UnifiedEmailServer runs on internal ports and receives forwarded traffic from SmartProxy
|
|
- SmartProxy handles external ports (25, 587, 465) and forwards to internal ports
|
|
- smartmta's Rust SMTP bridge handles SMTP protocol processing
|
|
|
|
### Port Mapping
|
|
- External port 25 → Internal port 10025 (SMTP)
|
|
- External port 587 → Internal port 10587 (Submission)
|
|
- External port 465 → Internal port 10465 (SMTPS)
|
|
|
|
### TLS Handling
|
|
- Ports 25 and 587: Use 'passthrough' mode (STARTTLS handled by smartmta)
|
|
- Port 465: Use 'terminate' mode (SmartProxy handles TLS termination)
|
|
|
|
## SmartMetrics Integration (2025-06-12) - COMPLETED
|
|
|
|
### Overview
|
|
Fixed the UI metrics display to show accurate CPU and memory data from SmartMetrics.
|
|
|
|
### Key Findings
|
|
1. **CPU Metrics:**
|
|
- SmartMetrics provides `cpuUsageText` as a string percentage
|
|
- MetricsManager parses it as `cpuUsage.user` (system is always 0)
|
|
- UI was incorrectly dividing by 2, showing half the actual CPU usage
|
|
|
|
2. **Memory Metrics:**
|
|
- SmartMetrics calculates `maxMemoryMB` as minimum of:
|
|
- V8 heap size limit
|
|
- System total memory
|
|
- Docker memory limit (if available)
|
|
- Provides `memoryUsageBytes` (total process memory including children)
|
|
- Provides `memoryPercentage` (pre-calculated percentage)
|
|
- UI was only showing heap usage, missing actual memory constraints
|
|
|
|
### Changes Made
|
|
1. **MetricsManager Enhanced:**
|
|
- Added `maxMemoryMB` from SmartMetrics instance
|
|
- Added `actualUsageBytes` from SmartMetrics data
|
|
- Added `actualUsagePercentage` from SmartMetrics data
|
|
- Kept existing memory fields for compatibility
|
|
|
|
2. **Interface Updated:**
|
|
- Added optional fields to `IServerStats.memoryUsage`
|
|
- Fields are optional to maintain backward compatibility
|
|
|
|
3. **UI Fixed:**
|
|
- Removed incorrect CPU division by 2
|
|
- Uses `actualUsagePercentage` when available (falls back to heap percentage)
|
|
- Shows actual memory usage vs max memory limit (not just heap)
|
|
|
|
### Result
|
|
- CPU now shows accurate usage percentage
|
|
- Memory shows percentage of actual constraints (Docker/system/V8 limits)
|
|
- Better monitoring for containerized environments
|
|
|
|
## Network UI Implementation (2025-06-20) - COMPLETED
|
|
|
|
### Overview
|
|
Revamped the Network UI to display real network data from SmartProxy instead of mock data.
|
|
|
|
### Architecture
|
|
1. **MetricsManager Integration:**
|
|
- Already integrates with SmartProxy via `dcRouter.smartProxy.getStats()`
|
|
- Extended with `getNetworkStats()` method to expose unused metrics:
|
|
- `getConnectionsByIP()` - Connection counts by IP address
|
|
- `getThroughputRate()` - Real-time bandwidth rates (bytes/second)
|
|
- `getTopIPs()` - Top connecting IPs sorted by connection count
|
|
- Note: SmartProxy base interface doesn't include all methods, manual implementation required
|
|
|
|
2. **Existing Infrastructure Leveraged:**
|
|
- `getActiveConnections` endpoint already exists in security.handler.ts
|
|
- Enhanced to include real SmartProxy data via MetricsManager
|
|
- IConnectionInfo interface already supports network data structures
|
|
|
|
3. **State Management:**
|
|
- Added `INetworkState` interface following existing patterns
|
|
- Created `networkStatePart` with connections, throughput, and IP data
|
|
- Integrated with existing auto-refresh mechanism
|
|
|
|
4. **UI Changes (Minimal):**
|
|
- Removed `generateMockData()` method and all mock generation
|
|
- Connected to real `networkStatePart` state
|
|
- Added `renderTopIPs()` section to display top connected IPs
|
|
- Updated traffic chart to show real request data
|
|
- Kept all existing UI components (DeesTable, DeesChartArea)
|
|
|
|
### Implementation Details
|
|
1. **Data Transformation:**
|
|
- Converts IConnectionInfo[] to INetworkRequest[] for table display
|
|
- Calculates traffic buckets based on selected time range
|
|
- Maps connection data to chart-compatible format
|
|
|
|
2. **Real Metrics Displayed:**
|
|
- Active connections count (from server stats)
|
|
- Requests per second (calculated from recent connections)
|
|
- Throughput rates (currently showing 0 until SmartProxy exposes rates)
|
|
- Top IPs with connection counts and percentages
|
|
|
|
3. **TypeScript Fixes:**
|
|
- SmartProxy methods like `getThroughputRate()` not in base interface
|
|
- Implemented manual fallbacks for missing methods
|
|
- Fixed `publicIpv4` → `publicIp` property name
|
|
|
|
### Result
|
|
- Network view now shows real connection activity
|
|
- Auto-refreshes with other stats every second
|
|
- Displays actual IPs and connection counts
|
|
- No more mock/demo data
|
|
- Minimal code changes (streamlined approach)
|
|
|
|
### Throughput Data Fix (2025-06-20)
|
|
The throughput was showing 0 because:
|
|
1. MetricsManager was hardcoding throughputRate to 0, assuming the method didn't exist
|
|
2. SmartProxy's `getStats()` returns `IProxyStats` interface, but the actual object (`MetricsCollector`) implements `IProxyStatsExtended`
|
|
3. `getThroughputRate()` only exists in the extended interface
|
|
|
|
**Solution implemented:**
|
|
1. Updated MetricsManager to check if methods exist at runtime and call them
|
|
2. Added property name mapping (`bytesInPerSec` → `bytesInPerSecond`)
|
|
3. Created new `getNetworkStats` endpoint in security.handler.ts
|
|
4. Updated frontend to call the new endpoint for complete network metrics
|
|
|
|
The throughput data now flows correctly from SmartProxy → MetricsManager → API → UI.
|
|
|
|
## Email Operations Dashboard (2026-02-01)
|
|
|
|
### Overview
|
|
Replaced mock data in the email UI with real backend data from the delivery queue and security logger.
|
|
|
|
### New Files Created
|
|
- `ts_interfaces/requests/email-ops.ts` - TypedRequest interfaces for email operations
|
|
- `ts/opsserver/handlers/email-ops.handler.ts` - Backend handler for email operations
|
|
|
|
### Key Interfaces
|
|
- `IReq_GetQueuedEmails` - Fetch emails from delivery queue by status
|
|
- `IReq_GetSentEmails` - Fetch delivered emails
|
|
- `IReq_GetFailedEmails` - Fetch failed emails
|
|
- `IReq_ResendEmail` - Re-queue a failed email for retry
|
|
- `IReq_GetSecurityIncidents` - Fetch security events from SecurityLogger
|
|
- `IReq_GetBounceRecords` - Fetch bounce records and suppression list
|
|
- `IReq_RemoveFromSuppressionList` - Remove email from suppression list
|
|
|
|
### UI Changes (ops-view-emails.ts)
|
|
- Replaced mock folders (inbox/sent/draft/trash) with operations views:
|
|
- **Queued**: Emails pending delivery
|
|
- **Sent**: Successfully delivered emails
|
|
- **Failed**: Failed emails with resend capability
|
|
- **Security**: Security incidents from SecurityLogger
|
|
- Removed `generateMockEmails()` method
|
|
- Added state management via `emailOpsStatePart` in appstate.ts
|
|
- Added resend button for failed emails
|
|
- Added security incident detail view
|
|
|
|
### Data Flow
|
|
```
|
|
UnifiedDeliveryQueue → EmailOpsHandler → TypedRequest → Frontend State → UI
|
|
SecurityLogger → EmailOpsHandler → TypedRequest → Frontend State → UI
|
|
BounceManager → EmailOpsHandler → TypedRequest → Frontend State → UI
|
|
```
|
|
|
|
### Backend Data Access
|
|
The handler accesses data from:
|
|
- `dcRouter.emailServer.deliveryQueue` - Email queue items (IQueueItem)
|
|
- `SecurityLogger.getInstance()` - Security events (ISecurityEvent)
|
|
- `emailServer.bounceManager` - Bounce records and suppression list
|
|
|
|
## OpsServer UI Fixes (2026-02-02)
|
|
|
|
### Configuration Page Fix
|
|
The configuration page had field name mismatches between frontend and backend:
|
|
- Frontend expected `server` and `storage` sections
|
|
- Backend returns `proxy` section (not `server`)
|
|
- Backend has no `storage` section
|
|
|
|
**Fix**: Updated `ops-view-config.ts` to use correct section names:
|
|
- `proxy` instead of `server`
|
|
- Removed non-existent `storage` section
|
|
- Added optional chaining (`?.`) for safety
|
|
|
|
### Auth Persistence Fix
|
|
Login state was using `'soft'` mode in Smartstate which is memory-only:
|
|
- User login was lost on page refresh
|
|
- State reset to logged out after browser restart
|
|
|
|
**Changes**:
|
|
1. `ts_web/appstate.ts`: Changed loginStatePart from `'soft'` to `'persistent'`
|
|
- Now uses IndexedDB to persist across browser sessions
|
|
2. `ts/opsserver/handlers/admin.handler.ts`: JWT expiry changed from 7 days to 24 hours
|
|
3. `ts_web/elements/ops-dashboard.ts`: Added JWT expiry check on session restore
|
|
- Validates stored JWT hasn't expired before auto-logging in
|
|
- Clears expired sessions and shows login form
|
|
|
|
## Config UI Read-Only Conversion (2026-02-03)
|
|
|
|
### Overview
|
|
The configuration UI has been converted from an editable interface to a read-only display. DcRouter is configured through code or remotely, not through the UI.
|
|
|
|
### Changes Made
|
|
|
|
1. **Backend (`ts/opsserver/handlers/config.handler.ts`)**:
|
|
- Removed `updateConfiguration` handler
|
|
- Removed `updateConfiguration()` private method
|
|
- Kept `getConfiguration` handler (read-only)
|
|
|
|
2. **Interfaces (`ts_interfaces/requests/config.ts`)**:
|
|
- Removed `IReq_UpdateConfiguration` interface
|
|
- Kept `IReq_GetConfiguration` interface
|
|
|
|
3. **Frontend (`ts_web/elements/ops-view-config.ts`)**:
|
|
- Removed `editingSection` and `editedConfig` state properties
|
|
- Removed `startEdit()`, `cancelEdit()`, `saveConfig()` methods
|
|
- Removed Edit/Save/Cancel buttons
|
|
- Removed warning banner about immediate changes
|
|
- Enhanced read-only display with:
|
|
- Status badges for boolean values (enabled/disabled)
|
|
- Array display as pills/tags with counts
|
|
- Section icons (mail, globe, network, shield)
|
|
- Better formatting for numbers and byte sizes
|
|
- Empty state handling ("Not configured", "None configured")
|
|
- Info note explaining configuration is read-only
|
|
|
|
4. **State Management (`ts_web/appstate.ts`)**:
|
|
- Removed `updateConfigurationAction`
|
|
- Kept `fetchConfigurationAction` (read-only)
|
|
|
|
5. **Tests (`test/test.protected-endpoint.ts`)**:
|
|
- Replaced `updateConfiguration` tests with `verifyIdentity` tests
|
|
- Added test for read-only config access
|
|
- Kept auth flow testing with different protected endpoint
|
|
|
|
6. **Documentation**:
|
|
- `readme.md`: Updated API endpoints to show config as read-only
|
|
- `ts_web/readme.md`: Removed `updateConfigurationAction` from actions list
|
|
- `ts_interfaces/readme.md`: Removed `IReq_UpdateConfiguration` from table
|
|
|
|
### Visual Display Features
|
|
- Boolean values shown as colored badges (green=enabled, red=disabled)
|
|
- Arrays displayed as pills with count summaries
|
|
- Section headers with relevant Lucide icons
|
|
- Numbers formatted with locale separators
|
|
- Byte sizes auto-formatted (B, KB, MB, GB)
|
|
- Time values shown with "seconds" suffix
|
|
- Nested objects with visual indentation
|
|
|
|
## Smartdata Cache System (2026-02-03)
|
|
|
|
### Overview
|
|
DcRouter now uses smartdata + LocalTsmDb for persistent caching. Data is stored at `~/.serve.zone/dcrouter/tsmdb`.
|
|
|
|
### Technology Stack
|
|
| Layer | Package | Purpose |
|
|
|-------|---------|---------|
|
|
| ORM | `@push.rocks/smartdata` | Document classes, decorators, queries |
|
|
| Database | `@push.rocks/smartmongo` (LocalTsmDb) | Embedded TsmDB via Unix socket |
|
|
|
|
### TC39 Decorators
|
|
The project uses TC39 Stage 3 decorators (not experimental decorators). The tsconfig was updated:
|
|
- Removed `experimentalDecorators: true`
|
|
- Removed `emitDecoratorMetadata: true`
|
|
|
|
This is required for smartdata v7+ compatibility.
|
|
|
|
### Cache Document Classes
|
|
Located in `ts/cache/documents/`:
|
|
|
|
| Class | Purpose | Default TTL |
|
|
|-------|---------|-------------|
|
|
| `CachedEmail` | Email queue items | 30 days |
|
|
| `CachedIPReputation` | IP reputation lookups | 24 hours |
|
|
|
|
Note: CachedBounce, CachedSuppression, and CachedDKIMKey were removed in the smartmta migration (smartmta handles its own persistence for those).
|
|
|
|
### Usage Pattern
|
|
```typescript
|
|
// Document classes use smartdata decorators
|
|
@plugins.smartdata.Collection(() => getDb())
|
|
export class CachedEmail extends CachedDocument<CachedEmail> {
|
|
@plugins.smartdata.svDb()
|
|
public createdAt: Date = new Date();
|
|
|
|
@plugins.smartdata.svDb()
|
|
public expiresAt: Date = new Date(Date.now() + TTL.DAYS_30);
|
|
|
|
@plugins.smartdata.unI()
|
|
@plugins.smartdata.svDb()
|
|
public id: string;
|
|
// ...
|
|
}
|
|
|
|
// Query examples
|
|
const email = await CachedEmail.getInstance({ id: 'abc123' });
|
|
const pending = await CachedEmail.getInstances({ status: 'pending' });
|
|
await email.save();
|
|
await email.delete();
|
|
```
|
|
|
|
### Configuration
|
|
```typescript
|
|
const dcRouter = new DcRouter({
|
|
cacheConfig: {
|
|
enabled: true,
|
|
storagePath: '~/.serve.zone/dcrouter/tsmdb',
|
|
dbName: 'dcrouter',
|
|
cleanupIntervalHours: 1,
|
|
ttlConfig: {
|
|
emails: 30, // days
|
|
ipReputation: 1, // days
|
|
bounces: 30, // days
|
|
dkimKeys: 90, // days
|
|
suppression: 30 // days
|
|
}
|
|
}
|
|
});
|
|
```
|
|
|
|
### Cache Cleaner
|
|
- Runs hourly by default (configurable via `cleanupIntervalHours`)
|
|
- Finds and deletes documents where `expiresAt < now()`
|
|
- Uses smartdata's `getInstances()` + `delete()` pattern
|
|
|
|
### Key Files
|
|
- `ts/cache/classes.cachedb.ts` - CacheDb singleton wrapper
|
|
- `ts/cache/classes.cached.document.ts` - Base class with TTL support
|
|
- `ts/cache/classes.cache.cleaner.ts` - Periodic cleanup service
|
|
- `ts/cache/documents/*.ts` - Document class definitions |