feat(docs,ops-dashboard): document unified database and reusable security profile and network target management

2026-04-02 20:31:08 +00:00
parent 947637eed7
commit 9d8354e58f
9 changed files with 164 additions and 83 deletions
--- a/readme.md
+++ b/readme.md
@@ -93,10 +93,11 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
 - **Socket-handler mode** — direct socket passing eliminates internal port hops
 - **Real-time metrics** via SmartMetrics (CPU, memory, connections, throughput)

-### 💾 Persistent Storage & Caching
- **Multiple storage backends**: filesystem, custom functions, or in-memory
- **Embedded cache database** via smartdata + smartdb (MongoDB-compatible)
+### 💾 Unified Database
+- **Two deployment modes**: embedded LocalSmartDb (zero-config) or external MongoDB
+- **15 document classes** covering routes, certs, VPN, RADIUS, security profiles, network targets, and caches
 - **Automatic TTL-based cleanup** for cached emails and IP reputation data
+- **Reusable references** — security profiles and network targets that propagate changes to all referencing routes

 ### 🖥️ OpsServer Dashboard
 - **Web-based management interface** with real-time monitoring
@@ -104,7 +105,9 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
 - **Live views** for connections, email queues, DNS queries, RADIUS sessions, certificates, remote ingress edges, VPN clients, and security events
 - **Domain-centric certificate overview** with backoff status and one-click reprovisioning
 - **Remote ingress management** with connection token generation and one-click copy
- **Read-only configuration display** — DcRouter is configured through code
+- **Security profiles & network targets** — reusable security configurations and host:port targets with propagation to referencing routes
+- **Global warning banners** when database is disabled (management features unavailable)
+- **Read-only configuration display** for system overview
 - **Smart tab visibility handling** — auto-pauses all polling, WebSocket connections, and chart updates when the browser tab is hidden, preventing resource waste and tab freezing

 ### 🔧 Programmatic API Client
@@ -269,11 +272,8 @@ const router = new DcRouter({
    ],
  },

-  // Persistent storage
-  storage: { fsPath: '/var/lib/dcrouter/data' },
-
-  // Cache database
-  cacheConfig: { enabled: true, storagePath: '~/.serve.zone/dcrouter/tsmdb' },
+  // Unified database (embedded LocalSmartDb or external MongoDB)
+  dbConfig: { enabled: true },

  // TLS & ACME
  tls: { contactEmail: 'admin@example.com' },
@@ -311,8 +311,7 @@ graph TB
        CM[Certificate Manager<br/><i>smartacme v9</i>]
        OS[OpsServer Dashboard]
        MM[Metrics Manager]
-        SM[Storage Manager]
-        CD[Cache Database]
+        DB2[DcRouterDb<br/><i>smartdata + smartdb</i>]
    end

    subgraph "Backend Services"
@@ -339,8 +338,7 @@ graph TB
    DC --> CM
    DC --> OS
    DC --> MM
-    DC --> SM
-    DC --> CD
+    DC --> DB2

    SP --> WEB
    SP --> API
@@ -365,8 +363,7 @@ graph TB
 | **RemoteIngress** | `@serve.zone/remoteingress` | Distributed edge tunneling with Rust data plane and TS management |
 | **OpsServer** | `@api.global/typedserver` | Web dashboard + TypedRequest API for monitoring and management |
 | **MetricsManager** | `@push.rocks/smartmetrics` | Real-time metrics collection (CPU, memory, email, DNS, security) |
-| **StorageManager** | built-in | Pluggable key-value storage (filesystem, custom, or in-memory) |
-| **CacheDb** | `@push.rocks/smartdb` | Embedded MongoDB-compatible database (LocalSmartDb) for persistent caching |
+| **DcRouterDb** | `@push.rocks/smartdata` + `@push.rocks/smartdb` | Unified database — embedded LocalSmartDb or external MongoDB for all persistence |

 ### How It Works

@@ -509,24 +506,16 @@ interface IDcRouterOptions {
  };
  dnsChallenge?: { cloudflareApiKey?: string };

-  // ── Storage & Caching ─────────────────────────────────────────
-  storage?: {
-    fsPath?: string;
-    readFunction?: (key: string) => Promise<string>;
-    writeFunction?: (key: string, value: string) => Promise<void>;
-  };
-  cacheConfig?: {
+  // ── Database ────────────────────────────────────────────────────
+  /** Unified database for all persistence (routes, certs, VPN, RADIUS, etc.) */
+  dbConfig?: {
    enabled?: boolean;                   // default: true
+    mongoDbUrl?: string;                 // External MongoDB URL (omit for embedded LocalSmartDb)
    storagePath?: string;                // default: '~/.serve.zone/dcrouter/tsmdb'
    dbName?: string;                     // default: 'dcrouter'
    cleanupIntervalHours?: number;       // default: 1
-    ttlConfig?: {
-      emails?: number;                   // default: 30 days
-      ipReputation?: number;             // default: 1 day
-      bounces?: number;                  // default: 30 days
-      dkimKeys?: number;                 // default: 90 days
-      suppression?: number;              // default: 30 days
-    };
+    seedOnEmpty?: boolean;               // Seed default profiles/targets if DB is empty
+    seedData?: object;                   // Custom seed data
  };
 }
 ```
@@ -1213,49 +1202,55 @@ The OpsServer includes a **Certificates** view showing:
 - One-click reprovisioning per domain
 - Certificate import and export

-## Storage & Caching
+## Storage & Database

-### StorageManager
+DcRouter uses a **unified database** (`DcRouterDb`) powered by [`@push.rocks/smartdata`](https://code.foss.global/push.rocks/smartdata) + [`@push.rocks/smartdb`](https://code.foss.global/push.rocks/smartdb) for all persistence. It supports two modes:

-Provides a unified key-value interface with three backends:
+### Embedded LocalSmartDb (Default)
+
+Zero-config, file-based MongoDB-compatible database — no external services needed:

 ```typescript
-// Filesystem backend
-storage: { fsPath: '/var/lib/dcrouter/data' }
-
-// Custom backend (Redis, S3, etc.)
-storage: {
-  readFunction: async (key) => await redis.get(key),
-  writeFunction: async (key, value) => await redis.set(key, value)
-}
-
-// In-memory (development only — data lost on restart)
-// Simply omit the storage config
+dbConfig: { enabled: true }
+// Data stored at ~/.serve.zone/dcrouter/tsmdb by default
 ```

-Used for: TLS certificates, DKIM keys, email routes, bounce/suppression lists, IP reputation data, domain configs, cert backoff state, remote ingress edge registrations.
+### External MongoDB

-### Cache Database
-
-An embedded MongoDB-compatible database (via smartdata + smartdb) for persistent caching with automatic TTL cleanup:
+Connect to an existing MongoDB instance:

 ```typescript
-cacheConfig: {
+dbConfig: {
  enabled: true,
-  storagePath: '~/.serve.zone/dcrouter/tsmdb',
+  mongoDbUrl: 'mongodb://localhost:27017',
  dbName: 'dcrouter',
-  cleanupIntervalHours: 1,
-  ttlConfig: {
-    emails: 30,         // days
-    ipReputation: 1,    // days
-    bounces: 30,        // days
-    dkimKeys: 90,       // days
-    suppression: 30     // days
-  }
 }
 ```

-Cached document types: `CachedEmail`, `CachedIPReputation`.
+### Disabling the Database
+
+For static, constructor-only deployments where no runtime management is needed:
+
+```typescript
+dbConfig: { enabled: false }
+// Routes come exclusively from constructor config — no CRUD, no persistence
+// OpsServer still runs but management features are disabled
+```
+
+### What's Stored
+
+DcRouterDb persists all runtime state across 15 document classes:
+
+| Category | Documents | Purpose |
+|----------|-----------|---------|
+| **Routes** | `StoredRouteDoc`, `RouteOverrideDoc` | Programmatic routes and hardcoded route overrides |
+| **Certificates** | `ProxyCertDoc`, `AcmeCertDoc`, `CertBackoffDoc` | TLS certs, ACME state, per-domain backoff |
+| **Auth** | `ApiTokenDoc` | API token storage |
+| **Remote Ingress** | `RemoteIngressEdgeDoc` | Edge node registrations |
+| **VPN** | `VpnServerKeysDoc`, `VpnClientDoc` | Server keys and client registrations |
+| **RADIUS** | `VlanMappingsDoc`, `AccountingSessionDoc` | VLAN mappings and accounting sessions |
+| **References** | `SecurityProfileDoc`, `NetworkTargetDoc` | Reusable security profiles and network targets |
+| **Cache** | `CachedEmailDoc`, `CachedIpReputationDoc` | TTL-based caches with automatic cleanup |

 ## Security Features

@@ -1324,6 +1319,8 @@ The OpsServer provides a web-based management interface served on port 3000 by d
 | 🔐 **Certificates** | Domain-centric certificate overview, status, backoff info, reprovisioning, import/export |
 | 🌍 **RemoteIngress** | Edge node management, connection status, token generation, enable/disable |
 | 🔐 **VPN** | VPN client management, server status, create/toggle/export/rotate/delete clients |
+| 🛡️ **Security Profiles** | Reusable security configurations (IP allow/block lists, rate limits) |
+| 🎯 **Network Targets** | Reusable host:port destinations for route references |
 | 📡 **RADIUS** | NAS client management, VLAN mappings, session monitoring, accounting |
 | 📜 **Logs** | Real-time log viewer with level filtering and search |
 | ⚙️ **Configuration** | Read-only view of current system configuration |
@@ -1410,6 +1407,22 @@ All management is done via TypedRequest over HTTP POST to `/typedrequest`:
 'setVlanMapping'                       // Add/update VLAN mapping
 'removeVlanMapping'                    // Remove VLAN mapping
 'testVlanAssignment'                   // Test what VLAN a MAC gets
+
+// Security Profiles
+'getSecurityProfiles'                  // List all security profiles
+'getSecurityProfile'                   // Get a single profile by ID
+'createSecurityProfile'                // Create a reusable security profile
+'updateSecurityProfile'                // Update a profile (propagates to referencing routes)
+'deleteSecurityProfile'                // Delete a profile (with optional force)
+'getSecurityProfileUsage'              // Get routes referencing a profile
+
+// Network Targets
+'getNetworkTargets'                    // List all network targets
+'getNetworkTarget'                     // Get a single target by ID
+'createNetworkTarget'                  // Create a reusable host:port target
+'updateNetworkTarget'                  // Update a target (propagates to referencing routes)
+'deleteNetworkTarget'                  // Delete a target (with optional force)
+'getNetworkTargetUsage'                // Get routes referencing a target
 ```

 ## API Client
@@ -1518,12 +1531,12 @@ const router = new DcRouter(options: IDcRouterOptions);
 | `remoteIngressManager` | `RemoteIngressManager` | Edge registration CRUD manager |
 | `tunnelManager` | `TunnelManager` | Tunnel lifecycle and status manager |
 | `vpnManager` | `VpnManager` | VPN server lifecycle and client CRUD manager |
-| `storageManager` | `StorageManager` | Storage backend |
 | `opsServer` | `OpsServer` | OpsServer/dashboard instance |
 | `metricsManager` | `MetricsManager` | Metrics collector |
-| `cacheDb` | `CacheDb` | Cache database instance |
-| `certProvisionScheduler` | `CertProvisionScheduler` | Per-domain backoff scheduler for cert provisioning |
-| `certificateStatusMap` | `Map<string, ...>` | Domain-keyed certificate status from SmartProxy events |
+| `dcRouterDb` | `DcRouterDb` | Unified database instance (smartdata + smartdb) |
+| `routeConfigManager` | `RouteConfigManager` | Programmatic route CRUD manager |
+| `apiTokenManager` | `ApiTokenManager` | API token management |
+| `referenceResolver` | `ReferenceResolver` | Security profile and network target resolver |

 ### Re-exported Types

@@ -1589,7 +1602,8 @@ tstest test/test.opsserver-api.ts --verbose --timeout 60
 | `test.jwt-auth.ts` | JWT login, verification, logout, invalid credentials | 8 |
 | `test.opsserver-api.ts` | Health, statistics, configuration, log APIs | 8 |
 | `test.protected-endpoint.ts` | Admin auth, identity verification, public endpoints | 8 |
-| `test.storagemanager.ts` | Memory, filesystem, custom backends, concurrency | 8 |
+| `test.reference-resolver.ts` | Security profiles, network targets, route resolution | 20 |
+| `test.security-profiles-api.ts` | Profile/target API endpoints, auth enforcement | 13 |

 ## Docker / OCI Container Deployment