BREAKING CHANGE(db): replace StorageManager and CacheDb with a unified smartdata-backed database layer
This commit is contained in:
@@ -1,120 +1,84 @@
|
||||
# DCRouter Storage Overview
|
||||
|
||||
DCRouter uses two complementary storage systems: **StorageManager** for configuration and state, and **CacheDb** for time-limited cached data.
|
||||
DCRouter uses a **unified database layer** backed by `@push.rocks/smartdata` for all persistent data. All data is stored as typed document classes in a single database.
|
||||
|
||||
## StorageManager (Key-Value Store)
|
||||
## Database Modes
|
||||
|
||||
A lightweight, pluggable key-value store for configuration, credentials, and runtime state. Data is persisted as flat JSON files on disk by default.
|
||||
|
||||
### Default Path
|
||||
|
||||
```
|
||||
~/.serve.zone/dcrouter/storage/
|
||||
```
|
||||
|
||||
Configurable via `options.storage.fsPath` or `options.baseDir`.
|
||||
|
||||
### Backends
|
||||
|
||||
```typescript
|
||||
// Filesystem (default)
|
||||
storage: { fsPath: '/var/lib/dcrouter/data' }
|
||||
|
||||
// Custom (Redis, S3, etc.)
|
||||
storage: {
|
||||
readFunction: async (key) => await redis.get(key),
|
||||
writeFunction: async (key, value) => await redis.set(key, value),
|
||||
}
|
||||
|
||||
// In-memory (omit storage config — data lost on restart)
|
||||
```
|
||||
|
||||
### What's Stored
|
||||
|
||||
| Prefix | Contents | Managed By |
|
||||
|--------|----------|------------|
|
||||
| `/vpn/server-keys` | VPN server Noise + WireGuard keypairs | `VpnManager` |
|
||||
| `/vpn/clients/{clientId}` | VPN client registrations (keys, tags, description, assigned IP) | `VpnManager` |
|
||||
| `/config-api/routes/{uuid}.json` | Programmatic routes (created via OpsServer API) | `RouteConfigManager` |
|
||||
| `/config-api/tokens/{uuid}.json` | API tokens (hashed secrets, scopes, expiry) | `ApiTokenManager` |
|
||||
| `/config-api/overrides/{routeName}.json` | Hardcoded route overrides (enable/disable) | `RouteConfigManager` |
|
||||
| `/email/bounces/suppression-list.json` | Email bounce suppression list | `smartmta` |
|
||||
| `/certs/*` | TLS certificates and ACME state | `SmartAcme` (via `StorageBackedCertManager`) |
|
||||
|
||||
### API
|
||||
|
||||
```typescript
|
||||
// Read/write JSON
|
||||
await storageManager.getJSON<T>(key);
|
||||
await storageManager.setJSON(key, value);
|
||||
|
||||
// Raw string read/write
|
||||
await storageManager.get(key);
|
||||
await storageManager.set(key, value);
|
||||
|
||||
// List keys by prefix
|
||||
await storageManager.list('/vpn/clients/');
|
||||
|
||||
// Delete
|
||||
await storageManager.delete(key);
|
||||
```
|
||||
|
||||
## CacheDb (Embedded MongoDB)
|
||||
|
||||
An embedded MongoDB-compatible database (via `@push.rocks/smartdb` + `@push.rocks/smartdata`) for cached data with automatic TTL-based cleanup.
|
||||
|
||||
### Default Path
|
||||
### Embedded Mode (default)
|
||||
When no external MongoDB URL is provided, DCRouter starts an embedded `LocalSmartDb` (Rust-based MongoDB-compatible engine) via `@push.rocks/smartdb`.
|
||||
|
||||
```
|
||||
~/.serve.zone/dcrouter/tsmdb/
|
||||
```
|
||||
|
||||
Configurable via `options.cacheConfig.storagePath`.
|
||||
|
||||
### What's Cached
|
||||
|
||||
| Document Type | Default TTL | Purpose |
|
||||
|--------------|-------------|---------|
|
||||
| `CachedEmail` | 30 days | Email metadata cache for dashboard display |
|
||||
| `CachedIPReputation` | 1 day | IP reputation lookup results (DNSBL checks) |
|
||||
|
||||
### Configuration
|
||||
### External Mode
|
||||
Connect to any MongoDB-compatible database by providing a connection URL.
|
||||
|
||||
```typescript
|
||||
cacheConfig: {
|
||||
enabled: true, // default: true
|
||||
storagePath: '~/.serve.zone/dcrouter/tsmdb', // default
|
||||
dbName: 'dcrouter', // default
|
||||
cleanupIntervalHours: 1, // how often to purge expired records
|
||||
ttlConfig: {
|
||||
emails: 30, // days
|
||||
ipReputation: 1, // days
|
||||
bounces: 30, // days (reserved)
|
||||
dkimKeys: 90, // days (reserved)
|
||||
suppression: 30, // days (reserved)
|
||||
},
|
||||
dbConfig: {
|
||||
mongoDbUrl: 'mongodb://host:27017',
|
||||
dbName: 'dcrouter',
|
||||
}
|
||||
```
|
||||
|
||||
### How It Works
|
||||
|
||||
1. `CacheDb` starts a `LocalSmartDb` instance (embedded MongoDB process)
|
||||
2. `smartdata` connects to it via Unix socket
|
||||
3. Document classes (`CachedEmail`, `CachedIPReputation`) are decorated with `@Collection` and use `smartdata` ORM
|
||||
4. `CacheCleaner` runs on a timer, purging records older than their configured TTL
|
||||
|
||||
### Disabling
|
||||
|
||||
For development or lightweight deployments, disable the cache to avoid starting a MongoDB process:
|
||||
## Configuration
|
||||
|
||||
```typescript
|
||||
cacheConfig: { enabled: false }
|
||||
dbConfig: {
|
||||
enabled: true, // default: true
|
||||
mongoDbUrl: undefined, // default: embedded LocalSmartDb
|
||||
storagePath: '~/.serve.zone/dcrouter/tsmdb', // default (embedded mode only)
|
||||
dbName: 'dcrouter', // default
|
||||
cleanupIntervalHours: 1, // TTL cleanup interval
|
||||
}
|
||||
```
|
||||
|
||||
## When to Use Which
|
||||
## Document Classes
|
||||
|
||||
| Use Case | System | Why |
|
||||
|----------|--------|-----|
|
||||
| VPN keys, API tokens, routes, certs | **StorageManager** | Small JSON blobs, key-value access, no queries needed |
|
||||
| Email metadata, IP reputation | **CacheDb** | Time-series data, TTL expiry, potential for queries/aggregation |
|
||||
| Runtime state (connected clients, metrics) | **Neither** | In-memory only, rebuilt on startup |
|
||||
All data is stored as smartdata document classes in `ts/db/documents/`.
|
||||
|
||||
| Document Class | Collection | Unique Key | Purpose |
|
||||
|---|---|---|---|
|
||||
| `StoredRouteDoc` | storedRoutes | `id` | Programmatic routes (created via API) |
|
||||
| `RouteOverrideDoc` | routeOverrides | `routeName` | Hardcoded route enable/disable overrides |
|
||||
| `ApiTokenDoc` | apiTokens | `id` | API tokens (hashed secrets, scopes, expiry) |
|
||||
| `VpnServerKeysDoc` | vpnServerKeys | `configId` (singleton) | VPN server Noise + WireGuard keypairs |
|
||||
| `VpnClientDoc` | vpnClients | `clientId` | VPN client registrations |
|
||||
| `AcmeCertDoc` | acmeCerts | `domainName` | ACME certificates and keys |
|
||||
| `ProxyCertDoc` | proxyCerts | `domain` | SmartProxy TLS certificates |
|
||||
| `CertBackoffDoc` | certBackoff | `domain` | Per-domain cert provision backoff state |
|
||||
| `RemoteIngressEdgeDoc` | remoteIngressEdges | `id` | Edge node registrations |
|
||||
| `VlanMappingsDoc` | vlanMappings | `configId` (singleton) | MAC-to-VLAN mapping table |
|
||||
| `AccountingSessionDoc` | accountingSessions | `sessionId` | RADIUS accounting sessions |
|
||||
| `CachedEmail` | cachedEmails | `id` | Email metadata (TTL: 30 days) |
|
||||
| `CachedIPReputation` | cachedIPReputation | `ipAddress` | IP reputation results (TTL: 24 hours) |
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
DcRouterDb (singleton)
|
||||
├── LocalSmartDb (embedded, Rust) ─── or ─── External MongoDB
|
||||
└── SmartdataDb (ORM)
|
||||
└── @Collection(() => getDb())
|
||||
├── StoredRouteDoc
|
||||
├── RouteOverrideDoc
|
||||
├── ApiTokenDoc
|
||||
├── VpnServerKeysDoc / VpnClientDoc
|
||||
├── AcmeCertDoc / ProxyCertDoc / CertBackoffDoc
|
||||
├── RemoteIngressEdgeDoc
|
||||
├── VlanMappingsDoc / AccountingSessionDoc
|
||||
├── CachedEmail (TTL)
|
||||
└── CachedIPReputation (TTL)
|
||||
```
|
||||
|
||||
### TTL Cleanup
|
||||
|
||||
`CacheCleaner` runs on a configurable interval (default: 1 hour) and removes expired documents where `expiresAt < now()`.
|
||||
|
||||
## Disabling
|
||||
|
||||
For tests or lightweight deployments without persistence:
|
||||
|
||||
```typescript
|
||||
dbConfig: { enabled: false }
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user