serve.zone/dcrouter
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(integration): components now play nicer with each other

Browse Source
This commit is contained in:
Philipp Kunz
2025-05-30 05:30:06 +00:00
parent 2c244c4a9a
commit 40db395591
19 changed files with 2849 additions and 264 deletions
Download Patch File Download Diff File Expand all files Collapse all files

96
readme.hints.md
View File

@ -741,4 +741,98 @@ The `ts/config/` directory cleanup has been completed. Removed ~500+ lines of un
- Config directory now contains only 2 files (validator.ts, index.ts)
- SMS configuration is self-contained in SMS module
- All deprecated email configuration removed
- Build passes successfully
- Build passes successfully
## Per-Domain Rate Limiting (2025-05-29) - COMPLETED
### Overview
Per-domain rate limiting has been implemented in the UnifiedRateLimiter. Each email domain can have its own rate limits that override global limits.
### Implementation Details
1. **UnifiedRateLimiter Enhanced:**
- Added `domains` property to IHierarchicalRateLimits
- Added `domainCounters` Map for tracking domain-specific counters
- Added `checkDomainMessageLimit()` method
- Added `applyDomainLimits()`, `removeDomainLimits()`, `getDomainLimits()` methods
2. **Domain Rate Limit Configuration:**
```typescript
interface IEmailDomainConfig {
domain: string;
rateLimits?: {
outbound?: {
messagesPerMinute?: number;
messagesPerHour?: number; // Note: Hour/day limits need additional implementation
messagesPerDay?: number;
};
inbound?: {
messagesPerMinute?: number;
connectionsPerIp?: number;
recipientsPerMessage?: number;
};
};
}
```
3. **Automatic Application:**
- UnifiedEmailServer applies domain rate limits during startup
- `applyDomainRateLimits()` method converts domain config to rate limiter format
- Domain limits override pattern and global limits
4. **Usage Pattern:**
```typescript
// Domain configuration with rate limits
{
domain: 'high-volume.com',
dnsMode: 'internal-dns',
rateLimits: {
outbound: {
messagesPerMinute: 200 // Higher than global limit
},
inbound: {
recipientsPerMessage: 100 // Higher recipient limit
}
}
}
```
5. **Rate Limit Precedence:**
- Domain-specific limits (highest priority)
- Pattern-specific limits
- Global limits (lowest priority)
### Integration Status
- ✅ Rate limiter supports per-domain limits
- ✅ UnifiedEmailServer applies domain limits on startup
- ✅ Domain limits properly override global/pattern limits
- ✅ SMTP server handlers now enforce rate limits (COMPLETED 2025-05-29)
- ⚠️ Hour/day limits need additional implementation in rate limiter
### SMTP Handler Integration (2025-05-29) - COMPLETED
Rate limiting is now fully integrated into SMTP server handlers:
1. **UnifiedEmailServer Enhancement:**
- Added `getRateLimiter()` method to provide access to the rate limiter
2. **ConnectionManager Integration:**
- Replaced custom rate limiting with UnifiedRateLimiter
- Now uses `rateLimiter.recordConnection(ip)` for all connection checks
- Maintains local IP tracking for resource cleanup only
3. **CommandHandler Integration:**
- `handleMailFrom()`: Checks message rate limits with domain context
- `handleRcptTo()`: Enforces recipient limits per message
- `handleAuth*()`: Records authentication failures and blocks after threshold
- Error handling: Records syntax/command errors and blocks after threshold
4. **SMTP Response Codes:**
- `421`: Temporary rate limit (client should retry later)
- `451`: Temporary recipient rejection
- `421 Too many errors`: IP blocked due to excessive errors
- `421 Too many authentication failures`: IP blocked due to auth failures
### Next Steps
The only remaining item is implementing hour/day rate limits in the UnifiedRateLimiter, which would require:
1. Additional counters for hourly and daily windows
2. Separate tracking for these longer time periods
3. Cleanup logic for expired hourly/daily counters

111
readme.plan.md
View File

@ -63,13 +63,11 @@ interface IDcRouterOptions {
emailConfig?: IUnifiedEmailServerOptions;
}
// Updated UnifiedEmailServerOptions with backward-compatible domains
// Updated UnifiedEmailServerOptions
interface IUnifiedEmailServerOptions {
ports: number[];
hostname: string;
// Backward compatible - can be strings or full config objects
domains: (string | IEmailDomainConfig)[];
domains: IEmailDomainConfig[];
// Pattern-based routing rules (evaluated after domain matching)
routes: IEmailRoute[];
@ -249,83 +247,83 @@ When a domain is configured with `dnsMode: 'forward'`:
### Phase 1: Storage Manager Implementation
#### 1.1 Create Storage Manager Core
- [ ] Create `ts/storage/classes.storagemanager.ts`
- [ ] Implement base StorageManager class
- [ ] Add storage backend detection logic
- [ ] Implement key namespacing for different components
- [x] Create `ts/storage/classes.storagemanager.ts`
- [x] Implement base StorageManager class
- [x] Add storage backend detection logic
- [x] Implement key namespacing for different components
#### 1.2 Implement Storage Backends
- [ ] Filesystem backend using `fsPath`
- [ ] Custom function backend using provided read/write functions
- [ ] Memory backend with Map<string, string>
- [ ] Add console warning for memory backend
- [x] Filesystem backend using `fsPath`
- [x] Custom function backend using provided read/write functions
- [x] Memory backend with Map<string, string>
- [x] Add console warning for memory backend
#### 1.3 Storage Interface Methods
- [ ] Implement get/set/delete/list/exists methods
- [ ] Add JSON serialization helpers
- [ ] Add atomic write support for filesystem
- [ ] Add key validation and sanitization
- [x] Implement get/set/delete/list/exists methods
- [x] Add JSON serialization helpers
- [x] Add atomic write support for filesystem
- [x] Add key validation and sanitization
#### 1.4 Integration Points
- [ ] Add StorageManager instance to DcRouter
- [ ] Pass storage to components that need it
- [x] Add StorageManager instance to DcRouter
- [x] Pass storage to components that need it
- [ ] Update component constructors to accept storage
### Phase 2: Email DNS Configuration Implementation
#### 2.1 Update Email Configuration
- [ ] Update IUnifiedEmailServerOptions to support string | IEmailDomainConfig
- [ ] Implement backward compatibility for string domains
- [ ] Add domain configuration validation
- [ ] Implement infrastructure vs routing separation
- [x] Update IUnifiedEmailServerOptions to support IEmailDomainConfig
- [x] Add domain configuration validation
- [x] Implement infrastructure vs routing separation
#### 2.2 Domain Configuration Processing
- [ ] Parse domains array (string vs IEmailDomainConfig)
- [ ] Apply global defaults to domain configs
- [ ] Validate each domain's DNS mode configuration
- [ ] Create domain registry for quick lookups
- [x] Apply global defaults to domain configs
- [x] Validate each domain's DNS mode configuration
- [x] Create domain registry for quick lookups
#### 2.3 DNS Mode Implementations
- [ ] **Forward Mode**: Skip DNS handling, validate target reachability
- [ ] **Internal DNS Mode**:
- [ ] Validate dnsDomain is set in DcRouter config
- [ ] Check NS delegation exists (query for NS records)
- [ ] Show instructions if NS delegation is missing
- [ ] Log success if NS delegation is properly configured
- [ ] Automatically create MX, SPF, DKIM, DMARC records in internal DNS
- [ ] Apply TTL (default: 3600) and MX priority (default: 10)
- [ ] Store records via StorageManager
- [ ] Register domains with DnsServer
- [ ] **External DNS Mode**:
- [ ] Use standard DNS resolution (or custom servers if specified)
- [ ] Always validate required records (default: MX, SPF, DKIM, DMARC)
- [ ] Always show setup instructions if records are missing
- [x] **Forward Mode**: Skip DNS handling, validate target reachability
- [x] **Internal DNS Mode**:
- [x] Validate dnsDomain is set in DcRouter config
- [x] Check NS delegation exists (query for NS records)
- [x] Show instructions if NS delegation is missing
- [x] Log success if NS delegation is properly configured
- [x] Automatically create MX, SPF, DKIM, DMARC records in internal DNS
- [x] Apply TTL (default: 3600) and MX priority (default: 10)
- [x] Store records via StorageManager
- [x] Register domains with DnsServer
- [x] **External DNS Mode**:
- [x] Use standard DNS resolution (or custom servers if specified)
- [x] Always validate required records (default: MX, SPF, DKIM, DMARC)
- [x] Always show setup instructions if records are missing
- [ ] Cache DNS query results in storage
#### 2.4 Per-Domain Features
- [ ] Implement per-domain DKIM key management
- [ ] Apply per-domain rate limits
- [x] Implement per-domain DKIM key management
- [x] Apply per-domain rate limits
- [x] Integrate rate limiting into SMTP server handlers
- [ ] Handle per-domain email processing rules
- [ ] Automatic DKIM key rotation based on domain config
- [x] Automatic DKIM key rotation based on domain config
### Phase 3: Storage Usage Implementation
#### 3.1 Email Component Storage
- [ ] DKIM keys storage
- [ ] Email routing rules storage
- [ ] Bounce/complaint tracking
- [ ] Reputation data persistence
- [x] DKIM keys storage (DKIMCreator updated with StorageManager)
- [x] Email routing rules storage (EmailRouter updated with persistence support)
- [x] Bounce/complaint tracking (BounceManager updated with StorageManager)
- [x] Reputation data persistence (SenderReputationMonitor and IPReputationChecker updated)
#### 3.2 DNS Component Storage
- [ ] DNS records storage
- [ ] DNSSEC keys storage
- [ ] Zone data persistence
- [ ] Cache storage
- [N/A] DNS records storage (handled by smartdns library internally)
- [N/A] DNSSEC keys storage (handled by smartdns library internally)
- [N/A] Zone data persistence (handled by smartdns library internally)
- [N/A] Cache storage (handled by smartdns library internally)
#### 3.3 Certificate Storage
- [ ] Let's Encrypt certificates
- [ ] Certificate renewal data
- [ ] ACME account keys
- [N/A] Let's Encrypt certificates (handled by SmartProxy library)
- [N/A] Certificate renewal data (handled by SmartProxy library)
- [N/A] ACME account keys (handled by SmartProxy library)
### Phase 4: Testing
@ -379,12 +377,7 @@ console.warn(
emailConfig: {
ports: [25, 587, 465],
hostname: 'mail.myservice.com',
// Mix of simple strings and full configs (backward compatible)
domains: [
// Simple domain (uses all defaults including DKIM)
'simple.com',
{
// Forward-only domain (no local DNS needed)
domain: 'forwarded.com',
@ -499,7 +492,6 @@ console.warn(
### Email DNS Benefits
- **Unified Configuration**: Infrastructure and routing cleanly separated
- **Backward Compatible**: Existing string domain configs continue to work
- **Flexible DNS Modes**: Choose per-domain how DNS is handled
- **External Mode**: Works with existing DNS infrastructure
- **Internal Mode**: Self-contained email system with automatic record creation
@ -522,7 +514,6 @@ console.warn(
- Keys are hierarchical (path-like)
- Values are strings (JSON for complex data)
- Internal DNS mode requires `dnsDomain` to be set in DcRouter
- Domain configuration is backward compatible (strings still work)
- Clean separation: domains = infrastructure, routes = handling
- Domain config only defines: which domains, DNS mode, DKIM settings, rate limits
- DKIM is always enabled for all domains (use dkim object to override defaults)

236
test/test.rate-limiting-integration.ts Normal file
View File

@ -0,0 +1,236 @@
import { expect, tap } from '@git.zone/tstest/tapbundle';
import * as plugins from './helpers/server.loader.js';
import { createTestSmtpClient } from './helpers/smtp.client.js';
import { SmtpClient } from '../ts/mail/delivery/smtpclient/smtp-client.js';
const TEST_PORT = 2525;
// Test email configuration with rate limits
const testEmailConfig = {
ports: [TEST_PORT],
hostname: 'localhost',
domains: [
{
domain: 'test.local',
dnsMode: 'forward' as const,
rateLimits: {
inbound: {
messagesPerMinute: 3, // Very low limit for testing
recipientsPerMessage: 2,
connectionsPerIp: 5
}
}
}
],
routes: [
{
name: 'test-route',
match: { recipients: '*@test.local' },
action: { type: 'process' as const, process: { scan: false, queue: 'normal' } }
}
],
rateLimits: {
global: {
maxMessagesPerMinute: 10,
maxConnectionsPerIP: 10,
maxErrorsPerIP: 3,
maxAuthFailuresPerIP: 2,
blockDuration: 5000 // 5 seconds for testing
}
}
};
tap.test('prepare server with rate limiting', async () => {
await plugins.startTestServer(testEmailConfig);
// Give server time to start
await new Promise(resolve => setTimeout(resolve, 1000));
});
tap.test('should enforce connection rate limits', async (tools) => {
const done = tools.defer();
const clients: SmtpClient[] = [];
try {
// Try to create many connections quickly
for (let i = 0; i < 12; i++) {
const client = createTestSmtpClient();
clients.push(client);
// Connection should fail after limit is exceeded
const verified = await client.verify().catch(() => false);
if (i < 10) {
// First 10 should succeed (global limit)
expect(verified).toBeTrue();
} else {
// After 10, should be rate limited
expect(verified).toBeFalse();
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
// Clean up connections
for (const client of clients) {
await client.close().catch(() => {});
}
}
});
tap.test('should enforce message rate limits per domain', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Send messages rapidly to test domain-specific rate limit
for (let i = 0; i < 5; i++) {
const email = {
from: `sender${i}@example.com`,
to: 'recipient@test.local',
subject: `Test ${i}`,
text: 'Test message'
};
const result = await client.sendMail(email).catch(err => err);
if (i < 3) {
// First 3 should succeed (domain limit is 3 per minute)
expect(result.accepted).toBeDefined();
expect(result.accepted.length).toEqual(1);
} else {
// After 3, should be rate limited
expect(result.code).toEqual('EENVELOPE');
expect(result.response).toContain('try again later');
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close();
}
});
tap.test('should enforce recipient limits', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Try to send to many recipients (domain limit is 2 per message)
const email = {
from: 'sender@example.com',
to: ['user1@test.local', 'user2@test.local', 'user3@test.local'],
subject: 'Test with multiple recipients',
text: 'Test message'
};
const result = await client.sendMail(email).catch(err => err);
// Should fail due to recipient limit
expect(result.code).toEqual('EENVELOPE');
expect(result.response).toContain('try again later');
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close();
}
});
tap.test('should enforce error rate limits', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Send multiple invalid commands to trigger error rate limit
const socket = (client as any).socket;
// Wait for connection
await new Promise(resolve => setTimeout(resolve, 100));
// Send invalid commands
for (let i = 0; i < 5; i++) {
socket.write('INVALID_COMMAND\r\n');
// Wait for response
await new Promise(resolve => {
socket.once('data', resolve);
});
}
// After 3 errors, connection should be blocked
const lastResponse = await new Promise<string>(resolve => {
socket.once('data', (data: Buffer) => resolve(data.toString()));
socket.write('NOOP\r\n');
});
expect(lastResponse).toContain('421 Too many errors');
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close().catch(() => {});
}
});
tap.test('should enforce authentication failure limits', async (tools) => {
const done = tools.defer();
// Create config with auth required
const authConfig = {
...testEmailConfig,
auth: {
required: true,
methods: ['PLAIN' as const]
}
};
// Restart server with auth config
await plugins.stopTestServer();
await plugins.startTestServer(authConfig);
await new Promise(resolve => setTimeout(resolve, 1000));
const client = createTestSmtpClient();
try {
// Try multiple failed authentications
for (let i = 0; i < 3; i++) {
const result = await client.sendMail({
from: 'sender@example.com',
to: 'recipient@test.local',
subject: 'Test',
text: 'Test'
}, {
auth: {
user: 'wronguser',
pass: 'wrongpass'
}
}).catch(err => err);
if (i < 2) {
// First 2 should fail with auth error
expect(result.code).toEqual('EAUTH');
} else {
// After 2 failures, should be blocked
expect(result.code).toEqual('ECONNECTION');
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close().catch(() => {});
}
});
tap.test('cleanup server', async () => {
await plugins.stopTestServer();
});
tap.start();

8
ts/classes.dcrouter.ts
View File

@ -9,6 +9,8 @@ import type { IEmailRoute } from './mail/routing/interfaces.js';
import { logger } from './logger.js';
// Import the email configuration helpers directly from mail/delivery
import { configureEmailStorage, configureEmailServer } from './mail/delivery/index.js';
// Import storage manager
import { StorageManager, type IStorageConfig } from './storage/index.js';
export interface IDcRouterOptions {
/**
@ -63,6 +65,9 @@ export interface IDcRouterOptions {
cloudflareApiKey?: string;
/** Other DNS providers can be added here */
};
/** Storage configuration */
storage?: IStorageConfig;
}
/**
@ -86,6 +91,7 @@ export class DcRouter {
public smartProxy?: plugins.smartproxy.SmartProxy;
public dnsServer?: plugins.smartdns.dnsServerMod.DnsServer;
public emailServer?: UnifiedEmailServer;
public storageManager: StorageManager;
// Environment access
@ -97,6 +103,8 @@ export class DcRouter {
...optionsArg
};
// Initialize storage manager
this.storageManager = new StorageManager(this.options.storage);
}
public async start() {

199
ts/deliverability/classes.senderreputationmonitor.ts
View File

@ -171,12 +171,14 @@ export class SenderReputationMonitor {
private reputationData: Map<string, IDomainReputationMetrics> = new Map();
private updateTimer: NodeJS.Timeout = null;
private isInitialized: boolean = false;
private storageManager?: any; // StorageManager instance
/**
* Constructor for SenderReputationMonitor
* @param config Configuration options
* @param storageManager Optional StorageManager instance
*/
constructor(config: IReputationMonitorConfig = {}) {
constructor(config: IReputationMonitorConfig = {}, storageManager?: any) {
// Merge with default config
this.config = {
...DEFAULT_CONFIG,
@ -191,18 +193,34 @@ export class SenderReputationMonitor {
}
};
// Initialize
this.initialize();
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager) {
logger.log('warn',
'⚠️ WARNING: SenderReputationMonitor initialized without StorageManager.\n' +
' Reputation data will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Initialize (async, but we don't await here to avoid blocking constructor)
this.initialize().catch(error => {
logger.log('error', `Failed to initialize SenderReputationMonitor: ${error.message}`, {
stack: error.stack
});
});
}
/**
* Get the singleton instance
* @param config Configuration options
* @param storageManager Optional StorageManager instance
* @returns Singleton instance
*/
public static getInstance(config: IReputationMonitorConfig = {}): SenderReputationMonitor {
public static getInstance(config: IReputationMonitorConfig = {}, storageManager?: any): SenderReputationMonitor {
if (!SenderReputationMonitor.instance) {
SenderReputationMonitor.instance = new SenderReputationMonitor(config);
SenderReputationMonitor.instance = new SenderReputationMonitor(config, storageManager);
}
return SenderReputationMonitor.instance;
}
@ -210,7 +228,7 @@ export class SenderReputationMonitor {
/**
* Initialize the reputation monitor
*/
private initialize(): void {
private async initialize(): Promise<void> {
if (this.isInitialized) return;
try {
@ -219,7 +237,7 @@ export class SenderReputationMonitor {
if (!isTestEnvironment) {
// Load existing reputation data
this.loadReputationData();
await this.loadReputationData();
}
// Initialize data for any new domains
@ -354,7 +372,7 @@ export class SenderReputationMonitor {
}
// Save all updated data
this.saveReputationData();
await this.saveReputationData();
logger.log('info', 'Completed reputation metrics update for all domains');
}
@ -977,7 +995,11 @@ export class SenderReputationMonitor {
// Skip in test environment
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (!isTestEnvironment && Math.random() < 0.01) { // ~1% chance to save on each event
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data: ${error.message}`, {
stack: error.stack
});
});
}
}
@ -1037,7 +1059,11 @@ export class SenderReputationMonitor {
this.config.domains.push(domain);
this.initializeDomainData(domain);
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data after adding domain: ${error.message}`, {
stack: error.stack
});
});
logger.log('info', `Added ${domain} to reputation monitoring`);
}
@ -1055,7 +1081,11 @@ export class SenderReputationMonitor {
this.config.domains.splice(index, 1);
this.reputationData.delete(domain);
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data after removing domain: ${error.message}`, {
stack: error.stack
});
});
logger.log('info', `Removed ${domain} from reputation monitoring`);
}
@ -1063,7 +1093,7 @@ export class SenderReputationMonitor {
/**
* Load reputation data from storage
*/
private loadReputationData(): void {
private async loadReputationData(): Promise<void> {
// Skip loading in test environment to prevent file system race conditions
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (isTestEnvironment) {
@ -1071,32 +1101,98 @@ export class SenderReputationMonitor {
}
try {
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
// Try to load from storage manager first
if (this.storageManager) {
try {
const data = await this.storageManager.get('/email/reputation/domain-reputation.json');
if (data) {
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains from StorageManager`);
return;
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
} catch (error) {
// Fall through to filesystem migration check
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains`);
// Check if data exists in filesystem and migrate it to storage manager
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
// Migrate to storage manager
logger.log('info', `Migrating reputation data for ${this.reputationData.size} domains from filesystem to StorageManager`);
await this.storageManager.set(
'/email/reputation/domain-reputation.json',
JSON.stringify(Array.from(this.reputationData.values()), null, 2)
);
logger.log('info', `Loaded and migrated reputation data for ${this.reputationData.size} domains`);
}
} else {
// No storage manager, use filesystem directly
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains from filesystem`);
}
}
} catch (error) {
logger.log('error', `Failed to load reputation data: ${error.message}`, {
@ -1108,7 +1204,7 @@ export class SenderReputationMonitor {
/**
* Save reputation data to storage
*/
private saveReputationData(): void {
private async saveReputationData(): Promise<void> {
// Skip saving in test environment to prevent file system race conditions
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (isTestEnvironment) {
@ -1116,18 +1212,29 @@ export class SenderReputationMonitor {
}
try {
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
const reputationEntries = Array.from(this.reputationData.values());
plugins.smartfile.memory.toFsSync(
JSON.stringify(reputationEntries, null, 2),
dataFile
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains`);
// Save to storage manager if available
if (this.storageManager) {
await this.storageManager.set(
'/email/reputation/domain-reputation.json',
JSON.stringify(reputationEntries, null, 2)
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains to StorageManager`);
} else {
// No storage manager, use filesystem directly
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
plugins.smartfile.memory.toFsSync(
JSON.stringify(reputationEntries, null, 2),
dataFile
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains to filesystem`);
}
} catch (error) {
logger.log('error', `Failed to save reputation data: ${error.message}`, {
stack: error.stack

133
ts/mail/core/classes.bouncemanager.ts
View File

@ -209,10 +209,13 @@ export class BounceManager {
expiresAt?: number; // undefined means permanent
}> = new Map();
private storageManager?: any; // StorageManager instance
constructor(options?: {
retryStrategy?: Partial<RetryStrategy>;
maxCacheSize?: number;
cacheTTL?: number;
storageManager?: any;
}) {
// Set retry strategy with defaults
if (options?.retryStrategy) {
@ -228,8 +231,24 @@ export class BounceManager {
ttl: options?.cacheTTL || 30 * 24 * 60 * 60 * 1000, // 30 days default
});
// Store storage manager reference
this.storageManager = options?.storageManager;
// If no storage manager provided, log warning
if (!this.storageManager) {
console.warn(
'⚠️ WARNING: BounceManager initialized without StorageManager.\n' +
' Bounce data will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Load suppression list from storage
this.loadSuppressionList();
// Note: This is async but we can't await in constructor
// The suppression list will be loaded asynchronously
this.loadSuppressionList().catch(error => {
logger.log('error', `Failed to load suppression list on startup: ${error.message}`);
});
}
/**
@ -479,7 +498,7 @@ export class BounceManager {
this.updateBounceCache(bounce);
// Save to permanent storage
this.saveBounceRecord(bounce);
await this.saveBounceRecord(bounce);
// Log hard bounce for monitoring
logger.log('warn', `Hard bounce for ${bounce.recipient}: ${bounce.bounceType}`, {
@ -545,7 +564,10 @@ export class BounceManager {
expiresAt
});
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after adding ${email}: ${error.message}`);
});
logger.log('info', `Added ${email} to suppression list`, {
reason,
@ -561,7 +583,10 @@ export class BounceManager {
const wasRemoved = this.suppressionList.delete(email.toLowerCase());
if (wasRemoved) {
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after removing ${email}: ${error.message}`);
});
logger.log('info', `Removed ${email} from suppression list`);
}
}
@ -582,7 +607,10 @@ export class BounceManager {
// Check if suppression has expired
if (suppression.expiresAt && Date.now() > suppression.expiresAt) {
this.suppressionList.delete(lowercaseEmail);
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after expiry cleanup: ${error.message}`);
});
return false;
}
@ -609,7 +637,10 @@ export class BounceManager {
// Check if suppression has expired
if (suppression.expiresAt && Date.now() > suppression.expiresAt) {
this.suppressionList.delete(lowercaseEmail);
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after expiry cleanup: ${error.message}`);
});
return null;
}
@ -619,13 +650,20 @@ export class BounceManager {
/**
* Save suppression list to disk
*/
private saveSuppressionList(): void {
private async saveSuppressionList(): Promise<void> {
try {
const suppressionData = JSON.stringify(Array.from(this.suppressionList.entries()));
plugins.smartfile.memory.toFsSync(
suppressionData,
plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json')
);
if (this.storageManager) {
// Use storage manager
await this.storageManager.set('/email/bounces/suppression-list.json', suppressionData);
} else {
// Fall back to filesystem
plugins.smartfile.memory.toFsSync(
suppressionData,
plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json')
);
}
} catch (error) {
logger.log('error', `Failed to save suppression list: ${error.message}`);
}
@ -634,14 +672,40 @@ export class BounceManager {
/**
* Load suppression list from disk
*/
private loadSuppressionList(): void {
private async loadSuppressionList(): Promise<void> {
try {
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
let entries = null;
let needsMigration = false;
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
const entries = JSON.parse(data);
if (this.storageManager) {
// Try to load from storage manager first
const suppressionData = await this.storageManager.get('/email/bounces/suppression-list.json');
if (suppressionData) {
entries = JSON.parse(suppressionData);
} else {
// Check if data exists in filesystem and migrate
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
entries = JSON.parse(data);
needsMigration = true;
logger.log('info', 'Migrating suppression list from filesystem to StorageManager');
}
}
} else {
// No storage manager, use filesystem directly
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
entries = JSON.parse(data);
}
}
if (entries) {
this.suppressionList = new Map(entries);
// Clean expired entries
@ -655,9 +719,9 @@ export class BounceManager {
}
}
if (expiredCount > 0) {
if (expiredCount > 0 || needsMigration) {
logger.log('info', `Cleaned ${expiredCount} expired entries from suppression list`);
this.saveSuppressionList();
await this.saveSuppressionList();
}
logger.log('info', `Loaded ${this.suppressionList.size} entries from suppression list`);
@ -671,21 +735,28 @@ export class BounceManager {
* Save bounce record to disk
* @param bounce Bounce record to save
*/
private saveBounceRecord(bounce: BounceRecord): void {
private async saveBounceRecord(bounce: BounceRecord): Promise<void> {
try {
const bounceData = JSON.stringify(bounce);
const bouncePath = plugins.path.join(
paths.dataDir,
'emails',
'bounces',
`${bounce.id}.json`
);
const bounceData = JSON.stringify(bounce, null, 2);
// Ensure directory exists
const bounceDir = plugins.path.join(paths.dataDir, 'emails', 'bounces');
plugins.smartfile.fs.ensureDirSync(bounceDir);
plugins.smartfile.memory.toFsSync(bounceData, bouncePath);
if (this.storageManager) {
// Use storage manager
await this.storageManager.set(`/email/bounces/records/${bounce.id}.json`, bounceData);
} else {
// Fall back to filesystem
const bouncePath = plugins.path.join(
paths.dataDir,
'emails',
'bounces',
`${bounce.id}.json`
);
// Ensure directory exists
const bounceDir = plugins.path.join(paths.dataDir, 'emails', 'bounces');
plugins.smartfile.fs.ensureDirSync(bounceDir);
plugins.smartfile.memory.toFsSync(bounceData, bouncePath);
}
} catch (error) {
logger.log('error', `Failed to save bounce record: ${error.message}`);
}

135
ts/mail/delivery/classes.unified.rate.limiter.ts
View File

@ -28,6 +28,9 @@ export interface IHierarchicalRateLimits {
// IP-specific rate limits (applied to specific IPs)
ips?: Record<string, IRateLimitConfig>;
// Domain-specific rate limits (applied to specific email domains)
domains?: Record<string, IRateLimitConfig>;
// Temporary blocks list and their expiry times
blocks?: Record<string, number>; // IP to expiry timestamp
}
@ -86,6 +89,7 @@ export class UnifiedRateLimiter extends EventEmitter {
private counters: Map<string, ILimitCounter> = new Map();
private patternCounters: Map<string, ILimitCounter> = new Map();
private ipCounters: Map<string, ILimitCounter> = new Map();
private domainCounters: Map<string, ILimitCounter> = new Map();
private cleanupInterval?: NodeJS.Timeout;
private stats: IRateLimiterStats;
@ -221,6 +225,13 @@ export class UnifiedRateLimiter extends EventEmitter {
}
}
// Clean domain counters
for (const [key, counter] of this.domainCounters.entries()) {
if (counter.lastReset < cutoff) {
this.domainCounters.delete(key);
}
}
// Update statistics
this.updateStats();
}
@ -231,9 +242,10 @@ export class UnifiedRateLimiter extends EventEmitter {
* @param ip IP address
* @param recipients Number of recipients
* @param pattern Matched pattern
* @param domain Domain name for domain-specific limits
* @returns Result of rate limit check
*/
public checkMessageLimit(email: string, ip: string, recipients: number, pattern?: string): IRateLimitResult {
public checkMessageLimit(email: string, ip: string, recipients: number, pattern?: string, domain?: string): IRateLimitResult {
// Check if IP is blocked
if (this.isIpBlocked(ip)) {
return {
@ -257,6 +269,14 @@ export class UnifiedRateLimiter extends EventEmitter {
}
}
// Check domain-specific limit if domain is provided
if (domain) {
const domainResult = this.checkDomainMessageLimit(domain);
if (!domainResult.allowed) {
return domainResult;
}
}
// Check IP-specific limit
const ipResult = this.checkIpMessageLimit(ip);
if (!ipResult.allowed) {
@ -264,7 +284,7 @@ export class UnifiedRateLimiter extends EventEmitter {
}
// Check recipient limit
const recipientResult = this.checkRecipientLimit(email, recipients, pattern);
const recipientResult = this.checkRecipientLimit(email, recipients, pattern, domain);
if (!recipientResult.allowed) {
return recipientResult;
}
@ -403,6 +423,64 @@ export class UnifiedRateLimiter extends EventEmitter {
return { allowed: true };
}
/**
* Check domain-specific message rate limit
* @param domain Domain to check
*/
private checkDomainMessageLimit(domain: string): IRateLimitResult {
const now = Date.now();
// Get domain-specific limit or use global
const domainConfig = this.config.domains?.[domain];
const limit = domainConfig?.maxMessagesPerMinute || this.config.global.maxMessagesPerMinute!;
if (!limit) {
return { allowed: true };
}
// Get or create counter
let counter = this.domainCounters.get(domain);
if (!counter) {
counter = {
count: 0,
lastReset: now,
recipients: 0,
errors: 0,
authFailures: 0,
connections: 0
};
this.domainCounters.set(domain, counter);
}
// Check if counter needs to be reset
if (now - counter.lastReset >= 60000) {
counter.count = 0;
counter.lastReset = now;
}
// Check if limit is exceeded
if (counter.count >= limit) {
// Calculate reset time
const resetIn = 60000 - (now - counter.lastReset);
logger.log('warn', `Domain ${domain} rate limit exceeded: ${counter.count}/${limit} messages per minute`);
return {
allowed: false,
reason: `Domain "${domain}" message rate limit exceeded`,
limit,
current: counter.count,
resetIn
};
}
// Increment counter
counter.count++;
return { allowed: true };
}
/**
* Check IP-specific message rate limit
* @param ip IP address
@ -485,15 +563,22 @@ export class UnifiedRateLimiter extends EventEmitter {
* @param email Email address
* @param recipients Number of recipients
* @param pattern Matched pattern
* @param domain Domain name
*/
private checkRecipientLimit(email: string, recipients: number, pattern?: string): IRateLimitResult {
// Get pattern-specific limit if available
private checkRecipientLimit(email: string, recipients: number, pattern?: string, domain?: string): IRateLimitResult {
// Get the most specific limit available
let limit = this.config.global.maxRecipientsPerMessage!;
// Check pattern-specific limit
if (pattern && this.config.patterns?.[pattern]?.maxRecipientsPerMessage) {
limit = this.config.patterns[pattern].maxRecipientsPerMessage!;
}
// Check domain-specific limit (overrides pattern if present)
if (domain && this.config.domains?.[domain]?.maxRecipientsPerMessage) {
limit = this.config.domains[domain].maxRecipientsPerMessage!;
}
if (!limit) {
return { allowed: true };
}
@ -923,4 +1008,46 @@ export class UnifiedRateLimiter extends EventEmitter {
public getConfig(): IHierarchicalRateLimits {
return { ...this.config };
}
/**
* Apply domain-specific rate limits
* Merges domain limits with existing configuration
* @param domain Domain name
* @param limits Rate limit configuration for the domain
*/
public applyDomainLimits(domain: string, limits: IRateLimitConfig): void {
if (!this.config.domains) {
this.config.domains = {};
}
// Merge the limits with any existing domain config
this.config.domains[domain] = {
...this.config.domains[domain],
...limits
};
logger.log('info', `Applied rate limits for domain ${domain}:`, limits);
}
/**
* Remove domain-specific rate limits
* @param domain Domain name
*/
public removeDomainLimits(domain: string): void {
if (this.config.domains && this.config.domains[domain]) {
delete this.config.domains[domain];
// Also remove the counter
this.domainCounters.delete(domain);
logger.log('info', `Removed rate limits for domain ${domain}`);
}
}
/**
* Get domain-specific rate limits
* @param domain Domain name
* @returns Domain rate limit config or undefined
*/
public getDomainLimits(domain: string): IRateLimitConfig | undefined {
return this.config.domains?.[domain];
}
}

103
ts/mail/delivery/smtpserver/command-handler.ts
View File

@ -142,15 +142,37 @@ export class CommandHandler implements ICommandHandler {
// For the ERR-01 test, an empty or invalid command is considered a syntax error (500)
if (!command || command.trim().length === 0) {
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
// Record error for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordError(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive errors`);
this.sendResponse(socket, `421 Too many errors - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
}
return;
}
// Handle unknown commands - this should happen before sequence validation
// RFC 5321: Use 500 for unrecognized commands, 501 for parameter errors
if (!Object.values(SmtpCommand).includes(command.toUpperCase() as SmtpCommand)) {
// Comply with RFC 5321 section 4.2.4: Use 500 for unrecognized commands
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
// Record error for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordError(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive errors`);
this.sendResponse(socket, `421 Too many errors - connection blocked`);
socket.end();
} else {
// Comply with RFC 5321 section 4.2.4: Use 500 for unrecognized commands
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
}
return;
}
@ -477,6 +499,12 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Get rate limiter for message-level checks
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Note: Connection-level rate limiting is already handled in ConnectionManager
// Special handling for commands that include "MAIL FROM:" in the args
let processedArgs = args;
@ -509,6 +537,26 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Check message rate limits for this sender
const senderAddress = validation.address || '';
const senderDomain = senderAddress.includes('@') ? senderAddress.split('@')[1] : undefined;
// Check rate limits with domain context if available
const messageResult = rateLimiter.checkMessageLimit(
senderAddress,
session.remoteAddress,
1, // We don't know recipients yet, check with 1
undefined, // No pattern matching for now
senderDomain // Pass domain for domain-specific limits
);
if (!messageResult.allowed) {
SmtpLogger.warn(`Message rate limit exceeded for ${senderAddress} from IP ${session.remoteAddress}: ${messageResult.reason}`);
// Use 421 for temporary rate limiting (client should retry later)
this.sendResponse(socket, `421 ${messageResult.reason} - try again later`);
return;
}
// Enhanced SIZE parameter handling
if (validation.params && validation.params.SIZE) {
const size = parseInt(validation.params.SIZE, 10);
@ -619,6 +667,29 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Check rate limits for recipients
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const recipientAddress = validation.address || '';
const recipientDomain = recipientAddress.includes('@') ? recipientAddress.split('@')[1] : undefined;
// Check rate limits with accumulated recipient count
const recipientCount = session.rcptTo.length + 1; // Including this new recipient
const messageResult = rateLimiter.checkMessageLimit(
session.mailFrom,
session.remoteAddress,
recipientCount,
undefined, // No pattern matching for now
recipientDomain // Pass recipient domain for domain-specific limits
);
if (!messageResult.allowed) {
SmtpLogger.warn(`Recipient rate limit exceeded for ${recipientAddress} from IP ${session.remoteAddress}: ${messageResult.reason}`);
// Use 451 for temporary recipient rejection
this.sendResponse(socket, `451 ${messageResult.reason} - try again later`);
return;
}
// Create recipient object
const recipient: IEnvelopeRecipient = {
address: validation.address || '',
@ -864,7 +935,18 @@ export class CommandHandler implements ICommandHandler {
session.username = username;
this.sendResponse(socket, `${SmtpResponseCode.AUTHENTICATION_SUCCESSFUL} Authentication successful`);
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
// Record authentication failure for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordAuthFailure(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive authentication failures`);
this.sendResponse(socket, `421 Too many authentication failures - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
}
}
} catch (error) {
SmtpLogger.error(`AUTH PLAIN error: ${error instanceof Error ? error.message : String(error)}`);
@ -945,7 +1027,18 @@ export class CommandHandler implements ICommandHandler {
session.username = username;
this.sendResponse(socket, `${SmtpResponseCode.AUTHENTICATION_SUCCESSFUL} Authentication successful`);
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
// Record authentication failure for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordAuthFailure(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive authentication failures`);
this.sendResponse(socket, `421 Too many authentication failures - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
}
}
}
} catch (error) {

38
ts/mail/delivery/smtpserver/connection-manager.ts
View File

@ -298,19 +298,20 @@ export class ConnectionManager implements IConnectionManager {
// Get client IP
const remoteAddress = socket.remoteAddress || '0.0.0.0';
// Check rate limits by IP
if (this.isIPRateLimited(remoteAddress)) {
this.rejectConnection(socket, 'Rate limit exceeded');
// Use UnifiedRateLimiter for connection rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Check connection limit with UnifiedRateLimiter
const connectionResult = rateLimiter.recordConnection(remoteAddress);
if (!connectionResult.allowed) {
this.rejectConnection(socket, connectionResult.reason || 'Rate limit exceeded');
this.connectionStats.rejectedConnections++;
return;
}
// Check per-IP connection limit
if (this.hasReachedIPConnectionLimit(remoteAddress)) {
this.rejectConnection(socket, 'Too many connections from this IP');
this.connectionStats.rejectedConnections++;
return;
}
// Still track IP connections locally for cleanup purposes
this.trackIPConnection(remoteAddress);
// Check if maximum global connections reached
if (this.hasReachedMaxConnections()) {
@ -454,19 +455,20 @@ export class ConnectionManager implements IConnectionManager {
// Get client IP
const remoteAddress = socket.remoteAddress || '0.0.0.0';
// Check rate limits by IP
if (this.isIPRateLimited(remoteAddress)) {
this.rejectConnection(socket, 'Rate limit exceeded');
// Use UnifiedRateLimiter for connection rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Check connection limit with UnifiedRateLimiter
const connectionResult = rateLimiter.recordConnection(remoteAddress);
if (!connectionResult.allowed) {
this.rejectConnection(socket, connectionResult.reason || 'Rate limit exceeded');
this.connectionStats.rejectedConnections++;
return;
}
// Check per-IP connection limit
if (this.hasReachedIPConnectionLimit(remoteAddress)) {
this.rejectConnection(socket, 'Too many connections from this IP');
this.connectionStats.rejectedConnections++;
return;
}
// Still track IP connections locally for cleanup purposes
this.trackIPConnection(remoteAddress);
// Check if maximum global connections reached
if (this.hasReachedMaxConnections()) {

333
ts/mail/routing/classes.dns.validator.ts Normal file
View File

@ -0,0 +1,333 @@
import * as plugins from '../../plugins.js';
import type { IEmailDomainConfig } from './interfaces.js';
import { logger } from '../../logger.js';
import type { DcRouter } from '../../classes.dcrouter.js';
import type { StorageManager } from '../../storage/index.js';
/**
* DNS validation result
*/
export interface IDnsValidationResult {
valid: boolean;
errors: string[];
warnings: string[];
requiredChanges: string[];
}
/**
* DNS records found for a domain
*/
interface IDnsRecords {
mx?: string[];
spf?: string;
dkim?: string;
dmarc?: string;
ns?: string[];
}
/**
* Validates DNS configuration for email domains
*/
export class DnsValidator {
private dcRouter: DcRouter;
private storageManager: StorageManager;
constructor(dcRouter: DcRouter) {
this.dcRouter = dcRouter;
this.storageManager = dcRouter.storageManager;
}
/**
* Validate all domain configurations
*/
async validateAllDomains(domainConfigs: IEmailDomainConfig[]): Promise<Map<string, IDnsValidationResult>> {
const results = new Map<string, IDnsValidationResult>();
for (const config of domainConfigs) {
const result = await this.validateDomain(config);
results.set(config.domain, result);
}
return results;
}
/**
* Validate a single domain configuration
*/
async validateDomain(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
switch (config.dnsMode) {
case 'forward':
return this.validateForwardMode(config);
case 'internal-dns':
return this.validateInternalDnsMode(config);
case 'external-dns':
return this.validateExternalDnsMode(config);
default:
return {
valid: false,
errors: [`Unknown DNS mode: ${config.dnsMode}`],
warnings: [],
requiredChanges: []
};
}
}
/**
* Validate forward mode configuration
*/
private async validateForwardMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
// Forward mode doesn't require DNS validation by default
if (!config.dns?.forward?.skipDnsValidation) {
logger.log('info', `DNS validation skipped for forward mode domain: ${config.domain}`);
}
// DKIM keys are still generated for consistency
result.warnings.push(
`Domain "${config.domain}" uses forward mode. DKIM keys will be generated but signing only happens if email is processed.`
);
return result;
}
/**
* Validate internal DNS mode configuration
*/
private async validateInternalDnsMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
// Check if dnsDomain is configured
const dnsDomain = (this.dcRouter as any).options?.dnsDomain;
if (!dnsDomain) {
result.valid = false;
result.errors.push(
`Domain "${config.domain}" is configured to use internal DNS, but dnsDomain is not set in DcRouter configuration.`
);
console.error(
`❌ ERROR: Domain "${config.domain}" is configured to use internal DNS,\n` +
' but dnsDomain is not set in DcRouter configuration.\n' +
' Please configure dnsDomain to enable the DNS server.\n' +
' Example: dnsDomain: "ns.myservice.com"'
);
return result;
}
// Check NS delegation
try {
const nsRecords = await this.resolveNs(config.domain);
const isDelegated = nsRecords.includes(dnsDomain);
if (!isDelegated) {
result.warnings.push(
`NS delegation not found for ${config.domain}. Please add NS record at your registrar.`
);
result.requiredChanges.push(
`Add NS record: ${config.domain}. NS ${dnsDomain}.`
);
console.log(
`📋 DNS Delegation Required for ${config.domain}:\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
'Please add this NS record at your domain registrar:\n' +
` ${config.domain}. NS ${dnsDomain}.\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
'This delegation is required for internal DNS mode to work.'
);
} else {
console.log(
`✅ NS delegation verified: ${config.domain} -> ${dnsDomain}`
);
}
} catch (error) {
result.warnings.push(
`Could not verify NS delegation for ${config.domain}: ${error.message}`
);
}
return result;
}
/**
* Validate external DNS mode configuration
*/
private async validateExternalDnsMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
try {
// Get current DNS records
const records = await this.checkDnsRecords(config);
const requiredRecords = config.dns?.external?.requiredRecords || ['MX', 'SPF', 'DKIM', 'DMARC'];
// Check MX record
if (requiredRecords.includes('MX') && !records.mx?.length) {
result.requiredChanges.push(
`Add MX record: ${this.getBaseDomain(config.domain)} -> ${config.domain} (priority 10)`
);
}
// Check SPF record
if (requiredRecords.includes('SPF') && !records.spf) {
result.requiredChanges.push(
`Add TXT record: ${this.getBaseDomain(config.domain)} -> "v=spf1 a mx ~all"`
);
}
// Check DKIM record
if (requiredRecords.includes('DKIM') && !records.dkim) {
const selector = config.dkim?.selector || 'default';
const dkimPublicKey = await this.storageManager.get(`/email/dkim/${config.domain}/public.key`);
if (dkimPublicKey) {
const publicKeyBase64 = dkimPublicKey
.replace(/-----BEGIN PUBLIC KEY-----/g, '')
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
result.requiredChanges.push(
`Add TXT record: ${selector}._domainkey.${config.domain} -> "v=DKIM1; k=rsa; p=${publicKeyBase64}"`
);
} else {
result.warnings.push(
`DKIM public key not found for ${config.domain}. It will be generated on first use.`
);
}
}
// Check DMARC record
if (requiredRecords.includes('DMARC') && !records.dmarc) {
result.requiredChanges.push(
`Add TXT record: _dmarc.${this.getBaseDomain(config.domain)} -> "v=DMARC1; p=none; rua=mailto:dmarc@${config.domain}"`
);
}
// Show setup instructions if needed
if (result.requiredChanges.length > 0) {
console.log(
`📋 DNS Configuration Required for ${config.domain}:\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
result.requiredChanges.map((change, i) => `${i + 1}. ${change}`).join('\n') +
'\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
);
}
} catch (error) {
result.errors.push(`DNS validation failed: ${error.message}`);
result.valid = false;
}
return result;
}
/**
* Check DNS records for a domain
*/
private async checkDnsRecords(config: IEmailDomainConfig): Promise<IDnsRecords> {
const records: IDnsRecords = {};
const baseDomain = this.getBaseDomain(config.domain);
const selector = config.dkim?.selector || 'default';
// Use custom DNS servers if specified
const resolver = new plugins.dns.promises.Resolver();
if (config.dns?.external?.servers?.length) {
resolver.setServers(config.dns.external.servers);
}
// Check MX records
try {
const mxRecords = await resolver.resolveMx(baseDomain);
records.mx = mxRecords.map(mx => mx.exchange);
} catch (error) {
logger.log('debug', `No MX records found for ${baseDomain}`);
}
// Check SPF record
try {
const txtRecords = await resolver.resolveTxt(baseDomain);
const spfRecord = txtRecords.find(records =>
records.some(record => record.startsWith('v=spf1'))
);
if (spfRecord) {
records.spf = spfRecord.join('');
}
} catch (error) {
logger.log('debug', `No SPF record found for ${baseDomain}`);
}
// Check DKIM record
try {
const dkimRecords = await resolver.resolveTxt(`${selector}._domainkey.${config.domain}`);
const dkimRecord = dkimRecords.find(records =>
records.some(record => record.includes('v=DKIM1'))
);
if (dkimRecord) {
records.dkim = dkimRecord.join('');
}
} catch (error) {
logger.log('debug', `No DKIM record found for ${selector}._domainkey.${config.domain}`);
}
// Check DMARC record
try {
const dmarcRecords = await resolver.resolveTxt(`_dmarc.${baseDomain}`);
const dmarcRecord = dmarcRecords.find(records =>
records.some(record => record.startsWith('v=DMARC1'))
);
if (dmarcRecord) {
records.dmarc = dmarcRecord.join('');
}
} catch (error) {
logger.log('debug', `No DMARC record found for _dmarc.${baseDomain}`);
}
return records;
}
/**
* Resolve NS records for a domain
*/
private async resolveNs(domain: string): Promise<string[]> {
try {
const resolver = new plugins.dns.promises.Resolver();
const nsRecords = await resolver.resolveNs(domain);
return nsRecords;
} catch (error) {
logger.log('warn', `Failed to resolve NS records for ${domain}: ${error.message}`);
return [];
}
}
/**
* Get base domain from email domain (e.g., mail.example.com -> example.com)
*/
private getBaseDomain(domain: string): string {
const parts = domain.split('.');
if (parts.length <= 2) {
return domain;
}
// For subdomains like mail.example.com, return example.com
// But preserve domain structure for longer TLDs like .co.uk
if (parts[parts.length - 2].length <= 3 && parts[parts.length - 1].length === 2) {
// Likely a country code TLD like .co.uk
return parts.slice(-3).join('.');
}
return parts.slice(-2).join('.');
}
}

139
ts/mail/routing/classes.domain.registry.ts Normal file
View File

@ -0,0 +1,139 @@
import type { IEmailDomainConfig } from './interfaces.js';
import { logger } from '../../logger.js';
/**
* Registry for email domain configurations
* Provides fast lookups and validation for domains
*/
export class DomainRegistry {
private domains: Map<string, IEmailDomainConfig> = new Map();
private defaults: IEmailDomainConfig['dkim'] & {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
rateLimits?: IEmailDomainConfig['rateLimits'];
};
constructor(
domainConfigs: IEmailDomainConfig[],
defaults?: {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
dkim?: IEmailDomainConfig['dkim'];
rateLimits?: IEmailDomainConfig['rateLimits'];
}
) {
// Set defaults
this.defaults = {
dnsMode: defaults?.dnsMode || 'external-dns',
...this.getDefaultDkimConfig(),
...defaults?.dkim,
rateLimits: defaults?.rateLimits
};
// Process and store domain configurations
for (const config of domainConfigs) {
const processedConfig = this.applyDefaults(config);
this.domains.set(config.domain.toLowerCase(), processedConfig);
logger.log('info', `Registered domain: ${config.domain} with DNS mode: ${processedConfig.dnsMode}`);
}
}
/**
* Get default DKIM configuration
*/
private getDefaultDkimConfig(): IEmailDomainConfig['dkim'] {
return {
selector: 'default',
keySize: 2048,
rotateKeys: false,
rotationInterval: 90
};
}
/**
* Apply defaults to a domain configuration
*/
private applyDefaults(config: IEmailDomainConfig): IEmailDomainConfig {
return {
...config,
dnsMode: config.dnsMode || this.defaults.dnsMode!,
dkim: {
...this.getDefaultDkimConfig(),
...this.defaults,
...config.dkim
},
rateLimits: {
...this.defaults.rateLimits,
...config.rateLimits,
outbound: {
...this.defaults.rateLimits?.outbound,
...config.rateLimits?.outbound
},
inbound: {
...this.defaults.rateLimits?.inbound,
...config.rateLimits?.inbound
}
}
};
}
/**
* Check if a domain is registered
*/
isDomainRegistered(domain: string): boolean {
return this.domains.has(domain.toLowerCase());
}
/**
* Check if an email address belongs to a registered domain
*/
isEmailRegistered(email: string): boolean {
const domain = this.extractDomain(email);
if (!domain) return false;
return this.isDomainRegistered(domain);
}
/**
* Get domain configuration
*/
getDomainConfig(domain: string): IEmailDomainConfig | undefined {
return this.domains.get(domain.toLowerCase());
}
/**
* Get domain configuration for an email address
*/
getEmailDomainConfig(email: string): IEmailDomainConfig | undefined {
const domain = this.extractDomain(email);
if (!domain) return undefined;
return this.getDomainConfig(domain);
}
/**
* Extract domain from email address
*/
private extractDomain(email: string): string | null {
const parts = email.toLowerCase().split('@');
if (parts.length !== 2) return null;
return parts[1];
}
/**
* Get all registered domains
*/
getAllDomains(): string[] {
return Array.from(this.domains.keys());
}
/**
* Get all domain configurations
*/
getAllConfigs(): IEmailDomainConfig[] {
return Array.from(this.domains.values());
}
/**
* Get domains by DNS mode
*/
getDomainsByMode(mode: 'forward' | 'internal-dns' | 'external-dns'): IEmailDomainConfig[] {
return Array.from(this.domains.values()).filter(config => config.dnsMode === mode);
}
}

213
ts/mail/routing/classes.email.router.ts
View File

@ -9,14 +9,29 @@ import type { Email } from '../core/classes.email.js';
export class EmailRouter extends EventEmitter {
private routes: IEmailRoute[];
private patternCache: Map<string, boolean> = new Map();
private storageManager?: any; // StorageManager instance
private persistChanges: boolean;
/**
* Create a new email router
* @param routes Array of email routes
* @param options Router options
*/
constructor(routes: IEmailRoute[]) {
constructor(routes: IEmailRoute[], options?: {
storageManager?: any;
persistChanges?: boolean;
}) {
super();
this.routes = this.sortRoutesByPriority(routes);
this.storageManager = options?.storageManager;
this.persistChanges = options?.persistChanges ?? !!this.storageManager;
// If storage manager is provided, try to load persisted routes
if (this.storageManager) {
this.loadRoutes({ merge: true }).catch(error => {
console.error(`Failed to load persisted routes: ${error.message}`);
});
}
}
/**
@ -43,19 +58,26 @@ export class EmailRouter extends EventEmitter {
/**
* Update routes
* @param routes New routes
* @param persist Whether to persist changes (defaults to persistChanges setting)
*/
public updateRoutes(routes: IEmailRoute[]): void {
public async updateRoutes(routes: IEmailRoute[], persist?: boolean): Promise<void> {
this.routes = this.sortRoutesByPriority(routes);
this.clearCache();
this.emit('routesUpdated', this.routes);
// Persist if requested or if persistChanges is enabled
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Set routes (alias for updateRoutes)
* @param routes New routes
* @param persist Whether to persist changes
*/
public setRoutes(routes: IEmailRoute[]): void {
this.updateRoutes(routes);
public async setRoutes(routes: IEmailRoute[], persist?: boolean): Promise<void> {
await this.updateRoutes(routes, persist);
}
/**
@ -367,4 +389,187 @@ export class EmailRouter extends EventEmitter {
return size;
}
/**
* Save current routes to storage
*/
public async saveRoutes(): Promise<void> {
if (!this.storageManager) {
this.emit('persistenceWarning', 'Cannot save routes: StorageManager not configured');
return;
}
try {
// Validate all routes before saving
for (const route of this.routes) {
if (!route.name || !route.match || !route.action) {
throw new Error(`Invalid route: ${JSON.stringify(route)}`);
}
}
const routesData = JSON.stringify(this.routes, null, 2);
await this.storageManager.set('/email/routes/config.json', routesData);
this.emit('routesPersisted', this.routes.length);
} catch (error) {
console.error(`Failed to save routes: ${error.message}`);
throw error;
}
}
/**
* Load routes from storage
* @param options Load options
*/
public async loadRoutes(options?: {
merge?: boolean; // Merge with existing routes
replace?: boolean; // Replace existing routes
}): Promise<IEmailRoute[]> {
if (!this.storageManager) {
this.emit('persistenceWarning', 'Cannot load routes: StorageManager not configured');
return [];
}
try {
const routesData = await this.storageManager.get('/email/routes/config.json');
if (!routesData) {
return [];
}
const loadedRoutes = JSON.parse(routesData) as IEmailRoute[];
// Validate loaded routes
for (const route of loadedRoutes) {
if (!route.name || !route.match || !route.action) {
console.warn(`Skipping invalid route: ${JSON.stringify(route)}`);
continue;
}
}
if (options?.replace) {
// Replace all routes
this.routes = this.sortRoutesByPriority(loadedRoutes);
} else if (options?.merge) {
// Merge with existing routes (loaded routes take precedence)
const routeMap = new Map<string, IEmailRoute>();
// Add existing routes
for (const route of this.routes) {
routeMap.set(route.name, route);
}
// Override with loaded routes
for (const route of loadedRoutes) {
routeMap.set(route.name, route);
}
this.routes = this.sortRoutesByPriority(Array.from(routeMap.values()));
}
this.clearCache();
this.emit('routesLoaded', loadedRoutes.length);
return loadedRoutes;
} catch (error) {
console.error(`Failed to load routes: ${error.message}`);
throw error;
}
}
/**
* Add a route
* @param route Route to add
* @param persist Whether to persist changes
*/
public async addRoute(route: IEmailRoute, persist?: boolean): Promise<void> {
// Validate route
if (!route.name || !route.match || !route.action) {
throw new Error('Invalid route: missing required fields');
}
// Check if route already exists
const existingIndex = this.routes.findIndex(r => r.name === route.name);
if (existingIndex >= 0) {
throw new Error(`Route '${route.name}' already exists`);
}
// Add route
this.routes.push(route);
this.routes = this.sortRoutesByPriority(this.routes);
this.clearCache();
this.emit('routeAdded', route);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Remove a route by name
* @param name Route name
* @param persist Whether to persist changes
*/
public async removeRoute(name: string, persist?: boolean): Promise<void> {
const index = this.routes.findIndex(r => r.name === name);
if (index < 0) {
throw new Error(`Route '${name}' not found`);
}
const removedRoute = this.routes.splice(index, 1)[0];
this.clearCache();
this.emit('routeRemoved', removedRoute);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Update a route
* @param name Route name
* @param route Updated route data
* @param persist Whether to persist changes
*/
public async updateRoute(name: string, route: IEmailRoute, persist?: boolean): Promise<void> {
// Validate route
if (!route.name || !route.match || !route.action) {
throw new Error('Invalid route: missing required fields');
}
const index = this.routes.findIndex(r => r.name === name);
if (index < 0) {
throw new Error(`Route '${name}' not found`);
}
// Update route
this.routes[index] = route;
this.routes = this.sortRoutesByPriority(this.routes);
this.clearCache();
this.emit('routeUpdated', route);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Get a route by name
* @param name Route name
* @returns Route or undefined
*/
public getRoute(name: string): IEmailRoute | undefined {
return this.routes.find(r => r.name === name);
}
}

400
ts/mail/routing/classes.unified.email.server.ts
View File

@ -16,8 +16,10 @@ import {
type IReputationMonitorConfig
} from '../../deliverability/index.js';
import { EmailRouter } from './classes.email.router.js';
import type { IEmailRoute, IEmailAction, IEmailContext } from './interfaces.js';
import type { IEmailRoute, IEmailAction, IEmailContext, IEmailDomainConfig } from './interfaces.js';
import { Email } from '../core/classes.email.js';
import { DomainRegistry } from './classes.domain.registry.js';
import { DnsValidator } from './classes.dns.validator.js';
import { BounceManager, BounceType, BounceCategory } from '../core/classes.bouncemanager.js';
import { createSmtpServer } from '../delivery/smtpserver/index.js';
import { createPooledSmtpClient } from '../delivery/smtpclient/create-client.js';
@ -46,7 +48,7 @@ export interface IUnifiedEmailServerOptions {
// Base server options
ports: number[];
hostname: string;
domains: string[]; // Domains to handle email for
domains: IEmailDomainConfig[]; // Domain configurations
banner?: string;
debug?: boolean;
useSocketHandler?: boolean; // Use socket-handler mode instead of port listening
@ -79,6 +81,13 @@ export interface IUnifiedEmailServerOptions {
// Email routing rules
routes: IEmailRoute[];
// Global defaults for all domains
defaults?: {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
dkim?: IEmailDomainConfig['dkim'];
rateLimits?: IEmailDomainConfig['rateLimits'];
};
// Outbound settings
outbound?: {
maxConnections?: number;
@ -88,14 +97,7 @@ export interface IUnifiedEmailServerOptions {
defaultFrom?: string;
};
// DKIM settings
dkim?: {
enabled: boolean;
selector?: string;
keySize?: number;
};
// Rate limiting
// Rate limiting (global limits, can be overridden per domain)
rateLimits?: IHierarchicalRateLimits;
// Deliverability options
@ -156,6 +158,7 @@ export class UnifiedEmailServer extends EventEmitter {
private dcRouter: DcRouter;
private options: IUnifiedEmailServerOptions;
private emailRouter: EmailRouter;
private domainRegistry: DomainRegistry;
private servers: any[] = [];
private stats: IServerStats;
@ -186,20 +189,21 @@ export class UnifiedEmailServer extends EventEmitter {
socketTimeout: options.socketTimeout || 60000 // 1 minute
};
// Initialize DKIM creator
this.dkimCreator = new DKIMCreator(paths.keysDir);
// Initialize DKIM creator with storage manager
this.dkimCreator = new DKIMCreator(paths.keysDir, dcRouter.storageManager);
// Initialize IP reputation checker
// Initialize IP reputation checker with storage manager
this.ipReputationChecker = IPReputationChecker.getInstance({
enableLocalCache: true,
enableDNSBL: true,
enableIPInfo: true
});
}, dcRouter.storageManager);
// Initialize bounce manager
// Initialize bounce manager with storage manager
this.bounceManager = new BounceManager({
maxCacheSize: 10000,
cacheTTL: 30 * 24 * 60 * 60 * 1000 // 30 days
cacheTTL: 30 * 24 * 60 * 60 * 1000, // 30 days
storageManager: dcRouter.storageManager
});
// Initialize IP warmup manager
@ -209,14 +213,23 @@ export class UnifiedEmailServer extends EventEmitter {
targetDomains: []
});
// Initialize sender reputation monitor
this.senderReputationMonitor = SenderReputationMonitor.getInstance(options.reputationMonitorConfig || {
enabled: true,
domains: []
});
// Initialize sender reputation monitor with storage manager
this.senderReputationMonitor = SenderReputationMonitor.getInstance(
options.reputationMonitorConfig || {
enabled: true,
domains: []
},
dcRouter.storageManager
);
// Initialize email router with routes
this.emailRouter = new EmailRouter(options.routes || []);
// Initialize domain registry
this.domainRegistry = new DomainRegistry(options.domains, options.defaults);
// Initialize email router with routes and storage manager
this.emailRouter = new EmailRouter(options.routes || [], {
storageManager: dcRouter.storageManager,
persistChanges: true
});
// Initialize rate limiter
this.rateLimiter = new UnifiedRateLimiter(options.rateLimits || {
@ -331,10 +344,40 @@ export class UnifiedEmailServer extends EventEmitter {
await this.deliverySystem.start();
logger.log('info', 'Email delivery system started');
// Set up automatic DKIM if DNS server is available
if (this.dcRouter.dnsServer && this.options.dkim?.enabled) {
await this.setupAutomaticDkim();
logger.log('info', 'Automatic DKIM configuration completed');
// Set up DKIM for all domains
await this.setupDkimForDomains();
logger.log('info', 'DKIM configuration completed for all domains');
// Set up DNS records for internal-dns mode domains
await this.setupInternalDnsRecords();
logger.log('info', 'DNS records created for internal-dns domains');
// Apply per-domain rate limits
this.applyDomainRateLimits();
logger.log('info', 'Per-domain rate limits configured');
// Check and rotate DKIM keys if needed
await this.checkAndRotateDkimKeys();
logger.log('info', 'DKIM key rotation check completed');
// Validate DNS configuration for all domains
const dnsValidator = new DnsValidator(this.dcRouter);
const validationResults = await dnsValidator.validateAllDomains(this.domainRegistry.getAllConfigs());
// Log validation results
let hasErrors = false;
for (const [domain, result] of validationResults) {
if (!result.valid) {
hasErrors = true;
logger.log('error', `DNS validation failed for ${domain}: ${result.errors.join(', ')}`);
}
if (result.warnings.length > 0) {
logger.log('warn', `DNS warnings for ${domain}: ${result.warnings.join(', ')}`);
}
}
if (hasErrors) {
logger.log('warn', 'Some domains have DNS configuration errors. Email handling may not work correctly.');
}
// Skip server creation in socket-handler mode
@ -984,17 +1027,20 @@ export class UnifiedEmailServer extends EventEmitter {
/**
* Set up automatic DKIM configuration with DNS server
* Set up DKIM configuration for all domains
*/
private async setupAutomaticDkim(): Promise<void> {
if (!this.options.domains || this.options.domains.length === 0) {
private async setupDkimForDomains(): Promise<void> {
const domainConfigs = this.domainRegistry.getAllConfigs();
if (domainConfigs.length === 0) {
logger.log('warn', 'No domains configured for DKIM');
return;
}
const selector = this.options.dkim?.selector || 'default';
for (const domain of this.options.domains) {
for (const domainConfig of domainConfigs) {
const domain = domainConfig.domain;
const selector = domainConfig.dkim?.selector || 'default';
try {
// Check if DKIM keys already exist for this domain
let keyPair: { privateKey: string; publicKey: string };
@ -1020,22 +1066,272 @@ export class UnifiedEmailServer extends EventEmitter {
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
// Register DNS handler for this domain's DKIM records
// Register DNS handler for internal-dns mode domains
if (domainConfig.dnsMode === 'internal-dns' && this.dcRouter.dnsServer) {
const ttl = domainConfig.dns?.internal?.ttl || 3600;
this.dcRouter.dnsServer.registerHandler(
`${selector}._domainkey.${domain}`,
['TXT'],
() => ({
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
})
);
logger.log('info', `DKIM DNS handler registered for domain: ${domain} with selector: ${selector}`);
}
} catch (error) {
logger.log('error', `Failed to set up DKIM for domain ${domain}: ${error.message}`);
}
}
}
/**
* Set up DNS records for internal-dns mode domains
* Creates MX, SPF, and DMARC records automatically
*/
private async setupInternalDnsRecords(): Promise<void> {
// Check if DNS server is available
if (!this.dcRouter.dnsServer) {
logger.log('warn', 'DNS server not available, skipping internal DNS record setup');
return;
}
// Get domains configured for internal-dns mode
const internalDnsDomains = this.domainRegistry.getDomainsByMode('internal-dns');
if (internalDnsDomains.length === 0) {
logger.log('info', 'No domains configured for internal-dns mode');
return;
}
logger.log('info', `Setting up DNS records for ${internalDnsDomains.length} internal-dns domains`);
for (const domainConfig of internalDnsDomains) {
const domain = domainConfig.domain;
const ttl = domainConfig.dns?.internal?.ttl || 3600;
const mxPriority = domainConfig.dns?.internal?.mxPriority || 10;
try {
// 1. Register MX record - points to the email domain itself
this.dcRouter.dnsServer.registerHandler(
`${selector}._domainkey.${domain}`,
['TXT'],
domain,
['MX'],
() => ({
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
name: domain,
type: 'MX',
class: 'IN',
ttl: 300,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
ttl: ttl,
data: {
priority: mxPriority,
exchange: domain
}
})
);
logger.log('info', `MX record registered for ${domain} -> ${domain} (priority ${mxPriority})`);
// Store MX record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/mx`,
JSON.stringify({
type: 'MX',
priority: mxPriority,
exchange: domain,
ttl: ttl
})
);
logger.log('info', `DKIM DNS handler registered for domain: ${domain} with selector: ${selector}`);
// 2. Register SPF record - allows the domain to send emails
const spfRecord = `v=spf1 a mx ~all`;
this.dcRouter.dnsServer.registerHandler(
domain,
['TXT'],
() => ({
name: domain,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: spfRecord
})
);
logger.log('info', `SPF record registered for ${domain}: "${spfRecord}"`);
// Store SPF record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/spf`,
JSON.stringify({
type: 'TXT',
data: spfRecord,
ttl: ttl
})
);
// 3. Register DMARC record - policy for handling email authentication
const dmarcRecord = `v=DMARC1; p=none; rua=mailto:dmarc@${domain}`;
this.dcRouter.dnsServer.registerHandler(
`_dmarc.${domain}`,
['TXT'],
() => ({
name: `_dmarc.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: dmarcRecord
})
);
logger.log('info', `DMARC record registered for _dmarc.${domain}: "${dmarcRecord}"`);
// Store DMARC record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/dmarc`,
JSON.stringify({
type: 'TXT',
name: `_dmarc.${domain}`,
data: dmarcRecord,
ttl: ttl
})
);
// 4. Register A record - points to the server IP (if available)
// This is needed for SPF 'a' mechanism to work
// Note: We'll skip A record for now since DnsServer doesn't expose getPublicIP
// This can be added later when the server's public IP is known
logger.log('info', `A record setup skipped for ${domain} - public IP detection not available`);
// Log summary of DNS records created
logger.log('info', `✅ DNS records created for ${domain}:
- MX: ${domain} (priority ${mxPriority})
- SPF: ${spfRecord}
- DMARC: ${dmarcRecord}
- DKIM: ${domainConfig.dkim?.selector || 'default'}._domainkey.${domain}`);
} catch (error) {
logger.log('error', `Failed to set up DKIM for domain ${domain}: ${error.message}`);
logger.log('error', `Failed to set up DNS records for ${domain}: ${error.message}`);
}
}
}
/**
* Apply per-domain rate limits from domain configurations
*/
private applyDomainRateLimits(): void {
const domainConfigs = this.domainRegistry.getAllConfigs();
for (const domainConfig of domainConfigs) {
if (domainConfig.rateLimits) {
const domain = domainConfig.domain;
const rateLimitConfig: any = {};
// Convert domain-specific rate limits to the format expected by UnifiedRateLimiter
if (domainConfig.rateLimits.outbound) {
if (domainConfig.rateLimits.outbound.messagesPerMinute) {
rateLimitConfig.maxMessagesPerMinute = domainConfig.rateLimits.outbound.messagesPerMinute;
}
// Note: messagesPerHour and messagesPerDay would need additional implementation in rate limiter
}
if (domainConfig.rateLimits.inbound) {
if (domainConfig.rateLimits.inbound.messagesPerMinute) {
rateLimitConfig.maxMessagesPerMinute = domainConfig.rateLimits.inbound.messagesPerMinute;
}
if (domainConfig.rateLimits.inbound.connectionsPerIp) {
rateLimitConfig.maxConnectionsPerIP = domainConfig.rateLimits.inbound.connectionsPerIp;
}
if (domainConfig.rateLimits.inbound.recipientsPerMessage) {
rateLimitConfig.maxRecipientsPerMessage = domainConfig.rateLimits.inbound.recipientsPerMessage;
}
}
// Apply the rate limits if we have any
if (Object.keys(rateLimitConfig).length > 0) {
this.rateLimiter.applyDomainLimits(domain, rateLimitConfig);
logger.log('info', `Applied rate limits for domain ${domain}:`, rateLimitConfig);
}
}
}
}
/**
* Check and rotate DKIM keys if needed
*/
private async checkAndRotateDkimKeys(): Promise<void> {
const domainConfigs = this.domainRegistry.getAllConfigs();
for (const domainConfig of domainConfigs) {
const domain = domainConfig.domain;
const selector = domainConfig.dkim?.selector || 'default';
const rotateKeys = domainConfig.dkim?.rotateKeys || false;
const rotationInterval = domainConfig.dkim?.rotationInterval || 90;
const keySize = domainConfig.dkim?.keySize || 2048;
if (!rotateKeys) {
logger.log('debug', `DKIM key rotation disabled for ${domain}`);
continue;
}
try {
// Check if keys need rotation
const needsRotation = await this.dkimCreator.needsRotation(domain, selector, rotationInterval);
if (needsRotation) {
logger.log('info', `DKIM keys need rotation for ${domain} (selector: ${selector})`);
// Rotate the keys
const newSelector = await this.dkimCreator.rotateDkimKeys(domain, selector, keySize);
// Update the domain config with new selector
domainConfig.dkim = {
...domainConfig.dkim,
selector: newSelector
};
// Re-register DNS handler for new selector if internal-dns mode
if (domainConfig.dnsMode === 'internal-dns' && this.dcRouter.dnsServer) {
// Get new public key
const keyPair = await this.dkimCreator.readDKIMKeysForSelector(domain, newSelector);
const publicKeyBase64 = keyPair.publicKey
.replace(/-----BEGIN PUBLIC KEY-----/g, '')
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
const ttl = domainConfig.dns?.internal?.ttl || 3600;
// Register new selector
this.dcRouter.dnsServer.registerHandler(
`${newSelector}._domainkey.${domain}`,
['TXT'],
() => ({
name: `${newSelector}._domainkey.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
})
);
logger.log('info', `DKIM DNS handler registered for new selector: ${newSelector}._domainkey.${domain}`);
// Store the updated public key in storage
await this.dcRouter.storageManager.set(
`/email/dkim/${domain}/public.key`,
keyPair.publicKey
);
}
// Clean up old keys after grace period (async, don't wait)
this.dkimCreator.cleanupOldKeys(domain, 30).catch(error => {
logger.log('warn', `Failed to cleanup old DKIM keys for ${domain}: ${error.message}`);
});
} else {
logger.log('debug', `DKIM keys for ${domain} are up to date`);
}
} catch (error) {
logger.log('error', `Failed to check/rotate DKIM keys for ${domain}: ${error.message}`);
}
}
}
@ -1118,6 +1414,11 @@ export class UnifiedEmailServer extends EventEmitter {
// Update options without restart
this.options = { ...this.options, ...options };
// Update domain registry if domains changed
if (options.domains) {
this.domainRegistry = new DomainRegistry(options.domains, options.defaults || this.options.defaults);
}
// Update email router if routes changed
if (options.routes) {
this.emailRouter.updateRoutes(options.routes);
@ -1140,6 +1441,13 @@ export class UnifiedEmailServer extends EventEmitter {
return { ...this.stats };
}
/**
* Get domain registry
*/
public getDomainRegistry(): DomainRegistry {
return this.domainRegistry;
}
/**
* Update email routes dynamically
*/
@ -1719,4 +2027,12 @@ export class UnifiedEmailServer extends EventEmitter {
receivingDomain
});
}
/**
* Get the rate limiter instance
* @returns The unified rate limiter
*/
public getRateLimiter(): UnifiedRateLimiter {
return this.rateLimiter;
}
}

3
ts/mail/routing/index.ts
View File

@ -2,4 +2,5 @@
export * from './classes.email.router.js';
export * from './classes.unified.email.server.js';
export * from './classes.dnsmanager.js';
export * from './interfaces.js';
export * from './interfaces.js';
export * from './classes.domain.registry.js';

64
ts/mail/routing/interfaces.ts
View File

@ -135,4 +135,68 @@ export interface IEmailContext {
email: Email;
/** The SMTP session */
session: IExtendedSmtpSession;
}
/**
* Email domain configuration
*/
export interface IEmailDomainConfig {
/** Domain name */
domain: string;
/** DNS handling mode */
dnsMode: 'forward' | 'internal-dns' | 'external-dns';
/** DNS configuration based on mode */
dns?: {
/** For 'forward' mode */
forward?: {
/** Skip DNS validation (default: false) */
skipDnsValidation?: boolean;
/** Target server's expected domain */
targetDomain?: string;
};
/** For 'internal-dns' mode */
internal?: {
/** TTL for DNS records in seconds (default: 3600) */
ttl?: number;
/** MX record priority (default: 10) */
mxPriority?: number;
};
/** For 'external-dns' mode */
external?: {
/** Custom DNS servers (default: system DNS) */
servers?: string[];
/** Which records to validate (default: ['MX', 'SPF', 'DKIM', 'DMARC']) */
requiredRecords?: ('MX' | 'SPF' | 'DKIM' | 'DMARC')[];
};
};
/** Per-domain DKIM settings (DKIM always enabled) */
dkim?: {
/** DKIM selector (default: 'default') */
selector?: string;
/** Key size in bits (default: 2048) */
keySize?: number;
/** Automatically rotate keys (default: false) */
rotateKeys?: boolean;
/** Days between key rotations (default: 90) */
rotationInterval?: number;
};
/** Per-domain rate limits */
rateLimits?: {
outbound?: {
messagesPerMinute?: number;
messagesPerHour?: number;
messagesPerDay?: number;
};
inbound?: {
messagesPerMinute?: number;
connectionsPerIp?: number;
recipientsPerMessage?: number;
};
};
}

344
ts/mail/security/classes.dkimcreator.ts
View File

@ -13,11 +13,31 @@ export interface IKeyPaths {
publicKeyPath: string;
}
export interface IDkimKeyMetadata {
domain: string;
selector: string;
createdAt: number;
rotatedAt?: number;
previousSelector?: string;
keySize: number;
}
export class DKIMCreator {
private keysDir: string;
private storageManager?: any; // StorageManager instance
constructor(keysDir = paths.keysDir) {
constructor(keysDir = paths.keysDir, storageManager?: any) {
this.keysDir = keysDir;
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager) {
console.warn(
'⚠️ WARNING: DKIMCreator initialized without StorageManager.\n' +
' DKIM keys will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
}
public async getKeyPathsForDomain(domainArg: string): Promise<IKeyPaths> {
@ -45,19 +65,63 @@ export class DKIMCreator {
await this.handleDKIMKeysForDomain(domain);
}
// Read DKIM keys from disk
// Read DKIM keys - always use storage manager, migrate from filesystem if needed
public async readDKIMKeys(domainArg: string): Promise<{ privateKey: string; publicKey: string }> {
const keyPaths = await this.getKeyPathsForDomain(domainArg);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
// Try to read from storage manager first
if (this.storageManager) {
try {
const [privateKey, publicKey] = await Promise.all([
this.storageManager.get(`/email/dkim/${domainArg}/private.key`),
this.storageManager.get(`/email/dkim/${domainArg}/public.key`)
]);
if (privateKey && publicKey) {
return { privateKey, publicKey };
}
} catch (error) {
// Fall through to migration check
}
// Check if keys exist in filesystem and migrate them to storage manager
const keyPaths = await this.getKeyPathsForDomain(domainArg);
try {
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
// Convert the buffers to strings
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Convert the buffers to strings
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Migrate to storage manager
console.log(`Migrating DKIM keys for ${domainArg} from filesystem to StorageManager`);
await Promise.all([
this.storageManager.set(`/email/dkim/${domainArg}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domainArg}/public.key`, publicKey)
]);
return { privateKey, publicKey };
} catch (error) {
if (error.code === 'ENOENT') {
// Keys don't exist anywhere
throw new Error(`DKIM keys not found for domain ${domainArg}`);
}
throw error;
}
} else {
// No storage manager, use filesystem directly
const keyPaths = await this.getKeyPathsForDomain(domainArg);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
return { privateKey, publicKey };
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
return { privateKey, publicKey };
}
}
// Create a DKIM key pair - changed to public for API access
@ -71,13 +135,27 @@ export class DKIMCreator {
return { privateKey, publicKey };
}
// Store a DKIM key pair to disk - changed to public for API access
// Store a DKIM key pair - uses storage manager if available, else disk
public async storeDKIMKeys(
privateKey: string,
publicKey: string,
privateKeyPath: string,
publicKeyPath: string
): Promise<void> {
// Store in storage manager if available
if (this.storageManager) {
// Extract domain from path (e.g., /path/to/keys/example.com-private.pem -> example.com)
const match = privateKeyPath.match(/\/([^\/]+)-private\.pem$/);
if (match) {
const domain = match[1];
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/public.key`, publicKey)
]);
}
}
// Also store to filesystem for backward compatibility
await Promise.all([writeFile(privateKeyPath, privateKey), writeFile(publicKeyPath, publicKey)]);
}
@ -117,4 +195,246 @@ export class DKIMCreator {
value: dnsRecordValue,
};
}
/**
* Get DKIM key metadata for a domain
*/
private async getKeyMetadata(domain: string, selector: string = 'default'): Promise<IDkimKeyMetadata | null> {
if (!this.storageManager) {
return null;
}
const metadataKey = `/email/dkim/${domain}/${selector}/metadata`;
const metadataStr = await this.storageManager.get(metadataKey);
if (!metadataStr) {
return null;
}
return JSON.parse(metadataStr) as IDkimKeyMetadata;
}
/**
* Save DKIM key metadata
*/
private async saveKeyMetadata(metadata: IDkimKeyMetadata): Promise<void> {
if (!this.storageManager) {
return;
}
const metadataKey = `/email/dkim/${metadata.domain}/${metadata.selector}/metadata`;
await this.storageManager.set(metadataKey, JSON.stringify(metadata));
}
/**
* Check if DKIM keys need rotation
*/
public async needsRotation(domain: string, selector: string = 'default', rotationIntervalDays: number = 90): Promise<boolean> {
const metadata = await this.getKeyMetadata(domain, selector);
if (!metadata) {
// No metadata means old keys, should rotate
return true;
}
const now = Date.now();
const keyAgeMs = now - metadata.createdAt;
const keyAgeDays = keyAgeMs / (1000 * 60 * 60 * 24);
return keyAgeDays >= rotationIntervalDays;
}
/**
* Rotate DKIM keys for a domain
*/
public async rotateDkimKeys(domain: string, currentSelector: string = 'default', keySize: number = 2048): Promise<string> {
console.log(`Rotating DKIM keys for ${domain}...`);
// Generate new selector based on date
const now = new Date();
const newSelector = `key${now.getFullYear()}${String(now.getMonth() + 1).padStart(2, '0')}`;
// Create new keys with custom key size
const { privateKey, publicKey } = await generateKeyPair('rsa', {
modulusLength: keySize,
publicKeyEncoding: { type: 'spki', format: 'pem' },
privateKeyEncoding: { type: 'pkcs1', format: 'pem' },
});
// Store new keys with new selector
const newKeyPaths = await this.getKeyPathsForSelector(domain, newSelector);
// Store in storage manager if available
if (this.storageManager) {
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/${newSelector}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/${newSelector}/public.key`, publicKey)
]);
}
// Also store to filesystem
await this.storeDKIMKeys(
privateKey,
publicKey,
newKeyPaths.privateKeyPath,
newKeyPaths.publicKeyPath
);
// Save metadata for new keys
const metadata: IDkimKeyMetadata = {
domain,
selector: newSelector,
createdAt: Date.now(),
previousSelector: currentSelector,
keySize
};
await this.saveKeyMetadata(metadata);
// Update metadata for old keys
const oldMetadata = await this.getKeyMetadata(domain, currentSelector);
if (oldMetadata) {
oldMetadata.rotatedAt = Date.now();
await this.saveKeyMetadata(oldMetadata);
}
console.log(`DKIM keys rotated for ${domain}. New selector: ${newSelector}`);
return newSelector;
}
/**
* Get key paths for a specific selector
*/
public async getKeyPathsForSelector(domain: string, selector: string): Promise<IKeyPaths> {
return {
privateKeyPath: plugins.path.join(this.keysDir, `${domain}-${selector}-private.pem`),
publicKeyPath: plugins.path.join(this.keysDir, `${domain}-${selector}-public.pem`),
};
}
/**
* Read DKIM keys for a specific selector
*/
public async readDKIMKeysForSelector(domain: string, selector: string): Promise<{ privateKey: string; publicKey: string }> {
// Try to read from storage manager first
if (this.storageManager) {
try {
const [privateKey, publicKey] = await Promise.all([
this.storageManager.get(`/email/dkim/${domain}/${selector}/private.key`),
this.storageManager.get(`/email/dkim/${domain}/${selector}/public.key`)
]);
if (privateKey && publicKey) {
return { privateKey, publicKey };
}
} catch (error) {
// Fall through to migration check
}
// Check if keys exist in filesystem and migrate them to storage manager
const keyPaths = await this.getKeyPathsForSelector(domain, selector);
try {
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Migrate to storage manager
console.log(`Migrating DKIM keys for ${domain}/${selector} from filesystem to StorageManager`);
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/${selector}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/${selector}/public.key`, publicKey)
]);
return { privateKey, publicKey };
} catch (error) {
if (error.code === 'ENOENT') {
throw new Error(`DKIM keys not found for domain ${domain} with selector ${selector}`);
}
throw error;
}
} else {
// No storage manager, use filesystem directly
const keyPaths = await this.getKeyPathsForSelector(domain, selector);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
return { privateKey, publicKey };
}
}
/**
* Get DNS record for a specific selector
*/
public async getDNSRecordForSelector(domain: string, selector: string): Promise<plugins.tsclass.network.IDnsRecord> {
const keys = await this.readDKIMKeysForSelector(domain, selector);
// Remove the PEM header and footer and newlines
const pemHeader = '-----BEGIN PUBLIC KEY-----';
const pemFooter = '-----END PUBLIC KEY-----';
const keyContents = keys.publicKey
.replace(pemHeader, '')
.replace(pemFooter, '')
.replace(/\n/g, '');
// Generate the DKIM DNS TXT record
const dnsRecordValue = `v=DKIM1; h=sha256; k=rsa; p=${keyContents}`;
return {
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
dnsSecEnabled: null,
value: dnsRecordValue,
};
}
/**
* Clean up old DKIM keys after grace period
*/
public async cleanupOldKeys(domain: string, gracePeriodDays: number = 30): Promise<void> {
if (!this.storageManager) {
return;
}
// List all selectors for the domain
const metadataKeys = await this.storageManager.list(`/email/dkim/${domain}/`);
for (const key of metadataKeys) {
if (key.endsWith('/metadata')) {
const metadataStr = await this.storageManager.get(key);
if (metadataStr) {
const metadata = JSON.parse(metadataStr) as IDkimKeyMetadata;
// Check if key is rotated and past grace period
if (metadata.rotatedAt) {
const gracePeriodMs = gracePeriodDays * 24 * 60 * 60 * 1000;
const now = Date.now();
if (now - metadata.rotatedAt > gracePeriodMs) {
console.log(`Cleaning up old DKIM keys for ${domain} selector ${metadata.selector}`);
// Delete key files
const keyPaths = await this.getKeyPathsForSelector(domain, metadata.selector);
try {
await plugins.fs.promises.unlink(keyPaths.privateKeyPath);
await plugins.fs.promises.unlink(keyPaths.publicKeyPath);
} catch (error) {
console.warn(`Failed to delete old key files: ${error.message}`);
}
// Delete metadata
await this.storageManager.delete(key);
}
}
}
}
}
}
}

159
ts/security/classes.ipreputationchecker.ts
View File

@ -64,6 +64,7 @@ export class IPReputationChecker {
private static instance: IPReputationChecker;
private reputationCache: LRUCache<string, IReputationResult>;
private options: Required<IIPReputationOptions>;
private storageManager?: any; // StorageManager instance
// Default DNSBL servers
private static readonly DEFAULT_DNSBL_SERVERS = [
@ -95,14 +96,26 @@ export class IPReputationChecker {
/**
* Constructor for IPReputationChecker
* @param options Configuration options
* @param storageManager Optional StorageManager instance for persistence
*/
constructor(options: IIPReputationOptions = {}) {
constructor(options: IIPReputationOptions = {}, storageManager?: any) {
// Merge with default options
this.options = {
...IPReputationChecker.DEFAULT_OPTIONS,
...options
};
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager && this.options.enableLocalCache) {
logger.log('warn',
'⚠️ WARNING: IPReputationChecker initialized without StorageManager.\n' +
' IP reputation cache will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Initialize reputation cache
this.reputationCache = new LRUCache<string, IReputationResult>({
max: this.options.maxCacheSize,
@ -111,18 +124,22 @@ export class IPReputationChecker {
// Load cache from disk if enabled
if (this.options.enableLocalCache) {
this.loadCache();
// Fire and forget the load operation
this.loadCache().catch(error => {
logger.log('error', `Failed to load IP reputation cache during initialization: ${error.message}`);
});
}
}
/**
* Get the singleton instance of the checker
* @param options Configuration options
* @param storageManager Optional StorageManager instance for persistence
* @returns Singleton instance
*/
public static getInstance(options: IIPReputationOptions = {}): IPReputationChecker {
public static getInstance(options: IIPReputationOptions = {}, storageManager?: any): IPReputationChecker {
if (!IPReputationChecker.instance) {
IPReputationChecker.instance = new IPReputationChecker(options);
IPReputationChecker.instance = new IPReputationChecker(options, storageManager);
}
return IPReputationChecker.instance;
}
@ -198,7 +215,10 @@ export class IPReputationChecker {
// Save cache if enabled
if (this.options.enableLocalCache) {
this.saveCache();
// Fire and forget the save operation
this.saveCache().catch(error => {
logger.log('error', `Failed to save IP reputation cache: ${error.message}`);
});
}
// Log the reputation check
@ -428,9 +448,9 @@ export class IPReputationChecker {
}
/**
* Save cache to disk
* Save cache to disk or storage manager
*/
private saveCache(): void {
private async saveCache(): Promise<void> {
try {
// Convert cache entries to serializable array
const entries = Array.from(this.reputationCache.entries()).map(([ip, data]) => ({
@ -443,52 +463,94 @@ export class IPReputationChecker {
return;
}
// Ensure directory exists
const cacheDir = plugins.path.join(paths.dataDir, 'security');
plugins.smartfile.fs.ensureDirSync(cacheDir);
const cacheData = JSON.stringify(entries);
// Save to file
const cacheFile = plugins.path.join(cacheDir, 'ip_reputation_cache.json');
plugins.smartfile.memory.toFsSync(
JSON.stringify(entries),
cacheFile
);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to disk`);
// Save to storage manager if available
if (this.storageManager) {
await this.storageManager.set('/security/ip-reputation-cache.json', cacheData);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to StorageManager`);
} else {
// Fall back to filesystem
const cacheDir = plugins.path.join(paths.dataDir, 'security');
plugins.smartfile.fs.ensureDirSync(cacheDir);
const cacheFile = plugins.path.join(cacheDir, 'ip_reputation_cache.json');
plugins.smartfile.memory.toFsSync(cacheData, cacheFile);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to disk`);
}
} catch (error) {
logger.log('error', `Failed to save IP reputation cache: ${error.message}`);
}
}
/**
* Load cache from disk
* Load cache from disk or storage manager
*/
private loadCache(): void {
private async loadCache(): Promise<void> {
try {
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
let cacheData: string | null = null;
let fromFilesystem = false;
// Check if file exists
if (!plugins.fs.existsSync(cacheFile)) {
return;
// Try to load from storage manager first
if (this.storageManager) {
try {
cacheData = await this.storageManager.get('/security/ip-reputation-cache.json');
if (!cacheData) {
// Check if data exists in filesystem and migrate it
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
if (plugins.fs.existsSync(cacheFile)) {
logger.log('info', 'Migrating IP reputation cache from filesystem to StorageManager');
cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
fromFilesystem = true;
// Migrate to storage manager
await this.storageManager.set('/security/ip-reputation-cache.json', cacheData);
logger.log('info', 'IP reputation cache migrated to StorageManager successfully');
// Optionally delete the old file after successful migration
try {
plugins.fs.unlinkSync(cacheFile);
logger.log('info', 'Old cache file removed after migration');
} catch (deleteError) {
logger.log('warn', `Could not delete old cache file: ${deleteError.message}`);
}
}
}
} catch (error) {
logger.log('error', `Error loading from StorageManager: ${error.message}`);
}
} else {
// No storage manager, load from filesystem
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
if (plugins.fs.existsSync(cacheFile)) {
cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
fromFilesystem = true;
}
}
// Read and parse cache
const cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
const entries = JSON.parse(cacheData);
// Validate and filter entries
const now = Date.now();
const validEntries = entries.filter(entry => {
const age = now - entry.data.timestamp;
return age < this.options.cacheTTL; // Only load entries that haven't expired
});
// Restore cache
for (const entry of validEntries) {
this.reputationCache.set(entry.ip, entry.data);
// Parse and restore cache if data was found
if (cacheData) {
const entries = JSON.parse(cacheData);
// Validate and filter entries
const now = Date.now();
const validEntries = entries.filter(entry => {
const age = now - entry.data.timestamp;
return age < this.options.cacheTTL; // Only load entries that haven't expired
});
// Restore cache
for (const entry of validEntries) {
this.reputationCache.set(entry.ip, entry.data);
}
const source = fromFilesystem ? 'disk' : 'StorageManager';
logger.log('info', `Loaded ${validEntries.length} IP reputation cache entries from ${source}`);
}
logger.log('info', `Loaded ${validEntries.length} IP reputation cache entries from disk`);
} catch (error) {
logger.log('error', `Failed to load IP reputation cache: ${error.message}`);
}
@ -510,4 +572,21 @@ export class IPReputationChecker {
return 'trusted';
}
}
/**
* Update the storage manager after instantiation
* This is useful when the storage manager is not available at construction time
* @param storageManager The StorageManager instance to use
*/
public updateStorageManager(storageManager: any): void {
this.storageManager = storageManager;
logger.log('info', 'IPReputationChecker storage manager updated');
// If cache is enabled and we have entries, save them to the new storage manager
if (this.options.enableLocalCache && this.reputationCache.size > 0) {
this.saveCache().catch(error => {
logger.log('error', `Failed to save cache to new storage manager: ${error.message}`);
});
}
}
}

397
ts/storage/classes.storagemanager.ts Normal file
View File

@ -0,0 +1,397 @@
import * as plugins from '../plugins.js';
import * as paths from '../paths.js';
import { logger } from '../logger.js';
// Promisify filesystem operations
const readFile = plugins.util.promisify(plugins.fs.readFile);
const writeFile = plugins.util.promisify(plugins.fs.writeFile);
const unlink = plugins.util.promisify(plugins.fs.unlink);
const rename = plugins.util.promisify(plugins.fs.rename);
const readdir = plugins.util.promisify(plugins.fs.readdir);
const stat = plugins.util.promisify(plugins.fs.stat);
/**
* Storage configuration interface
*/
export interface IStorageConfig {
/** Filesystem path for storage */
fsPath?: string;
/** Custom read function */
readFunction?: (key: string) => Promise<string>;
/** Custom write function */
writeFunction?: (key: string, value: string) => Promise<void>;
}
/**
* Storage backend type
*/
export type StorageBackend = 'filesystem' | 'custom' | 'memory';
/**
* Central storage manager for DcRouter
* Provides unified key-value storage with multiple backend support
*/
export class StorageManager {
private backend: StorageBackend;
private memoryStore: Map<string, string> = new Map();
private config: IStorageConfig;
private fsBasePath?: string;
constructor(config?: IStorageConfig) {
this.config = config || {};
// Check if both fsPath and custom functions are provided
if (config?.fsPath && (config?.readFunction || config?.writeFunction)) {
console.warn(
'⚠️ WARNING: Both fsPath and custom read/write functions are configured.\n' +
' Using custom read/write functions. fsPath will be ignored.'
);
}
// Determine backend based on configuration
if (config?.readFunction && config?.writeFunction) {
this.backend = 'custom';
} else if (config?.fsPath) {
// Set up internal read/write functions for filesystem
this.backend = 'custom'; // Use custom backend with internal functions
this.fsBasePath = plugins.path.resolve(config.fsPath);
this.ensureDirectory(this.fsBasePath);
// Set up internal filesystem read/write functions
this.config.readFunction = async (key: string) => {
return this.fsRead(key);
};
this.config.writeFunction = async (key: string, value: string) => {
await this.fsWrite(key, value);
};
} else {
this.backend = 'memory';
this.showMemoryWarning();
}
logger.log('info', `StorageManager initialized with ${this.backend} backend`);
}
/**
* Show warning when using memory backend
*/
private showMemoryWarning(): void {
console.warn(
'⚠️ WARNING: StorageManager is using in-memory storage.\n' +
' Data will be lost when the process restarts.\n' +
' Configure storage.fsPath or storage functions for persistence.'
);
}
/**
* Ensure directory exists for filesystem backend
*/
private async ensureDirectory(dirPath: string): Promise<void> {
try {
await plugins.smartfile.fs.ensureDir(dirPath);
} catch (error) {
logger.log('error', `Failed to create storage directory: ${error.message}`);
throw error;
}
}
/**
* Validate and sanitize storage key
*/
private validateKey(key: string): string {
if (!key || typeof key !== 'string') {
throw new Error('Storage key must be a non-empty string');
}
// Ensure key starts with /
if (!key.startsWith('/')) {
key = '/' + key;
}
// Remove any dangerous path elements
key = key.replace(/\.\./g, '').replace(/\/+/g, '/');
return key;
}
/**
* Convert key to filesystem path
*/
private keyToPath(key: string): string {
if (!this.fsBasePath) {
throw new Error('Filesystem base path not configured');
}
// Remove leading slash and convert to path
const relativePath = key.substring(1);
return plugins.path.join(this.fsBasePath, relativePath);
}
/**
* Internal filesystem read function
*/
private async fsRead(key: string): Promise<string> {
const filePath = this.keyToPath(key);
try {
const content = await readFile(filePath, 'utf8');
return content;
} catch (error) {
if (error.code === 'ENOENT') {
return null;
}
throw error;
}
}
/**
* Internal filesystem write function
*/
private async fsWrite(key: string, value: string): Promise<void> {
const filePath = this.keyToPath(key);
const dir = plugins.path.dirname(filePath);
// Ensure directory exists
await plugins.smartfile.fs.ensureDir(dir);
// Write atomically with temp file
const tempPath = `${filePath}.tmp`;
await writeFile(tempPath, value, 'utf8');
await rename(tempPath, filePath);
}
/**
* Get value by key
*/
async get(key: string): Promise<string | null> {
key = this.validateKey(key);
try {
switch (this.backend) {
case 'custom': {
if (!this.config.readFunction) {
throw new Error('Read function not configured');
}
try {
return await this.config.readFunction(key);
} catch (error) {
// Assume null if read fails (key doesn't exist)
return null;
}
}
case 'memory': {
return this.memoryStore.get(key) || null;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage get error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* Set value by key
*/
async set(key: string, value: string): Promise<void> {
key = this.validateKey(key);
if (typeof value !== 'string') {
throw new Error('Storage value must be a string');
}
try {
switch (this.backend) {
case 'filesystem': {
const filePath = this.keyToPath(key);
const dirPath = plugins.path.dirname(filePath);
// Ensure directory exists
await plugins.smartfile.fs.ensureDir(dirPath);
// Write atomically
const tempPath = filePath + '.tmp';
await writeFile(tempPath, value, 'utf8');
await rename(tempPath, filePath);
break;
}
case 'custom': {
if (!this.config.writeFunction) {
throw new Error('Write function not configured');
}
await this.config.writeFunction(key, value);
break;
}
case 'memory': {
this.memoryStore.set(key, value);
break;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage set error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* Delete value by key
*/
async delete(key: string): Promise<void> {
key = this.validateKey(key);
try {
switch (this.backend) {
case 'filesystem': {
const filePath = this.keyToPath(key);
try {
await unlink(filePath);
} catch (error) {
if (error.code !== 'ENOENT') {
throw error;
}
}
break;
}
case 'custom': {
// Try to delete by setting empty value
if (this.config.writeFunction) {
await this.config.writeFunction(key, '');
}
break;
}
case 'memory': {
this.memoryStore.delete(key);
break;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage delete error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* List keys by prefix
*/
async list(prefix?: string): Promise<string[]> {
prefix = prefix ? this.validateKey(prefix) : '/';
try {
switch (this.backend) {
case 'filesystem': {
const basePath = this.keyToPath(prefix);
const keys: string[] = [];
const walkDir = async (dir: string, baseDir: string): Promise<void> => {
try {
const entries = await readdir(dir, { withFileTypes: true });
for (const entry of entries) {
const fullPath = plugins.path.join(dir, entry.name);
if (entry.isDirectory()) {
await walkDir(fullPath, baseDir);
} else if (entry.isFile()) {
// Convert path back to key
const relativePath = plugins.path.relative(this.fsBasePath!, fullPath);
const key = '/' + relativePath.replace(/\\/g, '/');
if (key.startsWith(prefix)) {
keys.push(key);
}
}
}
} catch (error) {
if (error.code !== 'ENOENT') {
throw error;
}
}
};
await walkDir(basePath, basePath);
return keys.sort();
}
case 'custom': {
// Custom backends need to implement their own listing
logger.log('warn', 'List operation not supported for custom backend');
return [];
}
case 'memory': {
const keys: string[] = [];
for (const key of this.memoryStore.keys()) {
if (key.startsWith(prefix)) {
keys.push(key);
}
}
return keys.sort();
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage list error for prefix ${prefix}: ${error.message}`);
throw error;
}
}
/**
* Check if key exists
*/
async exists(key: string): Promise<boolean> {
key = this.validateKey(key);
try {
const value = await this.get(key);
return value !== null;
} catch (error) {
return false;
}
}
/**
* Get storage backend type
*/
getBackend(): StorageBackend {
return this.backend;
}
/**
* JSON helper: Get and parse JSON value
*/
async getJSON<T = any>(key: string): Promise<T | null> {
const value = await this.get(key);
if (value === null) {
return null;
}
try {
return JSON.parse(value) as T;
} catch (error) {
logger.log('error', `Failed to parse JSON for key ${key}: ${error.message}`);
throw error;
}
}
/**
* JSON helper: Set value as JSON
*/
async setJSON(key: string, value: any): Promise<void> {
const jsonString = JSON.stringify(value, null, 2);
await this.set(key, jsonString);
}
}

2
ts/storage/index.ts Normal file
View File

@ -0,0 +1,2 @@
// Storage module exports
export * from './classes.storagemanager.js';