serve.zone/dcrouter
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(integration): components now play nicer with each other

Browse Source
This commit is contained in:
Philipp Kunz 2025-05-30 05:30:06 +00:00
parent 2c244c4a9a
commit 40db395591
19 changed files with 2849 additions and 264 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

96
readme.hints.md
View File

@ -741,4 +741,98 @@ The `ts/config/` directory cleanup has been completed. Removed ~500+ lines of un
- Config directory now contains only 2 files (validator.ts, index.ts)
- SMS configuration is self-contained in SMS module
- All deprecated email configuration removed
- Build passes successfully
- Build passes successfully
## Per-Domain Rate Limiting (2025-05-29) - COMPLETED
### Overview
Per-domain rate limiting has been implemented in the UnifiedRateLimiter. Each email domain can have its own rate limits that override global limits.
### Implementation Details
1. **UnifiedRateLimiter Enhanced:**
- Added `domains` property to IHierarchicalRateLimits
- Added `domainCounters` Map for tracking domain-specific counters
- Added `checkDomainMessageLimit()` method
- Added `applyDomainLimits()`, `removeDomainLimits()`, `getDomainLimits()` methods
2. **Domain Rate Limit Configuration:**
```typescript
interface IEmailDomainConfig {
domain: string;
rateLimits?: {
outbound?: {
messagesPerMinute?: number;
messagesPerHour?: number; // Note: Hour/day limits need additional implementation
messagesPerDay?: number;
};
inbound?: {
messagesPerMinute?: number;
connectionsPerIp?: number;
recipientsPerMessage?: number;
};
};
}
```
3. **Automatic Application:**
- UnifiedEmailServer applies domain rate limits during startup
- `applyDomainRateLimits()` method converts domain config to rate limiter format
- Domain limits override pattern and global limits
4. **Usage Pattern:**
```typescript
// Domain configuration with rate limits
{
domain: 'high-volume.com',
dnsMode: 'internal-dns',
rateLimits: {
outbound: {
messagesPerMinute: 200 // Higher than global limit
},
inbound: {
recipientsPerMessage: 100 // Higher recipient limit
}
}
}
```
5. **Rate Limit Precedence:**
- Domain-specific limits (highest priority)
- Pattern-specific limits
- Global limits (lowest priority)
### Integration Status
- ✅ Rate limiter supports per-domain limits
- ✅ UnifiedEmailServer applies domain limits on startup
- ✅ Domain limits properly override global/pattern limits
- ✅ SMTP server handlers now enforce rate limits (COMPLETED 2025-05-29)
- ⚠️ Hour/day limits need additional implementation in rate limiter
### SMTP Handler Integration (2025-05-29) - COMPLETED
Rate limiting is now fully integrated into SMTP server handlers:
1. **UnifiedEmailServer Enhancement:**
- Added `getRateLimiter()` method to provide access to the rate limiter
2. **ConnectionManager Integration:**
- Replaced custom rate limiting with UnifiedRateLimiter
- Now uses `rateLimiter.recordConnection(ip)` for all connection checks
- Maintains local IP tracking for resource cleanup only
3. **CommandHandler Integration:**
- `handleMailFrom()`: Checks message rate limits with domain context
- `handleRcptTo()`: Enforces recipient limits per message
- `handleAuth*()`: Records authentication failures and blocks after threshold
- Error handling: Records syntax/command errors and blocks after threshold
4. **SMTP Response Codes:**
- `421`: Temporary rate limit (client should retry later)
- `451`: Temporary recipient rejection
- `421 Too many errors`: IP blocked due to excessive errors
- `421 Too many authentication failures`: IP blocked due to auth failures
### Next Steps
The only remaining item is implementing hour/day rate limits in the UnifiedRateLimiter, which would require:
1. Additional counters for hourly and daily windows
2. Separate tracking for these longer time periods
3. Cleanup logic for expired hourly/daily counters

111
readme.plan.md
View File

@ -63,13 +63,11 @@ interface IDcRouterOptions {
emailConfig?: IUnifiedEmailServerOptions;
}
// Updated UnifiedEmailServerOptions with backward-compatible domains
// Updated UnifiedEmailServerOptions
interface IUnifiedEmailServerOptions {
ports: number[];
hostname: string;
// Backward compatible - can be strings or full config objects
domains: (string | IEmailDomainConfig)[];
domains: IEmailDomainConfig[];
// Pattern-based routing rules (evaluated after domain matching)
routes: IEmailRoute[];
@ -249,83 +247,83 @@ When a domain is configured with `dnsMode: 'forward'`:
### Phase 1: Storage Manager Implementation
#### 1.1 Create Storage Manager Core
- [ ] Create `ts/storage/classes.storagemanager.ts`
- [ ] Implement base StorageManager class
- [ ] Add storage backend detection logic
- [ ] Implement key namespacing for different components
- [x] Create `ts/storage/classes.storagemanager.ts`
- [x] Implement base StorageManager class
- [x] Add storage backend detection logic
- [x] Implement key namespacing for different components
#### 1.2 Implement Storage Backends
- [ ] Filesystem backend using `fsPath`
- [ ] Custom function backend using provided read/write functions
- [ ] Memory backend with Map<string, string>
- [ ] Add console warning for memory backend
- [x] Filesystem backend using `fsPath`
- [x] Custom function backend using provided read/write functions
- [x] Memory backend with Map<string, string>
- [x] Add console warning for memory backend
#### 1.3 Storage Interface Methods
- [ ] Implement get/set/delete/list/exists methods
- [ ] Add JSON serialization helpers
- [ ] Add atomic write support for filesystem
- [ ] Add key validation and sanitization
- [x] Implement get/set/delete/list/exists methods
- [x] Add JSON serialization helpers
- [x] Add atomic write support for filesystem
- [x] Add key validation and sanitization
#### 1.4 Integration Points
- [ ] Add StorageManager instance to DcRouter
- [ ] Pass storage to components that need it
- [x] Add StorageManager instance to DcRouter
- [x] Pass storage to components that need it
- [ ] Update component constructors to accept storage
### Phase 2: Email DNS Configuration Implementation
#### 2.1 Update Email Configuration
- [ ] Update IUnifiedEmailServerOptions to support string | IEmailDomainConfig
- [ ] Implement backward compatibility for string domains
- [ ] Add domain configuration validation
- [ ] Implement infrastructure vs routing separation
- [x] Update IUnifiedEmailServerOptions to support IEmailDomainConfig
- [x] Add domain configuration validation
- [x] Implement infrastructure vs routing separation
#### 2.2 Domain Configuration Processing
- [ ] Parse domains array (string vs IEmailDomainConfig)
- [ ] Apply global defaults to domain configs
- [ ] Validate each domain's DNS mode configuration
- [ ] Create domain registry for quick lookups
- [x] Apply global defaults to domain configs
- [x] Validate each domain's DNS mode configuration
- [x] Create domain registry for quick lookups
#### 2.3 DNS Mode Implementations
- [ ] **Forward Mode**: Skip DNS handling, validate target reachability
- [ ] **Internal DNS Mode**:
- [ ] Validate dnsDomain is set in DcRouter config
- [ ] Check NS delegation exists (query for NS records)
- [ ] Show instructions if NS delegation is missing
- [ ] Log success if NS delegation is properly configured
- [ ] Automatically create MX, SPF, DKIM, DMARC records in internal DNS
- [ ] Apply TTL (default: 3600) and MX priority (default: 10)
- [ ] Store records via StorageManager
- [ ] Register domains with DnsServer
- [ ] **External DNS Mode**:
- [ ] Use standard DNS resolution (or custom servers if specified)
- [ ] Always validate required records (default: MX, SPF, DKIM, DMARC)
- [ ] Always show setup instructions if records are missing
- [x] **Forward Mode**: Skip DNS handling, validate target reachability
- [x] **Internal DNS Mode**:
- [x] Validate dnsDomain is set in DcRouter config
- [x] Check NS delegation exists (query for NS records)
- [x] Show instructions if NS delegation is missing
- [x] Log success if NS delegation is properly configured
- [x] Automatically create MX, SPF, DKIM, DMARC records in internal DNS
- [x] Apply TTL (default: 3600) and MX priority (default: 10)
- [x] Store records via StorageManager
- [x] Register domains with DnsServer
- [x] **External DNS Mode**:
- [x] Use standard DNS resolution (or custom servers if specified)
- [x] Always validate required records (default: MX, SPF, DKIM, DMARC)
- [x] Always show setup instructions if records are missing
- [ ] Cache DNS query results in storage
#### 2.4 Per-Domain Features
- [ ] Implement per-domain DKIM key management
- [ ] Apply per-domain rate limits
- [x] Implement per-domain DKIM key management
- [x] Apply per-domain rate limits
- [x] Integrate rate limiting into SMTP server handlers
- [ ] Handle per-domain email processing rules
- [ ] Automatic DKIM key rotation based on domain config
- [x] Automatic DKIM key rotation based on domain config
### Phase 3: Storage Usage Implementation
#### 3.1 Email Component Storage
- [ ] DKIM keys storage
- [ ] Email routing rules storage
- [ ] Bounce/complaint tracking
- [ ] Reputation data persistence
- [x] DKIM keys storage (DKIMCreator updated with StorageManager)
- [x] Email routing rules storage (EmailRouter updated with persistence support)
- [x] Bounce/complaint tracking (BounceManager updated with StorageManager)
- [x] Reputation data persistence (SenderReputationMonitor and IPReputationChecker updated)
#### 3.2 DNS Component Storage
- [ ] DNS records storage
- [ ] DNSSEC keys storage
- [ ] Zone data persistence
- [ ] Cache storage
- [N/A] DNS records storage (handled by smartdns library internally)
- [N/A] DNSSEC keys storage (handled by smartdns library internally)
- [N/A] Zone data persistence (handled by smartdns library internally)
- [N/A] Cache storage (handled by smartdns library internally)
#### 3.3 Certificate Storage
- [ ] Let's Encrypt certificates
- [ ] Certificate renewal data
- [ ] ACME account keys
- [N/A] Let's Encrypt certificates (handled by SmartProxy library)
- [N/A] Certificate renewal data (handled by SmartProxy library)
- [N/A] ACME account keys (handled by SmartProxy library)
### Phase 4: Testing
@ -379,12 +377,7 @@ console.warn(
emailConfig: {
ports: [25, 587, 465],
hostname: 'mail.myservice.com',
// Mix of simple strings and full configs (backward compatible)
domains: [
// Simple domain (uses all defaults including DKIM)
'simple.com',
{
// Forward-only domain (no local DNS needed)
domain: 'forwarded.com',
@ -499,7 +492,6 @@ console.warn(
### Email DNS Benefits
- **Unified Configuration**: Infrastructure and routing cleanly separated
- **Backward Compatible**: Existing string domain configs continue to work
- **Flexible DNS Modes**: Choose per-domain how DNS is handled
- **External Mode**: Works with existing DNS infrastructure
- **Internal Mode**: Self-contained email system with automatic record creation
@ -522,7 +514,6 @@ console.warn(
- Keys are hierarchical (path-like)
- Values are strings (JSON for complex data)
- Internal DNS mode requires `dnsDomain` to be set in DcRouter
- Domain configuration is backward compatible (strings still work)
- Clean separation: domains = infrastructure, routes = handling
- Domain config only defines: which domains, DNS mode, DKIM settings, rate limits
- DKIM is always enabled for all domains (use dkim object to override defaults)

236
test/test.rate-limiting-integration.ts Normal file
View File

@ -0,0 +1,236 @@
import { expect, tap } from '@git.zone/tstest/tapbundle';
import * as plugins from './helpers/server.loader.js';
import { createTestSmtpClient } from './helpers/smtp.client.js';
import { SmtpClient } from '../ts/mail/delivery/smtpclient/smtp-client.js';
const TEST_PORT = 2525;
// Test email configuration with rate limits
const testEmailConfig = {
ports: [TEST_PORT],
hostname: 'localhost',
domains: [
{
domain: 'test.local',
dnsMode: 'forward' as const,
rateLimits: {
inbound: {
messagesPerMinute: 3, // Very low limit for testing
recipientsPerMessage: 2,
connectionsPerIp: 5
}
}
}
],
routes: [
{
name: 'test-route',
match: { recipients: '*@test.local' },
action: { type: 'process' as const, process: { scan: false, queue: 'normal' } }
}
],
rateLimits: {
global: {
maxMessagesPerMinute: 10,
maxConnectionsPerIP: 10,
maxErrorsPerIP: 3,
maxAuthFailuresPerIP: 2,
blockDuration: 5000 // 5 seconds for testing
}
}
};
tap.test('prepare server with rate limiting', async () => {
await plugins.startTestServer(testEmailConfig);
// Give server time to start
await new Promise(resolve => setTimeout(resolve, 1000));
});
tap.test('should enforce connection rate limits', async (tools) => {
const done = tools.defer();
const clients: SmtpClient[] = [];
try {
// Try to create many connections quickly
for (let i = 0; i < 12; i++) {
const client = createTestSmtpClient();
clients.push(client);
// Connection should fail after limit is exceeded
const verified = await client.verify().catch(() => false);
if (i < 10) {
// First 10 should succeed (global limit)
expect(verified).toBeTrue();
} else {
// After 10, should be rate limited
expect(verified).toBeFalse();
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
// Clean up connections
for (const client of clients) {
await client.close().catch(() => {});
}
}
});
tap.test('should enforce message rate limits per domain', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Send messages rapidly to test domain-specific rate limit
for (let i = 0; i < 5; i++) {
const email = {
from: `sender${i}@example.com`,
to: 'recipient@test.local',
subject: `Test ${i}`,
text: 'Test message'
};
const result = await client.sendMail(email).catch(err => err);
if (i < 3) {
// First 3 should succeed (domain limit is 3 per minute)
expect(result.accepted).toBeDefined();
expect(result.accepted.length).toEqual(1);
} else {
// After 3, should be rate limited
expect(result.code).toEqual('EENVELOPE');
expect(result.response).toContain('try again later');
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close();
}
});
tap.test('should enforce recipient limits', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Try to send to many recipients (domain limit is 2 per message)
const email = {
from: 'sender@example.com',
to: ['user1@test.local', 'user2@test.local', 'user3@test.local'],
subject: 'Test with multiple recipients',
text: 'Test message'
};
const result = await client.sendMail(email).catch(err => err);
// Should fail due to recipient limit
expect(result.code).toEqual('EENVELOPE');
expect(result.response).toContain('try again later');
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close();
}
});
tap.test('should enforce error rate limits', async (tools) => {
const done = tools.defer();
const client = createTestSmtpClient();
try {
// Send multiple invalid commands to trigger error rate limit
const socket = (client as any).socket;
// Wait for connection
await new Promise(resolve => setTimeout(resolve, 100));
// Send invalid commands
for (let i = 0; i < 5; i++) {
socket.write('INVALID_COMMAND\r\n');
// Wait for response
await new Promise(resolve => {
socket.once('data', resolve);
});
}
// After 3 errors, connection should be blocked
const lastResponse = await new Promise<string>(resolve => {
socket.once('data', (data: Buffer) => resolve(data.toString()));
socket.write('NOOP\r\n');
});
expect(lastResponse).toContain('421 Too many errors');
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close().catch(() => {});
}
});
tap.test('should enforce authentication failure limits', async (tools) => {
const done = tools.defer();
// Create config with auth required
const authConfig = {
...testEmailConfig,
auth: {
required: true,
methods: ['PLAIN' as const]
}
};
// Restart server with auth config
await plugins.stopTestServer();
await plugins.startTestServer(authConfig);
await new Promise(resolve => setTimeout(resolve, 1000));
const client = createTestSmtpClient();
try {
// Try multiple failed authentications
for (let i = 0; i < 3; i++) {
const result = await client.sendMail({
from: 'sender@example.com',
to: 'recipient@test.local',
subject: 'Test',
text: 'Test'
}, {
auth: {
user: 'wronguser',
pass: 'wrongpass'
}
}).catch(err => err);
if (i < 2) {
// First 2 should fail with auth error
expect(result.code).toEqual('EAUTH');
} else {
// After 2 failures, should be blocked
expect(result.code).toEqual('ECONNECTION');
}
}
done.resolve();
} catch (error) {
done.reject(error);
} finally {
await client.close().catch(() => {});
}
});
tap.test('cleanup server', async () => {
await plugins.stopTestServer();
});
tap.start();

8
ts/classes.dcrouter.ts
View File

@ -9,6 +9,8 @@ import type { IEmailRoute } from './mail/routing/interfaces.js';
import { logger } from './logger.js';
// Import the email configuration helpers directly from mail/delivery
import { configureEmailStorage, configureEmailServer } from './mail/delivery/index.js';
// Import storage manager
import { StorageManager, type IStorageConfig } from './storage/index.js';
export interface IDcRouterOptions {
/**
@ -63,6 +65,9 @@ export interface IDcRouterOptions {
cloudflareApiKey?: string;
/** Other DNS providers can be added here */
};
/** Storage configuration */
storage?: IStorageConfig;
}
/**
@ -86,6 +91,7 @@ export class DcRouter {
public smartProxy?: plugins.smartproxy.SmartProxy;
public dnsServer?: plugins.smartdns.dnsServerMod.DnsServer;
public emailServer?: UnifiedEmailServer;
public storageManager: StorageManager;
// Environment access
@ -97,6 +103,8 @@ export class DcRouter {
...optionsArg
};
// Initialize storage manager
this.storageManager = new StorageManager(this.options.storage);
}
public async start() {

199
ts/deliverability/classes.senderreputationmonitor.ts
View File

@ -171,12 +171,14 @@ export class SenderReputationMonitor {
private reputationData: Map<string, IDomainReputationMetrics> = new Map();
private updateTimer: NodeJS.Timeout = null;
private isInitialized: boolean = false;
private storageManager?: any; // StorageManager instance
/**
* Constructor for SenderReputationMonitor
* @param config Configuration options
* @param storageManager Optional StorageManager instance
*/
constructor(config: IReputationMonitorConfig = {}) {
constructor(config: IReputationMonitorConfig = {}, storageManager?: any) {
// Merge with default config
this.config = {
...DEFAULT_CONFIG,
@ -191,18 +193,34 @@ export class SenderReputationMonitor {
}
};
// Initialize
this.initialize();
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager) {
logger.log('warn',
'⚠️ WARNING: SenderReputationMonitor initialized without StorageManager.\n' +
' Reputation data will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Initialize (async, but we don't await here to avoid blocking constructor)
this.initialize().catch(error => {
logger.log('error', `Failed to initialize SenderReputationMonitor: ${error.message}`, {
stack: error.stack
});
});
}
/**
* Get the singleton instance
* @param config Configuration options
* @param storageManager Optional StorageManager instance
* @returns Singleton instance
*/
public static getInstance(config: IReputationMonitorConfig = {}): SenderReputationMonitor {
public static getInstance(config: IReputationMonitorConfig = {}, storageManager?: any): SenderReputationMonitor {
if (!SenderReputationMonitor.instance) {
SenderReputationMonitor.instance = new SenderReputationMonitor(config);
SenderReputationMonitor.instance = new SenderReputationMonitor(config, storageManager);
}
return SenderReputationMonitor.instance;
}
@ -210,7 +228,7 @@ export class SenderReputationMonitor {
/**
* Initialize the reputation monitor
*/
private initialize(): void {
private async initialize(): Promise<void> {
if (this.isInitialized) return;
try {
@ -219,7 +237,7 @@ export class SenderReputationMonitor {
if (!isTestEnvironment) {
// Load existing reputation data
this.loadReputationData();
await this.loadReputationData();
}
// Initialize data for any new domains
@ -354,7 +372,7 @@ export class SenderReputationMonitor {
}
// Save all updated data
this.saveReputationData();
await this.saveReputationData();
logger.log('info', 'Completed reputation metrics update for all domains');
}
@ -977,7 +995,11 @@ export class SenderReputationMonitor {
// Skip in test environment
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (!isTestEnvironment && Math.random() < 0.01) { // ~1% chance to save on each event
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data: ${error.message}`, {
stack: error.stack
});
});
}
}
@ -1037,7 +1059,11 @@ export class SenderReputationMonitor {
this.config.domains.push(domain);
this.initializeDomainData(domain);
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data after adding domain: ${error.message}`, {
stack: error.stack
});
});
logger.log('info', `Added ${domain} to reputation monitoring`);
}
@ -1055,7 +1081,11 @@ export class SenderReputationMonitor {
this.config.domains.splice(index, 1);
this.reputationData.delete(domain);
this.saveReputationData();
this.saveReputationData().catch(error => {
logger.log('error', `Failed to save reputation data after removing domain: ${error.message}`, {
stack: error.stack
});
});
logger.log('info', `Removed ${domain} from reputation monitoring`);
}
@ -1063,7 +1093,7 @@ export class SenderReputationMonitor {
/**
* Load reputation data from storage
*/
private loadReputationData(): void {
private async loadReputationData(): Promise<void> {
// Skip loading in test environment to prevent file system race conditions
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (isTestEnvironment) {
@ -1071,32 +1101,98 @@ export class SenderReputationMonitor {
}
try {
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
// Try to load from storage manager first
if (this.storageManager) {
try {
const data = await this.storageManager.get('/email/reputation/domain-reputation.json');
if (data) {
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains from StorageManager`);
return;
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
} catch (error) {
// Fall through to filesystem migration check
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains`);
// Check if data exists in filesystem and migrate it to storage manager
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
// Migrate to storage manager
logger.log('info', `Migrating reputation data for ${this.reputationData.size} domains from filesystem to StorageManager`);
await this.storageManager.set(
'/email/reputation/domain-reputation.json',
JSON.stringify(Array.from(this.reputationData.values()), null, 2)
);
logger.log('info', `Loaded and migrated reputation data for ${this.reputationData.size} domains`);
}
} else {
// No storage manager, use filesystem directly
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
if (plugins.fs.existsSync(dataFile)) {
const data = plugins.fs.readFileSync(dataFile, 'utf8');
const reputationEntries = JSON.parse(data);
for (const entry of reputationEntries) {
// Restore Date objects
entry.lastUpdated = new Date(entry.lastUpdated);
for (const listing of entry.blocklist.activeListings) {
listing.listedSince = new Date(listing.listedSince);
}
for (const delisting of entry.blocklist.recentDelistings) {
delisting.listedFrom = new Date(delisting.listedFrom);
delisting.listedTo = new Date(delisting.listedTo);
}
this.reputationData.set(entry.domain, entry);
}
logger.log('info', `Loaded reputation data for ${this.reputationData.size} domains from filesystem`);
}
}
} catch (error) {
logger.log('error', `Failed to load reputation data: ${error.message}`, {
@ -1108,7 +1204,7 @@ export class SenderReputationMonitor {
/**
* Save reputation data to storage
*/
private saveReputationData(): void {
private async saveReputationData(): Promise<void> {
// Skip saving in test environment to prevent file system race conditions
const isTestEnvironment = process.env.NODE_ENV === 'test' || !!process.env.JEST_WORKER_ID;
if (isTestEnvironment) {
@ -1116,18 +1212,29 @@ export class SenderReputationMonitor {
}
try {
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
const reputationEntries = Array.from(this.reputationData.values());
plugins.smartfile.memory.toFsSync(
JSON.stringify(reputationEntries, null, 2),
dataFile
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains`);
// Save to storage manager if available
if (this.storageManager) {
await this.storageManager.set(
'/email/reputation/domain-reputation.json',
JSON.stringify(reputationEntries, null, 2)
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains to StorageManager`);
} else {
// No storage manager, use filesystem directly
const reputationDir = plugins.path.join(paths.dataDir, 'reputation');
plugins.smartfile.fs.ensureDirSync(reputationDir);
const dataFile = plugins.path.join(reputationDir, 'domain_reputation.json');
plugins.smartfile.memory.toFsSync(
JSON.stringify(reputationEntries, null, 2),
dataFile
);
logger.log('debug', `Saved reputation data for ${reputationEntries.length} domains to filesystem`);
}
} catch (error) {
logger.log('error', `Failed to save reputation data: ${error.message}`, {
stack: error.stack

133
ts/mail/core/classes.bouncemanager.ts
View File

@ -209,10 +209,13 @@ export class BounceManager {
expiresAt?: number; // undefined means permanent
}> = new Map();
private storageManager?: any; // StorageManager instance
constructor(options?: {
retryStrategy?: Partial<RetryStrategy>;
maxCacheSize?: number;
cacheTTL?: number;
storageManager?: any;
}) {
// Set retry strategy with defaults
if (options?.retryStrategy) {
@ -228,8 +231,24 @@ export class BounceManager {
ttl: options?.cacheTTL || 30 * 24 * 60 * 60 * 1000, // 30 days default
});
// Store storage manager reference
this.storageManager = options?.storageManager;
// If no storage manager provided, log warning
if (!this.storageManager) {
console.warn(
'⚠️ WARNING: BounceManager initialized without StorageManager.\n' +
' Bounce data will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Load suppression list from storage
this.loadSuppressionList();
// Note: This is async but we can't await in constructor
// The suppression list will be loaded asynchronously
this.loadSuppressionList().catch(error => {
logger.log('error', `Failed to load suppression list on startup: ${error.message}`);
});
}
/**
@ -479,7 +498,7 @@ export class BounceManager {
this.updateBounceCache(bounce);
// Save to permanent storage
this.saveBounceRecord(bounce);
await this.saveBounceRecord(bounce);
// Log hard bounce for monitoring
logger.log('warn', `Hard bounce for ${bounce.recipient}: ${bounce.bounceType}`, {
@ -545,7 +564,10 @@ export class BounceManager {
expiresAt
});
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after adding ${email}: ${error.message}`);
});
logger.log('info', `Added ${email} to suppression list`, {
reason,
@ -561,7 +583,10 @@ export class BounceManager {
const wasRemoved = this.suppressionList.delete(email.toLowerCase());
if (wasRemoved) {
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after removing ${email}: ${error.message}`);
});
logger.log('info', `Removed ${email} from suppression list`);
}
}
@ -582,7 +607,10 @@ export class BounceManager {
// Check if suppression has expired
if (suppression.expiresAt && Date.now() > suppression.expiresAt) {
this.suppressionList.delete(lowercaseEmail);
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after expiry cleanup: ${error.message}`);
});
return false;
}
@ -609,7 +637,10 @@ export class BounceManager {
// Check if suppression has expired
if (suppression.expiresAt && Date.now() > suppression.expiresAt) {
this.suppressionList.delete(lowercaseEmail);
this.saveSuppressionList();
// Save asynchronously without blocking
this.saveSuppressionList().catch(error => {
logger.log('error', `Failed to save suppression list after expiry cleanup: ${error.message}`);
});
return null;
}
@ -619,13 +650,20 @@ export class BounceManager {
/**
* Save suppression list to disk
*/
private saveSuppressionList(): void {
private async saveSuppressionList(): Promise<void> {
try {
const suppressionData = JSON.stringify(Array.from(this.suppressionList.entries()));
plugins.smartfile.memory.toFsSync(
suppressionData,
plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json')
);
if (this.storageManager) {
// Use storage manager
await this.storageManager.set('/email/bounces/suppression-list.json', suppressionData);
} else {
// Fall back to filesystem
plugins.smartfile.memory.toFsSync(
suppressionData,
plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json')
);
}
} catch (error) {
logger.log('error', `Failed to save suppression list: ${error.message}`);
}
@ -634,14 +672,40 @@ export class BounceManager {
/**
* Load suppression list from disk
*/
private loadSuppressionList(): void {
private async loadSuppressionList(): Promise<void> {
try {
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
let entries = null;
let needsMigration = false;
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
const entries = JSON.parse(data);
if (this.storageManager) {
// Try to load from storage manager first
const suppressionData = await this.storageManager.get('/email/bounces/suppression-list.json');
if (suppressionData) {
entries = JSON.parse(suppressionData);
} else {
// Check if data exists in filesystem and migrate
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
entries = JSON.parse(data);
needsMigration = true;
logger.log('info', 'Migrating suppression list from filesystem to StorageManager');
}
}
} else {
// No storage manager, use filesystem directly
const suppressionPath = plugins.path.join(paths.dataDir, 'emails', 'suppression_list.json');
if (plugins.fs.existsSync(suppressionPath)) {
const data = plugins.fs.readFileSync(suppressionPath, 'utf8');
entries = JSON.parse(data);
}
}
if (entries) {
this.suppressionList = new Map(entries);
// Clean expired entries
@ -655,9 +719,9 @@ export class BounceManager {
}
}
if (expiredCount > 0) {
if (expiredCount > 0 || needsMigration) {
logger.log('info', `Cleaned ${expiredCount} expired entries from suppression list`);
this.saveSuppressionList();
await this.saveSuppressionList();
}
logger.log('info', `Loaded ${this.suppressionList.size} entries from suppression list`);
@ -671,21 +735,28 @@ export class BounceManager {
* Save bounce record to disk
* @param bounce Bounce record to save
*/
private saveBounceRecord(bounce: BounceRecord): void {
private async saveBounceRecord(bounce: BounceRecord): Promise<void> {
try {
const bounceData = JSON.stringify(bounce);
const bouncePath = plugins.path.join(
paths.dataDir,
'emails',
'bounces',
`${bounce.id}.json`
);
const bounceData = JSON.stringify(bounce, null, 2);
// Ensure directory exists
const bounceDir = plugins.path.join(paths.dataDir, 'emails', 'bounces');
plugins.smartfile.fs.ensureDirSync(bounceDir);
plugins.smartfile.memory.toFsSync(bounceData, bouncePath);
if (this.storageManager) {
// Use storage manager
await this.storageManager.set(`/email/bounces/records/${bounce.id}.json`, bounceData);
} else {
// Fall back to filesystem
const bouncePath = plugins.path.join(
paths.dataDir,
'emails',
'bounces',
`${bounce.id}.json`
);
// Ensure directory exists
const bounceDir = plugins.path.join(paths.dataDir, 'emails', 'bounces');
plugins.smartfile.fs.ensureDirSync(bounceDir);
plugins.smartfile.memory.toFsSync(bounceData, bouncePath);
}
} catch (error) {
logger.log('error', `Failed to save bounce record: ${error.message}`);
}

135
ts/mail/delivery/classes.unified.rate.limiter.ts
View File

@ -28,6 +28,9 @@ export interface IHierarchicalRateLimits {
// IP-specific rate limits (applied to specific IPs)
ips?: Record<string, IRateLimitConfig>;
// Domain-specific rate limits (applied to specific email domains)
domains?: Record<string, IRateLimitConfig>;
// Temporary blocks list and their expiry times
blocks?: Record<string, number>; // IP to expiry timestamp
}
@ -86,6 +89,7 @@ export class UnifiedRateLimiter extends EventEmitter {
private counters: Map<string, ILimitCounter> = new Map();
private patternCounters: Map<string, ILimitCounter> = new Map();
private ipCounters: Map<string, ILimitCounter> = new Map();
private domainCounters: Map<string, ILimitCounter> = new Map();
private cleanupInterval?: NodeJS.Timeout;
private stats: IRateLimiterStats;
@ -221,6 +225,13 @@ export class UnifiedRateLimiter extends EventEmitter {
}
}
// Clean domain counters
for (const [key, counter] of this.domainCounters.entries()) {
if (counter.lastReset < cutoff) {
this.domainCounters.delete(key);
}
}
// Update statistics
this.updateStats();
}
@ -231,9 +242,10 @@ export class UnifiedRateLimiter extends EventEmitter {
* @param ip IP address
* @param recipients Number of recipients
* @param pattern Matched pattern
* @param domain Domain name for domain-specific limits
* @returns Result of rate limit check
*/
public checkMessageLimit(email: string, ip: string, recipients: number, pattern?: string): IRateLimitResult {
public checkMessageLimit(email: string, ip: string, recipients: number, pattern?: string, domain?: string): IRateLimitResult {
// Check if IP is blocked
if (this.isIpBlocked(ip)) {
return {
@ -257,6 +269,14 @@ export class UnifiedRateLimiter extends EventEmitter {
}
}
// Check domain-specific limit if domain is provided
if (domain) {
const domainResult = this.checkDomainMessageLimit(domain);
if (!domainResult.allowed) {
return domainResult;
}
}
// Check IP-specific limit
const ipResult = this.checkIpMessageLimit(ip);
if (!ipResult.allowed) {
@ -264,7 +284,7 @@ export class UnifiedRateLimiter extends EventEmitter {
}
// Check recipient limit
const recipientResult = this.checkRecipientLimit(email, recipients, pattern);
const recipientResult = this.checkRecipientLimit(email, recipients, pattern, domain);
if (!recipientResult.allowed) {
return recipientResult;
}
@ -403,6 +423,64 @@ export class UnifiedRateLimiter extends EventEmitter {
return { allowed: true };
}
/**
* Check domain-specific message rate limit
* @param domain Domain to check
*/
private checkDomainMessageLimit(domain: string): IRateLimitResult {
const now = Date.now();
// Get domain-specific limit or use global
const domainConfig = this.config.domains?.[domain];
const limit = domainConfig?.maxMessagesPerMinute || this.config.global.maxMessagesPerMinute!;
if (!limit) {
return { allowed: true };
}
// Get or create counter
let counter = this.domainCounters.get(domain);
if (!counter) {
counter = {
count: 0,
lastReset: now,
recipients: 0,
errors: 0,
authFailures: 0,
connections: 0
};
this.domainCounters.set(domain, counter);
}
// Check if counter needs to be reset
if (now - counter.lastReset >= 60000) {
counter.count = 0;
counter.lastReset = now;
}
// Check if limit is exceeded
if (counter.count >= limit) {
// Calculate reset time
const resetIn = 60000 - (now - counter.lastReset);
logger.log('warn', `Domain ${domain} rate limit exceeded: ${counter.count}/${limit} messages per minute`);
return {
allowed: false,
reason: `Domain "${domain}" message rate limit exceeded`,
limit,
current: counter.count,
resetIn
};
}
// Increment counter
counter.count++;
return { allowed: true };
}
/**
* Check IP-specific message rate limit
* @param ip IP address
@ -485,15 +563,22 @@ export class UnifiedRateLimiter extends EventEmitter {
* @param email Email address
* @param recipients Number of recipients
* @param pattern Matched pattern
* @param domain Domain name
*/
private checkRecipientLimit(email: string, recipients: number, pattern?: string): IRateLimitResult {
// Get pattern-specific limit if available
private checkRecipientLimit(email: string, recipients: number, pattern?: string, domain?: string): IRateLimitResult {
// Get the most specific limit available
let limit = this.config.global.maxRecipientsPerMessage!;
// Check pattern-specific limit
if (pattern && this.config.patterns?.[pattern]?.maxRecipientsPerMessage) {
limit = this.config.patterns[pattern].maxRecipientsPerMessage!;
}
// Check domain-specific limit (overrides pattern if present)
if (domain && this.config.domains?.[domain]?.maxRecipientsPerMessage) {
limit = this.config.domains[domain].maxRecipientsPerMessage!;
}
if (!limit) {
return { allowed: true };
}
@ -923,4 +1008,46 @@ export class UnifiedRateLimiter extends EventEmitter {
public getConfig(): IHierarchicalRateLimits {
return { ...this.config };
}
/**
* Apply domain-specific rate limits
* Merges domain limits with existing configuration
* @param domain Domain name
* @param limits Rate limit configuration for the domain
*/
public applyDomainLimits(domain: string, limits: IRateLimitConfig): void {
if (!this.config.domains) {
this.config.domains = {};
}
// Merge the limits with any existing domain config
this.config.domains[domain] = {
...this.config.domains[domain],
...limits
};
logger.log('info', `Applied rate limits for domain ${domain}:`, limits);
}
/**
* Remove domain-specific rate limits
* @param domain Domain name
*/
public removeDomainLimits(domain: string): void {
if (this.config.domains && this.config.domains[domain]) {
delete this.config.domains[domain];
// Also remove the counter
this.domainCounters.delete(domain);
logger.log('info', `Removed rate limits for domain ${domain}`);
}
}
/**
* Get domain-specific rate limits
* @param domain Domain name
* @returns Domain rate limit config or undefined
*/
public getDomainLimits(domain: string): IRateLimitConfig | undefined {
return this.config.domains?.[domain];
}
}

103
ts/mail/delivery/smtpserver/command-handler.ts
View File

@ -142,15 +142,37 @@ export class CommandHandler implements ICommandHandler {
// For the ERR-01 test, an empty or invalid command is considered a syntax error (500)
if (!command || command.trim().length === 0) {
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
// Record error for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordError(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive errors`);
this.sendResponse(socket, `421 Too many errors - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
}
return;
}
// Handle unknown commands - this should happen before sequence validation
// RFC 5321: Use 500 for unrecognized commands, 501 for parameter errors
if (!Object.values(SmtpCommand).includes(command.toUpperCase() as SmtpCommand)) {
// Comply with RFC 5321 section 4.2.4: Use 500 for unrecognized commands
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
// Record error for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordError(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive errors`);
this.sendResponse(socket, `421 Too many errors - connection blocked`);
socket.end();
} else {
// Comply with RFC 5321 section 4.2.4: Use 500 for unrecognized commands
this.sendResponse(socket, `${SmtpResponseCode.SYNTAX_ERROR} Command not recognized`);
}
return;
}
@ -477,6 +499,12 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Get rate limiter for message-level checks
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Note: Connection-level rate limiting is already handled in ConnectionManager
// Special handling for commands that include "MAIL FROM:" in the args
let processedArgs = args;
@ -509,6 +537,26 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Check message rate limits for this sender
const senderAddress = validation.address || '';
const senderDomain = senderAddress.includes('@') ? senderAddress.split('@')[1] : undefined;
// Check rate limits with domain context if available
const messageResult = rateLimiter.checkMessageLimit(
senderAddress,
session.remoteAddress,
1, // We don't know recipients yet, check with 1
undefined, // No pattern matching for now
senderDomain // Pass domain for domain-specific limits
);
if (!messageResult.allowed) {
SmtpLogger.warn(`Message rate limit exceeded for ${senderAddress} from IP ${session.remoteAddress}: ${messageResult.reason}`);
// Use 421 for temporary rate limiting (client should retry later)
this.sendResponse(socket, `421 ${messageResult.reason} - try again later`);
return;
}
// Enhanced SIZE parameter handling
if (validation.params && validation.params.SIZE) {
const size = parseInt(validation.params.SIZE, 10);
@ -619,6 +667,29 @@ export class CommandHandler implements ICommandHandler {
return;
}
// Check rate limits for recipients
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const recipientAddress = validation.address || '';
const recipientDomain = recipientAddress.includes('@') ? recipientAddress.split('@')[1] : undefined;
// Check rate limits with accumulated recipient count
const recipientCount = session.rcptTo.length + 1; // Including this new recipient
const messageResult = rateLimiter.checkMessageLimit(
session.mailFrom,
session.remoteAddress,
recipientCount,
undefined, // No pattern matching for now
recipientDomain // Pass recipient domain for domain-specific limits
);
if (!messageResult.allowed) {
SmtpLogger.warn(`Recipient rate limit exceeded for ${recipientAddress} from IP ${session.remoteAddress}: ${messageResult.reason}`);
// Use 451 for temporary recipient rejection
this.sendResponse(socket, `451 ${messageResult.reason} - try again later`);
return;
}
// Create recipient object
const recipient: IEnvelopeRecipient = {
address: validation.address || '',
@ -864,7 +935,18 @@ export class CommandHandler implements ICommandHandler {
session.username = username;
this.sendResponse(socket, `${SmtpResponseCode.AUTHENTICATION_SUCCESSFUL} Authentication successful`);
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
// Record authentication failure for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordAuthFailure(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive authentication failures`);
this.sendResponse(socket, `421 Too many authentication failures - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
}
}
} catch (error) {
SmtpLogger.error(`AUTH PLAIN error: ${error instanceof Error ? error.message : String(error)}`);
@ -945,7 +1027,18 @@ export class CommandHandler implements ICommandHandler {
session.username = username;
this.sendResponse(socket, `${SmtpResponseCode.AUTHENTICATION_SUCCESSFUL} Authentication successful`);
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
// Record authentication failure for rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
const shouldBlock = rateLimiter.recordAuthFailure(session.remoteAddress);
if (shouldBlock) {
SmtpLogger.warn(`IP ${session.remoteAddress} blocked due to excessive authentication failures`);
this.sendResponse(socket, `421 Too many authentication failures - connection blocked`);
socket.end();
} else {
this.sendResponse(socket, `${SmtpResponseCode.AUTH_FAILED} Authentication failed`);
}
}
}
} catch (error) {

38
ts/mail/delivery/smtpserver/connection-manager.ts
View File

@ -298,19 +298,20 @@ export class ConnectionManager implements IConnectionManager {
// Get client IP
const remoteAddress = socket.remoteAddress || '0.0.0.0';
// Check rate limits by IP
if (this.isIPRateLimited(remoteAddress)) {
this.rejectConnection(socket, 'Rate limit exceeded');
// Use UnifiedRateLimiter for connection rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Check connection limit with UnifiedRateLimiter
const connectionResult = rateLimiter.recordConnection(remoteAddress);
if (!connectionResult.allowed) {
this.rejectConnection(socket, connectionResult.reason || 'Rate limit exceeded');
this.connectionStats.rejectedConnections++;
return;
}
// Check per-IP connection limit
if (this.hasReachedIPConnectionLimit(remoteAddress)) {
this.rejectConnection(socket, 'Too many connections from this IP');
this.connectionStats.rejectedConnections++;
return;
}
// Still track IP connections locally for cleanup purposes
this.trackIPConnection(remoteAddress);
// Check if maximum global connections reached
if (this.hasReachedMaxConnections()) {
@ -454,19 +455,20 @@ export class ConnectionManager implements IConnectionManager {
// Get client IP
const remoteAddress = socket.remoteAddress || '0.0.0.0';
// Check rate limits by IP
if (this.isIPRateLimited(remoteAddress)) {
this.rejectConnection(socket, 'Rate limit exceeded');
// Use UnifiedRateLimiter for connection rate limiting
const emailServer = this.smtpServer.getEmailServer();
const rateLimiter = emailServer.getRateLimiter();
// Check connection limit with UnifiedRateLimiter
const connectionResult = rateLimiter.recordConnection(remoteAddress);
if (!connectionResult.allowed) {
this.rejectConnection(socket, connectionResult.reason || 'Rate limit exceeded');
this.connectionStats.rejectedConnections++;
return;
}
// Check per-IP connection limit
if (this.hasReachedIPConnectionLimit(remoteAddress)) {
this.rejectConnection(socket, 'Too many connections from this IP');
this.connectionStats.rejectedConnections++;
return;
}
// Still track IP connections locally for cleanup purposes
this.trackIPConnection(remoteAddress);
// Check if maximum global connections reached
if (this.hasReachedMaxConnections()) {

333
ts/mail/routing/classes.dns.validator.ts Normal file
View File

@ -0,0 +1,333 @@
import * as plugins from '../../plugins.js';
import type { IEmailDomainConfig } from './interfaces.js';
import { logger } from '../../logger.js';
import type { DcRouter } from '../../classes.dcrouter.js';
import type { StorageManager } from '../../storage/index.js';
/**
* DNS validation result
*/
export interface IDnsValidationResult {
valid: boolean;
errors: string[];
warnings: string[];
requiredChanges: string[];
}
/**
* DNS records found for a domain
*/
interface IDnsRecords {
mx?: string[];
spf?: string;
dkim?: string;
dmarc?: string;
ns?: string[];
}
/**
* Validates DNS configuration for email domains
*/
export class DnsValidator {
private dcRouter: DcRouter;
private storageManager: StorageManager;
constructor(dcRouter: DcRouter) {
this.dcRouter = dcRouter;
this.storageManager = dcRouter.storageManager;
}
/**
* Validate all domain configurations
*/
async validateAllDomains(domainConfigs: IEmailDomainConfig[]): Promise<Map<string, IDnsValidationResult>> {
const results = new Map<string, IDnsValidationResult>();
for (const config of domainConfigs) {
const result = await this.validateDomain(config);
results.set(config.domain, result);
}
return results;
}
/**
* Validate a single domain configuration
*/
async validateDomain(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
switch (config.dnsMode) {
case 'forward':
return this.validateForwardMode(config);
case 'internal-dns':
return this.validateInternalDnsMode(config);
case 'external-dns':
return this.validateExternalDnsMode(config);
default:
return {
valid: false,
errors: [`Unknown DNS mode: ${config.dnsMode}`],
warnings: [],
requiredChanges: []
};
}
}
/**
* Validate forward mode configuration
*/
private async validateForwardMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
// Forward mode doesn't require DNS validation by default
if (!config.dns?.forward?.skipDnsValidation) {
logger.log('info', `DNS validation skipped for forward mode domain: ${config.domain}`);
}
// DKIM keys are still generated for consistency
result.warnings.push(
`Domain "${config.domain}" uses forward mode. DKIM keys will be generated but signing only happens if email is processed.`
);
return result;
}
/**
* Validate internal DNS mode configuration
*/
private async validateInternalDnsMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
// Check if dnsDomain is configured
const dnsDomain = (this.dcRouter as any).options?.dnsDomain;
if (!dnsDomain) {
result.valid = false;
result.errors.push(
`Domain "${config.domain}" is configured to use internal DNS, but dnsDomain is not set in DcRouter configuration.`
);
console.error(
`❌ ERROR: Domain "${config.domain}" is configured to use internal DNS,\n` +
' but dnsDomain is not set in DcRouter configuration.\n' +
' Please configure dnsDomain to enable the DNS server.\n' +
' Example: dnsDomain: "ns.myservice.com"'
);
return result;
}
// Check NS delegation
try {
const nsRecords = await this.resolveNs(config.domain);
const isDelegated = nsRecords.includes(dnsDomain);
if (!isDelegated) {
result.warnings.push(
`NS delegation not found for ${config.domain}. Please add NS record at your registrar.`
);
result.requiredChanges.push(
`Add NS record: ${config.domain}. NS ${dnsDomain}.`
);
console.log(
`📋 DNS Delegation Required for ${config.domain}:\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
'Please add this NS record at your domain registrar:\n' +
` ${config.domain}. NS ${dnsDomain}.\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
'This delegation is required for internal DNS mode to work.'
);
} else {
console.log(
`✅ NS delegation verified: ${config.domain} -> ${dnsDomain}`
);
}
} catch (error) {
result.warnings.push(
`Could not verify NS delegation for ${config.domain}: ${error.message}`
);
}
return result;
}
/**
* Validate external DNS mode configuration
*/
private async validateExternalDnsMode(config: IEmailDomainConfig): Promise<IDnsValidationResult> {
const result: IDnsValidationResult = {
valid: true,
errors: [],
warnings: [],
requiredChanges: []
};
try {
// Get current DNS records
const records = await this.checkDnsRecords(config);
const requiredRecords = config.dns?.external?.requiredRecords || ['MX', 'SPF', 'DKIM', 'DMARC'];
// Check MX record
if (requiredRecords.includes('MX') && !records.mx?.length) {
result.requiredChanges.push(
`Add MX record: ${this.getBaseDomain(config.domain)} -> ${config.domain} (priority 10)`
);
}
// Check SPF record
if (requiredRecords.includes('SPF') && !records.spf) {
result.requiredChanges.push(
`Add TXT record: ${this.getBaseDomain(config.domain)} -> "v=spf1 a mx ~all"`
);
}
// Check DKIM record
if (requiredRecords.includes('DKIM') && !records.dkim) {
const selector = config.dkim?.selector || 'default';
const dkimPublicKey = await this.storageManager.get(`/email/dkim/${config.domain}/public.key`);
if (dkimPublicKey) {
const publicKeyBase64 = dkimPublicKey
.replace(/-----BEGIN PUBLIC KEY-----/g, '')
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
result.requiredChanges.push(
`Add TXT record: ${selector}._domainkey.${config.domain} -> "v=DKIM1; k=rsa; p=${publicKeyBase64}"`
);
} else {
result.warnings.push(
`DKIM public key not found for ${config.domain}. It will be generated on first use.`
);
}
}
// Check DMARC record
if (requiredRecords.includes('DMARC') && !records.dmarc) {
result.requiredChanges.push(
`Add TXT record: _dmarc.${this.getBaseDomain(config.domain)} -> "v=DMARC1; p=none; rua=mailto:dmarc@${config.domain}"`
);
}
// Show setup instructions if needed
if (result.requiredChanges.length > 0) {
console.log(
`📋 DNS Configuration Required for ${config.domain}:\n` +
'━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n' +
result.requiredChanges.map((change, i) => `${i + 1}. ${change}`).join('\n') +
'\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━'
);
}
} catch (error) {
result.errors.push(`DNS validation failed: ${error.message}`);
result.valid = false;
}
return result;
}
/**
* Check DNS records for a domain
*/
private async checkDnsRecords(config: IEmailDomainConfig): Promise<IDnsRecords> {
const records: IDnsRecords = {};
const baseDomain = this.getBaseDomain(config.domain);
const selector = config.dkim?.selector || 'default';
// Use custom DNS servers if specified
const resolver = new plugins.dns.promises.Resolver();
if (config.dns?.external?.servers?.length) {
resolver.setServers(config.dns.external.servers);
}
// Check MX records
try {
const mxRecords = await resolver.resolveMx(baseDomain);
records.mx = mxRecords.map(mx => mx.exchange);
} catch (error) {
logger.log('debug', `No MX records found for ${baseDomain}`);
}
// Check SPF record
try {
const txtRecords = await resolver.resolveTxt(baseDomain);
const spfRecord = txtRecords.find(records =>
records.some(record => record.startsWith('v=spf1'))
);
if (spfRecord) {
records.spf = spfRecord.join('');
}
} catch (error) {
logger.log('debug', `No SPF record found for ${baseDomain}`);
}
// Check DKIM record
try {
const dkimRecords = await resolver.resolveTxt(`${selector}._domainkey.${config.domain}`);
const dkimRecord = dkimRecords.find(records =>
records.some(record => record.includes('v=DKIM1'))
);
if (dkimRecord) {
records.dkim = dkimRecord.join('');
}
} catch (error) {
logger.log('debug', `No DKIM record found for ${selector}._domainkey.${config.domain}`);
}
// Check DMARC record
try {
const dmarcRecords = await resolver.resolveTxt(`_dmarc.${baseDomain}`);
const dmarcRecord = dmarcRecords.find(records =>
records.some(record => record.startsWith('v=DMARC1'))
);
if (dmarcRecord) {
records.dmarc = dmarcRecord.join('');
}
} catch (error) {
logger.log('debug', `No DMARC record found for _dmarc.${baseDomain}`);
}
return records;
}
/**
* Resolve NS records for a domain
*/
private async resolveNs(domain: string): Promise<string[]> {
try {
const resolver = new plugins.dns.promises.Resolver();
const nsRecords = await resolver.resolveNs(domain);
return nsRecords;
} catch (error) {
logger.log('warn', `Failed to resolve NS records for ${domain}: ${error.message}`);
return [];
}
}
/**
* Get base domain from email domain (e.g., mail.example.com -> example.com)
*/
private getBaseDomain(domain: string): string {
const parts = domain.split('.');
if (parts.length <= 2) {
return domain;
}
// For subdomains like mail.example.com, return example.com
// But preserve domain structure for longer TLDs like .co.uk
if (parts[parts.length - 2].length <= 3 && parts[parts.length - 1].length === 2) {
// Likely a country code TLD like .co.uk
return parts.slice(-3).join('.');
}
return parts.slice(-2).join('.');
}
}

139
ts/mail/routing/classes.domain.registry.ts Normal file
View File

@ -0,0 +1,139 @@
import type { IEmailDomainConfig } from './interfaces.js';
import { logger } from '../../logger.js';
/**
* Registry for email domain configurations
* Provides fast lookups and validation for domains
*/
export class DomainRegistry {
private domains: Map<string, IEmailDomainConfig> = new Map();
private defaults: IEmailDomainConfig['dkim'] & {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
rateLimits?: IEmailDomainConfig['rateLimits'];
};
constructor(
domainConfigs: IEmailDomainConfig[],
defaults?: {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
dkim?: IEmailDomainConfig['dkim'];
rateLimits?: IEmailDomainConfig['rateLimits'];
}
) {
// Set defaults
this.defaults = {
dnsMode: defaults?.dnsMode || 'external-dns',
...this.getDefaultDkimConfig(),
...defaults?.dkim,
rateLimits: defaults?.rateLimits
};
// Process and store domain configurations
for (const config of domainConfigs) {
const processedConfig = this.applyDefaults(config);
this.domains.set(config.domain.toLowerCase(), processedConfig);
logger.log('info', `Registered domain: ${config.domain} with DNS mode: ${processedConfig.dnsMode}`);
}
}
/**
* Get default DKIM configuration
*/
private getDefaultDkimConfig(): IEmailDomainConfig['dkim'] {
return {
selector: 'default',
keySize: 2048,
rotateKeys: false,
rotationInterval: 90
};
}
/**
* Apply defaults to a domain configuration
*/
private applyDefaults(config: IEmailDomainConfig): IEmailDomainConfig {
return {
...config,
dnsMode: config.dnsMode || this.defaults.dnsMode!,
dkim: {
...this.getDefaultDkimConfig(),
...this.defaults,
...config.dkim
},
rateLimits: {
...this.defaults.rateLimits,
...config.rateLimits,
outbound: {
...this.defaults.rateLimits?.outbound,
...config.rateLimits?.outbound
},
inbound: {
...this.defaults.rateLimits?.inbound,
...config.rateLimits?.inbound
}
}
};
}
/**
* Check if a domain is registered
*/
isDomainRegistered(domain: string): boolean {
return this.domains.has(domain.toLowerCase());
}
/**
* Check if an email address belongs to a registered domain
*/
isEmailRegistered(email: string): boolean {
const domain = this.extractDomain(email);
if (!domain) return false;
return this.isDomainRegistered(domain);
}
/**
* Get domain configuration
*/
getDomainConfig(domain: string): IEmailDomainConfig | undefined {
return this.domains.get(domain.toLowerCase());
}
/**
* Get domain configuration for an email address
*/
getEmailDomainConfig(email: string): IEmailDomainConfig | undefined {
const domain = this.extractDomain(email);
if (!domain) return undefined;
return this.getDomainConfig(domain);
}
/**
* Extract domain from email address
*/
private extractDomain(email: string): string | null {
const parts = email.toLowerCase().split('@');
if (parts.length !== 2) return null;
return parts[1];
}
/**
* Get all registered domains
*/
getAllDomains(): string[] {
return Array.from(this.domains.keys());
}
/**
* Get all domain configurations
*/
getAllConfigs(): IEmailDomainConfig[] {
return Array.from(this.domains.values());
}
/**
* Get domains by DNS mode
*/
getDomainsByMode(mode: 'forward' | 'internal-dns' | 'external-dns'): IEmailDomainConfig[] {
return Array.from(this.domains.values()).filter(config => config.dnsMode === mode);
}
}

213
ts/mail/routing/classes.email.router.ts
View File

@ -9,14 +9,29 @@ import type { Email } from '../core/classes.email.js';
export class EmailRouter extends EventEmitter {
private routes: IEmailRoute[];
private patternCache: Map<string, boolean> = new Map();
private storageManager?: any; // StorageManager instance
private persistChanges: boolean;
/**
* Create a new email router
* @param routes Array of email routes
* @param options Router options
*/
constructor(routes: IEmailRoute[]) {
constructor(routes: IEmailRoute[], options?: {
storageManager?: any;
persistChanges?: boolean;
}) {
super();
this.routes = this.sortRoutesByPriority(routes);
this.storageManager = options?.storageManager;
this.persistChanges = options?.persistChanges ?? !!this.storageManager;
// If storage manager is provided, try to load persisted routes
if (this.storageManager) {
this.loadRoutes({ merge: true }).catch(error => {
console.error(`Failed to load persisted routes: ${error.message}`);
});
}
}
/**
@ -43,19 +58,26 @@ export class EmailRouter extends EventEmitter {
/**
* Update routes
* @param routes New routes
* @param persist Whether to persist changes (defaults to persistChanges setting)
*/
public updateRoutes(routes: IEmailRoute[]): void {
public async updateRoutes(routes: IEmailRoute[], persist?: boolean): Promise<void> {
this.routes = this.sortRoutesByPriority(routes);
this.clearCache();
this.emit('routesUpdated', this.routes);
// Persist if requested or if persistChanges is enabled
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Set routes (alias for updateRoutes)
* @param routes New routes
* @param persist Whether to persist changes
*/
public setRoutes(routes: IEmailRoute[]): void {
this.updateRoutes(routes);
public async setRoutes(routes: IEmailRoute[], persist?: boolean): Promise<void> {
await this.updateRoutes(routes, persist);
}
/**
@ -367,4 +389,187 @@ export class EmailRouter extends EventEmitter {
return size;
}
/**
* Save current routes to storage
*/
public async saveRoutes(): Promise<void> {
if (!this.storageManager) {
this.emit('persistenceWarning', 'Cannot save routes: StorageManager not configured');
return;
}
try {
// Validate all routes before saving
for (const route of this.routes) {
if (!route.name || !route.match || !route.action) {
throw new Error(`Invalid route: ${JSON.stringify(route)}`);
}
}
const routesData = JSON.stringify(this.routes, null, 2);
await this.storageManager.set('/email/routes/config.json', routesData);
this.emit('routesPersisted', this.routes.length);
} catch (error) {
console.error(`Failed to save routes: ${error.message}`);
throw error;
}
}
/**
* Load routes from storage
* @param options Load options
*/
public async loadRoutes(options?: {
merge?: boolean; // Merge with existing routes
replace?: boolean; // Replace existing routes
}): Promise<IEmailRoute[]> {
if (!this.storageManager) {
this.emit('persistenceWarning', 'Cannot load routes: StorageManager not configured');
return [];
}
try {
const routesData = await this.storageManager.get('/email/routes/config.json');
if (!routesData) {
return [];
}
const loadedRoutes = JSON.parse(routesData) as IEmailRoute[];
// Validate loaded routes
for (const route of loadedRoutes) {
if (!route.name || !route.match || !route.action) {
console.warn(`Skipping invalid route: ${JSON.stringify(route)}`);
continue;
}
}
if (options?.replace) {
// Replace all routes
this.routes = this.sortRoutesByPriority(loadedRoutes);
} else if (options?.merge) {
// Merge with existing routes (loaded routes take precedence)
const routeMap = new Map<string, IEmailRoute>();
// Add existing routes
for (const route of this.routes) {
routeMap.set(route.name, route);
}
// Override with loaded routes
for (const route of loadedRoutes) {
routeMap.set(route.name, route);
}
this.routes = this.sortRoutesByPriority(Array.from(routeMap.values()));
}
this.clearCache();
this.emit('routesLoaded', loadedRoutes.length);
return loadedRoutes;
} catch (error) {
console.error(`Failed to load routes: ${error.message}`);
throw error;
}
}
/**
* Add a route
* @param route Route to add
* @param persist Whether to persist changes
*/
public async addRoute(route: IEmailRoute, persist?: boolean): Promise<void> {
// Validate route
if (!route.name || !route.match || !route.action) {
throw new Error('Invalid route: missing required fields');
}
// Check if route already exists
const existingIndex = this.routes.findIndex(r => r.name === route.name);
if (existingIndex >= 0) {
throw new Error(`Route '${route.name}' already exists`);
}
// Add route
this.routes.push(route);
this.routes = this.sortRoutesByPriority(this.routes);
this.clearCache();
this.emit('routeAdded', route);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Remove a route by name
* @param name Route name
* @param persist Whether to persist changes
*/
public async removeRoute(name: string, persist?: boolean): Promise<void> {
const index = this.routes.findIndex(r => r.name === name);
if (index < 0) {
throw new Error(`Route '${name}' not found`);
}
const removedRoute = this.routes.splice(index, 1)[0];
this.clearCache();
this.emit('routeRemoved', removedRoute);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Update a route
* @param name Route name
* @param route Updated route data
* @param persist Whether to persist changes
*/
public async updateRoute(name: string, route: IEmailRoute, persist?: boolean): Promise<void> {
// Validate route
if (!route.name || !route.match || !route.action) {
throw new Error('Invalid route: missing required fields');
}
const index = this.routes.findIndex(r => r.name === name);
if (index < 0) {
throw new Error(`Route '${name}' not found`);
}
// Update route
this.routes[index] = route;
this.routes = this.sortRoutesByPriority(this.routes);
this.clearCache();
this.emit('routeUpdated', route);
this.emit('routesUpdated', this.routes);
// Persist if requested
if (persist ?? this.persistChanges) {
await this.saveRoutes();
}
}
/**
* Get a route by name
* @param name Route name
* @returns Route or undefined
*/
public getRoute(name: string): IEmailRoute | undefined {
return this.routes.find(r => r.name === name);
}
}

400
ts/mail/routing/classes.unified.email.server.ts
View File

@ -16,8 +16,10 @@ import {
type IReputationMonitorConfig
} from '../../deliverability/index.js';
import { EmailRouter } from './classes.email.router.js';
import type { IEmailRoute, IEmailAction, IEmailContext } from './interfaces.js';
import type { IEmailRoute, IEmailAction, IEmailContext, IEmailDomainConfig } from './interfaces.js';
import { Email } from '../core/classes.email.js';
import { DomainRegistry } from './classes.domain.registry.js';
import { DnsValidator } from './classes.dns.validator.js';
import { BounceManager, BounceType, BounceCategory } from '../core/classes.bouncemanager.js';
import { createSmtpServer } from '../delivery/smtpserver/index.js';
import { createPooledSmtpClient } from '../delivery/smtpclient/create-client.js';
@ -46,7 +48,7 @@ export interface IUnifiedEmailServerOptions {
// Base server options
ports: number[];
hostname: string;
domains: string[]; // Domains to handle email for
domains: IEmailDomainConfig[]; // Domain configurations
banner?: string;
debug?: boolean;
useSocketHandler?: boolean; // Use socket-handler mode instead of port listening
@ -79,6 +81,13 @@ export interface IUnifiedEmailServerOptions {
// Email routing rules
routes: IEmailRoute[];
// Global defaults for all domains
defaults?: {
dnsMode?: 'forward' | 'internal-dns' | 'external-dns';
dkim?: IEmailDomainConfig['dkim'];
rateLimits?: IEmailDomainConfig['rateLimits'];
};
// Outbound settings
outbound?: {
maxConnections?: number;
@ -88,14 +97,7 @@ export interface IUnifiedEmailServerOptions {
defaultFrom?: string;
};
// DKIM settings
dkim?: {
enabled: boolean;
selector?: string;
keySize?: number;
};
// Rate limiting
// Rate limiting (global limits, can be overridden per domain)
rateLimits?: IHierarchicalRateLimits;
// Deliverability options
@ -156,6 +158,7 @@ export class UnifiedEmailServer extends EventEmitter {
private dcRouter: DcRouter;
private options: IUnifiedEmailServerOptions;
private emailRouter: EmailRouter;
private domainRegistry: DomainRegistry;
private servers: any[] = [];
private stats: IServerStats;
@ -186,20 +189,21 @@ export class UnifiedEmailServer extends EventEmitter {
socketTimeout: options.socketTimeout || 60000 // 1 minute
};
// Initialize DKIM creator
this.dkimCreator = new DKIMCreator(paths.keysDir);
// Initialize DKIM creator with storage manager
this.dkimCreator = new DKIMCreator(paths.keysDir, dcRouter.storageManager);
// Initialize IP reputation checker
// Initialize IP reputation checker with storage manager
this.ipReputationChecker = IPReputationChecker.getInstance({
enableLocalCache: true,
enableDNSBL: true,
enableIPInfo: true
});
}, dcRouter.storageManager);
// Initialize bounce manager
// Initialize bounce manager with storage manager
this.bounceManager = new BounceManager({
maxCacheSize: 10000,
cacheTTL: 30 * 24 * 60 * 60 * 1000 // 30 days
cacheTTL: 30 * 24 * 60 * 60 * 1000, // 30 days
storageManager: dcRouter.storageManager
});
// Initialize IP warmup manager
@ -209,14 +213,23 @@ export class UnifiedEmailServer extends EventEmitter {
targetDomains: []
});
// Initialize sender reputation monitor
this.senderReputationMonitor = SenderReputationMonitor.getInstance(options.reputationMonitorConfig || {
enabled: true,
domains: []
});
// Initialize sender reputation monitor with storage manager
this.senderReputationMonitor = SenderReputationMonitor.getInstance(
options.reputationMonitorConfig || {
enabled: true,
domains: []
},
dcRouter.storageManager
);
// Initialize email router with routes
this.emailRouter = new EmailRouter(options.routes || []);
// Initialize domain registry
this.domainRegistry = new DomainRegistry(options.domains, options.defaults);
// Initialize email router with routes and storage manager
this.emailRouter = new EmailRouter(options.routes || [], {
storageManager: dcRouter.storageManager,
persistChanges: true
});
// Initialize rate limiter
this.rateLimiter = new UnifiedRateLimiter(options.rateLimits || {
@ -331,10 +344,40 @@ export class UnifiedEmailServer extends EventEmitter {
await this.deliverySystem.start();
logger.log('info', 'Email delivery system started');
// Set up automatic DKIM if DNS server is available
if (this.dcRouter.dnsServer && this.options.dkim?.enabled) {
await this.setupAutomaticDkim();
logger.log('info', 'Automatic DKIM configuration completed');
// Set up DKIM for all domains
await this.setupDkimForDomains();
logger.log('info', 'DKIM configuration completed for all domains');
// Set up DNS records for internal-dns mode domains
await this.setupInternalDnsRecords();
logger.log('info', 'DNS records created for internal-dns domains');
// Apply per-domain rate limits
this.applyDomainRateLimits();
logger.log('info', 'Per-domain rate limits configured');
// Check and rotate DKIM keys if needed
await this.checkAndRotateDkimKeys();
logger.log('info', 'DKIM key rotation check completed');
// Validate DNS configuration for all domains
const dnsValidator = new DnsValidator(this.dcRouter);
const validationResults = await dnsValidator.validateAllDomains(this.domainRegistry.getAllConfigs());
// Log validation results
let hasErrors = false;
for (const [domain, result] of validationResults) {
if (!result.valid) {
hasErrors = true;
logger.log('error', `DNS validation failed for ${domain}: ${result.errors.join(', ')}`);
}
if (result.warnings.length > 0) {
logger.log('warn', `DNS warnings for ${domain}: ${result.warnings.join(', ')}`);
}
}
if (hasErrors) {
logger.log('warn', 'Some domains have DNS configuration errors. Email handling may not work correctly.');
}
// Skip server creation in socket-handler mode
@ -984,17 +1027,20 @@ export class UnifiedEmailServer extends EventEmitter {
/**
* Set up automatic DKIM configuration with DNS server
* Set up DKIM configuration for all domains
*/
private async setupAutomaticDkim(): Promise<void> {
if (!this.options.domains || this.options.domains.length === 0) {
private async setupDkimForDomains(): Promise<void> {
const domainConfigs = this.domainRegistry.getAllConfigs();
if (domainConfigs.length === 0) {
logger.log('warn', 'No domains configured for DKIM');
return;
}
const selector = this.options.dkim?.selector || 'default';
for (const domain of this.options.domains) {
for (const domainConfig of domainConfigs) {
const domain = domainConfig.domain;
const selector = domainConfig.dkim?.selector || 'default';
try {
// Check if DKIM keys already exist for this domain
let keyPair: { privateKey: string; publicKey: string };
@ -1020,22 +1066,272 @@ export class UnifiedEmailServer extends EventEmitter {
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
// Register DNS handler for this domain's DKIM records
// Register DNS handler for internal-dns mode domains
if (domainConfig.dnsMode === 'internal-dns' && this.dcRouter.dnsServer) {
const ttl = domainConfig.dns?.internal?.ttl || 3600;
this.dcRouter.dnsServer.registerHandler(
`${selector}._domainkey.${domain}`,
['TXT'],
() => ({
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
})
);
logger.log('info', `DKIM DNS handler registered for domain: ${domain} with selector: ${selector}`);
}
} catch (error) {
logger.log('error', `Failed to set up DKIM for domain ${domain}: ${error.message}`);
}
}
}
/**
* Set up DNS records for internal-dns mode domains
* Creates MX, SPF, and DMARC records automatically
*/
private async setupInternalDnsRecords(): Promise<void> {
// Check if DNS server is available
if (!this.dcRouter.dnsServer) {
logger.log('warn', 'DNS server not available, skipping internal DNS record setup');
return;
}
// Get domains configured for internal-dns mode
const internalDnsDomains = this.domainRegistry.getDomainsByMode('internal-dns');
if (internalDnsDomains.length === 0) {
logger.log('info', 'No domains configured for internal-dns mode');
return;
}
logger.log('info', `Setting up DNS records for ${internalDnsDomains.length} internal-dns domains`);
for (const domainConfig of internalDnsDomains) {
const domain = domainConfig.domain;
const ttl = domainConfig.dns?.internal?.ttl || 3600;
const mxPriority = domainConfig.dns?.internal?.mxPriority || 10;
try {
// 1. Register MX record - points to the email domain itself
this.dcRouter.dnsServer.registerHandler(
`${selector}._domainkey.${domain}`,
['TXT'],
domain,
['MX'],
() => ({
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
name: domain,
type: 'MX',
class: 'IN',
ttl: 300,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
ttl: ttl,
data: {
priority: mxPriority,
exchange: domain
}
})
);
logger.log('info', `MX record registered for ${domain} -> ${domain} (priority ${mxPriority})`);
// Store MX record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/mx`,
JSON.stringify({
type: 'MX',
priority: mxPriority,
exchange: domain,
ttl: ttl
})
);
logger.log('info', `DKIM DNS handler registered for domain: ${domain} with selector: ${selector}`);
// 2. Register SPF record - allows the domain to send emails
const spfRecord = `v=spf1 a mx ~all`;
this.dcRouter.dnsServer.registerHandler(
domain,
['TXT'],
() => ({
name: domain,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: spfRecord
})
);
logger.log('info', `SPF record registered for ${domain}: "${spfRecord}"`);
// Store SPF record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/spf`,
JSON.stringify({
type: 'TXT',
data: spfRecord,
ttl: ttl
})
);
// 3. Register DMARC record - policy for handling email authentication
const dmarcRecord = `v=DMARC1; p=none; rua=mailto:dmarc@${domain}`;
this.dcRouter.dnsServer.registerHandler(
`_dmarc.${domain}`,
['TXT'],
() => ({
name: `_dmarc.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: dmarcRecord
})
);
logger.log('info', `DMARC record registered for _dmarc.${domain}: "${dmarcRecord}"`);
// Store DMARC record in StorageManager
await this.dcRouter.storageManager.set(
`/email/dns/${domain}/dmarc`,
JSON.stringify({
type: 'TXT',
name: `_dmarc.${domain}`,
data: dmarcRecord,
ttl: ttl
})
);
// 4. Register A record - points to the server IP (if available)
// This is needed for SPF 'a' mechanism to work
// Note: We'll skip A record for now since DnsServer doesn't expose getPublicIP
// This can be added later when the server's public IP is known
logger.log('info', `A record setup skipped for ${domain} - public IP detection not available`);
// Log summary of DNS records created
logger.log('info', `✅ DNS records created for ${domain}:
- MX: ${domain} (priority ${mxPriority})
- SPF: ${spfRecord}
- DMARC: ${dmarcRecord}
- DKIM: ${domainConfig.dkim?.selector || 'default'}._domainkey.${domain}`);
} catch (error) {
logger.log('error', `Failed to set up DKIM for domain ${domain}: ${error.message}`);
logger.log('error', `Failed to set up DNS records for ${domain}: ${error.message}`);
}
}
}
/**
* Apply per-domain rate limits from domain configurations
*/
private applyDomainRateLimits(): void {
const domainConfigs = this.domainRegistry.getAllConfigs();
for (const domainConfig of domainConfigs) {
if (domainConfig.rateLimits) {
const domain = domainConfig.domain;
const rateLimitConfig: any = {};
// Convert domain-specific rate limits to the format expected by UnifiedRateLimiter
if (domainConfig.rateLimits.outbound) {
if (domainConfig.rateLimits.outbound.messagesPerMinute) {
rateLimitConfig.maxMessagesPerMinute = domainConfig.rateLimits.outbound.messagesPerMinute;
}
// Note: messagesPerHour and messagesPerDay would need additional implementation in rate limiter
}
if (domainConfig.rateLimits.inbound) {
if (domainConfig.rateLimits.inbound.messagesPerMinute) {
rateLimitConfig.maxMessagesPerMinute = domainConfig.rateLimits.inbound.messagesPerMinute;
}
if (domainConfig.rateLimits.inbound.connectionsPerIp) {
rateLimitConfig.maxConnectionsPerIP = domainConfig.rateLimits.inbound.connectionsPerIp;
}
if (domainConfig.rateLimits.inbound.recipientsPerMessage) {
rateLimitConfig.maxRecipientsPerMessage = domainConfig.rateLimits.inbound.recipientsPerMessage;
}
}
// Apply the rate limits if we have any
if (Object.keys(rateLimitConfig).length > 0) {
this.rateLimiter.applyDomainLimits(domain, rateLimitConfig);
logger.log('info', `Applied rate limits for domain ${domain}:`, rateLimitConfig);
}
}
}
}
/**
* Check and rotate DKIM keys if needed
*/
private async checkAndRotateDkimKeys(): Promise<void> {
const domainConfigs = this.domainRegistry.getAllConfigs();
for (const domainConfig of domainConfigs) {
const domain = domainConfig.domain;
const selector = domainConfig.dkim?.selector || 'default';
const rotateKeys = domainConfig.dkim?.rotateKeys || false;
const rotationInterval = domainConfig.dkim?.rotationInterval || 90;
const keySize = domainConfig.dkim?.keySize || 2048;
if (!rotateKeys) {
logger.log('debug', `DKIM key rotation disabled for ${domain}`);
continue;
}
try {
// Check if keys need rotation
const needsRotation = await this.dkimCreator.needsRotation(domain, selector, rotationInterval);
if (needsRotation) {
logger.log('info', `DKIM keys need rotation for ${domain} (selector: ${selector})`);
// Rotate the keys
const newSelector = await this.dkimCreator.rotateDkimKeys(domain, selector, keySize);
// Update the domain config with new selector
domainConfig.dkim = {
...domainConfig.dkim,
selector: newSelector
};
// Re-register DNS handler for new selector if internal-dns mode
if (domainConfig.dnsMode === 'internal-dns' && this.dcRouter.dnsServer) {
// Get new public key
const keyPair = await this.dkimCreator.readDKIMKeysForSelector(domain, newSelector);
const publicKeyBase64 = keyPair.publicKey
.replace(/-----BEGIN PUBLIC KEY-----/g, '')
.replace(/-----END PUBLIC KEY-----/g, '')
.replace(/\s/g, '');
const ttl = domainConfig.dns?.internal?.ttl || 3600;
// Register new selector
this.dcRouter.dnsServer.registerHandler(
`${newSelector}._domainkey.${domain}`,
['TXT'],
() => ({
name: `${newSelector}._domainkey.${domain}`,
type: 'TXT',
class: 'IN',
ttl: ttl,
data: `v=DKIM1; k=rsa; p=${publicKeyBase64}`
})
);
logger.log('info', `DKIM DNS handler registered for new selector: ${newSelector}._domainkey.${domain}`);
// Store the updated public key in storage
await this.dcRouter.storageManager.set(
`/email/dkim/${domain}/public.key`,
keyPair.publicKey
);
}
// Clean up old keys after grace period (async, don't wait)
this.dkimCreator.cleanupOldKeys(domain, 30).catch(error => {
logger.log('warn', `Failed to cleanup old DKIM keys for ${domain}: ${error.message}`);
});
} else {
logger.log('debug', `DKIM keys for ${domain} are up to date`);
}
} catch (error) {
logger.log('error', `Failed to check/rotate DKIM keys for ${domain}: ${error.message}`);
}
}
}
@ -1118,6 +1414,11 @@ export class UnifiedEmailServer extends EventEmitter {
// Update options without restart
this.options = { ...this.options, ...options };
// Update domain registry if domains changed
if (options.domains) {
this.domainRegistry = new DomainRegistry(options.domains, options.defaults || this.options.defaults);
}
// Update email router if routes changed
if (options.routes) {
this.emailRouter.updateRoutes(options.routes);
@ -1140,6 +1441,13 @@ export class UnifiedEmailServer extends EventEmitter {
return { ...this.stats };
}
/**
* Get domain registry
*/
public getDomainRegistry(): DomainRegistry {
return this.domainRegistry;
}
/**
* Update email routes dynamically
*/
@ -1719,4 +2027,12 @@ export class UnifiedEmailServer extends EventEmitter {
receivingDomain
});
}
/**
* Get the rate limiter instance
* @returns The unified rate limiter
*/
public getRateLimiter(): UnifiedRateLimiter {
return this.rateLimiter;
}
}

3
ts/mail/routing/index.ts
View File

@ -2,4 +2,5 @@
export * from './classes.email.router.js';
export * from './classes.unified.email.server.js';
export * from './classes.dnsmanager.js';
export * from './interfaces.js';
export * from './interfaces.js';
export * from './classes.domain.registry.js';

64
ts/mail/routing/interfaces.ts
View File

@ -135,4 +135,68 @@ export interface IEmailContext {
email: Email;
/** The SMTP session */
session: IExtendedSmtpSession;
}
/**
* Email domain configuration
*/
export interface IEmailDomainConfig {
/** Domain name */
domain: string;
/** DNS handling mode */
dnsMode: 'forward' | 'internal-dns' | 'external-dns';
/** DNS configuration based on mode */
dns?: {
/** For 'forward' mode */
forward?: {
/** Skip DNS validation (default: false) */
skipDnsValidation?: boolean;
/** Target server's expected domain */
targetDomain?: string;
};
/** For 'internal-dns' mode */
internal?: {
/** TTL for DNS records in seconds (default: 3600) */
ttl?: number;
/** MX record priority (default: 10) */
mxPriority?: number;
};
/** For 'external-dns' mode */
external?: {
/** Custom DNS servers (default: system DNS) */
servers?: string[];
/** Which records to validate (default: ['MX', 'SPF', 'DKIM', 'DMARC']) */
requiredRecords?: ('MX' | 'SPF' | 'DKIM' | 'DMARC')[];
};
};
/** Per-domain DKIM settings (DKIM always enabled) */
dkim?: {
/** DKIM selector (default: 'default') */
selector?: string;
/** Key size in bits (default: 2048) */
keySize?: number;
/** Automatically rotate keys (default: false) */
rotateKeys?: boolean;
/** Days between key rotations (default: 90) */
rotationInterval?: number;
};
/** Per-domain rate limits */
rateLimits?: {
outbound?: {
messagesPerMinute?: number;
messagesPerHour?: number;
messagesPerDay?: number;
};
inbound?: {
messagesPerMinute?: number;
connectionsPerIp?: number;
recipientsPerMessage?: number;
};
};
}

344
ts/mail/security/classes.dkimcreator.ts
View File

@ -13,11 +13,31 @@ export interface IKeyPaths {
publicKeyPath: string;
}
export interface IDkimKeyMetadata {
domain: string;
selector: string;
createdAt: number;
rotatedAt?: number;
previousSelector?: string;
keySize: number;
}
export class DKIMCreator {
private keysDir: string;
private storageManager?: any; // StorageManager instance
constructor(keysDir = paths.keysDir) {
constructor(keysDir = paths.keysDir, storageManager?: any) {
this.keysDir = keysDir;
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager) {
console.warn(
'⚠️ WARNING: DKIMCreator initialized without StorageManager.\n' +
' DKIM keys will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
}
public async getKeyPathsForDomain(domainArg: string): Promise<IKeyPaths> {
@ -45,19 +65,63 @@ export class DKIMCreator {
await this.handleDKIMKeysForDomain(domain);
}
// Read DKIM keys from disk
// Read DKIM keys - always use storage manager, migrate from filesystem if needed
public async readDKIMKeys(domainArg: string): Promise<{ privateKey: string; publicKey: string }> {
const keyPaths = await this.getKeyPathsForDomain(domainArg);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
// Try to read from storage manager first
if (this.storageManager) {
try {
const [privateKey, publicKey] = await Promise.all([
this.storageManager.get(`/email/dkim/${domainArg}/private.key`),
this.storageManager.get(`/email/dkim/${domainArg}/public.key`)
]);
if (privateKey && publicKey) {
return { privateKey, publicKey };
}
} catch (error) {
// Fall through to migration check
}
// Check if keys exist in filesystem and migrate them to storage manager
const keyPaths = await this.getKeyPathsForDomain(domainArg);
try {
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
// Convert the buffers to strings
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Convert the buffers to strings
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Migrate to storage manager
console.log(`Migrating DKIM keys for ${domainArg} from filesystem to StorageManager`);
await Promise.all([
this.storageManager.set(`/email/dkim/${domainArg}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domainArg}/public.key`, publicKey)
]);
return { privateKey, publicKey };
} catch (error) {
if (error.code === 'ENOENT') {
// Keys don't exist anywhere
throw new Error(`DKIM keys not found for domain ${domainArg}`);
}
throw error;
}
} else {
// No storage manager, use filesystem directly
const keyPaths = await this.getKeyPathsForDomain(domainArg);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
return { privateKey, publicKey };
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
return { privateKey, publicKey };
}
}
// Create a DKIM key pair - changed to public for API access
@ -71,13 +135,27 @@ export class DKIMCreator {
return { privateKey, publicKey };
}
// Store a DKIM key pair to disk - changed to public for API access
// Store a DKIM key pair - uses storage manager if available, else disk
public async storeDKIMKeys(
privateKey: string,
publicKey: string,
privateKeyPath: string,
publicKeyPath: string
): Promise<void> {
// Store in storage manager if available
if (this.storageManager) {
// Extract domain from path (e.g., /path/to/keys/example.com-private.pem -> example.com)
const match = privateKeyPath.match(/\/([^\/]+)-private\.pem$/);
if (match) {
const domain = match[1];
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/public.key`, publicKey)
]);
}
}
// Also store to filesystem for backward compatibility
await Promise.all([writeFile(privateKeyPath, privateKey), writeFile(publicKeyPath, publicKey)]);
}
@ -117,4 +195,246 @@ export class DKIMCreator {
value: dnsRecordValue,
};
}
/**
* Get DKIM key metadata for a domain
*/
private async getKeyMetadata(domain: string, selector: string = 'default'): Promise<IDkimKeyMetadata | null> {
if (!this.storageManager) {
return null;
}
const metadataKey = `/email/dkim/${domain}/${selector}/metadata`;
const metadataStr = await this.storageManager.get(metadataKey);
if (!metadataStr) {
return null;
}
return JSON.parse(metadataStr) as IDkimKeyMetadata;
}
/**
* Save DKIM key metadata
*/
private async saveKeyMetadata(metadata: IDkimKeyMetadata): Promise<void> {
if (!this.storageManager) {
return;
}
const metadataKey = `/email/dkim/${metadata.domain}/${metadata.selector}/metadata`;
await this.storageManager.set(metadataKey, JSON.stringify(metadata));
}
/**
* Check if DKIM keys need rotation
*/
public async needsRotation(domain: string, selector: string = 'default', rotationIntervalDays: number = 90): Promise<boolean> {
const metadata = await this.getKeyMetadata(domain, selector);
if (!metadata) {
// No metadata means old keys, should rotate
return true;
}
const now = Date.now();
const keyAgeMs = now - metadata.createdAt;
const keyAgeDays = keyAgeMs / (1000 * 60 * 60 * 24);
return keyAgeDays >= rotationIntervalDays;
}
/**
* Rotate DKIM keys for a domain
*/
public async rotateDkimKeys(domain: string, currentSelector: string = 'default', keySize: number = 2048): Promise<string> {
console.log(`Rotating DKIM keys for ${domain}...`);
// Generate new selector based on date
const now = new Date();
const newSelector = `key${now.getFullYear()}${String(now.getMonth() + 1).padStart(2, '0')}`;
// Create new keys with custom key size
const { privateKey, publicKey } = await generateKeyPair('rsa', {
modulusLength: keySize,
publicKeyEncoding: { type: 'spki', format: 'pem' },
privateKeyEncoding: { type: 'pkcs1', format: 'pem' },
});
// Store new keys with new selector
const newKeyPaths = await this.getKeyPathsForSelector(domain, newSelector);
// Store in storage manager if available
if (this.storageManager) {
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/${newSelector}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/${newSelector}/public.key`, publicKey)
]);
}
// Also store to filesystem
await this.storeDKIMKeys(
privateKey,
publicKey,
newKeyPaths.privateKeyPath,
newKeyPaths.publicKeyPath
);
// Save metadata for new keys
const metadata: IDkimKeyMetadata = {
domain,
selector: newSelector,
createdAt: Date.now(),
previousSelector: currentSelector,
keySize
};
await this.saveKeyMetadata(metadata);
// Update metadata for old keys
const oldMetadata = await this.getKeyMetadata(domain, currentSelector);
if (oldMetadata) {
oldMetadata.rotatedAt = Date.now();
await this.saveKeyMetadata(oldMetadata);
}
console.log(`DKIM keys rotated for ${domain}. New selector: ${newSelector}`);
return newSelector;
}
/**
* Get key paths for a specific selector
*/
public async getKeyPathsForSelector(domain: string, selector: string): Promise<IKeyPaths> {
return {
privateKeyPath: plugins.path.join(this.keysDir, `${domain}-${selector}-private.pem`),
publicKeyPath: plugins.path.join(this.keysDir, `${domain}-${selector}-public.pem`),
};
}
/**
* Read DKIM keys for a specific selector
*/
public async readDKIMKeysForSelector(domain: string, selector: string): Promise<{ privateKey: string; publicKey: string }> {
// Try to read from storage manager first
if (this.storageManager) {
try {
const [privateKey, publicKey] = await Promise.all([
this.storageManager.get(`/email/dkim/${domain}/${selector}/private.key`),
this.storageManager.get(`/email/dkim/${domain}/${selector}/public.key`)
]);
if (privateKey && publicKey) {
return { privateKey, publicKey };
}
} catch (error) {
// Fall through to migration check
}
// Check if keys exist in filesystem and migrate them to storage manager
const keyPaths = await this.getKeyPathsForSelector(domain, selector);
try {
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
// Migrate to storage manager
console.log(`Migrating DKIM keys for ${domain}/${selector} from filesystem to StorageManager`);
await Promise.all([
this.storageManager.set(`/email/dkim/${domain}/${selector}/private.key`, privateKey),
this.storageManager.set(`/email/dkim/${domain}/${selector}/public.key`, publicKey)
]);
return { privateKey, publicKey };
} catch (error) {
if (error.code === 'ENOENT') {
throw new Error(`DKIM keys not found for domain ${domain} with selector ${selector}`);
}
throw error;
}
} else {
// No storage manager, use filesystem directly
const keyPaths = await this.getKeyPathsForSelector(domain, selector);
const [privateKeyBuffer, publicKeyBuffer] = await Promise.all([
readFile(keyPaths.privateKeyPath),
readFile(keyPaths.publicKeyPath),
]);
const privateKey = privateKeyBuffer.toString();
const publicKey = publicKeyBuffer.toString();
return { privateKey, publicKey };
}
}
/**
* Get DNS record for a specific selector
*/
public async getDNSRecordForSelector(domain: string, selector: string): Promise<plugins.tsclass.network.IDnsRecord> {
const keys = await this.readDKIMKeysForSelector(domain, selector);
// Remove the PEM header and footer and newlines
const pemHeader = '-----BEGIN PUBLIC KEY-----';
const pemFooter = '-----END PUBLIC KEY-----';
const keyContents = keys.publicKey
.replace(pemHeader, '')
.replace(pemFooter, '')
.replace(/\n/g, '');
// Generate the DKIM DNS TXT record
const dnsRecordValue = `v=DKIM1; h=sha256; k=rsa; p=${keyContents}`;
return {
name: `${selector}._domainkey.${domain}`,
type: 'TXT',
dnsSecEnabled: null,
value: dnsRecordValue,
};
}
/**
* Clean up old DKIM keys after grace period
*/
public async cleanupOldKeys(domain: string, gracePeriodDays: number = 30): Promise<void> {
if (!this.storageManager) {
return;
}
// List all selectors for the domain
const metadataKeys = await this.storageManager.list(`/email/dkim/${domain}/`);
for (const key of metadataKeys) {
if (key.endsWith('/metadata')) {
const metadataStr = await this.storageManager.get(key);
if (metadataStr) {
const metadata = JSON.parse(metadataStr) as IDkimKeyMetadata;
// Check if key is rotated and past grace period
if (metadata.rotatedAt) {
const gracePeriodMs = gracePeriodDays * 24 * 60 * 60 * 1000;
const now = Date.now();
if (now - metadata.rotatedAt > gracePeriodMs) {
console.log(`Cleaning up old DKIM keys for ${domain} selector ${metadata.selector}`);
// Delete key files
const keyPaths = await this.getKeyPathsForSelector(domain, metadata.selector);
try {
await plugins.fs.promises.unlink(keyPaths.privateKeyPath);
await plugins.fs.promises.unlink(keyPaths.publicKeyPath);
} catch (error) {
console.warn(`Failed to delete old key files: ${error.message}`);
}
// Delete metadata
await this.storageManager.delete(key);
}
}
}
}
}
}
}

159
ts/security/classes.ipreputationchecker.ts
View File

@ -64,6 +64,7 @@ export class IPReputationChecker {
private static instance: IPReputationChecker;
private reputationCache: LRUCache<string, IReputationResult>;
private options: Required<IIPReputationOptions>;
private storageManager?: any; // StorageManager instance
// Default DNSBL servers
private static readonly DEFAULT_DNSBL_SERVERS = [
@ -95,14 +96,26 @@ export class IPReputationChecker {
/**
* Constructor for IPReputationChecker
* @param options Configuration options
* @param storageManager Optional StorageManager instance for persistence
*/
constructor(options: IIPReputationOptions = {}) {
constructor(options: IIPReputationOptions = {}, storageManager?: any) {
// Merge with default options
this.options = {
...IPReputationChecker.DEFAULT_OPTIONS,
...options
};
this.storageManager = storageManager;
// If no storage manager provided, log warning
if (!storageManager && this.options.enableLocalCache) {
logger.log('warn',
'⚠️ WARNING: IPReputationChecker initialized without StorageManager.\n' +
' IP reputation cache will only be stored to filesystem.\n' +
' Consider passing a StorageManager instance for better storage flexibility.'
);
}
// Initialize reputation cache
this.reputationCache = new LRUCache<string, IReputationResult>({
max: this.options.maxCacheSize,
@ -111,18 +124,22 @@ export class IPReputationChecker {
// Load cache from disk if enabled
if (this.options.enableLocalCache) {
this.loadCache();
// Fire and forget the load operation
this.loadCache().catch(error => {
logger.log('error', `Failed to load IP reputation cache during initialization: ${error.message}`);
});
}
}
/**
* Get the singleton instance of the checker
* @param options Configuration options
* @param storageManager Optional StorageManager instance for persistence
* @returns Singleton instance
*/
public static getInstance(options: IIPReputationOptions = {}): IPReputationChecker {
public static getInstance(options: IIPReputationOptions = {}, storageManager?: any): IPReputationChecker {
if (!IPReputationChecker.instance) {
IPReputationChecker.instance = new IPReputationChecker(options);
IPReputationChecker.instance = new IPReputationChecker(options, storageManager);
}
return IPReputationChecker.instance;
}
@ -198,7 +215,10 @@ export class IPReputationChecker {
// Save cache if enabled
if (this.options.enableLocalCache) {
this.saveCache();
// Fire and forget the save operation
this.saveCache().catch(error => {
logger.log('error', `Failed to save IP reputation cache: ${error.message}`);
});
}
// Log the reputation check
@ -428,9 +448,9 @@ export class IPReputationChecker {
}
/**
* Save cache to disk
* Save cache to disk or storage manager
*/
private saveCache(): void {
private async saveCache(): Promise<void> {
try {
// Convert cache entries to serializable array
const entries = Array.from(this.reputationCache.entries()).map(([ip, data]) => ({
@ -443,52 +463,94 @@ export class IPReputationChecker {
return;
}
// Ensure directory exists
const cacheDir = plugins.path.join(paths.dataDir, 'security');
plugins.smartfile.fs.ensureDirSync(cacheDir);
const cacheData = JSON.stringify(entries);
// Save to file
const cacheFile = plugins.path.join(cacheDir, 'ip_reputation_cache.json');
plugins.smartfile.memory.toFsSync(
JSON.stringify(entries),
cacheFile
);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to disk`);
// Save to storage manager if available
if (this.storageManager) {
await this.storageManager.set('/security/ip-reputation-cache.json', cacheData);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to StorageManager`);
} else {
// Fall back to filesystem
const cacheDir = plugins.path.join(paths.dataDir, 'security');
plugins.smartfile.fs.ensureDirSync(cacheDir);
const cacheFile = plugins.path.join(cacheDir, 'ip_reputation_cache.json');
plugins.smartfile.memory.toFsSync(cacheData, cacheFile);
logger.log('info', `Saved ${entries.length} IP reputation cache entries to disk`);
}
} catch (error) {
logger.log('error', `Failed to save IP reputation cache: ${error.message}`);
}
}
/**
* Load cache from disk
* Load cache from disk or storage manager
*/
private loadCache(): void {
private async loadCache(): Promise<void> {
try {
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
let cacheData: string | null = null;
let fromFilesystem = false;
// Check if file exists
if (!plugins.fs.existsSync(cacheFile)) {
return;
// Try to load from storage manager first
if (this.storageManager) {
try {
cacheData = await this.storageManager.get('/security/ip-reputation-cache.json');
if (!cacheData) {
// Check if data exists in filesystem and migrate it
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
if (plugins.fs.existsSync(cacheFile)) {
logger.log('info', 'Migrating IP reputation cache from filesystem to StorageManager');
cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
fromFilesystem = true;
// Migrate to storage manager
await this.storageManager.set('/security/ip-reputation-cache.json', cacheData);
logger.log('info', 'IP reputation cache migrated to StorageManager successfully');
// Optionally delete the old file after successful migration
try {
plugins.fs.unlinkSync(cacheFile);
logger.log('info', 'Old cache file removed after migration');
} catch (deleteError) {
logger.log('warn', `Could not delete old cache file: ${deleteError.message}`);
}
}
}
} catch (error) {
logger.log('error', `Error loading from StorageManager: ${error.message}`);
}
} else {
// No storage manager, load from filesystem
const cacheFile = plugins.path.join(paths.dataDir, 'security', 'ip_reputation_cache.json');
if (plugins.fs.existsSync(cacheFile)) {
cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
fromFilesystem = true;
}
}
// Read and parse cache
const cacheData = plugins.fs.readFileSync(cacheFile, 'utf8');
const entries = JSON.parse(cacheData);
// Validate and filter entries
const now = Date.now();
const validEntries = entries.filter(entry => {
const age = now - entry.data.timestamp;
return age < this.options.cacheTTL; // Only load entries that haven't expired
});
// Restore cache
for (const entry of validEntries) {
this.reputationCache.set(entry.ip, entry.data);
// Parse and restore cache if data was found
if (cacheData) {
const entries = JSON.parse(cacheData);
// Validate and filter entries
const now = Date.now();
const validEntries = entries.filter(entry => {
const age = now - entry.data.timestamp;
return age < this.options.cacheTTL; // Only load entries that haven't expired
});
// Restore cache
for (const entry of validEntries) {
this.reputationCache.set(entry.ip, entry.data);
}
const source = fromFilesystem ? 'disk' : 'StorageManager';
logger.log('info', `Loaded ${validEntries.length} IP reputation cache entries from ${source}`);
}
logger.log('info', `Loaded ${validEntries.length} IP reputation cache entries from disk`);
} catch (error) {
logger.log('error', `Failed to load IP reputation cache: ${error.message}`);
}
@ -510,4 +572,21 @@ export class IPReputationChecker {
return 'trusted';
}
}
/**
* Update the storage manager after instantiation
* This is useful when the storage manager is not available at construction time
* @param storageManager The StorageManager instance to use
*/
public updateStorageManager(storageManager: any): void {
this.storageManager = storageManager;
logger.log('info', 'IPReputationChecker storage manager updated');
// If cache is enabled and we have entries, save them to the new storage manager
if (this.options.enableLocalCache && this.reputationCache.size > 0) {
this.saveCache().catch(error => {
logger.log('error', `Failed to save cache to new storage manager: ${error.message}`);
});
}
}
}

397
ts/storage/classes.storagemanager.ts Normal file
View File

@ -0,0 +1,397 @@
import * as plugins from '../plugins.js';
import * as paths from '../paths.js';
import { logger } from '../logger.js';
// Promisify filesystem operations
const readFile = plugins.util.promisify(plugins.fs.readFile);
const writeFile = plugins.util.promisify(plugins.fs.writeFile);
const unlink = plugins.util.promisify(plugins.fs.unlink);
const rename = plugins.util.promisify(plugins.fs.rename);
const readdir = plugins.util.promisify(plugins.fs.readdir);
const stat = plugins.util.promisify(plugins.fs.stat);
/**
* Storage configuration interface
*/
export interface IStorageConfig {
/** Filesystem path for storage */
fsPath?: string;
/** Custom read function */
readFunction?: (key: string) => Promise<string>;
/** Custom write function */
writeFunction?: (key: string, value: string) => Promise<void>;
}
/**
* Storage backend type
*/
export type StorageBackend = 'filesystem' | 'custom' | 'memory';
/**
* Central storage manager for DcRouter
* Provides unified key-value storage with multiple backend support
*/
export class StorageManager {
private backend: StorageBackend;
private memoryStore: Map<string, string> = new Map();
private config: IStorageConfig;
private fsBasePath?: string;
constructor(config?: IStorageConfig) {
this.config = config || {};
// Check if both fsPath and custom functions are provided
if (config?.fsPath && (config?.readFunction || config?.writeFunction)) {
console.warn(
'⚠️ WARNING: Both fsPath and custom read/write functions are configured.\n' +
' Using custom read/write functions. fsPath will be ignored.'
);
}
// Determine backend based on configuration
if (config?.readFunction && config?.writeFunction) {
this.backend = 'custom';
} else if (config?.fsPath) {
// Set up internal read/write functions for filesystem
this.backend = 'custom'; // Use custom backend with internal functions
this.fsBasePath = plugins.path.resolve(config.fsPath);
this.ensureDirectory(this.fsBasePath);
// Set up internal filesystem read/write functions
this.config.readFunction = async (key: string) => {
return this.fsRead(key);
};
this.config.writeFunction = async (key: string, value: string) => {
await this.fsWrite(key, value);
};
} else {
this.backend = 'memory';
this.showMemoryWarning();
}
logger.log('info', `StorageManager initialized with ${this.backend} backend`);
}
/**
* Show warning when using memory backend
*/
private showMemoryWarning(): void {
console.warn(
'⚠️ WARNING: StorageManager is using in-memory storage.\n' +
' Data will be lost when the process restarts.\n' +
' Configure storage.fsPath or storage functions for persistence.'
);
}
/**
* Ensure directory exists for filesystem backend
*/
private async ensureDirectory(dirPath: string): Promise<void> {
try {
await plugins.smartfile.fs.ensureDir(dirPath);
} catch (error) {
logger.log('error', `Failed to create storage directory: ${error.message}`);
throw error;
}
}
/**
* Validate and sanitize storage key
*/
private validateKey(key: string): string {
if (!key || typeof key !== 'string') {
throw new Error('Storage key must be a non-empty string');
}
// Ensure key starts with /
if (!key.startsWith('/')) {
key = '/' + key;
}
// Remove any dangerous path elements
key = key.replace(/\.\./g, '').replace(/\/+/g, '/');
return key;
}
/**
* Convert key to filesystem path
*/
private keyToPath(key: string): string {
if (!this.fsBasePath) {
throw new Error('Filesystem base path not configured');
}
// Remove leading slash and convert to path
const relativePath = key.substring(1);
return plugins.path.join(this.fsBasePath, relativePath);
}
/**
* Internal filesystem read function
*/
private async fsRead(key: string): Promise<string> {
const filePath = this.keyToPath(key);
try {
const content = await readFile(filePath, 'utf8');
return content;
} catch (error) {
if (error.code === 'ENOENT') {
return null;
}
throw error;
}
}
/**
* Internal filesystem write function
*/
private async fsWrite(key: string, value: string): Promise<void> {
const filePath = this.keyToPath(key);
const dir = plugins.path.dirname(filePath);
// Ensure directory exists
await plugins.smartfile.fs.ensureDir(dir);
// Write atomically with temp file
const tempPath = `${filePath}.tmp`;
await writeFile(tempPath, value, 'utf8');
await rename(tempPath, filePath);
}
/**
* Get value by key
*/
async get(key: string): Promise<string | null> {
key = this.validateKey(key);
try {
switch (this.backend) {
case 'custom': {
if (!this.config.readFunction) {
throw new Error('Read function not configured');
}
try {
return await this.config.readFunction(key);
} catch (error) {
// Assume null if read fails (key doesn't exist)
return null;
}
}
case 'memory': {
return this.memoryStore.get(key) || null;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage get error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* Set value by key
*/
async set(key: string, value: string): Promise<void> {
key = this.validateKey(key);
if (typeof value !== 'string') {
throw new Error('Storage value must be a string');
}
try {
switch (this.backend) {
case 'filesystem': {
const filePath = this.keyToPath(key);
const dirPath = plugins.path.dirname(filePath);
// Ensure directory exists
await plugins.smartfile.fs.ensureDir(dirPath);
// Write atomically
const tempPath = filePath + '.tmp';
await writeFile(tempPath, value, 'utf8');
await rename(tempPath, filePath);
break;
}
case 'custom': {
if (!this.config.writeFunction) {
throw new Error('Write function not configured');
}
await this.config.writeFunction(key, value);
break;
}
case 'memory': {
this.memoryStore.set(key, value);
break;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage set error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* Delete value by key
*/
async delete(key: string): Promise<void> {
key = this.validateKey(key);
try {
switch (this.backend) {
case 'filesystem': {
const filePath = this.keyToPath(key);
try {
await unlink(filePath);
} catch (error) {
if (error.code !== 'ENOENT') {
throw error;
}
}
break;
}
case 'custom': {
// Try to delete by setting empty value
if (this.config.writeFunction) {
await this.config.writeFunction(key, '');
}
break;
}
case 'memory': {
this.memoryStore.delete(key);
break;
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage delete error for key ${key}: ${error.message}`);
throw error;
}
}
/**
* List keys by prefix
*/
async list(prefix?: string): Promise<string[]> {
prefix = prefix ? this.validateKey(prefix) : '/';
try {
switch (this.backend) {
case 'filesystem': {
const basePath = this.keyToPath(prefix);
const keys: string[] = [];
const walkDir = async (dir: string, baseDir: string): Promise<void> => {
try {
const entries = await readdir(dir, { withFileTypes: true });
for (const entry of entries) {
const fullPath = plugins.path.join(dir, entry.name);
if (entry.isDirectory()) {
await walkDir(fullPath, baseDir);
} else if (entry.isFile()) {
// Convert path back to key
const relativePath = plugins.path.relative(this.fsBasePath!, fullPath);
const key = '/' + relativePath.replace(/\\/g, '/');
if (key.startsWith(prefix)) {
keys.push(key);
}
}
}
} catch (error) {
if (error.code !== 'ENOENT') {
throw error;
}
}
};
await walkDir(basePath, basePath);
return keys.sort();
}
case 'custom': {
// Custom backends need to implement their own listing
logger.log('warn', 'List operation not supported for custom backend');
return [];
}
case 'memory': {
const keys: string[] = [];
for (const key of this.memoryStore.keys()) {
if (key.startsWith(prefix)) {
keys.push(key);
}
}
return keys.sort();
}
default:
throw new Error(`Unknown backend: ${this.backend}`);
}
} catch (error) {
logger.log('error', `Storage list error for prefix ${prefix}: ${error.message}`);
throw error;
}
}
/**
* Check if key exists
*/
async exists(key: string): Promise<boolean> {
key = this.validateKey(key);
try {
const value = await this.get(key);
return value !== null;
} catch (error) {
return false;
}
}
/**
* Get storage backend type
*/
getBackend(): StorageBackend {
return this.backend;
}
/**
* JSON helper: Get and parse JSON value
*/
async getJSON<T = any>(key: string): Promise<T | null> {
const value = await this.get(key);
if (value === null) {
return null;
}
try {
return JSON.parse(value) as T;
} catch (error) {
logger.log('error', `Failed to parse JSON for key ${key}: ${error.message}`);
throw error;
}
}
/**
* JSON helper: Set value as JSON
*/
async setJSON(key: string, value: any): Promise<void> {
const jsonString = JSON.stringify(value, null, 2);
await this.set(key, jsonString);
}
}

2
ts/storage/index.ts Normal file
View File

@ -0,0 +1,2 @@
// Storage module exports
export * from './classes.storagemanager.js';