push.rocks/smartmta
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(readme): clarify architecture and IPC, document outbound flow and testing, and update module and crate descriptions in README

Browse Source
This commit is contained in:
Jürgen Kunz
2026-02-11 07:36:54 +00:00
parent 7853ef67b6
commit 2088c9f76e
3 changed files with 144 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files

322
readme.md
View File

@@ -18,14 +18,14 @@ After installation, run `pnpm build` to compile the Rust binary (`mailer-bin`).
## Overview
`@push.rocks/smartmta` is a **complete mail server solution** — SMTP server, SMTP client, email security, content scanning, and delivery management — all built with a custom SMTP implementation. The SMTP server itself runs as a Rust binary for maximum performance, communicating with the TypeScript orchestration layer via IPC.
`@push.rocks/smartmta` is a **complete mail server solution** — SMTP server, SMTP client, email security, content scanning, and delivery management — all built with a custom SMTP implementation. The SMTP engine runs as a Rust binary for maximum performance, communicating with the TypeScript orchestration layer via JSON-over-stdin/stdout IPC.
### ⚡ What's Inside
| Module | What It Does |
|---|---|
| **Rust SMTP Server** | High-performance SMTP engine written in Rust — TCP/TLS listener, STARTTLS, AUTH, pipelining, per-connection rate limiting |
| **SMTP Client** | Outbound delivery with connection pooling, retry logic, TLS negotiation |
| **Rust SMTP Server** | High-performance SMTP engine in Rust — TCP/TLS listener, STARTTLS, AUTH, pipelining, per-connection rate limiting |
| **Rust SMTP Client** | Outbound delivery with connection pooling, retry logic, TLS negotiation, DKIM signing — all in Rust |
| **DKIM** | Key generation, signing, and verification — per domain, with automatic rotation |
| **SPF** | Full SPF record validation via Rust |
| **DMARC** | Policy enforcement and verification |
@@ -37,8 +37,7 @@ After installation, run `pnpm build` to compile the Rust binary (`mailer-bin`).
| **Delivery Queue** | Persistent queue with exponential backoff retry |
| **Template Engine** | Email templates with variable substitution |
| **Domain Registry** | Multi-domain management with per-domain configuration |
| **DNS Manager** | Automatic DNS record management with Cloudflare API integration |
| **Rust Security Bridge** | All security ops (DKIM+SPF+DMARC+DNSBL+content scanning) run in Rust via IPC |
| **DNS Manager** | Automatic DNS record management (MX, SPF, DKIM, DMARC) |
### 🏗️ Architecture
@@ -52,9 +51,9 @@ After installation, run `pnpm build` to compile the Rust binary (`mailer-bin`).
│ ┌──────┐ │ ┌───────┐ │ ┌──────────┐ │ ┌────────────────┐ │
│ │Match │ │ │ DKIM │ │ │ Queue │ │ │ DomainRegistry │ │
│ │Route │ │ │ SPF │ │ │ Rate Lim │ │ │ DnsManager │ │
│ │ Act │ │ │ DMARC │ │ │ SMTP Cli │ │ │ DKIMCreator │ │
│ └──────┘ │ │ IPRep │ │ │ Retry │ │ │ Templates │ │
│ │ │ Scan │ │ └──────────┘ │ └────────────────┘ │
│ │ Act │ │ │ DMARC │ │ │ Retry │ │ │ DKIMCreator │ │
│ └──────┘ │ │ IPRep │ │ └──────────┘ │ │ Templates │ │
│ │ │ Scan │ │ │ └────────────────┘ │
│ │ └───────┘ │ │ │
├───────────┴───────────┴──────────────┴───────────────────────┤
│ Rust Security Bridge (smartrust IPC) │
@@ -63,18 +62,25 @@ After installation, run `pnpm build` to compile the Rust binary (`mailer-bin`).
│ ┌──────────────┐ ┌───────────────┐ ┌──────────────────┐ │
│ │ mailer-smtp │ │mailer-security│ │ mailer-core │ │
│ │ SMTP Server │ │DKIM/SPF/DMARC │ │ Types/Validation │ │
│ │ TLS/AUTH │ │IP Rep/Content │ │ MIME/Bounce │ │
│ │ SMTP Client │ │IP Rep/Content │ │ MIME/Bounce │ │
│ │ TLS/AUTH │ │ Scanning │ │ Detection │ │
│ └──────────────┘ └───────────────┘ └──────────────────┘ │
└──────────────────────────────────────────────────────────────┘
```
**Data flow for inbound mail:**
1. Rust SMTP server accepts the connection and handles the full SMTP protocol
2. On `DATA` completion, Rust runs the security pipeline **in-process** (DKIM/SPF/DMARC verification, content scanning, IP reputation check) — zero IPC round-trips
3. Rust emits an `emailReceived` event via IPC with pre-computed security results attached
4. TypeScript processes the email (routing decisions using the pre-computed results, delivery)
5. Rust sends the final SMTP response to the client
1. 📨 Rust SMTP server accepts the connection and handles the full SMTP protocol
2. 🔒 On `DATA` completion, Rust runs the security pipeline **in-process** (DKIM/SPF/DMARC verification, content scanning, IP reputation check) — zero IPC round-trips
3. 📤 Rust emits an `emailReceived` event via IPC with pre-computed security results attached
4. 🔀 TypeScript processes the email (routing decisions using the pre-computed results, delivery)
5. ✅ Rust sends the final SMTP response to the client
**Data flow for outbound mail:**
1. 📝 TypeScript constructs the email and resolves DKIM keys for the sender domain
2. 🦀 Sends to Rust via IPC — Rust builds the RFC 2822 message, signs with DKIM, and delivers via its SMTP client with connection pooling
3. 📬 Result (accepted/rejected recipients, server response) returned to TypeScript
## Usage
@@ -163,32 +169,19 @@ await emailServer.start();
> 🔒 **Note:** `start()` will throw if the Rust binary is not compiled. Run `pnpm build` first.
### 📧 Sending Emails with the SMTP Client
### 📧 Sending Outbound Emails
Create and send emails using the built-in SMTP client with connection pooling:
All outbound email delivery goes through the Rust SMTP client, accessed via `UnifiedEmailServer.sendOutboundEmail()`. The Rust client handles connection pooling, TLS negotiation, and DKIM signing automatically:
```typescript
import { Email, Delivery } from '@push.rocks/smartmta';
// Create a client with connection pooling
const client = Delivery.smtpClientMod.createSmtpClient({
host: 'smtp.example.com',
port: 587,
secure: false, // will upgrade via STARTTLS
pool: true,
maxConnections: 5,
auth: {
user: 'sender@example.com',
pass: 'your-password',
},
});
import { Email, UnifiedEmailServer } from '@push.rocks/smartmta';
// Build an email
const email = new Email({
from: 'sender@example.com',
to: ['recipient@example.com'],
cc: ['cc@example.com'],
subject: 'Hello from smartmta!',
subject: 'Hello from smartmta! 🚀',
text: 'Plain text body',
html: '<h1>Hello!</h1><p>HTML body with <strong>formatting</strong></p>',
priority: 'high',
@@ -201,32 +194,32 @@ const email = new Email({
],
});
// Send it
const result = await client.sendMail(email);
console.log(`Message sent: ${result.messageId}`);
// Send via the Rust SMTP client (connection pooling, TLS, DKIM signing)
const result = await emailServer.sendOutboundEmail('smtp.example.com', 587, email, {
auth: { user: 'sender@example.com', pass: 'your-password' },
dkimDomain: 'example.com',
dkimSelector: 'default',
});
console.log(`Accepted: ${result.accepted.join(', ')}`);
console.log(`Response: ${result.response}`);
// -> Accepted: recipient@example.com
// -> Response: 2.0.0 Ok: queued
```
Additional client factories are available:
The `sendOutboundEmail` method:
- 🔑 Automatically resolves DKIM keys from the `DKIMCreator` for the specified domain
- 🔗 Uses connection pooling in Rust — reuses TCP/TLS connections across sends
- ⏱️ Configurable connection and socket timeouts via `outbound` options on the server
```typescript
// Pooled client for high-throughput scenarios
const pooled = Delivery.smtpClientMod.createPooledSmtpClient({ /* ... */ });
### 🔑 DKIM Signing & Key Management
// Optimized for bulk sending
const bulk = Delivery.smtpClientMod.createBulkSmtpClient({ /* ... */ });
// Optimized for transactional emails
const transactional = Delivery.smtpClientMod.createTransactionalSmtpClient({ /* ... */ });
```
### 🔑 DKIM Signing
DKIM key management is handled by `DKIMCreator`, which generates, stores, and rotates keys per domain. Signing is performed automatically by `UnifiedEmailServer` during outbound delivery:
DKIM key management is handled by `DKIMCreator`, which generates, stores, and rotates keys per domain. Signing is performed automatically by the Rust SMTP client during outbound delivery:
```typescript
import { DKIMCreator } from '@push.rocks/smartmta';
const dkimCreator = new DKIMCreator('/path/to/keys');
const dkimCreator = new DKIMCreator('/path/to/keys', storageManager);
// Auto-generate keys if they don't exist
await dkimCreator.handleDKIMKeysForDomain('example.com');
@@ -244,30 +237,34 @@ if (needsRotation) {
}
```
When `UnifiedEmailServer.start()` is called, DKIM signing is applied to all outbound mail automatically using the Rust security bridge's `signDkim()` method for maximum performance.
When `UnifiedEmailServer.start()` is called:
- DKIM keys are generated or loaded for every configured domain
- Signing is applied to all outbound mail via the Rust security bridge
- Key rotation is checked automatically based on your `rotationInterval` config
### 🛡️ Email Authentication (SPF, DKIM, DMARC)
Verify incoming emails against all three authentication standards. All verification is powered by the Rust binary:
All verification is powered by the Rust binary. For inbound mail, `UnifiedEmailServer` runs the full security pipeline **automatically** — DKIM, SPF, DMARC, content scanning, and IP reputation in a single Rust pass. Results are attached as headers (`Received-SPF`, `X-DKIM-Result`, `X-DMARC-Result`).
You can also use the individual verifiers directly:
```typescript
import { DKIMVerifier, SpfVerifier, DmarcVerifier } from '@push.rocks/smartmta';
// SPF verification — first arg is an Email object
// SPF verification
const spfVerifier = new SpfVerifier();
const spfResult = await spfVerifier.verify(email, senderIP, heloDomain);
// -> { result: 'pass' | 'fail' | 'softfail' | 'neutral' | 'none' | 'temperror' | 'permerror',
// domain: string, ip: string }
// -> { result: 'pass' | 'fail' | 'softfail' | 'neutral' | 'none', domain, ip }
// DKIM verification — takes raw email content
// DKIM verification
const dkimVerifier = new DKIMVerifier();
const dkimResult = await dkimVerifier.verify(rawEmailContent);
// -> [{ is_valid: true, domain: 'example.com', selector: 'default', status: 'pass' }]
// DMARC verification — first arg is an Email object
// DMARC verification
const dmarcVerifier = new DmarcVerifier();
const dmarcResult = await dmarcVerifier.verify(email, spfResult, dkimResult);
// -> { action: 'pass' | 'quarantine' | 'reject', hasDmarc: boolean,
// spfDomainAligned: boolean, dkimDomainAligned: boolean, ... }
// -> { action: 'pass' | 'quarantine' | 'reject', policy, spfDomainAligned, dkimDomainAligned }
```
### 🔀 Email Routing
@@ -294,7 +291,7 @@ const router = new EmailRouter([
priority: 50,
match: {
recipients: '*@example.com',
sizeRange: { max: 10 * 1024 * 1024 }, // under 10MB
sizeRange: { max: 10 * 1024 * 1024 }, // under 10MB
},
action: {
type: 'forward',
@@ -326,7 +323,16 @@ const router = new EmailRouter([
const matchedRoute = await router.evaluateRoutes(emailContext);
```
**Match criteria available:**
#### Route Action Types
| Action | Description |
|---|---|
| `forward` | Forward the email to another SMTP server via the Rust SMTP client |
| `deliver` | Queue for local MTA delivery |
| `process` | Queue for processing (with optional content scanning and DKIM signing) |
| `reject` | Reject with a configurable SMTP error code and message |
#### Match Criteria
| Criterion | Description |
|---|---|
@@ -339,52 +345,6 @@ const matchedRoute = await router.evaluateRoutes(emailContext);
| `subject` | Subject line pattern (string or RegExp) |
| `hasAttachments` | Filter by attachment presence |
### 🔍 Content Scanning
Built-in content scanner for detecting spam, phishing, malware, and other threats. Text pattern scanning runs in Rust for performance; binary attachment scanning (PE headers, VBA macros) runs in TypeScript:
```typescript
import { ContentScanner } from '@push.rocks/smartmta';
const scanner = new ContentScanner({
scanSubject: true,
scanBody: true,
scanAttachments: true,
blockExecutables: true,
blockMacros: true,
minThreatScore: 30,
highThreatScore: 70,
customRules: [
{
pattern: /bitcoin.*wallet/i,
type: 'scam',
score: 80,
description: 'Cryptocurrency scam pattern',
},
],
});
const result = await scanner.scanEmail(email);
// -> { isClean: false, threatScore: 85, threatType: 'phishing', scannedElements: [...] }
```
### 🌐 IP Reputation Checking
Check sender IP addresses against DNSBL blacklists and classify IP types. DNSBL lookups run in Rust:
```typescript
import { IPReputationChecker } from '@push.rocks/smartmta';
const ipChecker = IPReputationChecker.getInstance({
enableDNSBL: true,
dnsblServers: ['zen.spamhaus.org', 'bl.spamcop.net'],
cacheTTL: 24 * 60 * 60 * 1000, // 24 hours
});
const reputation = await ipChecker.checkReputation('192.168.1.1');
// -> { score: 85, isSpam: false, isProxy: false, isTor: false, blacklists: [] }
```
### ⏱️ Rate Limiting
Hierarchical rate limiting to protect your server and maintain deliverability:
@@ -445,7 +405,7 @@ const bounce = await bounceManager.processSmtpFailure(
// Check if an address is suppressed due to bounces
const suppressed = bounceManager.isEmailSuppressed('recipient@example.com');
// Manually manage the suppression list
// Manage the suppression list
bounceManager.addToSuppressionList('bad@example.com', 'repeated hard bounces');
bounceManager.removeFromSuppressionList('recovered@example.com');
```
@@ -484,7 +444,7 @@ const email = await templates.createEmail('welcome', {
### 🌍 DNS Management
DNS record management for email authentication is handled automatically by `UnifiedEmailServer`. When the server starts, it ensures MX, SPF, DKIM, and DMARC records are in place for all configured domains via the Cloudflare API:
When `UnifiedEmailServer.start()` is called, it automatically ensures MX, SPF, DKIM, and DMARC records are in place for all configured domains:
```typescript
const emailServer = new UnifiedEmailServer(dcRouterRef, {
@@ -492,7 +452,7 @@ const emailServer = new UnifiedEmailServer(dcRouterRef, {
domains: [
{
domain: 'example.com',
dnsMode: 'external-dns', // managed via Cloudflare API
dnsMode: 'external-dns', // managed via Cloudflare API
},
],
// ... other config
@@ -506,99 +466,43 @@ const emailServer = new UnifiedEmailServer(dcRouterRef, {
await emailServer.start();
```
### 🦀 RustSecurityBridge
The `RustSecurityBridge` is the singleton that manages the Rust binary process. It handles security verification, content scanning, bounce detection, and the SMTP server lifecycle — all via `@push.rocks/smartrust` IPC:
```typescript
import { RustSecurityBridge } from '@push.rocks/smartmta';
const bridge = RustSecurityBridge.getInstance();
await bridge.start();
// Compound verification: DKIM + SPF + DMARC in a single IPC call
const securityResult = await bridge.verifyEmail({
rawMessage: rawEmailString,
ip: '203.0.113.10',
heloDomain: 'sender.example.com',
mailFrom: 'user@example.com',
});
// -> { dkim: [...], spf: { result, explanation }, dmarc: { result, policy } }
// Individual security operations
const dkimResults = await bridge.verifyDkim(rawEmailString);
const spfResult = await bridge.checkSpf({
ip: '203.0.113.10',
heloDomain: 'sender.example.com',
mailFrom: 'user@example.com',
});
const reputationResult = await bridge.checkIpReputation('203.0.113.10');
// DKIM signing
const signed = await bridge.signDkim({
email: rawEmailString,
domain: 'example.com',
selector: 'default',
privateKeyPem: privateKey,
});
// Content scanning
const scanResult = await bridge.scanContent({
subject: 'Win a free iPhone!!!',
body: '<a href="http://phishing.example.com">Click here</a>',
from: 'scammer@evil.com',
});
// Bounce detection
const bounceResult = await bridge.detectBounce({
subject: 'Delivery Status Notification (Failure)',
body: '550 5.1.1 User unknown',
from: 'mailer-daemon@example.com',
});
await bridge.stop();
```
> ⚠️ **Important:** The Rust bridge is **mandatory**. There are no TypeScript fallbacks. If the Rust binary is unavailable, `UnifiedEmailServer.start()` will throw an error.
## 🦀 Rust Acceleration Layer
Performance-critical operations are implemented in Rust and communicate with the TypeScript runtime via `@push.rocks/smartrust` (JSON-over-stdin/stdout IPC). The Rust workspace lives at `rust/` with five crates:
Performance-critical operations are implemented in Rust and communicate with the TypeScript runtime via `@push.rocks/smartrust` (JSON-over-stdin/stdout IPC). The Rust workspace lives at `rust/` with four crates:
| Crate | Status | Purpose |
|---|---|---|
| `mailer-core` | ✅ Complete (26 tests) | Email types, validation, MIME building, bounce detection |
| `mailer-security` | ✅ Complete (22 tests) | DKIM sign/verify, SPF, DMARC, IP reputation/DNSBL, content scanning |
| `mailer-smtp` | ✅ Complete (77 tests) | Full SMTP protocol engine — TCP/TLS server, STARTTLS, AUTH, pipelining, in-process security pipeline, rate limiting |
| `mailer-bin` | ✅ Complete | CLI + smartrust IPC bridge — security, content scanning, SMTP server lifecycle |
| `mailer-napi` | 🔜 Planned | Native Node.js addon (N-API) |
| `mailer-smtp` | ✅ Complete (106 tests) | Full SMTP protocol engine — TCP/TLS server + client, STARTTLS, AUTH, pipelining, connection pooling, in-process security pipeline |
| `mailer-bin` | ✅ Complete | CLI + smartrust IPC bridge — wires everything together |
### What Runs in Rust
### What Runs Where
| Operation | Runs In | Why |
|---|---|---|
| SMTP server (port listening, protocol, TLS) | Rust | Performance, memory safety, zero-copy parsing |
| DKIM signing & verification | Rust | Crypto-heavy, benefits from native speed |
| SPF validation | Rust | DNS lookups with async resolver |
| DMARC policy checking | Rust | Integrates with SPF/DKIM results |
| IP reputation / DNSBL | Rust | Parallel DNS queries |
| Content scanning (text patterns) | Rust | Regex engine performance |
| Bounce detection (pattern matching) | Rust | Regex engine performance |
| Email validation & MIME building | Rust | Parsing performance |
| Binary attachment scanning | TypeScript | Buffer data too large for IPC |
| Email routing & orchestration | TypeScript | Business logic, flexibility |
| Delivery queue & retry | TypeScript | State management, persistence |
| Template rendering | TypeScript | String interpolation |
| SMTP server (port listening, protocol, TLS) | 🦀 Rust | Performance, memory safety, zero-copy parsing |
| SMTP client (outbound delivery, connection pooling) | 🦀 Rust | Connection management, TLS negotiation |
| DKIM signing & verification | 🦀 Rust | Crypto-heavy, benefits from native speed |
| SPF validation | 🦀 Rust | DNS lookups with async resolver |
| DMARC policy checking | 🦀 Rust | Integrates with SPF/DKIM results |
| IP reputation / DNSBL | 🦀 Rust | Parallel DNS queries |
| Content scanning (text patterns) | 🦀 Rust | Regex engine performance |
| Bounce detection (pattern matching) | 🦀 Rust | Regex engine performance |
| Email validation & MIME building | 🦀 Rust | Parsing performance |
| Email routing & orchestration | 🟦 TypeScript | Business logic, flexibility |
| Delivery queue & retry | 🟦 TypeScript | State management, persistence |
| Template rendering | 🟦 TypeScript | String interpolation |
| Domain & DNS management | 🟦 TypeScript | API integrations |
## Project Structure
## 📁 Project Structure
```
smartmta/
├── ts/ # TypeScript source
│ ├── mail/
│ │ ├── core/ # Email, EmailValidator, BounceManager, TemplateManager
│ │ ├── delivery/ # DeliverySystem, Queue, RateLimiter
│ │ │ └── smtpclient/ # SMTP client with connection pooling
│ │ ├── delivery/ # DeliveryQueue, DeliverySystem, RateLimiter
│ │ ├── routing/ # UnifiedEmailServer, EmailRouter, DomainRegistry, DnsManager
│ │ └── security/ # DKIMCreator, DKIMVerifier, SpfVerifier, DmarcVerifier
│ └── security/ # ContentScanner, IPReputationChecker, RustSecurityBridge
@@ -606,14 +510,56 @@ smartmta/
│ └── crates/
│ ├── mailer-core/ # Email types, validation, MIME, bounce detection
│ ├── mailer-security/ # DKIM, SPF, DMARC, IP reputation, content scanning
│ ├── mailer-smtp/ # Full SMTP server (TCP/TLS, state machine, rate limiting)
│ ├── mailer-bin/ # CLI + smartrust IPC bridge
│ └── mailer-napi/ # N-API addon (planned)
├── test/ # Test suite
│ ├── mailer-smtp/ # Full SMTP server + client (TCP/TLS, rate limiting, pooling)
│ └── mailer-bin/ # CLI + smartrust IPC bridge
├── test/ # Test suite (116 TypeScript + 154 Rust tests)
├── dist_ts/ # Compiled TypeScript output
└── dist_rust/ # Compiled Rust binaries
```
## 🧪 Testing
The project has comprehensive test coverage with both unit and end-to-end tests:
```bash
# Build Rust binary first
pnpm build
# Run all tests
pnpm test
# Run specific test files
tstest test/test.e2e.server-lifecycle.node.ts --verbose --timeout 60
tstest test/test.e2e.inbound-smtp.node.ts --verbose --timeout 60
tstest test/test.e2e.routing-actions.node.ts --verbose --timeout 60
tstest test/test.e2e.outbound-delivery.node.ts --verbose --timeout 60
```
**E2E tests** exercise the full pipeline — starting `UnifiedEmailServer`, connecting via raw TCP sockets, sending SMTP transactions, verifying routing actions, and testing outbound delivery through a mock SMTP receiver.
## API Reference
### Exported Classes (top-level)
| Class | Description |
|---|---|
| `UnifiedEmailServer` | 🎯 Main entry point — orchestrates SMTP server, routing, security, and delivery |
| `Email` | Email message class with validation, attachments, headers, and RFC 822 serialization |
| `EmailRouter` | Pattern-based route matching and evaluation engine |
| `DomainRegistry` | Multi-domain configuration manager |
| `DnsManager` | Automatic DNS record management |
| `DKIMCreator` | DKIM key generation, storage, rotation |
| `DKIMVerifier` | DKIM signature verification (delegates to Rust) |
| `SpfVerifier` | SPF record validation (delegates to Rust) |
| `DmarcVerifier` | DMARC policy enforcement (delegates to Rust) |
### Namespaced Exports
| Namespace | Classes |
|---|---|
| `Core` | `Email`, `EmailValidator`, `TemplateManager`, `BounceManager` |
| `Delivery` | `UnifiedDeliveryQueue`, `MultiModeDeliverySystem`, `DeliveryStatus`, `UnifiedRateLimiter` |
## License and Legal Information
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.