update

readme.plan.md

@@ -1,91 +1,59 @@
- # DcRouter Implementation Plan
+# Plan for Improving Smartmail Integration
 
- This document outlines the plan for developing **DcRouter**, a central routing
- component for the platform service. DcRouter will provide:
-   - HTTP/HTTPS reverse proxy with TLS (ACME) support
-   - TCP/SNI-based proxying
-   - Mail Transfer Agent (MTA) for inbound/outbound SMTP
-   - Rule-based routing for web and mail traffic
-   - DNS-based dynamic routing
-   - Configuration, persistence, logging, and monitoring
+## Current State Analysis
 
- ## 1. Read and Summarize Dependencies
- We will leverage existing modules; key points from their READMEs:
- - **@push.rocks/smartproxy**: high-performance HTTP/HTTPS reverse proxy,
-   TLS termination, ACME HTTP-01, WebSockets, low-level port forwarding (nftables),
-   TCP/SNI proxy with dynamic routing and IP filtering.
- - **@push.rocks/smartmail**: unified mail data structures and utilities.
- - **mailparser**: parse raw SMTP messages into structured objects.
- - **mailauth**: SPF/DKIM/DMARC validation and signing (DKIM).
- - **@api.global/typedserver** / **typedsocket**: typed HTTP and TCP server abstractions.
- - **@push.rocks/smartrule**: rule engine for dynamic decision-making on traffic.
- - **@push.rocks/smartdns**: DNS resolution and caching for routing decisions.
- - **@push.rocks/smartdata**: persistent key/value or document store for configs.
- - **@push.rocks/smartlog** and **@push.rocks/smartrx**: logging and metrics/event tracking.
- - **@push.rocks/qenv**: environment-based configuration loading.
- - **ts/mta**: built-in MTA implementation (MtaService, SMTPServer, EmailSendJob, DKIMCreator/Verifier, DNSManager, ApiManager, Email class).
+The platformservice currently uses @push.rocks/smartmail version 2.0.1, primarily for email handling in:
+- Email service via connector.mta.ts
+- Template management 
+- API endpoints for sending emails
 
- ## 2. High-Level Architecture
- DcRouter will instantiate and coordinate:
- 1. **HTTP(S) Proxy Layer**
-    - Use `NetworkProxy` and `Port80Handler` from smartproxy
-    - Manage virtual hosts, certificates, redirects
- 2. **TCP/SNI Proxy Layer (SmartProxy)**
-    - Use `SmartProxy` from `@push.rocks/smartproxy` to:
-      - Forward TCP connections by port or port ranges to backends (nftables-like port forwarding).
-      - Route TLS connections based on SNI to different target IPs/ports (SNI bridge).
-      - Integrate HTTP reverse proxy on select ports via `NetworkProxyBridge`.
-      - Manage ACME certificates with built-in `Port80Handler` and `CertProvisioner`.
-      - Apply IP filtering, connection rate limiting, timeouts, and detailed logging.
- 3. **SMTP/MTA Layer**
-    - Leverage `MtaService` from `ts/mta`, coordinating subcomponents:
-      - `SMTPServer` (classes.smtpserver): SMTP listener on ports 25/587 with STARTTLS/TLS
-      - `Email` and `EmailSendJob`: queue-based outbound sending with retries and rate limiting
-      - `DKIMCreator` / `DKIMVerifier`: automatic key management, signing, and verification
-      - `DNSManager`: generate and update DNS records for DKIM/SPF/MTA
-      - `ApiManager`: certificate provisioning and renewal via typedrequest/typedsocket
-    - Integrate inbound processing (mailparser, SPF/DKIM validation) and routing rules
- 4. **Rule Engine**
-    - Use smartrule to define routing rules for HTTP domains and mail addresses
-    - Support wildcard matching, priority, and dynamic updates
- 5. **DNS Integration**
-    - Use smartdns to resolve backend targets by name, with caching
- 6. **Persistence & Config**
-    - Load environment config via qenv
-    - Store dynamic state (rules, domains, MTA settings) in smartdata
- 7. **Logging & Metrics**
-    - Instrument all layers with smartlog and smartrx
- 8. **Management API / CLI**
-    - Expose REST/CLI commands via typedserver for: add/remove domains, rules,
-      view status, reload config, etc.
+## Identified Integration Improvements
 
- ## 3. Feature Breakdown and Milestones
- - [ ] A. Core Router scaffolding (`DcRouter` class)
- - [ ] B. HTTP/HTTPS proxy integration
- - [ ] C. SmartProxy Integration
-   - [ ] C1. Define `IPortProxySettings` with `fromPort`, `toPort`, `domainConfigs`, and global port ranges
-   - [ ] C2. Configure `domainConfigs` to mix port forwarding, SNI routing, and `useNetworkProxy` flags
-   - [ ] C3. Initialize and start `SmartProxy`, including `Port80Handler` for ACME and `CertProvisioner`
-   - [ ] C4. Wire SmartProxy events (e.g., certificate issuance, connection logging) into DcRouter
-   - [ ] C5. Support runtime updates to `domainConfigs` for dynamic routing changes
- - [ ] D. MTA Integration (ts/mta)
-  - [ ] D1. Scaffold integration of `MtaService` into `DcRouter`
-  - [ ] D2. Configure and start `SMTPServer` with TLS/STARTTLS on ports 25/587
-  - [ ] D3. Wire inbound email handling via `processIncomingEmail` and rule engine
-  - [ ] D4. Implement outbound queue processing (`EmailSendJob`), retries, rate limiting
-  - [ ] D5. Manage DKIM keys and DNS records (`DKIMCreator`, `DNSManager`)
-  - [ ] D6. Provision and auto-renew certificates (`ApiManager`)
- - [ ] E. Rule engine wiring for HTTP and SMTP
- - [ ] F. DNS-based dynamic resolution
- - [ ] G. Persistence layer for configs and state
- - [ ] H. Logging/metrics instrumentation
- - [ ] I. CLI and REST API endpoints
- - [ ] J. Automated tests for each layer
- - [ ] K. Documentation and examples in README
+### 1. Update smartmail to Latest Version
 
- ## 4. Next Steps
- 1. Review and refine this plan.
- 2. Begin implementation starting with the core scaffolding (A).
- 3. Iterate through features in milestone order.
+- [x] Update package.json to use the latest smartmail version
+- [ ] Handle any breaking API changes during the upgrade
 
- _Last updated: 2025-05-04_
+### 2. Email Validation Improvements
+
+- [ ] Implement advanced email validation using EmailAddressValidator 
+- [ ] Add MX record checking for outbound emails
+- [ ] Integrate domain reputation checking for spam prevention
+
+### 3. Template System Enhancement
+
+- [ ] Refactor TemplateManager to leverage smartmail's templating capabilities fully
+- [ ] Add support for typed email templates with proper interfaces
+- [ ] Create a comprehensive template catalog with standardized formats
+
+### 4. MIME Handling Improvements
+
+- [ ] Utilize smartmail's MIME conversion capabilities in the MTA connector
+- [ ] Streamline attachment handling between smartmail and internal Email class
+- [ ] Handle content encoding more efficiently
+
+### 5. Integration with MTA Service
+
+- [ ] Refactor Email class to extend or delegate to smartmail.Smartmail
+- [ ] Align the interfaces between internal Email and Smartmail classes
+- [ ] Simplify conversion between MTA Email format and Smartmail
+
+### 6. Enhanced Email Processing
+
+- [ ] Add proper DKIM signature verification using smartmail capabilities
+- [ ] Implement proper email tagging and categorization
+- [ ] Better handling of incoming emails with smart parsing
+
+### 7. Testing & Documentation
+
+- [ ] Create comprehensive tests for the smartmail integration
+- [ ] Document the integration patterns for future reference
+- [ ] Create example implementations for common use cases
+
+## Implementation Strategy
+
+1. First focus on non-breaking improvements (validation, template enhancement)
+2. Then tackle the deeper integration with MTA service
+3. Finally implement advanced features (MIME handling, DKIM)
+
+Each change should be made incrementally with thorough testing to ensure no disruption to the existing email functionality.