diff --git a/changelog.md b/changelog.md index a4acdd0..47fb150 100644 --- a/changelog.md +++ b/changelog.md @@ -1,5 +1,12 @@ # Changelog +## 2026-03-21 - 6.3.0 - feat(readme) +document distributed cluster mode, erasure coding, and QUIC-based architecture + +- Expand README overview and feature matrix to highlight clustering, multi-drive awareness, and distributed storage capabilities +- Add standalone and cluster mode usage examples plus cluster configuration options +- Document clustering internals including erasure coding, quorum behavior, QUIC transport, self-healing, and on-disk layout + ## 2026-03-21 - 6.2.0 - feat(cluster) add shard healing, drive health heartbeats, and clustered policy directory support diff --git a/readme.md b/readme.md index 700a2f0..aa6d02f 100644 --- a/readme.md +++ b/readme.md @@ -1,6 +1,6 @@ # @push.rocks/smartstorage -A high-performance, S3-compatible local storage server powered by a **Rust core** with a clean TypeScript API. Drop-in replacement for AWS S3 during development and testing β€” no cloud, no Docker, no MinIO. Just `npm install` and go. +A high-performance, S3-compatible storage server powered by a **Rust core** with a clean TypeScript API. Runs standalone for dev/test β€” or scales out as a **distributed, erasure-coded cluster** with QUIC-based inter-node communication. No cloud, no Docker. Just `npm install` and go. πŸš€ ## Issue Reporting and Security @@ -15,23 +15,34 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community | Large file uploads | Streaming, zero-copy | Yes | OOM risk | | Range requests | Seek-based | Yes | Full read | | Language | Rust + TypeScript | Go | JavaScript | -| Multipart uploads | Full support | Yes | No | +| Multipart uploads | βœ… Full support | Yes | No | | Auth | AWS SigV4 (full verification) | Full IAM | Basic | | Bucket policies | IAM-style evaluation | Yes | No | +| Clustering | βœ… Erasure-coded, QUIC | Yes | No | +| Multi-drive awareness | βœ… Per-drive health | Yes | No | ### Core Features -- **Rust-powered HTTP server** β€” hyper 1.x with streaming I/O, zero-copy, backpressure -- **Full S3-compatible API** β€” works with AWS SDK v3, SmartBucket, any S3 client -- **Filesystem-backed storage** β€” buckets map to directories, objects to files -- **Streaming multipart uploads** β€” large files without memory pressure -- **Byte-range requests** β€” `seek()` directly to the requested byte offset -- **AWS SigV4 authentication** β€” full signature verification with constant-time comparison and 15-min clock skew enforcement -- **Bucket policies** β€” IAM-style JSON policies with Allow/Deny evaluation, wildcard matching, and anonymous access support -- **CORS middleware** β€” configurable cross-origin support -- **Structured logging** β€” tracing-based, error through debug levels -- **Clean slate mode** β€” wipe storage on startup for test isolation -- **Test-first design** β€” start/stop in milliseconds, no port conflicts +- πŸ¦€ **Rust-powered HTTP server** β€” hyper 1.x with streaming I/O, zero-copy, backpressure +- πŸ“¦ **Full S3-compatible API** β€” works with AWS SDK v3, SmartBucket, any S3 client +- πŸ’Ύ **Filesystem-backed storage** β€” buckets map to directories, objects to files +- πŸ“€ **Streaming multipart uploads** β€” large files without memory pressure +- πŸ“ **Byte-range requests** β€” `seek()` directly to the requested byte offset +- πŸ” **AWS SigV4 authentication** β€” full signature verification with constant-time comparison +- πŸ“‹ **Bucket policies** β€” IAM-style JSON policies with Allow/Deny evaluation and wildcard matching +- 🌐 **CORS middleware** β€” configurable cross-origin support +- 🧹 **Clean slate mode** β€” wipe storage on startup for test isolation +- ⚑ **Test-first design** β€” start/stop in milliseconds, no port conflicts + +### Clustering Features + +- πŸ”— **Erasure coding** β€” Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance +- πŸš„ **QUIC transport** β€” multiplexed, encrypted inter-node communication via `quinn` with zero head-of-line blocking +- πŸ’½ **Multi-drive awareness** β€” each node manages multiple independent storage paths with health monitoring +- 🀝 **Cluster membership** β€” static seed config + runtime join, heartbeat-based failure detection +- ✍️ **Quorum writes** β€” data is only acknowledged after k+1 shards are persisted +- πŸ“– **Quorum reads** β€” reconstruct from any k available shards, local-first fast path +- 🩹 **Self-healing** β€” background scanner detects and reconstructs missing/corrupt shards ## Installation @@ -43,6 +54,8 @@ pnpm add @push.rocks/smartstorage -D ## Quick Start +### Standalone Mode (Dev & Test) + ```typescript import { SmartStorage } from '@push.rocks/smartstorage'; @@ -63,6 +76,31 @@ const descriptor = await storage.getStorageDescriptor(); await storage.stop(); ``` +### Cluster Mode (Distributed) + +```typescript +import { SmartStorage } from '@push.rocks/smartstorage'; + +const storage = await SmartStorage.createAndStart({ + server: { port: 3000 }, + cluster: { + enabled: true, + nodeId: 'node-1', + quicPort: 4000, + seedNodes: ['192.168.1.11:4000', '192.168.1.12:4000'], + erasure: { + dataShards: 4, // k: minimum shards to reconstruct data + parityShards: 2, // m: fault tolerance (can lose up to m shards) + }, + drives: { + paths: ['/mnt/disk1', '/mnt/disk2', '/mnt/disk3'], + }, + }, +}); +``` + +Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data. + ## Configuration All config fields are optional β€” sensible defaults are applied automatically. @@ -75,7 +113,7 @@ const config: ISmartStorageConfig = { port: 3000, // Default: 3000 address: '0.0.0.0', // Default: '0.0.0.0' silent: false, // Default: false - region: 'us-east-1', // Default: 'us-east-1' β€” used for SigV4 signing + region: 'us-east-1', // Default: 'us-east-1' β€” used for SigV4 signing }, storage: { directory: './my-data', // Default: .nogit/bucketsDir @@ -111,6 +149,22 @@ const config: ISmartStorageConfig = { expirationDays: 7, cleanupIntervalMinutes: 60, }, + cluster: { // Optional β€” omit for standalone mode + enabled: true, + nodeId: 'node-1', // Auto-generated UUID if omitted + quicPort: 4000, // Default: 4000 + seedNodes: [], // Addresses of existing cluster members + erasure: { + dataShards: 4, // Default: 4 + parityShards: 2, // Default: 2 + chunkSizeBytes: 4194304, // Default: 4 MB + }, + drives: { + paths: ['/mnt/disk1', '/mnt/disk2'], + }, + heartbeatIntervalMs: 5000, // Default: 5000 + heartbeatTimeoutMs: 30000, // Default: 30000 + }, }; const storage = await SmartStorage.createAndStart(config); @@ -207,7 +261,7 @@ const files = await dir.listFiles(); ## Multipart Uploads -For files larger than 5 MB, use multipart uploads. smartstorage handles them with **streaming I/O** β€” parts are written directly to disk, never buffered in memory. +For files larger than 5 MB, use multipart uploads. smartstorage handles them with **streaming I/O** β€” parts are written directly to disk, never buffered in memory. In cluster mode, each part is independently erasure-coded and distributed. ```typescript import { @@ -255,8 +309,6 @@ When `auth.enabled` is `true`, the auth pipeline works as follows: ### Setting a Bucket Policy -Use the S3 `PutBucketPolicy` API (or any S3 client that supports it): - ```typescript import { PutBucketPolicyCommand } from '@aws-sdk/client-s3'; @@ -294,6 +346,81 @@ await client.send(new PutBucketPolicyCommand({ Deleting a bucket automatically removes its associated policy. +## Clustering Deep Dive πŸ”— + +smartstorage can run as a distributed storage cluster where multiple nodes cooperate to store and retrieve data with built-in redundancy. + +### How It Works + +``` +Client ──HTTP PUT──▢ Node A (coordinator) + β”‚ + β”œβ”€ Split object into 4 MB chunks + β”œβ”€ Erasure-code each chunk (4 data + 2 parity = 6 shards) + β”‚ + β”œβ”€β”€QUIC──▢ Node B (shard writes) + β”œβ”€β”€QUIC──▢ Node C (shard writes) + └─ Local disk (shard writes) +``` + +1. **Any node can coordinate** β€” the client connects to any cluster member +2. **Objects are chunked** β€” large objects split into fixed-size pieces (default 4 MB) +3. **Each chunk is erasure-coded** β€” Reed-Solomon produces k data + m parity shards +4. **Shards are distributed** β€” placed across different nodes and drives for fault isolation +5. **Quorum guarantees consistency** β€” writes need k+1 acks, reads need k shards + +### Erasure Coding + +With the default `4+2` configuration: +- Storage overhead: **33%** (vs. 200% for 3x replication) +- Fault tolerance: **any 2 drives/nodes can fail** simultaneously +- Read efficiency: only **4 of 6 shards** needed to reconstruct data + +| Config | Total Shards | Overhead | Tolerance | Min Nodes | +|--------|-------------|----------|-----------|-----------| +| 4+2 | 6 | 33% | 2 failures | 3 | +| 6+3 | 9 | 50% | 3 failures | 5 | +| 2+1 | 3 | 50% | 1 failure | 2 | + +### QUIC Transport + +Inter-node communication uses [QUIC](https://en.wikipedia.org/wiki/QUIC) via the `quinn` library: +- πŸ”’ **Built-in TLS** β€” self-signed certs auto-generated at cluster init +- πŸ”€ **Multiplexed streams** β€” concurrent shard transfers without head-of-line blocking +- ⚑ **Connection pooling** β€” persistent connections to peer nodes +- 🌊 **Natural backpressure** β€” QUIC flow control prevents overloading slow peers + +### Cluster Membership + +- **Static seed nodes** β€” initial cluster defined in config +- **Runtime join** β€” new nodes can join a running cluster +- **Heartbeat monitoring** β€” every 5s (configurable), with suspect/offline detection +- **Split-brain prevention** β€” nodes only mark peers offline when they have majority + +### Self-Healing + +A background scanner periodically (default: every 24h): +1. Checks shard checksums (CRC32C) for bit-rot detection +2. Identifies shards on offline nodes +3. Reconstructs missing shards from remaining data using Reed-Solomon +4. Places healed shards on healthy drives + +Healing runs at low priority to avoid impacting foreground I/O. + +### Erasure Set Formation + +Drives are organized into fixed **erasure sets** at cluster initialization: + +``` +3 nodes Γ— 4 drives each = 12 drives total +With 6-shard erasure sets β†’ 2 erasure sets + +Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1 +Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3 +``` + +Drives are interleaved across nodes for maximum fault isolation. New nodes form new erasure sets β€” existing data is never rebalanced. + ## Testing Integration ```typescript @@ -358,31 +485,37 @@ Get connection details for S3-compatible clients. Returns: smartstorage uses a **hybrid Rust + TypeScript** architecture: ``` -β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” -β”‚ Your Code (AWS SDK, etc.) β”‚ -β”‚ ↕ HTTP (localhost:3000) β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ ruststorage binary (Rust) β”‚ -β”‚ β”œβ”€ hyper 1.x HTTP server β”‚ -β”‚ β”œβ”€ S3 path-style routing β”‚ -β”‚ β”œβ”€ Streaming storage layer β”‚ -β”‚ β”œβ”€ Multipart manager β”‚ -β”‚ β”œβ”€ SigV4 auth + policy engine β”‚ -β”‚ β”œβ”€ CORS middleware β”‚ -β”‚ └─ S3 XML response builder β”‚ -β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ -β”‚ TypeScript (thin IPC wrapper) β”‚ -β”‚ β”œβ”€ SmartStorage class β”‚ -β”‚ β”œβ”€ RustBridge (stdin/stdout) β”‚ -β”‚ └─ Config & S3 descriptor β”‚ -β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Your Code (AWS SDK, SmartBucket, etc.) β”‚ +β”‚ ↕ HTTP (localhost:3000) β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ ruststorage binary (Rust) β”‚ +β”‚ β”œβ”€ hyper 1.x HTTP server β”‚ +β”‚ β”œβ”€ S3 path-style routing β”‚ +β”‚ β”œβ”€ StorageBackend (Standalone or Clustered) β”‚ +β”‚ β”‚ β”œβ”€ FileStore (single-node mode) β”‚ +β”‚ β”‚ └─ DistributedStore (cluster mode) β”‚ +β”‚ β”‚ β”œβ”€ ErasureCoder (Reed-Solomon) β”‚ +β”‚ β”‚ β”œβ”€ ShardStore (per-drive storage) β”‚ +β”‚ β”‚ β”œβ”€ QuicTransport (quinn) β”‚ +β”‚ β”‚ β”œβ”€ ClusterState & Membership β”‚ +β”‚ β”‚ └─ HealingService β”‚ +β”‚ β”œβ”€ SigV4 auth + policy engine β”‚ +β”‚ β”œβ”€ CORS middleware β”‚ +β”‚ └─ S3 XML response builder β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ TypeScript (thin IPC wrapper) β”‚ +β”‚ β”œβ”€ SmartStorage class β”‚ +β”‚ β”œβ”€ RustBridge (stdin/stdout JSON IPC) β”‚ +β”‚ └─ Config & S3 descriptor β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ ``` -**Why Rust?** The TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests. +**Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests. -**IPC Protocol:** TypeScript spawns the `ruststorage` binary with `--management` and communicates via newline-delimited JSON over stdin/stdout. Commands: `start`, `stop`, `createBucket`. +**IPC Protocol:** TypeScript spawns the `ruststorage` binary with `--management` and communicates via newline-delimited JSON over stdin/stdout. Commands: `start`, `stop`, `createBucket`, `clusterStatus`. -### S3-Compatible Operations Supported +### S3-Compatible Operations | Operation | Method | Path | |-----------|--------|------| @@ -407,26 +540,40 @@ smartstorage uses a **hybrid Rust + TypeScript** architecture: ### On-Disk Format +**Standalone mode:** ``` {storage.directory}/ {bucket}/ - {key}._storage_object # Object data + {key}._storage_object # Object data {key}._storage_object.metadata.json # Metadata (content-type, x-amz-meta-*, etc.) - {key}._storage_object.md5 # Cached MD5 hash + {key}._storage_object.md5 # Cached MD5 hash .multipart/ {upload-id}/ - metadata.json # Upload metadata (bucket, key, parts) - part-1 # Part data files - part-2 - ... + metadata.json # Upload metadata + part-1, part-2, ... # Part data files .policies/ - {bucket}.policy.json # Bucket policy (IAM JSON format) + {bucket}.policy.json # Bucket policy (IAM JSON format) +``` + +**Cluster mode:** +``` +{drive_path}/.smartstorage/ + format.json # Drive metadata (cluster ID, erasure set) + data/{bucket}/{key_hash}/{key}/ + chunk-{N}/shard-{M}.dat # Erasure-coded shard data + chunk-{N}/shard-{M}.meta # Shard metadata (checksum, size) + +{storage.directory}/ + .manifests/{bucket}/ + {key}.manifest.json # Object manifest (shard placements, checksums) + .buckets/{bucket}/ # Bucket metadata + .policies/{bucket}.policy.json # Bucket policies ``` ## Related Packages - [`@push.rocks/smartbucket`](https://code.foss.global/push.rocks/smartbucket) β€” High-level S3-compatible abstraction layer -- [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) β€” TypeScript <-> Rust IPC bridge +- [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) β€” TypeScript ↔ Rust IPC bridge - [`@git.zone/tsrust`](https://code.foss.global/git.zone/tsrust) β€” Rust cross-compilation for npm packages ## License and Legal Information diff --git a/ts/00_commitinfo_data.ts b/ts/00_commitinfo_data.ts index e29fcd6..406986b 100644 --- a/ts/00_commitinfo_data.ts +++ b/ts/00_commitinfo_data.ts @@ -3,6 +3,6 @@ */ export const commitinfo = { name: '@push.rocks/smartstorage', - version: '6.2.0', + version: '6.3.0', description: 'A Node.js TypeScript package to create a local S3-compatible storage server using mapped local directories for development and testing purposes.' }