"description":"A Node.js TypeScript package to create a local S3-compatible storage server using mapped local directories for development and testing purposes.",
"description":"A Node.js TypeScript package to create a local S3-compatible storage server using mapped local directories for development and testing purposes.",
A high-performance, S3-compatible local storage server powered by a **Rust core** with a clean TypeScript API. Drop-in replacement for AWS S3 during development and testing — no cloud, no Docker, no MinIO. Just `npm install` and go.
A high-performance, S3-compatible storage server powered by a **Rust core** with a clean TypeScript API. Runs standalone for dev/test — or scales out as a **distributed, erasure-coded cluster** with QUIC-based inter-node communication. No cloud, no Docker. Just `npm install` and go. 🚀
## Issue Reporting and Security
## Issue Reporting and Security
@@ -15,23 +15,34 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
dataShards: 4,// k: minimum shards to reconstruct data
parityShards: 2,// m: fault tolerance (can lose up to m shards)
},
drives:{
paths:['/mnt/disk1','/mnt/disk2','/mnt/disk3'],
},
},
});
```
Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data.
## Configuration
## Configuration
All config fields are optional — sensible defaults are applied automatically.
All config fields are optional — sensible defaults are applied automatically.
For files larger than 5 MB, use multipart uploads. smartstorage handles them with **streaming I/O** — parts are written directly to disk, never buffered in memory.
For files larger than 5 MB, use multipart uploads. smartstorage handles them with **streaming I/O** — parts are written directly to disk, never buffered in memory. In cluster mode, each part is independently erasure-coded and distributed.
```typescript
```typescript
import{
import{
@@ -255,8 +309,6 @@ When `auth.enabled` is `true`, the auth pipeline works as follows:
### Setting a Bucket Policy
### Setting a Bucket Policy
Use the S3 `PutBucketPolicy` API (or any S3 client that supports it):
- **Static seed nodes** — initial cluster defined in config
- **Runtime join** — new nodes can join a running cluster
- **Heartbeat monitoring** — every 5s (configurable), with suspect/offline detection
- **Split-brain prevention** — nodes only mark peers offline when they have majority
### Self-Healing
A background scanner periodically (default: every 24h):
1. Checks shard checksums (CRC32C) for bit-rot detection
2. Identifies shards on offline nodes
3. Reconstructs missing shards from remaining data using Reed-Solomon
4. Places healed shards on healthy drives
Healing runs at low priority to avoid impacting foreground I/O.
### Erasure Set Formation
Drives are organized into fixed **erasure sets** at cluster initialization:
```
3 nodes × 4 drives each = 12 drives total
With 6-shard erasure sets → 2 erasure sets
Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1
Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3
```
Drives are interleaved across nodes for maximum fault isolation. New nodes form new erasure sets — existing data is never rebalanced.
## Testing Integration
## Testing Integration
```typescript
```typescript
@@ -358,31 +485,37 @@ Get connection details for S3-compatible clients. Returns:
smartstorage uses a **hybrid Rust + TypeScript** architecture:
smartstorage uses a **hybrid Rust + TypeScript** architecture:
```
```
┌─────────────────────────────────┐
┌──────────────────────────────────────────────┐
│ Your Code (AWS SDK, etc.) │
│ Your Code (AWS SDK, SmartBucket, etc.) │
│ ↕ HTTP (localhost:3000) │
│ ↕ HTTP (localhost:3000) │
├─────────────────────────────────┤
├──────────────────────────────────────────────┤
│ ruststorage binary (Rust) │
│ ruststorage binary (Rust) │
│ ├─ hyper 1.x HTTP server │
│ ├─ hyper 1.x HTTP server │
│ ├─ S3 path-style routing │
│ ├─ S3 path-style routing │
│ ├─ Streaming storage layer │
│ ├─ StorageBackend (Standalone or Clustered) │
│ ├─ Multipart manager │
│ │ ├─ FileStore (single-node mode) │
│ ├─ SigV4 auth + policy engine │
│ │ └─ DistributedStore (cluster mode) │
│ ├─ CORS middleware │
│ │ ├─ ErasureCoder (Reed-Solomon) │
│ └─ S3 XML response builder │
│ │ ├─ ShardStore (per-drive storage) │
├─────────────────────────────────┤
│ │ ├─ QuicTransport (quinn) │
│ TypeScript (thin IPC wrapper) │
│ │ ├─ ClusterState & Membership │
│ ├─ SmartStorage class │
│ │ └─ HealingService │
│ ├─ RustBridge (stdin/stdout) │
│ ├─ SigV4 auth + policy engine │
│ └─ Config & S3 descriptor │
│ ├─ CORS middleware │
└─────────────────────────────────┘
│ └─ S3 XML response builder │
├──────────────────────────────────────────────┤
│ TypeScript (thin IPC wrapper) │
│ ├─ SmartStorage class │
│ ├─ RustBridge (stdin/stdout JSON IPC) │
│ └─ Config & S3 descriptor │
└──────────────────────────────────────────────┘
```
```
**Why Rust?** The TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests.
**Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests.
**IPC Protocol:** TypeScript spawns the `ruststorage` binary with `--management` and communicates via newline-delimited JSON over stdin/stdout. Commands: `start`, `stop`, `createBucket`.
**IPC Protocol:** TypeScript spawns the `ruststorage` binary with `--management` and communicates via newline-delimited JSON over stdin/stdout. Commands: `start`, `stop`, `createBucket`, `clusterStatus`.
description:'A Node.js TypeScript package to create a local S3-compatible storage server using mapped local directories for development and testing purposes.'
description:'A Node.js TypeScript package to create a local S3-compatible storage server using mapped local directories for development and testing purposes.'
}
}
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
