@push.rocks/smartstorage

A high-performance, S3-compatible storage server powered by a Rust core with a clean TypeScript API. Runs standalone for dev/test — or scales out as a distributed, erasure-coded cluster with QUIC-based inter-node communication. No cloud, no Docker. Just install the package and go. 🚀

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why smartstorage?

Feature	smartstorage	MinIO	s3rver
Install	pnpm add	Docker / binary	npm install
Startup time	~20ms	seconds	~200ms
Large file uploads	Streaming, bounded memory	Yes	OOM risk
Range requests	Seek-based	Yes	Full read
Language	Rust + TypeScript	Go	JavaScript
Multipart uploads	✅ Full support	Yes	No
Auth	AWS SigV4 (full verification)	Full IAM	Basic
Bucket policies	IAM-style evaluation	Yes	No
Clustering	✅ Erasure-coded, QUIC	Yes	No
Multi-drive awareness	✅ Per-drive health	Yes	No

Core Features

• 🦀 Rust-powered HTTP server — hyper 1.x with streaming I/O, bounded buffering, backpressure
• 📦 Full S3-compatible API — works with AWS SDK v3, SmartBucket, any S3 client
• 💾 Filesystem-backed storage — buckets map to directories, objects to files
• 📤 Streaming multipart uploads — large files with bounded memory use
• 📐 Byte-range requests — seek() directly to the requested byte offset
• 🔐 AWS SigV4 authentication — full signature verification with constant-time comparison
• 📋 Bucket policies — IAM-style JSON policies with Allow/Deny evaluation and wildcard matching
• 🌐 CORS middleware — configurable cross-origin support
• 🧹 Clean slate mode — wipe storage on startup for test isolation
• 📊 Runtime storage stats — cheap bucket summaries and global counts without S3 list scans
• 🔑 Runtime credential rotation — list and replace active auth credentials without mutating internals
• 🧩 Bucket tenants — provision one scoped S3 credential per bucket with restart persistence
• ⚡ Test-first design — start/stop in milliseconds, no port conflicts

Clustering Features

• 🔗 Erasure coding — Reed-Solomon (configurable k data + m parity shards) for storage efficiency and fault tolerance
• 🚄 QUIC transport — multiplexed, encrypted inter-node communication via quinn with zero head-of-line blocking
• 💽 Multi-drive awareness — each node manages multiple independent storage paths with health monitoring
• 🩺 Cluster health introspection — query native node, drive, quorum, and healing status for product dashboards
• 🤝 Cluster membership — static seed config + runtime join, heartbeat-based failure detection
• ✍️ Quorum writes — data is only acknowledged after k+1 shards are persisted
• 📖 Quorum reads — reconstruct from any k available shards, local-first fast path
• 🩹 Self-healing — background scanner detects and reconstructs missing/corrupt shards

Installation

pnpm add @push.rocks/smartstorage -D

Note: The package ships with precompiled Rust binaries for linux_amd64 and linux_arm64. No Rust toolchain needed on your machine.

Quick Start

Standalone Mode (Dev & Test)

import { SmartStorage } from '@push.rocks/smartstorage';

// Start a local S3-compatible storage server
const storage = await SmartStorage.createAndStart({
  server: { port: 3000 },
  storage: { cleanSlate: true },
});

// Create a bucket
await storage.createBucket('my-bucket');

// Get connection details for any S3 client
const descriptor = await storage.getStorageDescriptor();
// → { endpoint: 'localhost', port: 3000, accessKey: 'STORAGE', accessSecret: 'STORAGE', useSsl: false }

// When done
await storage.stop();

Cluster Mode (Distributed)

Cluster mode requires the control root and every drive root to exist before startup. The process user must own them, no path component may be a symlink, and group/world write permission must be disabled.

import { SmartStorage } from '@push.rocks/smartstorage';

const storage = await SmartStorage.createAndStart({
  server: { port: 3000 },
  storage: {
    directory: '/var/lib/smartstorage/control',
    cleanSlate: false,
  },
  cluster: {
    enabled: true,
    nodeId: 'node-1',
    quicPort: 4000,
    seedNodes: ['192.168.1.11:4000', '192.168.1.12:4000'],
    erasure: {
      dataShards: 4,      // k: minimum shards to reconstruct data
      parityShards: 2,    // m: fault tolerance (can lose up to m shards)
    },
    drives: {
      paths: ['/mnt/disk1', '/mnt/disk2', '/mnt/disk3'],
    },
  },
});

Objects are automatically split into chunks (default 4 MB), erasure-coded into 6 shards (4 data + 2 parity), and distributed across drives/nodes. Any 4 of 6 shards can reconstruct the original data.

Configuration

All config fields are optional — sensible defaults are applied automatically.

import { SmartStorage, ISmartStorageConfig } from '@push.rocks/smartstorage';

const config: ISmartStorageConfig = {
  server: {
    port: 3000,              // Default: 3000
    address: '0.0.0.0',      // Default: '0.0.0.0'
    silent: false,           // Default: false
    region: 'us-east-1',     // Default: 'us-east-1' — used for SigV4 signing
  },
  storage: {
    directory: './my-data',  // Default: .nogit/bucketsDir
    cleanSlate: false,       // Default: false — set true to wipe on start
    pool: {                  // Optional explicit identity for this process' one pool
      id: 'fast-local',
      directory: './my-data',
      backend: { kind: 'localFs' },
    },
  },
  auth: {
    enabled: false,          // Default: false
    credentials: [{
      accessKeyId: 'MY_KEY',
      secretAccessKey: 'MY_SECRET',
    }],
  },
  cors: {
    enabled: false,          // Default: false
    allowedOrigins: ['*'],
    allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'],
    allowedHeaders: ['*'],
    exposedHeaders: ['ETag', 'x-amz-request-id', 'x-amz-version-id'],
    maxAge: 86400,
    allowCredentials: false,
  },
  logging: {
    level: 'info',           // 'error' | 'warn' | 'info' | 'debug'
    format: 'text',          // 'text' | 'json'
    enabled: true,
  },
  limits: {
    maxObjectSize: 5 * 1024 * 1024 * 1024, // 5 GB
    maxMetadataSize: 2048,
    requestTimeout: 300000,  // 5 minutes
  },
  multipart: {
    expirationDays: 7,             // Must be greater than zero
    cleanupIntervalMinutes: 60,    // Must be greater than zero
  },
  cluster: {                 // Optional — omit for standalone mode
    enabled: true,
    nodeId: 'node-1',        // Auto-generated UUID if omitted
    quicPort: 4000,          // Default: 4000
    seedNodes: [],           // Addresses of existing cluster members
    erasure: {
      dataShards: 4,         // Default: 4
      parityShards: 2,       // Default: 2
      chunkSizeBytes: 4194304, // Default: 4 MB
    },
    drives: {
      paths: ['/mnt/disk1', '/mnt/disk2'],
    },
    heartbeatIntervalMs: 5000,  // Default: 5000
    heartbeatTimeoutMs: 30000,  // Default: 30000
  },
};

const storage = await SmartStorage.createAndStart(config);

Storage Pools and Mounted-Filesystem Serving

One SmartStorage process serves one storage pool. Existing storage.directory configuration remains compatible and normalizes to a default localFs pool. For localFs, storage.directory and storage.pool.directory must resolve to the same path when both are supplied. For mountedFs, both values must instead be lexically exact absolute mountpoints and identical strings. A root is durably bound to its pool ID and cannot later be reassigned to a different pool.

For standalone localFs pools, SmartStorage creates a missing root with 0700 permissions. An existing root must be owned by the process user and must not be group- or world-writable; startup fails before opening the S3 listener when this contract is not met. SmartStorage never silently changes an existing root's ownership or permissions.

On Linux, standalone SmartStorage can serve an exact NFS or SMB mount. The configured directory itself must be the absolute mountpoint (not a symlink or canonical alias), expectedSource must contain a numeric IPv4 or IPv6 address, the client mount must enable the Linux nosymfollow VFS option, and cleanSlate is forbidden. Mounted pools are not supported in cluster mode. Onebox and other mount-provisioning clients must include nosymfollow in their NFS or CIFS mount options before SmartStorage starts. NFS mounts must keep server-coordinated locking enabled: nolock and local_lock=all|flock are rejected, while an absent local_lock option, local_lock=none, or local_lock=posix is accepted because SmartStorage uses flock(2) for its provider locks. CIFS mounts must not use nobrl. Accepted source forms are IP:/absolute/export for NFS and //IP/share[/path] for SMB. IPv6 addresses use brackets, for example [2001:db8::10]:/archive and //[2001:db8::10]/archive.

Startup opens the mountpoint once and confines every standalone storage, credential, policy, retention, fencing, and background operation beneath that descriptor through /proc/self/fd. It verifies the descriptor's mount ID, device, NFS/CIFS filesystem magic, and authoritative ST_NOSYMFOLLOW VFS flag, plus the effective mount and superblock locking options, then rereads the complete mount activation before serving. Every storage boundary revalidates the activation and ST_NOSYMFOLLOW. Any drift permanently poisons the process, fails storage-derived HTTP and management boundaries closed, never follows descendant symlinks, and never falls back to a host directory at the configured path. Lifecycle, liveness, cluster-status, and cached-diagnostic operations remain available for shutdown and diagnosis.

Every server mode stops accepting HTTP connections, gracefully drains each accepted connection, and joins independently owned request and multipart workers before releasing the mount descriptor or root-maintenance lock. The drain is bounded to 30 seconds. A timeout is reported as an incomplete stop: the Rust process retains the server, descriptors, locks, and unfinished task handles so the caller can retry stop() after hard-mounted I/O recovers. Streamed object reads move their file descriptor into the same owned worker lifecycle, so a disconnected client cannot detach a blocked filesystem read. Active HTTP request lifecycles are capped at 4,096; excess requests receive a retryable 503 Service Unavailable response with Retry-After: 1. A terminal shutdown result is reported only after all owned work is complete and remains available to retrying stop waiters until the management process terminates.

const storage = await SmartStorage.createAndStart({
  storage: {
    directory: '/mnt/archive',
    pool: {
      id: 'archive',
      directory: '/mnt/archive',
      backend: {
        kind: 'mountedFs',
        expectedFilesystemType: 'nfs',
        expectedSource: '192.0.2.10:/archive',
        // IPv6 is canonicalized as: [2001:db8::10]:/archive
      },
    },
  },
});

Serving startup also runs the root-ownership, same-filesystem, server-coordinated lock-contention, directory-fsync, and durable provider-identity checks through the descriptor anchor. A mounted runtime observation is productionEligible after these runtime checks pass and while the activation is unpoisoned. Atomic directory exchange is reported separately as an optional capability: NFS filesystems without it can still serve S3, atomic create-only uploads, fenced bucket ensure/delete, and migration cleanup, while atomic whole-bucket replacement and isolated restore remain unavailable and fail closed. Its remoteLocking check and filesystem.remoteLockingMode observation distinguish server-coordinated locking from unsafe local or disabled locking. getStoragePoolDiagnostic() remains available after poison and returns cached activation identity and poison evidence without accessing the storage filesystem.

Probe an existing exact mount without starting the S3 listener:

const observation = await SmartStorage.probeStoragePool({
  id: 'archive',
  directory: '/mnt/archive',
  backend: {
    kind: 'mountedFs',
    expectedFilesystemType: 'nfs',
    expectedSource: '192.0.2.10:/archive',
  },
});

console.log(observation.supported);
console.log(observation.productionEligible); // false: this read-only probe skips write semantics
console.log(observation.checks);

The read-only probe verifies the exact active mount, numeric source, descriptor-bound mount ID, device, filesystem magic, ST_NOSYMFOLLOW, and server-coordinated locking options, and activation stability. It reports observed block and transfer sizes but does not change mount tuning or write to the target. Mutating semantic checks remain notRun, so a successful diagnostic probe keeps supported at null and productionEligible at false; production eligibility is established only by serving startup.

Common Configurations

CI/CD testing — silent, clean, fast:

const storage = await SmartStorage.createAndStart({
  server: { port: 9999, silent: true },
  storage: { cleanSlate: true },
});

Auth enabled:

const storage = await SmartStorage.createAndStart({
  auth: {
    enabled: true,
    credentials: [{ accessKeyId: 'test', secretAccessKey: 'test123' }],
  },
});

CORS for local web dev:

const storage = await SmartStorage.createAndStart({
  cors: {
    enabled: true,
    allowedOrigins: ['http://localhost:5173'],
    allowCredentials: true,
  },
});

Runtime Credentials

const credentials = await storage.listCredentials();

await storage.replaceCredentials([
  {
    accessKeyId: 'ADMINA',
    secretAccessKey: 'super-secret-a',
  },
  {
    accessKeyId: 'ADMINB',
    secretAccessKey: 'super-secret-b',
  },
]);

interface IStorageCredential {
  accessKeyId: string;
  secretAccessKey: string;
  bucketName?: string;
  region?: string;
}

• listCredentials() returns the Rust core's current runtime credential set.
• replaceCredentials() swaps the full set atomically and persists it under the storage root. On success, new requests use the new set immediately and the old credentials stop authenticating immediately.
• Requests that were already authenticated before the replacement keep running; auth is evaluated when each request starts.
• No restart is required, and runtime-created credentials survive restart unless storage.cleanSlate clears the bounded storage contents; the owned root, maintenance-lock inode, and any durable provider/pool identity are preserved.
• Replacement input must contain at least one credential, each accessKeyId and secretAccessKey must be non-empty, and accessKeyId values must be unique.

Bucket Tenants

Bucket tenants are designed for platform services that need one bucket and one scoped S3 credential per app. Tenant credentials are enforced by the auth layer before the normal bucket-policy/default-auth pipeline, so a scoped credential cannot list all buckets or access another bucket even when it has a valid SigV4 signature.

const tenant = await storage.createBucketTenant({
  bucketName: 'workapp-123',
});

// Directly usable by AWS SDK v3 or env injection
const client = new S3Client({
  endpoint: `http://${tenant.endpoint}:${tenant.port}`,
  region: tenant.region,
  credentials: {
    accessKeyId: tenant.accessKeyId,
    secretAccessKey: tenant.secretAccessKey,
  },
  forcePathStyle: true,
});

console.log(tenant.env.S3_BUCKET);
console.log(tenant.env.AWS_ACCESS_KEY_ID);

await storage.rotateBucketTenantCredentials({ bucketName: 'workapp-123' });
await storage.deleteBucketTenant({ bucketName: 'workapp-123', accessKeyId: tenant.accessKeyId });
const descriptor = await storage.getBucketTenantDescriptor({ bucketName: 'workapp-123' });
const tenants = await storage.listBucketTenants();

• createBucketTenant() creates the bucket if needed and stores a scoped credential for that bucket.
• rotateBucketTenantCredentials() replaces the active scoped credential for the bucket and persists the new credential.
• deleteBucketTenant({ bucketName, accessKeyId }) revokes one scoped credential and keeps the bucket.
• deleteBucketTenant({ bucketName }) revokes scoped credentials for an existing tenant bucket and deletes that bucket's contents recursively.
• Tenant credentials can list, read, write, and delete objects in their assigned bucket, but cannot list all buckets, access other buckets, copy from other buckets, delete buckets, or mutate bucket policies.
• Bucket tenant APIs require auth.enabled: true.

Bucket Backup/Restore

const appBackup = await storage.exportBucket({ bucketName: 'workapp-123' });
await storage.importBucket({ bucketName: 'workapp-123-restore', source: appBackup });

• exportBucket() returns a self-contained smartstorage.bucket.v1 JSON export with only the selected bucket's objects and object metadata. The legacy whole-bucket management export is intentionally bounded to 10,000 objects, 48 MiB of payload, and 8 MiB of serialized keys plus metadata; use the migration transfer surface for larger buckets.
• importBucket() validates object payload size and MD5 before creating the target bucket if needed, then restores the exported objects into that bucket.
• Exports do not include credentials, policies, or unrelated tenant data.

Bucket Migration Control

Standalone storage exposes durable, provider-bound migration control records. Discover the local provider identity, receipt key, and cleanup support with getBucketMigrationCapability(). Create each side with createSourceMigration() and createDestinationMigration(), then advance only the explicit source (active → sealed → cutoverCommitted → retired) and destination (holding → verified → cutoverAuthorized → published) paths. Before cutover, a destination aborted terminal receipt can release a sealed source. Every call binds a positive fence token, exact migration binding, and opaque controller capability; exact same-token retries replay the durable receipt while stale or mismatched requests fail.

Source creation additionally requires sourceCleanupAccessKeyId. SmartStorage verifies that it is the bucket's one exact scoped credential, durably binds its hash before creating migration state, and revalidates that ownership before every source phase transition and before persisting a cleanup request. A provider can have only one nonterminal migration per role and logical bucket; terminal records remain as history and do not prevent a later migration ID. The standalone registry has a finite capacity of 100,000 migration states. Terminal receipts and completed-cleanup tombstones continue to consume that capacity because they are retained permanently; controllers must treat migration IDs as lifecycle records and capacity-plan before exhaustion. SmartStorage serializes registry creation globally and atomically rejects a new migration ID when 100,000 states already exist. Exact replays of existing IDs remain available at the bound; the provider never deletes history or permits manual registry-file reuse to make capacity. Registry enumeration budgets durable locks, canonical crash-left atomic temporary files, and unrecognized junk separately. This leaves bounded recovery headroom at full state capacity without deleting a temporary file another process may still own; every partition remains finite and fails closed when its own limit is exceeded.

Source sealing drains bucket mutations and blocks later object, multipart, policy, bucket, tenant, and fenced mutation paths. Reads, heads, listings, policy reads, and multipart listings remain available while sealed, but they hold the bucket data-plane read guard so a later retirement cleanup drains every accepted source read before deletion. A retained bucket cannot become a migration source.

After the destination is durably published and the source is durably retired, the controller can permanently retire the source data:

const sourceState = await source.inspectBucketMigration({ migrationId });
const destinationState = await destination.inspectBucketMigration({ migrationId });

const cleanup = await source.cleanupSourceMigration({
  binding,
  cleanupFenceToken: 1,
  opaqueCapability: sourceCapability,
  sourceRetiredReceiptSha256: sourceState!.terminalReceipt!.receiptSha256,
  destinationPublishedReceipt: destinationState!.terminalReceipt!,
  resourceFence: {
    version: 1,
    scopeId: `${binding.authorityId}.${binding.logicalBucket}`,
    token: 42,
    mutationId: `${migrationId}.source-cleanup`,
    payloadSha256: cleanupIntentSha256,
  },
  accessKeyId: sourceTenant.accessKeyId,
});

cleanupSourceMigration() first persists a permanent requested cleanup subrecord that binds the exact retired-source receipt, fully verified destination-published receipt, cleanup token, resource fence, and hashed access key. It then reuses exact fenced deletion to remove bucket data, every matching multipart upload, the exact credential, and the bucket policy before persisting the canonical cleanup receipt and generic delete-result digest. Multipart deletion immediately reconciles the runtime admission counter and cleanup queue. The lock order drains reads and mutations before credential revocation.

The controller owns recovery: SmartStorage never starts source deletion automatically. Retry the exact cleanupSourceMigration() request after any timeout, process exit, or ambiguous response. A requested-only retry repeats the exact resource deletion; if deletion was already durably complete, its generic receipt is replayed and the migration cleanup record is finalized. After completion the request cannot be rebound, the bucket can never be recreated or credentialed, and unrelated global credential replacement resumes. Generic bucket deletion and multipart abort remain blocked throughout the migration seal.

inspectBucketMigration() returns the durable phase, terminal receipts, and the optional sourceCleanup requested/completed subrecord. Terminal receipts and completed cleanup receipts remain available across restart. storage.cleanSlate: true cannot erase or bypass that history: standalone startup is refused whenever any durable migration state exists, including a terminal receipt or completed-cleanup tombstone.

Durable Resource Fencing

Standalone storage with cleanSlate: false supports monotonic, crash-safe bucket replacement and deletion. The legacy immediate mode (holdPublication omitted or false) still requires callers to drain normal writes before issuing a fenced mutation.

const fence = {
  version: 1 as const,
  scopeId: 'corestore-node-1.workapp-123',
  token: 42,
  mutationId: 'delete-workapp-123',
  payloadSha256: 'a'.repeat(64),
};

const result = await storage.deleteBucket({
  bucketName: 'workapp-123',
  accessKeyId: tenant.accessKeyId,
  fence,
});

For control planes that cannot atomically publish the provider result, request a durable write-publication hold:

const held = await storage.importBucket({
  bucketName: 'workapp-123',
  source: appBackup,
  accessKeyId: tenant.accessKeyId,
  fence,
  holdPublication: true,
});

// Persist the provider result and update the control-plane resource first.
const release = await storage.commitResourcePublication({
  barrier: held.resourcePublicationBarrier!,
  resourceFenceReceipt: held.resourceFence!,
});

holdPublication: true is part of the effective mutation identity. SmartStorage drains active target-bucket mutations before the destructive provider operation, then keeps all new target mutations behind a durable barrier after the provider effects and receipt are fully persisted. Mutations include object, multipart, bucket, policy, tenant, and credential changes. They receive retryable S3 WritePublicationHeld (503, Retry-After: 1) or EFENCE_PUBLICATION_HELD through the management API. Unrelated buckets remain available. Reads are not blocked, so this protocol controls write publication; it does not hide newly restored data from readers.

The exact mutation replay returns the same barrier. Different and higher-token mutations cannot clear it. commitResourcePublication() validates the complete barrier, the canonical fence receipt, and its 256-bit opaque commit capability; it is safe to retry and returns the same release receipt. Treat commitCapability as a secret: do not log it or persist it outside protected control-plane state. A successful commit removes the raw capability from provider state and retains only its hash.

There is no timeout, shutdown release, or administrative bypass. A held barrier survives restart and is restored before the listener accepts requests. Invalid, unknown, oversized, or insecure publication state fails startup closed. replaceCredentials() is also rejected while any publication remains held. Capability discovery keeps requiresDrain: true for legacy immediate mode and reports publicationHoldSupported: true, publicationHoldVersion: 1, and publicationHoldRequiresExternalDrain: false when the provider can enforce the hold internally.

When supplied, the exact accessKeyId is bound into the durable mutation identity. Its delete receipt is committed only after bucket data is durably absent, that exact bucket-scoped credential is durably revoked, and the bucket policy is durably absent. Retrying the same fence returns the stored receipt. Recovery after a process crash repeats the credential and policy cleanup idempotently, so deletedCredentials can be 0; a higher fence against an already absent bucket can likewise return bucketDeleted: false. Completed exact deletes return credentialsPreserved: false and policyPreserved: false.

For backward compatibility, omitting accessKeyId retains the original fenced data-only delete identity and receipt: tenant credentials and bucket policy are preserved, and both preservation fields are true.

After an exact delete, a new higher-token authority can recreate the same target without removing its durable fence history. Call ensureBucketTenant() with the new fence first, then issue an exact fenced importBucket() with the same resource token. Ensure receipts use the smartstorage.bucket.ensure.v1 profile and attest the applied access key, provider-computed secret hash, and effective region; the secret itself is never written to the receipt or fencing registry.

computeSmartStorageResourceFenceEffectivePayloadSha256V1() computes the same canonical effective payload digest as the Rust provider for replacement, exact-delete, and ensure profiles. A completed fenced ensure returns its receipt on descriptor.resourceFence.

Absent-bucket fenced ensure removes any residual policy before creating the new resource; existing-bucket ensure preserves policy and object data. Once a bucket has durable fence history, legacy createBucket(), createBucketTenant(), rotateBucketTenantCredentials(), deleteBucketTenant(), and unfenced importBucket() mutations fail with EFENCE_REQUIRED. Use fenced ensure, exact import, and exact delete for its lifecycle.

replaceCredentials() is a trusted process-local administrative override, not a fenced lifecycle API. If it changes exact-owned credential material, completed receipt replay detects the drift and fails closed with an ownership or durable state error.

Health and Metrics APIs

const health = await storage.getHealth();
const metrics = await storage.getMetrics();

• getHealth() reports running state, storage directory and pool observation, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
• getMetrics() returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, and cluster-enabled metrics.

Runtime Stats

const stats = await storage.getStorageStats();
const bucketSummaries = await storage.listBucketSummaries();

console.log(stats.bucketCount);
console.log(stats.totalObjectCount);
console.log(stats.totalStorageBytes);
console.log(bucketSummaries[0]?.name, bucketSummaries[0]?.objectCount);

interface IBucketSummary {
  name: string;
  objectCount: number;
  totalSizeBytes: number;
  creationDate?: number;
}

interface IStorageLocationSummary {
  path: string;
  totalBytes?: number;
  availableBytes?: number;
  usedBytes?: number;
  pool?: IStoragePoolObservation;
}

interface IStorageStats {
  bucketCount: number;
  totalObjectCount: number;
  totalStorageBytes: number;
  buckets: IBucketSummary[];
  storageDirectory: string;
  storageLocations?: IStorageLocationSummary[];
}

• bucketCount, totalObjectCount, totalStorageBytes, and per-bucket totals are logical object stats maintained by the Rust runtime. They count object payload bytes, not sidecar files or erasure-coded shard overhead.
• smartstorage initializes these values from native on-disk state at startup, then keeps them in memory and updates them when bucket/object mutations succeed. Stats reads do not issue S3 ListObjects or rescan every object.
• Values are exact for mutations performed through smartstorage after startup. Direct filesystem edits outside smartstorage are not watched; restart the server to resync.
• storageLocations is a cheap filesystem-capacity snapshot. Standalone mode reports the storage directory plus its pool observation. Cluster mode reports the configured drive paths.

Cluster Health

const clusterHealth = await storage.getClusterHealth();

if (!clusterHealth.enabled) {
  console.log('Cluster mode is disabled');
} else {
  console.log(clusterHealth.nodeId, clusterHealth.quorumHealthy);
  console.log(clusterHealth.peers);
  console.log(clusterHealth.drives);
}

interface IClusterHealth {
  enabled: boolean;
  nodeId?: string;
  quorumHealthy?: boolean;
  majorityHealthy?: boolean;
  peers?: IClusterPeerHealth[];
  drives?: IClusterDriveHealth[];
  erasure?: IClusterErasureHealth;
  repairs?: IClusterRepairHealth;
}

• getClusterHealth() is served by the Rust core. The TypeScript wrapper does not infer values from static config.
• Standalone mode returns { enabled: false }.
• Peer status is the local node's current view of cluster membership and heartbeats, so it is best-effort and may lag real network state.
• Drive health is based on live native probe checks on the configured local drive paths. Capacity values are cheap filesystem snapshots.
• quorumHealthy means the local node currently sees majority quorum and enough available placements in every erasure set to satisfy the configured write quorum.
• Repair fields expose the background healer's currently available runtime state. They are best-effort and limited to what the engine tracks today, such as whether a scan is active, the last completed run, and the last error.

Usage with AWS SDK v3

import { S3Client, PutObjectCommand, GetObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';

const descriptor = await storage.getStorageDescriptor();

const client = new S3Client({
  endpoint: `http://${descriptor.endpoint}:${descriptor.port}`,
  region: 'us-east-1',
  credentials: {
    accessKeyId: descriptor.accessKey,
    secretAccessKey: descriptor.accessSecret,
  },
  forcePathStyle: true,  // Required for path-style access
});

// Upload
await client.send(new PutObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
  Body: 'Hello, Storage!',
  ContentType: 'text/plain',
}));

// Download
const { Body } = await client.send(new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
}));
const content = await Body.transformToString(); // "Hello, Storage!"

// Delete
await client.send(new DeleteObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
}));

Usage with SmartBucket

import { SmartBucket } from '@push.rocks/smartbucket';

const smartbucket = new SmartBucket(await storage.getStorageDescriptor());
const bucket = await smartbucket.createBucket('my-bucket');
const dir = await bucket.getBaseDirectory();

// Upload
await dir.fastPut({ path: 'docs/readme.txt', contents: 'Hello!' });

// Download
const content = await dir.fastGet('docs/readme.txt');

// List
const files = await dir.listFiles();

Standalone SmartStorage honors If-None-Match: * on both PutObject and CompleteMultipartUpload, publishes through an atomic no-replace operation, and implements bounded ListParts responses so exact-streaming clients can verify multipart abort cleanup. Conditional create-only writes and active ListParts fail closed in cluster mode until those semantics have complete durable cluster metadata. Descriptor-confined standalone startup reclaims abandoned private object materializations from interrupted create-only, copy, or multipart publication without removing an already published hard link.

Standalone DeleteObject also honors If-Match: * and comma-separated strong ETags. It compares the current ETag and deletes inside the same bucket mutation gate used by PUT, COPY, and multipart completion. A mismatch returns the S3 PreconditionFailed response without changing object or runtime-stat state; an absent conditional target returns NoSuchKey. This supplies the conditional delete semantics used by SmartBucket's verified exact-path purge capability. Conditional deletion returns NotImplemented in cluster mode until the precondition can be enforced across cluster nodes.

AWS SDK streaming PutObject and UploadPart requests are decoded before storage. SmartStorage verifies the declared decoded length and CRC32 trailer for STREAMING-UNSIGNED-PAYLOAD-TRAILER, and verifies the complete chained signature sequence for STREAMING-AWS4-HMAC-SHA256-PAYLOAD. The aws-chunked transport encoding is not retained as object metadata. Every aws-chunked request requires a verified SigV4 envelope, even when auth.enabled is false; ordinary non-streaming SDK requests retain the configured auth-disabled behavior. Unsupported, malformed, truncated, checksum-invalid, or signature-invalid bodies fail closed. Standalone overwrites use private same-filesystem staging and are atomically published only after terminal verification succeeds. Body verification failures and cancellation before publication preserve the previous object. Decoder memory is globally bounded, and limits.requestTimeout (milliseconds) bounds inactivity while waiting for each incoming body frame; timeout or memory-pressure failures publish no partial object.

Multipart Uploads

For files larger than 5 MB, use multipart uploads. smartstorage handles them with bounded-memory streaming I/O. AWS streaming transport chunks are decoded through per-chunk buffers of at most 16 MiB under a shared 64 MiB decoder budget before the part data is written. In cluster mode, each part is independently erasure-coded and distributed.

import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
} from '@aws-sdk/client-s3';

// 1. Initiate
const { UploadId } = await client.send(new CreateMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
}));

// 2. Upload parts
const parts = [];
for (let i = 0; i < chunks.length; i++) {
  const { ETag } = await client.send(new UploadPartCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    UploadId,
    PartNumber: i + 1,
    Body: chunks[i],
  }));
  parts.push({ PartNumber: i + 1, ETag });
}

// 3. Complete
await client.send(new CompleteMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
  UploadId,
  MultipartUpload: { Parts: parts },
}));

Bucket Policies

smartstorage supports AWS-style bucket policies for fine-grained access control. Policies use the same IAM JSON format as real S3 — so you can develop and test your policy logic locally before deploying.

When auth.enabled is true, the auth pipeline works as follows:

1. Authenticate — verify the AWS SigV4 signature (anonymous requests skip this step)
2. Authorize — evaluate bucket policies against the request action, resource, and caller identity
3. Default — authenticated users get full access; anonymous requests are denied unless a policy explicitly allows them

Setting a Bucket Policy

import { PutBucketPolicyCommand } from '@aws-sdk/client-s3';

// Allow anonymous read access to all objects in a bucket
await client.send(new PutBucketPolicyCommand({
  Bucket: 'public-assets',
  Policy: JSON.stringify({
    Version: '2012-10-17',
    Statement: [{
      Sid: 'PublicRead',
      Effect: 'Allow',
      Principal: '*',
      Action: ['s3:GetObject'],
      Resource: ['arn:aws:s3:::public-assets/*'],
    }],
  }),
}));

Policy Features

• Effect: Allow and Deny (explicit Deny always wins)
• Principal: "*" (everyone) or { "AWS": ["arn:..."] } for specific identities
• Action: IAM-style actions like s3:GetObject, s3:PutObject, s3:*, or prefix wildcards like s3:Get*
• Resource: ARN patterns with * and ? wildcards (e.g. arn:aws:s3:::my-bucket/*)
• Persistence: Policies survive server restarts — stored as JSON on disk alongside your data

Policy CRUD Operations

Operation	AWS SDK Command	HTTP
Get policy	GetBucketPolicyCommand	GET /{bucket}?policy
Set policy	PutBucketPolicyCommand	PUT /{bucket}?policy
Delete policy	DeleteBucketPolicyCommand	DELETE /{bucket}?policy

Deleting a bucket automatically removes its associated policy.

Clustering Deep Dive 🔗

smartstorage can run as a distributed storage cluster where multiple nodes cooperate to store and retrieve data with built-in redundancy.

How It Works

Client ──HTTP PUT──▶ Node A (coordinator)
                       │
                       ├─ Split object into 4 MB chunks
                       ├─ Erasure-code each chunk (4 data + 2 parity = 6 shards)
                       │
                       ├──QUIC──▶ Node B (shard writes)
                       ├──QUIC──▶ Node C (shard writes)
                       └─ Local disk (shard writes)

1. Any node can coordinate — the client connects to any cluster member
2. Objects are chunked — large objects split into fixed-size pieces (default 4 MB)
3. Each chunk is erasure-coded — Reed-Solomon produces k data + m parity shards
4. Shards are distributed — placed across different nodes and drives for fault isolation
5. Quorum guarantees consistency — writes need k+1 acks, reads need k shards

Cluster Root and Upgrade Requirements

Cluster mode never creates configured control or drive roots. Before startup, create storage.directory and every cluster.drives.paths entry, make them owned by the SmartStorage process user, and remove group/world write permission. Every configured root must be a real directory with no symlink path component, and every drive root must resolve to a distinct device/inode identity. The control root may be distinct or may be the same identity as one drive root; it cannot make two drive entries aliases of one another. Every root must support the filesystem durability and locking semantics validated at startup. SmartStorage fails closed before accepting traffic when a root does not meet these requirements.

This release migrates clustered control records, manifests, and shards to descriptor-anchored, digest-addressed formats with durable tombstones. Legacy records are migrated in bounded steps and the new records become authoritative. Before a manifest listing can issue a continuation token, SmartStorage drains all remaining legacy manifests through batches retaining at most 32 MiB, so a later migration cannot introduce a key behind an issued token. Background healing can stop that drain between committed record migrations during shutdown. Upgrade every member during one coordinated maintenance window. Once any member has written or migrated the new state, do not restart an older binary and do not run a mixed-version cluster. Back up the cluster roots before upgrading; rollback requires restoring those roots from the pre-upgrade backup.

Erasure Coding

With the default 4+2 configuration:

• Storage overhead: 33% (vs. 200% for 3x replication)
• Fault tolerance: any 2 drives/nodes can fail simultaneously
• Read efficiency: only 4 of 6 shards needed to reconstruct data

Config	Total Shards	Overhead	Tolerance	Min Nodes
4+2	6	33%	2 failures	3
6+3	9	50%	3 failures	5
2+1	3	50%	1 failure	2

QUIC Transport

Inter-node communication uses QUIC via the quinn library:

• 🔒 Built-in TLS — self-signed certs auto-generated at cluster init
• 🔀 Multiplexed streams — concurrent shard transfers without head-of-line blocking
• ⚡ Connection pooling — persistent connections to peer nodes
• 🌊 Natural backpressure — QUIC flow control prevents overloading slow peers

Cluster Membership

• Static seed nodes — initial cluster defined in config
• Runtime join — new nodes can join a running cluster
• Heartbeat monitoring — every 5s (configurable), with suspect/offline detection
• Split-brain prevention — nodes only mark peers offline when they have majority

Self-Healing

A background scanner periodically (default: every 24h):

1. Checks shard checksums (CRC32C) for bit-rot detection
2. Identifies shards on offline nodes
3. Reconstructs missing shards from remaining data using Reed-Solomon
4. Places healed shards on healthy drives

Healing runs at low priority to avoid impacting foreground I/O.

Erasure Set Formation

Drives are organized into fixed erasure sets at cluster initialization:

3 nodes × 4 drives each = 12 drives total
With 6-shard erasure sets → 2 erasure sets

Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1
Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3

Drives are interleaved across nodes for maximum fault isolation. New nodes form new erasure sets — existing data is never rebalanced.

Testing Integration

import { SmartStorage } from '@push.rocks/smartstorage';
import { tap, expect } from '@git.zone/tstest/tapbundle';

let storage: SmartStorage;

tap.test('setup', async () => {
  storage = await SmartStorage.createAndStart({
    server: { port: 4567, silent: true },
    storage: { cleanSlate: true },
  });
});

tap.test('should store and retrieve objects', async () => {
  await storage.createBucket('test');
  // ... your test logic using AWS SDK or SmartBucket
});

tap.test('teardown', async () => {
  await storage.stop();
});

export default tap.start();

API Reference

SmartStorage Class

static createAndStart(config?: ISmartStorageConfig): Promise<SmartStorage>

Create and start a server in one call.

static probeStoragePool(pool: IMountedStoragePoolConfig): Promise<IStoragePoolObservation>

Inspect an exact host-mounted NFS or SMB pool without writing to it or starting the S3 listener. The observation reports the active mount source, filesystem type, capacity/tuning metadata, stability checks, and current production eligibility.

start(): Promise<void>

Spawn the Rust binary and start the HTTP server.

stop(): Promise<void>

Gracefully stop the server and then terminate the Rust process. In every server mode, an incomplete 30-second drain rejects this call without terminating the process; call stop() again after the blocked operation recovers. A terminal shutdown error is reported only after the completed Rust server has released its task and resource ownership; the wrapper terminates the management process before rejecting with that error.

The Rust management process reads stdin on a dedicated task. Ordinary commands enter a bounded nonblocking queue and continue to execute one at a time, while stop uses an independent retained lifecycle signal plus bounded waiters. The first stop intent immediately closes management and HTTP admission and signals HTTP, multipart, and clustered background producers even when an active mounted filesystem operation is blocked. Already queued ordinary commands are rejected as stopping. A stop waiter that reaches its deadline receives one incomplete error and is never sent a later completion; a retry uses a new request ID and receives the retained terminal result when ownership completes.

IPC EOF requests the same graceful shutdown. If an owned blocked operation cannot finish by the EOF ownership deadline, the Rust process exits nonzero instead of returning through runtime teardown that could wait forever on a blocked native filesystem worker. Output is serialized through a dedicated writer with a 144 MiB per-line ceiling, a 160 MiB owned queue budget, and a five-second Unix stdout write deadline; writer failure or sustained stdout backpressure fails closed rather than growing memory without bound.

getStoragePoolDiagnostic(): Promise<IStoragePoolObservation | null>

Return the active pool observation. For a poisoned mounted pool, this uses cached activation evidence and remains available without touching the storage filesystem. It returns null before the Rust server has started.

createBucket(name: string): Promise<{ name: string }>

Create a storage bucket.

createBucketTenant(options): Promise<IBucketTenantDescriptor>

Create a bucket tenant with a generated or supplied scoped credential. Options: { bucketName, accessKeyId?, secretAccessKey?, region? }.

ensureBucketTenant(options): Promise<IBucketTenantDescriptor>

Atomically create an absent bucket and exact scoped credential, resume an exact orphan credential, or rotate the credential for an existing exact-owned bucket. Foreign or ambiguous ownership is rejected before mutation. Options: { bucketName, accessKeyId, secretAccessKey, region?, fence? }. Once a bucket has durable fence history, supply fence; the returned descriptor then includes the durable resourceFence receipt.

deleteBucketTenant(options): Promise<void>

Revoke a tenant credential or delete a bucket that still has tenant credentials. Options: { bucketName, accessKeyId? }.

rotateBucketTenantCredentials(options): Promise<IBucketTenantDescriptor>

Replace the scoped credential for a bucket tenant. Options: { bucketName, accessKeyId?, secretAccessKey?, region? }.

listBucketTenants(): Promise<IBucketTenantMetadata[]>

List scoped tenant credential metadata without returning secrets.

getBucketTenantDescriptor(options): Promise<IBucketTenantDescriptor>

Return endpoint, port, region, bucket, access key, secret key, SSL flag, legacy descriptor fields, and S3/AWS env values for the bucket tenant.

getBucketMigrationCapability(): Promise<ISmartStorageBucketMigrationCapability>

Return standalone provider identity and receipt-key evidence plus source seal, destination publication, and controller-retried source-cleanup capabilities. Cluster mode reports this migration metadata surface as unsupported.

createSourceMigration(options): Promise<ISmartStorageMigrationTransitionReceipt>

Create durable source migration ownership for an existing non-retained bucket. sourceCleanupAccessKeyId is required and must name the bucket's one exact scoped credential; its hash is permanently bound to the migration.

createDestinationMigration(options): Promise<ISmartStorageMigrationTransitionReceipt>

Create durable destination migration ownership for an existing bucket.

transitionSourceMigration(options): Promise<ISmartStorageMigrationTransitionReceipt>

Advance the source across an allowed phase edge. Sealing introduces the manifest digest; releasing requires the exact signed destination-abort receipt.

transitionDestinationMigration(options): Promise<ISmartStorageMigrationTransitionReceipt>

Advance the destination across an allowed phase edge. Verification introduces the manifest digest and terminal publication produces a signed receipt.

cleanupSourceMigration(options): Promise<ISmartStorageSourceMigrationCleanupReceipt>

Permanently delete a retired source after verifying the exact signed destination-published receipt. The request includes the migration binding, cleanup fence token, source capability, retired receipt hash, published receipt, exact resource fence, and exact source accessKeyId. Exact retries return the same durable cleanup receipt.

inspectBucketMigration(options): Promise<ISmartStorageMigrationInspection | null>

Return durable migration, terminal-receipt, and source-cleanup state for one migrationId.

exportBucket(options): Promise<IBucketExport>

Export one bucket's objects and metadata into a smartstorage.bucket.v1 JSON object, subject to the bounded legacy management-export limits documented above.

importBucket(options): Promise<ISmartStorageBucketMutationResult>

Import a smartstorage.bucket.v1 JSON object into the target bucket after validating object size and MD5. An exact fenced import binds the replacement to the bucket's exclusive credential. Options: { bucketName, source, accessKeyId?, fence?, holdPublication? }. Setting holdPublication: true requires a fence and returns resourcePublicationBarrier; omitted or false preserves immediate behavior.

deleteBucket(options): Promise<ISmartStorageBucketMutationResult>

Delete a fenced bucket. Supplying accessKeyId also durably revokes that exact tenant credential and deletes the policy; omitting it preserves the legacy credential/policy behavior. Options: { bucketName, accessKeyId?, fence, holdPublication? }. Setting holdPublication: true returns a durable resourcePublicationBarrier; omitted or false preserves immediate behavior.

commitResourcePublication(options): Promise<ISmartStorageResourcePublicationReleaseReceipt>

Release a durable write-publication hold after the caller has persisted and published the exact provider receipt. Options: { barrier, resourceFenceReceipt }. The operation is exact-match and idempotent.

getResourceFencingCapability(): Promise<ISmartStorageResourceFencingCapability>

Report resource-fencing support, drain requirements, publication-hold version, whether publication holds are supported, and whether the provider must perform an additional external drain.

getStorageDescriptor(options?): Promise<IS3Descriptor>

Get connection details for S3-compatible clients. Returns:

Field	Type	Description
endpoint	string	Server hostname (localhost by default)
port	number	Server port
accessKey	string	Access key from first configured credential
accessSecret	string	Secret key from first configured credential
useSsl	boolean	Always false (plain HTTP)

getStorageStats(): Promise<IStorageStats>

Read cached logical bucket and object totals from the Rust runtime without issuing S3 list calls.

listBucketSummaries(): Promise<IBucketSummary[]>

Get per-bucket logical object counts and total payload sizes.

listCredentials(): Promise<IStorageCredentialMetadata[]>

Return metadata for the currently active runtime credential set without secretAccessKey values.

replaceCredentials(credentials: IStorageCredential[]): Promise<void>

Atomically replace the active runtime credential set without restarting the server.

getClusterHealth(): Promise<IClusterHealth>

Read the Rust core's current cluster, drive, quorum, and repair health snapshot. Standalone mode returns { enabled: false }.

getHealth(): Promise<ISmartStorageHealth>

Return running state, storage directory, per-location pool observations, auth state, credential counts, bucket count, object count, total bytes, cluster health, resource-fencing capability, and publication-hold capability fields.

getMetrics(): Promise<ISmartStorageMetrics>

Return numeric metrics plus a Prometheus text snippet for operational scraping.

Architecture

smartstorage uses a hybrid Rust + TypeScript architecture:

┌──────────────────────────────────────────────┐
│  Your Code (AWS SDK, SmartBucket, etc.)       │
│  ↕ HTTP (localhost:3000)                     │
├──────────────────────────────────────────────┤
│  ruststorage binary (Rust)                    │
│  ├─ hyper 1.x HTTP server                   │
│  ├─ S3 path-style routing                   │
│  ├─ StorageBackend (Standalone or Clustered) │
│  │   ├─ FileStore (single-node mode)        │
│  │   └─ DistributedStore (cluster mode)     │
│  │       ├─ ErasureCoder (Reed-Solomon)     │
│  │       ├─ ShardStore (per-drive storage)  │
│  │       ├─ QuicTransport (quinn)           │
│  │       ├─ ClusterState & Membership       │
│  │       └─ HealingService                  │
│  ├─ SigV4 auth + policy engine              │
│  ├─ CORS middleware                          │
│  └─ S3 XML response builder                 │
├──────────────────────────────────────────────┤
│  TypeScript (thin IPC wrapper)               │
│  ├─ SmartStorage class                       │
│  ├─ RustBridge (stdin/stdout JSON IPC)       │
│  └─ Config & S3 descriptor                  │
└──────────────────────────────────────────────┘

Why Rust? The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves these with bounded-memory streaming I/O, backpressure, and direct seek() for range requests.

IPC Protocol: TypeScript communicates with the ruststorage binary over newline-delimited JSON via stdin/stdout. The current management commands are probeStoragePool, getStoragePoolDiagnostic, start, stop, createBucket, createBucketTenant, ensureBucketTenant, deleteBucketTenant, rotateBucketTenantCredentials, listBucketTenants, getBucketTenantCredential, getBucketRetentionCapability, getBucketRetentionReceipt, getBucketMigrationCapability, createSourceMigration, createDestinationMigration, transitionSourceMigration, transitionDestinationMigration, cleanupSourceMigration, inspectBucketMigration, exportBucket, importBucket, deleteBucket, commitResourcePublication, getResourceFencingCapability, getStorageStats, listBucketSummaries, listCredentials, replaceCredentials, and getClusterHealth. getStoragePoolDiagnostic reads a weak, cached diagnostic source and remains responsive while another management command owns the server; it does not retain the mounted root after server ownership completes.

S3-Compatible Operations

Bucket names are validated consistently at HTTP, management, policy, and storage boundaries: they must be 3-63 lowercase letters, digits, dots, or hyphens, begin and end with a letter or digit, contain no consecutive dots, and not be formatted as an IPv4 address.

Object keys remain S3-compatible UTF-8 strings of up to 1,024 bytes, including leading, repeated, embedded, and trailing / characters. On disk, every new object key is encoded below the reserved .smartstorage-objects-v2 bucket directory as canonical, prefix-free lowercase-hex frames. Frames and fixed payload/sidecar names stay within the common 255-byte NAME_MAX, remain safe on CIFS, and listings return the exact original key.

Existing v7 identity-layout objects remain readable on local and NFS storage when their raw path can be reconstructed without traversal. Their first mutation stages and publishes the canonical v2 object set before removing the legacy payload and sidecars. The migration uses descriptor-confined, digest-bound durable state and reconciles every preparation, publication, and partial-cleanup crash on startup or next access. Canonical and legacy payloads for the same key without that exact owned state, noncanonical entries below the reserved v2 root, and unsafe legacy aliases on case-folding CIFS fail closed instead of choosing an ambiguous object.

The standalone v2 object-layout upgrade is one-way. After the first canonical v2 object write or legacy-object migration in a standalone storage root, downgrading that root to SmartStorage 7.0.0 or earlier is unsupported because those versions cannot read the canonical v2 objects.

Operation	Method	Path
ListBuckets	GET /
CreateBucket	PUT /{bucket}
DeleteBucket	DELETE /{bucket}
HeadBucket	HEAD /{bucket}
ListObjects (v1/v2)	GET /{bucket}	?list-type=2 for v2
PutObject	PUT /{bucket}/{key}
GetObject	GET /{bucket}/{key}	Supports Range header
HeadObject	HEAD /{bucket}/{key}
DeleteObject	DELETE /{bucket}/{key}	Standalone If-Match: * or strong ETag list
CopyObject	PUT /{bucket}/{key}	x-amz-copy-source header
InitiateMultipartUpload	POST /{bucket}/{key}?uploads
UploadPart	PUT /{bucket}/{key}?partNumber&uploadId
ListParts	GET /{bucket}/{key}?uploadId	Optional max-parts and part-number-marker
CompleteMultipartUpload	POST /{bucket}/{key}?uploadId
AbortMultipartUpload	DELETE /{bucket}/{key}?uploadId
ListMultipartUploads	GET /{bucket}?uploads
GetBucketPolicy	GET /{bucket}?policy
PutBucketPolicy	PUT /{bucket}?policy
DeleteBucketPolicy	DELETE /{bucket}?policy

On-Disk Format

Standalone mode:

{storage.directory}/
  {bucket}/
    .smartstorage-objects-v2/
      {whole-key-hex-frames}/
        object._storage_object_v2                # Object data; completed multipart objects include
                                                 # an atomic payload/ETag/metadata trailer
        object._storage_object_v2.metadata.json  # Legacy/direct-write metadata sidecar
        object._storage_object_v2.md5            # Legacy/direct-write cached MD5 hash
  .multipart/
    {upload-id}/
      metadata.json                      # Upload metadata
      part-1, part-2, ...               # Part data files
  .policies/
    {bucket}.policy.json                 # Bucket policy (IAM JSON format)

Cluster mode:

{storage.directory}/
  .buckets/
    v2/{bucket-address}.bucket.json       # Active/tombstoned bucket state
    .policies/{bucket}.policy.json        # Bucket policies
  .manifests/
    v2/{bucket-prefix}/{bucket-address}/{object-prefix}/
      {object-address}.state.json
                                         # Active/tombstoned object manifest
    .multipart/{upload-id}/               # Descriptor-anchored multipart state
    .get-tmp/                             # Unlinked GET reconstruction files
  .smartstorage/cluster/
    identity.json                         # Node and cluster identity
    topology.json                         # Last durable topology
    generation-inventory/{address}.json   # Deferred physical-generation cleanup

{drive_path}/.smartstorage/data/
  v2/{bucket-prefix}/{bucket-address}/{key-prefix}/{generation-address}/
    {shard-address}.shard                 # Immutable erasure-coded shard record
    {shard-address}.tombstone             # Crash-safe deletion authority

Superseded physical generations are queued with their exact shard placements. Cluster reads are bounded, cleanup waits a 24-hour reader grace period, and the hourly cleanup worker purges both shard records and tombstones. Multipart uploads older than multipart.expirationDays, plus committed uploads whose first cleanup attempt failed, are retried during bounded cleanup passes scheduled every multipart.cleanupIntervalMinutes; large registries are visited page by page across successive passes. Generation cleanup retains at most 32 MiB of serialized records per page, multipart session listings process documents one at a time, policy startup loading is bounded and streaming, and both workers observe shutdown within an active pass. Both multipart settings must be greater than zero; SmartStorage rejects invalid values during startup. Shutdown never aborts or detaches an active multipart descriptor worker; it waits for the worker under the same retryable bounded drain used by mounted HTTP requests. Multipart admission is bounded at 100,000 registry entries per storage process, including cleanup-pending and unreadable entries. Cleanup only removes descriptor-verified expired uploads and cleanup-marked committed entries; unreadable or incomplete descriptors fail closed and require operator repair or removal after their bucket and clustered shard ownership has been established.

Related Packages

• @push.rocks/smartbucket — High-level S3-compatible abstraction layer
• @push.rocks/smartrust — TypeScript ↔ Rust IPC bridge
• @git.zone/tsrust — Rust cross-compilation for npm packages

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.