jkunz a17fb4469d
Default (tags) / security (push) Failing after 0s
Default (tags) / test (push) Failing after 1s
Default (tags) / metadata (push) Skipped
v11.3.1
2026-11-14 11:18:47 +00:00
2026-11-14 11:18:47 +00:00
2026-11-14 11:18:47 +00:00
2026-11-14 11:18:47 +00:00
2026-11-14 11:18:47 +00:00

@lossless.org/objectstorage

An S3-compatible object storage server that you install with pnpm add and run from TypeScript. A Rust engine serves the S3 API with streaming I/O and bounded memory; a typed TypeScript API starts it, provisions credentials and reads its health. Run it standalone on one directory for development, tests and single-node services, or as an erasure-coded cluster across nodes and drives. Any S3 client works against it: the AWS SDK, @lossless.org/client, or your own.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why objectstorage

  • No Docker, no toolchain. Prebuilt engines for Linux (amd64, arm64) and macOS (Intel, Apple Silicon) ship inside the npm package.
  • S3 semantics, not a mock. AWS SigV4 verification with constant-time comparison (headers and streaming chunk signatures), bucket policies with Allow/Deny, multipart uploads, byte ranges, conditional requests, bucket versioning and ListObjectsV2 pagination. A request it does not implement is refused (501 NotImplemented, or 400 InvalidRequest where cluster mode cannot enforce a condition yet), never answered with a silent wrong result.
  • Bounded memory. Request and response bodies stream with backpressure, ranges seek in the file, and upload decoding runs under a fixed shared budget.
  • Crash-safe writes. Standalone writes are staged, synced and published atomically: a reader, and a crash, sees the previous object or the new one, never a mix.
  • Built for tests. port: 0 takes a free port, cleanSlate wipes the root, and stop() resolves only once the listener, the work and the engine process are gone.
  • Credentials for platforms. Rotate credentials without a restart, give each app one bucket and one scoped key (bucket tenants), or hand out expiring keys limited to one prefix.
  • Operable. /-/live, /-/ready, /-/health and Prometheus /-/metrics endpoints, cached storage stats, a read-only dashboard, bucket export and import, fenced bucket replacement and deletion, and migration control.
  • Clustering. Reed-Solomon erasure coding (default 4 data + 2 parity shards), quorum writes and reads, QUIC transport with optional mutual TLS, per-drive health and background self-healing.

Installation

pnpm add @lossless.org/objectstorage      # add -D when you only use it in tests
Platform Engine Requirements
Linux amd64, arm64 ruststorage_linux_amd64_musl, ruststorage_linux_arm64_musl (static; the TypeScript API runs these) none beyond the kernel
Linux amd64, arm64 (custom packaging through binaryPath) ruststorage_linux_amd64, ruststorage_linux_arm64 (dynamically linked GNU builds) glibc 2.34 or later with its dynamic loader, libm.so.6, libgcc_s.so.1
macOS Intel, Apple Silicon ruststorage_macos_amd64, ruststorage_macos_arm64 local APFS storage; built for macOS 11 and later, ad-hoc signed, system libraries only

Native runtime notices (Cargo licenses, the glibc LGPL terms, the native distribution inventory) are in third-party notices; keep them when you redistribute the engine binaries. What each platform is qualified on, and how to build from source, is under Platform support.

Quick Start

Standalone

import { ObjectStorage } from '@lossless.org/objectstorage';
import { S3Client, CreateBucketCommand, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';

const storage = await ObjectStorage.createAndStart({
  server: { port: 0, silent: true }, // 0: any free port
  storage: { cleanSlate: true },     // wipes ./.nogit/bucketsDir on start
});
const { endpoint, port, accessKey, accessSecret } = await storage.getStorageDescriptor();
const s3 = new S3Client({
  endpoint: `http://${endpoint}:${port}`,
  region: 'us-east-1',
  credentials: { accessKeyId: accessKey, secretAccessKey: accessSecret },
  forcePathStyle: true, // objectstorage routes path-style requests
});

await s3.send(new CreateBucketCommand({ Bucket: 'my-bucket' }));
await s3.send(new PutObjectCommand({ Bucket: 'my-bucket', Key: 'hello.txt', Body: 'Hello!' }));
const object = await s3.send(new GetObjectCommand({ Bucket: 'my-bucket', Key: 'hello.txt' }));
console.log(await object.Body!.transformToString()); // Hello!

s3.destroy();
await storage.stop();

With port: 0 the operating system picks the port. After start(), storage.port, getStorageDescriptor(), bucket tenant descriptors (and their S3_PORT), the HTTP health endpoints and cluster peer advertisements report the bound port; config.server.port keeps the requested value, and before startup and after shutdown storage.port returns it. A restart with port 0 takes a new port.

In tests

import { ObjectStorage } from '@lossless.org/objectstorage';
import { tap, expect } from '@git.zone/tsrepo/tapbundle';

let storage: ObjectStorage;

tap.test('setup', async () => {
  storage = await ObjectStorage.createAndStart({
    server: { port: 0, silent: true },
    storage: { cleanSlate: true },
  });
});

tap.test('stores objects', async () => {
  await storage.createBucket('test');
  // ... your test, through the AWS SDK or @lossless.org/client
});

tap.test('teardown', async () => {
  await storage.stop();
});

export default tap.start();

stop() resolves only once the engine released its listener, its work and its process, so a test can remove a disposable storage root right after it.

Cluster

import { ObjectStorage } from '@lossless.org/objectstorage';

const storage = await ObjectStorage.createAndStart({
  server: { port: 3000 },
  storage: {
    directory: '/var/lib/objectstorage/control',
    cleanSlate: false,
  },
  cluster: {
    enabled: true,
    nodeId: 'node-1',
    quicPort: 4000,
    seedNodes: ['192.168.1.11:4000', '192.168.1.12:4000'],
    erasure: {
      dataShards: 4,      // k: shards needed to reconstruct
      parityShards: 2,    // m: shards that may be lost
    },
    drives: {
      paths: ['/mnt/disk1', '/mnt/disk2', '/mnt/disk3'],
    },
  },
});

Objects are split into chunks (default 4 MiB); each chunk is erasure-coded into 6 shards (4 data + 2 parity) spread across drives and nodes, and any 4 of them rebuild it.

Before the first start, create the control root and every drive root, owned by the process user, with no symlink in their paths and without group or world write permission; cluster mode never creates them (Cluster roots). Each drive root counts as one independent drive, so startup refuses two drive roots on the same device (st_dev). Only the device number is checked: partitions of one disk, LVM volumes, btrfs subvolumes and datasets of one ZFS pool report distinct devices and are not detected, so keeping them apart is the operator's job. The control root may share a device with a drive. For dev and test clusters whose drive roots are directories on one filesystem, opt in explicitly; startup then logs which roots share a device:

drives: {
  paths: ['/srv/objectstorage-dev/drive-1', '/srv/objectstorage-dev/drive-2', '/srv/objectstorage-dev/drive-3'],
  allowSharedDevice: true, // dev/test only: these roots are not independent drives
},

Without cluster.tls the cluster transport does not authenticate nodes: run cluster mode only on a network where every host that can reach quicPort is a cluster node, or set cluster.tls on every node (see Cluster transport trust).

Configuration

Every field is optional; ObjectStorage.createAndStart(config) and new ObjectStorage(config) take an IObjectStorageConfig.

Key Default Notes
server.port 3000 0 lets the OS allocate a port (see Quick Start)
server.address '0.0.0.0' descriptors report localhost for 0.0.0.0
server.silent false suppresses the startup lines
server.region 'us-east-1' SigV4 signing region, returned by GetBucketLocation
storage.directory <cwd>/.nogit/bucketsDir standalone storage root; control root in cluster mode
storage.cleanSlate false wipes the root's contents on start (the root, its maintenance lock and pool identity stay); refused for mountedFs pools, with durable migration state and on a root with retained buckets
storage.pool a default localFs pool at storage.directory see Storage pools
auth.enabled false SigV4 authentication and bucket policies
auth.credentials [{ accessKeyId: 'STORAGE', secretAccessKey: 'STORAGE' }] initial root credentials; { accessKeyId, secretAccessKey, bucketName?, region? }
cors.enabled false
cors.allowedOrigins / allowedMethods / allowedHeaders ['*'] / ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'] / ['*']
cors.exposedHeaders ['ETag', 'x-amz-request-id', 'x-amz-version-id']
cors.maxAge / cors.allowCredentials 86400 / false
limits.requestTimeout 300000 milliseconds the engine waits for the next frame of a streaming (aws-chunked) upload body and of a bucket policy or versioning document
limits.maxObjectSize / limits.maxMetadataSize 5368709120 / 2048 accepted, not applied by the current engine
logging.level / format / enabled 'info' / 'text' / true accepted, not applied: the engine logs at info to its stderr, which is forwarded verbatim to the host's stderr
multipart.expirationDays 7 uploads older than this are removed; must be greater than zero
multipart.cleanupIntervalMinutes 60 cleanup pass interval; must be greater than zero
multipart.maxUploadsPerPrefixScope 100 open uploads per prefix credential scope; must be non-zero
cluster.enabled true selects cluster mode; omit cluster for standalone
cluster.nodeId generated and persisted required with tls, then a lowercase DNS name
cluster.quicPort 4000 UDP; 0 lets the OS assign it (read it from getClusterHealth().quicAddress)
cluster.seedNodes [] host:port QUIC addresses of existing members
cluster.erasure.dataShards / parityShards 4 / 2
cluster.erasure.chunkSizeBytes 4194304 (4 MiB) 1 byte to 64 MiB; startup refuses anything else
cluster.drives.paths drive roots of this node
cluster.drives.allowSharedDevice false dev/test only
cluster.heartbeatIntervalMs / heartbeatTimeoutMs 5000 / 30000
cluster.getDeadlineMs 21600000 (6 h) 60,000 (1 min) to 604,800,000 (7 days); the same on every node (Streaming Reads)
cluster.tls unset { caFile, certFile, keyFile }; on every node or on none (Cluster transport trust)
dashboard.enabled false read-only dashboard (Dashboard)
dashboard.address / dashboard.port '127.0.0.1' / 3001 port 0 allocates one; never the S3 port
binaryPath package discovery one exact engine executable (Engine selection)
onUnexpectedEngineExit unset ({ code, signal }) => void, called when the engine ends without a stop request (Process lifecycle); not sent to the engine

A few common shapes:

// Auth on, one root credential
await ObjectStorage.createAndStart({
  auth: { enabled: true, credentials: [{ accessKeyId: 'test', secretAccessKey: 'test123' }] },
});

// CORS for local web development
await ObjectStorage.createAndStart({
  cors: { enabled: true, allowedOrigins: ['http://localhost:5173'], allowCredentials: true },
});

Storage pools and mounted filesystems

One ObjectStorage process serves one storage pool. storage.directory alone is a default localFs pool. With both storage.directory and storage.pool.directory, a localFs pool needs both to resolve to the same path and a mountedFs pool needs both to be the same exact absolute mountpoint string. storage.pool.id starts with a lowercase letter and has at most 63 lowercase letters, digits or hyphens. A root is durably bound to its pool ID and cannot be reassigned to another pool.

A missing standalone localFs root is created with mode 0700. An existing root must be owned by the process user and must not be group- or world-writable, otherwise startup fails before the S3 listener opens; ObjectStorage never changes an existing root's ownership or permissions.

On Linux, standalone ObjectStorage can serve an exact NFS or SMB mount (mountedFs; not in cluster mode):

  • The configured directory is the absolute mountpoint itself, not a symlink or canonical alias, and cleanSlate is forbidden.
  • expectedSource holds a numeric IPv4 or IPv6 address: IP:/absolute/export for NFS, //IP/share[/path] for SMB, IPv6 in brackets ([2001:db8::10]:/archive, //[2001:db8::10]/archive).
  • The client mount must carry the Linux nosymfollow option; whoever provisions the mount adds it before ObjectStorage starts.
  • NFS must keep server-coordinated locking: nolock and local_lock=all|flock are refused; no local_lock, local_lock=none and local_lock=posix are accepted, because ObjectStorage uses flock(2). CIFS mounts must not use nobrl.
const storage = await ObjectStorage.createAndStart({
  storage: {
    directory: '/mnt/archive',
    pool: {
      id: 'archive',
      directory: '/mnt/archive',
      backend: {
        kind: 'mountedFs',
        expectedFilesystemType: 'nfs',
        expectedSource: '192.0.2.10:/archive',
        // IPv6 is canonicalized as: [2001:db8::10]:/archive
      },
    },
  },
});

Startup opens the mountpoint once and confines every storage, credential, policy, retention, fencing and background operation beneath that descriptor. It verifies the mount ID, device, NFS/CIFS filesystem magic, the ST_NOSYMFOLLOW flag and the effective locking options, rereads the complete mount activation before serving, and runs the root-ownership, same-filesystem, lock-contention, directory-fsync and provider-identity checks through the descriptor. Every storage boundary revalidates the activation. Any drift permanently poisons the process: storage-backed HTTP and management calls fail closed, descendant symlinks are never followed, and the process never falls back to a host directory at the configured path. Lifecycle, liveness, cluster status and cached diagnostics stay available for shutdown and diagnosis; getStoragePoolDiagnostic() returns the cached activation identity and poison evidence without touching the filesystem.

A serving pool observation is productionEligible once these checks pass and while the activation is unpoisoned. Atomic directory exchange is reported as a separate, optional capability: NFS without it still serves S3, atomic create-only uploads, fenced bucket ensure and delete, and migration cleanup, while atomic whole-bucket replacement and isolated restore fail closed. The remoteLocking check and filesystem.remoteLockingMode tell server-coordinated locking apart from unsafe local or disabled locking.

Probe an existing mount without starting the S3 listener:

const observation = await ObjectStorage.probeStoragePool({
  id: 'archive',
  directory: '/mnt/archive',
  backend: {
    kind: 'mountedFs',
    expectedFilesystemType: 'nfs',
    expectedSource: '192.0.2.10:/archive',
  },
});

console.log(observation.supported);
console.log(observation.productionEligible); // false: this read-only probe skips write semantics
console.log(observation.checks);

The probe verifies the exact active mount, numeric source, mount ID, device, filesystem magic, ST_NOSYMFOLLOW, the locking options and activation stability, and reports block and transfer sizes. It never writes to the target or changes mount tuning: write checks stay notRun, so supported stays null and productionEligible false. Only a serving startup establishes production eligibility.

Engine selection (binaryPath)

binaryPath selects one exact ruststorage executable, for example a verified engine an installer placed beside a compiled application. When it is set, OBJECTSTORAGE_RUST_BINARY, the package's dist_rust binaries, development builds and PATH are never searched, even if the path is unusable: a missing, non-file or non-executable path rejects start() (and createAndStart()) with RustBinaryLocatorError, code ERR_RUST_BINARY_EXPLICIT_PATH_INVALID, and an empty string throws at construction. Validation never changes the file's permissions. The option is not sent to the engine and does not appear in storage.config. Without it, normal discovery runs, with OBJECTSTORAGE_RUST_BINARY as a hint for qualification runs. probeStoragePool() and findOrphanObjectLeaves() take the same option.

const storage = await ObjectStorage.createAndStart({
  binaryPath: '/opt/app/engines/ruststorage',
  storage: { directory: '/var/lib/app/objects' },
});

Using objectstorage

AWS SDK v3

Point any S3 client at getStorageDescriptor() (or a bucket tenant descriptor) with path-style addressing, as in the Quick Start. getStorageDescriptor() returns { endpoint, port, accessKey, accessSecret, useSsl: false, bucketName: '' }, using the first configured credential (STORAGE/STORAGE by default); pass an object to override fields. The engine serves plain HTTP.

AWS SDK streaming PutObject and UploadPart requests are decoded before storage. For STREAMING-UNSIGNED-PAYLOAD-TRAILER the engine verifies the declared decoded length and the CRC32 trailer, for STREAMING-AWS4-HMAC-SHA256-PAYLOAD the complete chained signature sequence. Every aws-chunked request needs a verified SigV4 envelope, also with auth.enabled: false; ordinary requests keep the auth-disabled behavior. Unsupported, malformed, truncated, checksum-invalid or signature-invalid bodies fail closed, and the aws-chunked transport encoding is not stored as object metadata. Decoder memory is bounded globally (per-chunk buffers of at most 16 MiB under a shared 64 MiB budget), and limits.requestTimeout bounds the wait for each body frame. A standalone overwrite is staged on the same filesystem and published atomically only after verification succeeds, so a failed, timed-out or cancelled body keeps the previous object.

Client library (@lossless.org/client)

@lossless.org/client is the lossless.org client for NoSQLDB, SQLDB and ObjectStorage. Its @lossless.org/client/objectstorage entry point provides SmartBucket with buckets, directories, files, metadata, streaming and exact (create-only) uploads, and needs @aws-sdk/client-s3 as a peer:

pnpm add @lossless.org/client @aws-sdk/client-s3
import { SmartBucket } from '@lossless.org/client/objectstorage';

const smartBucket = new SmartBucket(await storage.getStorageDescriptor());
const bucket = await smartBucket.createBucket('my-bucket');

await bucket.fastPut({ path: 'docs/readme.txt', contents: 'Hello!' });
const content = await bucket.fastGet({ path: 'docs/readme.txt' }); // Buffer
const exists = await bucket.fastExists({ path: 'docs/readme.txt' }); // true

await smartBucket.close();

The client's exact uploads and exact-path purge use the standalone conditional writes and deletes described under Conditional requests.

Credentials

With auth.enabled: true three kinds of credential authenticate S3 requests: root credentials (the runtime set), bucket tenant credentials (one bucket each) and prefix credentials (one prefix of one bucket, expiring).

Runtime credentials

const credentials = await storage.listCredentials(); // metadata, no secrets

await storage.replaceCredentials([
  { accessKeyId: 'ADMINA', secretAccessKey: 'super-secret-a' },
  { accessKeyId: 'ADMINB', secretAccessKey: 'super-secret-b' },
]);
  • replaceCredentials() swaps the whole set atomically and persists it under the storage root. New requests use the new set at once and the old credentials stop authenticating at once; requests already authenticated keep running, because auth is evaluated when a request starts.
  • No restart is needed, and the set survives restarts; storage.cleanSlate clears it with the rest of the root's contents.
  • The input holds at least one credential ({ accessKeyId, secretAccessKey, bucketName?, region? }), every accessKeyId and secretAccessKey is non-empty, and access key IDs are unique.
  • A new accessKeyId must not be held by a live or revoked prefix credential (ECREDENTIAL_CONFLICT); bucket tenant create, ensure and rotate apply the same check. replaceCredentials() is refused while any resource publication is held (Durable resource fencing).
  • Prefix credentials are not part of listCredentials() (Prefix credentials).
  • replaceCredentials() is a trusted, process-local administrative override, not a fenced lifecycle API: if it changes credential material that a fenced bucket owns exactly, replaying that bucket's receipt detects the drift and fails closed.

Bucket tenants

A bucket tenant is one bucket with its own scoped S3 credential, for platforms that give every app one bucket. The scope is enforced before bucket policies and default auth, so a validly signed tenant request still cannot list all buckets or reach another bucket. Tenant APIs require auth.enabled: true.

const tenant = await storage.createBucketTenant({ bucketName: 'workapp-123' });

const client = new S3Client({
  endpoint: `http://${tenant.endpoint}:${tenant.port}`,
  region: tenant.region,
  credentials: { accessKeyId: tenant.accessKeyId, secretAccessKey: tenant.secretAccessKey },
  forcePathStyle: true,
});
console.log(tenant.env.S3_BUCKET, tenant.env.AWS_ACCESS_KEY_ID); // ready for env injection

await storage.rotateBucketTenantCredentials({ bucketName: 'workapp-123' });
const descriptor = await storage.getBucketTenantDescriptor({ bucketName: 'workapp-123' });
const tenants = await storage.listBucketTenants(); // metadata, no secrets
await storage.deleteBucketTenant({ bucketName: 'workapp-123', accessKeyId: descriptor.accessKeyId });
  • createBucketTenant() creates the bucket if needed and stores a scoped credential, generated unless you pass accessKeyId and secretAccessKey. ensureBucketTenant() does the same atomically with a supplied credential: it creates an absent bucket with that credential, resumes an orphaned exact credential, or rotates the credential of an existing bucket it owns exactly, and refuses foreign or ambiguous ownership before changing anything.
  • The descriptor holds the endpoint, port, region, bucket, key pair, SSL flag and an env map (S3_ENDPOINT, S3_PORT, S3_REGION, S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_USE_SSL, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION).
  • deleteBucketTenant({ bucketName, accessKeyId }) revokes one credential and keeps the bucket; deleteBucketTenant({ bucketName }) revokes the bucket's credentials and deletes the bucket with its contents, every object version included.
  • Tenant credentials list, read, write and delete objects in their bucket. They cannot list all buckets, reach or copy from other buckets, delete buckets, change bucket policies or change versioning.

Immutable retention

createBucketTenant() and ensureBucketTenant() accept retention: { mode: 'compliance', policyId, retentionDurationSeconds } for engine-enforced retention in standalone mode. Objects of a retained bucket cannot be overwritten with a different payload or metadata (an exact replay of the same PUT succeeds with the same ETag) and cannot be deleted by any credential before their retainUntil (creation time plus the duration). CopyObject and multipart uploads into a retained bucket are refused. HeadObject reports ObjectLockMode: COMPLIANCE and the smartstorage-retention-* metadata. The policy ID cannot change and the duration cannot shorten; the retention receipt survives restarts, and cleanSlate refuses to start on a retained root. getBucketRetentionCapability() reports support and getBucketRetentionReceipt({ bucketName }) the bucket's receipt (also on the tenant descriptor and in listBucketTenants()). Retention and versioning exclude each other, and a retained bucket cannot become a migration source.

Prefix credentials

A prefix credential is an ordinary SigV4 key that reaches only the keys below one prefix of one bucket, and only until it expires: a short-lived key for a guest (a VM, a job, a mounted drive) that must see one folder and nothing else.

const credential = await storage.createPrefixCredential({
  bucketName: 'drives',
  prefix: 'drives/user-42/',
  ttlSeconds: 3600,          // 1 to 43200 (12 h)
});
// { accessKeyId, secretAccessKey, bucketName, prefix, region, createdAt, expiresAt }

const client = new S3Client({
  endpoint: 'http://localhost:3000',
  region: credential.region,
  credentials: {
    accessKeyId: credential.accessKeyId,
    secretAccessKey: credential.secretAccessKey,
  },
  forcePathStyle: true,
});

await storage.revokePrefixCredential({ accessKeyId: credential.accessKeyId }); // { revoked: true }
const live = await storage.listPrefixCredentials({ bucketName: 'drives' });   // metadata, no secrets
interface IPrefixCredentialInput {
  bucketName: string;
  prefix: string;
  ttlSeconds?: number;      // first node: lifetime in seconds
  expiresAt?: number;       // further nodes: the first node's expiresAt (epoch ms)
  accessKeyId?: string;     // supplied together with secretAccessKey and expiresAt only
  secretAccessKey?: string;
  region?: string;          // default: server.region
}

Input rules.

  • Exactly one of ttlSeconds (an integer, 1 to 43 200) and expiresAt (an integer epoch-ms time after now and at most 43 200 s ahead).
  • Key material is generated when omitted (SP plus 20 uppercase hex characters, a 64-hex-character secret). Supplied material needs expiresAt and both accessKeyId and secretAccessKey: an ID of 16 to 128 characters of A-Z0-9, and a secret of 32 to 128 printable ASCII characters without spaces.
  • The prefix is validated, never normalised: 1 to 1023 bytes of A-Z a-z 0-9 - . _ ~ / (RFC 3986 unreserved characters plus /, no percent-encoding), ending with /, not starting with /, with no empty, . or .. segment.
  • The bucket must exist. The access key ID must not belong to a static or bucket tenant credential and must not have been revoked.
  • The secret appears only in the create result. It is never listed, logged or stored in storage.config.
  • Errors are ObjectStorageCredentialError with a code:
Code Meaning
ECREDENTIAL_AUTH_DISABLED auth.enabled is false
ECREDENTIAL_INVALID_PREFIX the prefix fails the grammar
ECREDENTIAL_INVALID_TTL not exactly one of ttlSeconds and expiresAt, or ttlSeconds out of range
ECREDENTIAL_INVALID_EXPIRY expiresAt is not in the future, or is more than 12 h ahead
ECREDENTIAL_INVALID_KEY the key material or the region is malformed, or only part of it was supplied
ECREDENTIAL_NO_SUCH_BUCKET the bucket does not exist
ECREDENTIAL_BUCKET_UNAVAILABLE the bucket is a sealed or retired migration source
ECREDENTIAL_CONFLICT the ID is held by another credential, or a replay differs from the stored record
ECREDENTIAL_REVOKED the ID was revoked
ECREDENTIAL_NOT_PREFIX a revoke named a static or bucket tenant credential
ECREDENTIAL_LIMIT 10 000 live credentials and revocations are stored
ECREDENTIAL_PERSISTENCE the store could not be written
ECREDENTIAL_STORE_UNAVAILABLE the store file is unreadable (see below)

Access. A key is covered when its bytes start with the prefix; a key equal to the prefix (its folder marker) is covered.

  • Objects. GetObject, HeadObject, PutObject, DeleteObject and the multipart operations work on covered keys.
    • Writes also refuse keys with ., .. or empty segments after the prefix, and keys with control characters.
    • CopyObject and UploadPartCopy need the source to be covered too.
    • Object requests accept only the query parameters the operation uses: uploads, uploadId, partNumber, max-parts and part-number-marker, the six response-* header overrides on GetObject and HeadObject, plus the x-id operation label that AWS SDKs append. Any other object query (?acl, ?tagging, ?versionId, ?retention, …) is refused with AccessDenied (403), also a name S3 does not define (?v=1), and so is any repeated query parameter. A prefix credential never names a version: ?versionId is refused even as null, and so is a copy source with a ?versionId= suffix.
    • Requests that pass these checks still go through the router's allowlist (Supported S3 Operations), so a parameter the operation does not implement gets NotImplemented (501).
  • DeleteObjects. Every member is checked on its own: covered keys are deleted, and the others get a per-key AccessDenied, as does every member that names a VersionId.
  • Listings. ListObjects V1, ListObjectsV2, ListObjectVersions and ListMultipartUploads need a prefix parameter that is covered. A missing or wider prefix is refused, never clamped.
  • Bucket level. ListBuckets returns only the credential's bucket; HeadBucket and GET /{bucket}?location work on that bucket. Any other bucket-level query (?acl, ?versioning, ?lifecycle, …) and any repeated query parameter is refused with AccessDenied, so a prefix credential can neither read nor change a bucket's versioning. Creating or deleting buckets and bucket policy operations are refused.
  • Policies. The scope check runs before bucket policies, so a policy Allow (including Principal: "*") never widens a prefix credential. A policy Deny still applies.

Expiry and deadlines.

  • A credential authenticates while the engine clock is before expiresAt. After that, or after revocation, requests get InvalidAccessKeyId (403).
  • Admitted requests end at expiresAt as well: a trickled upload fails with nothing stored, and a slowly read download is cut off.
  • Revocation affects only requests admitted after it; requests already running continue until they finish or until expiresAt.
  • There is no extend operation. To refresh, create a new key and revoke the old one.
  • Multipart cap. All prefix credentials of one scope (the same bucket and prefix) together hold at most multipart.maxUploadsPerPrefixScope (default 100) open multipart uploads. The next CreateMultipartUpload of that scope gets AccessDenied (403) until one of them ends: completed or aborted by any credential, or removed by the multipart expiry cleanup. Revoking, expiring or replacing a credential frees none of its scope's slots, so issuing new keys does not raise the cap. The count is held in memory and resets when the engine restarts. The engine-wide limit of 100 000 uploads still applies.

Bucket deletion. Deletion attempts revoke first; callers re-issue.

  • Every path that deletes a bucket revokes the bucket's prefix credentials before it removes anything: S3 DeleteBucket, deleteBucket(), deleteBucketTenant() without an accessKeyId, and migration source cleanup. A recreated bucket of the same name is not reachable with them.
  • S3 DeleteBucket checks that the bucket is empty before it revokes, so a BucketNotEmpty (409) keeps the credentials.
  • If the revocation cannot be stored, the deletion is refused.
  • A deletion that fails or is refused after the revocation (for example a stale resource fence, EFENCE_STALE) returns that error, never a success, and leaves the credentials revoked. Callers that keep the bucket create new prefix credentials.

Persistence.

  • Credentials and revocations live in .smartstorage/prefix-credentials.json, under the same root as credentials.json. The engine writes the file atomically with mode 0600, and storage.cleanSlate clears it.

  • A sweep every 60 seconds drops expired entries.

  • Engines older than prefix credentials never read the file, so a rollback disables every prefix credential rather than widening it.

  • Fail closed. If the file is unreadable, larger than 32 MiB, malformed, or inconsistent (duplicate IDs, an ID held by a static credential, a record that fails validation), the store is unavailable:

    • every prefix credential is refused;
    • create, revoke and list fail with ECREDENTIAL_STORE_UNAVAILABLE;
    • no new static or tenant access key ID can be introduced, because the ID collision check cannot run: replaceCredentials(), createBucketTenant(), ensureBucketTenant() and rotateBucketTenantCredentials() throw ObjectStorageCredentialError with code ECREDENTIAL_STORE_UNAVAILABLE when they would add an access key ID that is not in the current set;
    • bucket deletion is refused, because the bucket's credentials cannot be revoked: deleteBucket() and deleteBucketTenant() without an accessKeyId throw ObjectStorageCredentialError with code ECREDENTIAL_STORE_UNAVAILABLE, and S3 DeleteBucket returns InvalidRequest (400) with a message starting ECREDENTIAL_STORE_UNAVAILABLE:;
    • static and tenant credentials keep working;
    • getHealth().auth.prefixCredentialStoreAvailable is false;
    • the engine logs the condition without secrets and leaves the file as found.

    To recover, repair or delete the file and restart. Deleting it revokes every prefix credential and forgets every revocation, so never reuse a revoked ID afterwards. Generated IDs are random.

Cluster mode. Prefix credentials are node-local, like runtime credentials: nothing replicates them, and the caller fans out.

  1. Create on one node with ttlSeconds.
  2. Create on every other node with the first node's accessKeyId, secretAccessKey, expiresAt and region.
    • Each node records its own createdAt; every other field matches the first node's record.
    • An identical replay returns the stored record unchanged. Any difference gives ECREDENTIAL_CONFLICT.
    • A late create never extends the lifetime.
  3. To revoke, call revokePrefixCredential on every node and retry until each node acknowledges.
    • A node that never saw the ID records the revocation for 12 h, so a create that reaches it later is refused with ECREDENTIAL_REVOKED.
    • revoked is true only where a live credential was removed.

Revocation guarantee.

  • Once a node has acknowledged a revocation, it refuses every request admitted after the acknowledgement. It also refuses any later create for that ID until the ID could no longer be created anyway: an expiresAt create must lie within 12 h of the first node's create, and the revocation is kept at least that long.
  • A node that never acknowledged serves the key until the caller's retry reaches it, or until expiresAt.
  • The hard bound is expiresAt, at most 12 h after the first create. Each node judges expiry by its own clock, so clock offsets shift these instants.
  • listPrefixCredentials() lets the caller reconcile nodes.

Versioning

Standalone buckets support S3 versioning (PutBucketVersioning with Enabled or Suspended, version ids, delete markers, versionId on GET, HEAD, DELETE and copy sources, ListObjectVersions). Cluster buckets stay unversioned. The full behavior, permissions and limits are in Bucket versioning.

Conditional requests

  • Reads. GetObject and HeadObject evaluate If-Match, If-None-Match, If-Modified-Since and If-Unmodified-Since, in standalone and cluster mode (Conditional GET and HEAD).
  • Create-only writes. Standalone PutObject and CompleteMultipartUpload honor If-None-Match: * and publish through an atomic no-replace operation; of two racing create-only writes the first to commit wins.
  • Conditional deletes. Standalone DeleteObject honors If-Match: * and comma-separated strong ETags. It compares the current ETag and deletes inside the bucket gate that PUT, copy and multipart completion commit under. A mismatch returns PreconditionFailed and changes neither the object nor the stats; an absent target returns NoSuchKey.
  • Copies. CopyObject and UploadPartCopy evaluate x-amz-copy-source-if-match, -if-none-match, -if-modified-since and -if-unmodified-since with the same rules as reads, except that every failed condition is 412.
  • Cluster mode refuses create-only writes with InvalidRequest (400) and conditional deletes with NotImplemented (501) until the conditions can be enforced across nodes.

Conditional GET and HEAD

GetObject and HeadObject evaluate the conditions against the version the request selects (the current one, or the one its versionId names), in the order of RFC 9110 §13.2.2, which is also the order S3 documents:

  1. If-Match, else If-Unmodified-Since: when false, 412 PreconditionFailed.
  2. If-None-Match, else If-Modified-Since: when false, 304 Not Modified.

So a matching If-Match decides over a failing If-Unmodified-Since (200), and If-None-Match decides alone over If-Modified-Since. If-Match compares entity tags strongly (a W/"…" tag never matches), If-None-Match weakly; both take * (any existing object) or a comma-separated list, quoted or not. A * beside a tag, or an empty or unterminated tag, is InvalidArgument (400). Dates compare in whole seconds, as Last-Modified carries them; a value that is not one valid HTTP-date (IMF-fixdate, RFC 850 or asctime), such as If-Modified-Since: yesterday or a list of dates, is ignored, as RFC 9110 requires. A 304 has no body and carries ETag, Last-Modified and the Cache-Control/Expires overrides (response overrides). Preconditions are evaluated before the Range header, so a failed one answers 304 or 412 where the range alone would be 206 or 416, and before a cluster GET reads any shard. A missing key, a current delete marker (404) or a delete marker named by versionId (405) answers as without preconditions.

Multipart uploads

Multipart uploads stream with bounded memory; in cluster mode each part is erasure-coded and distributed on its own.

import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
} from '@aws-sdk/client-s3';

const { UploadId } = await client.send(new CreateMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
}));

const parts = [];
for (let i = 0; i < chunks.length; i++) {
  const { ETag } = await client.send(new UploadPartCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    UploadId,
    PartNumber: i + 1,
    Body: chunks[i],
  }));
  parts.push({ PartNumber: i + 1, ETag });
}

await client.send(new CompleteMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
  UploadId,
  MultipartUpload: { Parts: parts },
}));
  • CompleteMultipartUpload refuses, in standalone and cluster mode, a part list that names a part that was never uploaded or an ETag that does not match the uploaded part with InvalidPart (400), and a part listed twice with InvalidPartOrder (400). The upload stays open, so a completion with the correct parts still succeeds.
  • Standalone ListParts responses are bounded (max-parts, part-number-marker), so exact-streaming clients can verify abort cleanup; cluster mode refuses ListParts with InvalidRequest (400).
  • Uploads older than multipart.expirationDays are removed by cleanup passes every multipart.cleanupIntervalMinutes. At most 100,000 uploads are registered per storage process, cleanup-pending and unreadable entries included.
  • UploadPartCopy enforces the 5 GiB copy-source cap but not the S3 rule that a ranged copy needs a source larger than 5 MB; plain UploadPart has no size cap.

Bucket policies

With auth.enabled: true, requests are authorized in three steps:

  1. Authenticate: verify the AWS SigV4 signature (anonymous requests skip this step).
  2. Authorize: evaluate the bucket policy against the request's action, resource and caller.
  3. Default: authenticated callers get full access; anonymous requests are denied unless a policy allows them.
import { PutBucketPolicyCommand } from '@aws-sdk/client-s3';

// Allow anonymous read access to all objects in a bucket
await client.send(new PutBucketPolicyCommand({
  Bucket: 'public-assets',
  Policy: JSON.stringify({
    Version: '2012-10-17',
    Statement: [{
      Sid: 'PublicRead',
      Effect: 'Allow',
      Principal: '*',
      Action: ['s3:GetObject'],
      Resource: ['arn:aws:s3:::public-assets/*'],
    }],
  }),
}));
  • Effect: Allow and Deny; an explicit Deny always wins.
  • Principal: "*" or { "AWS": ["arn:..."] }.
  • Action: IAM-style actions such as s3:GetObject, s3:PutObject, s3:*, or prefix wildcards such as s3:Get*.
  • Resource: ARN patterns with * and ? wildcards (arn:aws:s3:::my-bucket/*).
  • Policies are stored with the data and survive restarts; GetBucketPolicy, PutBucketPolicy and DeleteBucketPolicy manage them (GET, PUT, DELETE /{bucket}?policy), and deleting a bucket removes its policy.

Observability

HTTP endpoints

The S3 listener answers four operational GET endpoints without authentication:

Path Answer
/-/live (/-/livez) 200 { ok: true, status: 'alive', startedAt }; also while a mounted pool is poisoned
/-/ready (/-/readyz) 200 ready; 503 degraded when a cluster node lacks majority or write quorum
/-/health (/-/healthz) 200 with the engine version, listener address and port, storage stats (bucket names included), cluster health and request counters
/-/metrics Prometheus text: requests, error responses, buckets, objects, cluster enabled and quorum, noncurrent versions and bytes, delete markers

Every Prometheus metric family is named objectstorage_*. Because /-/health lists bucket names, keep the S3 port off networks that must not see them.

Health and metrics

const health = await storage.getHealth();
const metrics = await storage.getMetrics();
  • getHealth() reports running state, storage directory and per-location pool observations, auth state, credential counts, bucket count, object count, total bytes, cluster health, and the resource-fencing and publication-hold capability fields.
  • getHealth().auth also reports prefixCredentialCount (live prefix credentials), prefixCredentialTombstoneCount (unexpired revocations) and prefixCredentialStoreAvailable (false while the prefix credential file is unreadable and every prefix credential is refused). These are counts and the store state only, never an access key ID, secret, bucket or prefix.
  • getHealth().stampClockAheadOfWallMs is present while version commit times run ahead of the wall clock (a wall clock that went backwards, see Bucket versioning).
  • getHealth().sweepClockBehindWallMs is present while the wall clock runs ahead of the version sweep's clock (a forward jump since the engine started, see Version retention).
  • getMetrics() returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (objectstorage_prefix_credentials_total, objectstorage_prefix_credential_tombstones_total, objectstorage_prefix_credential_store_available), version (objectstorage_noncurrent_versions_total, objectstorage_noncurrent_bytes_total, objectstorage_delete_markers_total), version sweep (objectstorage_versions_expired_total, objectstorage_delete_markers_expired_total, objectstorage_version_bytes_expired_total) and cluster metrics.

Storage stats

const stats = await storage.getStorageStats();
const bucketSummaries = await storage.listBucketSummaries();

console.log(stats.bucketCount, stats.totalObjectCount, stats.totalStorageBytes);
console.log(bucketSummaries[0]?.name, bucketSummaries[0]?.objectCount);
interface IBucketSummary {
  name: string;
  objectCount: number;
  totalSizeBytes: number;
  creationDate?: number;
  // Versioned buckets only:
  versioning?: 'Enabled' | 'Suspended';
  noncurrentVersionCount?: number;
  noncurrentBytes?: number;
  deleteMarkerCount?: number;
  versionsExpiredTotal?: number;
  deleteMarkersExpiredTotal?: number;
  bytesExpiredTotal?: number;
  lastSweepCompletedAt?: number;
  sweepDurationMs?: number;
  // Versioned buckets and buckets with a lifecycle or thinning configuration:
  thinning?:
    | { state: 'active' }
    | { state: 'inactive'; reason: 'notConfigured' | 'bucketUnversioned' | 'noNoncurrentExpirationRule' };
}

interface IStorageLocationSummary {
  path: string;
  totalBytes?: number;
  availableBytes?: number;
  usedBytes?: number;
  pool?: IStoragePoolObservation;
}

interface IStorageStats {
  bucketCount: number;
  totalObjectCount: number;
  totalStorageBytes: number;
  buckets: IBucketSummary[];
  storageDirectory: string;
  storageLocations?: IStorageLocationSummary[];
  // With at least one versioned bucket:
  totalNoncurrentVersionCount?: number;
  totalNoncurrentBytes?: number;
  totalDeleteMarkerCount?: number;
  totalVersionsExpired?: number;
  totalDeleteMarkersExpired?: number;
  totalBytesExpired?: number;
  stampClockAheadOfWallMs?: number;
  sweepClockBehindWallMs?: number;
}
  • Stats are logical: object payload bytes, not sidecar files or erasure-coded shard overhead. Object counts and bytes count current objects only; a versioned bucket also reports its noncurrent data versions and their bytes and its delete markers, current and noncurrent, and the totals add them up. Stored capacity is current plus noncurrent bytes.
  • The engine initializes the stats from its on-disk state at startup and updates them in memory as mutations succeed, so reads never list or rescan objects. They are exact for mutations made through objectstorage; direct filesystem edits are not watched, and a restart resyncs them.
  • storageLocations is a cheap filesystem-capacity snapshot: the storage directory with its pool observation in standalone mode, the drive paths in cluster mode.

Cluster health

const clusterHealth = await storage.getClusterHealth();

if (!clusterHealth.enabled) {
  console.log('Cluster mode is disabled');
} else {
  console.log(clusterHealth.nodeId, clusterHealth.quorumHealthy);
  console.log(clusterHealth.peers);
  console.log(clusterHealth.drives);
}
interface IClusterHealth {
  enabled: boolean;
  nodeId?: string;
  quorumHealthy?: boolean;
  majorityHealthy?: boolean;
  peers?: IClusterPeerHealth[];
  drives?: IClusterDriveHealth[];
  erasure?: IClusterErasureHealth;
  repairs?: IClusterRepairHealth;
  transport?: IClusterTransportHealth;
}

interface IClusterTransportHealth {
  mode: 'mutualTls' | 'unauthenticated';
  certificate?: { sha256: string; notAfter: number };      // only with mutualTls
  caCertificates?: { sha256: string; notAfter: number }[]; // file order; only with mutualTls
}

type TDriveMedia = 'rotational' | 'smr' | 'flash' | 'zoned';

interface IClusterDriveHealth {
  // index, path, status, capacity, error and erasure set fields, plus:
  media: {
    declared?: TDriveMedia;              // not reported yet: drives cannot declare their media
    detected: TDriveMedia | 'unknown';   // detected once at startup
  };
}
  • The engine computes every value; nothing is inferred from static config. Standalone mode returns { enabled: false }.
  • In cluster mode quicAddress is the local QUIC/UDP socket's bound address. With cluster.quicPort: 0 the OS assigns the port; use (await seed.getClusterHealth()).quicAddress in another node's seedNodes. Membership advertisements and peer quicAddress values carry that address, also after a restart.
  • Peer status is the local node's view of membership and heartbeats, so it is best-effort and may lag the network.
  • Drive health comes from live probes of the local drive paths on every call (a write, sync and read of a probe file); capacity values are cheap filesystem snapshots. Poll getClusterSnapshot() instead for a cheap periodic view (Dashboard).
  • quorumHealthy means the node currently sees majority quorum and enough available placements in every erasure set for the write quorum.
  • drives[].media.detected is detected once at startup. On Linux it follows the drive root's filesystem to the whole disk behind it (st_dev through /sys/dev/block; a partition counts as its disk) and reads queue/zoned and queue/rotational: zoned for host-managed or host-aware devices (SMR or ZNS), otherwise rotational or flash as the disk's driver sets queue/rotational. Virtual and network disks report what their driver or hypervisor sets: QEMU disks show rotational unless configured with rotation_rate=1, NVMe volumes show flash. It is unknown, never a guess, for filesystems without a block device of their own (tmpfs, overlay, network and FUSE filesystems, btrfs subvolumes), for device-mapper, md, loop and other virtual block devices (LVM, dm-crypt and software RAID included), when sysfs gives no usable answer, and on every platform but Linux. Drive-managed SMR reports itself as rotational and cannot be detected; smr can only ever be declared, which is planned. Detected media change no behavior today.
  • transport.mode is mutualTls when cluster.tls is set and unauthenticated otherwise. With mutual TLS, certificate and caCertificates describe the files loaded at startup (SHA-256 of each DER certificate in lowercase hex, notAfter in epoch milliseconds), so during a rotation they show what each node actually runs.
  • repairs exposes the healer's runtime state: whether a scan is active, the last completed run and the last error. repairs.shardsChecked, shardsHealed and failed count the last completed scan's placement checks, shards written and failures; repairs.shardsPending counts the pending shards that scan left pending (Self-healing). While a scan runs, lastRunStartedAt is its start and the counts are still the previous scan's; runHealingScan() returns one scan's start and counts together.

Dashboard

Every node can serve a small read-only dashboard: the cluster's nodes, drives, erasure layout and quorum, object and byte totals, request, error and byte rates, healing state and recent engine events. It is off by default:

const storage = await ObjectStorage.createAndStart({
  // ...
  dashboard: { enabled: true, address: '10.0.0.5', port: 3001 },
});
console.log(storage.dashboardPort); // the bound port; undefined while off

Open http://<address>:<port>/ in a browser. The page ships in the package, so nothing is fetched from elsewhere, and it updates itself over one TypedSocket connection to the same origin.

  • Configuration. dashboard.enabled must be set. address defaults to 127.0.0.1 and port to 3001; port: 0 asks the OS for a port. A port equal to server.port is refused at construction. An enabled dashboard that cannot bind fails start() and stops the engine: it never runs silently without it. The dashboard section is not sent to the engine.
  • What it shows. Nodes (status and last heartbeat in this node's view, start time); local drives of every node (status, usage, errors, last probe); the erasure layout and quorum; objects and bytes per node and summed; request, error and byte rates per node and summed, with two-minute sparklines kept in the browser; healing per node (last scan, checked, healed, pending shards, failures); and the newest events of each node (peer and drive status changes, healing scan results, engine start). A standalone node shows its totals, rates and events. Rates come from counters of the S3 listener; byte counts are taken at the TCP stream and include HTTP headers. Nothing is stored: a reload starts the sparklines again.
  • Where the data comes from. getClusterSnapshot() (also callable directly), built by the engine from memory only: the drive states the last heartbeat probe left, the runtime stats, healing state, S3 listener counters and a ring of the newest 64 events. It never probes a drive, syncs, walks a directory or reads the control store. local is this node's snapshot; in cluster mode cluster adds quorum, the erasure layout and every peer with the peer's own snapshot, asked for over QUIC in parallel with one second per peer (a peer that is not online or does not answer in time shows as unavailable in that sample). Standalone mode returns cluster: null. One snapshot runs at a time; a call while another runs rejects. Any node with the dashboard on therefore shows the whole cluster.
  • Cost. With no dashboard connected there is no timer, IPC or QUIC traffic. While at least one is connected, one shared sampler takes a snapshot every two seconds, keeps at most one in flight, skips ticks while one runs, and pushes the same sample (a few KB per node) to every connected dashboard. At most 16 dashboard sockets are admitted at once; the 17th is refused with 503. On the engine side, request and byte counting is a relaxed atomic add per response and per socket read or write. Measured on a three-node cluster: an enabled dashboard with nobody watching cost within 0.4 % CPU per request and no engine memory; its server modules and page add about 50 MiB to each node's Node.js process once, at start, without growing over time or with stored objects; with 16 dashboards open on every node, large uploads ran about 1.5 % slower and p99 latency was about 5 % higher.
  • Security. The dashboard is read-only by construction: its socket registers one read handler and nothing else, so it cannot start a healing scan, change configuration or touch S3 traffic or stored data. It has no authentication: protect it with the bind address and a firewall, for example by binding it to a private interface. It shows node IDs, addresses, drive paths, counts and event messages, never credentials or object keys.
  • Versions. Run every node on the same version, as clusters already require. An engine that predates the snapshot request logs a stream error and closes the stream, and the asking node shows it as unavailable.

Operations

Process lifecycle

  • start() spawns the engine and starts the HTTP server. Concurrent starts join the same attempt, and starting a running instance succeeds without a restart. A start is rejected while shutdown or unconfirmed startup cleanup is pending; after a completed shutdown, a start begins a new lifecycle. A start has no time limit, because startup grows with the stored data (a standalone root with bucket versions checks every version chain), but it fails at once when the engine process exits. A long start shows its progress on the host's stderr, where every engine log line is forwarded.
  • stop() stops admission, drains every accepted connection and joins request and multipart workers, then terminates the engine. The drain is bounded to 30 seconds; an incomplete drain rejects the call without terminating the process and keeps the server, descriptors, locks and unfinished work, so call stop() again once the blocked operation (for example hard-mounted I/O) recovers. Streamed reads move their file descriptor into the same owned lifecycle, so a disconnected client cannot detach a blocked filesystem read. Management calls still queued when a stop begins are rejected. A terminal shutdown error is reported only after the server released its work and resources; the wrapper terminates the engine before rejecting with it.
  • Concurrent stops join the same attempt, including any admitted startup. A stop after a failed startup joins the remaining cleanup without sending a command to a terminated process. Repeated completed stops replay the same outcome, terminal errors included; an incomplete drain or process termination stays retryable. A successful stop therefore confirms that the instance owns no listener, work or process any more; no health polling is needed before removing a disposable root.
  • If the engine has already exited, stop() cleans up the Node side without another command. Nonzero exits and signals other than SIGTERM are logged with exit code and signal, also with server.silent, and so is every unexpected exit below.
  • Unexpected engine exit. onUnexpectedEngineExit is called when the engine process ends while the instance is running (its start() completed) and stop() was not requested. Who asked decides, not the exit code: an outside SIGTERM, which the engine answers with exit code 0, calls it as an OOM kill does. A deliberate stop(), a failed start() (which rejects instead) and probeStoragePool() / findOrphanObjectLeaves() never call it. It is called once per started lifetime, synchronously, after the instance reports not running, with { code, signal } (null when absent); the wrapper neither awaits nor catches it. The instance does not restart itself: the host decides, for example by calling stop() and exiting so its supervisor restarts it.
const storage = new ObjectStorage({
  onUnexpectedEngineExit: () => void storage.stop().finally(() => process.exit(69)),
});
await storage.start();
  • Load shedding. At most 4,096 HTTP requests are in flight; further requests get a retryable 503 Service Unavailable with Retry-After: 1.
  • Signals. On SIGTERM or SIGINT the engine closes admission and runs the same graceful drain as stop(), admitted uploads with their file flush and publication barriers included, then releases its resources, flushes its output and exits with code 0, even while the parent's stdin pipe stays open. Shutdown has a 35-second deadline, then at most five seconds to flush output; a watchdog bounds the whole signal shutdown to 40 seconds. A second SIGTERM or SIGINT, a missed deadline or an output failure forces exit code 70, and a completed drain with terminal errors exits nonzero after flushing, so a silent parent logger cannot hide it. Every exit with code 70 first writes its reason to the engine's stderr as one line starting ruststorage forced exit (code 70): , whatever the log level, so it appears on the host's stderr ahead of the unexpected-exit line. Forced exits can lose unacknowledged writes and are reported as abnormal exits; incomplete client uploads never publish a partial object. This covers signals sent to the whole process group, by a watcher restart or Ctrl-C. Closing the engine's stdin (IPC EOF) requests the same graceful shutdown; an owned operation that cannot finish by the deadline makes the engine exit nonzero instead of waiting forever on a blocked filesystem worker.
  • Management replies. The engine writes its replies through one output writer with a 144 MiB line ceiling and a 160 MiB queue budget. A reader that keeps reading, however slowly, receives a reply of any length: the engine gives up on a reply only when its stdout accepts no byte for 30 seconds. Such a reply, and one beyond the line ceiling or the free queue budget, fails only its own call with an error starting Management reply; a partly written line is closed with a newline first, so the reader keeps its framing, and the engine keeps serving S3 and later calls. A failed or closed stdout, a full output queue of 256 lines or an event that cannot be queued ends the engine with code 70.

Bucket backup and restore

const appBackup = await storage.exportBucket({ bucketName: 'workapp-123' });
await storage.importBucket({ bucketName: 'workapp-123-restore', source: appBackup });
  • exportBucket() returns a self-contained smartstorage.bucket.v1 JSON export with only that bucket's objects and object metadata: no credentials, policies or other tenants' data. It is bounded to 10,000 objects, 48 MiB of payload and 8 MiB of serialized keys plus metadata; use migration control for larger buckets.
  • importBucket() validates each object's size and MD5, creates the target bucket if needed and restores the objects. In standalone mode each object is written like a PUT (staged and made durable without locking the bucket, then published), so a reader, and a crash during the import, sees an existing key's previous object or the imported one, never a mix. The bucket is not locked for the import as a whole: requests to it are served while the import runs, a fence on the bucket waits only for the object being committed, and an import whose bucket was deleted, versioned, put under retention, given a lifecycle or version thinning configuration or fenced meanwhile fails at the next object, keeping the objects it published before (requests in flight).
  • Export, import, fenced replacement and deletion, and migrations refuse versioned buckets, because none of them carries version history.

Bucket migration control

Standalone storage exposes durable, provider-bound migration control records; cluster mode reports this surface as unsupported. getBucketMigrationCapability() returns the local provider identity, the receipt key evidence, and source seal, destination publication and cleanup support. Create each side with createSourceMigration() and createDestinationMigration(), then advance only the explicit source path (active → sealed → cutoverCommitted → retired) with transitionSourceMigration() and the destination path (holding → verified → cutoverAuthorized → published) with transitionDestinationMigration(). Sealing introduces the source manifest digest, verification the destination's; terminal publication produces a signed receipt. Before cutover, a signed destination aborted terminal receipt can release a sealed source. Every call binds a positive fence token, the exact migration binding and an opaque controller capability; exact same-token retries replay the durable receipt, and stale or mismatched requests fail.

Source creation needs an existing bucket without retention and sourceCleanupAccessKeyId: ObjectStorage verifies that it is the bucket's one exact scoped credential, durably binds its hash before creating migration state, and revalidates that ownership before every source transition and before persisting a cleanup request. A provider has at most one nonterminal migration per role and logical bucket; terminal records stay as history and do not block a later migration ID. The standalone registry holds at most 100,000 migration states. Terminal receipts and completed-cleanup tombstones are kept permanently and count against that capacity, so controllers treat migration IDs as lifecycle records and plan capacity before it runs out. Registry creation is serialized globally and refuses a new migration ID atomically once 100,000 states exist; exact replays of existing IDs keep working at the bound, and the provider never deletes history or lets a manual registry-file reuse make room. Registry enumeration budgets durable locks, crash-left atomic temporary files and unrecognized entries separately, which leaves bounded recovery headroom at full capacity without deleting a temporary file another process may still own; every partition fails closed when it exceeds its own limit.

Sealing a source blocks later object, multipart, policy, bucket, tenant and fenced mutations. It refuses new writes from the moment it starts, waits for the commits already in flight, and refuses writes whose bodies were still arriving at their commit (503 MigrationSourceSealed), so nothing is published into the source once the seal returns (requests in flight). Reads, heads, listings, policy reads and multipart listings stay available while sealed. A read holds the bucket's publication guard only while it opens its object, so a later retirement cleanup waits for the opens in flight, not for responses still streaming, and such a response keeps returning the object it opened. Generic bucket deletion and multipart abort stay blocked throughout the seal. Versioning can be enabled on an active source, so sealing checks again and refuses a source that became versioned (EMIGRATION_VERSIONED; the source stays active).

After the destination is durably published and the source durably retired, the controller can permanently retire the source data:

const sourceState = await source.inspectBucketMigration({ migrationId });
const destinationState = await destination.inspectBucketMigration({ migrationId });

const cleanup = await source.cleanupSourceMigration({
  binding,
  cleanupFenceToken: 1,
  opaqueCapability: sourceCapability,
  sourceRetiredReceiptSha256: sourceState!.terminalReceipt!.receiptSha256,
  destinationPublishedReceipt: destinationState!.terminalReceipt!,
  resourceFence: {
    version: 1,
    scopeId: `${binding.authorityId}.${binding.logicalBucket}`,
    token: 42,
    mutationId: `${migrationId}.source-cleanup`,
    payloadSha256: cleanupIntentSha256,
  },
  accessKeyId: sourceTenant.accessKeyId,
});

cleanupSourceMigration() first persists a permanent requested-cleanup subrecord binding the exact retired-source receipt, the fully verified destination-published receipt, the cleanup token, the resource fence and the hashed access key. It then reuses exact fenced deletion to remove the bucket data, every matching multipart upload, the exact credential and the bucket policy, and persists the canonical cleanup receipt and the generic delete-result digest. Multipart deletion reconciles the runtime admission counter and cleanup queue at once. The cleanup waits for the commits and object opens in flight before it revokes credentials, and refuses at their commit the writes still receiving a body.

The controller owns recovery: ObjectStorage never starts source deletion on its own. Retry the exact cleanupSourceMigration() request after any timeout, process exit or ambiguous response. A requested-only retry repeats the exact resource deletion; if the deletion was already durably complete, its generic receipt is replayed and the migration cleanup record is finalized. After completion the request cannot be rebound, the bucket can never be recreated or credentialed, and unrelated global credential replacement resumes.

inspectBucketMigration({ migrationId }) returns the durable phase, the terminal receipts and the optional sourceCleanup requested/completed subrecord, all kept across restarts. storage.cleanSlate: true cannot erase or bypass that history: standalone startup is refused while any durable migration state exists, a terminal receipt or completed-cleanup tombstone included.

Durable resource fencing

Standalone storage with cleanSlate: false supports monotonic, crash-safe bucket replacement and deletion. In immediate mode (holdPublication omitted or false) callers drain normal writes before issuing a fenced mutation.

const fence = {
  version: 1 as const,
  scopeId: 'corestore-node-1.workapp-123',
  token: 42,
  mutationId: 'delete-workapp-123',
  payloadSha256: 'a'.repeat(64),
};

const result = await storage.deleteBucket({
  bucketName: 'workapp-123',
  accessKeyId: tenant.accessKeyId,
  fence,
});

Control planes that cannot publish the provider result atomically request a durable write-publication hold:

const held = await storage.importBucket({
  bucketName: 'workapp-123',
  source: appBackup,
  accessKeyId: tenant.accessKeyId,
  fence,
  holdPublication: true,
});

// Persist the provider result and update the control-plane resource first.
const release = await storage.commitResourcePublication({
  barrier: held.resourcePublicationBarrier!,
  resourceFenceReceipt: held.resourceFence!,
});

holdPublication: true is part of the effective mutation identity. ObjectStorage refuses new target-bucket mutations from the moment the held mutation starts, waits for the commits in flight before the destructive provider operation, then keeps all new target mutations behind a durable barrier once the provider effects and receipt are fully persisted. Mutations include object, multipart, bucket, policy, tenant and credential changes; they receive a retryable S3 WritePublicationHeld (503, Retry-After: 1) or EFENCE_PUBLICATION_HELD through the management API. Unrelated buckets stay available. Reads are not blocked: the protocol controls write publication and does not hide restored data from readers.

The exact mutation replay returns the same barrier; different and higher-token mutations cannot clear it. commitResourcePublication() validates the complete barrier, the canonical fence receipt and its 256-bit opaque commit capability, is safe to retry and returns the same release receipt. Treat commitCapability as a secret: do not log it or persist it outside protected control-plane state. A successful commit removes the raw capability from provider state and keeps only its hash.

There is no timeout, shutdown release or administrative bypass. A held barrier survives restart and is restored before the listener accepts requests. Invalid, unknown, oversized or insecure publication state fails startup closed. getResourceFencingCapability() reports requiresDrain: true for immediate mode and publicationHoldSupported: true, publicationHoldVersion: 1 and publicationHoldRequiresExternalDrain: false when the provider enforces the hold itself.

With accessKeyId, that exact key is bound into the durable mutation identity. A delete receipt is committed only after the bucket data is durably absent, that bucket-scoped credential is durably revoked and the bucket policy is durably absent. Retrying the same fence returns the stored receipt. Recovery after a crash repeats the credential and policy cleanup idempotently, so deletedCredentials can be 0, and a higher fence against an already absent bucket can return bucketDeleted: false. Completed exact deletes return credentialsPreserved: false and policyPreserved: false. Without accessKeyId, the fenced delete removes data only: tenant credentials and the bucket policy stay, and both preservation fields are true. In both cases the bucket's prefix credentials are revoked first, and the deletion is refused if that revocation cannot be stored.

After an exact delete, a new higher-token authority can recreate the same target without removing its fence history: call ensureBucketTenant() with the new fence first, then an exact fenced importBucket() with the same resource token. Ensure receipts use the smartstorage.bucket.ensure.v1 profile and attest the applied access key, the provider-computed secret hash and the effective region; the secret itself is never written to the receipt or the fencing registry. A completed fenced ensure returns its receipt on descriptor.resourceFence. computeObjectStorageResourceFenceEffectivePayloadSha256V1() computes the same canonical effective payload digest as the engine for the replacement, exact-delete and ensure profiles.

An ensure on an absent bucket removes any residual policy before creating the resource; on an existing bucket it keeps policy and object data. Once a bucket has durable fence history, createBucket(), createBucketTenant(), rotateBucketTenantCredentials(), deleteBucketTenant() and unfenced importBucket() fail with EFENCE_REQUIRED, and S3 CreateBucket and DeleteBucket get InvalidBucketState (409) with the EFENCE_REQUIRED: … message; use fenced ensure, exact import and exact delete for its lifecycle.

getBucketResourceFenceState({ bucketName }) reads one bucket's validated durable fence state under the provider's resource lock: the provider-root identity digest, bucket and resource identities, opaque scope, highest safe token and publication state (none, requested, publishing, held or released). It exposes no mutation receipts, payload evidence, commit capabilities, credentials or release proofs. null means that no fence state, sentinel or publication journal existed at the read's linearization point; a later mutation still applies the normal fencing checks. Management errors carry an ObjectStorageResourceFenceError with an EFENCE_* code (Errors).

Fences and requests in flight

A fenced replacement, deletion or publication commit and a migration transition wait only for the commits and object opens already in flight on their bucket, never for a request body that is still arriving or a response that is still streaming, and requests to the bucket do not queue behind them while they wait. A write that receives a body, copies or assembles (PutObject, CopyObject, UploadPart, UploadPartCopy, CompleteMultipartUpload, DeleteObjects, PutBucketPolicy, PutBucketVersioning, and the PUT and DELETE of ?lifecycle and ?lossless-version-thinning) holds the bucket's publication guard only while it commits, and a copy also while it opens its source. If such a mutation ran on its bucket after the request arrived, the write is refused at its commit and nothing of it is published: 503 WritePublicationHeld or MigrationSourceSealed while the bucket is held or sealed, 404 NoSuchBucket once it was deleted, otherwise 503 WritePublicationHeld. Every such mutation on the bucket counts, also one that fails, replays a stored receipt or changes nothing, so a write spanning it is refused with a retryable 503 even when the bucket did not change. A retry is authorized and applied against the bucket as it is now, so a write never publishes into contents, or with credentials, that a fence replaced or revoked on its bucket while it was in flight.

A copy's source bucket is checked when the copy reads its source:

  • CopyObject reads its source at its commit, so a mutation of the source bucket after the request arrived refuses it there (503 WritePublicationHeld, or 404 NoSuchBucket once the bucket is gone).
  • UploadPartCopy reads its source when it opens it, like a GET. A mutation that ends on the source bucket before the open refuses the request with the same codes. After the open, the part copies what it opened, like a GET already streaming, also when a fence then replaces or deletes the source bucket; only a mutation of the destination bucket refuses the part at its commit.

An unfenced importBucket() is a write like these for each of its objects: it holds the guard only while it commits one object, so a fence waits for that commit, and the import then fails at its next object with the same codes, keeping the objects it published before. A response already streaming keeps returning the object version it opened, also after a fence replaced or deleted it. A GET or HEAD that has to read its object in full to answer (the SHA-256 that verifies a retained object against its receipt, the ETag of an older engine's object without its .md5) releases the guard once it opened the object, so a fence never waits for that read and the request still answers the bytes it opened. Some operations still read an object in full while they hold the guard: a listing, for each such older object it lists; a copy or part copy from a retained bucket, which verifies its source while it opens it; a DeleteObject of a retained object, or one with If-Match of such an older object; in a versioned bucket, a PUT, CopyObject, CompleteMultipartUpload or DELETE whose key's current object is such an older object, which hashes it for the record of the version it supersedes; and a PUT into a retained bucket that meets an existing or concurrently created object, which verifies that object against its receipt. Cluster mode has no fences, and its writes take no publication guard.

Orphan object leaves

ObjectStorage.findOrphanObjectLeaves(directory, { reportLimit? }) scans a standalone storage directory read-only for canonical v2 object leaves whose payload file is missing, without starting the S3 listener. Such a leaf makes reads, writes and deletes of its key fail, and every listing of its bucket whose prefix and cursor range covers the key. The walk only opens, stats and reads directories. Its memory is bounded by the key-frame depth and reportLimit (default 1,000, at most 100,000), not by the number of keys. The report gives the scanned bucket and leaf counts, the total orphanCount, and up to reportLimit orphans with bucket, decoded key, leaf path and the leaf's first entry names.

Startup removes empty orphan leaves; a leaf that still holds sidecar files stays until it is repaired. From engine 10.7.0 on, a leaf appears only together with its complete payload, but a DeleteObject still removes the payload before the leaf, so a scan of a live pool can briefly report a key being deleted. Engines before 10.7.0 create the leaf before the upload body is complete: scan a pool such an engine serves while it is stopped, scan a snapshot, or scan twice and treat only leaves both runs report as orphans.

Clustering Deep Dive

Multiple nodes cooperate to store and serve data with built-in redundancy.

How it works

Client ──HTTP PUT──▶ Node A (coordinator)
                       │
                       ├─ Split object into 4 MiB chunks
                       ├─ Erasure-code each chunk (4 data + 2 parity = 6 shards)
                       │
                       ├──QUIC──▶ Node B (shard writes)
                       ├──QUIC──▶ Node C (shard writes)
                       └─ Local disk (shard writes)
  1. Any node coordinates. A client connects to any member.
  2. Objects are chunked into fixed-size pieces (cluster.erasure.chunkSizeBytes, default 4 MiB).
  3. Each chunk is erasure-coded with Reed-Solomon into k data and m parity shards.
  4. Shards are distributed across nodes and drives for fault isolation.
  5. Quorums. A write is acknowledged once k+1 shards are persisted; a read needs k shards and prefers local ones. A chunk with fewer than k+m written shards is published with the missing ones as pending shards, which healing completes on their assigned drives.

Buckets, manifests and credentials are per node: a manifest lives on the node that coordinated the write, and runtime and prefix credentials are not replicated.

Failure timing. A node that membership marks offline is never contacted: its shards of a write are pending at once, reads take their shards from the other nodes, and below either quorum a request fails at once. Until a stopped node is marked offline (five missed heartbeat rounds), each chunk that needs it waits up to the QUIC idle timeout (10 s). Cluster connections send keep-alives every 2 s from both ends, so a slow but live peer stays connected however long a request takes; there is no whole-request deadline. A heartbeat to a silent peer therefore fails after 10 s, and because each round goes to every peer at once, stopped nodes are marked offline after about 50 s by default, however many stop together (a whole location, for example). A node whose restart takes longer comes back to pending shards and healing work. heartbeatTimeoutMs (default 30 s) bounds how long a live peer may take to answer a heartbeat, but cannot extend the wait for a silent one beyond 10 s.

Cluster roots

Cluster mode never creates its roots. Before startup, create storage.directory and every cluster.drives.paths entry, owned by the ObjectStorage process user, without group or world write permission. Every root must be a real directory with no symlink path component, and every drive root must resolve to a distinct device/inode identity and, unless cluster.drives.allowSharedDevice is set, to a distinct device. The control root may be its own directory or the same identity as one drive root, but it cannot make two drive entries aliases of each other. Every root must support the durability and locking semantics checked at startup; ObjectStorage fails closed before accepting traffic otherwise. Upgrades are one-way and every member runs the same version (Upgrades and rollbacks).

Erasure coding

With the default 4+2 configuration the storage overhead is 50 % (against 200 % for 3x replication), any 2 drives or nodes may fail at once, and any 4 of 6 shards rebuild a chunk.

Config Total shards Overhead Tolerance Min nodes
4+2 6 50% 2 failures 3
6+3 9 50% 3 failures 5
2+1 3 50% 1 failure 2

Erasure set formation

Drives are organized into fixed erasure sets at cluster initialization, interleaved across nodes for fault isolation:

3 nodes × 4 drives each = 12 drives total
With 6-shard erasure sets → 2 erasure sets

Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1
Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3

New nodes form new erasure sets; existing data is never rebalanced.

Streaming Reads

A cluster GET streams the object chunk by chunk; nothing is written to the control root. It reads from the generation of the manifest it loaded at its start, without holding the object's operation gate, so overwrites, deletes and other GETs of the same key run while it streams. Each chunk is rebuilt from the first k valid shards, read in parallel, and at most one rebuilt chunk waits ahead of the client.

  • Errors after the headers abort the response. The first chunk is rebuilt before the status line, so an unreadable object still gets an error status. A failure in a later chunk ends the body with an error (the connection is closed), never with a clean end of body. A whole-object GET sends its last bytes only after the total length and the object MD5 match the manifest; range GETs are checked for length only.
  • One deadline for the whole GET. cluster.getDeadlineMs is the longest a single GET may run, counted from its start and including the client's transfer time; a GET that takes longer is aborted, and if its client stopped reading, its connection is closed. The default is 6 hours, the allowed range 1 minute to 7 days. The cleanup grace is derived from it: superseded and deleted generations keep their shards for the deadline plus the smaller of the deadline and one hour (7 hours by default, 2 minutes at the 1-minute floor), so every GET that started on a generation has ended before cleanup frees it. Writes in progress hold their object's gate and are never cleaned up, however short the grace. The setting must be the same on every node; a mismatch can abort streams, but never serves wrong bytes. CopyObject and CompleteMultipartUpload read their sources under the object gate with their own fixed 6-hour limit, counted from the start of the operation.
  • GET memory budget. Rebuilding chunks is charged to an engine-wide budget of max(1 GiB, 16 × chunk footprint) at the configured chunk size. A chunk's footprint is ⌈chunk/k⌉ × (k + min(k, m)) + chunk: the k fetched shards, the data shards rebuilt for missing ones (a read rebuilds no parity shards) and the decoded chunk; 4+2 at 4 MiB is 10 MiB, 1+15 at 64 MiB is 192 MiB. A GET whose client went away keeps its charge until its local shard reads have ended, and shutdown waits for them before it releases the drive roots. The budget is separate from the upload budget. The first chunk waits at most 5 seconds for budget and is otherwise refused with 503 SlowDown and Retry-After: 1; later chunks wait. Slow downloaders hold up to two rebuilt chunks each, so many of them can exhaust the budget and turn new GETs into 503s under load.

Superseded generations are queued with their exact shard placements; the cleanup worker runs once per grace period and at least hourly, and purges their shard records and tombstones once the grace has passed.

QUIC transport

Nodes talk over QUIC (quinn): multiplexed streams carry concurrent shard transfers without head-of-line blocking, connections to peers are kept and reused, and QUIC flow control applies backpressure to slow peers. With cluster.tls, every connection is TLS 1.3 with a CA-signed certificate on both ends; without it, each node generates a self-signed certificate at startup and accepts any peer (Cluster transport trust). The QUIC protocol identity is smartstorage.

Cluster transport trust

Without cluster.tls the cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's quicPort can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port. Run such a cluster only on a network where every host that can reach quicPort (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with quicPort firewalled from everything else.

With cluster.tls set, the transport is mutual TLS (TLS 1.3, no session resumption, no early data) with a cluster CA you run:

  • Identity. A node's identity is the single DNS subject alternative name (SAN) of its certificate, and it equals cluster.nodeId. With cluster.tls, nodeId is required and must be a lowercase DNS name such as node-1 or node-1.storage.internal; it is never generated, and it must still match the node id persisted in the control root. One certificate serves as both the server and the client certificate, so it must be valid for both: an extended key usage extension listing serverAuth and clientAuth, or no extended key usage extension at all (the recipe below issues both). Use a dedicated CA per cluster: every certificate it signs with one lowercase DNS SAN is a member of the cluster.
  • Checks. A node connects to a known peer only if the peer's certificate names exactly that peer's node id (one DNS SAN, no wildcard), and to a seed (seedNodes, known only by its address) only if the seed presents a certificate from the cluster CA. A node accepts a connection only with a client certificate from the cluster CA, and a join or heartbeat only for the node id that certificate names. A peer without such a certificate fails the handshake before it can send any request. Every CA-signed node is a full member: it may sync topology and read, write, tombstone and purge any shard, so the way to remove a compromised node is to stop trusting its certificate (a new CA, see the rotation below).
  • Startup. A node checks its own certificate the way its peers will (chain, validity, server and client authentication, the SAN, and that keyFile matches certFile) and refuses to start if that check or any file check below fails. The error names the file and the reason. A node never falls back to the unauthenticated transport. At startup it logs one line with its certificate's notAfter and the SHA-256 of each CA certificate, and getClusterHealth().transport reports them (see Cluster health).
  • Files. caFile holds one CA certificate, or two during a CA rotation; anything else is refused, which keeps a system CA bundle out. certFile holds the node certificate, optionally followed by intermediates, and nothing but certificates. keyFile holds exactly one unencrypted PKCS#8, SEC1 or PKCS#1 key; ECDSA P-256 is recommended. Each path must be absolute and canonical, with no symbolic link anywhere, and lie outside the control root and every drive root (roots are backed up and replaced, and a key must not travel with them). That check compares paths, so a bind mount that makes a root reachable under another path is not detected. Each file must be a regular file of at most 64 KiB. The files and their directories must be owned by the engine's user or by root and must not be group- or world-writable, and keyFile must be owned by the engine's user and grant no group or world permission (mode 0600 or 0400).
  • Reload. The files are read once at startup; a restart is the reload. Certificates are checked at every handshake, so a connection established before a certificate's notAfter keeps running until it reconnects, and new handshakes after notAfter fail. Renew certificates and restart the nodes before they expire.

Making the CA and the node certificates

The engine ships no CA tooling. With OpenSSL 3 or LibreSSL 3.3 (the openssl of macOS), create the CA once and keep ca.key offline. ec_param_enc:named_curve is required for LibreSSL, which otherwise writes the curve as explicit parameters that the engine refuses; OpenSSL 3 writes the named curve either way:

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
  -pkeyopt ec_param_enc:named_curve -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
  -subj "/CN=objectstorage cluster CA" \
  -addext "basicConstraints=critical,CA:TRUE" \
  -addext "keyUsage=critical,keyCertSign,cRLSign" \
  -out ca.pem

Then issue one certificate per node, with the node id as its only DNS SAN:

NODE_ID=node-1
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
  -pkeyopt ec_param_enc:named_curve -out "$NODE_ID.key"
openssl req -new -key "$NODE_ID.key" -subj "/CN=$NODE_ID" -out "$NODE_ID.csr"
cat > "$NODE_ID.ext" <<EXT
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature
extendedKeyUsage = serverAuth,clientAuth
subjectAltName = DNS:$NODE_ID
EXT
openssl x509 -req -in "$NODE_ID.csr" -CA ca.pem -CAkey ca.key -CAcreateserial \
  -days 365 -sha256 -extfile "$NODE_ID.ext" -out "$NODE_ID.pem"

Install ca.pem and the node's certificate and key on the node, as the user that runs the engine and outside its control and drive roots:

install -d -m 0755 /etc/objectstorage/tls
install -m 0644 ca.pem "$NODE_ID.pem" /etc/objectstorage/tls/
install -m 0600 "$NODE_ID.key" /etc/objectstorage/tls/

Then set cluster.tls to the three paths (see Configuration). The test suite runs exactly these commands (test/test.cluster-tls.node.ts).

Docker secrets are mounted owned by root with mode 0444 by default, which the key check refuses. Set the key secret's uid to the engine's user and its mode to 0400 in the service's long secrets syntax. Docker Compose ignores uid, gid and mode for secrets read from a file, because it bind-mounts the file; there the host file itself must be owned by the engine's uid with mode 0600 or 0400. Kubernetes secret volumes present their files through symbolic links, which the canonical-path rule refuses.

Enabling mutual TLS

Nodes with and without cluster.tls cannot talk to each other: a TLS node refuses peers without a client certificate and refuses self-signed servers. Enabling TLS is therefore not a rolling change: stop every node, configure cluster.tls on every node, then start every node. A node whose nodeId was generated keeps that id (the control root persists it, and getClusterHealth().nodeId reports it): configure it as nodeId and issue the node's certificate for it, or the node refuses to start. Disabling TLS takes the same full stop.

Rotating certificates and the CA

A node certificate is renewed by issuing a new one from the same CA and restarting the node; its peers need no change. A new CA takes three rolling restarts, with no cluster stop:

  1. Put both CAs into every node's caFile (two certificates) and restart the nodes one at a time.
  2. Issue every node a certificate from the new CA and restart the nodes one at a time.
  3. Remove the old CA from every caFile and restart the nodes one at a time.

Every pair of nodes stays connected throughout, because trust only grows before the certificates change and only shrinks after. The order matters: a node with a certificate from the new CA cannot connect to a node that trusts only the old one.

Cluster membership

  • Static seed nodes define the initial cluster in config; new nodes can join a running cluster.
  • Heartbeats every heartbeatIntervalMs (default 5 s) detect suspect and offline peers; nodes mark peers offline only while they hold a majority, which prevents split-brain decisions.

Self-healing

Every 24 hours, and whenever runHealingScan() asks for it, each node scans the manifests it coordinated (manifests live on the node that wrote them):

  1. Every shard placement is checked at the drive it names, locally or through its node, against the checksum and size the manifest recorded. The check reads the shard record's header and the record's length only, never its payload, so a scan costs a few small reads per shard. Verifying payloads is not the scan's job: a read detects a rotted payload through its CRC32C and rebuilds the chunk from the other shards.
  2. A missing shard is rebuilt from k valid shards with Reed-Solomon, checked against the recorded checksum, and written to the drive its placement names. The manifest does not change.
  3. A pending shard is written to the drive its manifest recorded and then moved into the chunk's placements, in the same version and generation; the completed manifest is identical to one whose write never failed.

Healing never places a shard on a drive of its own choosing. Placements and pending shards on an offline node wait for that node; a pending shard whose drive refuses writes stays pending and is reported in repairs.shardsPending (replacing a permanently lost drive is not supported yet). A shard record that cannot be read, such as a truncated one, is reported in repairs.failed and left in place, because shard records are immutable; a node that cannot read a record answers with an error, never as if the shard were missing. A minority partition does not heal: its scan fails. Placements are checked without the object's operation gate, as a GET reads them. The chunks of an object that need a write are healed in batches of up to 1 GiB of chunk data: each batch takes the object's gate, reloads the manifest, heals only chunks that are unchanged in the same generation, publishes its pending shards with one manifest store and releases the gate before the next batch. An object whose gate is busy is skipped until the next scan.

Scans never overlap; a scan that outlasts the 24-hour interval delays the next one.

runHealingScan(options?) runs a scan on this node and returns its startedAt, completedAt, durationMs, shardsChecked, shardsHealed, shardsPending and failed. It joins the next scan that starts after the request arrives, never one already running, and requests that arrive together share one scan; so it covers every write that finished before the call. It rejects in standalone mode, when the scan fails (without a majority, or when a bucket's manifests cannot be listed) and when the engine shuts down before the scan completes, and it is refused while 16 requests already wait for a scan to start. The engine answers other management calls while the scan runs. options.timeoutMs (default 300,000) bounds the wait; the engine cannot cancel a scan, so one that outlasts it keeps running, its result is discarded, and a later call joins the next scan that starts after it.

Supported S3 Operations

The router fails closed, in standalone and cluster mode alike. A request selects one operation from its method, its path (/, /{bucket}, /{bucket}/{key}) and its query names, and each query name must be one that operation implements. Anything else gets NotImplemented (501, with the S3 XML error body) before the request takes a lock, reads its body or touches a bucket or object. A credential the request is not allowed for still gets AccessDenied (403) first.

Operation Request Query names besides the selector Notes
ListBuckets GET / none
CreateBucket, DeleteBucket, HeadBucket PUT, DELETE, HEAD /{bucket} none
ListObjects GET /{bucket} prefix, delimiter, max-keys, marker, encoding-type pagination
ListObjectsV2 GET /{bucket}?list-type=2 prefix, delimiter, max-keys, continuation-token, start-after, fetch-owner, encoding-type pagination
ListObjectVersions GET /{bucket}?versions prefix, delimiter, max-keys, key-marker, version-id-marker, encoding-type every version and delete marker
ListMultipartUploads GET /{bucket}?uploads prefix, encoding-type not paginated
GetBucketLocation GET /{bucket}?location none returns server.region; an empty constraint for us-east-1
GetBucketVersioning, PutBucketVersioning GET, PUT /{bucket}?versioning none PUT standalone only, NotImplemented in cluster mode
GetBucketLifecycleConfiguration, PutBucketLifecycleConfiguration, DeleteBucketLifecycle GET, PUT, DELETE /{bucket}?lifecycle none PUT and DELETE standalone only, NotImplemented in cluster mode; PUT needs CRC32 or Content-MD5 integrity; version retention
Version thinning (extension) GET, PUT, DELETE /{bucket}?lossless-version-thinning none PUT and DELETE standalone only, NotImplemented in cluster mode; version retention
GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy GET, PUT, DELETE /{bucket}?policy none
DeleteObjects POST /{bucket}?delete none up to 1,000 keys, CRC32 or Content-MD5 integrity
GetObject, HeadObject GET, HEAD /{bucket}/{key} versionId, response-cache-control, response-content-disposition, response-content-encoding, response-content-language, response-content-type, response-expires Range on GET; conditions
DeleteObject DELETE /{bucket}/{key} versionId standalone If-Match
PutObject, CopyObject PUT /{bucket}/{key}, CopyObject with x-amz-copy-source none a copy needs s3:GetObject on the source and s3:PutObject on the target; x-amz-copy-source-if-*
CreateMultipartUpload POST /{bucket}/{key}?uploads none
UploadPart, UploadPartCopy PUT /{bucket}/{key}?partNumber=…&uploadId=… none UploadPartCopy: x-amz-copy-source, optional x-amz-copy-source-range: bytes=first-last and x-amz-copy-source-if-*, same permissions as CopyObject
CompleteMultipartUpload POST /{bucket}/{key}?uploadId=… none
AbortMultipartUpload DELETE /{bucket}/{key}?uploadId=… none
ListParts GET /{bucket}/{key}?uploadId=… max-parts, part-number-marker standalone

Request rules

Every operation also accepts the x-id label that AWS SDKs append and the SigV4 query-signing parameters of a presigned URL (X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Signature, X-Amz-Security-Token, X-Amz-Content-Sha256). Query-string signatures are not verified: a presigned request is served as an unsigned one, which is anonymous when auth is enabled.

An object GET or HEAD (GetObject, HeadObject, ListParts) ignores query names S3 does not define, as S3 does, so links with cache-busters (?v=1) or tracking parameters (?fbclid=…) keep working. A name S3 defines that the operation does not implement (?acl, ?partNumber=1, …) is still refused there, and every other request (bucket level, every write and delete) refuses unknown names too. The set of names S3 defines is taken from the AWS SDK v3 model (3.1039.0). With CORS enabled, the 501 and 405 responses carry the CORS headers.

The response-* parameters of GetObject and HeadObject replace the Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Type and Expires headers of a 200 or 206 response with the given value, verbatim. A value that is not a valid HTTP header value (a control character such as CR or LF) returns InvalidArgument (400). With auth enabled, an anonymous request that carries one returns InvalidRequest (400), as S3 does, because an override such as response-content-type=text/html would let a public object serve script from the storage origin; a presigned URL is such a request. With auth disabled the overrides apply. A 304 Not Modified carries the Cache-Control and Expires overrides (the headers RFC 9110 requires a 304 to repeat from the 200) and none of the others.

Bucket versioning

Standalone buckets support S3 versioning exactly as S3 defines it: a bucket is unversioned until PutBucketVersioning sets Enabled or Suspended, and it never goes back to unversioned. New buckets are unversioned. A bucket that was never versioned behaves exactly as before versioning existed, on disk and on the wire: no version ids, no x-amz-version-id header, null as its only version id. In cluster mode PUT ?versioning answers NotImplemented (501), so every cluster bucket stays unversioned for now.

  • Version ids. A version written while versioning is Enabled gets a lowercase UUID as its version id. Versions written before versioning was enabled, and every version written while it is Suspended, are the key's null version; a key has at most one.
  • Enabled. PUT, CopyObject and CompleteMultipartUpload add a new current version and keep the previous one as noncurrent; the response carries x-amz-version-id. A DELETE without a version id adds a delete marker as the current version (also for a key with no object, as S3 does): GET and HEAD then answer NoSuchKey (404) with x-amz-delete-marker: true and the marker's x-amz-version-id, and ListObjects no longer lists the key. The DELETE response carries x-amz-delete-marker: true and the marker's id.
  • Suspended. Writes create the null version and replace any previous null version of the key, wherever it is in the history; versions with ids stay. A DELETE replaces the null version with a null delete marker. Responses carry x-amz-version-id: null.
  • By version id. GET, HEAD and DELETE take ?versionId= (an id or null). Reading a delete marker by its id answers MethodNotAllowed (405) with x-amz-delete-marker: true, its id and its Last-Modified. DELETE with a version id removes that version for good; when it was the current one, the next newest version becomes current again (a delete marker, or the object it covered). A well-formed id that names no version of the key is NoSuchVersion (404), also on DELETE; a malformed one is InvalidArgument. DeleteObjects accepts a VersionId per object and reports VersionId, DeleteMarker and DeleteMarkerVersionId per deleted entry.
  • Copies. A copy source may name a version (?versionId= after the source); x-amz-copy-source-if-* apply to that version, and the response carries x-amz-copy-source-version-id (and x-amz-version-id for the new version in a versioned bucket). A delete marker as a copy source is InvalidRequest (400); a source whose current version is a delete marker is NoSuchKey. UploadPartCopy works the same way.
  • Conditional requests. If-None-Match: * on PUT and CompleteMultipartUpload succeeds when the key has no current version or its current version is a delete marker. If-Match on DELETE is evaluated against the current version, also with a version id; a current delete marker fails it (412). GET and HEAD with a versionId evaluate their conditional headers against that version (see Conditional GET and HEAD).
  • ListObjectVersions lists every version and delete marker in key order, each key newest first, with IsLatest on the current one; versions and delete markers, and common prefixes, all count against max-keys. A truncated page carries NextKeyMarker and, after a version, NextVersionIdMarker; key-marker alone resumes after every version of that key, with version-id-marker after that version of it. A version-id-marker naming a version that was deleted between the pages resumes at that key's newest remaining version: no version of the key is skipped, but versions the previous page already listed are listed again. Last-Modified of a version is its commit time.
  • Permissions. PutBucketVersioning is authorized as s3:PutBucketVersioning, GetBucketVersioning as s3:GetBucketVersioning, ListObjectVersions as s3:ListBucketVersions; a versionId (also null, also in DeleteObjects) as s3:GetObjectVersion or s3:DeleteObjectVersion. Only root credentials may change a bucket's versioning: bucket tenant credentials get AccessDenied for PutBucketVersioning (they may read it with GetBucketVersioning, list versions, and read and delete versions in their bucket), because the operator's bucket flows (export, import, fenced replacement and deletion, migrations) do not carry version history and refuse versioned buckets. The same holds for the lifecycle and thinning configuration (see Version retention). This lifts once a format that carries history exists. Prefix credentials never name a version and cannot change versioning; they can list versions inside their prefix, and their DELETEs add delete markers.
  • Not combined with. A bucket with engine-enforced immutable retention refuses versioning, and a versioned bucket refuses retention (InvalidBucketState, 409). Bucket export and import, fenced bucket replacement and deletion, and bucket migrations refuse versioned buckets, because none of them carries version history; a migration whose source became versioned after it was created cannot be sealed (EMIGRATION_VERSIONED, the source stays active). Export, import, fenced replacement and migrations also refuse a bucket with a lifecycle or a thinning configuration, which they do not carry either (same codes, the message names the reason); deleting a bucket, fenced or not, removes its configuration. MfaDelete is not implemented (501). A versioned bucket is empty for DeleteBucket only once every version and delete marker is deleted (BucketNotEmpty before); deleting it forgets its versioning, so a new bucket of the same name starts unversioned. deleteBucketTenant deletes every version with the bucket.
  • Requests by version are served under the bucket's gate. Every request whose answer depends on a bucket's versioning state reads that state, and the history, under the bucket's gate: DELETE and each DeleteObjects member, GET and HEAD with a version id (also null on a bucket that was never versioned), a copy or UploadPartCopy source with a version id, the delete-marker answer of a key without a current object, and ListObjectVersions. A request that arrives while PutBucketVersioning or a write's commit is in progress is therefore served by the state that change leaves. A write takes the gate only to commit: PUT, CopyObject and CompleteMultipartUpload first receive, copy or assemble their payload without it, and their commit checks again that the bucket exists, its retention, its versioning state and If-None-Match: *. A write that PutBucketVersioning, retention or DeleteBucket overtook while its body arrived commits under the new state. A copy publishes the source as it is at its commit: its x-amz-copy-source-if-* conditions are evaluated again there (a condition that no longer holds answers 412), and a copy of the current object (or of versionId=null) whose source was replaced (by a write or an importBucket()) while it was copied is refused with 409 ConditionalRequestConflict and can be retried. A copy that names any other version id copies that version, also when the key changed meanwhile, and UploadPartCopy copies its source as it was when the part copy began. Requests by version, like every listing and delete, therefore never wait for uploads to the bucket in progress: measured with one 1 GiB upload into the bucket (one container capped at 8 CPUs and 12 GiB, xfs, auth off), GETs by version id issued every 50 ms during it answered in 1.3 to 1.5 ms (median; p99 2.4 to 7.8 ms), at full loopback speed and at 50 MB/s alike, against 1.4 to 1.6 ms without an upload; 11.2.0 answered such a GET only once the upload had finished, after 2.8 to 3.0 s at full loopback speed and 20.4 s at 50 MB/s. GET and HEAD without a version id read an object this engine wrote outside a retained bucket without the gate; every object of a retained bucket, an object in the sidecar layout of an older engine, and a key that a write or delete changes while the request looks it up are read under it, so such a request waits for a commit in progress, never for an upload. A read that has to hash its object in full (the SHA-256 that verifies a retained object against its receipt, the ETag of an older object without its .md5), with or without a version id, does so after it released the gate, so a commit never waits for that hash. Every GET and HEAD reports the size, ETag, Last-Modified, metadata and version id of the bytes its body returns.
  • Startup. A standalone engine checks every version chain once when it starts, reading every version's record and opening its payload (and repairing what a crash left), and takes the bucket counters from that check. Startup time therefore grows with the number of stored versions. Measured with 1,000,000 versions (100,000 keys × 10) on 8 CPUs with a 12 GiB memory limit, on a KVM host whose page cache did not keep the root between starts: the first start after writing them took 346 s; six clean starts took 127 to 264 s (127, 145, 150, 193, 203 and 264 s, median about 170 s); three starts after a SIGKILL during writes by 32 concurrent writers took 215 to 256 s. A root on which no bucket was ever versioned pays nothing for this and starts exactly as before. start() waits for as long as startup takes; once recovery passes 10 s, the engine logs its progress (chains, versions, elapsed time) to the host's stderr every 10 s.
  • Commit times. Version commit times come from one stamp clock per engine: max(wall clock, last stamp + monotonic time since), persisted to .smartstorage/versioning/stamp-clock.json by every PutBucketVersioning that changes a bucket's state (before the state itself, and before the response), every 60 seconds and on a graceful stop. The monotonic time keeps counting while the host is suspended (CLOCK_BOOTTIME on Linux, CLOCK_MONOTONIC on macOS). Within a process, a wall clock that jumps back, also across a suspend, never produces a commit time earlier than true time. After a restart with the wall clock behind, commit times can lag true time by at most the downtime plus 60 seconds. A forward jump that is corrected later keeps commit times ahead by the jump for the life of the process. While commit times run ahead of the wall clock, getStorageStats() and getHealth() report stampClockAheadOfWallMs. Within a key, commit times never go backwards, and a version becomes noncurrent at its successor's commit time.

The reads of an unversioned bucket keep their 11.0.0 semantics:

  • GET ?versioning returns the empty <VersioningConfiguration/> of a bucket that was never versioned.
  • GET ?lifecycle returns NoSuchLifecycleConfiguration (404) for a bucket without a lifecycle configuration, which is every cluster bucket; in cluster mode PUT and DELETE ?lifecycle are NotImplemented.
  • GET ?versions lists every current object as its S3 null version (VersionId null, IsLatest true), with CommonPrefixes and paging on key-marker and version-id-marker. A truncated page returns the last key or common prefix as NextKeyMarker, plus NextVersionIdMarker null after a key. A version-id-marker needs a key-marker.
  • versionId on GET, HEAD and DELETE, a VersionId of a DeleteObjects member, and a ?versionId= after the copy source of CopyObject and UploadPartCopy: null is the current object. A well-formed id (a lowercase UUID) names no version and returns NoSuchVersion (404; NoSuchBucket when the bucket is missing). Anything else, the empty string included, returns InvalidArgument (400); in DeleteObjects both fail only their member. A ? inside a copy-source key must be encoded as %3F.
  • These reads are authorized as s3:GetBucketVersioning, s3:GetLifecycleConfiguration and s3:ListBucketVersions. A versionId, null included, is authorized as s3:GetObjectVersion (GET, HEAD, and a copy source's ?versionId=) or s3:DeleteObjectVersion (DELETE), not as s3:GetObject or s3:DeleteObject, so a policy that allows DeleteObject and denies DeleteObjectVersion keeps protecting history once buckets are versioned. Bucket tenants may use both actions in their bucket.

Version retention: lifecycle and thinning

Standalone buckets expire noncurrent versions by S3 lifecycle rules, and an extension overlay keeps a thinned history on top of them. Only lifecycle NoncurrentVersionExpiration ever deletes a noncurrent version; the overlay only keeps versions a rule would otherwise expire. GetBucketLifecycleConfiguration is therefore always an upper bound on what can be deleted, and DeleteBucketLifecycle stops all deletion: once it has answered, the sweep removes nothing more under the old rules. The owner's three policies:

Policy Lifecycle rule Overlay
Keep the last 6 versions NoncurrentDays 1, NewerNoncurrentVersions 5 (the current version plus 5; S3 requires a days floor) none
Keep every version for 24 hours NoncurrentDays 1 none
24 hourly, 7 daily, 4 weekly, 12 monthly NoncurrentDays 1 Hourly 24, Daily 7, Weekly 4, Monthly 12

S3 rounds NoncurrentDays up to the next midnight UTC: a version that became noncurrent at 10:30 with one day is due at 00:00 two days later, so "24 hours" keeps every version for more than 24 and at most 48 hours (and covers any hourly count up to 24).

  • The lifecycle subset. PUT ?lifecycle (PutBucketLifecycleConfiguration) accepts rules with an empty Filter, a Filter Prefix or the old rule Prefix, a Status, NoncurrentVersionExpiration (NoncurrentDays from 1, required; NewerNoncurrentVersions from 1 to 100) and Expiration{ExpiredObjectDeleteMarker}. Any other element (Transition, Expiration Days or Date, tag or size filters, NoncurrentVersionTransition, AbortIncompleteMultipartUpload) refuses the whole document with NotImplemented (501), and so does the x-amz-transition-default-minimum-object-size header; nothing is accepted and then ignored. Up to 1,000 rules and 64 KiB. The body needs Content-MD5 or x-amz-checksum-crc32 (AWS SDKs send one), and a wrong one is BadDigest. GET ?lifecycle returns the document; DELETE ?lifecycle answers 204, also without one. Within a rule a version expires only when it is past NoncurrentDays and has at least NewerNoncurrentVersions newer noncurrent versions; any matching rule can expire it, as in S3. Delete markers never count as newer versions. A noncurrent delete marker expires with its rule when no kept data version lies directly below it, and a current delete marker with no version left below it is removed when an enabled rule sets ExpiredObjectDeleteMarker.
  • Version thinning (?lossless-version-thinning, an extension): <VersionThinningConfiguration><Hourly>24</Hourly><Daily>7</Daily><Weekly>4</Weekly><Monthly>12</Monthly></VersionThinningConfiguration>, each count from 1 to 1,000, at least one. Periods are UTC hours, days, ISO weeks (from Monday 00:00) and calendar months; the window of a count n is the n periods ending with the current one, and in each of them the newest data version is kept (the current version fills its own period). GET returns the document or NoSuchVersionThinningConfiguration (404); DELETE answers 204. The overlay keeps versions only for keys a noncurrent rule covers; without such a rule it has nothing to keep.
  • Unversioned buckets store both configurations and report them inactive; nothing expires until the bucket is versioned. Versioning-suspended buckets expire noncurrent versions exactly like enabled ones, as S3 does.
  • Permissions. GET is authorized as s3:GetLifecycleConfiguration, PUT and DELETE (of both subresources) as s3:PutLifecycleConfiguration, so bucket policies apply. Bucket tenant credentials may read both and get AccessDenied for PUT and DELETE: a stored configuration refuses the operator's export, import, replacement and migration flows, so a tenant must not be able to set one. Prefix credentials are refused all six. Both configurations are bucket mutations for publication holds and migration seals (a sealed source answers MigrationSourceSealed).
  • Storage. Both are kept in .smartstorage/versioning/lifecycle/<bucket>.json, written durably before the response; a file that fails validation stops the engine from starting. A bucket deletion removes it.

The sweep. The engine removes expired versions in passes: the first 15 minutes after it started, then 15 minutes after each pass ended, requested or timed. A pass visits every versioned bucket with an enabled rule that can expire something (no other bucket is touched) and reads each version chain without holding the bucket's gate. It then removes what expired in holds of the bucket's write gate. A hold covers at most 32 consecutive keys with at most 256 noncurrent versions and delete markers together, expired or not, because each key costs the hold a sync of its chain directory per phase and a reload of its chain; a key with a longer history runs alone, in holds of at most 256 removals, as many as it needs. Under the gate it checks that each key's chain is still the one it evaluated, apart from newer versions on top; a key that changed leaves the hold and is evaluated again. After every hold the sweep waits seven times as long as that hold lasted before it takes the bucket's gate again, so it holds the gate at most an eighth of the time and writes to the bucket get through between its holds. The pause is longer than the hold because the filesystem's work for a hold outlives it: the writes after a hold pay for part of it (measured on xfs, with ext4 as the control), so a client writing back to back loses more of its rate than the share of time the gate is held. Within a hold the work runs in phases across all its keys: the expired records of every key, then a sync of each key's chain directory, then their payloads and a sync of each, then the expired current delete markers and a sync of each. A key's payloads go only once its records are durable, and its delete marker only after its versions. A hold that fails part way repairs every key it touched. A crash at any point leaves a consistent history (startup recovery finishes the job), and the next pass completes the expiry. After a restart a pass starts over; no position is kept.

Sweeping on request. runVersionSweep(options?) runs a pass and returns its startedAt, completedAt, durationMs, versionsExpired, deleteMarkersExpired, bytesExpired, keysRefused and keysDeferred. It joins the next pass that starts after the request arrives, never one already running, so the pass applies every rule change acknowledged before the call; requests that arrive together share one pass. It rejects in cluster mode, when the pass fails and when the engine shuts down before the pass completes, and it is refused while 16 requests already wait for a pass to start. The engine answers other management calls while the pass runs. options.timeoutMs (default 300,000) bounds the wait; the engine cannot cancel a pass, so one that outlasts it keeps running, its result is discarded, and a later call joins the next pass that starts after it.

The sweep's clock. A version is due by its stored noncurrent time and the sweep's now, which is min(wall clock, wall clock at the start + monotonic time since). Within a process a backward jump of the wall clock only delays expiry (by the jump, until the clock is corrected), and a forward jump changes nothing: versions written after it carry commit times ahead of now, and their keys wait until now passes them. sweepClockBehindWallMs in getStorageStats() and getHealth() reports a forward jump while it lasts. A single host cannot see a forward jump across a restart, and a restart with the wall clock behind can make a version due up to one UTC midnight early when its commit time lagged across that midnight (see Commit times above).

Stats. Every versioned bucket reports versionsExpiredTotal (noncurrent data versions the sweep removed), deleteMarkersExpiredTotal and bytesExpiredTotal (payload bytes of those data versions); they reconcile with noncurrentVersionCount, deleteMarkerCount and noncurrentBytes. It also reports lastSweepCompletedAt and sweepDurationMs of its last completed pass; lastSweepCompletedAt is the sweep's now when that pass ended (see The sweep's clock above), so after a forward jump of the wall clock it trails the wall clock by sweepClockBehindWallMs, while the startedAt and completedAt of a runVersionSweep() report are wall-clock times. Every bucket with a versioning state or a configuration reports thinning ({ state: 'active' }, or inactive with the reason notConfigured, bucketUnversioned or noNoncurrentExpirationRule). The expired totals live in memory and are written to .smartstorage/versioning/sweep/<bucket>.json every 60 seconds, at the end of a pass and on a graceful stop; after a crash (kill -9) they can miss what was expired in the last 60 seconds. The live counters are recounted at startup and stay exact. getMetrics() exports the totals as objectstorage_versions_expired_total, objectstorage_delete_markers_expired_total and objectstorage_version_bytes_expired_total.

encoding-type=url percent-encodes, on ListObjects, ListObjectsV2, ListObjectVersions and ListMultipartUploads, every UTF-8 byte outside A-Z a-z 0-9 - . _ ~ and / (a space is %20, never +) in the elements S3 encodes for that operation, and the response carries <EncodingType>url</EncodingType>:

  • ListObjects: Prefix, Marker, NextMarker, Delimiter, Key and CommonPrefixes/Prefix.
  • ListObjectsV2: Prefix, Delimiter, StartAfter, Key and CommonPrefixes/Prefix; ContinuationToken and NextContinuationToken are opaque tokens, the same with or without encoding-type.
  • ListObjectVersions: Prefix, KeyMarker, NextKeyMarker, Delimiter, Key and CommonPrefixes/Prefix.
  • ListMultipartUploads: Prefix and Key.

Without the parameter keys are written as they are and no EncodingType is returned; any other value returns InvalidArgument (400). An object key may hold any UTF-8 character, also those XML 1.0 cannot carry (U+0001 and the other control characters except tab, line feed and carriage return): a listing that contains such a key is valid XML only with encoding-type=url. ListObjectsV2 continuation tokens use only A-Z a-z 0-9 - _, so they never break the XML.

Accepted but not applied: owners are never returned (fetch-owner); responses say so by omitting Owner. Refused with NotImplemented although S3 defines them: every other subresource (?acl, ?tagging, ?retention, ?legal-hold, ?cors, ?website, ?object-lock, ?attributes, ?restore, ?select, …), partNumber on GET and HEAD, the paging parameters of ListMultipartUploads (key-marker, upload-id-marker, max-uploads, delimiter), ListObjects V1 with V2 cursors, the ListBuckets parameters (prefix, max-buckets, continuation-token, bucket-region), headers hoisted into a presigned URL's query (x-amz-checksum-crc32=…) on anything but an object read, and any POST that is not one of the operations above.

Object listing and pagination

ListObjects pages identically in standalone and cluster mode, because both backends assemble a page through the same builder. A standalone listing walks only the part of the bucket its prefix and cursor can reach, and blocks only commits to the listed bucket while it walks: reads of every bucket, writes to other buckets and other listings of the same bucket proceed. A listing without a prefix still walks the whole bucket.

ListObjectsV2 (?list-type=2) reads prefix, delimiter, max-keys, continuation-token and start-after. Both cursors are exclusive: listing resumes strictly after the named key. A request may carry both; the continuation token wins, because it is the server-issued resume point of an interrupted page while start-after only states where a first page begins. The response echoes ContinuationToken and StartAfter when the request carried them, reports KeyCount as contents plus common prefixes, and returns NextContinuationToken whenever the page is truncated.

A continuation token is opaque: pass NextContinuationToken back unchanged. It is the unpadded base64url (RFC 4648 §5) encoding of the UTF-8 key the next page resumes after, at most 1,366 characters, and is the same with or without encoding-type=url, so a client may change encoding-type between pages. A token the server could not have issued (outside the base64url alphabet, padded, not canonical, or not decoding to a 1 to 1,024-byte UTF-8 key) returns InvalidArgument (400), and so do the raw-key tokens of engines before 11.0.0.

ListObjects v1 (no list-type) reads prefix, delimiter, max-keys and marker. The response always carries Marker, empty when the request carried none, and carries NextMarker whenever the page is truncated. Feeding NextMarker back as marker walks the listing.

Paging rules shared by both versions:

  • max-keys counts contents and common prefixes. A delimited page of max-keys=1 therefore returns one group or one key, never both.
  • The next cursor is the last emitted item of the page, which may be a common prefix. Resuming from a common prefix skips the whole group: a common prefix that is not lexicographically greater than the cursor is never returned again.
  • max-keys=0 answers an empty, untruncated page with KeyCount 0 and no next cursor.
  • max-keys above 1000 is clamped to 1000 and the response echoes the clamped value in MaxKeys, as S3 does. A max-keys value that is not a non-negative integer is rejected with InvalidRequest, and so is a start-after or marker longer than the 1,024-byte S3 key bound.
  • An empty start-after, continuation-token or marker value means no cursor.
  • Keys are returned in ascending UTF-8 byte order, and a key is never returned twice across pages of an unchanged bucket.

ListMultipartUploads is not paginated: it returns every in-flight upload of the bucket with IsTruncated false. An optional prefix limits the result to uploads whose keys start with it and is echoed in the response; prefix credentials must supply one inside their prefix.

Bucket names and object keys

Bucket names are validated the same way at the HTTP, management, policy and storage boundaries: 3 to 63 lowercase letters, digits, dots or hyphens, beginning and ending with a letter or digit, without consecutive dots, and not formatted as an IPv4 address. Object keys are S3-compatible UTF-8 strings of up to 1,024 bytes, including leading, repeated, embedded and trailing / characters; listings return the exact original key.

API Reference

All methods are on ObjectStorage unless marked static. Option objects are named after their exported I… types in ts/index.ts.

Lifecycle

Member Description
new ObjectStorage(config?: IObjectStorageConfig) Create an instance without starting it; validates the configuration
static createAndStart(config?): Promise<ObjectStorage> Construct and start() in one call
start(): Promise<void> / stop(): Promise<void> Start the engine and listener / drain and stop them (Process lifecycle)
config: TObjectStorageResolvedConfig The resolved configuration the engine receives
port: number The bound TCP port while running; the requested port before and after
dashboardPort: number | undefined The dashboard's bound port while it runs
getStorageDescriptor(overrides?): Promise<IS3Descriptor> Connection details for S3 clients (AWS SDK v3)

Buckets and data

Member Description
createBucket(name): Promise<{ name }> Create a bucket
exportBucket({ bucketName }): Promise<IBucketExport> Bounded smartstorage.bucket.v1 export (Backup and restore)
importBucket({ bucketName, source, accessKeyId?, fence?, holdPublication? }): Promise<IObjectStorageBucketMutationResult> Import an export; with fence an exact fenced replacement bound to the bucket's credential; holdPublication: true (needs a fence) returns resourcePublicationBarrier
deleteBucket({ bucketName, accessKeyId?, fence, holdPublication? }): Promise<IObjectStorageBucketMutationResult> Fenced deletion; with accessKeyId it also revokes that credential and deletes the policy (Durable resource fencing)

Credentials

Member Description
listCredentials(): Promise<IStorageCredentialMetadata[]> The runtime credential set without secrets
replaceCredentials(credentials: IStorageCredential[]): Promise<void> Replace the runtime set atomically (Runtime credentials)
createBucketTenant({ bucketName, accessKeyId?, secretAccessKey?, region?, retention? }): Promise<IBucketTenantDescriptor> Bucket plus scoped credential (Bucket tenants)
ensureBucketTenant({ bucketName, accessKeyId, secretAccessKey, region?, retention?, fence? }): Promise<IBucketTenantDescriptor> Atomic create, resume or rotate with a supplied credential; with fence the descriptor carries resourceFence
rotateBucketTenantCredentials({ bucketName, accessKeyId?, secretAccessKey?, region? }): Promise<IBucketTenantDescriptor> Replace a tenant's credential
deleteBucketTenant({ bucketName, accessKeyId? }): Promise<void> Revoke one credential, or delete the bucket with its credentials
listBucketTenants(): Promise<IBucketTenantMetadata[]> Tenant metadata without secrets
getBucketTenantDescriptor({ bucketName }): Promise<IBucketTenantDescriptor> Endpoint, credential and env values of a tenant
createPrefixCredential(input: IPrefixCredentialInput): Promise<IPrefixCredential> Expiring prefix credential, or install one created on another node (Prefix credentials)
revokePrefixCredential({ accessKeyId }): Promise<{ revoked: boolean }> Revoke and keep the ID refused while it could still be created
listPrefixCredentials({ bucketName? }?): Promise<IPrefixCredentialMetadata[]> Live prefix credentials without secrets
getBucketRetentionCapability(): Promise<IBucketRetentionCapability> Engine-enforced retention support (Immutable retention)
getBucketRetentionReceipt({ bucketName }): Promise<IBucketRetentionReceipt | null> A bucket's retention receipt

Migration and fencing

Member Description
getBucketMigrationCapability() Provider identity, receipt key evidence and supported migration phases (Migration control)
createSourceMigration(options) / createDestinationMigration(options) Create durable source or destination migration ownership
transitionSourceMigration(options) / transitionDestinationMigration(options) Advance one allowed phase edge; returns a transition receipt
cleanupSourceMigration(options) Permanently delete a retired source; exact retries return the same receipt
inspectBucketMigration({ migrationId }) Durable phase, terminal receipts and cleanup state, or null
commitResourcePublication({ barrier, resourceFenceReceipt }) Release a write-publication hold; exact-match and idempotent
getResourceFencingCapability() Fencing support, drain requirement and publication-hold version
getBucketResourceFenceState({ bucketName }) One bucket's validated durable fence state, or null
computeObjectStorageResourceFenceEffectivePayloadSha256V1(payload) Exported function: the engine's canonical effective payload digest

Observability and maintenance

Member Description
getHealth(): Promise<IObjectStorageHealth> Running state, storage, auth, counts, cluster and fencing fields (Health and metrics)
getMetrics(): Promise<IObjectStorageMetrics> Numeric metrics plus Prometheus text
getStorageStats(): Promise<IStorageStats> / listBucketSummaries(): Promise<IBucketSummary[]> Cached logical totals, without S3 list calls (Storage stats)
getStoragePoolDiagnostic(): Promise<IStoragePoolObservation | null> The active pool observation, cached for a poisoned mounted pool; null before start
getClusterHealth(): Promise<IClusterHealth> Live node, drive, quorum, repair and transport health (Cluster health)
getClusterSnapshot(): Promise<IObjectStorageClusterSnapshot> The dashboard's memory-only snapshot (Dashboard)
runHealingScan({ timeoutMs? }?): Promise<IClusterHealingScanResult> Run and await one healing scan (Self-healing)
runVersionSweep({ timeoutMs? }?): Promise<IVersionSweepResult> Run and await one version sweep pass (Version retention)
static probeStoragePool(pool, { binaryPath? }?): Promise<IStoragePoolObservation> Read-only probe of a mounted pool (Storage pools)
static findOrphanObjectLeaves(directory, { binaryPath?, reportLimit? }?): Promise<IOrphanObjectLeafReport> Read-only scan for payloadless object leaves (Orphan object leaves)

Errors

Management calls reject with typed errors where the engine gives a code:

  • ObjectStorageCredentialError (code in objectStorageCredentialErrorCodes): the ECREDENTIAL_* codes listed under Prefix credentials, also from tenant, credential and bucket deletion calls.
  • ObjectStorageResourceFenceError (code in objectStorageResourceFenceErrorCodes): EFENCE_REQUIRED, EFENCE_STALE, EFENCE_SCOPE_MISMATCH, EFENCE_UNSUPPORTED, EFENCE_MUTATION_MISMATCH, EFENCE_STATE_CORRUPT, EFENCE_RECEIPT_LIMIT, EFENCE_BUSY, EFENCE_OWNERSHIP_MISMATCH, EFENCE_PUBLICATION_HELD, EFENCE_PUBLICATION_MISMATCH, EFENCE_PUBLICATION_CAPABILITY.
  • RustBinaryLocatorError (ERR_RUST_BINARY_EXPLICIT_PATH_INVALID) from start() for an unusable binaryPath.

Architecture

┌──────────────────────────────────────────────┐
│  Your code (AWS SDK, @lossless.org/client)   │
│  ↕ HTTP (localhost:3000)                     │
├──────────────────────────────────────────────┤
│  ruststorage binary (Rust)                   │
│  ├─ hyper 1.x HTTP server                    │
│  ├─ S3 path-style routing                    │
│  ├─ StorageBackend (Standalone or Clustered) │
│  │   ├─ FileStore (single-node mode)         │
│  │   └─ DistributedStore (cluster mode)      │
│  │       ├─ ErasureCoder (Reed-Solomon)      │
│  │       ├─ ShardBackend (per-drive shards)  │
│  │       ├─ QuicTransport (quinn)            │
│  │       ├─ ClusterState & Membership        │
│  │       └─ HealingService                   │
│  ├─ SigV4 auth + policy engine               │
│  ├─ CORS middleware                          │
│  └─ S3 XML response builder                  │
├──────────────────────────────────────────────┤
│  TypeScript (thin IPC wrapper)               │
│  ├─ ObjectStorage class                      │
│  ├─ RustBridge (stdin/stdout JSON IPC)       │
│  └─ Config & S3 descriptor                   │
└──────────────────────────────────────────────┘

The engine executable is ruststorage. The TypeScript API spawns it and drives it over newline-delimited JSON on stdin/stdout; the engine's log lines (tracing, default level info) go to its stderr and are forwarded verbatim to the host's stderr. The dashboard runs in the Node.js process and reads the engine only through that channel.

Platform support

Qualification: the amd64 suite runs natively on Ubuntu 24.04 (glibc 2.39). ARM64 storage and restart checks run in a full-system QEMU guest (Alpine 6.18.35 kernel, Ubuntu 24.04 glibc 2.39 libraries), which does not qualify physical ARM hardware or a complete Ubuntu ARM installation. The macOS engines are qualified on macOS 26.3.1 on Apple Silicon, the Intel engine through Rosetta 2; older macOS releases and physical Intel Macs are not qualified. On macOS every storage feature works on local APFS (standalone, multipart, retention, bucket replacement, clustering) with the full 1,024-byte UTF-8 key limit; NFS/SMB mountedFs pools are Linux-only.

To build from source, pnpm build builds the engine for the host (GNU and musl on Linux, Rust 1.95.0 pinned in rust-toolchain.toml) and the TypeScript API.

On-Disk Format

Format identifiers keep their smartstorage.* names (smartstorage.bucket.v1, smartstorage.bucket-versioning.v1, smartstorage.object-version.v1, smartstorage.bucket.ensure.v1, and the hash domains of the same prefix), filesystem names their .smartstorage* names, and retention metadata keys their smartstorage-retention-* names. They are part of the stored formats and never change in existing data.

Standalone mode:

{storage.directory}/
  {bucket}/
    .smartstorage-objects-v2/
      {whole-key-hex-frames}/
        object._storage_object_v2                # Object data; completed multipart objects include
                                                 # an atomic payload/ETag/metadata trailer
        object._storage_object_v2.metadata.json  # Legacy metadata sidecar
        object._storage_object_v2.md5            # Legacy cached MD5 hash
  .multipart/
    {upload-id}/
      metadata.json                      # Upload metadata
      part-1, part-2, ...               # Part data files
  .policies/
    {bucket}.policy.json                 # Bucket policy (IAM JSON format)
  .smartstorage/staging/v1/              # Private staging: a write lands here first; its
                                         # key leaf appears only with the complete payload
  .smartstorage/credentials.json         # Runtime and bucket tenant credentials (0600)
  .smartstorage/prefix-credentials.json  # Prefix credentials and revocations (0600)
  .smartstorage/versioning/v1/{bucket}.json
                                         # Versioning state of a versioned bucket
  .smartstorage/versioning/stamp-clock.json
                                         # Persisted floor of the version stamp clock
  .smartstorage/versions/{bucket}/{whole-key-hex-frames}/
    {seq:016x}-{generation}              # Record of a noncurrent version or delete marker
    {seq:016x}-{generation}.object       # Payload of a noncurrent data version

Every object key is encoded below the reserved .smartstorage-objects-v2 bucket directory as canonical, prefix-free lowercase-hex frames. Frames and the fixed payload and sidecar names stay within the common 255-byte NAME_MAX and are safe on CIFS. Objects of the older v7 identity layout stay readable on local and NFS storage when their raw path can be reconstructed without traversal; their first mutation stages and publishes the canonical v2 object set before removing the legacy payload and sidecars, under descriptor-confined, digest-bound durable state that startup or the next access reconciles after any crash. Canonical and legacy payloads of one key without that exact owned state, noncanonical entries below the reserved v2 root, and unsafe legacy aliases on case-folding CIFS fail closed instead of choosing an ambiguous object. Descriptor-confined standalone startup reclaims abandoned private object materializations of interrupted create-only, copy or multipart publications without removing an already published hard link.

Bucket versioning keeps the layout above for current objects and adds a format that later standalone engines read unchanged:

  • .smartstorage/versioning/v1/{bucket}.json is {"format":"smartstorage.bucket-versioning.v1","bucket":…,"status":"Enabled"|"Suspended"}. A bucket without it was never versioned.
  • The current object of a key stays at its key path. A payload written while its bucket is versioned ends in a trailer record of schema 2 whose version holds its generation UUID, whether it is the null version, its sequence number and its commit time in epoch milliseconds; payloads written before keep schema 1 and are the key's null version with sequence 0.
  • Every other version lives in the key's chain directory below .smartstorage/versions/{bucket}/, whose frame directories are the same hexadecimal frames as below .smartstorage-objects-v2. A record {seq:016x}-{generation} (JSON, "format":"smartstorage.object-version.v1", with the key, sequence, generation, null flag, delete marker flag, commit time, the time it became noncurrent, payload length, ETag and metadata) exists for every noncurrent version and every delete marker; a data version also has {seq:016x}-{generation}.object, the payload inode it had while it was current. Sequence order is chain order; the version id is the generation UUID, or null.
  • A payload is linked into the chain before its record exists and unlinked after its record is gone. Startup recovery brings every chain to a consistent state after a crash (it removes payloads without a record, finishes or undoes an interrupted change of the current version, clears the noncurrent time of a delete marker that is current again and keeps one null version per key) and refuses to start on a record that is corrupt or misses its payload, on a chain directory with a foreign entry, and on versions of a bucket without versioning state.

Cluster mode:

{storage.directory}/
  .buckets/
    v2/{bucket-address}.bucket.json       # Active/tombstoned bucket state
    .policies/{bucket}.policy.json        # Bucket policies
  .manifests/
    v2/{bucket-prefix}/{bucket-address}/{object-prefix}/
      {object-address}.state.json
                                         # Active/tombstoned object manifest
    .multipart/{upload-id}/               # Descriptor-anchored multipart state
  .smartstorage/credentials.json          # This node's runtime credentials (not replicated)
  .smartstorage/prefix-credentials.json   # This node's prefix credentials (not replicated)
  .smartstorage/cluster/
    identity.json                         # Node and cluster identity
    topology.json                         # Last durable topology
    generation-inventory/{address}.json   # Deferred physical-generation cleanup

{drive_path}/.smartstorage/data/
  v2/{bucket-prefix}/{bucket-address}/{key-prefix}/{generation-address}/
    {shard-address}.shard                 # Immutable erasure-coded shard record
    {shard-address}.tombstone             # Crash-safe deletion authority

Cluster control records, manifests and shards use descriptor-anchored, digest-addressed formats with durable tombstones. Records of older layouts are migrated in bounded steps, and the new records become authoritative; before a manifest listing can issue a continuation token, all remaining legacy manifests are migrated, so a later migration never puts a key behind an issued token. Older engines spooled GET responses into .manifests/.get-tmp/; startup removes a leftover spool directory that holds only spool files and otherwise leaves it in place, unused. Cleanup removes only descriptor-verified expired multipart uploads and cleanup-marked committed entries (retried in bounded passes every multipart.cleanupIntervalMinutes, large registries page by page); unreadable or incomplete descriptors fail closed and need operator repair or removal once their bucket and shard ownership is established. Shutdown never aborts or detaches an active multipart worker; it waits for it within the same retryable bounded drain.

Upgrades and rollbacks

Storage layout upgrades are one-way, and a cluster never runs mixed versions.

  • Standalone v2 object layout. After the first canonical v2 object write or legacy-object migration in a root, engines 7.0.0 and earlier cannot read it, so the root cannot go back to them.
  • Standalone versioning. Engines before 11.2.0 cannot read a payload with a schema-2 trailer: a standalone root on which any bucket was ever versioned (PutBucketVersioning with Enabled or Suspended) cannot be rolled back to an engine before 11.2.0.
  • Staged leaves (8.0.3 to 10.6.0). A crash between staging and publication of a PUT, CopyObject or CompleteMultipartUpload can leave a staged leaf directory .smartstorage/staging/v1/.smartstorage-<uuid>.object.tmp.leaf/. Engines 10.7.0 and later remove it at startup; 8.0.3 to 10.6.0 refuse to start with "Standalone staging directory contains an unowned entry". Before rolling such a root back, start and stop the newer engine on it once. If it cannot start, back up the root and remove only those *.object.tmp.leaf directories: they hold payloads that were never published.
  • Prefix credentials. Engines before prefix credentials never read .smartstorage/prefix-credentials.json, so a rollback disables every prefix credential rather than widening it.
  • Cluster format. Upgrade every member in one coordinated maintenance window and back up the cluster roots first. Once any member has written or migrated the new state, do not restart an older engine and do not run a mixed-version cluster; a rollback restores the roots from the pre-upgrade backup.
  • Pending shards. Engines that predate pending shards refuse manifests that carry them, because their chunks are short of placements. Before rolling a cluster back to such an engine, stop writing, complete or abort every open multipart upload (part descriptors record pending shards too, the scan does not count them, and an older engine drops them, so such an upload could not complete after the rollback), then run runHealingScan() on every node until it reports shardsPending 0. Its scan starts after the request arrives, so it covers every write that finished before the call; a scan that could not list every manifest fails instead of reporting counts.
  • @lossless.org/client: one TypeScript client for lossless.org NoSQLDB, SQLDB and ObjectStorage; @lossless.org/client/objectstorage is its S3 client library
  • @push.rocks/smartrust: the TypeScript-to-Rust IPC bridge the API runs the engine with
  • @git.zone/rsrepo: Rust builds and cross-compilation for npm packages

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.

S
Description
No description provided
Readme MIT
16 MiB
0 Stars 3 Watchers 0 Forks
Languages
Rust 67.1%
HTML 16.7%
TypeScript 16%
JavaScript 0.2%