@lossless.org/objectstorage
An S3-compatible object storage server that you install with pnpm add and run from TypeScript. A Rust engine serves the S3 API with streaming I/O and bounded memory; a typed TypeScript API starts it, provisions credentials and reads its health. Run it standalone on one directory for development, tests and single-node services, or as an erasure-coded cluster across nodes and drives. Any S3 client works against it: the AWS SDK, @lossless.org/client, or your own.
Issue Reporting and Security
For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.
Why objectstorage
- No Docker, no toolchain. Prebuilt engines for Linux (amd64, arm64) and macOS (Intel, Apple Silicon) ship inside the npm package.
- S3 semantics, not a mock. AWS SigV4 verification with constant-time comparison (headers and streaming chunk signatures), bucket policies with Allow/Deny, multipart uploads, byte ranges, conditional requests, bucket versioning and ListObjectsV2 pagination. A request it does not implement is refused (
501 NotImplemented, or400 InvalidRequestwhere cluster mode cannot enforce a condition yet), never answered with a silent wrong result. - Bounded memory. Request and response bodies stream with backpressure, ranges seek in the file, and upload decoding runs under a fixed shared budget.
- Crash-safe writes. Standalone writes are staged, synced and published atomically: a reader, and a crash, sees the previous object or the new one, never a mix.
- Built for tests.
port: 0takes a free port,cleanSlatewipes the root, andstop()resolves only once the listener, the work and the engine process are gone. - Credentials for platforms. Rotate credentials without a restart, give each app one bucket and one scoped key (bucket tenants), or hand out expiring keys limited to one prefix.
- Operable.
/-/live,/-/ready,/-/healthand Prometheus/-/metricsendpoints, cached storage stats, a read-only dashboard, bucket export and import, fenced bucket replacement and deletion, and migration control. - Clustering. Reed-Solomon erasure coding (default 4 data + 2 parity shards), quorum writes and reads, QUIC transport with optional mutual TLS, per-drive health and background self-healing.
Installation
pnpm add @lossless.org/objectstorage # add -D when you only use it in tests
| Platform | Engine | Requirements |
|---|---|---|
| Linux amd64, arm64 | ruststorage_linux_amd64_musl, ruststorage_linux_arm64_musl (static; the TypeScript API runs these) |
none beyond the kernel |
Linux amd64, arm64 (custom packaging through binaryPath) |
ruststorage_linux_amd64, ruststorage_linux_arm64 (dynamically linked GNU builds) |
glibc 2.34 or later with its dynamic loader, libm.so.6, libgcc_s.so.1 |
| macOS Intel, Apple Silicon | ruststorage_macos_amd64, ruststorage_macos_arm64 |
local APFS storage; built for macOS 11 and later, ad-hoc signed, system libraries only |
Native runtime notices (Cargo licenses, the glibc LGPL terms, the native distribution inventory) are in third-party notices; keep them when you redistribute the engine binaries. What each platform is qualified on, and how to build from source, is under Platform support.
Quick Start
Standalone
import { ObjectStorage } from '@lossless.org/objectstorage';
import { S3Client, CreateBucketCommand, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
const storage = await ObjectStorage.createAndStart({
server: { port: 0, silent: true }, // 0: any free port
storage: { cleanSlate: true }, // wipes ./.nogit/bucketsDir on start
});
const { endpoint, port, accessKey, accessSecret } = await storage.getStorageDescriptor();
const s3 = new S3Client({
endpoint: `http://${endpoint}:${port}`,
region: 'us-east-1',
credentials: { accessKeyId: accessKey, secretAccessKey: accessSecret },
forcePathStyle: true, // objectstorage routes path-style requests
});
await s3.send(new CreateBucketCommand({ Bucket: 'my-bucket' }));
await s3.send(new PutObjectCommand({ Bucket: 'my-bucket', Key: 'hello.txt', Body: 'Hello!' }));
const object = await s3.send(new GetObjectCommand({ Bucket: 'my-bucket', Key: 'hello.txt' }));
console.log(await object.Body!.transformToString()); // Hello!
s3.destroy();
await storage.stop();
With port: 0 the operating system picks the port. After start(), storage.port, getStorageDescriptor(), bucket tenant descriptors (and their S3_PORT), the HTTP health endpoints and cluster peer advertisements report the bound port; config.server.port keeps the requested value, and before startup and after shutdown storage.port returns it. A restart with port 0 takes a new port.
In tests
import { ObjectStorage } from '@lossless.org/objectstorage';
import { tap, expect } from '@git.zone/tsrepo/tapbundle';
let storage: ObjectStorage;
tap.test('setup', async () => {
storage = await ObjectStorage.createAndStart({
server: { port: 0, silent: true },
storage: { cleanSlate: true },
});
});
tap.test('stores objects', async () => {
await storage.createBucket('test');
// ... your test, through the AWS SDK or @lossless.org/client
});
tap.test('teardown', async () => {
await storage.stop();
});
export default tap.start();
stop() resolves only once the engine released its listener, its work and its process, so a test can remove a disposable storage root right after it.
Cluster
import { ObjectStorage } from '@lossless.org/objectstorage';
const storage = await ObjectStorage.createAndStart({
server: { port: 3000 },
storage: {
directory: '/var/lib/objectstorage/control',
cleanSlate: false,
},
cluster: {
enabled: true,
nodeId: 'node-1',
quicPort: 4000,
seedNodes: ['192.168.1.11:4000', '192.168.1.12:4000'],
erasure: {
dataShards: 4, // k: shards needed to reconstruct
parityShards: 2, // m: shards that may be lost
},
drives: {
paths: ['/mnt/disk1', '/mnt/disk2', '/mnt/disk3'],
},
},
});
Objects are split into chunks (default 4 MiB); each chunk is erasure-coded into 6 shards (4 data + 2 parity) spread across drives and nodes, and any 4 of them rebuild it.
Before the first start, create the control root and every drive root, owned by the process user, with no symlink in their paths and without group or world write permission; cluster mode never creates them (Cluster roots). Each drive root counts as one independent drive, so startup refuses two drive roots on the same device (st_dev). Only the device number is checked: partitions of one disk, LVM volumes, btrfs subvolumes and datasets of one ZFS pool report distinct devices and are not detected, so keeping them apart is the operator's job. The control root may share a device with a drive. For dev and test clusters whose drive roots are directories on one filesystem, opt in explicitly; startup then logs which roots share a device:
drives: {
paths: ['/srv/objectstorage-dev/drive-1', '/srv/objectstorage-dev/drive-2', '/srv/objectstorage-dev/drive-3'],
allowSharedDevice: true, // dev/test only: these roots are not independent drives
},
Without cluster.tls the cluster transport does not authenticate nodes: run cluster mode only on a network where every host that can reach quicPort is a cluster node, or set cluster.tls on every node (see Cluster transport trust).
Configuration
Every field is optional; ObjectStorage.createAndStart(config) and new ObjectStorage(config) take an IObjectStorageConfig.
| Key | Default | Notes |
|---|---|---|
server.port |
3000 |
0 lets the OS allocate a port (see Quick Start) |
server.address |
'0.0.0.0' |
descriptors report localhost for 0.0.0.0 |
server.silent |
false |
suppresses the startup lines |
server.region |
'us-east-1' |
SigV4 signing region, returned by GetBucketLocation |
storage.directory |
<cwd>/.nogit/bucketsDir |
standalone storage root; control root in cluster mode |
storage.cleanSlate |
false |
wipes the root's contents on start (the root, its maintenance lock and pool identity stay); refused for mountedFs pools, with durable migration state and on a root with retained buckets |
storage.pool |
a default localFs pool at storage.directory |
see Storage pools |
auth.enabled |
false |
SigV4 authentication and bucket policies |
auth.credentials |
[{ accessKeyId: 'STORAGE', secretAccessKey: 'STORAGE' }] |
initial root credentials; { accessKeyId, secretAccessKey, bucketName?, region? } |
cors.enabled |
false |
|
cors.allowedOrigins / allowedMethods / allowedHeaders |
['*'] / ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'] / ['*'] |
|
cors.exposedHeaders |
['ETag', 'x-amz-request-id', 'x-amz-version-id'] |
|
cors.maxAge / cors.allowCredentials |
86400 / false |
|
limits.requestTimeout |
300000 |
milliseconds the engine waits for the next frame of a streaming (aws-chunked) upload body and of a bucket policy or versioning document |
limits.maxObjectSize / limits.maxMetadataSize |
5368709120 / 2048 |
accepted, not applied by the current engine |
logging.level / format / enabled |
'info' / 'text' / true |
accepted, not applied: the engine logs at info to its stderr, which is forwarded verbatim to the host's stderr |
multipart.expirationDays |
7 |
uploads older than this are removed; must be greater than zero |
multipart.cleanupIntervalMinutes |
60 |
cleanup pass interval; must be greater than zero |
multipart.maxUploadsPerPrefixScope |
100 |
open uploads per prefix credential scope; must be non-zero |
cluster.enabled |
true selects cluster mode; omit cluster for standalone |
|
cluster.nodeId |
generated and persisted | required with tls, then a lowercase DNS name |
cluster.quicPort |
4000 |
UDP; 0 lets the OS assign it (read it from getClusterHealth().quicAddress) |
cluster.seedNodes |
[] |
host:port QUIC addresses of existing members |
cluster.erasure.dataShards / parityShards |
4 / 2 |
|
cluster.erasure.chunkSizeBytes |
4194304 (4 MiB) |
1 byte to 64 MiB; startup refuses anything else |
cluster.drives.paths |
drive roots of this node | |
cluster.drives.allowSharedDevice |
false |
dev/test only |
cluster.heartbeatIntervalMs / heartbeatTimeoutMs |
5000 / 30000 |
|
cluster.getDeadlineMs |
21600000 (6 h) |
60,000 (1 min) to 604,800,000 (7 days); the same on every node (Streaming Reads) |
cluster.tls |
unset | { caFile, certFile, keyFile }; on every node or on none (Cluster transport trust) |
dashboard.enabled |
false |
read-only dashboard (Dashboard) |
dashboard.address / dashboard.port |
'127.0.0.1' / 3001 |
port 0 allocates one; never the S3 port |
binaryPath |
package discovery | one exact engine executable (Engine selection) |
onUnexpectedEngineExit |
unset | ({ code, signal }) => void, called when the engine ends without a stop request (Process lifecycle); not sent to the engine |
A few common shapes:
// Auth on, one root credential
await ObjectStorage.createAndStart({
auth: { enabled: true, credentials: [{ accessKeyId: 'test', secretAccessKey: 'test123' }] },
});
// CORS for local web development
await ObjectStorage.createAndStart({
cors: { enabled: true, allowedOrigins: ['http://localhost:5173'], allowCredentials: true },
});
Storage pools and mounted filesystems
One ObjectStorage process serves one storage pool. storage.directory alone is a default localFs pool. With both storage.directory and storage.pool.directory, a localFs pool needs both to resolve to the same path and a mountedFs pool needs both to be the same exact absolute mountpoint string. storage.pool.id starts with a lowercase letter and has at most 63 lowercase letters, digits or hyphens. A root is durably bound to its pool ID and cannot be reassigned to another pool.
A missing standalone localFs root is created with mode 0700. An existing root must be owned by the process user and must not be group- or world-writable, otherwise startup fails before the S3 listener opens; ObjectStorage never changes an existing root's ownership or permissions.
On Linux, standalone ObjectStorage can serve an exact NFS or SMB mount (mountedFs; not in cluster mode):
- The configured directory is the absolute mountpoint itself, not a symlink or canonical alias, and
cleanSlateis forbidden. expectedSourceholds a numeric IPv4 or IPv6 address:IP:/absolute/exportfor NFS,//IP/share[/path]for SMB, IPv6 in brackets ([2001:db8::10]:/archive,//[2001:db8::10]/archive).- The client mount must carry the Linux
nosymfollowoption; whoever provisions the mount adds it before ObjectStorage starts. - NFS must keep server-coordinated locking:
nolockandlocal_lock=all|flockare refused; nolocal_lock,local_lock=noneandlocal_lock=posixare accepted, because ObjectStorage usesflock(2). CIFS mounts must not usenobrl.
const storage = await ObjectStorage.createAndStart({
storage: {
directory: '/mnt/archive',
pool: {
id: 'archive',
directory: '/mnt/archive',
backend: {
kind: 'mountedFs',
expectedFilesystemType: 'nfs',
expectedSource: '192.0.2.10:/archive',
// IPv6 is canonicalized as: [2001:db8::10]:/archive
},
},
},
});
Startup opens the mountpoint once and confines every storage, credential, policy, retention, fencing and background operation beneath that descriptor. It verifies the mount ID, device, NFS/CIFS filesystem magic, the ST_NOSYMFOLLOW flag and the effective locking options, rereads the complete mount activation before serving, and runs the root-ownership, same-filesystem, lock-contention, directory-fsync and provider-identity checks through the descriptor. Every storage boundary revalidates the activation. Any drift permanently poisons the process: storage-backed HTTP and management calls fail closed, descendant symlinks are never followed, and the process never falls back to a host directory at the configured path. Lifecycle, liveness, cluster status and cached diagnostics stay available for shutdown and diagnosis; getStoragePoolDiagnostic() returns the cached activation identity and poison evidence without touching the filesystem.
A serving pool observation is productionEligible once these checks pass and while the activation is unpoisoned. Atomic directory exchange is reported as a separate, optional capability: NFS without it still serves S3, atomic create-only uploads, fenced bucket ensure and delete, and migration cleanup, while atomic whole-bucket replacement and isolated restore fail closed. The remoteLocking check and filesystem.remoteLockingMode tell server-coordinated locking apart from unsafe local or disabled locking.
Probe an existing mount without starting the S3 listener:
const observation = await ObjectStorage.probeStoragePool({
id: 'archive',
directory: '/mnt/archive',
backend: {
kind: 'mountedFs',
expectedFilesystemType: 'nfs',
expectedSource: '192.0.2.10:/archive',
},
});
console.log(observation.supported);
console.log(observation.productionEligible); // false: this read-only probe skips write semantics
console.log(observation.checks);
The probe verifies the exact active mount, numeric source, mount ID, device, filesystem magic, ST_NOSYMFOLLOW, the locking options and activation stability, and reports block and transfer sizes. It never writes to the target or changes mount tuning: write checks stay notRun, so supported stays null and productionEligible false. Only a serving startup establishes production eligibility.
Engine selection (binaryPath)
binaryPath selects one exact ruststorage executable, for example a verified engine an installer placed beside a compiled application. When it is set, OBJECTSTORAGE_RUST_BINARY, the package's dist_rust binaries, development builds and PATH are never searched, even if the path is unusable: a missing, non-file or non-executable path rejects start() (and createAndStart()) with RustBinaryLocatorError, code ERR_RUST_BINARY_EXPLICIT_PATH_INVALID, and an empty string throws at construction. Validation never changes the file's permissions. The option is not sent to the engine and does not appear in storage.config. Without it, normal discovery runs, with OBJECTSTORAGE_RUST_BINARY as a hint for qualification runs. probeStoragePool() and findOrphanObjectLeaves() take the same option.
const storage = await ObjectStorage.createAndStart({
binaryPath: '/opt/app/engines/ruststorage',
storage: { directory: '/var/lib/app/objects' },
});
Using objectstorage
AWS SDK v3
Point any S3 client at getStorageDescriptor() (or a bucket tenant descriptor) with path-style addressing, as in the Quick Start. getStorageDescriptor() returns { endpoint, port, accessKey, accessSecret, useSsl: false, bucketName: '' }, using the first configured credential (STORAGE/STORAGE by default); pass an object to override fields. The engine serves plain HTTP.
AWS SDK streaming PutObject and UploadPart requests are decoded before storage. For STREAMING-UNSIGNED-PAYLOAD-TRAILER the engine verifies the declared decoded length and the CRC32 trailer, for STREAMING-AWS4-HMAC-SHA256-PAYLOAD the complete chained signature sequence. Every aws-chunked request needs a verified SigV4 envelope, also with auth.enabled: false; ordinary requests keep the auth-disabled behavior. Unsupported, malformed, truncated, checksum-invalid or signature-invalid bodies fail closed, and the aws-chunked transport encoding is not stored as object metadata. Decoder memory is bounded globally (per-chunk buffers of at most 16 MiB under a shared 64 MiB budget), and limits.requestTimeout bounds the wait for each body frame. A standalone overwrite is staged on the same filesystem and published atomically only after verification succeeds, so a failed, timed-out or cancelled body keeps the previous object.
Client library (@lossless.org/client)
@lossless.org/client is the lossless.org client for NoSQLDB, SQLDB and ObjectStorage. Its @lossless.org/client/objectstorage entry point provides SmartBucket with buckets, directories, files, metadata, streaming and exact (create-only) uploads, and needs @aws-sdk/client-s3 as a peer:
pnpm add @lossless.org/client @aws-sdk/client-s3
import { SmartBucket } from '@lossless.org/client/objectstorage';
const smartBucket = new SmartBucket(await storage.getStorageDescriptor());
const bucket = await smartBucket.createBucket('my-bucket');
await bucket.fastPut({ path: 'docs/readme.txt', contents: 'Hello!' });
const content = await bucket.fastGet({ path: 'docs/readme.txt' }); // Buffer
const exists = await bucket.fastExists({ path: 'docs/readme.txt' }); // true
await smartBucket.close();
The client's exact uploads and exact-path purge use the standalone conditional writes and deletes described under Conditional requests.
Credentials
With auth.enabled: true three kinds of credential authenticate S3 requests: root credentials (the runtime set), bucket tenant credentials (one bucket each) and prefix credentials (one prefix of one bucket, expiring).
Runtime credentials
const credentials = await storage.listCredentials(); // metadata, no secrets
await storage.replaceCredentials([
{ accessKeyId: 'ADMINA', secretAccessKey: 'super-secret-a' },
{ accessKeyId: 'ADMINB', secretAccessKey: 'super-secret-b' },
]);
replaceCredentials()swaps the whole set atomically and persists it under the storage root. New requests use the new set at once and the old credentials stop authenticating at once; requests already authenticated keep running, because auth is evaluated when a request starts.- No restart is needed, and the set survives restarts;
storage.cleanSlateclears it with the rest of the root's contents. - The input holds at least one credential (
{ accessKeyId, secretAccessKey, bucketName?, region? }), everyaccessKeyIdandsecretAccessKeyis non-empty, and access key IDs are unique. - A new
accessKeyIdmust not be held by a live or revoked prefix credential (ECREDENTIAL_CONFLICT); bucket tenant create, ensure and rotate apply the same check.replaceCredentials()is refused while any resource publication is held (Durable resource fencing). - Prefix credentials are not part of
listCredentials()(Prefix credentials). replaceCredentials()is a trusted, process-local administrative override, not a fenced lifecycle API: if it changes credential material that a fenced bucket owns exactly, replaying that bucket's receipt detects the drift and fails closed.
Bucket tenants
A bucket tenant is one bucket with its own scoped S3 credential, for platforms that give every app one bucket. The scope is enforced before bucket policies and default auth, so a validly signed tenant request still cannot list all buckets or reach another bucket. Tenant APIs require auth.enabled: true.
const tenant = await storage.createBucketTenant({ bucketName: 'workapp-123' });
const client = new S3Client({
endpoint: `http://${tenant.endpoint}:${tenant.port}`,
region: tenant.region,
credentials: { accessKeyId: tenant.accessKeyId, secretAccessKey: tenant.secretAccessKey },
forcePathStyle: true,
});
console.log(tenant.env.S3_BUCKET, tenant.env.AWS_ACCESS_KEY_ID); // ready for env injection
await storage.rotateBucketTenantCredentials({ bucketName: 'workapp-123' });
const descriptor = await storage.getBucketTenantDescriptor({ bucketName: 'workapp-123' });
const tenants = await storage.listBucketTenants(); // metadata, no secrets
await storage.deleteBucketTenant({ bucketName: 'workapp-123', accessKeyId: descriptor.accessKeyId });
createBucketTenant()creates the bucket if needed and stores a scoped credential, generated unless you passaccessKeyIdandsecretAccessKey.ensureBucketTenant()does the same atomically with a supplied credential: it creates an absent bucket with that credential, resumes an orphaned exact credential, or rotates the credential of an existing bucket it owns exactly, and refuses foreign or ambiguous ownership before changing anything.- The descriptor holds the endpoint, port, region, bucket, key pair, SSL flag and an
envmap (S3_ENDPOINT,S3_PORT,S3_REGION,S3_BUCKET,S3_ACCESS_KEY_ID,S3_SECRET_ACCESS_KEY,S3_USE_SSL,AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_REGION). deleteBucketTenant({ bucketName, accessKeyId })revokes one credential and keeps the bucket;deleteBucketTenant({ bucketName })revokes the bucket's credentials and deletes the bucket with its contents, every object version included.- Tenant credentials list, read, write and delete objects in their bucket. They cannot list all buckets, reach or copy from other buckets, delete buckets, change bucket policies or change versioning.
Immutable retention
createBucketTenant() and ensureBucketTenant() accept retention: { mode: 'compliance', policyId, retentionDurationSeconds } for engine-enforced retention in standalone mode. Objects of a retained bucket cannot be overwritten with a different payload or metadata (an exact replay of the same PUT succeeds with the same ETag) and cannot be deleted by any credential before their retainUntil (creation time plus the duration). CopyObject and multipart uploads into a retained bucket are refused. HeadObject reports ObjectLockMode: COMPLIANCE and the smartstorage-retention-* metadata. The policy ID cannot change and the duration cannot shorten; the retention receipt survives restarts, and cleanSlate refuses to start on a retained root. getBucketRetentionCapability() reports support and getBucketRetentionReceipt({ bucketName }) the bucket's receipt (also on the tenant descriptor and in listBucketTenants()). Retention and versioning exclude each other, and a retained bucket cannot become a migration source.
Prefix credentials
A prefix credential is an ordinary SigV4 key that reaches only the keys below one prefix of one bucket, and only until it expires: a short-lived key for a guest (a VM, a job, a mounted drive) that must see one folder and nothing else.
const credential = await storage.createPrefixCredential({
bucketName: 'drives',
prefix: 'drives/user-42/',
ttlSeconds: 3600, // 1 to 43200 (12 h)
});
// { accessKeyId, secretAccessKey, bucketName, prefix, region, createdAt, expiresAt }
const client = new S3Client({
endpoint: 'http://localhost:3000',
region: credential.region,
credentials: {
accessKeyId: credential.accessKeyId,
secretAccessKey: credential.secretAccessKey,
},
forcePathStyle: true,
});
await storage.revokePrefixCredential({ accessKeyId: credential.accessKeyId }); // { revoked: true }
const live = await storage.listPrefixCredentials({ bucketName: 'drives' }); // metadata, no secrets
interface IPrefixCredentialInput {
bucketName: string;
prefix: string;
ttlSeconds?: number; // first node: lifetime in seconds
expiresAt?: number; // further nodes: the first node's expiresAt (epoch ms)
accessKeyId?: string; // supplied together with secretAccessKey and expiresAt only
secretAccessKey?: string;
region?: string; // default: server.region
}
Input rules.
- Exactly one of
ttlSeconds(an integer, 1 to 43 200) andexpiresAt(an integer epoch-ms time after now and at most 43 200 s ahead). - Key material is generated when omitted (
SPplus 20 uppercase hex characters, a 64-hex-character secret). Supplied material needsexpiresAtand bothaccessKeyIdandsecretAccessKey: an ID of 16 to 128 characters ofA-Z0-9, and a secret of 32 to 128 printable ASCII characters without spaces. - The prefix is validated, never normalised: 1 to 1023 bytes of
A-Z a-z 0-9 - . _ ~ /(RFC 3986 unreserved characters plus/, no percent-encoding), ending with/, not starting with/, with no empty,.or..segment. - The bucket must exist. The access key ID must not belong to a static or bucket tenant credential and must not have been revoked.
- The secret appears only in the create result. It is never listed, logged or stored in
storage.config. - Errors are
ObjectStorageCredentialErrorwith acode:
| Code | Meaning |
|---|---|
ECREDENTIAL_AUTH_DISABLED |
auth.enabled is false |
ECREDENTIAL_INVALID_PREFIX |
the prefix fails the grammar |
ECREDENTIAL_INVALID_TTL |
not exactly one of ttlSeconds and expiresAt, or ttlSeconds out of range |
ECREDENTIAL_INVALID_EXPIRY |
expiresAt is not in the future, or is more than 12 h ahead |
ECREDENTIAL_INVALID_KEY |
the key material or the region is malformed, or only part of it was supplied |
ECREDENTIAL_NO_SUCH_BUCKET |
the bucket does not exist |
ECREDENTIAL_BUCKET_UNAVAILABLE |
the bucket is a sealed or retired migration source |
ECREDENTIAL_CONFLICT |
the ID is held by another credential, or a replay differs from the stored record |
ECREDENTIAL_REVOKED |
the ID was revoked |
ECREDENTIAL_NOT_PREFIX |
a revoke named a static or bucket tenant credential |
ECREDENTIAL_LIMIT |
10 000 live credentials and revocations are stored |
ECREDENTIAL_PERSISTENCE |
the store could not be written |
ECREDENTIAL_STORE_UNAVAILABLE |
the store file is unreadable (see below) |
Access. A key is covered when its bytes start with the prefix; a key equal to the prefix (its folder marker) is covered.
- Objects. GetObject, HeadObject, PutObject, DeleteObject and the multipart operations work on covered keys.
- Writes also refuse keys with
.,..or empty segments after the prefix, and keys with control characters. - CopyObject and UploadPartCopy need the source to be covered too.
- Object requests accept only the query parameters the operation uses:
uploads,uploadId,partNumber,max-partsandpart-number-marker, the sixresponse-*header overrides on GetObject and HeadObject, plus thex-idoperation label that AWS SDKs append. Any other object query (?acl,?tagging,?versionId,?retention, …) is refused withAccessDenied(403), also a name S3 does not define (?v=1), and so is any repeated query parameter. A prefix credential never names a version:?versionIdis refused even asnull, and so is a copy source with a?versionId=suffix. - Requests that pass these checks still go through the router's allowlist (Supported S3 Operations), so a parameter the operation does not implement gets
NotImplemented(501).
- Writes also refuse keys with
- DeleteObjects. Every member is checked on its own: covered keys are deleted, and the others get a per-key
AccessDenied, as does every member that names aVersionId. - Listings. ListObjects V1, ListObjectsV2, ListObjectVersions and ListMultipartUploads need a
prefixparameter that is covered. A missing or wider prefix is refused, never clamped. - Bucket level. ListBuckets returns only the credential's bucket; HeadBucket and
GET /{bucket}?locationwork on that bucket. Any other bucket-level query (?acl,?versioning,?lifecycle, …) and any repeated query parameter is refused withAccessDenied, so a prefix credential can neither read nor change a bucket's versioning. Creating or deleting buckets and bucket policy operations are refused. - Policies. The scope check runs before bucket policies, so a policy Allow (including
Principal: "*") never widens a prefix credential. A policy Deny still applies.
Expiry and deadlines.
- A credential authenticates while the engine clock is before
expiresAt. After that, or after revocation, requests getInvalidAccessKeyId(403). - Admitted requests end at
expiresAtas well: a trickled upload fails with nothing stored, and a slowly read download is cut off. - Revocation affects only requests admitted after it; requests already running continue until they finish or until
expiresAt. - There is no extend operation. To refresh, create a new key and revoke the old one.
- Multipart cap. All prefix credentials of one scope (the same bucket and prefix) together hold at most
multipart.maxUploadsPerPrefixScope(default 100) open multipart uploads. The next CreateMultipartUpload of that scope getsAccessDenied(403) until one of them ends: completed or aborted by any credential, or removed by the multipart expiry cleanup. Revoking, expiring or replacing a credential frees none of its scope's slots, so issuing new keys does not raise the cap. The count is held in memory and resets when the engine restarts. The engine-wide limit of 100 000 uploads still applies.
Bucket deletion. Deletion attempts revoke first; callers re-issue.
- Every path that deletes a bucket revokes the bucket's prefix credentials before it removes anything: S3 DeleteBucket,
deleteBucket(),deleteBucketTenant()without anaccessKeyId, and migration source cleanup. A recreated bucket of the same name is not reachable with them. - S3 DeleteBucket checks that the bucket is empty before it revokes, so a
BucketNotEmpty(409) keeps the credentials. - If the revocation cannot be stored, the deletion is refused.
- A deletion that fails or is refused after the revocation (for example a stale resource fence,
EFENCE_STALE) returns that error, never a success, and leaves the credentials revoked. Callers that keep the bucket create new prefix credentials.
Persistence.
-
Credentials and revocations live in
.smartstorage/prefix-credentials.json, under the same root ascredentials.json. The engine writes the file atomically with mode0600, andstorage.cleanSlateclears it. -
A sweep every 60 seconds drops expired entries.
-
Engines older than prefix credentials never read the file, so a rollback disables every prefix credential rather than widening it.
-
Fail closed. If the file is unreadable, larger than 32 MiB, malformed, or inconsistent (duplicate IDs, an ID held by a static credential, a record that fails validation), the store is unavailable:
- every prefix credential is refused;
- create, revoke and list fail with
ECREDENTIAL_STORE_UNAVAILABLE; - no new static or tenant access key ID can be introduced, because the ID collision check cannot run:
replaceCredentials(),createBucketTenant(),ensureBucketTenant()androtateBucketTenantCredentials()throwObjectStorageCredentialErrorwith codeECREDENTIAL_STORE_UNAVAILABLEwhen they would add an access key ID that is not in the current set; - bucket deletion is refused, because the bucket's credentials cannot be revoked:
deleteBucket()anddeleteBucketTenant()without anaccessKeyIdthrowObjectStorageCredentialErrorwith codeECREDENTIAL_STORE_UNAVAILABLE, and S3 DeleteBucket returnsInvalidRequest(400) with a message startingECREDENTIAL_STORE_UNAVAILABLE:; - static and tenant credentials keep working;
getHealth().auth.prefixCredentialStoreAvailableisfalse;- the engine logs the condition without secrets and leaves the file as found.
To recover, repair or delete the file and restart. Deleting it revokes every prefix credential and forgets every revocation, so never reuse a revoked ID afterwards. Generated IDs are random.
Cluster mode. Prefix credentials are node-local, like runtime credentials: nothing replicates them, and the caller fans out.
- Create on one node with
ttlSeconds. - Create on every other node with the first node's
accessKeyId,secretAccessKey,expiresAtandregion.- Each node records its own
createdAt; every other field matches the first node's record. - An identical replay returns the stored record unchanged. Any difference gives
ECREDENTIAL_CONFLICT. - A late create never extends the lifetime.
- Each node records its own
- To revoke, call
revokePrefixCredentialon every node and retry until each node acknowledges.- A node that never saw the ID records the revocation for 12 h, so a create that reaches it later is refused with
ECREDENTIAL_REVOKED. revokedistrueonly where a live credential was removed.
- A node that never saw the ID records the revocation for 12 h, so a create that reaches it later is refused with
Revocation guarantee.
- Once a node has acknowledged a revocation, it refuses every request admitted after the acknowledgement. It also refuses any later create for that ID until the ID could no longer be created anyway: an
expiresAtcreate must lie within 12 h of the first node's create, and the revocation is kept at least that long. - A node that never acknowledged serves the key until the caller's retry reaches it, or until
expiresAt. - The hard bound is
expiresAt, at most 12 h after the first create. Each node judges expiry by its own clock, so clock offsets shift these instants. listPrefixCredentials()lets the caller reconcile nodes.
Versioning
Standalone buckets support S3 versioning (PutBucketVersioning with Enabled or Suspended, version ids, delete markers, versionId on GET, HEAD, DELETE and copy sources, ListObjectVersions). Cluster buckets stay unversioned. The full behavior, permissions and limits are in Bucket versioning.
Conditional requests
- Reads. GetObject and HeadObject evaluate
If-Match,If-None-Match,If-Modified-SinceandIf-Unmodified-Since, in standalone and cluster mode (Conditional GET and HEAD). - Create-only writes. Standalone PutObject and CompleteMultipartUpload honor
If-None-Match: *and publish through an atomic no-replace operation; of two racing create-only writes the first to commit wins. - Conditional deletes. Standalone DeleteObject honors
If-Match: *and comma-separated strong ETags. It compares the current ETag and deletes inside the bucket gate that PUT, copy and multipart completion commit under. A mismatch returnsPreconditionFailedand changes neither the object nor the stats; an absent target returnsNoSuchKey. - Copies. CopyObject and UploadPartCopy evaluate
x-amz-copy-source-if-match,-if-none-match,-if-modified-sinceand-if-unmodified-sincewith the same rules as reads, except that every failed condition is 412. - Cluster mode refuses create-only writes with
InvalidRequest(400) and conditional deletes withNotImplemented(501) until the conditions can be enforced across nodes.
Conditional GET and HEAD
GetObject and HeadObject evaluate the conditions against the version the request selects (the current one, or the one its versionId names), in the order of RFC 9110 §13.2.2, which is also the order S3 documents:
If-Match, elseIf-Unmodified-Since: when false, 412PreconditionFailed.If-None-Match, elseIf-Modified-Since: when false, 304 Not Modified.
So a matching If-Match decides over a failing If-Unmodified-Since (200), and If-None-Match decides alone over If-Modified-Since. If-Match compares entity tags strongly (a W/"…" tag never matches), If-None-Match weakly; both take * (any existing object) or a comma-separated list, quoted or not. A * beside a tag, or an empty or unterminated tag, is InvalidArgument (400). Dates compare in whole seconds, as Last-Modified carries them; a value that is not one valid HTTP-date (IMF-fixdate, RFC 850 or asctime), such as If-Modified-Since: yesterday or a list of dates, is ignored, as RFC 9110 requires. A 304 has no body and carries ETag, Last-Modified and the Cache-Control/Expires overrides (response overrides). Preconditions are evaluated before the Range header, so a failed one answers 304 or 412 where the range alone would be 206 or 416, and before a cluster GET reads any shard. A missing key, a current delete marker (404) or a delete marker named by versionId (405) answers as without preconditions.
Multipart uploads
Multipart uploads stream with bounded memory; in cluster mode each part is erasure-coded and distributed on its own.
import {
CreateMultipartUploadCommand,
UploadPartCommand,
CompleteMultipartUploadCommand,
} from '@aws-sdk/client-s3';
const { UploadId } = await client.send(new CreateMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
}));
const parts = [];
for (let i = 0; i < chunks.length; i++) {
const { ETag } = await client.send(new UploadPartCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
PartNumber: i + 1,
Body: chunks[i],
}));
parts.push({ PartNumber: i + 1, ETag });
}
await client.send(new CompleteMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
MultipartUpload: { Parts: parts },
}));
- CompleteMultipartUpload refuses, in standalone and cluster mode, a part list that names a part that was never uploaded or an ETag that does not match the uploaded part with
InvalidPart(400), and a part listed twice withInvalidPartOrder(400). The upload stays open, so a completion with the correct parts still succeeds. - Standalone ListParts responses are bounded (
max-parts,part-number-marker), so exact-streaming clients can verify abort cleanup; cluster mode refuses ListParts withInvalidRequest(400). - Uploads older than
multipart.expirationDaysare removed by cleanup passes everymultipart.cleanupIntervalMinutes. At most 100,000 uploads are registered per storage process, cleanup-pending and unreadable entries included. - UploadPartCopy enforces the 5 GiB copy-source cap but not the S3 rule that a ranged copy needs a source larger than 5 MB; plain UploadPart has no size cap.
Bucket policies
With auth.enabled: true, requests are authorized in three steps:
- Authenticate: verify the AWS SigV4 signature (anonymous requests skip this step).
- Authorize: evaluate the bucket policy against the request's action, resource and caller.
- Default: authenticated callers get full access; anonymous requests are denied unless a policy allows them.
import { PutBucketPolicyCommand } from '@aws-sdk/client-s3';
// Allow anonymous read access to all objects in a bucket
await client.send(new PutBucketPolicyCommand({
Bucket: 'public-assets',
Policy: JSON.stringify({
Version: '2012-10-17',
Statement: [{
Sid: 'PublicRead',
Effect: 'Allow',
Principal: '*',
Action: ['s3:GetObject'],
Resource: ['arn:aws:s3:::public-assets/*'],
}],
}),
}));
- Effect:
AllowandDeny; an explicit Deny always wins. - Principal:
"*"or{ "AWS": ["arn:..."] }. - Action: IAM-style actions such as
s3:GetObject,s3:PutObject,s3:*, or prefix wildcards such ass3:Get*. - Resource: ARN patterns with
*and?wildcards (arn:aws:s3:::my-bucket/*). - Policies are stored with the data and survive restarts; GetBucketPolicy, PutBucketPolicy and DeleteBucketPolicy manage them (
GET,PUT,DELETE /{bucket}?policy), and deleting a bucket removes its policy.
Observability
HTTP endpoints
The S3 listener answers four operational GET endpoints without authentication:
| Path | Answer |
|---|---|
/-/live (/-/livez) |
200 { ok: true, status: 'alive', startedAt }; also while a mounted pool is poisoned |
/-/ready (/-/readyz) |
200 ready; 503 degraded when a cluster node lacks majority or write quorum |
/-/health (/-/healthz) |
200 with the engine version, listener address and port, storage stats (bucket names included), cluster health and request counters |
/-/metrics |
Prometheus text: requests, error responses, buckets, objects, cluster enabled and quorum, noncurrent versions and bytes, delete markers |
Every Prometheus metric family is named objectstorage_*. Because /-/health lists bucket names, keep the S3 port off networks that must not see them.
Health and metrics
const health = await storage.getHealth();
const metrics = await storage.getMetrics();
getHealth()reports running state, storage directory and per-location pool observations, auth state, credential counts, bucket count, object count, total bytes, cluster health, and the resource-fencing and publication-hold capability fields.getHealth().authalso reportsprefixCredentialCount(live prefix credentials),prefixCredentialTombstoneCount(unexpired revocations) andprefixCredentialStoreAvailable(falsewhile the prefix credential file is unreadable and every prefix credential is refused). These are counts and the store state only, never an access key ID, secret, bucket or prefix.getHealth().stampClockAheadOfWallMsis present while version commit times run ahead of the wall clock (a wall clock that went backwards, see Bucket versioning).getHealth().sweepClockBehindWallMsis present while the wall clock runs ahead of the version sweep's clock (a forward jump since the engine started, see Version retention).getMetrics()returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, prefix credential (objectstorage_prefix_credentials_total,objectstorage_prefix_credential_tombstones_total,objectstorage_prefix_credential_store_available), version (objectstorage_noncurrent_versions_total,objectstorage_noncurrent_bytes_total,objectstorage_delete_markers_total), version sweep (objectstorage_versions_expired_total,objectstorage_delete_markers_expired_total,objectstorage_version_bytes_expired_total) and cluster metrics.
Storage stats
const stats = await storage.getStorageStats();
const bucketSummaries = await storage.listBucketSummaries();
console.log(stats.bucketCount, stats.totalObjectCount, stats.totalStorageBytes);
console.log(bucketSummaries[0]?.name, bucketSummaries[0]?.objectCount);
interface IBucketSummary {
name: string;
objectCount: number;
totalSizeBytes: number;
creationDate?: number;
// Versioned buckets only:
versioning?: 'Enabled' | 'Suspended';
noncurrentVersionCount?: number;
noncurrentBytes?: number;
deleteMarkerCount?: number;
versionsExpiredTotal?: number;
deleteMarkersExpiredTotal?: number;
bytesExpiredTotal?: number;
lastSweepCompletedAt?: number;
sweepDurationMs?: number;
// Versioned buckets and buckets with a lifecycle or thinning configuration:
thinning?:
| { state: 'active' }
| { state: 'inactive'; reason: 'notConfigured' | 'bucketUnversioned' | 'noNoncurrentExpirationRule' };
}
interface IStorageLocationSummary {
path: string;
totalBytes?: number;
availableBytes?: number;
usedBytes?: number;
pool?: IStoragePoolObservation;
}
interface IStorageStats {
bucketCount: number;
totalObjectCount: number;
totalStorageBytes: number;
buckets: IBucketSummary[];
storageDirectory: string;
storageLocations?: IStorageLocationSummary[];
// With at least one versioned bucket:
totalNoncurrentVersionCount?: number;
totalNoncurrentBytes?: number;
totalDeleteMarkerCount?: number;
totalVersionsExpired?: number;
totalDeleteMarkersExpired?: number;
totalBytesExpired?: number;
stampClockAheadOfWallMs?: number;
sweepClockBehindWallMs?: number;
}
- Stats are logical: object payload bytes, not sidecar files or erasure-coded shard overhead. Object counts and bytes count current objects only; a versioned bucket also reports its noncurrent data versions and their bytes and its delete markers, current and noncurrent, and the totals add them up. Stored capacity is current plus noncurrent bytes.
- The engine initializes the stats from its on-disk state at startup and updates them in memory as mutations succeed, so reads never list or rescan objects. They are exact for mutations made through objectstorage; direct filesystem edits are not watched, and a restart resyncs them.
storageLocationsis a cheap filesystem-capacity snapshot: the storage directory with its pool observation in standalone mode, the drive paths in cluster mode.
Cluster health
const clusterHealth = await storage.getClusterHealth();
if (!clusterHealth.enabled) {
console.log('Cluster mode is disabled');
} else {
console.log(clusterHealth.nodeId, clusterHealth.quorumHealthy);
console.log(clusterHealth.peers);
console.log(clusterHealth.drives);
}
interface IClusterHealth {
enabled: boolean;
nodeId?: string;
quorumHealthy?: boolean;
majorityHealthy?: boolean;
peers?: IClusterPeerHealth[];
drives?: IClusterDriveHealth[];
erasure?: IClusterErasureHealth;
repairs?: IClusterRepairHealth;
transport?: IClusterTransportHealth;
}
interface IClusterTransportHealth {
mode: 'mutualTls' | 'unauthenticated';
certificate?: { sha256: string; notAfter: number }; // only with mutualTls
caCertificates?: { sha256: string; notAfter: number }[]; // file order; only with mutualTls
}
type TDriveMedia = 'rotational' | 'smr' | 'flash' | 'zoned';
interface IClusterDriveHealth {
// index, path, status, capacity, error and erasure set fields, plus:
media: {
declared?: TDriveMedia; // not reported yet: drives cannot declare their media
detected: TDriveMedia | 'unknown'; // detected once at startup
};
}
- The engine computes every value; nothing is inferred from static config. Standalone mode returns
{ enabled: false }. - In cluster mode
quicAddressis the local QUIC/UDP socket's bound address. Withcluster.quicPort: 0the OS assigns the port; use(await seed.getClusterHealth()).quicAddressin another node'sseedNodes. Membership advertisements and peerquicAddressvalues carry that address, also after a restart. - Peer status is the local node's view of membership and heartbeats, so it is best-effort and may lag the network.
- Drive health comes from live probes of the local drive paths on every call (a write, sync and read of a probe file); capacity values are cheap filesystem snapshots. Poll
getClusterSnapshot()instead for a cheap periodic view (Dashboard). quorumHealthymeans the node currently sees majority quorum and enough available placements in every erasure set for the write quorum.drives[].media.detectedis detected once at startup. On Linux it follows the drive root's filesystem to the whole disk behind it (st_devthrough/sys/dev/block; a partition counts as its disk) and readsqueue/zonedandqueue/rotational:zonedfor host-managed or host-aware devices (SMR or ZNS), otherwiserotationalorflashas the disk's driver setsqueue/rotational. Virtual and network disks report what their driver or hypervisor sets: QEMU disks showrotationalunless configured withrotation_rate=1, NVMe volumes showflash. It isunknown, never a guess, for filesystems without a block device of their own (tmpfs, overlay, network and FUSE filesystems, btrfs subvolumes), for device-mapper, md, loop and other virtual block devices (LVM, dm-crypt and software RAID included), when sysfs gives no usable answer, and on every platform but Linux. Drive-managed SMR reports itself as rotational and cannot be detected;smrcan only ever be declared, which is planned. Detected media change no behavior today.transport.modeismutualTlswhencluster.tlsis set andunauthenticatedotherwise. With mutual TLS,certificateandcaCertificatesdescribe the files loaded at startup (SHA-256 of each DER certificate in lowercase hex,notAfterin epoch milliseconds), so during a rotation they show what each node actually runs.repairsexposes the healer's runtime state: whether a scan is active, the last completed run and the last error.repairs.shardsChecked,shardsHealedandfailedcount the last completed scan's placement checks, shards written and failures;repairs.shardsPendingcounts the pending shards that scan left pending (Self-healing). While a scan runs,lastRunStartedAtis its start and the counts are still the previous scan's;runHealingScan()returns one scan's start and counts together.
Dashboard
Every node can serve a small read-only dashboard: the cluster's nodes, drives, erasure layout and quorum, object and byte totals, request, error and byte rates, healing state and recent engine events. It is off by default:
const storage = await ObjectStorage.createAndStart({
// ...
dashboard: { enabled: true, address: '10.0.0.5', port: 3001 },
});
console.log(storage.dashboardPort); // the bound port; undefined while off
Open http://<address>:<port>/ in a browser. The page ships in the package, so nothing is fetched from elsewhere, and it updates itself over one TypedSocket connection to the same origin.
- Configuration.
dashboard.enabledmust be set.addressdefaults to127.0.0.1andportto3001;port: 0asks the OS for a port. A port equal toserver.portis refused at construction. An enabled dashboard that cannot bind failsstart()and stops the engine: it never runs silently without it. Thedashboardsection is not sent to the engine. - What it shows. Nodes (status and last heartbeat in this node's view, start time); local drives of every node (status, usage, errors, last probe); the erasure layout and quorum; objects and bytes per node and summed; request, error and byte rates per node and summed, with two-minute sparklines kept in the browser; healing per node (last scan, checked, healed, pending shards, failures); and the newest events of each node (peer and drive status changes, healing scan results, engine start). A standalone node shows its totals, rates and events. Rates come from counters of the S3 listener; byte counts are taken at the TCP stream and include HTTP headers. Nothing is stored: a reload starts the sparklines again.
- Where the data comes from.
getClusterSnapshot()(also callable directly), built by the engine from memory only: the drive states the last heartbeat probe left, the runtime stats, healing state, S3 listener counters and a ring of the newest 64 events. It never probes a drive, syncs, walks a directory or reads the control store.localis this node's snapshot; in cluster modeclusteradds quorum, the erasure layout and every peer with the peer's own snapshot, asked for over QUIC in parallel with one second per peer (a peer that is not online or does not answer in time shows as unavailable in that sample). Standalone mode returnscluster: null. One snapshot runs at a time; a call while another runs rejects. Any node with the dashboard on therefore shows the whole cluster. - Cost. With no dashboard connected there is no timer, IPC or QUIC traffic. While at least one is connected, one shared sampler takes a snapshot every two seconds, keeps at most one in flight, skips ticks while one runs, and pushes the same sample (a few KB per node) to every connected dashboard. At most 16 dashboard sockets are admitted at once; the 17th is refused with
503. On the engine side, request and byte counting is a relaxed atomic add per response and per socket read or write. Measured on a three-node cluster: an enabled dashboard with nobody watching cost within 0.4 % CPU per request and no engine memory; its server modules and page add about 50 MiB to each node's Node.js process once, at start, without growing over time or with stored objects; with 16 dashboards open on every node, large uploads ran about 1.5 % slower and p99 latency was about 5 % higher. - Security. The dashboard is read-only by construction: its socket registers one read handler and nothing else, so it cannot start a healing scan, change configuration or touch S3 traffic or stored data. It has no authentication: protect it with the bind address and a firewall, for example by binding it to a private interface. It shows node IDs, addresses, drive paths, counts and event messages, never credentials or object keys.
- Versions. Run every node on the same version, as clusters already require. An engine that predates the snapshot request logs a stream error and closes the stream, and the asking node shows it as unavailable.
Operations
Process lifecycle
start()spawns the engine and starts the HTTP server. Concurrent starts join the same attempt, and starting a running instance succeeds without a restart. A start is rejected while shutdown or unconfirmed startup cleanup is pending; after a completed shutdown, a start begins a new lifecycle. A start has no time limit, because startup grows with the stored data (a standalone root with bucket versions checks every version chain), but it fails at once when the engine process exits. A long start shows its progress on the host's stderr, where every engine log line is forwarded.stop()stops admission, drains every accepted connection and joins request and multipart workers, then terminates the engine. The drain is bounded to 30 seconds; an incomplete drain rejects the call without terminating the process and keeps the server, descriptors, locks and unfinished work, so callstop()again once the blocked operation (for example hard-mounted I/O) recovers. Streamed reads move their file descriptor into the same owned lifecycle, so a disconnected client cannot detach a blocked filesystem read. Management calls still queued when a stop begins are rejected. A terminal shutdown error is reported only after the server released its work and resources; the wrapper terminates the engine before rejecting with it.- Concurrent stops join the same attempt, including any admitted startup. A stop after a failed startup joins the remaining cleanup without sending a command to a terminated process. Repeated completed stops replay the same outcome, terminal errors included; an incomplete drain or process termination stays retryable. A successful stop therefore confirms that the instance owns no listener, work or process any more; no health polling is needed before removing a disposable root.
- If the engine has already exited,
stop()cleans up the Node side without another command. Nonzero exits and signals other than SIGTERM are logged with exit code and signal, also withserver.silent, and so is every unexpected exit below. - Unexpected engine exit.
onUnexpectedEngineExitis called when the engine process ends while the instance is running (itsstart()completed) andstop()was not requested. Who asked decides, not the exit code: an outside SIGTERM, which the engine answers with exit code 0, calls it as an OOM kill does. A deliberatestop(), a failedstart()(which rejects instead) andprobeStoragePool()/findOrphanObjectLeaves()never call it. It is called once per started lifetime, synchronously, after the instance reports not running, with{ code, signal }(nullwhen absent); the wrapper neither awaits nor catches it. The instance does not restart itself: the host decides, for example by callingstop()and exiting so its supervisor restarts it.
const storage = new ObjectStorage({
onUnexpectedEngineExit: () => void storage.stop().finally(() => process.exit(69)),
});
await storage.start();
- Load shedding. At most 4,096 HTTP requests are in flight; further requests get a retryable
503 Service UnavailablewithRetry-After: 1. - Signals. On SIGTERM or SIGINT the engine closes admission and runs the same graceful drain as
stop(), admitted uploads with their file flush and publication barriers included, then releases its resources, flushes its output and exits with code 0, even while the parent's stdin pipe stays open. Shutdown has a 35-second deadline, then at most five seconds to flush output; a watchdog bounds the whole signal shutdown to 40 seconds. A second SIGTERM or SIGINT, a missed deadline or an output failure forces exit code 70, and a completed drain with terminal errors exits nonzero after flushing, so a silent parent logger cannot hide it. Every exit with code 70 first writes its reason to the engine's stderr as one line startingruststorage forced exit (code 70):, whatever the log level, so it appears on the host's stderr ahead of the unexpected-exit line. Forced exits can lose unacknowledged writes and are reported as abnormal exits; incomplete client uploads never publish a partial object. This covers signals sent to the whole process group, by a watcher restart or Ctrl-C. Closing the engine's stdin (IPC EOF) requests the same graceful shutdown; an owned operation that cannot finish by the deadline makes the engine exit nonzero instead of waiting forever on a blocked filesystem worker. - Management replies. The engine writes its replies through one output writer with a 144 MiB line ceiling and a 160 MiB queue budget. A reader that keeps reading, however slowly, receives a reply of any length: the engine gives up on a reply only when its stdout accepts no byte for 30 seconds. Such a reply, and one beyond the line ceiling or the free queue budget, fails only its own call with an error starting
Management reply; a partly written line is closed with a newline first, so the reader keeps its framing, and the engine keeps serving S3 and later calls. A failed or closed stdout, a full output queue of 256 lines or an event that cannot be queued ends the engine with code 70.
Bucket backup and restore
const appBackup = await storage.exportBucket({ bucketName: 'workapp-123' });
await storage.importBucket({ bucketName: 'workapp-123-restore', source: appBackup });
exportBucket()returns a self-containedsmartstorage.bucket.v1JSON export with only that bucket's objects and object metadata: no credentials, policies or other tenants' data. It is bounded to 10,000 objects, 48 MiB of payload and 8 MiB of serialized keys plus metadata; use migration control for larger buckets.importBucket()validates each object's size and MD5, creates the target bucket if needed and restores the objects. In standalone mode each object is written like a PUT (staged and made durable without locking the bucket, then published), so a reader, and a crash during the import, sees an existing key's previous object or the imported one, never a mix. The bucket is not locked for the import as a whole: requests to it are served while the import runs, a fence on the bucket waits only for the object being committed, and an import whose bucket was deleted, versioned, put under retention, given a lifecycle or version thinning configuration or fenced meanwhile fails at the next object, keeping the objects it published before (requests in flight).- Export, import, fenced replacement and deletion, and migrations refuse versioned buckets, because none of them carries version history.
Bucket migration control
Standalone storage exposes durable, provider-bound migration control records; cluster mode reports this surface as unsupported. getBucketMigrationCapability() returns the local provider identity, the receipt key evidence, and source seal, destination publication and cleanup support. Create each side with createSourceMigration() and createDestinationMigration(), then advance only the explicit source path (active → sealed → cutoverCommitted → retired) with transitionSourceMigration() and the destination path (holding → verified → cutoverAuthorized → published) with transitionDestinationMigration(). Sealing introduces the source manifest digest, verification the destination's; terminal publication produces a signed receipt. Before cutover, a signed destination aborted terminal receipt can release a sealed source. Every call binds a positive fence token, the exact migration binding and an opaque controller capability; exact same-token retries replay the durable receipt, and stale or mismatched requests fail.
Source creation needs an existing bucket without retention and sourceCleanupAccessKeyId: ObjectStorage verifies that it is the bucket's one exact scoped credential, durably binds its hash before creating migration state, and revalidates that ownership before every source transition and before persisting a cleanup request. A provider has at most one nonterminal migration per role and logical bucket; terminal records stay as history and do not block a later migration ID. The standalone registry holds at most 100,000 migration states. Terminal receipts and completed-cleanup tombstones are kept permanently and count against that capacity, so controllers treat migration IDs as lifecycle records and plan capacity before it runs out. Registry creation is serialized globally and refuses a new migration ID atomically once 100,000 states exist; exact replays of existing IDs keep working at the bound, and the provider never deletes history or lets a manual registry-file reuse make room. Registry enumeration budgets durable locks, crash-left atomic temporary files and unrecognized entries separately, which leaves bounded recovery headroom at full capacity without deleting a temporary file another process may still own; every partition fails closed when it exceeds its own limit.
Sealing a source blocks later object, multipart, policy, bucket, tenant and fenced mutations. It refuses new writes from the moment it starts, waits for the commits already in flight, and refuses writes whose bodies were still arriving at their commit (503 MigrationSourceSealed), so nothing is published into the source once the seal returns (requests in flight). Reads, heads, listings, policy reads and multipart listings stay available while sealed. A read holds the bucket's publication guard only while it opens its object, so a later retirement cleanup waits for the opens in flight, not for responses still streaming, and such a response keeps returning the object it opened. Generic bucket deletion and multipart abort stay blocked throughout the seal. Versioning can be enabled on an active source, so sealing checks again and refuses a source that became versioned (EMIGRATION_VERSIONED; the source stays active).
After the destination is durably published and the source durably retired, the controller can permanently retire the source data:
const sourceState = await source.inspectBucketMigration({ migrationId });
const destinationState = await destination.inspectBucketMigration({ migrationId });
const cleanup = await source.cleanupSourceMigration({
binding,
cleanupFenceToken: 1,
opaqueCapability: sourceCapability,
sourceRetiredReceiptSha256: sourceState!.terminalReceipt!.receiptSha256,
destinationPublishedReceipt: destinationState!.terminalReceipt!,
resourceFence: {
version: 1,
scopeId: `${binding.authorityId}.${binding.logicalBucket}`,
token: 42,
mutationId: `${migrationId}.source-cleanup`,
payloadSha256: cleanupIntentSha256,
},
accessKeyId: sourceTenant.accessKeyId,
});
cleanupSourceMigration() first persists a permanent requested-cleanup subrecord binding the exact retired-source receipt, the fully verified destination-published receipt, the cleanup token, the resource fence and the hashed access key. It then reuses exact fenced deletion to remove the bucket data, every matching multipart upload, the exact credential and the bucket policy, and persists the canonical cleanup receipt and the generic delete-result digest. Multipart deletion reconciles the runtime admission counter and cleanup queue at once. The cleanup waits for the commits and object opens in flight before it revokes credentials, and refuses at their commit the writes still receiving a body.
The controller owns recovery: ObjectStorage never starts source deletion on its own. Retry the exact cleanupSourceMigration() request after any timeout, process exit or ambiguous response. A requested-only retry repeats the exact resource deletion; if the deletion was already durably complete, its generic receipt is replayed and the migration cleanup record is finalized. After completion the request cannot be rebound, the bucket can never be recreated or credentialed, and unrelated global credential replacement resumes.
inspectBucketMigration({ migrationId }) returns the durable phase, the terminal receipts and the optional sourceCleanup requested/completed subrecord, all kept across restarts. storage.cleanSlate: true cannot erase or bypass that history: standalone startup is refused while any durable migration state exists, a terminal receipt or completed-cleanup tombstone included.
Durable resource fencing
Standalone storage with cleanSlate: false supports monotonic, crash-safe bucket replacement and deletion. In immediate mode (holdPublication omitted or false) callers drain normal writes before issuing a fenced mutation.
const fence = {
version: 1 as const,
scopeId: 'corestore-node-1.workapp-123',
token: 42,
mutationId: 'delete-workapp-123',
payloadSha256: 'a'.repeat(64),
};
const result = await storage.deleteBucket({
bucketName: 'workapp-123',
accessKeyId: tenant.accessKeyId,
fence,
});
Control planes that cannot publish the provider result atomically request a durable write-publication hold:
const held = await storage.importBucket({
bucketName: 'workapp-123',
source: appBackup,
accessKeyId: tenant.accessKeyId,
fence,
holdPublication: true,
});
// Persist the provider result and update the control-plane resource first.
const release = await storage.commitResourcePublication({
barrier: held.resourcePublicationBarrier!,
resourceFenceReceipt: held.resourceFence!,
});
holdPublication: true is part of the effective mutation identity. ObjectStorage refuses new target-bucket mutations from the moment the held mutation starts, waits for the commits in flight before the destructive provider operation, then keeps all new target mutations behind a durable barrier once the provider effects and receipt are fully persisted. Mutations include object, multipart, bucket, policy, tenant and credential changes; they receive a retryable S3 WritePublicationHeld (503, Retry-After: 1) or EFENCE_PUBLICATION_HELD through the management API. Unrelated buckets stay available. Reads are not blocked: the protocol controls write publication and does not hide restored data from readers.
The exact mutation replay returns the same barrier; different and higher-token mutations cannot clear it. commitResourcePublication() validates the complete barrier, the canonical fence receipt and its 256-bit opaque commit capability, is safe to retry and returns the same release receipt. Treat commitCapability as a secret: do not log it or persist it outside protected control-plane state. A successful commit removes the raw capability from provider state and keeps only its hash.
There is no timeout, shutdown release or administrative bypass. A held barrier survives restart and is restored before the listener accepts requests. Invalid, unknown, oversized or insecure publication state fails startup closed. getResourceFencingCapability() reports requiresDrain: true for immediate mode and publicationHoldSupported: true, publicationHoldVersion: 1 and publicationHoldRequiresExternalDrain: false when the provider enforces the hold itself.
With accessKeyId, that exact key is bound into the durable mutation identity. A delete receipt is committed only after the bucket data is durably absent, that bucket-scoped credential is durably revoked and the bucket policy is durably absent. Retrying the same fence returns the stored receipt. Recovery after a crash repeats the credential and policy cleanup idempotently, so deletedCredentials can be 0, and a higher fence against an already absent bucket can return bucketDeleted: false. Completed exact deletes return credentialsPreserved: false and policyPreserved: false. Without accessKeyId, the fenced delete removes data only: tenant credentials and the bucket policy stay, and both preservation fields are true. In both cases the bucket's prefix credentials are revoked first, and the deletion is refused if that revocation cannot be stored.
After an exact delete, a new higher-token authority can recreate the same target without removing its fence history: call ensureBucketTenant() with the new fence first, then an exact fenced importBucket() with the same resource token. Ensure receipts use the smartstorage.bucket.ensure.v1 profile and attest the applied access key, the provider-computed secret hash and the effective region; the secret itself is never written to the receipt or the fencing registry. A completed fenced ensure returns its receipt on descriptor.resourceFence. computeObjectStorageResourceFenceEffectivePayloadSha256V1() computes the same canonical effective payload digest as the engine for the replacement, exact-delete and ensure profiles.
An ensure on an absent bucket removes any residual policy before creating the resource; on an existing bucket it keeps policy and object data. Once a bucket has durable fence history, createBucket(), createBucketTenant(), rotateBucketTenantCredentials(), deleteBucketTenant() and unfenced importBucket() fail with EFENCE_REQUIRED, and S3 CreateBucket and DeleteBucket get InvalidBucketState (409) with the EFENCE_REQUIRED: … message; use fenced ensure, exact import and exact delete for its lifecycle.
getBucketResourceFenceState({ bucketName }) reads one bucket's validated durable fence state under the provider's resource lock: the provider-root identity digest, bucket and resource identities, opaque scope, highest safe token and publication state (none, requested, publishing, held or released). It exposes no mutation receipts, payload evidence, commit capabilities, credentials or release proofs. null means that no fence state, sentinel or publication journal existed at the read's linearization point; a later mutation still applies the normal fencing checks. Management errors carry an ObjectStorageResourceFenceError with an EFENCE_* code (Errors).
Fences and requests in flight
A fenced replacement, deletion or publication commit and a migration transition
wait only for the commits and object opens already in flight on their bucket,
never for a request body that is still arriving or a response that is still
streaming, and requests to the bucket do not queue behind them while they wait.
A write that receives a body, copies or assembles (PutObject, CopyObject,
UploadPart, UploadPartCopy, CompleteMultipartUpload, DeleteObjects,
PutBucketPolicy, PutBucketVersioning, and the PUT and DELETE of ?lifecycle
and ?lossless-version-thinning) holds the bucket's publication guard only
while it commits, and a copy also while it opens its source. If such a mutation
ran on its bucket after the request arrived, the write is refused at its commit
and nothing of it is published: 503 WritePublicationHeld or
MigrationSourceSealed while the bucket is held or sealed, 404 NoSuchBucket
once it was deleted, otherwise 503 WritePublicationHeld. Every such mutation
on the bucket counts, also one that fails, replays a stored receipt or changes
nothing, so a write spanning it is refused with a retryable 503 even when the
bucket did not change. A retry is authorized and applied against the bucket as
it is now, so a write never publishes into contents, or with credentials, that
a fence replaced or revoked on its bucket while it was in flight.
A copy's source bucket is checked when the copy reads its source:
- CopyObject reads its source at its commit, so a mutation of the source
bucket after the request arrived refuses it there (503
WritePublicationHeld, or 404NoSuchBucketonce the bucket is gone). - UploadPartCopy reads its source when it opens it, like a GET. A mutation that ends on the source bucket before the open refuses the request with the same codes. After the open, the part copies what it opened, like a GET already streaming, also when a fence then replaces or deletes the source bucket; only a mutation of the destination bucket refuses the part at its commit.
An unfenced importBucket() is a write like these for each of its objects: it
holds the guard only while it commits one object, so a fence waits for that
commit, and the import then fails at its next object with the same codes,
keeping the objects it published before. A response
already streaming keeps returning the object version it opened, also after a
fence replaced or deleted it. A GET or HEAD that has to read its object in full
to answer (the SHA-256 that verifies a retained object against its receipt, the
ETag of an older engine's object without its .md5) releases the guard once it
opened the object, so a fence never waits for that read and the request still
answers the bytes it opened. Some operations still read an object in full
while they hold the guard: a listing, for each such older object it lists; a
copy or part copy from a retained bucket, which verifies its source while it
opens it; a DeleteObject of a retained object, or one with If-Match of
such an older object; in a versioned bucket, a PUT, CopyObject,
CompleteMultipartUpload or DELETE whose key's current object is such an
older object, which hashes it for the record of the version it supersedes;
and a PUT into a retained bucket that meets an existing or concurrently
created object, which verifies that object against its receipt. Cluster mode
has no fences, and its writes take no publication guard.
Orphan object leaves
ObjectStorage.findOrphanObjectLeaves(directory, { reportLimit? }) scans a standalone storage directory read-only for canonical v2 object leaves whose payload file is missing, without starting the S3 listener. Such a leaf makes reads, writes and deletes of its key fail, and every listing of its bucket whose prefix and cursor range covers the key. The walk only opens, stats and reads directories. Its memory is bounded by the key-frame depth and reportLimit (default 1,000, at most 100,000), not by the number of keys. The report gives the scanned bucket and leaf counts, the total orphanCount, and up to reportLimit orphans with bucket, decoded key, leaf path and the leaf's first entry names.
Startup removes empty orphan leaves; a leaf that still holds sidecar files stays until it is repaired. From engine 10.7.0 on, a leaf appears only together with its complete payload, but a DeleteObject still removes the payload before the leaf, so a scan of a live pool can briefly report a key being deleted. Engines before 10.7.0 create the leaf before the upload body is complete: scan a pool such an engine serves while it is stopped, scan a snapshot, or scan twice and treat only leaves both runs report as orphans.
Clustering Deep Dive
Multiple nodes cooperate to store and serve data with built-in redundancy.
How it works
Client ──HTTP PUT──▶ Node A (coordinator)
│
├─ Split object into 4 MiB chunks
├─ Erasure-code each chunk (4 data + 2 parity = 6 shards)
│
├──QUIC──▶ Node B (shard writes)
├──QUIC──▶ Node C (shard writes)
└─ Local disk (shard writes)
- Any node coordinates. A client connects to any member.
- Objects are chunked into fixed-size pieces (
cluster.erasure.chunkSizeBytes, default 4 MiB). - Each chunk is erasure-coded with Reed-Solomon into k data and m parity shards.
- Shards are distributed across nodes and drives for fault isolation.
- Quorums. A write is acknowledged once k+1 shards are persisted; a read needs k shards and prefers local ones. A chunk with fewer than k+m written shards is published with the missing ones as pending shards, which healing completes on their assigned drives.
Buckets, manifests and credentials are per node: a manifest lives on the node that coordinated the write, and runtime and prefix credentials are not replicated.
Failure timing. A node that membership marks offline is never contacted: its shards of a write are pending at once, reads take their shards from the other nodes, and below either quorum a request fails at once. Until a stopped node is marked offline (five missed heartbeat rounds), each chunk that needs it waits up to the QUIC idle timeout (10 s). Cluster connections send keep-alives every 2 s from both ends, so a slow but live peer stays connected however long a request takes; there is no whole-request deadline. A heartbeat to a silent peer therefore fails after 10 s, and because each round goes to every peer at once, stopped nodes are marked offline after about 50 s by default, however many stop together (a whole location, for example). A node whose restart takes longer comes back to pending shards and healing work. heartbeatTimeoutMs (default 30 s) bounds how long a live peer may take to answer a heartbeat, but cannot extend the wait for a silent one beyond 10 s.
Cluster roots
Cluster mode never creates its roots. Before startup, create storage.directory and every cluster.drives.paths entry, owned by the ObjectStorage process user, without group or world write permission. Every root must be a real directory with no symlink path component, and every drive root must resolve to a distinct device/inode identity and, unless cluster.drives.allowSharedDevice is set, to a distinct device. The control root may be its own directory or the same identity as one drive root, but it cannot make two drive entries aliases of each other. Every root must support the durability and locking semantics checked at startup; ObjectStorage fails closed before accepting traffic otherwise. Upgrades are one-way and every member runs the same version (Upgrades and rollbacks).
Erasure coding
With the default 4+2 configuration the storage overhead is 50 % (against 200 % for 3x replication), any 2 drives or nodes may fail at once, and any 4 of 6 shards rebuild a chunk.
| Config | Total shards | Overhead | Tolerance | Min nodes |
|---|---|---|---|---|
| 4+2 | 6 | 50% | 2 failures | 3 |
| 6+3 | 9 | 50% | 3 failures | 5 |
| 2+1 | 3 | 50% | 1 failure | 2 |
Erasure set formation
Drives are organized into fixed erasure sets at cluster initialization, interleaved across nodes for fault isolation:
3 nodes × 4 drives each = 12 drives total
With 6-shard erasure sets → 2 erasure sets
Set 0: Node1-Disk0, Node2-Disk0, Node3-Disk0, Node1-Disk1, Node2-Disk1, Node3-Disk1
Set 1: Node1-Disk2, Node2-Disk2, Node3-Disk2, Node1-Disk3, Node2-Disk3, Node3-Disk3
New nodes form new erasure sets; existing data is never rebalanced.
Streaming Reads
A cluster GET streams the object chunk by chunk; nothing is written to the control root. It reads from the generation of the manifest it loaded at its start, without holding the object's operation gate, so overwrites, deletes and other GETs of the same key run while it streams. Each chunk is rebuilt from the first k valid shards, read in parallel, and at most one rebuilt chunk waits ahead of the client.
- Errors after the headers abort the response. The first chunk is rebuilt before the status line, so an unreadable object still gets an error status. A failure in a later chunk ends the body with an error (the connection is closed), never with a clean end of body. A whole-object GET sends its last bytes only after the total length and the object MD5 match the manifest; range GETs are checked for length only.
- One deadline for the whole GET.
cluster.getDeadlineMsis the longest a single GET may run, counted from its start and including the client's transfer time; a GET that takes longer is aborted, and if its client stopped reading, its connection is closed. The default is 6 hours, the allowed range 1 minute to 7 days. The cleanup grace is derived from it: superseded and deleted generations keep their shards for the deadline plus the smaller of the deadline and one hour (7 hours by default, 2 minutes at the 1-minute floor), so every GET that started on a generation has ended before cleanup frees it. Writes in progress hold their object's gate and are never cleaned up, however short the grace. The setting must be the same on every node; a mismatch can abort streams, but never serves wrong bytes. CopyObject and CompleteMultipartUpload read their sources under the object gate with their own fixed 6-hour limit, counted from the start of the operation. - GET memory budget. Rebuilding chunks is charged to an engine-wide budget of
max(1 GiB, 16 × chunk footprint)at the configured chunk size. A chunk's footprint is ⌈chunk/k⌉ × (k + min(k, m)) + chunk: the k fetched shards, the data shards rebuilt for missing ones (a read rebuilds no parity shards) and the decoded chunk; 4+2 at 4 MiB is 10 MiB, 1+15 at 64 MiB is 192 MiB. A GET whose client went away keeps its charge until its local shard reads have ended, and shutdown waits for them before it releases the drive roots. The budget is separate from the upload budget. The first chunk waits at most 5 seconds for budget and is otherwise refused with503 SlowDownandRetry-After: 1; later chunks wait. Slow downloaders hold up to two rebuilt chunks each, so many of them can exhaust the budget and turn new GETs into503s under load.
Superseded generations are queued with their exact shard placements; the cleanup worker runs once per grace period and at least hourly, and purges their shard records and tombstones once the grace has passed.
QUIC transport
Nodes talk over QUIC (quinn): multiplexed streams carry concurrent shard transfers without head-of-line blocking, connections to peers are kept and reused, and QUIC flow control applies backpressure to slow peers. With cluster.tls, every connection is TLS 1.3 with a CA-signed certificate on both ends; without it, each node generates a self-signed certificate at startup and accepts any peer (Cluster transport trust). The QUIC protocol identity is smartstorage.
Cluster transport trust
Without cluster.tls the cluster transport does not authenticate nodes. Each node generates a self-signed certificate at startup and accepts any peer's certificate without verifying it. QUIC encrypts the traffic against passive eavesdropping, but not against an attacker on the path. Any host that can reach a node's quicPort can join the cluster, sync topology, and read, write, tombstone and purge shards. S3 credentials and bucket policies do not apply to this port. Run such a cluster only on a network where every host that can reach quicPort (UDP, default 4000) is a cluster node: a private network or VPN between the nodes, with quicPort firewalled from everything else.
With cluster.tls set, the transport is mutual TLS (TLS 1.3, no session resumption, no early data) with a cluster CA you run:
- Identity. A node's identity is the single DNS subject alternative name (SAN) of its certificate, and it equals
cluster.nodeId. Withcluster.tls,nodeIdis required and must be a lowercase DNS name such asnode-1ornode-1.storage.internal; it is never generated, and it must still match the node id persisted in the control root. One certificate serves as both the server and the client certificate, so it must be valid for both: an extended key usage extension listingserverAuthandclientAuth, or no extended key usage extension at all (the recipe below issues both). Use a dedicated CA per cluster: every certificate it signs with one lowercase DNS SAN is a member of the cluster. - Checks. A node connects to a known peer only if the peer's certificate names exactly that peer's node id (one DNS SAN, no wildcard), and to a seed (
seedNodes, known only by its address) only if the seed presents a certificate from the cluster CA. A node accepts a connection only with a client certificate from the cluster CA, and a join or heartbeat only for the node id that certificate names. A peer without such a certificate fails the handshake before it can send any request. Every CA-signed node is a full member: it may sync topology and read, write, tombstone and purge any shard, so the way to remove a compromised node is to stop trusting its certificate (a new CA, see the rotation below). - Startup. A node checks its own certificate the way its peers will (chain, validity, server and client authentication, the SAN, and that
keyFilematchescertFile) and refuses to start if that check or any file check below fails. The error names the file and the reason. A node never falls back to the unauthenticated transport. At startup it logs one line with its certificate'snotAfterand the SHA-256 of each CA certificate, andgetClusterHealth().transportreports them (see Cluster health). - Files.
caFileholds one CA certificate, or two during a CA rotation; anything else is refused, which keeps a system CA bundle out.certFileholds the node certificate, optionally followed by intermediates, and nothing but certificates.keyFileholds exactly one unencrypted PKCS#8, SEC1 or PKCS#1 key; ECDSA P-256 is recommended. Each path must be absolute and canonical, with no symbolic link anywhere, and lie outside the control root and every drive root (roots are backed up and replaced, and a key must not travel with them). That check compares paths, so a bind mount that makes a root reachable under another path is not detected. Each file must be a regular file of at most 64 KiB. The files and their directories must be owned by the engine's user or by root and must not be group- or world-writable, andkeyFilemust be owned by the engine's user and grant no group or world permission (mode 0600 or 0400). - Reload. The files are read once at startup; a restart is the reload. Certificates are checked at every handshake, so a connection established before a certificate's
notAfterkeeps running until it reconnects, and new handshakes afternotAfterfail. Renew certificates and restart the nodes before they expire.
Making the CA and the node certificates
The engine ships no CA tooling. With OpenSSL 3 or LibreSSL 3.3 (the openssl of macOS), create the CA once and keep ca.key offline. ec_param_enc:named_curve is required for LibreSSL, which otherwise writes the curve as explicit parameters that the engine refuses; OpenSSL 3 writes the named curve either way:
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-pkeyopt ec_param_enc:named_curve -out ca.key
openssl req -x509 -new -key ca.key -sha256 -days 3650 \
-subj "/CN=objectstorage cluster CA" \
-addext "basicConstraints=critical,CA:TRUE" \
-addext "keyUsage=critical,keyCertSign,cRLSign" \
-out ca.pem
Then issue one certificate per node, with the node id as its only DNS SAN:
NODE_ID=node-1
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 \
-pkeyopt ec_param_enc:named_curve -out "$NODE_ID.key"
openssl req -new -key "$NODE_ID.key" -subj "/CN=$NODE_ID" -out "$NODE_ID.csr"
cat > "$NODE_ID.ext" <<EXT
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature
extendedKeyUsage = serverAuth,clientAuth
subjectAltName = DNS:$NODE_ID
EXT
openssl x509 -req -in "$NODE_ID.csr" -CA ca.pem -CAkey ca.key -CAcreateserial \
-days 365 -sha256 -extfile "$NODE_ID.ext" -out "$NODE_ID.pem"
Install ca.pem and the node's certificate and key on the node, as the user that runs the engine and outside its control and drive roots:
install -d -m 0755 /etc/objectstorage/tls
install -m 0644 ca.pem "$NODE_ID.pem" /etc/objectstorage/tls/
install -m 0600 "$NODE_ID.key" /etc/objectstorage/tls/
Then set cluster.tls to the three paths (see Configuration). The test suite runs exactly these commands (test/test.cluster-tls.node.ts).
Docker secrets are mounted owned by root with mode 0444 by default, which the key check refuses. Set the key secret's uid to the engine's user and its mode to 0400 in the service's long secrets syntax. Docker Compose ignores uid, gid and mode for secrets read from a file, because it bind-mounts the file; there the host file itself must be owned by the engine's uid with mode 0600 or 0400. Kubernetes secret volumes present their files through symbolic links, which the canonical-path rule refuses.
Enabling mutual TLS
Nodes with and without cluster.tls cannot talk to each other: a TLS node refuses peers without a client certificate and refuses self-signed servers. Enabling TLS is therefore not a rolling change: stop every node, configure cluster.tls on every node, then start every node. A node whose nodeId was generated keeps that id (the control root persists it, and getClusterHealth().nodeId reports it): configure it as nodeId and issue the node's certificate for it, or the node refuses to start. Disabling TLS takes the same full stop.
Rotating certificates and the CA
A node certificate is renewed by issuing a new one from the same CA and restarting the node; its peers need no change. A new CA takes three rolling restarts, with no cluster stop:
- Put both CAs into every node's
caFile(two certificates) and restart the nodes one at a time. - Issue every node a certificate from the new CA and restart the nodes one at a time.
- Remove the old CA from every
caFileand restart the nodes one at a time.
Every pair of nodes stays connected throughout, because trust only grows before the certificates change and only shrinks after. The order matters: a node with a certificate from the new CA cannot connect to a node that trusts only the old one.
Cluster membership
- Static seed nodes define the initial cluster in config; new nodes can join a running cluster.
- Heartbeats every
heartbeatIntervalMs(default 5 s) detect suspect and offline peers; nodes mark peers offline only while they hold a majority, which prevents split-brain decisions.
Self-healing
Every 24 hours, and whenever runHealingScan() asks for it, each node scans the manifests it coordinated (manifests live on the node that wrote them):
- Every shard placement is checked at the drive it names, locally or through its node, against the checksum and size the manifest recorded. The check reads the shard record's header and the record's length only, never its payload, so a scan costs a few small reads per shard. Verifying payloads is not the scan's job: a read detects a rotted payload through its CRC32C and rebuilds the chunk from the other shards.
- A missing shard is rebuilt from k valid shards with Reed-Solomon, checked against the recorded checksum, and written to the drive its placement names. The manifest does not change.
- A pending shard is written to the drive its manifest recorded and then moved into the chunk's placements, in the same version and generation; the completed manifest is identical to one whose write never failed.
Healing never places a shard on a drive of its own choosing. Placements and pending shards on an offline node wait for that node; a pending shard whose drive refuses writes stays pending and is reported in repairs.shardsPending (replacing a permanently lost drive is not supported yet). A shard record that cannot be read, such as a truncated one, is reported in repairs.failed and left in place, because shard records are immutable; a node that cannot read a record answers with an error, never as if the shard were missing. A minority partition does not heal: its scan fails. Placements are checked without the object's operation gate, as a GET reads them. The chunks of an object that need a write are healed in batches of up to 1 GiB of chunk data: each batch takes the object's gate, reloads the manifest, heals only chunks that are unchanged in the same generation, publishes its pending shards with one manifest store and releases the gate before the next batch. An object whose gate is busy is skipped until the next scan.
Scans never overlap; a scan that outlasts the 24-hour interval delays the next one.
runHealingScan(options?) runs a scan on this node and returns its startedAt, completedAt, durationMs, shardsChecked, shardsHealed, shardsPending and failed. It joins the next scan that starts after the request arrives, never one already running, and requests that arrive together share one scan; so it covers every write that finished before the call. It rejects in standalone mode, when the scan fails (without a majority, or when a bucket's manifests cannot be listed) and when the engine shuts down before the scan completes, and it is refused while 16 requests already wait for a scan to start. The engine answers other management calls while the scan runs. options.timeoutMs (default 300,000) bounds the wait; the engine cannot cancel a scan, so one that outlasts it keeps running, its result is discarded, and a later call joins the next scan that starts after it.
Supported S3 Operations
The router fails closed, in standalone and cluster mode alike. A request selects one operation from its method, its path (/, /{bucket}, /{bucket}/{key}) and its query names, and each query name must be one that operation implements. Anything else gets NotImplemented (501, with the S3 XML error body) before the request takes a lock, reads its body or touches a bucket or object. A credential the request is not allowed for still gets AccessDenied (403) first.
| Operation | Request | Query names besides the selector | Notes |
|---|---|---|---|
| ListBuckets | GET / |
none | |
| CreateBucket, DeleteBucket, HeadBucket | PUT, DELETE, HEAD /{bucket} |
none | |
| ListObjects | GET /{bucket} |
prefix, delimiter, max-keys, marker, encoding-type |
pagination |
| ListObjectsV2 | GET /{bucket}?list-type=2 |
prefix, delimiter, max-keys, continuation-token, start-after, fetch-owner, encoding-type |
pagination |
| ListObjectVersions | GET /{bucket}?versions |
prefix, delimiter, max-keys, key-marker, version-id-marker, encoding-type |
every version and delete marker |
| ListMultipartUploads | GET /{bucket}?uploads |
prefix, encoding-type |
not paginated |
| GetBucketLocation | GET /{bucket}?location |
none | returns server.region; an empty constraint for us-east-1 |
| GetBucketVersioning, PutBucketVersioning | GET, PUT /{bucket}?versioning |
none | PUT standalone only, NotImplemented in cluster mode |
| GetBucketLifecycleConfiguration, PutBucketLifecycleConfiguration, DeleteBucketLifecycle | GET, PUT, DELETE /{bucket}?lifecycle |
none | PUT and DELETE standalone only, NotImplemented in cluster mode; PUT needs CRC32 or Content-MD5 integrity; version retention |
| Version thinning (extension) | GET, PUT, DELETE /{bucket}?lossless-version-thinning |
none | PUT and DELETE standalone only, NotImplemented in cluster mode; version retention |
| GetBucketPolicy, PutBucketPolicy, DeleteBucketPolicy | GET, PUT, DELETE /{bucket}?policy |
none | |
| DeleteObjects | POST /{bucket}?delete |
none | up to 1,000 keys, CRC32 or Content-MD5 integrity |
| GetObject, HeadObject | GET, HEAD /{bucket}/{key} |
versionId, response-cache-control, response-content-disposition, response-content-encoding, response-content-language, response-content-type, response-expires |
Range on GET; conditions |
| DeleteObject | DELETE /{bucket}/{key} |
versionId |
standalone If-Match |
| PutObject, CopyObject | PUT /{bucket}/{key}, CopyObject with x-amz-copy-source |
none | a copy needs s3:GetObject on the source and s3:PutObject on the target; x-amz-copy-source-if-* |
| CreateMultipartUpload | POST /{bucket}/{key}?uploads |
none | |
| UploadPart, UploadPartCopy | PUT /{bucket}/{key}?partNumber=…&uploadId=… |
none | UploadPartCopy: x-amz-copy-source, optional x-amz-copy-source-range: bytes=first-last and x-amz-copy-source-if-*, same permissions as CopyObject |
| CompleteMultipartUpload | POST /{bucket}/{key}?uploadId=… |
none | |
| AbortMultipartUpload | DELETE /{bucket}/{key}?uploadId=… |
none | |
| ListParts | GET /{bucket}/{key}?uploadId=… |
max-parts, part-number-marker |
standalone |
Request rules
Every operation also accepts the x-id label that AWS SDKs append and the SigV4 query-signing parameters of a presigned URL (X-Amz-Algorithm, X-Amz-Credential, X-Amz-Date, X-Amz-Expires, X-Amz-SignedHeaders, X-Amz-Signature, X-Amz-Security-Token, X-Amz-Content-Sha256). Query-string signatures are not verified: a presigned request is served as an unsigned one, which is anonymous when auth is enabled.
An object GET or HEAD (GetObject, HeadObject, ListParts) ignores query names S3 does not define, as S3 does, so links with cache-busters (?v=1) or tracking parameters (?fbclid=…) keep working. A name S3 defines that the operation does not implement (?acl, ?partNumber=1, …) is still refused there, and every other request (bucket level, every write and delete) refuses unknown names too. The set of names S3 defines is taken from the AWS SDK v3 model (3.1039.0). With CORS enabled, the 501 and 405 responses carry the CORS headers.
The response-* parameters of GetObject and HeadObject replace the Cache-Control, Content-Disposition, Content-Encoding, Content-Language, Content-Type and Expires headers of a 200 or 206 response with the given value, verbatim. A value that is not a valid HTTP header value (a control character such as CR or LF) returns InvalidArgument (400). With auth enabled, an anonymous request that carries one returns InvalidRequest (400), as S3 does, because an override such as response-content-type=text/html would let a public object serve script from the storage origin; a presigned URL is such a request. With auth disabled the overrides apply. A 304 Not Modified carries the Cache-Control and Expires overrides (the headers RFC 9110 requires a 304 to repeat from the 200) and none of the others.
Bucket versioning
Standalone buckets support S3 versioning exactly as S3 defines it: a bucket is
unversioned until PutBucketVersioning sets Enabled or Suspended, and it
never goes back to unversioned. New buckets are unversioned. A bucket that was
never versioned behaves exactly as before versioning existed, on disk and on
the wire: no version ids, no x-amz-version-id header, null as its only
version id. In cluster mode PUT ?versioning answers NotImplemented (501),
so every cluster bucket stays unversioned for now.
- Version ids. A version written while versioning is
Enabledgets a lowercase UUID as its version id. Versions written before versioning was enabled, and every version written while it isSuspended, are the key'snullversion; a key has at most one. - Enabled. PUT, CopyObject and CompleteMultipartUpload add a new current
version and keep the previous one as noncurrent; the response carries
x-amz-version-id. A DELETE without a version id adds a delete marker as the current version (also for a key with no object, as S3 does): GET and HEAD then answerNoSuchKey(404) withx-amz-delete-marker: trueand the marker'sx-amz-version-id, and ListObjects no longer lists the key. The DELETE response carriesx-amz-delete-marker: trueand the marker's id. - Suspended. Writes create the
nullversion and replace any previousnullversion of the key, wherever it is in the history; versions with ids stay. A DELETE replaces thenullversion with anulldelete marker. Responses carryx-amz-version-id: null. - By version id. GET, HEAD and DELETE take
?versionId=(an id ornull). Reading a delete marker by its id answersMethodNotAllowed(405) withx-amz-delete-marker: true, its id and itsLast-Modified. DELETE with a version id removes that version for good; when it was the current one, the next newest version becomes current again (a delete marker, or the object it covered). A well-formed id that names no version of the key isNoSuchVersion(404), also on DELETE; a malformed one isInvalidArgument. DeleteObjects accepts aVersionIdper object and reportsVersionId,DeleteMarkerandDeleteMarkerVersionIdper deleted entry. - Copies. A copy source may name a version (
?versionId=after the source);x-amz-copy-source-if-*apply to that version, and the response carriesx-amz-copy-source-version-id(andx-amz-version-idfor the new version in a versioned bucket). A delete marker as a copy source isInvalidRequest(400); a source whose current version is a delete marker isNoSuchKey. UploadPartCopy works the same way. - Conditional requests.
If-None-Match: *on PUT and CompleteMultipartUpload succeeds when the key has no current version or its current version is a delete marker.If-Matchon DELETE is evaluated against the current version, also with a version id; a current delete marker fails it (412). GET and HEAD with aversionIdevaluate their conditional headers against that version (see Conditional GET and HEAD). - ListObjectVersions lists every version and delete marker in key order,
each key newest first, with
IsLateston the current one; versions and delete markers, and common prefixes, all count againstmax-keys. A truncated page carriesNextKeyMarkerand, after a version,NextVersionIdMarker;key-markeralone resumes after every version of that key, withversion-id-markerafter that version of it. Aversion-id-markernaming a version that was deleted between the pages resumes at that key's newest remaining version: no version of the key is skipped, but versions the previous page already listed are listed again.Last-Modifiedof a version is its commit time. - Permissions.
PutBucketVersioningis authorized ass3:PutBucketVersioning,GetBucketVersioningass3:GetBucketVersioning, ListObjectVersions ass3:ListBucketVersions; aversionId(alsonull, also in DeleteObjects) ass3:GetObjectVersionors3:DeleteObjectVersion. Only root credentials may change a bucket's versioning: bucket tenant credentials getAccessDeniedforPutBucketVersioning(they may read it withGetBucketVersioning, list versions, and read and delete versions in their bucket), because the operator's bucket flows (export, import, fenced replacement and deletion, migrations) do not carry version history and refuse versioned buckets. The same holds for the lifecycle and thinning configuration (see Version retention). This lifts once a format that carries history exists. Prefix credentials never name a version and cannot change versioning; they can list versions inside their prefix, and their DELETEs add delete markers. - Not combined with. A bucket with engine-enforced immutable retention
refuses versioning, and a versioned bucket refuses retention
(
InvalidBucketState, 409). Bucket export and import, fenced bucket replacement and deletion, and bucket migrations refuse versioned buckets, because none of them carries version history; a migration whose source became versioned after it was created cannot be sealed (EMIGRATION_VERSIONED, the source stays active). Export, import, fenced replacement and migrations also refuse a bucket with a lifecycle or a thinning configuration, which they do not carry either (same codes, the message names the reason); deleting a bucket, fenced or not, removes its configuration.MfaDeleteis not implemented (501). A versioned bucket is empty for DeleteBucket only once every version and delete marker is deleted (BucketNotEmptybefore); deleting it forgets its versioning, so a new bucket of the same name starts unversioned.deleteBucketTenantdeletes every version with the bucket. - Requests by version are served under the bucket's gate. Every request
whose answer depends on a bucket's versioning state reads that state, and
the history, under the bucket's gate: DELETE and each DeleteObjects member,
GET and HEAD with a version id (also
nullon a bucket that was never versioned), a copy or UploadPartCopy source with a version id, the delete-marker answer of a key without a current object, and ListObjectVersions. A request that arrives whilePutBucketVersioningor a write's commit is in progress is therefore served by the state that change leaves. A write takes the gate only to commit: PUT, CopyObject and CompleteMultipartUpload first receive, copy or assemble their payload without it, and their commit checks again that the bucket exists, its retention, its versioning state andIf-None-Match: *. A write thatPutBucketVersioning, retention or DeleteBucket overtook while its body arrived commits under the new state. A copy publishes the source as it is at its commit: itsx-amz-copy-source-if-*conditions are evaluated again there (a condition that no longer holds answers 412), and a copy of the current object (or ofversionId=null) whose source was replaced (by a write or animportBucket()) while it was copied is refused with 409ConditionalRequestConflictand can be retried. A copy that names any other version id copies that version, also when the key changed meanwhile, and UploadPartCopy copies its source as it was when the part copy began. Requests by version, like every listing and delete, therefore never wait for uploads to the bucket in progress: measured with one 1 GiB upload into the bucket (one container capped at 8 CPUs and 12 GiB, xfs, auth off), GETs by version id issued every 50 ms during it answered in 1.3 to 1.5 ms (median; p99 2.4 to 7.8 ms), at full loopback speed and at 50 MB/s alike, against 1.4 to 1.6 ms without an upload; 11.2.0 answered such a GET only once the upload had finished, after 2.8 to 3.0 s at full loopback speed and 20.4 s at 50 MB/s. GET and HEAD without a version id read an object this engine wrote outside a retained bucket without the gate; every object of a retained bucket, an object in the sidecar layout of an older engine, and a key that a write or delete changes while the request looks it up are read under it, so such a request waits for a commit in progress, never for an upload. A read that has to hash its object in full (the SHA-256 that verifies a retained object against its receipt, the ETag of an older object without its.md5), with or without a version id, does so after it released the gate, so a commit never waits for that hash. Every GET and HEAD reports the size, ETag, Last-Modified, metadata and version id of the bytes its body returns. - Startup. A standalone engine checks every version chain once when it
starts, reading every version's record and opening its payload (and
repairing what a crash left), and takes the bucket counters from that
check. Startup time therefore grows with the number of stored versions.
Measured with 1,000,000 versions (100,000 keys × 10) on 8 CPUs with a
12 GiB memory limit, on a KVM host whose page cache did not keep the root
between starts: the first start after writing them took 346 s; six clean
starts took 127 to 264 s (127, 145, 150, 193, 203 and 264 s, median about
170 s); three starts after a SIGKILL during writes by 32 concurrent
writers took 215 to 256 s. A root on which no bucket was ever versioned
pays nothing for this and starts exactly as before.
start()waits for as long as startup takes; once recovery passes 10 s, the engine logs its progress (chains, versions, elapsed time) to the host's stderr every 10 s. - Commit times. Version commit times come from one stamp clock per
engine:
max(wall clock, last stamp + monotonic time since), persisted to.smartstorage/versioning/stamp-clock.jsonby everyPutBucketVersioningthat changes a bucket's state (before the state itself, and before the response), every 60 seconds and on a graceful stop. The monotonic time keeps counting while the host is suspended (CLOCK_BOOTTIMEon Linux,CLOCK_MONOTONICon macOS). Within a process, a wall clock that jumps back, also across a suspend, never produces a commit time earlier than true time. After a restart with the wall clock behind, commit times can lag true time by at most the downtime plus 60 seconds. A forward jump that is corrected later keeps commit times ahead by the jump for the life of the process. While commit times run ahead of the wall clock,getStorageStats()andgetHealth()reportstampClockAheadOfWallMs. Within a key, commit times never go backwards, and a version becomes noncurrent at its successor's commit time.
The reads of an unversioned bucket keep their 11.0.0 semantics:
GET ?versioningreturns the empty<VersioningConfiguration/>of a bucket that was never versioned.GET ?lifecyclereturnsNoSuchLifecycleConfiguration(404) for a bucket without a lifecycle configuration, which is every cluster bucket; in cluster modePUTandDELETE ?lifecycleareNotImplemented.GET ?versionslists every current object as its S3 null version (VersionIdnull,IsLatesttrue), withCommonPrefixesand paging onkey-markerandversion-id-marker. A truncated page returns the last key or common prefix asNextKeyMarker, plusNextVersionIdMarkernullafter a key. Aversion-id-markerneeds akey-marker.versionIdon GET, HEAD and DELETE, aVersionIdof a DeleteObjects member, and a?versionId=after the copy source of CopyObject and UploadPartCopy:nullis the current object. A well-formed id (a lowercase UUID) names no version and returnsNoSuchVersion(404;NoSuchBucketwhen the bucket is missing). Anything else, the empty string included, returnsInvalidArgument(400); in DeleteObjects both fail only their member. A?inside a copy-source key must be encoded as%3F.- These reads are authorized as
s3:GetBucketVersioning,s3:GetLifecycleConfigurationands3:ListBucketVersions. AversionId,nullincluded, is authorized ass3:GetObjectVersion(GET, HEAD, and a copy source's?versionId=) ors3:DeleteObjectVersion(DELETE), not ass3:GetObjectors3:DeleteObject, so a policy that allows DeleteObject and denies DeleteObjectVersion keeps protecting history once buckets are versioned. Bucket tenants may use both actions in their bucket.
Version retention: lifecycle and thinning
Standalone buckets expire noncurrent versions by S3 lifecycle rules, and an
extension overlay keeps a thinned history on top of them. Only lifecycle
NoncurrentVersionExpiration ever deletes a noncurrent version; the overlay
only keeps versions a rule would otherwise expire. GetBucketLifecycleConfiguration
is therefore always an upper bound on what can be deleted, and
DeleteBucketLifecycle stops all deletion: once it has answered, the sweep
removes nothing more under the old rules. The owner's three policies:
| Policy | Lifecycle rule | Overlay |
|---|---|---|
| Keep the last 6 versions | NoncurrentDays 1, NewerNoncurrentVersions 5 (the current version plus 5; S3 requires a days floor) |
none |
| Keep every version for 24 hours | NoncurrentDays 1 |
none |
| 24 hourly, 7 daily, 4 weekly, 12 monthly | NoncurrentDays 1 |
Hourly 24, Daily 7, Weekly 4, Monthly 12 |
S3 rounds NoncurrentDays up to the next midnight UTC: a version that became
noncurrent at 10:30 with one day is due at 00:00 two days later, so "24 hours"
keeps every version for more than 24 and at most 48 hours (and covers any
hourly count up to 24).
- The lifecycle subset.
PUT ?lifecycle(PutBucketLifecycleConfiguration) accepts rules with an emptyFilter, aFilterPrefixor the old rulePrefix, aStatus,NoncurrentVersionExpiration(NoncurrentDaysfrom 1, required;NewerNoncurrentVersionsfrom 1 to 100) andExpiration{ExpiredObjectDeleteMarker}. Any other element (Transition,ExpirationDaysorDate, tag or size filters,NoncurrentVersionTransition,AbortIncompleteMultipartUpload) refuses the whole document withNotImplemented(501), and so does thex-amz-transition-default-minimum-object-sizeheader; nothing is accepted and then ignored. Up to 1,000 rules and 64 KiB. The body needsContent-MD5orx-amz-checksum-crc32(AWS SDKs send one), and a wrong one isBadDigest.GET ?lifecyclereturns the document;DELETE ?lifecycleanswers 204, also without one. Within a rule a version expires only when it is pastNoncurrentDaysand has at leastNewerNoncurrentVersionsnewer noncurrent versions; any matching rule can expire it, as in S3. Delete markers never count as newer versions. A noncurrent delete marker expires with its rule when no kept data version lies directly below it, and a current delete marker with no version left below it is removed when an enabled rule setsExpiredObjectDeleteMarker. - Version thinning (
?lossless-version-thinning, an extension):<VersionThinningConfiguration><Hourly>24</Hourly><Daily>7</Daily><Weekly>4</Weekly><Monthly>12</Monthly></VersionThinningConfiguration>, each count from 1 to 1,000, at least one. Periods are UTC hours, days, ISO weeks (from Monday 00:00) and calendar months; the window of a countnis thenperiods ending with the current one, and in each of them the newest data version is kept (the current version fills its own period).GETreturns the document orNoSuchVersionThinningConfiguration(404);DELETEanswers 204. The overlay keeps versions only for keys a noncurrent rule covers; without such a rule it has nothing to keep. - Unversioned buckets store both configurations and report them inactive; nothing expires until the bucket is versioned. Versioning-suspended buckets expire noncurrent versions exactly like enabled ones, as S3 does.
- Permissions. GET is authorized as
s3:GetLifecycleConfiguration, PUT and DELETE (of both subresources) ass3:PutLifecycleConfiguration, so bucket policies apply. Bucket tenant credentials may read both and getAccessDeniedfor PUT and DELETE: a stored configuration refuses the operator's export, import, replacement and migration flows, so a tenant must not be able to set one. Prefix credentials are refused all six. Both configurations are bucket mutations for publication holds and migration seals (a sealed source answersMigrationSourceSealed). - Storage. Both are kept in
.smartstorage/versioning/lifecycle/<bucket>.json, written durably before the response; a file that fails validation stops the engine from starting. A bucket deletion removes it.
The sweep. The engine removes expired versions in passes: the first 15 minutes after it started, then 15 minutes after each pass ended, requested or timed. A pass visits every versioned bucket with an enabled rule that can expire something (no other bucket is touched) and reads each version chain without holding the bucket's gate. It then removes what expired in holds of the bucket's write gate. A hold covers at most 32 consecutive keys with at most 256 noncurrent versions and delete markers together, expired or not, because each key costs the hold a sync of its chain directory per phase and a reload of its chain; a key with a longer history runs alone, in holds of at most 256 removals, as many as it needs. Under the gate it checks that each key's chain is still the one it evaluated, apart from newer versions on top; a key that changed leaves the hold and is evaluated again. After every hold the sweep waits seven times as long as that hold lasted before it takes the bucket's gate again, so it holds the gate at most an eighth of the time and writes to the bucket get through between its holds. The pause is longer than the hold because the filesystem's work for a hold outlives it: the writes after a hold pay for part of it (measured on xfs, with ext4 as the control), so a client writing back to back loses more of its rate than the share of time the gate is held. Within a hold the work runs in phases across all its keys: the expired records of every key, then a sync of each key's chain directory, then their payloads and a sync of each, then the expired current delete markers and a sync of each. A key's payloads go only once its records are durable, and its delete marker only after its versions. A hold that fails part way repairs every key it touched. A crash at any point leaves a consistent history (startup recovery finishes the job), and the next pass completes the expiry. After a restart a pass starts over; no position is kept.
Sweeping on request. runVersionSweep(options?) runs a pass and returns
its startedAt, completedAt, durationMs, versionsExpired,
deleteMarkersExpired, bytesExpired, keysRefused and keysDeferred. It
joins the next pass that starts after the request arrives, never one already
running, so the pass applies every rule change acknowledged before the call;
requests that arrive together share one pass. It rejects in cluster mode, when
the pass fails and when the engine shuts down before the pass completes, and
it is refused while 16 requests already wait for a pass to start. The engine
answers other management calls while the pass runs. options.timeoutMs
(default 300,000) bounds the wait; the engine cannot cancel a pass, so one
that outlasts it keeps running, its result is discarded, and a later call
joins the next pass that starts after it.
The sweep's clock. A version is due by its stored noncurrent time and the
sweep's now, which is min(wall clock, wall clock at the start + monotonic time since). Within a process a backward jump of the wall clock only delays
expiry (by the jump, until the clock is corrected), and a forward jump changes
nothing: versions written after it carry commit times ahead of now, and their
keys wait until now passes them. sweepClockBehindWallMs in
getStorageStats() and getHealth() reports a forward jump while it lasts. A
single host cannot see a forward jump across a restart, and a restart with the
wall clock behind can make a version due up to one UTC midnight early when its
commit time lagged across that midnight (see Commit times above).
Stats. Every versioned bucket reports versionsExpiredTotal (noncurrent
data versions the sweep removed), deleteMarkersExpiredTotal and
bytesExpiredTotal (payload bytes of those data versions); they reconcile
with noncurrentVersionCount, deleteMarkerCount and noncurrentBytes. It
also reports lastSweepCompletedAt and sweepDurationMs of its last
completed pass; lastSweepCompletedAt is the sweep's now when that pass
ended (see The sweep's clock above), so after a forward jump of the wall clock
it trails the wall clock by sweepClockBehindWallMs, while the startedAt
and completedAt of a runVersionSweep() report are wall-clock times. Every
bucket with a versioning state or a configuration
reports thinning ({ state: 'active' }, or inactive with the reason
notConfigured, bucketUnversioned or noNoncurrentExpirationRule). The
expired totals live in memory and are written to
.smartstorage/versioning/sweep/<bucket>.json every 60 seconds, at the end of
a pass and on a graceful stop; after a crash (kill -9) they can miss what was
expired in the last 60 seconds. The live counters are recounted at startup and
stay exact. getMetrics() exports the totals as
objectstorage_versions_expired_total, objectstorage_delete_markers_expired_total
and objectstorage_version_bytes_expired_total.
encoding-type=url percent-encodes, on ListObjects, ListObjectsV2,
ListObjectVersions and ListMultipartUploads, every UTF-8 byte outside
A-Z a-z 0-9 - . _ ~ and / (a space is %20, never +) in the elements S3
encodes for that operation, and the response carries
<EncodingType>url</EncodingType>:
- ListObjects:
Prefix,Marker,NextMarker,Delimiter,KeyandCommonPrefixes/Prefix. - ListObjectsV2:
Prefix,Delimiter,StartAfter,KeyandCommonPrefixes/Prefix;ContinuationTokenandNextContinuationTokenare opaque tokens, the same with or withoutencoding-type. - ListObjectVersions:
Prefix,KeyMarker,NextKeyMarker,Delimiter,KeyandCommonPrefixes/Prefix. - ListMultipartUploads:
PrefixandKey.
Without the parameter keys are written as they are and no EncodingType is
returned; any other value returns InvalidArgument (400). An object key may
hold any UTF-8 character, also those XML 1.0 cannot carry (U+0001 and the other
control characters except tab, line feed and carriage return): a listing that
contains such a key is valid XML only with encoding-type=url. ListObjectsV2
continuation tokens use only A-Z a-z 0-9 - _, so they never break the XML.
Accepted but not applied: owners are never returned (fetch-owner); responses
say so by omitting Owner. Refused with NotImplemented although S3
defines them: every other subresource (?acl, ?tagging, ?retention,
?legal-hold, ?cors, ?website, ?object-lock, ?attributes, ?restore,
?select, …), partNumber on GET and HEAD, the paging parameters of
ListMultipartUploads (key-marker, upload-id-marker, max-uploads,
delimiter), ListObjects V1 with V2 cursors, the ListBuckets parameters
(prefix, max-buckets, continuation-token, bucket-region), headers
hoisted into a presigned URL's query (x-amz-checksum-crc32=…) on anything but
an object read, and any POST that is not one of the operations above.
Object listing and pagination
ListObjects pages identically in standalone and cluster mode, because both backends assemble a page through the same builder. A standalone listing walks only the part of the bucket its prefix and cursor can reach, and blocks only commits to the listed bucket while it walks: reads of every bucket, writes to other buckets and other listings of the same bucket proceed. A listing without a prefix still walks the whole bucket.
ListObjectsV2 (?list-type=2) reads prefix, delimiter, max-keys, continuation-token and start-after. Both cursors are exclusive: listing resumes strictly after the named key. A request may carry both; the continuation token wins, because it is the server-issued resume point of an interrupted page while start-after only states where a first page begins. The response echoes ContinuationToken and StartAfter when the request carried them, reports KeyCount as contents plus common prefixes, and returns NextContinuationToken whenever the page is truncated.
A continuation token is opaque: pass NextContinuationToken back unchanged. It is the unpadded base64url (RFC 4648 §5) encoding of the UTF-8 key the next page resumes after, at most 1,366 characters, and is the same with or without encoding-type=url, so a client may change encoding-type between pages. A token the server could not have issued (outside the base64url alphabet, padded, not canonical, or not decoding to a 1 to 1,024-byte UTF-8 key) returns InvalidArgument (400), and so do the raw-key tokens of engines before 11.0.0.
ListObjects v1 (no list-type) reads prefix, delimiter, max-keys and marker. The response always carries Marker, empty when the request carried none, and carries NextMarker whenever the page is truncated. Feeding NextMarker back as marker walks the listing.
Paging rules shared by both versions:
max-keyscounts contents and common prefixes. A delimited page ofmax-keys=1therefore returns one group or one key, never both.- The next cursor is the last emitted item of the page, which may be a common prefix. Resuming from a common prefix skips the whole group: a common prefix that is not lexicographically greater than the cursor is never returned again.
max-keys=0answers an empty, untruncated page withKeyCount0 and no next cursor.max-keysabove 1000 is clamped to 1000 and the response echoes the clamped value inMaxKeys, as S3 does. Amax-keysvalue that is not a non-negative integer is rejected withInvalidRequest, and so is astart-afterormarkerlonger than the 1,024-byte S3 key bound.- An empty
start-after,continuation-tokenormarkervalue means no cursor. - Keys are returned in ascending UTF-8 byte order, and a key is never returned twice across pages of an unchanged bucket.
ListMultipartUploads is not paginated: it returns every in-flight upload of the bucket with IsTruncated false. An optional prefix limits the result to uploads whose keys start with it and is echoed in the response; prefix credentials must supply one inside their prefix.
Bucket names and object keys
Bucket names are validated the same way at the HTTP, management, policy and storage boundaries: 3 to 63 lowercase letters, digits, dots or hyphens, beginning and ending with a letter or digit, without consecutive dots, and not formatted as an IPv4 address. Object keys are S3-compatible UTF-8 strings of up to 1,024 bytes, including leading, repeated, embedded and trailing / characters; listings return the exact original key.
API Reference
All methods are on ObjectStorage unless marked static. Option objects are named after their exported I… types in ts/index.ts.
Lifecycle
| Member | Description |
|---|---|
new ObjectStorage(config?: IObjectStorageConfig) |
Create an instance without starting it; validates the configuration |
static createAndStart(config?): Promise<ObjectStorage> |
Construct and start() in one call |
start(): Promise<void> / stop(): Promise<void> |
Start the engine and listener / drain and stop them (Process lifecycle) |
config: TObjectStorageResolvedConfig |
The resolved configuration the engine receives |
port: number |
The bound TCP port while running; the requested port before and after |
dashboardPort: number | undefined |
The dashboard's bound port while it runs |
getStorageDescriptor(overrides?): Promise<IS3Descriptor> |
Connection details for S3 clients (AWS SDK v3) |
Buckets and data
| Member | Description |
|---|---|
createBucket(name): Promise<{ name }> |
Create a bucket |
exportBucket({ bucketName }): Promise<IBucketExport> |
Bounded smartstorage.bucket.v1 export (Backup and restore) |
importBucket({ bucketName, source, accessKeyId?, fence?, holdPublication? }): Promise<IObjectStorageBucketMutationResult> |
Import an export; with fence an exact fenced replacement bound to the bucket's credential; holdPublication: true (needs a fence) returns resourcePublicationBarrier |
deleteBucket({ bucketName, accessKeyId?, fence, holdPublication? }): Promise<IObjectStorageBucketMutationResult> |
Fenced deletion; with accessKeyId it also revokes that credential and deletes the policy (Durable resource fencing) |
Credentials
| Member | Description |
|---|---|
listCredentials(): Promise<IStorageCredentialMetadata[]> |
The runtime credential set without secrets |
replaceCredentials(credentials: IStorageCredential[]): Promise<void> |
Replace the runtime set atomically (Runtime credentials) |
createBucketTenant({ bucketName, accessKeyId?, secretAccessKey?, region?, retention? }): Promise<IBucketTenantDescriptor> |
Bucket plus scoped credential (Bucket tenants) |
ensureBucketTenant({ bucketName, accessKeyId, secretAccessKey, region?, retention?, fence? }): Promise<IBucketTenantDescriptor> |
Atomic create, resume or rotate with a supplied credential; with fence the descriptor carries resourceFence |
rotateBucketTenantCredentials({ bucketName, accessKeyId?, secretAccessKey?, region? }): Promise<IBucketTenantDescriptor> |
Replace a tenant's credential |
deleteBucketTenant({ bucketName, accessKeyId? }): Promise<void> |
Revoke one credential, or delete the bucket with its credentials |
listBucketTenants(): Promise<IBucketTenantMetadata[]> |
Tenant metadata without secrets |
getBucketTenantDescriptor({ bucketName }): Promise<IBucketTenantDescriptor> |
Endpoint, credential and env values of a tenant |
createPrefixCredential(input: IPrefixCredentialInput): Promise<IPrefixCredential> |
Expiring prefix credential, or install one created on another node (Prefix credentials) |
revokePrefixCredential({ accessKeyId }): Promise<{ revoked: boolean }> |
Revoke and keep the ID refused while it could still be created |
listPrefixCredentials({ bucketName? }?): Promise<IPrefixCredentialMetadata[]> |
Live prefix credentials without secrets |
getBucketRetentionCapability(): Promise<IBucketRetentionCapability> |
Engine-enforced retention support (Immutable retention) |
getBucketRetentionReceipt({ bucketName }): Promise<IBucketRetentionReceipt | null> |
A bucket's retention receipt |
Migration and fencing
| Member | Description |
|---|---|
getBucketMigrationCapability() |
Provider identity, receipt key evidence and supported migration phases (Migration control) |
createSourceMigration(options) / createDestinationMigration(options) |
Create durable source or destination migration ownership |
transitionSourceMigration(options) / transitionDestinationMigration(options) |
Advance one allowed phase edge; returns a transition receipt |
cleanupSourceMigration(options) |
Permanently delete a retired source; exact retries return the same receipt |
inspectBucketMigration({ migrationId }) |
Durable phase, terminal receipts and cleanup state, or null |
commitResourcePublication({ barrier, resourceFenceReceipt }) |
Release a write-publication hold; exact-match and idempotent |
getResourceFencingCapability() |
Fencing support, drain requirement and publication-hold version |
getBucketResourceFenceState({ bucketName }) |
One bucket's validated durable fence state, or null |
computeObjectStorageResourceFenceEffectivePayloadSha256V1(payload) |
Exported function: the engine's canonical effective payload digest |
Observability and maintenance
| Member | Description |
|---|---|
getHealth(): Promise<IObjectStorageHealth> |
Running state, storage, auth, counts, cluster and fencing fields (Health and metrics) |
getMetrics(): Promise<IObjectStorageMetrics> |
Numeric metrics plus Prometheus text |
getStorageStats(): Promise<IStorageStats> / listBucketSummaries(): Promise<IBucketSummary[]> |
Cached logical totals, without S3 list calls (Storage stats) |
getStoragePoolDiagnostic(): Promise<IStoragePoolObservation | null> |
The active pool observation, cached for a poisoned mounted pool; null before start |
getClusterHealth(): Promise<IClusterHealth> |
Live node, drive, quorum, repair and transport health (Cluster health) |
getClusterSnapshot(): Promise<IObjectStorageClusterSnapshot> |
The dashboard's memory-only snapshot (Dashboard) |
runHealingScan({ timeoutMs? }?): Promise<IClusterHealingScanResult> |
Run and await one healing scan (Self-healing) |
runVersionSweep({ timeoutMs? }?): Promise<IVersionSweepResult> |
Run and await one version sweep pass (Version retention) |
static probeStoragePool(pool, { binaryPath? }?): Promise<IStoragePoolObservation> |
Read-only probe of a mounted pool (Storage pools) |
static findOrphanObjectLeaves(directory, { binaryPath?, reportLimit? }?): Promise<IOrphanObjectLeafReport> |
Read-only scan for payloadless object leaves (Orphan object leaves) |
Errors
Management calls reject with typed errors where the engine gives a code:
ObjectStorageCredentialError(codeinobjectStorageCredentialErrorCodes): theECREDENTIAL_*codes listed under Prefix credentials, also from tenant, credential and bucket deletion calls.ObjectStorageResourceFenceError(codeinobjectStorageResourceFenceErrorCodes):EFENCE_REQUIRED,EFENCE_STALE,EFENCE_SCOPE_MISMATCH,EFENCE_UNSUPPORTED,EFENCE_MUTATION_MISMATCH,EFENCE_STATE_CORRUPT,EFENCE_RECEIPT_LIMIT,EFENCE_BUSY,EFENCE_OWNERSHIP_MISMATCH,EFENCE_PUBLICATION_HELD,EFENCE_PUBLICATION_MISMATCH,EFENCE_PUBLICATION_CAPABILITY.RustBinaryLocatorError(ERR_RUST_BINARY_EXPLICIT_PATH_INVALID) fromstart()for an unusablebinaryPath.
Architecture
┌──────────────────────────────────────────────┐
│ Your code (AWS SDK, @lossless.org/client) │
│ ↕ HTTP (localhost:3000) │
├──────────────────────────────────────────────┤
│ ruststorage binary (Rust) │
│ ├─ hyper 1.x HTTP server │
│ ├─ S3 path-style routing │
│ ├─ StorageBackend (Standalone or Clustered) │
│ │ ├─ FileStore (single-node mode) │
│ │ └─ DistributedStore (cluster mode) │
│ │ ├─ ErasureCoder (Reed-Solomon) │
│ │ ├─ ShardBackend (per-drive shards) │
│ │ ├─ QuicTransport (quinn) │
│ │ ├─ ClusterState & Membership │
│ │ └─ HealingService │
│ ├─ SigV4 auth + policy engine │
│ ├─ CORS middleware │
│ └─ S3 XML response builder │
├──────────────────────────────────────────────┤
│ TypeScript (thin IPC wrapper) │
│ ├─ ObjectStorage class │
│ ├─ RustBridge (stdin/stdout JSON IPC) │
│ └─ Config & S3 descriptor │
└──────────────────────────────────────────────┘
The engine executable is ruststorage. The TypeScript API spawns it and drives it over newline-delimited JSON on stdin/stdout; the engine's log lines (tracing, default level info) go to its stderr and are forwarded verbatim to the host's stderr. The dashboard runs in the Node.js process and reads the engine only through that channel.
Platform support
Qualification: the amd64 suite runs natively on Ubuntu 24.04 (glibc 2.39). ARM64 storage and restart checks run in a full-system QEMU guest (Alpine 6.18.35 kernel, Ubuntu 24.04 glibc 2.39 libraries), which does not qualify physical ARM hardware or a complete Ubuntu ARM installation. The macOS engines are qualified on macOS 26.3.1 on Apple Silicon, the Intel engine through Rosetta 2; older macOS releases and physical Intel Macs are not qualified. On macOS every storage feature works on local APFS (standalone, multipart, retention, bucket replacement, clustering) with the full 1,024-byte UTF-8 key limit; NFS/SMB mountedFs pools are Linux-only.
To build from source, pnpm build builds the engine for the host (GNU and musl on Linux, Rust 1.95.0 pinned in rust-toolchain.toml) and the TypeScript API.
On-Disk Format
Format identifiers keep their smartstorage.* names (smartstorage.bucket.v1, smartstorage.bucket-versioning.v1, smartstorage.object-version.v1, smartstorage.bucket.ensure.v1, and the hash domains of the same prefix), filesystem names their .smartstorage* names, and retention metadata keys their smartstorage-retention-* names. They are part of the stored formats and never change in existing data.
Standalone mode:
{storage.directory}/
{bucket}/
.smartstorage-objects-v2/
{whole-key-hex-frames}/
object._storage_object_v2 # Object data; completed multipart objects include
# an atomic payload/ETag/metadata trailer
object._storage_object_v2.metadata.json # Legacy metadata sidecar
object._storage_object_v2.md5 # Legacy cached MD5 hash
.multipart/
{upload-id}/
metadata.json # Upload metadata
part-1, part-2, ... # Part data files
.policies/
{bucket}.policy.json # Bucket policy (IAM JSON format)
.smartstorage/staging/v1/ # Private staging: a write lands here first; its
# key leaf appears only with the complete payload
.smartstorage/credentials.json # Runtime and bucket tenant credentials (0600)
.smartstorage/prefix-credentials.json # Prefix credentials and revocations (0600)
.smartstorage/versioning/v1/{bucket}.json
# Versioning state of a versioned bucket
.smartstorage/versioning/stamp-clock.json
# Persisted floor of the version stamp clock
.smartstorage/versions/{bucket}/{whole-key-hex-frames}/
{seq:016x}-{generation} # Record of a noncurrent version or delete marker
{seq:016x}-{generation}.object # Payload of a noncurrent data version
Every object key is encoded below the reserved .smartstorage-objects-v2 bucket directory as canonical, prefix-free lowercase-hex frames. Frames and the fixed payload and sidecar names stay within the common 255-byte NAME_MAX and are safe on CIFS. Objects of the older v7 identity layout stay readable on local and NFS storage when their raw path can be reconstructed without traversal; their first mutation stages and publishes the canonical v2 object set before removing the legacy payload and sidecars, under descriptor-confined, digest-bound durable state that startup or the next access reconciles after any crash. Canonical and legacy payloads of one key without that exact owned state, noncanonical entries below the reserved v2 root, and unsafe legacy aliases on case-folding CIFS fail closed instead of choosing an ambiguous object. Descriptor-confined standalone startup reclaims abandoned private object materializations of interrupted create-only, copy or multipart publications without removing an already published hard link.
Bucket versioning keeps the layout above for current objects and adds a format that later standalone engines read unchanged:
.smartstorage/versioning/v1/{bucket}.jsonis{"format":"smartstorage.bucket-versioning.v1","bucket":…,"status":"Enabled"|"Suspended"}. A bucket without it was never versioned.- The current object of a key stays at its key path. A payload written while its bucket is versioned ends in a trailer record of schema 2 whose
versionholds its generation UUID, whether it is the null version, its sequence number and its commit time in epoch milliseconds; payloads written before keep schema 1 and are the key's null version with sequence 0. - Every other version lives in the key's chain directory below
.smartstorage/versions/{bucket}/, whose frame directories are the same hexadecimal frames as below.smartstorage-objects-v2. A record{seq:016x}-{generation}(JSON,"format":"smartstorage.object-version.v1", with the key, sequence, generation, null flag, delete marker flag, commit time, the time it became noncurrent, payload length, ETag and metadata) exists for every noncurrent version and every delete marker; a data version also has{seq:016x}-{generation}.object, the payload inode it had while it was current. Sequence order is chain order; the version id is the generation UUID, ornull. - A payload is linked into the chain before its record exists and unlinked after its record is gone. Startup recovery brings every chain to a consistent state after a crash (it removes payloads without a record, finishes or undoes an interrupted change of the current version, clears the noncurrent time of a delete marker that is current again and keeps one null version per key) and refuses to start on a record that is corrupt or misses its payload, on a chain directory with a foreign entry, and on versions of a bucket without versioning state.
Cluster mode:
{storage.directory}/
.buckets/
v2/{bucket-address}.bucket.json # Active/tombstoned bucket state
.policies/{bucket}.policy.json # Bucket policies
.manifests/
v2/{bucket-prefix}/{bucket-address}/{object-prefix}/
{object-address}.state.json
# Active/tombstoned object manifest
.multipart/{upload-id}/ # Descriptor-anchored multipart state
.smartstorage/credentials.json # This node's runtime credentials (not replicated)
.smartstorage/prefix-credentials.json # This node's prefix credentials (not replicated)
.smartstorage/cluster/
identity.json # Node and cluster identity
topology.json # Last durable topology
generation-inventory/{address}.json # Deferred physical-generation cleanup
{drive_path}/.smartstorage/data/
v2/{bucket-prefix}/{bucket-address}/{key-prefix}/{generation-address}/
{shard-address}.shard # Immutable erasure-coded shard record
{shard-address}.tombstone # Crash-safe deletion authority
Cluster control records, manifests and shards use descriptor-anchored, digest-addressed formats with durable tombstones. Records of older layouts are migrated in bounded steps, and the new records become authoritative; before a manifest listing can issue a continuation token, all remaining legacy manifests are migrated, so a later migration never puts a key behind an issued token. Older engines spooled GET responses into .manifests/.get-tmp/; startup removes a leftover spool directory that holds only spool files and otherwise leaves it in place, unused. Cleanup removes only descriptor-verified expired multipart uploads and cleanup-marked committed entries (retried in bounded passes every multipart.cleanupIntervalMinutes, large registries page by page); unreadable or incomplete descriptors fail closed and need operator repair or removal once their bucket and shard ownership is established. Shutdown never aborts or detaches an active multipart worker; it waits for it within the same retryable bounded drain.
Upgrades and rollbacks
Storage layout upgrades are one-way, and a cluster never runs mixed versions.
- Standalone v2 object layout. After the first canonical v2 object write or legacy-object migration in a root, engines 7.0.0 and earlier cannot read it, so the root cannot go back to them.
- Standalone versioning. Engines before 11.2.0 cannot read a payload with a schema-2 trailer: a standalone root on which any bucket was ever versioned (
PutBucketVersioningwithEnabledorSuspended) cannot be rolled back to an engine before 11.2.0. - Staged leaves (8.0.3 to 10.6.0). A crash between staging and publication of a PUT, CopyObject or CompleteMultipartUpload can leave a staged leaf directory
.smartstorage/staging/v1/.smartstorage-<uuid>.object.tmp.leaf/. Engines 10.7.0 and later remove it at startup; 8.0.3 to 10.6.0 refuse to start with "Standalone staging directory contains an unowned entry". Before rolling such a root back, start and stop the newer engine on it once. If it cannot start, back up the root and remove only those*.object.tmp.leafdirectories: they hold payloads that were never published. - Prefix credentials. Engines before prefix credentials never read
.smartstorage/prefix-credentials.json, so a rollback disables every prefix credential rather than widening it. - Cluster format. Upgrade every member in one coordinated maintenance window and back up the cluster roots first. Once any member has written or migrated the new state, do not restart an older engine and do not run a mixed-version cluster; a rollback restores the roots from the pre-upgrade backup.
- Pending shards. Engines that predate pending shards refuse manifests that carry them, because their chunks are short of placements. Before rolling a cluster back to such an engine, stop writing, complete or abort every open multipart upload (part descriptors record pending shards too, the scan does not count them, and an older engine drops them, so such an upload could not complete after the rollback), then run
runHealingScan()on every node until it reportsshardsPending0. Its scan starts after the request arrives, so it covers every write that finished before the call; a scan that could not list every manifest fails instead of reporting counts.
Related Packages
@lossless.org/client: one TypeScript client for lossless.org NoSQLDB, SQLDB and ObjectStorage;@lossless.org/client/objectstorageis its S3 client library@push.rocks/smartrust: the TypeScript-to-Rust IPC bridge the API runs the engine with@git.zone/rsrepo: Rust builds and cross-compilation for npm packages
License and Legal Information
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.
Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
Trademarks
This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.
Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.
Company Information
Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany
For any legal inquiries or further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.