@serve.zone/containerarchive is a content-addressed incremental backup engine with a Rust core and TypeScript API for deduplicated, compressed, optionally encrypted snapshots of pipe-compatible JavaScript streams.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Container workloads do not only need file copies. They need repeatable point-in-time snapshots, low storage amplification, safe restores, and integrity checks that can run in automation. containerarchive packages those primitives behind a small TypeScript interface while leaving chunking, hashing, pack I/O, encryption, and repair work to Rust.

Highlights

📦 Immutable snapshot manifests with tags and multi-item backup support
🧩 FastCDC content-defined chunking with SHA-256 content addressing
♻️ Cross-snapshot deduplication through a global chunk index
🗜️ gzip by default with zstd support in the Rust core
🔐 Optional AES-256-GCM encryption with Argon2id-derived passphrase wrapping
🧱 8 MB target pack files with sidecar .idx lookup data
🔍 Quick, standard, and full repository verification modes
🔒 OS-enforced descriptor locks retained across each repository mutation
🧯 Bounded, fail-closed parsing and restore preflight checks for repository-controlled metadata

Current Safety Gates

This release deliberately refuses maintenance operations that require an atomic, sharded index-generation format which is not implemented yet:

Destructive prune() calls reject before deleting snapshots or packs. dryRun: true remains available for retention analysis and fails if a retained snapshot manifest references a chunk that is absent from the global index.
deleteSnapshot() durably removes one restore-point manifest under the repository lock and is idempotent for lost-response retries. Shared immutable pack and index data is retained until safe garbage collection is available.
repair() and reindex() reject before mutation. A bounded recovery-open workflow and atomic generation publication are required before they can be enabled safely.
Reed-Solomon parity configuration remains part of the repository schema, but parity generation and recovery are disabled until groups persist across ingests and protect both .pack and .idx data.
Ingest refuses to create a 4,097th index segment. A repository at this boundary cannot accept another snapshot until an atomic index-generation format is implemented.

These gates prevent partial cleanup, stale-index resurrection, and repair of an unopenable repository through an unsafe normal-open path.

Install

pnpm add @serve.zone/containerarchive

ContainerArchive currently ships static Linux binaries for x64 and arm64. It requires Linux filesystem support for Unix-domain sockets, flock, descriptor-relative no-follow traversal, no-replace/exchange renames, and durable file and directory synchronization; initialization and open fail closed when required capabilities are unavailable.

Architecture

The TypeScript class manages the developer-facing API and uses @push.rocks/smartrust to control the compiled Rust binary. Large data does not travel through JSON IPC; the TypeScript side opens temporary Unix sockets and streams bytes directly to or from Rust.

Node.js app
  |
  | TypeScript API: ContainerArchive
  |
  | JSON IPC for commands, Unix sockets for data streams
  v
Rust engine
  |
  | chunk -> hash -> compress -> encrypt -> pack -> snapshot
  v
repository directory

Filesystem and Stream Security

Every mutation in the current repository format, including repository initialization, acquires locks/repository.lock with an operating-system flock and retains the verified repository-root descriptor for the full operation. During a locked mutation, config/index state, capacity checks, pack shards, index segments, and snapshot manifests are resolved from that exact descriptor. Renaming or replacing the original repository path therefore cannot redirect a locked publication. Immutable config, key, pack, index, and snapshot files use no-replace publication; an existing name is accepted only when its complete bytes match exactly. Encrypted key material is loaded during open() and cached in memory rather than re-resolved through the mutation lock.

Initialization creates only the final repository path component. Its parent must already exist and either belong to the current user without group/world write permission or be a root-owned sticky directory such as /tmp. A supplied empty destination must already be a current-owner 0700 directory. Initialization rejects symlink components and nonempty destinations without creating locks/; it retains the exact parent/root descriptors, takes an auxiliary root-directory flock, then holds the authoritative locks/repository.lock flock until config.json has been written last and the root has been synced and revalidated.

After the requested repository path is resolved, the completed repository root must belong to the current user and must not be group/world writable. During locked mutations, descriptor-relative child directories used for config/index refresh and publication receive the same ownership and write-permission checks, do not follow symlinks, and must remain on the root filesystem. Path-based read-only helpers canonicalize requested children and constrain them to their expected parent directories, but they do not provide the same retained-descriptor guarantee. Existing completed 0755 roots and descriptor-relative mutation directories are accepted when opened, while new initialization directories are exact 0700. The authoritative lock file is stricter: it must be a current-owner, single-link regular file with exact mode 0600. New writes create it with that mode and bind its metadata before truncation.

Repositories initialized by version 0.1.4 normally do not contain the fixed repository.lock file. If an existing or externally created lock file has mode 0644 (or otherwise fails validation), this release rejects it without modifying or automatically repairing it. Stop every repository process, verify the file and repository ownership, then perform any operator-directed remediation; do not change or remove a lock file while a process may still hold it.

Each ingest/restore stream uses a cryptographically random 0700 temporary directory, a current-owner 0600 Unix socket, and a separate 32-byte random capability. The resolved temporary-directory chain must consist only of root/current-owner non-writable ancestors or a root-owned sticky directory such as /tmp; endpoint directory and socket device/inode identities are retained and revalidated before Rust receives the path and again during cleanup. Peers must send the fixed-size capability frame before data transfer; comparison is constant-time. Incorrect peers are disconnected while the listener remains available for the legitimate Rust peer. Authentication is limited to five seconds with at most eight provisional peers, and cleanup never recursively deletes an unexpected path.

After authentication, ingest uses bounded data frames followed by an explicit completion frame. Socket EOF without that completion is an error, so a Node.js source that emits bytes and then raises an error cannot be acknowledged as a shorter successful snapshot. ingestMulti() installs listeners for every source before waiting for any socket; an error from a later item therefore fails the whole ingest even while Rust is still reading an earlier item. Before a new index segment is durable, failure cleanup removes only pack and .idx inodes created by that ingest and never removes an exact replay of a pre-existing immutable object. If the index segment is already durable and snapshot publication then fails, the indexed packs are retained as safe unreferenced deduplication data.

Ingest and closure-import sources implement IContainerArchiveInputStream: on(), off(), pipe(), and destroy() are required, while unpipe() is optional. This supports Node.js Readable streams and streamx-compatible producers while retaining deterministic error cleanup.

Quick Start

import { createReadStream, createWriteStream } from 'node:fs';
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.init('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshot = await repo.ingest(createReadStream('/tmp/database.sql'), {
  inactivityTimeoutMs: 5 * 60 * 1000,
  tags: {
    service: 'postgres',
    environment: 'production',
  },
  items: [{ name: 'database.sql', type: 'database-dump' }],
});

console.log(snapshot.id, snapshot.newChunks, snapshot.reusedChunks);

const restored = await repo.restore(snapshot.id, { item: 'database.sql' });
restored.pipe(createWriteStream('/tmp/restored-database.sql'));

await repo.close();

Open an Existing Repository

import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.open('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshots = await repo.listSnapshots({
  tags: { service: 'postgres' },
});

Repositories initialized without a passphrase are unencrypted. Encrypted repositories require the passphrase on open().

init() accepts a new-repository passphrase, partial FastCDC chunking overrides, and packTargetSize. Passphrases must contain 1 to 1024 UTF-8 bytes. Chunk sizes must be positive safe integers with minSize <= avgSize <= maxSize <= 64 MiB; the only supported algorithm is fastcdc. packTargetSize must be a positive safe integer no larger than 1 GiB. New repositories omit parity configuration while parity publication/recovery remains safety-gated, so every individually valid chunk/pack combination has the documented init contract. TypeScript validates these values before starting the Rust process, and Rust independently validates the strict IPC shape and merged configuration before creating the destination.

Multi-Item Snapshots

Use ingestMulti() when a single restore point needs several streams, for example a DB dump plus a config archive.

import { createReadStream } from 'node:fs';

const snapshot = await repo.ingestMulti([
  {
    name: 'database.sql',
    type: 'database-dump',
    stream: createReadStream('/tmp/database.sql'),
  },
  {
    name: 'volumes.tar',
    type: 'volume-tar',
    stream: createReadStream('/tmp/volumes.tar'),
  },
], {
  tags: { service: 'nextcloud', kind: 'full-backup' },
});

console.log(snapshot.items.map((item) => item.name));

Listing, Filtering, and Restore

const allSnapshots = await repo.listSnapshots();

const recentProductionSnapshots = await repo.listSnapshots({
  tags: { environment: 'production' },
  after: '2026-05-01T00:00:00Z',
});

const snapshot = await repo.getSnapshot(recentProductionSnapshots[0].id);

const stream = await repo.restore(snapshot.id, {
  item: snapshot.items[0].name,
});

Verification

const quick = await repo.verify({ level: 'quick' });
const full = await repo.verify({ level: 'full' });

if (!full.ok) {
  console.error(full.errors);
}

Verification levels are intentionally different tradeoffs: quick checks index consistency, standard reads pack metadata/checksums, and full rehydrates chunk content for the strongest validation. Verification reports observed snapshot enumeration, pack and index I/O, index parsing, and completed chunk checks as management activity, so a progressing full verification can run longer than five minutes while five minutes without verification progress still fails closed.

repair() and reindex() are currently safety-gated as described above. unlock() cannot break a live operating-system lock; process exit releases lock ownership automatically.

Retention Pruning

Retention analysis is available in dry-run mode. Destructive pruning is currently safety-gated and rejects before mutation.

const preview = await repo.prune({ keepLast: 7, keepDays: 30 }, true);
console.log('would free bytes', preview.freedBytes);

Calling prune() without dryRun: true returns an error until atomic index generations are available.

Resource Limits and Cancellation

Repository-controlled data is bounded before allocation or output. The most relevant public restore defaults are:

Limit	Default	Supported maximum
Chunk plaintext	8 MiB	64 MiB
Selected item	1 GiB	1 TiB
Selected total output	1 GiB	1 TiB
Referenced stored bytes	2 GiB	2 TiB
Chunk references	100,000	1,000,000

Pass restore(snapshotId, { limits: { ... } }) to lower or raise these limits. Both ingest()/ingestMulti() and restore() accept inactivityTimeoutMs; it must be a positive safe integer, and the default and supported maximum are five minutes. Closure export options likewise accept inactivityTimeoutMs plus limits.maxTotalBytes and limits.maxObjects. Import adds limits.maxVerifiedPlaintextBytes, which bounds the unique plaintext decrypted, decompressed, and hashed before publication. Import defaults are 64 GiB, 100,000 objects, and 64 GiB of verified plaintext; callers can raise them up to 2 TiB, 400,506 objects, and 1 TiB respectively. The ingest timer starts only after the legitimate Rust peer authenticates, so a later multi-item socket does not expire while Rust is still processing an earlier item. Long-running bridge commands use the same five-minute inactivity boundary. Authenticated ingest-frame consumption, successful restore or closure socket I/O, closure preflight and validation progress, and completed durable publication phases reset it; elapsed wall-clock time alone does not generate liveness. Destroying a returned restore or closure-export stream cancels its Unix socket so the serial Rust management loop can continue. Closing, aborting, or erroring an import source before framed completion cancels the import; after transfer, the authenticated socket remains open so peer loss cancels validation before publication starts.

ingest() requires exactly one metadata item. ingestMulti() requires between one and 64 items, and one ContainerArchive instance admits at most 64 aggregate live ingest items across concurrent calls. Admission limits are checked before private socket directories or listeners are allocated.

Each snapshot manifest is limited to 32 MiB, and one listSnapshots() response is limited to 32 MiB of source manifests. Listing fails closed beyond that boundary; paginated listing and bounded maintenance iteration require a future index/snapshot metadata format. The in-memory global index is capped at 1,000,000 unique chunks and 512 MiB of index-segment source data.

Snapshot item names must be non-empty and unique, one management process can own only one open repository at a time, and verification stops after 10,000 reported integrity errors instead of growing an unbounded response.

Events

ContainerArchive#on() exposes RxJS subscriptions for progress and integrity signals.

const subscription = repo.on('ingest:progress', (event) => {
  console.log(event.operation, event.percentage, event.message);
});

repo.on('ingest:complete', (event) => {
  console.log('snapshot complete', event.snapshotId);
});

repo.on('verify:error', (event) => {
  console.error('verification error', event.pack, event.chunk, event.error);
});

subscription.unsubscribe();

Repository Layout

An initialized repository is a directory with predictable data stores.

repo/
  config.json
  packs/
    data/
    parity/
  snapshots/
  index/
  keys/
  locks/

Path	Purpose
`config.json`	Repository ID, chunking config, compression, encryption, pack target size, and parity config.
`packs/data`	Binary pack files and pack indexes.
`packs/parity`	Reserved for the safety-gated parity format.
`snapshots`	Immutable JSON snapshot manifests.
`index`	Global content-addressed chunk index.
`keys`	Wrapped encryption keys for passphrase-protected repositories.
`locks`	Persistent metadata file whose open descriptor carries the operating-system lock.

Portable Snapshot Closure Export

exportClosure() emits an opaque stream containing the selected snapshot manifests, a deterministic scoped index, encryption key files when required, and only the immutable pack/index files reachable from those snapshots. Packs are indivisible immutable objects, so a reachable pack can contain physical bytes that are not referenced by the selected manifests. Unrelated manifests, unreachable packs, and unselected entries in the scoped global index are excluded, but a copied immutable pack sidecar index can still describe unselected chunks that share a reachable pack.

import { pipeline } from 'node:stream/promises';

const exported = await repository.exportClosure([snapshot.id]);
const [, receipt] = await Promise.all([
  pipeline(exported.stream, destination),
  exported.completion,
]);

const importedReceipt = await ContainerArchive.importClosure(
  stagingRepositoryPath,
  source,
  {
    expectedReceipt: receipt,
    passphrase: process.env.ARCHIVE_PASSPHRASE,
  },
);

console.log('imported closure', importedReceipt.sha256);

Each object and the whole stream are SHA-256 integrity-hashed. The local Unix peer is capability-authenticated, but the exported artifact is not self-authenticating: bind the complete receipt — format version, snapshot IDs, object, pack, and chunk counts, payload and archive byte lengths, and digest — to an authenticated outer protocol before remote transport. The closure config disables parity because a selected closure does not contain or claim repository-wide parity coverage.

Export is fail-before-data for preflight errors and is bounded to 256 selected snapshots, 1,000,000 aggregate chunk references, 400,506 objects, 32 MiB of aggregate selected-manifest source data, and 2 TiB of encoded output. Import requires an absolute path beneath a safe existing parent and accepts only a new destination or a current-owner, exact-0700, empty destination. It creates the repository skeleton itself, validates the exact selected object set, and publishes config.json last. It has no live-merge or force mode. After staging starts, a failure before config publication can leave an invalid, config-less repository. A durability error after config publication is reported as an ambiguous commit and can leave a valid config present. Treat either outcome as disposable staging evidence and discard only the exact staging directory that the caller allocated exclusively for this import before retrying. A failure can also occur before creating the destination, and a pre-existing destination that fails the import preconditions must never be deleted as cleanup. Do not parse closures in application code, and do not open or otherwise trust a staging repository until importClosure() resolves successfully.

Export acquires the repository operation lock, refreshes the published index, and retains one verified root descriptor through preflight and streaming. It therefore cannot combine cached state with a replaced repository path. Import checks restore-equivalent chunk bounds and item sizes, and decrypts, decompresses, and content-hashes every selected unique chunk before publication. Immediately before publishing config.json, Rust sends a fixed publication marker over the authenticated socket and waits for the local peer's acknowledgement. Cancellation remains safe before that acknowledgement; after it, the short durable commit and one-shot Rust process finish even if the peer or management transport disappears.

Encrypted closures include wrapped repository key files and must be classified and protected like the source repository. Import requires the repository passphrase so every selected chunk can be decrypted and content-verified before publication; the passphrase is also required when the imported repository is opened. Supplying a passphrase for an unencrypted closure is rejected.

API Surface

API	Purpose
`ContainerArchive.init(path, options?)`	Create a new repository and return an open instance.
`ContainerArchive.open(path, options?)`	Open an existing repository.
`IContainerArchiveInputStream`	Pipe-compatible ingest/import source contract; requires `on()`, `off()`, `pipe()`, and `destroy()`, with optional `unpipe()`.
`ingest(stream, options?)`	Store one stream as a snapshot.
`ingestMulti(items, options?)`	Store several streams as one snapshot.
`restore(snapshotId, options?)`	Return a readable stream for a full snapshot or item.
`listSnapshots(filter?)`	List snapshots, optionally filtered by tags or date.
`getSnapshot(id)`	Load one snapshot manifest.
`deleteSnapshot(id)`	Idempotently remove one restore point while retaining shared immutable pack data.
`exportClosure(snapshotIds, options?)`	Stream an exact portable closure and return its integrity receipt.
`ContainerArchive.importClosure(path, stream, options)`	Verify a closure, construct a new private staging repository, and return the authenticated and validated import receipt.
`verify(options?)`	Verify repository integrity.
`prune(retention, dryRun?)`	Analyze retention with `dryRun: true`; destructive mode is currently gated.
`repair()`	Currently rejects before mutation pending atomic recovery generations.
`reindex()`	Currently rejects before mutation pending atomic sharded generations.
`unlock(options?)`	Refuses to break a live operating-system lock.
`on(event, handler)`	Subscribe to ingest/verify events.
`close()`	Close the repository and terminate the Rust process.

Development

pnpm run build
pnpm test

Useful source entry points:

ts/index.ts exports the public API.
ts/classes.containerarchive.ts owns the TypeScript facade and stream socket handling.
ts/interfaces.ts defines snapshot, retention, verification, repair, and IPC shapes.
rust/src/main.rs starts the Rust management loop.
rust/src/ingest.rs, restore.rs, verify.rs, prune.rs, and repair.rs implement the core workflows.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license.md file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

@serve.zone/containerarchive