@serve.zone/containerarchive is a content-addressed incremental backup engine with a Rust core and TypeScript API for deduplicated, compressed, optionally encrypted, parity-protected snapshots of arbitrary Node.js streams.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Container workloads do not only need file copies. They need repeatable point-in-time snapshots, low storage amplification, safe restores, and integrity checks that can run in automation. containerarchive packages those primitives behind a small TypeScript interface while leaving chunking, hashing, pack I/O, encryption, and repair work to Rust.

Highlights

📦 Immutable snapshot manifests with tags and multi-item backup support
🧩 FastCDC content-defined chunking with SHA-256 content addressing
♻️ Cross-snapshot deduplication through a global chunk index
🗜️ gzip by default with zstd support in the Rust core
🔐 Optional AES-256-GCM encryption with Argon2id-derived passphrase wrapping
🧱 8 MB target pack files with sidecar .idx lookup data
🛟 Reed-Solomon parity, default RS(20,1), to recover one missing/corrupt pack per group
🔍 Quick, standard, and full repository verification modes
🧹 Retention pruning, stale lock handling, index rebuilds, and parity repair

Install

pnpm add @serve.zone/containerarchive

Architecture

The TypeScript class manages the developer-facing API and uses @push.rocks/smartrust to control the compiled Rust binary. Large data does not travel through JSON IPC; the TypeScript side opens temporary Unix sockets and streams bytes directly to or from Rust.

Node.js app
  |
  | TypeScript API: ContainerArchive
  |
  | JSON IPC for commands, Unix sockets for data streams
  v
Rust engine
  |
  | chunk -> hash -> compress -> encrypt -> pack -> snapshot
  v
repository directory

Quick Start

import { createReadStream, createWriteStream } from 'node:fs';
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.init('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshot = await repo.ingest(createReadStream('/tmp/database.sql'), {
  tags: {
    service: 'postgres',
    environment: 'production',
  },
  items: [{ name: 'database.sql', type: 'database-dump' }],
});

console.log(snapshot.id, snapshot.newChunks, snapshot.reusedChunks);

const restored = await repo.restore(snapshot.id, { item: 'database.sql' });
restored.pipe(createWriteStream('/tmp/restored-database.sql'));

await repo.close();

Open an Existing Repository

import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.open('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshots = await repo.listSnapshots({
  tags: { service: 'postgres' },
});

Repositories initialized without a passphrase are unencrypted. Encrypted repositories require the passphrase on open().

Multi-Item Snapshots

Use ingestMulti() when a single restore point needs several streams, for example a DB dump plus a config archive.

import { createReadStream } from 'node:fs';

const snapshot = await repo.ingestMulti([
  {
    name: 'database.sql',
    type: 'database-dump',
    stream: createReadStream('/tmp/database.sql'),
  },
  {
    name: 'volumes.tar',
    type: 'volume-tar',
    stream: createReadStream('/tmp/volumes.tar'),
  },
], {
  tags: { service: 'nextcloud', kind: 'full-backup' },
});

console.log(snapshot.items.map((item) => item.name));

Listing, Filtering, and Restore

const allSnapshots = await repo.listSnapshots();

const recentProductionSnapshots = await repo.listSnapshots({
  tags: { environment: 'production' },
  after: '2026-05-01T00:00:00Z',
});

const snapshot = await repo.getSnapshot(recentProductionSnapshots[0].id);

const stream = await repo.restore(snapshot.id, {
  item: snapshot.items[0].name,
});

Verification and Repair

const quick = await repo.verify({ level: 'quick' });
const full = await repo.verify({ level: 'full' });

if (!full.ok) {
  console.error(full.errors);
}

const repair = await repo.repair();
console.log(repair.indexRebuilt, repair.packsRepaired, repair.errors);

await repo.reindex();
await repo.unlock();

Verification levels are intentionally different tradeoffs: quick checks index consistency, standard reads pack metadata/checksums, and full rehydrates chunk content for the strongest validation.

Retention Pruning

Always dry-run retention policies before deleting data.

const preview = await repo.prune({ keepLast: 7, keepDays: 30 }, true);
console.log('would free bytes', preview.freedBytes);

const result = await repo.prune({
  keepLast: 7,
  keepDays: 30,
  keepWeeks: 12,
  keepMonths: 6,
});

console.log(result.removedSnapshots, result.removedPacks, result.freedBytes);

Events

ContainerArchive#on() exposes RxJS subscriptions for progress and integrity signals.

const subscription = repo.on('ingest:progress', (event) => {
  console.log(event.operation, event.percentage, event.message);
});

repo.on('ingest:complete', (event) => {
  console.log('snapshot complete', event.snapshotId);
});

repo.on('verify:error', (event) => {
  console.error('verification error', event.pack, event.chunk, event.error);
});

subscription.unsubscribe();

Repository Layout

An initialized repository is a directory with predictable data stores.

repo/
  config.json
  packs/
    data/
    parity/
  snapshots/
  index/
  keys/
  locks/

Path	Purpose
`config.json`	Repository ID, chunking config, compression, encryption, pack target size, and parity config.
`packs/data`	Binary pack files and pack indexes.
`packs/parity`	Reed-Solomon parity shards and parity manifests.
`snapshots`	Immutable JSON snapshot manifests.
`index`	Global content-addressed chunk index.
`keys`	Wrapped encryption keys for passphrase-protected repositories.
`locks`	Advisory lock records for write operations.

API Surface

API	Purpose
`ContainerArchive.init(path, options?)`	Create a new repository and return an open instance.
`ContainerArchive.open(path, options?)`	Open an existing repository.
`ingest(stream, options?)`	Store one stream as a snapshot.
`ingestMulti(items, options?)`	Store several streams as one snapshot.
`restore(snapshotId, options?)`	Return a readable stream for a full snapshot or item.
`listSnapshots(filter?)`	List snapshots, optionally filtered by tags or date.
`getSnapshot(id)`	Load one snapshot manifest.
`verify(options?)`	Verify repository integrity.
`prune(retention, dryRun?)`	Apply retention rules and garbage collect unreferenced packs.
`repair()`	Rebuild index data, remove stale locks, and attempt parity recovery.
`reindex()`	Rebuild the global index from pack `.idx` files.
`unlock(options?)`	Remove advisory locks.
`on(event, handler)`	Subscribe to ingest/verify events.
`close()`	Close the repository and terminate the Rust process.

Development

pnpm run build
pnpm test

Useful source entry points:

ts/index.ts exports the public API.
ts/classes.containerarchive.ts owns the TypeScript facade and stream socket handling.
ts/interfaces.ts defines snapshot, retention, verification, repair, and IPC shapes.
rust/src/main.rs starts the Rust management loop.
rust/src/ingest.rs, restore.rs, verify.rs, prune.rs, and repair.rs implement the core workflows.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

@serve.zone/containerarchive