containerarchive/readme.md

# @serve.zone/containerarchive

`@serve.zone/containerarchive` is a content-addressed incremental backup engine with a Rust core and TypeScript API for deduplicated, compressed, optionally encrypted, parity-protected snapshots of arbitrary Node.js streams.

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Why It Exists

Container workloads do not only need file copies. They need repeatable point-in-time snapshots, low storage amplification, safe restores, and integrity checks that can run in automation. `containerarchive` packages those primitives behind a small TypeScript interface while leaving chunking, hashing, pack I/O, encryption, and repair work to Rust.

## Highlights

- 📦 Immutable snapshot manifests with tags and multi-item backup support
- 🧩 FastCDC content-defined chunking with SHA-256 content addressing
- ♻️ Cross-snapshot deduplication through a global chunk index
- 🗜️ gzip by default with zstd support in the Rust core
- 🔐 Optional AES-256-GCM encryption with Argon2id-derived passphrase wrapping
- 🧱 8 MB target pack files with sidecar `.idx` lookup data
- 🛟 Reed-Solomon parity, default RS(20,1), to recover one missing/corrupt pack per group
- 🔍 Quick, standard, and full repository verification modes
- 🧹 Retention pruning, stale lock handling, index rebuilds, and parity repair

## Install

```bash
pnpm add @serve.zone/containerarchive
```

## Architecture

The TypeScript class manages the developer-facing API and uses `@push.rocks/smartrust` to control the compiled Rust binary. Large data does not travel through JSON IPC; the TypeScript side opens temporary Unix sockets and streams bytes directly to or from Rust.

```
Node.js app
  |
  | TypeScript API: ContainerArchive
  |
  | JSON IPC for commands, Unix sockets for data streams
  v
Rust engine
  |
  | chunk -> hash -> compress -> encrypt -> pack -> snapshot
  v
repository directory
```

## Quick Start

```typescript
import { createReadStream, createWriteStream } from 'node:fs';
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.init('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshot = await repo.ingest(createReadStream('/tmp/database.sql'), {
  tags: {
    service: 'postgres',
    environment: 'production',
  },
  items: [{ name: 'database.sql', type: 'database-dump' }],
});

console.log(snapshot.id, snapshot.newChunks, snapshot.reusedChunks);

const restored = await repo.restore(snapshot.id, { item: 'database.sql' });
restored.pipe(createWriteStream('/tmp/restored-database.sql'));

await repo.close();
```

## Open an Existing Repository

```typescript
import { ContainerArchive } from '@serve.zone/containerarchive';

const repo = await ContainerArchive.open('/backups/my-service', {
  passphrase: process.env.ARCHIVE_PASSPHRASE,
});

const snapshots = await repo.listSnapshots({
  tags: { service: 'postgres' },
});
```

Repositories initialized without a passphrase are unencrypted. Encrypted repositories require the passphrase on `open()`.

## Multi-Item Snapshots

Use `ingestMulti()` when a single restore point needs several streams, for example a DB dump plus a config archive.

```typescript
import { createReadStream } from 'node:fs';

const snapshot = await repo.ingestMulti([
  {
    name: 'database.sql',
    type: 'database-dump',
    stream: createReadStream('/tmp/database.sql'),
  },
  {
    name: 'volumes.tar',
    type: 'volume-tar',
    stream: createReadStream('/tmp/volumes.tar'),
  },
], {
  tags: { service: 'nextcloud', kind: 'full-backup' },
});

console.log(snapshot.items.map((item) => item.name));
```

## Listing, Filtering, and Restore

```typescript
const allSnapshots = await repo.listSnapshots();

const recentProductionSnapshots = await repo.listSnapshots({
  tags: { environment: 'production' },
  after: '2026-05-01T00:00:00Z',
});

const snapshot = await repo.getSnapshot(recentProductionSnapshots[0].id);

const stream = await repo.restore(snapshot.id, {
  item: snapshot.items[0].name,
});
```

## Verification and Repair

```typescript
const quick = await repo.verify({ level: 'quick' });
const full = await repo.verify({ level: 'full' });

if (!full.ok) {
  console.error(full.errors);
}

const repair = await repo.repair();
console.log(repair.indexRebuilt, repair.packsRepaired, repair.errors);

await repo.reindex();
await repo.unlock();
```

Verification levels are intentionally different tradeoffs: quick checks index consistency, standard reads pack metadata/checksums, and full rehydrates chunk content for the strongest validation.

## Retention Pruning

Always dry-run retention policies before deleting data.

```typescript
const preview = await repo.prune({ keepLast: 7, keepDays: 30 }, true);
console.log('would free bytes', preview.freedBytes);

const result = await repo.prune({
  keepLast: 7,
  keepDays: 30,
  keepWeeks: 12,
  keepMonths: 6,
});

console.log(result.removedSnapshots, result.removedPacks, result.freedBytes);
```

## Events

`ContainerArchive#on()` exposes RxJS subscriptions for progress and integrity signals.

```typescript
const subscription = repo.on('ingest:progress', (event) => {
  console.log(event.operation, event.percentage, event.message);
});

repo.on('ingest:complete', (event) => {
  console.log('snapshot complete', event.snapshotId);
});

repo.on('verify:error', (event) => {
  console.error('verification error', event.pack, event.chunk, event.error);
});

subscription.unsubscribe();
```

## Repository Layout

An initialized repository is a directory with predictable data stores.

```text
repo/
  config.json
  packs/
    data/
    parity/
  snapshots/
  index/
  keys/
  locks/
```

| Path | Purpose |
| --- | --- |
| `config.json` | Repository ID, chunking config, compression, encryption, pack target size, and parity config. |
| `packs/data` | Binary pack files and pack indexes. |
| `packs/parity` | Reed-Solomon parity shards and parity manifests. |
| `snapshots` | Immutable JSON snapshot manifests. |
| `index` | Global content-addressed chunk index. |
| `keys` | Wrapped encryption keys for passphrase-protected repositories. |
| `locks` | Advisory lock records for write operations. |

## API Surface

| API | Purpose |
| --- | --- |
| `ContainerArchive.init(path, options?)` | Create a new repository and return an open instance. |
| `ContainerArchive.open(path, options?)` | Open an existing repository. |
| `ingest(stream, options?)` | Store one stream as a snapshot. |
| `ingestMulti(items, options?)` | Store several streams as one snapshot. |
| `restore(snapshotId, options?)` | Return a readable stream for a full snapshot or item. |
| `listSnapshots(filter?)` | List snapshots, optionally filtered by tags or date. |
| `getSnapshot(id)` | Load one snapshot manifest. |
| `verify(options?)` | Verify repository integrity. |
| `prune(retention, dryRun?)` | Apply retention rules and garbage collect unreferenced packs. |
| `repair()` | Rebuild index data, remove stale locks, and attempt parity recovery. |
| `reindex()` | Rebuild the global index from pack `.idx` files. |
| `unlock(options?)` | Remove advisory locks. |
| `on(event, handler)` | Subscribe to ingest/verify events. |
| `close()` | Close the repository and terminate the Rust process. |

## Development

```bash
pnpm run build
pnpm test
```

Useful source entry points:

- `ts/index.ts` exports the public API.
- `ts/classes.containerarchive.ts` owns the TypeScript facade and stream socket handling.
- `ts/interfaces.ts` defines snapshot, retention, verification, repair, and IPC shapes.
- `rust/src/main.rs` starts the Rust management loop.
- `rust/src/ingest.rs`, `restore.rs`, `verify.rs`, `prune.rs`, and `repair.rs` implement the core workflows.

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.