Onebox is a self-hosted application platform for a single server. It combines Docker, CoreTraffic routing, a typed web control plane, app templates, platform services, and containerarchive-powered backups into one NodeNext TypeScript package and service.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

What Onebox Does

Onebox turns a Linux host into a small PaaS that can run your own containers and curated app templates without a separate control plane. It is designed for the "one good server" use case: one machine, one local Docker runtime, one web dashboard, one operational surface.

Deploys Docker workloads from external images or Onebox App Store templates.
Uses the local Docker socket and creates the onebox-network network automatically.
Runs workloads as Docker Swarm services when Swarm is active, otherwise as standalone containers.
Starts a CoreTraffic-backed reverse proxy for HTTP/S routing and WebSocket traffic.
Serves the web UI and TypedRequest/TypedSocket API through OpsServer on port 3000 by default.
Reports every App Store-managed service's platform OIDC eligibility through getUsersAndAppRoles.response.platformOidcQualifications, including a stable qualification code/reason, declared roles, and an optional registration view.
Shows every App Store-managed service in Access, including the same stable qualification code/reason when platform OIDC is unavailable.
Exposes an admin-authenticated read-only MCP endpoint at /mcp for safe service, domain, and platform-service summaries.
Stores platform state in a local SmartDB database, read and written through @lossless.org/client/nosqldb — the document client that supersedes @push.rocks/smartdata.
Provisions database and object-storage dependencies through one Corestore platform service. Database clients intentionally use the MongoDB wire protocol and standard MONGODB_* connection variables.
Tracks domains, Cloudflare DNS records, ACME certificates, service logs, metrics, backup schedules, and app template metadata.
Can sync routes and import certificates from an external dcrouter gateway when configured.

Architecture

browser / CLI
    |
    v
OpsServer :3000
  - bundled web UI
  - TypedRequest handlers
  - TypedSocket dashboard events
    |
    v
Onebox coordinator
  - SmartDB repositories
  - Docker manager
  - CoreTraffic route manager
  - DNS and SSL managers
  - platform service providers
  - app store manager
  - backup manager and scheduler
    |
    v
Docker host
  - onebox-network
  - CoreTraffic
  - user services
  - optional platform services

Onebox is the central class. It initializes the database, Docker, CoreTraffic, DNS, SSL, platform services, App Store, backup subsystem, optional external gateway integration, and the web/API server.

Routes across stop, delete and daemon restart

Every service record publishes its hostnames for as long as it exists, whatever its lifecycle. Onebox builds all of them from the database: CoreTraffic's hostname routes and certificates, the public TCP/UDP listeners, and, when a dcrouter gateway is configured, the gateway routes with their DNS records.

Stop (service stop) stops the runtime and records the service as desired stopped. Its hostname, certificate, public listeners, gateway route and DNS record all stay, so a resolver never caches a missing name during maintenance. Only the answer changes: CoreTraffic answers every request to a stopped service's hostname itself, over HTTP and, with the service's own certificate, over HTTPS, with 503 Service Unavailable, Retry-After: 30, Cache-Control: no-store and the plain-text body The service at <hostname> is stopped. Thirty seconds keeps clients that honour Retry-After to two requests a minute during a stop that usually lasts minutes, and brings them back within half a minute of the start; no-store keeps caches from serving the 503 after it. The answer is a SmartProxy respond route, rendered in one place, buildStoppedServiceAction() in ts/classes/coretraffic.ts. A service whose runtime Onebox observes running is always forwarded, even if its record says stopped.
Start and restart record the service as desired running and forward its hostnames to the runtime again. The gateway route is re-synced idempotently.
Delete (service remove) is the only lifecycle step that removes the gateway route, the DNS record, the CoreTraffic route and the certificate. Removing a domain through a service update unpublishes that domain the same way.
Daemon restart (systemctl stop/start onebox) keeps ingress up. CoreTraffic runs as a Swarm service and keeps serving the last applied routes while the daemon is down. On start, Onebox rebuilds the complete route set from the database and adopts the running CoreTraffic service when its specification matches the one it would create, applying the rebuilt routes in one update. A missing or drifted service is (re)created. Only systemd disable and the shutdown of a server --ephemeral run remove CoreTraffic.
A refused route set: when SmartProxy refuses a route, CoreTraffic answers 422 and Onebox fails the change with CoreTrafficRoutesRefusedError, which names every refused route, what it serves (for example HTTPS route of app.example.com, stopped) and SmartProxy's reasons, and states what serves ingress now. A running service that refuses an update keeps serving its previous route set. A service Onebox has just created has no previous set: when it refuses its first route set, Onebox removes it and the error states that ingress is down. That happens at the first start after a CoreTraffic image change, and when a public listener change replaces the service; for a listener change Onebox first hands the new route set to the running service, so a set it refuses keeps that service and its routes in place.
A refused takeover is reported, not silent. The system status reports coreTraffic.ownership: owned while the daemon's route changes reach CoreTraffic, refused when CoreTraffic refused the route set of the daemon's last takeover (the adoption or creation at start, a new takeover after a refusal, or the new service of a public listener change), failed (with startError) when the start failed for another reason, and released before the start and after shutdown. coreTraffic.lastRefusal holds the latest refused set (when, what serves ingress, the message and every refused route) until CoreTraffic accepts a set. While refused, the next route change that alters the route set takes CoreTraffic over again with it, adopting the running service or creating a new one; the refused set itself is never sent again, so nothing retries on its own. Every route change that does not reach CoreTraffic is logged as an error naming why. Takeovers and route pushes run one at a time in call order: a change that arrives during a takeover applies after it, a start during a listener replacement waits for the replacement, and a listener change is compared with the listeners of the service that runs when its turn comes. A daemon stop takes priority over them: the shutdown stops the takeover or route push in flight at its next safe point, so it waits for at most one Docker call in flight, or for one service replacement in flight, then runs alone and leaves the daemon released. A replacement, which removes the old service, pauses two seconds and creates the new one, runs to its end, so a stop during it never leaves the old service removed and no new one created. A takeover stopped this way never marks CoreTraffic owned; a service it had just created keeps running, without routes until the next start adopts it and hands it the route set, or, when an ephemeral run shuts down, is removed once by that shutdown. Every change still queued is refused by name, for example CoreTraffic route change not applied: this process is releasing CoreTraffic. After a failed start, route changes apply at the next daemon start or when the CoreTraffic platform service is started.

Each service record stores two lifecycle fields. desiredState (running or stopped) is written only by deploy, start, restart and stop. status is the observed runtime state that the monitoring loop records. The monitoring loop never changes desiredState or removes a route. It skips a service while a lifecycle command is working on it, and it changes nothing when Docker cannot answer. The service-desired-state data step (0.23.0 → 0.24.0) derives desiredState for existing services from their last recorded status.

The dcrouter gateway client credential needs every capability Onebox uses. The one declaration of that set is oneboxGatewayCapabilities in ts/classes/external-gateway.ts, which also states what each capability is for: currently readDomains, readDnsRecords, readRoutes, syncRoutes, syncDnsRecords, readCertificates and manageMail. Onebox reads the credential's live capabilities from getGatewayClientContext when the gateway starts and on every later context read. It logs one error naming every missing capability, and again only when the gap changes, and reports it in the system status as externalGateway.missingCapabilities. Onebox does not refuse to start over a gap. Each step that needs a missing capability says it is skipped: without readRoutes the stale public-port route cleanup is skipped, and without readDnsRecords the stale hostname cleanup is. Listing gateway routes or DNS records never answers an empty list for a missing capability or a failed request: it names the missing capability or passes the failure on. An admin credential is not checked. A capability change on the dcrouter side bumps the gateway client's policy generation, so the credential has to be reissued and stored again with config set dcrouterGatewayApiToken --stdin. The server --ephemeral foreground run removes CoreTraffic on shutdown; the daemon and a plain server run leave it serving.

Platform contracts and images

Onebox speaks the serve.zone 32 contracts: @serve.zone/interfaces 32.15.0, @serve.zone/api 32.0.0, @serve.zone/appstore 32.2.0, @serve.zone/corestore 32.4.1 and @serve.zone/workloadinit 32.1.1. The installed interfaces release is the protocol version — nothing on a wire, in a record or in a digest input states a version of ours — and every reader is exact-key, so a peer one major away is refused by shape rather than half-understood.

The platform images Onebox runs are declared once, in ts/platform-images.ts, and are pinned by version or by manifest digest. serve.zone publishes version tags only from 32.0.0 on, and latest is frozen at its last 31-line build, so a latest reference would silently pin a pre-32 peer:

Service	Reference
CoreMail	`code.foss.global/serve.zone/coremail:32.0.0`
CoreTraffic	`code.foss.global/serve.zone/coretraffic:32.1.0`
dcrouter (managed gateway default)	`code.foss.global/serve.zone/dcrouter:32.0.0`
Corestore 32.2.3	`code.foss.global/serve.zone/corestore@sha256:5cc449ce…`
MariaDB 11.8.9	`mariadb@sha256:79d59758…`
ClickHouse 26.9.1.1629	`clickhouse/clickhouse-server@sha256:42acb460…`

Corestore is pinned by digest because the platform migration compares the image a host already ran against the image this release expects, and that comparison is only meaningful between immutable references. MariaDB and ClickHouse are upstream images whose tags their publishers can move to another build, so they are pinned by the digest of their multi-platform image index. The Cloudly image is not pinned here at all: it lives in the App Store catalog that Onebox reads live from @serve.zone/appstore.

Installation

Install the latest released Linux binary:

curl -sSL 'https://code.foss.global/api/v1/repos/serve.zone/onebox/raw/install.sh?ref=main' | sudo bash

The installer downloads onebox-linux-x64 or onebox-linux-arm64 from the latest Gitea release, installs it under /opt/onebox, and links /usr/local/bin/onebox. Use --version vX.Y.Z to pin a release, --install-dir /path to change the target directory, or --source to clone the tag and build the NodeNext package locally.

curl -sSL 'https://code.foss.global/api/v1/repos/serve.zone/onebox/raw/install.sh?ref=main' | sudo bash -s -- --source

For source checkouts, install dependencies and build directly:

pnpm install
pnpm run build
node ./cli.js --help

This repository currently marks the npm package as private; use the installer or a source checkout until public npm release packaging is enabled.

Quick Start

Run a foreground development instance:

onebox server --ephemeral

Open the dashboard:

http://localhost:3000

On the first browser visit, Onebox asks you to create the administrator passkey. Enter the short-lived bootstrap code printed in the local Onebox service logs, then complete the browser passkey prompt. The code authorizes first-time enrollment only; Onebox never accepts a human password.

Production passkeys require a stable HTTPS hostname. While the Onebox coordinator is running, persist it from another terminal with onebox config set adminUiDomain onebox.example.com, then restart Onebox before opening the production UI and enrolling the passkey.

ONEBOX_ADMIN_UI_DOMAIN=onebox.example.com is an alternative only when added to the Onebox systemd service environment; exporting it in an interactive shell does not update the generated unit. After enrollment, the hostname is pinned to the passkey relying party.

If every passkey for the configured administrator is lost, use local host access to reset enrollment:

sudo onebox systemd stop
sudo onebox auth reset-passkeys --confirm
sudo onebox systemd start
sudo onebox systemd logs

The reset invalidates every passkey and session for the configured administrator. It never enables password authentication. The command opens the database under the production state root /var/lib/onebox (or ONEBOX_STATE_ROOT) and runs the startup migrations as a boot does, including the backup archive conversion; use --state-root <path> only for a nonstandard installation. The former --database-path flag and a --state-root without a path are refused before any database is opened.

Deploy a simple service:

onebox service add web --image nginx:latest --domain web.example.com --port 80

For production, install and run the systemd service:

sudo onebox systemd enable
sudo onebox systemd start
sudo onebox systemd logs

The systemd unit runs onebox systemd start-daemon with /var/lib/onebox as its working directory. From source or foreground runs, the default SmartDB path is ./.nogit/smartdb relative to the current working directory.

CLI Reference

onebox <command> [options]

A failed command exits with status 1 and prints its error message first, then every attached cause on its own line under caused by:, with the members of a combined failure listed under -. The chain stops after eight levels and never repeats an error, and it prints messages only, never stack traces. A command that runs in the daemon reports the daemon's error message only. Onebox never quotes secret-bearing input that fails to parse. When App Store secret env stdin, a decrypted credential, a persisted signing key, the managed dcrouter config, a Corestore control API response, backup content or App Store migration script output is not valid JSON, Onebox names the input by what it is, and a malformed control request or response line by its byte length, never with the parser's error, whose message quotes the text it could not read.

Core commands:

Command	Purpose
`selftest typed-rpc [--port <port>]`	Start and stop an isolated server to verify the packaged Typed RPC runtime graph.
`server [--ephemeral] [--port <port>] [--monitor]`	Start the web/API server in the foreground.
`service add <name> --image <image> [--domain <domain>] [--port <port>] [--env KEY=VALUE]`	Deploy a workload.
`service inspect-metadata <name>`	Print narrow service ownership metadata and sorted environment and secret key names through the root-local daemon control socket, without values or secret-file contents.
`service patch-env <name> --stdin`	Atomically merge a JSON `{ "set": { ... }, "unset": [ ... ], "unsetSecrets": [ ... ] }` patch through the root-local daemon control socket. `unsetSecrets` is optional. Input is limited to 64 KiB.
`service set-secret <name> <key> --stdin`	Replace one encrypted service secret through the root-local daemon control socket. Accepts a single-line value up to 64 KiB on stdin.
`service list`	List known services.
`service start <name>`	Start a stopped service.
`service stop <name>`	Stop a running service. Its hostnames, certificates, gateway routes and DNS records stay published.
`service restart <name>`	Restart a service.
`service remove <name>`	Remove a service with its routes, gateway routes, DNS records and certificates.
`service logs <name>`	Print Docker logs for a service.
`service stage-workloadinit-approval <name> --stdin`	Verify and deliver a new WorkloadInit approval or matching higher-generation revocation through the root-local coordinator.
`service resume-workloadinit-approval <name>`	Resume a pending approval from its encrypted custody record.
`service workloadinit-approval-status <name>`	Inspect current authority and pending delivery without printing the artifact.
`appstore list`	List remote app templates.
`appstore config <app-id> [--version <version>]`	Print app metadata and version config.
`appstore install <app-id> --name <name> [--domain <domain>] [--version <version>] [--env KEY=VALUE] [--secrets-stdin]`	Install an app template. Pass declared secret overrides as one JSON object through stdin; `--env` accepts only non-secret values. The object may span several lines and is limited to 2 MiB; each value is limited to 500 KiB and all values together to 1 MiB, the sizes the encrypted secret store holds. A value may contain line breaks only when its declaration delivers it as a file, such as Cloudly's `SERVEZONE_WORKLOADINIT_APPROVAL_FILE`.
`appstore upgrade <service-name> [--version <version>]`	Upgrade an App Store-managed service through the forward-only catalog gate.
`registry add --url <url> --username <user> --password-stdin`	Store encrypted external registry credentials without exposing the password in process arguments.
`registry remove --url <url>`	Remove registry credentials.
`registry list`	List configured registries.
`dns add <domain>`	Add a DNS record through the configured DNS manager.
`dns sync`	Sync Cloudflare domains into Onebox.
`ssl renew [domain]`	Renew one certificate or expiring certificates.
`ssl list`	List stored certificates.
`ssl force-renew <domain>`	Force certificate renewal for a domain.
`proxy reload`	Reload routes and certificates into CoreTraffic.
`proxy test`	Check reverse proxy state.
`proxy status`	Print route/certificate counts and ports.
`systemd enable`	Install and enable the systemd unit.
`systemd disable`	Stop, disable, and remove the systemd unit, then remove the CoreTraffic service that otherwise keeps serving across daemon restarts.
`systemd start`	Start Onebox through systemd.
`systemd stop`	Stop Onebox through systemd.
`systemd status`	Show service status.
`systemd logs`	Follow `journalctl` logs.
`config show`	Show stored settings with secret values masked.
`config set <key> <value>`	Store a non-secret setting.
`config set <secret-key> --stdin`	Store one bounded, single-line secret without placing it in process arguments.
`backup create <service-name>`	Create a containerarchive backup through the running root-local coordinator.
`backup list [service-name]`	List completed backups, optionally for one service.
`backup import <service-name> --file <absolute path> --password-stdin`	Import a backup download, read by the daemon, as a backup of a service installed on this host; the download's backup password is read from stdin.
`backup restore <backup-id>`	Restore a backup in place over the service it belongs to.
`auth reset-passkeys --confirm [--state-root <path>]`	Locally invalidate every passkey for the configured administrator and prepare fresh enrollment.
`migration corestore-platform --status`	Read the redacted Corestore platform migration status without initializing or mutating SmartDB.
`migration corestore-platform --approve --checkpoint-reference <reference> [--allow-empty-legacy-mongodb]`	Offline, resumable cutover from split legacy storage/database services to one Corestore owner.
`migration corestore-engine --approve --checkpoint-reference <reference>`	Offline, resumable SmartDB 5 to NoSQLDB 10 whole-owner conversion. Leaves all captured workloads stopped.
`migration corestore-engine --status`	Read the value-free conversion journal through SmartDB's bounded stopped-engine inspector without initializing Onebox.
`migration corestore-engine --adopt-target-image --checkpoint-reference <reference> --expected-prior-image <repository@sha256:digest>`	After a converter-image fix, adopt only this Onebox release's pinned Corestore digest at the fenced pre-export boundary.
`migration corestore-engine --replace-active-target-image --checkpoint-reference <reference> --expected-prior-image <repository@sha256:digest> --expected-target-container <64-hex-id>`	Replace the exact active target with this release's pinned Corestore digest while retaining the same staged root and recovery containers.
`migration corestore-engine --replace-completed-target-image <repository@sha256:digest> --migration-id <32-hex-id> --expected-prior-image <repository@sha256:digest> --expected-target-container <64-hex-id>`	Move a completed conversion to this release's pinned, newer Corestore release on the same staged volume; restores the prior target on any failure.
`migration corestore-engine --stopped-image <service> <repository@sha256:digest>`	Select a pinned replacement image for a captured stopped workload while Onebox remains offline.
`migration corestore-engine --stage-approval <service> --stdin`	Stage a WorkloadInit approval through its owner on a captured stopped workload.
`migration corestore-engine --complete --checkpoint-reference <reference> --external-acceptance-reference <reference>`	Verify stopped workloads, target, resource credentials, and the operator's external-owner acceptance reference before normal startup.
`migration corestore-data --approve --checkpoint-reference <reference>`	Offline, resumable move of a completed conversion's Corestore data root out of its staged Docker volume into the Onebox-owned volume `corestore`.
`migration corestore-data --rollback`	Return an unaccepted data relocation to the staged volume and the prior Corestore target.
`migration corestore-data --complete --external-acceptance-reference <reference>`	Record the operator's acceptance of the relocated Corestore owner; afterwards it is not rolled back.
`migration corestore-data --status`	Read the value-free relocation record through SmartDB's bounded stopped-engine inspector without initializing Onebox.
`status`	Print JSON system status.
`upgrade`	Install the latest released package build. Requires root.

Corestore converter-image adoption has two strict pre-export boundaries. Without a journaled source-digest proof, the export volume must be empty. With a journaled proof, the export volume may contain only its canonical root-private proof file, whose checksum and complete captured-source identity must still match; Onebox then reruns authenticated digests against the isolated source helper before the journal compare-and-swap and on every exact replay. Successful retargets form a bounded, ordered, immutable image history. The first successful retarget lazily moves a version 1 journal to version 2, preserving its existing transition when present, and later retargets append without rewriting prior evidence.

The offline migration corestore-platform, migration corestore-engine and migration corestore-data commands always shut Onebox's offline maintenance down afterwards, and a shutdown failure never replaces the command's own outcome. After a failed command, the command's error is still reported first and carries the shutdown failure, named Onebox offline maintenance did not shut down after the command failed, as its cause beside any cause it already had, so the operator sees both. After a successful command, the shutdown failure fails the command as Onebox offline maintenance did not shut down after the command completed.

App Store upgrades fail closed unless the fresh index, app metadata, and target config agree. Semantic-version targets must be newer than the installed version. Branch and digest-tracked upgrades must use the fresh latest target. Onebox rejects stale operations when the service template or installed version changes before migration or apply.

Service deployment and replacement derive imageDigest from digest-pinned image references, including curated templates without resolver metadata. Explicit digests must be valid SHA256 values and match the pin. Replacing a mutable image or registry location, or forcing a mutable-image pull, clears the previous digest unless a new one is supplied; configuration-only updates preserve it. Failed replacements restore the prior image tuple.

An installed Cloudly WorkloadInit approval file mapping is reserved. Generic service updates and App Store upgrades must preserve its exact root-owned 0400 mapping, and the approval artifact itself is written only by the approval owner, never through a generic secret mutation.

service set-secret preserves other encrypted secrets and removes any public environment copy of the selected key. It uses normal service replacement, retaining the image, storage and prior running/stopped state. Repeating the command delivers the value again; matching stored ciphertext alone is not proof of runtime delivery. The reserved WorkloadInit approval key still requires its dedicated command. Updating a bootstrap password input does not rotate an application's persisted credential: complete rotation through that application's authenticated credential operation before treating the previous password as revoked.

service patch-env reads the latest public and encrypted service environment while holding the service mutation lock, applies public unset followed by set, and removes only encrypted aliases explicitly named by optional unsetSecrets. A key may appear in set and unsetSecrets to move an encrypted alias to a public template; set and public unset remain disjoint. Secret-file source keys and the reserved WorkloadInit approval source cannot be removed by this command. One normal replacement receives the complete merged public environment and, when an encrypted alias changes, the full remaining encrypted map. A public-only patch does not rewrite encrypted storage. An empty patch is rejected; replaying a patch that makes no semantic change across both maps returns without replacing the runtime or invalidating an approval. Image identity, secret files, approval state and prior running or stopped state remain owned by the normal replacement lifecycle.

service inspect-metadata reports only the service name and status, persisted service and runtime IDs, image and digest, App Store template identity, sorted public and encrypted environment key names, and secret-file source and target names. Values, ciphertext and file contents never enter the result.

The legacy nginx command name is still accepted as an alias for proxy, but CoreTraffic is the active proxy backend.

Ongoing WorkloadInit approval ownership

Use the service commands above with a digest-pinned service and the canonical root-owned 0400 approval mapping. The artifact is accepted only through bounded multiline stdin (500 KiB), never through command arguments or public service metadata.

sudo onebox service workloadinit-approval-status cloudly
sudo onebox service stage-workloadinit-approval cloudly --stdin < /root/workloadinit-approval.json
sudo onebox service resume-workloadinit-approval cloudly

Custody that predates 8.6.0 was imported once, by data migrations that are now below the ledger floor: approval custody arrived as an observed, unaccepted intent with no invented generations, history, acceptance or runtime timestamps, and legacy platform credential copies moved out of public service environment metadata into the encrypted bundle. Startup holds such a workload at zero before routing, verifies the artifact, and performs a controlled runtime replacement. Only successful delivery creates one imported-current-state history event at the artifact's actual generation, explicitly recording that prior history is unavailable. An unverifiable or revoked-only observation cannot establish the first accepted authority.

The owner atomically records its verified decision, encrypted service secret, and durable hold through SmartData transactions. It recreates the runtime using normal service lifecycle management, proves the exact image, secret mapping, and previous running/stopped state, then atomically appends immutable history, advances current authority, and releases the hold. Generic service mutations and Docker runtime mutation paths cannot bypass a pending hold. After a crash, startup discovers the uniquely owned runtime, repairs uncertain pointers, and proves zero replicas before reconciliation.

Exact accepted replay is a no-op. Any different artifact must advance the authority generation; an explicit revocation must name the current approval digest. A pending accepted operation must finish before another artifact can be staged. Resume after acceptance uses the persisted decision and does not reinterpret a registry outage as revocation. WorkloadInit 1.3 has no approval expiry. Invalid active-shaped artifacts never become revocation authorities.

A service that maps the approval file receives its first approval with its deployment: an App Store install of Cloudly supplies it as the template's SERVEZONE_WORKLOADINIT_APPROVAL_FILE secret input, and an ops API deployment as that key among its encrypted environment. The deployment never writes the approval itself. It creates the service without it, hands it to the owner's verified stage, which verifies it, requires an active approval, records it as the service's first accepted authority at the artifact's own generation and creates the runtime with its root-only mapping, and then starts the service. A deployment that maps the file without offering an approval is refused by name before a service exists. When the stage refuses the artifact, for example because it fails verification or is revoked-only, the deployment is rolled back and no service remains. A backup never supplies a first approval: an import or clone of a backup that carries one stays refused. Recovering Cloudly on another host therefore installs it there with a freshly issued approval, imports the old host's backup download for that service with onebox backup import, and restores it in place with onebox backup restore, which keeps the new approval; see Backups.

Removing a service ends with one transaction that removes the service record, its encrypted secrets, its approval owner record and its whole approval history together, so a removal that fails at any point leaves the service with its approval and generation floor or removes all of them. It is refused while an approval operation is pending; resume that operation first. A new service receives the highest stored service id plus one, so a service created after the highest-numbered one was removed receives its id, and it starts with no approval and no history. Earlier releases left that state behind; the workloadinit-orphaned-approvals data migration removes it for every service that no longer exists and for every existing service that carries no approval file mapping, which can only have inherited it. A pending operation of a deleted service is removed with it, because nothing can resume, deliver or hold an operation whose service is gone, and start-up removes such an operation the same way instead of refusing to start.

Corestore platform cutover

Fresh database and object-storage resources are owned by one pinned Corestore service. SmartDB replaces the MongoDB backend while deliberately retaining the MongoDB wire protocol, port 27017, driver compatibility, and MONGODB_* workload variables. SmartDB is the engine; Onebox reaches it through the @lossless.org/client document client, and the two are independent. Onebox does not run a standalone MongoDB platform service for new deployments.

Legacy split ownership is converted only by the explicit migration in ts_migration/. Before running it, stop Onebox and create a separately stored, verified, restorable full checkpoint of the authoritative Corestore volume. Pass the operator checkpoint identifier unchanged on every retry:

sudo onebox systemd stop
sudo onebox migration corestore-platform --status
sudo onebox migration corestore-platform --approve \
  --checkpoint-reference <verified-checkpoint-reference>
sudo onebox migration corestore-platform --status
sudo onebox systemd start

The migration captures exact platform/resource ownership, fences affected workloads, renames the legacy object-storage owner to Corestore, provisions target resources, converts non-empty legacy databases with the pinned Corestore migration image, atomically replaces resource credentials, and restarts only workloads that were running before the cutover. Each phase is persisted and replay-safe. Database conversion temporarily leases the canonical onebox-mongodb name to the exact captured source container so its persisted replica-set identity remains valid. A stopped canonical-name holder is displaced and retained only when its exact command, networks, data volume, key-file bind, and mount topology prove it is a legacy twin of the captured source; running or unrelated owners fail closed without MongoDB mutation. The migration restores the exact captured recovery name before advancing the phase. Replaying an already-complete migration re-provisions and verifies the canonical credential tuple and reports repaired when persisted credentials or mapped service settings changed. Running services are restarted only when credential reconciliation changes their delivered settings. Recovery containers are renamed and retained; the command prints their names so an operator can remove them only after production verification and the agreed recovery window.

An empty legacy database is recorded as a persisted inspection proof and fails closed on the first run. After reviewing that proof, rerun the same command with --allow-empty-legacy-mongodb. This approval skips only databases re-inspected as empty; it does not disable verification for non-empty databases. Normal Onebox startup never performs this infrastructure cutover implicitly.

Corestore engine conversion

A Corestore root written by SmartDB 5 cannot be opened by NoSQLDB 10. Use the separate corestore-engine offline command only after stopping Onebox and preserving a verified, restorable full checkpoint of its owner database, Corestore source volume, image, and approval/keyring authority. The command captures every workload, fences each Docker runtime and the original Corestore, exports through an isolated source with no external network, copies the entire non-database root to its final volume, verifies held database imports, and retains the exact old stopped container and volume. The target starts only after activation is journaled. Before the first ordinary credential rotation, Onebox durably snapshots affected environment templates from the service's exact installed App Store version, or from templates already persisted for a manual service; it never infers aliases from frozen literal values. The snapshot must resolve to the exact old stopped runtime. Every ordinary database and object-storage credential is then rotated through Corestore's durable API, its value-free receipt is saved before protected material is requested, and each captured workload receives canonical encrypted mappings plus its re-resolved templates while stopped. Onebox proves the exact stopped replacement runtime before advancing. A lost response reuses the recorded operation identity. The source and target volumes must not be renamed, remounted, or copied after staging.

corestore-engine --status is observational. While Onebox is stopped, it asks SmartDB's bounded stopped-engine inspector for the single conversion-journal value, strictly validates the persisted V1, V2, V3, or V4 owner shape, and emits only the value-free public projection. It does not initialize Onebox, start Docker or backup services, seed defaults, or run startup migrations. Missing or malformed owner state and a running database engine are refused.

The non-database copy preserves a hardlinked regular file only when every link to its exact device and inode is present inside the copied root. An alias outside that root, an alias crossing into the excluded legacy smartdb root, a symbolic link, or an unsupported entry stops the conversion before copying. Capacity counts each admitted inode once and retains the 8 GiB rollback reserve, including a fresh reserve check after verification and syncing. Verification requires the same path-to-inode topology, metadata and bytes; timestamps are deliberately normalized to the integral-microsecond precision that the Node utimes API can restore. The copy syncs every staged file and directory and accepts a nonempty stage only when it is the exact durable result of a lost acknowledgement. A partial or altered stage remains held and requires separately fenced replacement; the conversion never deletes it implicitly.

If the pinned converter image fails before a source export succeeds, install a published Onebox release with a qualified new Corestore digest, then run --adopt-target-image --checkpoint-reference <the same checkpoint> --expected-prior-image <the exact journaled old digest> while Onebox remains stopped. The new image is selected solely by that Onebox release. The command refuses an altered source, running captured workload, export artifact or unjournaled export residue, existing target stage, or any later conversion phase. The exact canonical source-digest proof is the only permitted export-volume entry when its hash and source identity are already journaled. Each successful adoption appends the old and new immutable digests to the migration history. Repeat --approve with the same checkpoint afterward. An exact retry of the adoption command is safe after a lost response.

If a released target defect is found after activation but before the conversion advances beyond active, install a published Onebox release with the corrected Corestore digest and run --replace-active-target-image with the same checkpoint, exact prior digest and exact current target container id. Onebox journals the intent before stopping anything, retains the prior target under a deterministic recovery name, creates and starts an exact candidate on the same staged volume with the same durable administration authority, moves the platform owner through exact compare-and-update operations, and appends the completed handoff. Candidate health is checked through its own loopback-bound control endpoint and persisted administration token; a stopped retained runtime cannot satisfy that check or compete as an active canonical endpoint. A retry resumes the same operation at every Docker, health and journal boundary. Prior active targets remain stopped and are never restarted or removed by this flow: their shared stage may already contain partial mutations from the failed release, so restarting old code requires a separately qualified recovery contract. The conversion remains active; rerun --approve with the same checkpoint to finish ordinary credential reconciliation. Active-target replacement preserves valid receipts and operation identities, including the case where every receipt was already journaled before the prior process stopped. Credential reconciliation resumes those receipts only from a V4 journal that already contains the pre-rotation template snapshot. A legacy V1, V2, or V3 active journal with any receipt and no snapshot fails closed; a zero-receipt active journal may capture and promote, while legacy credentials-reconciled and complete journals replay unchanged.

sudo onebox systemd stop
sudo onebox migration corestore-engine --approve \
  --checkpoint-reference <verified-checkpoint-reference>
sudo onebox migration corestore-engine --status
# Only when replacing an already activated target with a corrected released pin:
sudo onebox migration corestore-engine --replace-active-target-image \
  --checkpoint-reference <verified-checkpoint-reference> \
  --expected-prior-image <repository@sha256:digest> \
  --expected-target-container <64-hex-id>
sudo onebox migration corestore-engine --approve \
  --checkpoint-reference <verified-checkpoint-reference>
sudo onebox migration corestore-engine --stopped-image \
  <service> <repository@sha256:digest>
sudo onebox migration corestore-engine --stopped-secret \
  <service> SERVEZONE_SECRET_KEYRING_FILE --stdin \
  < /root/converted-cloudly-keyring.single-line.json
sudo onebox migration corestore-engine --stage-approval <service> --stdin \
  < /root/approved-workloadinit-artifact.json
# Finish external consumer maintenance through that component's owning command.
sudo onebox migration corestore-engine --complete \
  --checkpoint-reference <verified-checkpoint-reference> \
  --external-acceptance-reference <protected-owner-evidence-reference>
sudo onebox systemd start

At capture, Onebox records whether a Cloudly workload exists from its owner identity and reserved secret-file mappings. If present, both root-only mappings must already be exact, and the image, keyring and WorkloadInit approval actions become mandatory in that order. Each offline command holds the coordinator lock through its owner mutation; each Cloudly evidence transition and final acceptance uses an exact SmartData compare-and-update. Each command journals its exact intent before touching its owner and then records verified, value-free completion evidence; replay accepts only the same input. A keyring replay verifies an already replaced stopped runtime without replacing it again, or resumes the owning runtime refresh if only the encrypted value was saved. The image must be a new immutable Cloudly digest, the keyring must differ from the captured value, and the accepted approval must advance beyond the captured authority. Other workloads do not acquire Cloudly-specific requirements.

The stopped-secret command requires the existing root-only Cloudly keyring file mapping and a protected single-line JSON value. It uses Onebox's encrypted service-secret owner to replace the stopped runtime, preserving its image and other secrets. Deliver the converted keyring after selecting the Cloudly 32 image and before Cloudly's owning maintenance or startup. --complete rejects missing or altered Cloudly action evidence; it also checks that every captured workload is still stopped, the new Corestore is running on the exact staged volume, the old source remains stopped and retained, every resource receipt is current, Onebox's encrypted resource credentials match Corestore's protected material, workload secrets match those credentials, and the accepted approval is bound to the current stopped runtime and image with no pending approval. It records the external owner's acceptance reference only after these Onebox-owned checks; the operator must separately verify that reference through Cloudly's owner. Completion does not start any workload or remove the rollback source. Normal Onebox startup refuses an incomplete conversion or an exact target identity mismatch. Start captured workloads only through their normal owner after application-specific maintenance and approval delivery are complete.

Moving a completed conversion to a newer Corestore release. A completed conversion keeps Corestore on the image its journal names, so a newer Corestore pin in a later Onebox release does not apply by itself. --replace-completed-target-image applies it. The digest argument must equal the running Onebox release's compiled Corestore pin; for a new attempt, --migration-id, --expected-prior-image and --expected-target-container must equal the migrationId, targetImage and targetContainerId that --status reports, while a pending attempt resumes only with its own arguments (see below). The conversion must be complete, every Onebox service must be stopped with its Swarm runtime at zero, and a platform service that consumes this Corestore must not be running. Stop external consumers through their own owners as well: Onebox fences only its own writers.

Before it journals anything, Onebox pulls the candidate by digest and reads the release version from both images' org.opencontainers.image.version and version labels. It admits only a newer release of the same Corestore major: a downgrade, a re-release of the same version, and a major change are refused, because an engine may write records only its own or a later release reads, and a major is where Corestore rewrites persisted data on first start. It also proves the running prior owner's readiness and records a digest of its value-free resource inventory together with priorConsumerServiceIds, the Corestore service ids of the Onebox platform services that inventory names as consumers; Onebox keeps no binding of its own for them. Then it journals the attempt in a record of its own, retains the prior target stopped under its deterministic recovery name, and starts an exact candidate on the same staged volume with the same administration authority. The volume is never exported again. The journal names the candidate only after the candidate answered Corestore's authenticated readiness, served exactly the prior owner's resource inventory, and became the running platform owner. The conversion journal then carries the same completed handoff an active-target replacement appends, so its phase stays complete and the released Onebox 32.4 reader still accepts it.

Any failure before that commit restores the prior target. The candidate is stopped and kept, never removed, as onebox-migration-engine-target-retired-<migration-id>-<attempt>; the prior runtime returns to its canonical name, answers readiness and serves its recorded inventory again; and the conversion journal stays byte for byte unchanged. The command then exits non-zero and names the cause, and a later run starts a new attempt. An interrupted run resumes with the same arguments and never repeats a finished step; an interrupted restore resumes as a restore. A restore that finds the prior inventory changed stays pending for review.

A failure after the commit never restores: the journal already names the candidate, and the prior release may not read what the new engine wrote. The attempt stays pending, Onebox refuses to start, and the command exits non-zero with the cause and the exact command that finishes the attempt; startup's refusal and --status print the same command. Fix the cause and run it. It carries the attempt's own arguments: --expected-prior-image and --expected-target-container name the prior target, not the candidate that --status now reports as targetImage and targetContainerId, and a command built from those is refused with the same resume command. The rerun keeps the attempt's writer fence. It first proves that every Onebox service is stopped with its Swarm runtime at zero, that no consumer the attempt recorded runs, that the journal names the candidate, that the attempt is owner-selected, that its prior target is stopped under its recovery name and that no retired candidate runs; it refuses by name while a service, a recorded consumer, the prior target or a retired candidate runs, before it starts anything. It then starts the exact candidate if it was stopped, for example by hand, and proves its readiness again. A running consumer the candidate's inventory names that the admission did not record is refused by name at this point, and the attempt stays pending until the consumer is stopped; otherwise the rerun closes the attempt without repeating a handoff step. It never restarts a running candidate, never starts the prior target and never creates a container. A journal commit that cannot be confirmed is reported the same way; its rerun finishes a commit that landed, or proves the candidate again and then commits or restores. A failed journal or record write is always named with its own error, never reported as a concurrent change.

--status lists every attempt as completedTargetReplacements and states completedTargetState: none without a record, settled when no attempt is pending, pending before the journal commit, committed after it while the record's completion is missing, and diverged when the record and the journal disagree. While the last attempt is pending or committed, completedTargetResumeCommand is the exact command that finishes it. A diverged record is shown, not refused: completedTargetDivergence names the broken rule (foreign-conversion, handoff-mismatch or target-mismatch) and shows both sides, the record's attempt with its step and journal handoff index against the journal's target, handoff count and the handoff it holds at that index. Onebox refuses to start in every state but none and settled, and while a retired candidate runs. It refuses a diverged record by the broken rule and points to --status, also while the record's last attempt is pending: a diverged record gets no resume command, because the command refuses it too.

Never roll back the Onebox binary while an attempt is pending. The retained 32.4 binary does not read the completed-target record, so it would neither refuse to start nor resume the attempt, and a release with a different Corestore pin cannot resume it either, because the command accepts only its own release's pin. Finish the attempt first with the command it names, which completes it or finishes a restore in progress, and replace the binary in either direction only while --status reports completedTargetState as none or settled.

Neither Corestore nor NoSQLDB states a storage-format version that Onebox could compare. Format compatibility therefore rests on the release rule above, on NoSQLDB refusing an unsupported root at startup before it writes, and on the readiness and inventory proofs.

sudo onebox service stop <each-running-service>
sudo onebox systemd stop
sudo onebox migration corestore-engine --status
sudo onebox migration corestore-engine --replace-completed-target-image <pinned-repository@sha256:digest> \
  --migration-id <migration-id> \
  --expected-prior-image <current-target-repository@sha256:digest> \
  --expected-target-container <current-target-64-hex-id>
sudo onebox migration corestore-engine --status
sudo onebox systemd start
sudo onebox service start <each-service>

Moving Corestore's data out of Docker

The engine conversion wrote Corestore's data root into its stage volume, onebox-corestore-engine-stage-<migration-id>, and the first maintenance container that named that volume made Docker create it on its own: a plain local volume whose data lives under /var/lib/docker/volumes. Every later mount asked for a volume bound to an Onebox directory, but Docker keeps an existing volume's options and ignores the ones a mount states, so Corestore's database and object storage stayed inside Docker's own data root, where removing Docker deletes them. migration corestore-data moves the root into the Onebox-owned volume corestore, bound to the directory the Corestore platform storage class names for it (/var/lib/onebox/corestore on a default host), which is where a fresh install keeps it.

This release creates every platform data volume explicitly and refuses an existing one whose storage differs, so on a host converted before it, Corestore cannot be recreated through the platform deploy path until the relocation has run: recreating onebox-corestore refuses the plain stage volume by name and names migration corestore-data as its remedy. Upgrading and the conversion commands are unaffected, but run the relocation soon after the upgrade.

The root is renamed, never copied. NoSQLDB binds its storage root's path and inode into its fencing profile and refuses a copy as another store, and Corestore binds its isolated-restore claims to device and inode, so only the same inodes at the same in-container path /data/corestore are the same data. The staged volume and the target directory must therefore be on one filesystem; otherwise the command refuses by name before anything moves, and the move needs a cross-filesystem relocation capability from NoSQLDB and Corestore first.

Admission requires a complete conversion with no pending completed target replacement, a running owner on the staged volume, every Onebox service stopped with its Swarm runtime at zero, every platform service that consumes this Corestore stopped, a plain local staged volume, a target directory that is absent or empty, and no volume corestore other than one bound to that directory. Stop external consumers through their own owners as well. Onebox then reads, from the running owner, Corestore's readiness, a digest of its value-free resource inventory and the content digest of every database, and journals the attempt in its own record, corestore-data-relocation. It stops the owner and keeps it as onebox-migration-corestore-data-recovery-<relocation-id>, records an inventory of the stopped root that covers every entry's inode, link count, owner, mode, size and modification time, renames the root, and requires the same inventory at the target. It creates the volume corestore explicitly with the storage class's bind, starts a candidate on it with the same image and administration authority, makes it the platform owner, and requires its readiness, the same resource inventory and the same content digest of every database. Only then does the conversion journal name the relocated target. Every step is journaled; an interrupted run resumes with the same --approve --checkpoint-reference and never renames or creates twice. Onebox refuses to start while an attempt is between admission and relocated, and names both commands that end it.

A relocated attempt starts normally. Start the services, verify them, and accept the move with --complete --external-acceptance-reference <reference>. Until then, --rollback returns everything: it stops the candidate and keeps it as onebox-migration-corestore-data-retired-<relocation-id>, gives the owner back to the prior target, removes the relocation from the conversion journal, renames the root back into the staged volume, and restarts the prior target. The root keeps its inodes both ways, so whatever the relocated Corestore wrote stays in it; a rollback before the relocated owner ran requires the captured inventory and content digests again. Stop every Onebox service before a rollback. An interrupted rollback resumes with the same command, and a later --approve starts a new attempt. An accepted relocation is not rolled back, and a completed target replacement runs only while no relocation is pending or awaiting acceptance; afterwards its candidates run on corestore.

The relocation removes nothing. The prior target stays stopped on the emptied staged volume, and the export volume, the retained pre-conversion source and the conversion's other recovery containers stay as they were; migration storage-leftovers retires them once the relocation is accepted (see below). Once a relocation is committed, the conversion journal names it, and an Onebox release before 32.7.0 refuses to start against that journal.

sudo onebox service stop <each-running-service>
sudo onebox systemd stop
sudo onebox migration corestore-data --approve \
  --checkpoint-reference <verified-checkpoint-reference>
sudo onebox migration corestore-data --status
sudo onebox systemd start
sudo onebox service start <each-service>
# After verifying the services:
sudo onebox systemd stop
sudo onebox migration corestore-data --complete \
  --external-acceptance-reference <acceptance-reference>
sudo onebox systemd start

Retiring the storage leftovers

After an accepted relocation, the Corestore conversions have left behind data that nothing reads any more: the stopped recovery containers (the relocation's prior target, every prior target of an active or completed target replacement, and the original SmartDB 5 source), the emptied stage volume and the empty directory a storage-class resolution once created for it, the export volume with the conversion's source export, the pre-conversion smartstorage volume and its directory, the legacy mongodb volume and its directory, and the MinIO directory /var/lib/onebox/minio that the 4.2.0 decommission kept on disk. They are what stands between the host and removing Docker. onebox migration storage-leftovers --approve --checkpoint-reference <reference> retires exactly these, and nothing an operator removes by hand.

Each artifact is named by what a journal recorded for it, never by a name pattern alone: the containers by the ids and recovery names the conversion journal holds, the stage, export and source volumes by the names it holds, the mongodb volume only when the Corestore platform cutover record names a legacy MongoDB owner (by the name the removed provider used), and the MinIO directory only where the platform storage class places minio and only when it holds MinIO's .minio.sys/format.json. The command refuses unless the conversion is complete, the relocation is accepted and settled with the journal, no completed target replacement is pending, the platform cutover (when the host has one) is complete, and Corestore's owner runs on the relocated volume corestore. At approval it records every artifact in its own record, storage-leftover-retirement: each container's identity, each volume's storage, and each directory's inventory of every entry's inode, links, owner, mode, size and modification time. It refuses a volume that a container it does not retire still uses, data in the stage volume or the stage directory, and any path that overlaps Corestore's own data. Every read that can fail for reasons of the host runs before the record is written: these preconditions, the inventory and hashing of every artifact, and, when anything is to be archived, the backup repository's listing under the new retirement's tags. A refusal there, such as a repository that cannot be listed, leaves no record, and Onebox starts as before.

The export volume, the mongodb data and the MinIO directory are archived first, into Onebox's backup repository, as the deterministic bundle every backup's platform data and volumes use: every file's bytes and the directory tree, without owners, times or permission bits other than the executable bit. At approval the command hashes every file of each of them; after the archive it reads the bundle back from the repository and requires the same entries, sizes and SHA-256 digests, and the same directory inventory as at approval, before it records the snapshot id. Each archive is pinned in the repository when it is stored, with the reason Onebox storage-leftover retirement <retirement id> keeps the only copy of <artifact id>, and the read-back also requires that pin. An archive that fails this read-back is unpinned and removed from the repository again, and a rerun first unpins and removes every other archive stored under the artifact's retirement tags that is not the one it adopts, and pins the one it adopts again if its pin is gone, so the repository keeps exactly the archives the record names. Nothing is removed before every archive is proven. The MinIO data stays in MinIO's own on-disk format: nothing reads its objects, and whoever needs one restores the directory and reads it with the last community MinIO release offline. These snapshots belong to no service backup and no backup row names them; the retirement record holds their ids. Because they are pinned, retention never removes them and takes no account of them: a retention prune keeps each with the reason pinned, the daily prune applies no retention policy and removes no snapshot, and neither a schedule's retention nor a snapshot deletion removes one. Onebox never unpins an archive the retirement record names; one leaves the repository only when an operator unpins it with unpinSnapshot from @serve.zone/containerarchive.

The smartstorage source is removed without an archive. The conversion copied it byte for byte into the stage and proved the copy, the accepted, relocated Corestore data now holds what it held and more, and the Cloudly backups taken before and after the conversion are the checkpoints; a second copy would also double the largest dataset on the host.

Then the command removes, in order, the recovery containers (each only while it is stopped and still the recorded container, and never forced), the volumes (never forced, so Docker itself refuses one a container still uses), and the directories (each only while its inventory is the one captured at approval; the stage directory only while empty). Every step is journaled before it acts, and an interrupted run resumes with the same --approve --checkpoint-reference: an artifact whose removal was recorded and which is gone counts as removed, and an archive already stored under this retirement is adopted by the same read-back proof instead of stored twice. A resumed removal proves its artifact again before it acts: a volume still held by Docker must be the local volume of the recorded kind at the recorded path with no container mounting it, and what is left of a directory must still be the captured directory, by device and inode, with no other filesystem mounted anywhere inside it, because a recursive removal would descend into a mount. Onebox refuses to start while a retirement is in progress, and names both ways out: the resume with the same --approve --checkpoint-reference, and --abandon while nothing has changed. --status prints the value-free record without initializing Onebox, with each archive's snapshot id, and --restore-archive <artifact id> --to <absolute empty directory> restores one archived artifact, for example directory:minio, directory:mongodb or volume:export.

A retirement whose record was written but which has archived and removed nothing, because every artifact in its record is still present, is abandoned with onebox migration storage-leftovers --abandon. It removes the storage-leftover-retirement record, only while the record is exactly the one it checked, after which Onebox starts again and a later --approve captures anew under any checkpoint reference. The data version is not rewound; the 0.22.0 fence step accepts a host without a record. Because an archive is stored before the record names it, the command also lists the backup repository under the retirement's tags: an unpinned attempt that an interrupted cleanup left behind is removed by its exact tags, and a pinned archive refuses the abandonment, since the resume adopts it. Once an artifact is archived, removing or removed, or the retirement is complete, --abandon refuses by name and the resume is the only way on.

There is no rollback. A removed container, volume or directory is gone; only the archived three can be restored, as files. Once a retirement has removed a recovery container, the recovery-history check at startup and in --replace-completed-target-image expects no container the record names as removed, and --replace-completed-target-image keeps working: its new prior target is retained as before. Because Onebox releases up to 32.7.0 expect those containers at startup, this release moves the data version to 0.22.0 (storage-leftover-retirement-fence), and those releases refuse its database by name with TARGET_NOT_REACHABLE instead of failing on a missing container; take the usual database copy before upgrading if a downgrade must stay possible.

sudo onebox systemd stop
sudo onebox migration storage-leftovers --approve \
  --checkpoint-reference <verified-checkpoint-reference>
sudo onebox migration storage-leftovers --status
sudo onebox systemd start

# Only while the retirement has archived and removed nothing: drop its record instead.
sudo onebox migration storage-leftovers --abandon

Startup migrations run through @push.rocks/smartmigration. Each is a step from one data version to the next, evaluated on boot; several run in sequence when an install fast-forwards across releases, and a boot already at the target costs one ledger read. The data version is its own series (0.0.0 upward), not the app version, because it describes how persisted data evolves rather than what was released. Adding a migration therefore means adding a step and moving the oneboxDataVersion constant to that step's toVersion; a release that changes no data changes neither.

The chain is matched strictly rather than bridged to the app version, deliberately. Bridging would stamp the ledger at the app version, and every later step in the 0.x series would then sit below the ledger where the planner can never select it — the migration would silently stop running on upgraded installs while still running on fresh ones.

The chain has a floor at data version 0.14.0, which is where every released Onebox up to 8.6.0 left it. The steps below it are deleted, so a ledger below the floor is refused by name with LEDGER_BELOW_OLDEST_SUPPORTED_VERSION on the first ledger read — before the migration lock and before anything is written. Such an install must first run 8.6.0, which still carries those steps, and only then this release. A database with no user collection at all is a fresh install and is stamped straight at the chain target without running a step. A populated database that carries no ledger at all is bootstrapped at the floor and migrated from there — the same assumption Cloudly's floor carries — which is why the 8.6.0 stop matters: it is the release that leaves the ledger where this one expects it.

The chain above the floor is:

From → to	Step	What it does
`0.14.0` → `0.15.0`	`cutover-archive-name`	Moves the retired cutover's provenance collection from `CloudlySecretsV2CutoverArchive` to `CloudlySecretsCutoverArchive`. The records are moved, never rewritten.
`0.15.0` → `0.16.0`	`workloadinit-approval-fields`	Removes `_schemaVersion` from both WorkloadInit approval collections. First above the floor, because they are read through exact persistence: a released document is refused before any later step can open it.
`0.16.0` → `0.17.0`	`credential-encryption-context`	Re-encrypts every stored credential under the encryption context this release derives keys with.
`0.17.0` → `0.18.0`	`service-storage-binding-fields`	Brings every persisted service storage record onto the 32 storage contract: drops `schemaVersion`, rewrites the feature ids and the object-storage file delivery format, and recomputes each binding's `requestDigest` with the contract function, so every rewritten binding carries the digest `@serve.zone/interfaces` states rather than the value a released Onebox derived.
`0.18.0` → `0.19.0`	`appstore-ingress-recipient-fields`	Removes `schemaVersion` from the stored App Store secret-ingress key pair, which interfaces 32 reads exact-key. The key pair cannot be regenerated: every secret already sealed to the published recipient would become unreadable.
`0.19.0` → `0.20.0`	`corestore-cutover-state-name`	Moves the Corestore platform cutover state from the metadata key `corestorePlatformCutoverStateV1` to `corestorePlatformCutoverState`. The record's bytes are not touched.
`0.20.0` → `0.21.0`	`workloadinit-orphaned-approvals`	Removes the WorkloadInit approval owner record, pending operation and every history event that no service owns: those of a service that no longer exists, and those under the id of an existing service that carries no approval file mapping, which deletions by earlier releases left for the next service to reuse the id. Every approval Onebox recorded belonged to a service carrying the mapping, and nothing removes the mapping from a service, so state under a service without it was inherited. Each service id's state goes through the approval owner in one transaction, which re-reads the service and refuses by name when it carries the mapping or is held by an approval operation, so a mapped service's approval state is never touched.
`0.21.0` → `0.22.0`	`storage-leftover-retirement-fence`	A version fence: releases whose chain ends at `0.21.0` expect recovery containers at startup that a storage leftover retirement removes, and they refuse a ledger at `0.22.0` by name (`TARGET_NOT_REACHABLE`) before they touch Docker. The step writes nothing; it refuses a `storage-leftover-retirement` record that does not parse as `storage-leftover-retirement-fence-blocked: retirement-record-malformed`.
`0.22.0` → `0.23.0`	`backup-archive-index-generations`	Converts the backup archive repository to the index generations of `@serve.zone/containerarchive` 0.7.0 with `ContainerArchive.migrate()`; see Backups. An install without a repository passes. A repository that cannot be inspected or converted refuses the boot as `backup-archive-migration-blocked: <stage>`, and the ContainerArchive error behind it is logged beside the refusal. Releases whose chain ends at `0.22.0`, which cannot read a converted repository, refuse a ledger at `0.23.0` by name.
`0.23.0` → `0.24.0`	`service-desired-state`	Records each service's desired lifecycle (`desiredState`) apart from its observed `status`; see Routes across stop, delete and daemon restart. A service last recorded as `stopped` or `stopping` becomes desired stopped, every other one desired running, and a service that already carries the field keeps it. A failed update refuses the boot as `service-desired-state-blocked: ServiceDoc:<id>-<stage>`. Releases up to 32.10.0 know data versions only up to `0.23.0` and refuse a ledger at `0.24.0` by name, so returning to them needs a database copy taken before the upgrade.
`0.24.0` → `0.25.0`	`storage-target-credentials`	Encrypts the SMB password a storage target held in its configuration into `passwordEncrypted` and removes it from the configuration. Rewrites every service volume on an NFS or SMB target to its storage class and `subPath: "."`, the target's root where the volume lives, removing the stored driver and mount options; no data moves. Removes any credential element left in another volume's mount options; Docker keeps the volume it created from them. A refusal is reported as `storage-target-credentials-blocked: <document reference>-<stage>`. See NFS and SMB storage targets. Releases up to 32.11.0 know data versions only up to `0.24.0` and refuse a ledger at `0.25.0` by name, so returning to them needs a database copy taken before the upgrade; without one, recovery is forward-only.

Every step is one-way. None carries a revert handler, so there is no rewind: migration rewind is gone, and forcing a step to run again is not an operator procedure. The steps above are idempotent in the sense that matters — a boot that is already at the target reads the ledger once and does nothing.

After credential-encryption-context there is no rollback to 8.x. The credential key is derived from the machine plus two domain strings, and this release derives it under onebox-credential-encryption / onebox-salt where 8.x used onebox-credential-encryption-v1 / onebox-salt-v1. The step opens every stored credential under the retired context and writes it back under the current one with a fresh nonce; an 8.x binary started on the converted database cannot read a single credential. Take a restorable copy of the SmartDB root before the upgrade — that copy, not a downgrade, is the way back.

The step walks one collection at a time in chunks of 64 documents. Each chunk's rewrites commit in one transaction, and where the walk stands is recorded in the step's own smartmigration checkpoint, which the library drops in the very write that stamps the step — so a boot that is interrupted resumes at the chunk it committed, and no progress record can survive a finished step. The checkpoint also names the chunk that was being committed when a run stopped, so a resumed run can tell a document that was already written from one that was not, and converts neither twice. A credential that does not open under the retired context refuses start-up by name — credential-encryption-context-blocked with the collection and key that failed, never a value — while the credential it names is still there untouched. Nothing is ever trial-decrypted to find out whether it was already converted. It covers the service secret bundles, registry passwords, platform service admin credentials, platform resource credentials, secret settings, and WorkloadInit approval artifacts, and it is where the last two shapes that were still kept in the clear — a registry password stored as base64 and a secret-settings row written before those rows were encrypted — are encrypted for good, so the runtime readers refuse an unmarked value by name instead of reading it as a plaintext. The stored marker on the two prefixed columns loses its version token with the context it stood for: enc:v1: becomes enc:. Backup archives are encrypted under an operator backup password instead and keep the bytes they were written with. Setting ONEBOX_ENCRYPTION_KEY pins the key explicitly, and such an install keeps the same key across the change.

migration corestore-platform --status uses SmartDB's bounded Linux read-only management sidecar while Onebox is stopped. It reads only the exact migration-state string and does not start a database listener, initialize SmartData, run storage migrations, repair storage, or write files. The packaged SmartDB Rust sidecar must be available. Output is limited to the phase, SHA-256 hashes of the checkpoint reference and captured authority, resource/progress/recovery counts, and creation/update timestamps; checkpoint references, service and resource names, owner IDs, database names, and container identities are never emitted.

Retired Cloudly Secrets v2 cutover

The Cloudly Secrets v2 transition was a one-time, manually gated App Store cutover. It completed in production, the startup migration cloudly-secrets-v2-cutover-retirement (0.11.0 → 0.12.0, now below the ledger floor) took its state off every service document, and the engine, its CLI commands and its service-document field are gone. What remains is the archive: one record per service — which service moved, the App Store version and pinned image it came from and went to, and when the cutover was opened, settled and promoted. The record has its own self-contained shape and carries no fence, evidence or lineage structure, so it stays readable without the implementation that wrote it.

The cutover-archive-name step moves that collection to CloudlySecretsCutoverArchive, a name that no longer claims to belong to a numbered secrets release, and moves the records byte for byte. It refuses untouched if a namespace under either name is not an ordinary collection, or if both names hold a collection at once — a state a rename cannot produce. Ongoing WorkloadInit approval changes use the service commands above.

Configuration Notes

Useful settings include:

Setting	Purpose
`adminUiDomain`	Exact HTTPS hostname used as the WebAuthn relying-party ID.
`serverIP`	IP address used for DNS records.
`cloudflareToken`	Cloudflare API token. `cloudflareAPIKey` is accepted as a legacy alias.
`cloudflareZoneId`	Cloudflare zone identifier.
`acmeEmail`	ACME account email for certificate issuance.
`httpPort`	OpsServer/web UI port. Defaults to `3000`.
`metricsInterval`	Metrics collection interval in milliseconds.
`backupPassword`	Secret passphrase for encrypted backup repositories.
`dcrouterGatewayUrl`	Optional external dcrouter API endpoint.
`dcrouterGatewayApiToken`	Optional external dcrouter API token.
`dcrouterGatewayClientId`	Optional stable external gateway client identity used for route ownership.
`dcrouterTargetHost`	Optional target host advertised to dcrouter.
`dcrouterTargetPort`	Optional target port advertised to dcrouter.

Example:

onebox config set serverIP 203.0.113.10
onebox config set acmeEmail ops@example.com
onebox config set cloudflareToken --stdin
onebox config set cloudflareZoneId zone-id

For the stdin form, enter or pipe one secret line and then close stdin.

External registry credentials remain encrypted in Onebox and are decrypted only for the exact Docker image-pull request that needs them. Onebox does not persist them through docker login.

NFS and SMB storage targets

A storage target has one typed configuration per kind: local and backup take an optional path, nfs takes server, exportPath and an optional version, smb takes server, share and optional username, domain and version, and custom takes a driver and a device. Every kind takes options. A member the kind does not have is refused.

The SMB password is write-only. It is not part of the configuration: it is set in the settings UI or with the setStorageTargetPassword request, stored with Onebox's credential encryption, and every response reports only passwordSet. Sending null removes it. Changing the target's server or share, or its kind, removes the stored password, so a password is only ever sent to the host it was set for.

Docker's local volume driver hands an NFS or SMB mount one comma-separated o string. The server, the username, the domain, the password and every option value are therefore refused when they contain a comma or a control character, and option keys are limited to letters, digits, dots, dashes and underscores; an equals sign inside a value is kept, because the kernel splits each element at its first one. Options may not restate what the target states (addr, nfsvers, vers, username, password, domain and their aliases). A refusal is reported as storage-target-config-invalid: <field> <reason> and never contains the value.

A service stores a volume on an NFS or SMB target as its storage class and its subPath below the target's root, never as mount options. Onebox composes the mount, and decrypts the password, only when it creates the runtime. It creates the Docker volume first and mounts it by name, so the container's and the Swarm service's specifications carry no mount options.

Docker still holds the SMB password. The local driver mounts with the mount(2) system call and never runs the mount.cifs helper, and the kernel ignores credentials=, so there is no credentials file it could read instead: the password is part of the Docker volume's options, and docker volume inspect shows it to anyone with access to the Docker socket. This is a limit of the Docker runtime and ends with it.

subPath is a relative path below the export or the share; . is the root. Volumes created before this release keep the root, which the storage-target-credentials data migration records as ., so no data moves. A new volume also takes the root unless its request names a subPath, and that directory must already exist on the export: Onebox cannot create a directory on it yet. A volume keeps its subPath across updates and refuses to move to another one. Platform data volumes (MariaDB, ClickHouse, Valkey, Corestore) have no record besides the Docker volume and always mount the target's root.

Backups and database copies taken by an earlier release contain the SMB password in the clear, in the target and in every service record that used it. Change the password on the SMB server after upgrading, and set the new one on the target.

Platform mail (CoreMail)

Onebox runs one CoreMail workload as a platform service, next to Corestore, and is that workload's control plane. Hosted apps never talk to dcrouter for mail: they submit to CoreMail, and dcrouter delivers inbound mail to it.

A service asks for mail with platformRequirements.mail (App Store template) or enableMail (direct deploy). Onebox then provisions one CoreMail binding per service, bindingId = onebox-service-<serviceId>, and injects it into the encrypted service-secret bundle:

Variable	Meaning
`MAIL_COREMAIL_URL`	`https://<coremail-hostname>/socket`
`MAIL_COREMAIL_BINDING_ID`	The binding identity, also the SMTP username
`MAIL_COREMAIL_CREDENTIAL_ID` / `_VERSION` / `_SECRET`	Rotating binding credential
`MAIL_FROM`	Default sender for the service
`SMTP_HOST` / `SMTP_PORT` / `SMTP_TLS_MODE`	`onebox-coremail`, `587`, `starttls`
`SMTP_USERNAME` / `SMTP_PASSWORD`	The binding id and the same credential secret

@serve.zone/platformclient consumes the MAIL_COREMAIL_* form; apps that only speak SMTP use the SMTP_* form against CoreMail's submission listener.

The CoreMail hostname

One hostname is CoreMail's whole public identity: it is the strict-surface name CoreMail matches in Host, the origin its transfer grants resolve against, the EHLO name of its submission listener, and the host in MAIL_COREMAIL_URL. Onebox resolves it in this order:

the coremailHostname setting, if an operator set one;
otherwise coremail.<default wildcard domain>, when Onebox owns a non-obsolete default wildcard domain;
otherwise the placeholder coremail.serve.zone.

Onebox publishes that hostname as a managed proxy route to the CoreMail container's control port (onebox-coremail:3000) and installs the certificate it already acquires for it, so MAIL_COREMAIL_URL and the transfer origin actually resolve. The route is re-published on every route reload alongside hosted-app and Admin UI routes. The transfer origin, the SMTP EHLO name and the URL handed to apps are all derived from this same published hostname, so they cannot drift apart.

In case 3 Onebox publishes no route and mail fails closed. Claiming coremail.serve.zone on a host that does not own it would point apps and CoreMail's transfer grants at someone else's name, so provisioning a binding, composing desired state and registering the dcrouter gateway peer all refuse with an error naming the missing coremailHostname setting — the same way an unreachable control plane refuses a new binding. The CoreMail platform service itself still runs, so an operator can set the hostname afterwards and the refused paths recover with no teardown. getCoreMailControlStatus reports hostnamePublished: false and omits hostname while this holds.

An operator overrides the hostname with the coremailHostname setting; changing it re-publishes the route and moves the transfer origin with it, which requires re-pushing desired state (the monitoring tick does this) and a CoreMail restart to pick up the new strict-surface name.

Onebox owns CoreMail's whole runtime contract. It resolves the image to an immutable digest before any platform state changes, provisions CoreMail's own database and bucket from Corestore under the platform-owned consumer identity onebox-platform-coremail, and supplies the replica identity CoreMail requires. The rollout generation is a monotonic counter on the platform-service record; each deployment mints a fresh task id, because a container id does not exist until after the container is created.

Credentials are minted by Onebox and only ever leave it as verifiers. CoreMail receives argon2id verifier material in COREMAIL_CONTROL_BOOTSTRAP, dcrouter receives a verifier in its gateway peer, and the matching plaintext exists only in CoreMail's own runtime environment. Onebox emits the canonical argon2id PHC form the contract fixes (m=65536,t=3,p=1), re-encoding argon2's own m,p,t parameter order and proving every parameter against coreMailCredentialVerifierPolicy.

Desired state is pushed with CoreMailControlClient from @serve.zone/api, fenced on the config epoch Onebox last proved CoreMail applied, so a concurrent controller loses instead of overwriting. It converges on app install, upgrade and removal, and on the daemon monitoring tick. Mail bindings fail closed: if CoreMail cannot be reached, or has not accepted the binding, no credential is handed to the hosted app.

When Onebox is enrolled as a dcrouter gateway client, it provisions dcrouter too: syncCoreMailGatewayPeer registers the CoreMail workload with gateway.endpointUrl set to dcrouter's wss://<ops-host>/ socket and transferOrigin set to the bare https://<coremail-hostname> origin Onebox's own proxy terminates, and each app address is bound with syncMailAddressBinding targeting { type: 'coreMail' }. Without an enrolled gateway there is no desired state and mail stays unavailable.

Binding credentials rotate without an outage. rotateCoreMailBindingCredential mints the next version as current and demotes the one it replaces to retiring with a 15-minute acceptUntil, pushes desired state, then re-injects the new secret into the app's environment; a workload still holding the previous secret keeps authenticating until the window closes, after which the retired entry is dropped. Setting coremailCredentialMaxAgeDays also rotates any binding older than that age on the monitoring tick.

getCoreMailControlStatus reports the applied epoch and last error; getCoreMailServiceMailStatistics reads CoreMail's own per-service counters straight through Onebox's control session, so an operator client never needs CoreMail credentials. Persisted control errors are truncated and scrubbed of bearer tokens and long opaque strings before an operator can read them.

SMTP listener TLS

CoreMail's submission listener uses the certificate Onebox already manages for the CoreMail hostname. Onebox delivers the PEMs as runtime secret environment, which the desired state references by key name only (COREMAIL_SMTP_CERTIFICATE_PEM, COREMAIL_SMTP_PRIVATE_KEY_PEM) and never by value.

v1 behaviour: because that material is delivered as container environment, a renewed certificate only reaches CoreMail through a restart. The monitoring tick fingerprints the current certificate and restarts the CoreMail platform service when it changes, through an owner-internal restart that bypasses the operator stop guard — that guard refuses a stop while bindings exist, which is exactly when a renewal matters. Submission is therefore briefly unavailable at renewal. If the listener has no resolvable TLS material it stays disabled rather than accepting unprotected submissions.

App Store

The App Store manager fetches metadata from serve.zone/appstore through @serve.zone/appstore and caches it briefly. Templates can declare public, secret, and generated environment inputs, container arguments, file-mounted secrets, and platform requirements. Installing an app can automatically provision Corestore database/object-storage, ClickHouse, Valkey, or MariaDB resources, and a CoreMail mail binding (platformRequirements.mail).

Service secrets and provisioned platform credentials are encrypted at rest. The browser seals each declared secret to Onebox's active X25519 recipient with install-specific authenticated context before the TypedRequest call. The root CLI accepts secrets only through bounded stdin over its mode-0600 local control socket; the daemon seals them before constructing the install request. Generated values are created inside Onebox. Plaintext install secrets never enter the public environment, browser request cache, process arguments, or persisted App Store metadata. Environment secrets work with standalone Docker containers and Swarm services. File-delivered secrets stay out of the environment and are mounted through Docker secrets, so Onebox rejects them when the host is not using Swarm mode.

Hosted apps authenticate to Onebox with signed, audience-scoped RS256 machine JWTs stored in the encrypted service-secret bundle. Onebox verifies both the signature and the exact currently active token, rotates legacy random tokens and near-expiry JWTs, and reconciles changed identities through the normal service update path. Bootstrap actions use typed message/setup-route variants and revision-fenced atomic transitions; concurrent lifecycle changes fail closed.

Named storage contracts

App Store versions can declare template-local storageClasses and stable storageRequests for filesystem and objectStorage resources. These declarations contain portable requirements only. Provider names, host paths, network shares, mount options, and authentication details remain Onebox operator configuration and never enter an app manifest.

Onebox normalizes every install through @serve.zone/appstore, negotiates the declared storage feature IDs (storage.bindings, storage.filesystem, storage.object-storage, storage.object-storage.file), selects a compatible operator class deterministically, and persists the canonical resolved binding on the service. Filesystem requests become service-owned Docker volumes. Each object-storage request receives its own bucket, endpoint, and scoped credential material through Corestore's /control/storage/bindings/* API; multiple named bindings use disjoint environment keys and encrypted service-secret storage. Every binding carries the request digest @serve.zone/interfaces states — bare hex SHA-256 over the normalized request — derived by Onebox and by Corestore with that one contract function, so the binding a control plane issues and the request Onebox declared name one value.

Legacy volumes and platformRequirements.s3: true declarations are normalized for new installs. Existing pre-binding services remain on their legacy resources during compatible upgrades. Adding or changing an explicit named request requires a storage migration instead of silently replacing data.

The current fulfillment adapter advertises only local/backup filesystem classes and the standard managed object-storage policy with environment delivery. Capacity requests, secret-file object delivery, hard quotas, snapshots, versioning, non-local filesystem targets, and Kerberos-authenticated NFS remain unadvertised and fail closed. Onebox does not fall back to local storage when a requested capability or mapped class is unavailable. Reclaim policy is honored as retain or delete during rollback and service removal.

Backups export and restore opted-in filesystem volume data by mount path while preserving the portable storage specification, together with supported legacy platform resources. Named object-storage bindings and requests remain unsupported; backup creation fails closed when either is present.

onebox appstore list
onebox appstore config cloudly
onebox appstore install cloudly --name cloudly --domain cloudly.example.com

Backups

Backups are built around @serve.zone/containerarchive. Onebox exports service configuration, encrypted service-secret ciphertext, platform resource metadata, supported platform data, and optionally Docker images into a content-addressed archive repository. Platform data is streamed as a fixed, bounded set of deterministic logical tar items rather than one archive item per file. Restore derives large-item limits from authenticated snapshot metadata and validates tar paths, entry types, sizes, and symlink boundaries before importing data. Backup creation streams a backup from its sources into the archive instead of copying it to disk first: each volume is read with docker cp <mount>/. - and its tar rewritten into service-volumes.tar (a hard link is held as a file of its own), the image with docker save, a Corestore database as its closure stream, an S3 bucket object by object after its whole listing, and Valkey data from memory. Only MariaDB and ClickHouse dumps are staged, in a private directory the backup owns — created with mkdtemp, mode 0700, a name nothing can predict — and removed after the snapshot is recorded as well as after every failure; a backup without such a database stages nothing.

Onebox reaches the Docker daemon through its API client, except for these backup data paths: volumes are read with docker cp and written back with it on restore, and the image is carried with docker save and docker load. A host that backs up or restores a service with volumes or an included image therefore needs the Docker CLI beside the Docker Engine; onebox systemd enable installs Docker Engine with its CLI when docker is missing. Without an executable docker on the Onebox process's PATH, such a backup is refused by name as docker-cli-missing before it reads anything, and such a restore before it stages anything; a backup of a service with neither volumes nor an included image needs no CLI. Every CLI step passes --host with the socket Onebox's Docker API client connects to (/var/run/docker.sock), so DOCKER_HOST and Docker CLI contexts never point a backup or restore at another daemon.

Every backup, import, restore and download leaves a reserve free on the backup repository's filesystem and on the filesystem it stages in (a restore's writes into the service's volumes, databases and image are not measured): 5 % of the filesystem, and never less than 4 GiB. A backup or an import is refused before it reads anything when the backup repository's filesystem already holds no more than its reserve, and every write of its ingest keeps that reserve: ContainerArchive measures the repository's filesystem before each pack and before the snapshot manifest and stops the ingest at the reserve, leaving no snapshot, no pin and no backup row (packs a stopped ingest had already indexed are freed by the next prune). Backups, imports, restores and downloads that run at the same time share the staging filesystem: each holds a reservation in one ledger per Onebox, grouped by the filesystem's device, from the moment it creates its private directory until it has written everything it stages, and releases it however it ends — succeeded, failed or refused. An operation announces the bytes it is about to stage and is admitted only when the staging filesystem, measured at that moment, holds them above its reserve and above what the other operations still hold there; the filesystem is measured again before every write of staged data, so neither a concurrent backup operation nor another writer on the host takes a staged write past the reserve. A MariaDB or ClickHouse dump announces its estimate — the data length of the database's MariaDB tables, the uncompressed bytes of its ClickHouse tables — and is refused before it starts when that does not fit, and stopped when a write of the dump would cross the reserve; the MariaDB data length comes from InnoDB's statistics, which can lag behind recent writes, so for a freshly filled table the stop while writing is what holds the reserve. An import announces the downloaded file's size and is refused before it decrypts, a restore announces the backup's data items and is refused before it restores its first item, and a download announces those items twice and is refused before it restores its first item; each is stopped as well when one of its writes would cross the reserve. Every refusal is named backup-insufficient-space and states the filesystem (repository or staging), the path measured, the operation, the bytes it needs, the bytes available, the bytes other backup operations hold on that filesystem, and the reserve; a refused operation leaves no snapshot, backup row or staged file behind (packs a stopped ingest had already indexed are freed by the next prune). A scheduled run stops once, with the status refused, when the repository is at its reserve, and skips a service whose dumps the staging filesystem cannot hold, by name, while it backs up the others. A run ends refused when nothing failed but something was refused, and failed when something failed. Nothing is skipped silently, and a backup never fills the filesystem the host runs on.

The archive repository is opened at start-up and re-opened on demand, and an install whose archive does not open keeps serving everything that is not a backup. It is the one thing Onebox never works around: creation, restore, deletion, download, snapshot verification, and the scheduler's run and its archive prune all refuse by name as backup-archive-unavailable with the cause the archive failed for, the download route answers 503 for it, a scheduled run is skipped once under that name instead of failing service by service, and a deletion that cannot remove snapshot data leaves the backup row that points at it, marked and hidden, for the archive to finish once it opens.

S3-compatible resources store object bodies under SHA-256 identifiers and retain exact object keys in a validated versioned manifest, so an object key is never interpreted as a filesystem path. Host-side export of a Corestore-owned bucket requires the canonical internal endpoint http://onebox-corestore:9000, the exact onebox-corestore container identity, and its loopback-published S3 port. Stale onebox-smartstorage credentials, malformed Corestore endpoints, and missing host publications fail closed and require the explicit Corestore migration replay above; Onebox does not add a host alias or persist a loopback endpoint into workload credentials.

The repository uses the index generations of @serve.zone/containerarchive 0.7.0. A repository written by an earlier Onebox is in ContainerArchive's index/ layout, which 0.7.0 refuses to open, so the data migration backup-archive-index-generations (0.22.0 → 0.23.0) converts it at the first start of this release, before the archive is opened, and logs the result: Backup archive <path> migrated from the legacyIndex format to index generations: <n> indexed chunk(s) from <m> index segment(s). The conversion needs no backup password, holds the repository lock, completes a conversion a crash interrupted when the next boot runs it again, and leaves a converted repository unchanged. It rewrites only the index; snapshots, pins and packs stay as they are. A repository in the old layout that the step did not convert, such as one put back by hand beside a database at 0.23.0, is refused at open as backup-archive-migration-required, and Onebox never initializes a new repository over it. A non-empty repository path that ContainerArchive can neither open nor inspect is refused as backup-archive-unreadable, naming both errors; an absent path or an empty directory means no repository, which the step passes and Onebox initializes. To confirm the conversion, check the log line, the ledger at 0.23.0, and a restore or download of one backup.

Recovery after the conversion is forward only. ContainerArchive 0.6.x, and with it every Onebox release up to 32.9.0, refuses a converted repository (Index directory is missing; repository repair is required), and those releases refuse the 0.23.0 ledger before they start. A downgrade therefore needs both the database copy and a copy of the backup repository taken before the upgrade; without them, fix forward.

Onebox's backup rows decide which backups exist. A deletion — from the UI, the API or a schedule's retention — first marks the row with deletionStartedAt, which hides the backup from listing, restore, download and retention, then deletes the snapshot, then deletes the row. A crash between the two deletions leaves a marked row, never a row that names a missing snapshot, and every marked row is finished when the archive opens and again before each archive prune. Deleting a snapshot does not free its storage by itself: the daily archive prune (03:00) runs prune({}, false), which applies no retention policy, keeps every snapshot and deletes only the packs that no remaining snapshot references, whole packs only, so a pack that still holds one referenced chunk stays. Onebox requires that the prune removed no snapshot and reports a result that says otherwise as backup-archive-prune-removed-snapshots. Each run logs one line with the packs removed, the bytes freed, the snapshots kept, the interrupted deletions it finished and left, and the snapshots that no backup row names and no pin holds, by id; those are kept and only reported.

MariaDB and ClickHouse databases are backed up per platform resource, as a directory named after the resource inside platform-data/<type>.tar. A MariaDB directory holds the database's mariadb-dump output as dump.sql — its tables, views, sequences, triggers, routines and events — and a manifest.json that names the source database, the MariaDB user that took the dump and, for every table the dump holds data for, the number of rows it holds, counted from the dump itself. A ClickHouse directory holds a manifest.json that records each table's name, its CREATE TABLE statement and the number of rows its data file holds, and, for the table at position n of the manifest, its rows in TabSeparatedWithNames format as table-<n>.tsv; a table whose engine a restore cannot recreate, such as a view, fails the backup, naming the table and its engine. Both stream from the database client into their files as raw bytes, within the staging reserve described above, and an export that fails leaves no file behind. A MariaDB export's first command is the size estimate, so a wrong password fails it there, before the dump starts.

A restore checks the data of every MariaDB and ClickHouse resource before it changes the service, and refuses data without a manifest, without its dump.sql, or without one of the table-<n>.tsv files its manifest lists. It then streams each MariaDB dump to one mariadb client on stdin, which stops at the first statement that fails, and drops and recreates each backed-up ClickHouse table from its statement before it streams the table's data file to clickhouse-client. Docker cannot tell a command that its stdin was cut short, so the restore counts the rows of every table it restored and compares them with the manifest: a missing table or a differing count fails it as mariadb-restore-row-count-mismatch or clickhouse-restore-row-count-mismatch, a client that ended before it read all of its input fails it as mariadb-restore-input-incomplete or clickhouse-restore-input-incomplete, and any other failed statement fails it with the client's error. A restore into the same service keeps the DEFINER of the dump's views, triggers, routines and events as dumped. An import or clone restores as the new service's MariaDB user, which is not the user the manifest records, so it removes those DEFINER clauses on the way and the new user owns every object, with its SQL SECURITY unchanged.

A database restore replaces the backed-up tables one after another; it is not atomic. When it fails midway, the tables before the failing one hold the backed-up rows, the failing table has been recreated and may hold part of its rows (the rows of every INSERT statement that ran before the failing one), and the later tables are as they were. Tables the backup does not name are never touched. The restore's own rollback covers the service's configuration and runtime, not database rows: a failed import or clone removes the service it created together with its databases, and a failed restore over an existing service returns the service to its previous configuration and image when the backup included an image, but leaves the database as the failed restore left it. Running the restore again is the recovery: the MariaDB dump drops every table, view, sequence, routine and event before it creates it, and a table's triggers go with the table, while the ClickHouse restore drops every backed-up table before it recreates it, so a second run replaces every backed-up table whatever the first one left.

MariaDB and ClickHouse data in a backup taken by 32.5.1 or earlier cannot be restored. It records no row counts, so nothing could tell a partial restore from a complete one, and a restore of such a backup is refused as mariadb-backup-without-row-counts or clickhouse-backup-without-row-counts before the service changes, unless the restore sets skipPlatformData, which leaves out all of the backup's platform data. Take fresh backups of every service with a MariaDB or ClickHouse database after upgrading.

Onebox writes and reads exactly one snapshot format: every snapshot it ingests carries the logical-items format tag, and restore refuses any other snapshot by name as unrestorable-snapshot-format before it restores a single item — an untagged snapshot written before 6.11.0 as well as a snapshot tagged by a release line this one does not read. A backup row written before the archive existed carries no snapshot, and restore refuses it by name as pre-snapshot-backup-row before it opens the archive: such a row can only be deleted, and the .tar.enc file it names is left on disk for the operator to remove. Restore migrates plaintext control and platform credentials in a backup's public environment into the current representation whenever that backup carries no encrypted service-secret bundle.

A backup download is always one encrypted file. GET /backups/<id>/download restores the snapshot's logical items into a private directory the request owns, writes them as a deterministic tar straight through an AES-256-GCM cipher into that same directory, streams the result as the response body, and removes the directory once the body has been sent, has failed, or the client has gone away. Nothing is ever served unencrypted: an install without a backupPassword secret setting is refused by name as backup-password-unset, because the file carries the service's secret-setting ciphertext and the plaintext environment values beside it. Every other cause is named as well, and the route answers the status it deserves — 404 for a backup Onebox does not hold (backup-not-found), 409 for a refusal about the stored state (pre-snapshot-backup-row, unrestorable-snapshot-format, backup-password-unset), 503 for an archive that is not open (backup-archive-unavailable), 507 for an export the staging filesystem cannot hold above its reserve (backup-insufficient-space), 500 for a server-side failure. A download refuses exactly what restore refuses, and a failed export is never reported as a missing backup.

The file is served as <service-name>-<createdAt>.tar.enc, and its bytes are, in order: a 32-byte salt, a 12-byte initialization vector, the AES-256-GCM ciphertext, and its 16-byte authentication tag. The key is the 32 bytes PBKDF2-HMAC-SHA256 derives from the backup password with that salt and 100 000 iterations. onebox backup import reads the file back on this or another host, as described below, and an operator can open it with the password and standard tools:

export ONEBOX_BACKUP_PASSWORD='the backup password'
node -e '
const { createDecipheriv, pbkdf2Sync } = require("node:crypto");
const { readFileSync, writeFileSync } = require("node:fs");
const file = readFileSync(process.argv[1]);
const key = pbkdf2Sync(process.env.ONEBOX_BACKUP_PASSWORD, file.subarray(0, 32), 100000, 32, "sha256");
const decipher = createDecipheriv("aes-256-gcm", key, file.subarray(32, 44));
decipher.setAuthTag(file.subarray(file.length - 16));
writeFileSync(process.argv[2], Buffer.concat([decipher.update(file.subarray(44, file.length - 16)), decipher.final()]));
' mail-1730000000000.tar.enc backup.tar
tar -tvf backup.tar

The tar holds the snapshot's logical items under their own names — service-config, platform-resources-meta, platform-data/<type>.tar, service-volumes.tar, docker-image — and a backup-manifest JSON entry beside them that names the snapshot format, when the backup was taken, the Onebox release that took it when the archive recorded it, the service it belonged to, and whether it includes an image. Every entry has fixed modes, zero ownership and an epoch timestamp, so the same snapshot always produces the same tar. The service-secret bundle inside it stays encrypted under the credential-encryption key described below.

onebox backup import <service-name> --file <absolute path> --password-stdin turns a download into a backup of a service installed on this host, so that it restores like a backup the service took itself. The daemon opens the file itself: the path must be absolute and name a regular file, not a symbolic link or a directory, and the command runs only through the root-only control socket, with the backup password of the host the file was downloaded from read from stdin. The import decrypts the file into a private directory it owns and removes on every path, and nothing of it leaves that directory before the AES-256-GCM authentication tag verified; a wrong password and a file altered after the download are one refusal, backup-download-unauthenticated, and nothing of such a file is kept. It then requires exactly the logical items the file's own metadata names, as a restore does, and requires the encrypted service-secret bundle to decrypt under this host's credential-encryption key: an import never re-encrypts, so the new host must run with the old host's ONEBOX_ENCRYPTION_KEY. The items are ingested as a snapshot tagged with the importing service and recorded as its backup row, which keeps the time the backup was taken. The import is idempotent by the SHA-256 of the file: importing the same file for the same service again returns the backup it created the first time. A download written by 32.5.7 or earlier carries no manifest; its image is recognized by its docker-image item, and its row records the time of the import, because nothing in the file states when it was taken. Imports and backup creation of one service run one after another.

onebox backup restore <backup-id> restores a backup in place over the service it belongs to, with the same checks and rollback as a restore from the web UI, and prints the restore's warnings. An imported backup restores into the service it was imported for even when that service has another id than the one the backup was taken of: a Corestore database is restored from the backup's closure into the importing service's own database, whichever service and whichever Corestore wrote it.

Image-inclusive backups always resolve the active Docker image ID and export by that immutable ID. A canonical digest or mutable tag is retained only while it resolves to the active image; an absent or moved mutable tag is discarded in favor of the verified image ID, while a conflicting immutable digest fails closed. Restore verifies the archived image ID before changing service state, never reassigns a mutable tag, and never replaces archived content with a registry pull.

A restore leaves the Cloudly WorkloadInit approval to its owner. The backup's encrypted environment holds the approval the service carried when the backup was taken, but a restore never writes it: under the service mutation lock that every approval operation takes, it asks the approval owner which approval the replaced service carries, and the owner answers with the approval the service holds now, proven against the owner's current decision when the owner holds one. Everything else in the backup's encrypted environment is restored, except what the host owns for the service: its hosted-app runtime identity (SERVEZONE_APP_INSTANCE_ID, SERVEZONE_APP_HOST_TYPE, SERVEZONE_RUNTIME_URL and SERVEZONE_APP_CONTROL_TOKEN) and the credentials of its platform resources. A backup may come from another host or predate a credential rotation, so an in-place restore keeps the values the service holds now, and an import or clone receives its own from its deployment. A backup taken at approval generation N therefore restores in place into a service whose approval has since moved to N+1 and keeps N+1, and no restore or restore rollback moves an approval back to an older generation. While an approval operation is pending, the owner refuses and the restore changes nothing until the operation finishes. The owner admits a first approval only through its verified stage, so a backup whose environment carries an approval cannot seed a service that holds none: an import or clone of such a backup is refused before a service is created, and so is an in-place restore into a service without an approval. The backup's reserved approval file mapping must still match the service's, as for every service update.

An in-place restore configures the service exactly as the backup records it: every configuration field the backup holds is written, and a field the backup records as absent, such as a primary domain, a registry, Onebox registry settings, platform requirements or an App Store template, is removed from the service rather than kept or emptied. When a failed restore returns an existing service to its previous configuration and image, that configuration is exactly the previous one in the same way. A restore from an archived image is the one exception: the service runs the verified archived image by its image ID or immutable digest, so its image reference, registry, image digest and Onebox registry use follow that identity rather than the recorded tag.

A backup records the service's image digest and App Store template, imageDigest, appTemplateId and appTemplateVersion, and writes null for one the service lacks. After rolling back an App Store upgrade the service therefore reports the template version it runs, and the same upgrade is accepted again; an import or clone creates its service with them. A backup written before these fields were recorded lacks their keys, which a restore tells apart from null: restoring it in place keeps the service's template as it stands, and an image restored by its tag keeps the service's digest only while its image and registry are unchanged and otherwise runs unpinned; the restore's warnings name both, as the restored service records them. Every other field has been written by every backup this release restores, so a backup without it means the service had none. A backup that records only one of the two template fields is refused before the service changes. The added keys leave the backup format and its logical-items tag unchanged: a 32.x release before this one never reads them and restores a new backup as it restores its own.

A backup archive written before this release cannot be restored by it. The Corestore database backup receipt lost its shape version and its closure format version in 32, and the snapshot format token lost its version segment (v2-logical-items became logical-items), so a pre-32 archive is refused by both readers rather than half-read. No 8.6.0 binary is retained to read them: a pre-32 archive stays exactly the artifact it is and nothing in 32 restores it, so take fresh backups immediately after the upgrade.

Encrypted service-secret bundles can be decrypted only with the same credential-encryption key. Set the same 32-byte, base64-encoded ONEBOX_ENCRYPTION_KEY on another host before restoring there; without an explicit key, Onebox derives one from the hostname and /etc/machine-id, so ciphertext is portable only to a host with the same machine identity.

Backup and schedule operations are primarily exposed through the OpsServer/web UI handlers. Root operators can also run onebox backup create <service-name>, onebox backup list [service-name], onebox backup import <service-name> --file <absolute path> --password-stdin, onebox backup restore <backup-id>, onebox backup schedules and onebox backup prune --dry-run; every one of them executes inside the sole coordinator through the root-only Unix control socket. Backup creation and import are serialized per service across CLI, UI, and scheduled callers.

onebox backup schedules lists every schedule with its scope, cron expression, enabled state, next run, last run, last status and last error. The next run is the one the scheduler has planned for the schedule's registered task, so a disabled schedule, and one the daemon does not run, shows none; the schedules API (getBackupSchedules, getBackupSchedule) reports the same nextRunAt. onebox backup prune --dry-run runs the daily prune's decisions with prune({}, true) and prints the packs it would remove, the bytes that frees and the snapshots it keeps, without changing anything. Interrupted backup deletions are not finished by a dry run, so the packs only their snapshots hold are not counted; they are listed by backup id, as are the snapshots no backup row names and no pin holds. The command refuses to run without --dry-run: the garbage collection itself stays the daily 03:00 prune.

Development

Requirements:

Node.js for the application runtime.
pnpm for package scripts.
Docker for any runtime path that initializes Onebox fully.

Common tasks:

pnpm run watch
pnpm run build
pnpm test
node ./cli.ts.js server --ephemeral --monitor

Release binaries use Deno 2.9.4 and the committed deno.lock. After changing runtime dependencies, build the entry point, regenerate the lock with pnpm exec tsdeno install --entrypoint --lockfile-only --frozen=false --lock=deno.lock binary/onebox.ts, then warm and verify the frozen runtime graph before compiling. The release workflow performs those last two checks under the same runtime-only manifest that tsdeno compile uses.

Source map:

Path	Purpose
`cli.js`	Built CLI entry point.
`cli.ts.js`	Source CLI entry point for development.
`ts/cli.ts`	CLI router and command help.
`ts/classes/onebox.ts`	Main coordinator.
`ts/classes/docker.ts`	Docker client, networks, containers, and Swarm services.
`ts/classes/coretraffic.ts`	CoreTraffic Docker service and Admin API manager.
`ts/classes/reverseproxy.ts`	CoreTraffic route and certificate bridge.
`ts/classes/platform-services/`	Local platform service providers.
`ts/classes/appstore.ts`	Remote App Store catalog and upgrade logic.
`ts/classes/workloadinit-approvals.ts`	Ongoing approval verification, runtime delivery, startup holds, and recovery.
`ts/database/workloadinit-approvals.ts`	Transactional approval owner, encrypted custody, and immutable history.
`ts_migration/credential-encryption-context.ts`	Re-encrypts every stored credential under the current encryption context.
`ts/classes/storage-manager.ts`	Portable storage capability negotiation, class selection, binding fulfillment, and lifecycle.
`ts/classes/backup-manager.ts`	Backup and restore orchestration.
`ts/opsserver/`	Web UI server and TypedRequest handlers.
`ts/database/`	SmartDB repositories.
`ts_migration/`	Versioned persisted-data and infrastructure migrations.
`ts_web/`	Dashboard source.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in license.md.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

@serve.zone/onebox