@serve.zone/cloudly

Cloudly is the serve.zone control plane: a TypeScript service and browser dashboard that stores desired infrastructure state, authenticates humans and machines, coordinates clusters, serves an OCI registry, manages workload metadata, and pushes runtime configuration to connected node components.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Cloudly is the place where serve.zone operators describe what should run. It does not directly run every workload itself. Instead, it keeps the authoritative desired state in MongoDB and exposes TypedRequest/TypedSocket APIs so runtime components can reconcile that state where the containers actually live.

The current runtime pattern is reverse-connect:

browser / CLI / SDK
  -> Cloudly HTTP + TypedSocket API
      -> MongoDB-backed managers
      -> S3-backed image and artifact storage
      <- Coreflow cluster agents connect outward
          -> Docker Swarm reconciliation
          -> Coretraffic routing updates
          -> Corestore platform resources and backups

What Cloudly Manages

Cloudly currently coordinates these areas:

• Authentication and identity: human admin login, JWT identities, machine tokens, and cluster identities.
• Clusters: desired cluster records and machine users used by Coreflow to authenticate back to Cloudly.
• Services: workload definitions, image references, domains, ports, scale factors, value-free secret configuration, service mail addresses, volumes, and deployment metadata.
• Deployments: deployment records, node placement metadata, health/resource fields, restart/scale API stubs, and DNS activation/deactivation hooks.
• Immutable deployment operations: scoped deployer grants, durable reservations, exact OCI release evidence, rollout fences, runtime digest status, route promotion, and HTTPS readiness evidence.
• Images and registries: image metadata, S3-backed image storage, external registry records, and an embedded OCI registry mounted at /v2.
• Secrets: encrypted Secret/SecretVersion material, organization SecretSets, service attachments, lifecycle state, revision fences, retention metadata, and value-free resolution previews.
• Domains, DNS, and mail: domain records, DNS entries, optional domain sync from a dcrouter external gateway, and service mail binding reconciliation.
• Platform bindings: capabilities such as database, objectstorage, pushnotification, logging, backup, and RPC-style platform services. Web Push is provider-managed; storage capabilities remain Corestore-managed.
• Backups: backup records, service backup/restore requests, scheduled backup tasks, and archive replication handshakes with Coreflow/Corestore.
• BaseOS: managed BaseOS node registration, heartbeat handling, desired-state response, image build tracking, and image download URLs.
• CoreBuild workers: selection of external build workers for BaseOS ISO and balena raw-image artifact generation.
• Tasks: TaskBuffer-backed operational tasks with execution history, metrics, logs, manual triggers, cancellation, and cron schedules.
• Node and bare-metal inventory: Hetzner-backed node creation paths and bare-metal metadata records where configured.
• Dashboard: a web component UI rendered from ts_web, including service-scoped Secrets at /secrets/secrets and organization SecretSets at /secrets/secretsets.

Runtime Components

Component	Role
Cloudly	Main service coordinator. Creates connectors and managers, then starts the API server.
CloudlyServer	TypedServer/TypedSocket HTTP server, dashboard static server, OCI registry HTTP bridge, and BaseOS HTTP endpoints.
MongodbConnector	SmartData persistence layer for Cloudly records.
CloudflareConnector	Optional Cloudflare account used by ACME DNS-01 when the encrypted CLOUDFLARE_TOKEN system secret is configured.
LetsencryptConnector	SmartACME certificate issuance and certificate lookup.
CloudlyCoreflowManager	Authenticates Coreflow, returns cluster config payloads, and pushes config updates to connected Coreflow clients.
CloudlyJumpManager	Creates short-lived Jump Codes for onboarding existing systems into clusters.
CloudlyRegistryManager	Embedded OCI registry backed by configured S3 storage, including exact-tag release evidence and immutable rollout promotion.
DeploymentOperationManager	Durable existing-service and greenfield deployment operations, resource/route claims, retry/cleanup, rollout status, and public HTTPS verification.
CloudlyBaseOsManager	BaseOS registration, heartbeat, image build orchestration, worker selection, and artifact downloads.
CloudlyBackupManager	Service backup/restore orchestration and remote archive object replication.
CloudlyTaskManager	Registers predefined and runtime tasks, tracks task executions, schedules cron jobs, and exposes task APIs.
CloudlySettingsManager	Stores an allowlisted public settings DTO in MongoDB and refreshes gateway/Coreflow state after relevant changes. Credential material is not part of the settings response.

Configuration

Cloudly uses @push.rocks/smartconfig AppData with environment mappings. The runtime entry point loads .nogit/environment values through @push.rocks/qenv, and embedded callers can override public ICloudlyConfig values by constructing new Cloudly(config) programmatically. The secret keyring and WorkloadInit approval paths are deployment-local and must come from their environment variables or the AppData store; neither is part of the public constructor DTO.

Required runtime configuration:

Variable	Purpose
SERVEZONE_ENVIRONMENT	ACME/runtime environment, currently production or integration.
SERVEZONE_PUBLIC_ORIGIN	Canonical externally reachable Cloudly origin, including protocol and optional public port.
SERVEZONE_LISTENPORT	Internal port the Cloudly TypedServer binds to. Defaults should use 3000 for containerized deployments.
SERVEZONE_SSLMODE	none, external, or letsencrypt.
SERVEZONE_ADMINACCOUNT	First-run admin bootstrap in username:password format.
SERVEZONE_SECRET_KEYRING_PATH	Path to the mounted Secrets v2 keyring. Startup fails before secret model and manager initialization when it is absent or invalid; earlier non-secret startup migrations may already have run.
MONGODB_URL	MongoDB connection URL used by SmartData.
MONGODB_NAME	MongoDB database name.
MONGODB_USER	MongoDB username.
MONGODB_PASS	MongoDB password.
S3_ENDPOINT	S3-compatible endpoint for registry, images, and artifacts.
S3_ACCESSKEY	S3 access key.
S3_SECRETKEY	S3 secret key.
S3_BUCKET	S3 bucket name.
S3_PORT	S3 endpoint port.
S3_USESSL	Boolean SSL flag for the S3 endpoint.

Common optional public settings are stored through the Cloudly settings manager:

Setting	Purpose
deploymentArchiveRetentionDays	Retention period for archived deployments.
backupExternalTarget / backupNodeCacheKeepDays	Value-free backup target descriptor and node-cache retention.
corebuildWorkerUrl / corebuildWorkerUrls	One or more CoreBuild worker HTTP(S) URLs.
dcrouterGatewayUrl	External gateway URL. The matching credential is encrypted system-secret material, not a public setting.
dcrouterGatewayClientId	Required stable Cloudly gateway client ID used for route, DNS, certificate, domain, mail, and Web Push calls; there is no cluster-ID fallback.
dcrouterTargetHost / dcrouterTargetPort	Optional target address that dcrouter should forward workload traffic to.
dcrouterTargetHostsByNode	Optional map for pinned service route targets by swarm node name.
dcrouterMailSubmissionHost / dcrouterMailSubmissionPort / dcrouterMailSubmissionTlsMode	SMTP submission routing used when Cloudly provisions service-owned mail secrets.
dcrouterMailForwardTargetHost / dcrouterMailForwardTargetHostsByNode	Gateway-reachable SMTP forward target for inbound app mail, optionally overridden per pinned node.

Credentials for these integrations are stored as encrypted system-owned Secrets v2 values under stable keys such as CLOUDFLARE_TOKEN, HETZNER_TOKEN, BACKUP_EXTERNAL_TARGET_CREDENTIALS, DCROUTER_GATEWAY_API_TOKEN, BASEOS_JOIN_TOKEN, COREBUILD_WORKER_TOKEN, and COREBUILD_WORKERS. They are not serialized into the public settings document or returned by settings APIs.

Service Web Push

Set service.data.webPush.enabled to true, or declare pushnotification in an immutable deployment's requiredCapabilities, to request browser Web Push for a service. Cloudly first verifies that its enrolled dcrouter gateway credential has the exact readWebPush and manageWebPush capabilities and that dcrouter advertises binding, delivery, cancellation, and VAPID-rotation support.

After dcrouter creates a service-owned binding, Cloudly authenticates the one-time application credential before publishing three service-owned Secrets v2 entries:

Workload variable	Purpose
WEB_PUSH_TYPED_URL	dcrouter base URL used by the typed Web Push client.
WEB_PUSH_API_CREDENTIAL_ID	Service-scoped application credential ID.
WEB_PUSH_API_CREDENTIAL_SECRET	Service-scoped application credential secret.

The dcrouter gateway control credential never enters a workload. Cloudly stores only public VAPID metadata in the platform binding; the VAPID private key remains encrypted inside dcrouter. pushnotification does not create a Corestore namespace or resource.

Provider reconciliation is service-scoped and CAS-fenced. A crash before the first workload secret publication may replace that unpublished provider generation. Once a generation is published, missing secrets or credential drift fail closed without rotating or replacing provider state. Disabling Web Push or deleting the service removes the provider binding and managed application credentials before local ownership state is removed.

Optional runtime environment variables:

Variable	Purpose
SERVEZONE_INSTALL_DEMO_DATA	Runs the destructive demo data installer when set to true.
SERVEZONE_WORKLOADINIT_APPROVAL_PATH	Private WorkloadInit release-approval artifact used for Coreflow secret-runtime registration. Cloudly can start without it; after target and recipient authority are ready, registration returns workloadinit-unconfigured until an exact active approval verifies.
CLOUDLY_BACKUP_CRON	Enables the scheduled backup-all-services task with the supplied cron expression.
CLOUDLY_BACKUP_KEEP_LAST	Number of completed/failed backups to retain per service; defaults to 24.
CLOUDLY_BASEOS_IMAGE_CLEANUP_INTERVAL_MS	BaseOS image artifact cleanup interval; defaults to 12 hours.

The cache tier always uses Cloudly's canonical runtime s3Descriptor. Configure an optional long-term tier with the value-free backupExternalTarget setting. S3 and SMB credentials must be stored only in the encrypted BACKUP_EXTERNAL_TARGET_CREDENTIALS system secret; missing, malformed, or revoked credentials invalidate the cached external writer and fail closed. NFS targets use only their value-free path descriptor and do not read a credential secret.

Starting Cloudly

Install and build with pnpm:

pnpm install
pnpm build
pnpm start

Cloudly runs its marked data migrations during startup. The deployment-claim v2 migration changes exact route owners from hostname-keyed BSON objects to sorted hostname/value entries. Stop every previous Cloudly instance before starting this version; old and new writers must not overlap during that migration.

Secrets v2 requires a separately authorized offline cutover before the first normal startup. The immutable Cloudly image contains the supported tool and dispatches it before importing or constructing the normal Cloudly runtime. Set the existing MONGODB_URL, MONGODB_NAME, MONGODB_USER, MONGODB_PASS, and SERVEZONE_ENVIRONMENT variables for database commands by default. The inspection-only --mongodb-uri-stdin mode instead rejects all four MongoDB variables and accepts one LF-terminated URI from stdin, bounded to 4096 bytes with a separate 10-second read deadline.

Every CLI invocation suppresses command-time stdout and stderr, then emits one bounded JSON object followed by a newline on stdout. Exit status 0 is used only when that object has ok: true; blockers and errors return ok: false and exit status 1. status on an uninitialized database is a successful query with state: "uninitialized" and completionVerified: false. inspect with blockers returns its bounded blocker summary with ok: false. Diagnostic output contains only finite blocker codes and reasons. The schema is identified by diagnosticsVersion: 5: blockers contains at most 64 {code, reason} samples, blockerCount contains the total occurrence count, blockersTruncated states whether samples were omitted, and sorted blockerCounts entries contain {code, reason, count} for the complete distribution. Sorted diagnosticFacets entries contain finite {name, count} aggregates for ownerless-bundle evidence predicates, recognized Corestore binding shape and known-field presence, and nonterminal deployment-operation document and distinct-identity counts. The field is always present, including as [], and contains at most 64 unique approved names sorted by name. Each count is an integer from 1 through 10,000; invalid or excessive facet evidence fails with the fixed DIAGNOSTIC_FACET_LIMIT_EXCEEDED code instead of being truncated. It never emits source references, identifiers, secret keys or values, paths, URLs, or hashes derived from those values. Schema 5 retains schema 4's exact released Corestore null-reference and binding environment shapes, source-region evidence, unknown legacy binding fields, and ownerless-bundle evidence reasons while adding aggregate-only facets. Corestore shape classes form an exhaustive legacy-only, known-only hybrid, or unknown-key partition for every recognized candidate binding. Ownerless generated-shape facets include an evaluated denominator, and deployment facets distinguish source documents from distinct valid identities. Object-storage candidates still report that target-region preservation remains unproven. Facets are diagnostic evidence only: they do not authorize a rewrite, retirement, settlement, or any other mutation.

The offline secrets-v2 CLI and the WorkloadInit runtime approval loader accept private JSON inputs only through a lexically normalized absolute path. This applies to SERVEZONE_SECRET_KEYRING_PATH, apply --authorization, prepare --rehearsal-receipt, every settlement artifact path, and SERVEZONE_WORKLOADINIT_APPROVAL_PATH. The existing parent directory must be owned by the process user and not writable by group or other users. The input must be a regular, non-symlink file owned by the process user, have no group or other permissions, have exactly one hard link, and be no larger than 64 KiB. Every settlement artifact, including the plaintext authority handoff, uses the released 768 KiB limit because operation-entry envelopes can exceed the ordinary private JSON bound. Normal Cloudly startup reads the configured keyring through the runtime loader; it does not apply these CLI-specific ownership, link-count, mode, and size checks. The WorkloadInit approval loader does apply the strict rules and the default 64 KiB limit. Keyring, authorization, rehearsal-receipt, and settlement outputs use the same atomic, no-clobber private writer with mode 0600. SERVEZONE_SECRET_CUTOVER_AUTHORIZATION_PATH is retired and ignored. The only accepted cutover authorization input is the file named explicitly by apply --authorization.

1. Stop every Cloudly instance and every process that can write the legacy SecretGroup, SecretBundle, service-secret, settings, hosted-app OIDC, or platform-binding state. Take and verify a restorable database backup.

2. Create an existing, non-symlinked output directory owned by the CLI user and not writable by group or other users (mode 0700 is recommended). Generate the deployment keyring at a new absolute path in that directory. The writer holds the verified directory descriptor, uses a random same-directory mode 0600 temporary file, fsync, and an atomic no-clobber publish. It reconciles only old temporary files from dead writers and never follows an output or temporary-file symlink. --kek-id and --recipient-key-id are optional safe identifiers.

   node cli.js secrets-v2 keyring-generate \
     --output /run/cloudly-secrets/keyring.json
   export SERVEZONE_SECRET_KEYRING_PATH=/run/cloudly-secrets/keyring.json
   node cli.js secrets-v2 keyring-check
   

3. Run the mandatory rehearsal first against a restored scratch database from the exact verified backup that will be restored again for production. Prepare later compares canonical source, topology, target-baseline, and KEK evidence; it does not compare raw database serialization byte for byte. Set RUN_ID, IMAGE_DIGEST, CONFIG_DIGEST, REHEARSAL_ID, CORESTORE_ALLOCATION_ID, CORESTORE_ALLOCATION_GENERATION, and BACKUP_PROOF_DIGEST from the approved run, exact image/configuration, scratch allocation, and verified backup. Corestore allocation ID and generation are opaque identifiers.

   node cli.js secrets-v2 prepare \
     --purpose scratch-rehearsal \
     --run-id "$RUN_ID" \
     --target-image-digest "$IMAGE_DIGEST" \
     --target-config-digest "$CONFIG_DIGEST" \
     --rehearsal-id "$REHEARSAL_ID" \
     --corestore-allocation-id "$CORESTORE_ALLOCATION_ID" \
     --corestore-allocation-generation "$CORESTORE_ALLOCATION_GENERATION" \
     --backup-proof-digest "$BACKUP_PROOF_DIGEST"
   printf '%s\n' "$MONGODB_URL" | env \
     -u MONGODB_URL -u MONGODB_NAME -u MONGODB_USER -u MONGODB_PASS \
     node cli.js secrets-v2 inspect --mongodb-uri-stdin
   node cli.js secrets-v2 authorize \
     --run-id "$RUN_ID" \
     --confirm-runtime-quiescent \
     --expires-in-seconds 900 \
     --output /run/cloudly-secrets/scratch-cutover-authorization.json
   node cli.js secrets-v2 status
   node cli.js secrets-v2 apply \
     --authorization /run/cloudly-secrets/scratch-cutover-authorization.json
   node cli.js secrets-v2 status
   node cli.js secrets-v2 rehearsal-receipt \
     --output /run/cloudly-secrets/scratch-rehearsal-receipt.json
   

   Require the scratch result to report state: "completed" and completionVerified: true. The receipt command repeats full completion and exact-index verification, then writes a private, KEK-authenticated receipt for that scratch binding and baseline. Keep the same active KEK and retain the receipt-signing KEK in the deployment keyring through production preparation, authorization, and completion, then discard the scratch database. Restore a fresh production clone from the same verified baseline; do not promote or reuse the mutated scratch database.

4. Prepare that exact stopped production clone with the private receipt. Do not repeat rehearsal or Corestore values on the command line; they are authenticated inside the receipt. Before inserting the schema-v2 production binding, prepare verifies the receipt MAC and exact Cloudly version, environment, KEK, source inventory, initial namespace topology, and empty target baseline. The binding embeds the receipt and is insert-only. Before cutover state exists, an exact prepare replay is idempotent; a changed receipt or binding fails. After cutover state exists, every prepare call fails as too late.

   node cli.js secrets-v2 prepare \
     --purpose production \
     --run-id "$RUN_ID" \
     --target-image-digest "$IMAGE_DIGEST" \
     --target-config-digest "$CONFIG_DIGEST" \
     --rehearsal-receipt /run/cloudly-secrets/scratch-rehearsal-receipt.json
   

5. Inspect the exact prepared database. Inspection proves transaction support and returns transaction support plus bounded blocker code/reason samples and complete aggregate reason counts. Source references are retained only inside the migration process for deterministic gating and are never serialized or included in error messages. Internal SecretV2CutoverError messages contain only aggregate code:reason:count entries. A standalone MongoDB topology is unsupported. Any blocker makes the result ok: false and the command exits with status 1.

   printf '%s\n' "$MONGODB_URL" | env \
     -u MONGODB_URL -u MONGODB_NAME -u MONGODB_USER -u MONGODB_PASS \
     node cli.js secrets-v2 inspect --mongodb-uri-stdin
   

6. Authorize the exact inspection for at most 900 seconds. The confirmation flag is an operator assertion that every legacy writer is quiescent. The authorization is written privately and is never printed.

   node cli.js secrets-v2 authorize \
     --run-id "$RUN_ID" \
     --confirm-runtime-quiescent \
     --expires-in-seconds 900 \
     --output /run/cloudly-secrets/cutover-authorization.json
   

7. Check the safe state summary, then apply the signed authorization. Before preparation, status reports state: "uninitialized"; after preparation it reports state: "prepared". Both states report completionVerified: false. apply deliberately has no run-ID flag. It validates the signature, proves transactions, initializes indexes for exactly the five cutover target models, rechecks authorization expiry and the empty baseline, then enters the existing transactional migration. It requires exact target topology again after the migration before returning ok: true.

   node cli.js secrets-v2 status
   node cli.js secrets-v2 apply \
     --authorization /run/cloudly-secrets/cutover-authorization.json
   node cli.js secrets-v2 status
   

8. Only after status reports both state: "completed" and completionVerified: true, start Cloudly with SERVEZONE_SECRET_KEYRING_PATH mounted. A completed startup does not need a cutover authorization, but it re-verifies the persisted completion receipt, target integrity, and the standalone execution-binding record against the binding embedded in cutover state and completion. Completed verification also requires a production-purpose binding, its authenticated rehearsal receipt, the rehearsed baseline evidence embedded in cutover state, and the exact model-owned index set on all five target collections. A completed scratch database cannot start normal Cloudly.

   To rotate the KEK after production completion, add the new active KEK while retaining the rehearsal/completion signing KEK. Offline status verifies the completed database and rotates only its completion receipt; it does not rewrap encrypted target materials. Do not remove an old KEK until every material using it has been rewrapped through a supported full-material process and a subsequent status succeeds. This CLI does not provide that rewrap command. Normal Cloudly startup is read-only and deliberately does not rotate the completion receipt.

Normal Cloudly startup never consumes a cutover authorization and never creates Secrets v2 target schema, freezes legacy collections, backfills data, or advances cutover state. Missing or non-completed state fails before secret model initialization. Only the offline secrets-v2 apply command owns those mutations.

Authorization is bound to the immutable execution binding, migration key, run ID, exact Cloudly version and environment, KEK, source inventory, plan, namespace topology, target baseline, prior state, and expiry. Empty absent target collections, partially initialized ordinary target collections containing only canonical indexes, and fully initialized empty target collections intentionally hash identically, so a crash during schema-only initialization can be authenticated and retried. Apply requires the exact canonical indexes after initialization. Views, special namespaces, unexpected or changed index definitions, restrictive index options, and any target documents still block.

For an interrupted run, begin with status. If it reports completed, do not reauthorize; require completionVerified: true and continue with completed-state verification. If it reports prepared, no migration lease exists: rerun inspect, authorize with the bound --run-id, and apply. If it reports inventory, backfill, or verify, wait for the persisted 30-second migration lease to expire after the interrupted process exits, then rerun inspect, authorize with the exact same --run-id, and apply with the new file. A premature retry returns top-level code: "CUTOVER_BLOCKED" with LEASE_HELD in blockers[].code. The tool derives expectedPriorState: 'in-progress' and the persisted run ID; a mismatch is rejected. Do not put key material, database credentials, the authorization payload, or the rehearsal receipt on the command line or in logs.

The migration first freezes the legacy namespaces, then converts and verifies Secrets v2 documents, service/settings references, and encrypted material. Only after integrity verification does it delete the frozen SecretGroup and SecretBundle documents and scrub legacy fields and bearer authorizations. status.oldImageRollbackFence becomes true as soon as cutover state exists. From that point an old image is forbidden: after namespace freezing it can read the legacy names as empty views instead of failing loudly. An in-progress run may only resume with the same run ID or use a full offline restore. Completed state reports full-restore-only; downgrade and partial collection restore are unsupported.

Legacy Deployment Settlement

The supported offline settlement commands convert only the authenticated legacy deployment-operation census that blocks Secrets v2 preparation. They are not a general database repair interface. Keep every Cloudly writer stopped, use an authority handoff issued for the exact maintenance operation, and run the scratch-rehearsal purpose against the approved restored backup before production. The production check and apply require the scratch execution bundle produced by the matching rehearsal. Both apply modes require MongoDB transaction support and are exact-replay safe.

The four settlement commands are:

Command	Purpose
settlement-plan	Read the stopped database census and write its authenticated deterministic plan.
settlement-request	Bind the authenticated plan to the approved rehearsal and production request. This command does not connect to MongoDB.
settlement-check	Validate every artifact and precondition without mutating the database, then write the authenticated check result.
settlement-apply	Apply the exact transaction or return an exact replay, then write the authenticated execution bundle and receipt.

settlement-check reports ready with disposition unapplied, exact-replay, or no-op, or blocked with disposition unapplied. settlement-apply reports completed with disposition applied, exact-replay, or no-op, or rejected with disposition unapplied and no receipt.

settlement-plan, settlement-check, and settlement-apply accept the database URI only as one LF-terminated stdin value and reject MONGODB_URL, MONGODB_NAME, MONGODB_USER, and MONGODB_PASS in that mode. All artifact paths must satisfy the private-file rules above. Timestamps are positive integer milliseconds supplied by the approved operation.

printf '%s\n' "$MONGODB_URL" | env \
  -u MONGODB_URL -u MONGODB_NAME -u MONGODB_USER -u MONGODB_PASS \
  node cli.js secrets-v2 settlement-plan \
    --mongodb-uri-stdin \
    --authority-handoff /run/cloudly-secrets/settlement-authority.json \
    --effective-at "$EFFECTIVE_AT" \
    --output /run/cloudly-secrets/settlement-plan.json

node cli.js secrets-v2 settlement-request \
  --authority-handoff /run/cloudly-secrets/settlement-authority.json \
  --plan /run/cloudly-secrets/settlement-plan.json \
  --requested-at "$REQUESTED_AT" \
  --output /run/cloudly-secrets/settlement-request.json

For --purpose scratch-rehearsal, omit --scratch-execution. Require a ready check before apply and preserve the resulting scratch execution bundle:

printf '%s\n' "$MONGODB_URL" | env \
  -u MONGODB_URL -u MONGODB_NAME -u MONGODB_USER -u MONGODB_PASS \
  node cli.js secrets-v2 settlement-check \
    --mongodb-uri-stdin \
    --purpose scratch-rehearsal \
    --authority-handoff /run/cloudly-secrets/settlement-authority.json \
    --plan /run/cloudly-secrets/settlement-plan.json \
    --request-bundle /run/cloudly-secrets/settlement-request.json \
    --output /run/cloudly-secrets/settlement-scratch-check.json

printf '%s\n' "$MONGODB_URL" | env \
  -u MONGODB_URL -u MONGODB_NAME -u MONGODB_USER -u MONGODB_PASS \
  node cli.js secrets-v2 settlement-apply \
    --mongodb-uri-stdin \
    --purpose scratch-rehearsal \
    --authority-handoff /run/cloudly-secrets/settlement-authority.json \
    --plan /run/cloudly-secrets/settlement-plan.json \
    --request-bundle /run/cloudly-secrets/settlement-request.json \
    --output /run/cloudly-secrets/settlement-scratch-execution.json

Restore the production database from the approved baseline and reuse the exact authenticated plan and request bundle proven by the rehearsal; the production check verifies that the restored census still matches their source-state HMAC. Use --purpose production with the matching --scratch-execution for both check and apply. Never generate a replacement plan after rehearsal, reuse the mutated scratch database as production, or bypass a non-ready check.

Cloudly releases publish only the semantic-version image code.foss.global/serve.zone/cloudly:<version>. The mutable latest tag is a legacy manual channel and is not changed by the release workflow.

Publishing a newer Cloudly image does not promote the repository-owned App Store channel automatically. servezone.appstore.json remains pinned to the pre-cutover 15.0.1 image digest until Onebox can provide transaction-capable MongoDB, the mandatory scratch rehearsal, and the declared secret-file inputs for the keyring, signed cutover authorization, and rehearsal receipt. Update that digest only as a separate, verified cutover-readiness change.

Run the TypeScript entry point during development:

pnpm run startTs

Start from code when embedding the control plane in another Node.js process:

The embedding process must set SERVEZONE_SECRET_KEYRING_PATH before every normal start(); it is intentionally not part of the public constructor DTO. The keyring is mandatory even when no workload secret delivery is currently expected because it authenticates Secrets v2 completion and owns encrypted system material. Set SERVEZONE_WORKLOADINIT_APPROVAL_PATH as well before enabling Coreflow secret-runtime registration.

import { Cloudly } from '@serve.zone/cloudly';

const cloudly = new Cloudly({
  environment: 'production',
  publicOrigin: 'https://cloudly.example.com',
  listenPort: '3000',
  sslMode: 'external',
  servezoneAdminaccount: 'admin:change-me',
  mongoDescriptor: {
    mongoDbUrl: process.env.MONGODB_URL,
    mongoDbName: 'cloudly',
    mongoDbUser: process.env.MONGODB_USER,
    mongoDbPass: process.env.MONGODB_PASS,
  },
  s3Descriptor: {
    endpoint: process.env.S3_ENDPOINT,
    accessKey: process.env.S3_ACCESSKEY,
    accessSecret: process.env.S3_SECRETKEY,
    bucketName: process.env.S3_BUCKET,
    port: process.env.S3_PORT,
    useSsl: true,
  },
});

await cloudly.start();

Set SERVEZONE_INSTALL_DEMO_DATA=true only when you intentionally want the demo data installer to run. The code labels that path destructive.

API Model

Cloudly exposes a single composed TypedRouter. Managers add their own typed handlers to the main router, and CloudlyServer exposes that router through the HTTP/WebSocket server.

Cloudly also exposes an admin-JWT authenticated read-only MCP endpoint at /mcp. The MCP tools return safe summaries for clusters, services, deployments, domains, and nodes without machine tokens, SSH keys, secret values, provider zone IDs, or deployment logs.

Secrets v2 Safety Foundation

Cloudly persists secret material only as encrypted SecretVersion records. A mounted keyring supplies the active KEK and X25519 ingress recipient; startup fails closed before secret model and manager initialization when that keyring cannot be validated or authenticated completion verification fails. Earlier non-secret startup migrations may already have run. Completion verification and runtime preparation use the same loaded keyring instance, whose sole ownership is transferred to the secret manager only after verification. Admin create and rotation requests accept generated material or recipient-sealed input, and API responses contain metadata only.

Service-owned secrets use { kind: 'service', serviceId }. Shared material uses organization-owned SecretSets attached through service.data.secretConfiguration.secretSetAttachments. Generic service writes cannot mutate secret configuration or deployment authority, and unknown wire fields are rejected before persistence. Even an unchanged attachment request revalidates every referenced SecretSet and fails closed when one is missing, retired, or owned by another organization. Mail and Web Push use exact management scopes and revision-fenced service ownership; service deletion revokes service-owned material without deleting organization SecretSets. Retiring a SecretSet atomically detaches every current service consumer and publishes each post-detachment deployment manifest in the same transaction; deletion remains blocked while any attachment survives.

Secret and attachment mutations publish one immutable resolved manifest for every current target cluster, preserving accepted rollout history while dropping stale cluster states. A valid service that has not received an immutable image deployment yet is an explicit zero-manifest target, so its secret configuration can be prepared before deployment. Exact mutation replay never republishes manifests, but it retries the post-commit Coreflow config push when the persisted SecretVersion says the original mutation published runtime state. Coreflow broadcast delivery is concurrency-bounded and uses native TypedSocket request deadlines; overlapping broadcast requests coalesce into one active pass and at most one trailing pass. Copied plaintext bytes are wiped before any post-commit network dispatch begins.

Cloudly exposes 14 admin-only Secrets v2 handlers for secret and SecretSet lifecycle, attachments, resolution previews, and purge preflight. getSecretIngressRecipient additionally allows an exact-peer-bound registered Coreflow Secrets v2 runtime and returns only active public recipient metadata. Physical purge execution remains disabled because no erasure worker is authorized. The control plane also implements recipient enrollment, Spark swarm observation intake, target-authority reconciliation, live Coreflow registration, resolved-manifest rollover, recipient-sealed material delivery, deployment report replay protection, and fenced Corestore credential publication. These runtime contracts are supplied by @serve.zone/interfaces 27.2.0 and @serve.zone/api 14.0.0. End-to-end workload delivery is not claimed until the matching Coreflow and Spark integrations are released, configured, and enrolled. Spark observation replay returns the persisted exact receipt before age checks. Every non-replay observation, including the first report of a replacement reporter session, must advance the node-wide persisted observedAt high-water; startup backfills that authority through a bounded, idempotent migration.

Every normal Cloudly startup requires the mounted keyring named by SERVEZONE_SECRET_KEYRING_PATH; there is no keyless mode or empty-secret exception. Startup validates the keyring and authenticated cutover completion before secret models and managers initialize, then transfers sole keyring ownership to the secret manager for runtime use and shutdown wiping.

Coreflow runtime registration additionally requires the private artifact named by SERVEZONE_WORKLOADINIT_APPROVAL_PATH. Cloudly parses an exact active or revoked schema and cryptographically verifies the detached statement and bundle, public key, registry evidence, trusted signer and revocation policy, expected version, image-index digest, and platforms through @serve.zone/workloadinit. Missing, malformed, or otherwise unverifiable approval material returns workloadinit-unconfigured; an explicit revoked artifact, revoked trusted signer, or failed revocation/timeline policy returns workloadinit-revoked. Cloudly reloads the approval while validating a live registered session. A generation, digest, or status change invalidates that session and requires Coreflow to register again.

Hosted-App OIDC

The Access view lets administrators enable or disable platform OIDC for hosted apps whose App Store version declares a valid platformOidc capability, and assign the declared app roles to Cloudly human users. Cloudly exposes the matching getHostedAppAccessConfiguration, setHostedAppRoleAssignment, and setHostedAppPlatformOidc admin requests. Browser authorization handoff uses the shared getHostedAppOidcAuthorization, completeHostedAppOidcAuthorization, and cancelHostedAppOidcAuthorization requests from @serve.zone/interfaces 27.2.0.

Cloudly owns its hosted-app signing JWK as an encrypted system Secrets v2 value. Each enabled app gets a generated service-owned client-secret record with launcher-environment delivery metadata for its declared variable. Cloudly includes that entry in resolved manifests and can seal its material to the enrolled cluster recipient; actual workload injection still depends on the downstream runtime rollout described above. Registration records and API responses contain only value-free metadata. Enable, disable, restart recovery, and service cleanup are fenced to the exact service lifecycle and Secrets v2 management scope. The app must have a canonical HTTPS domain, and template upgrades across an OIDC contract change require OIDC to be disabled first.

On first startup, Cloudly bootstraps the first human admin from SERVEZONE_ADMINACCOUNT. Human clients authenticate through adminLoginWithUsernameAndPassword; machine clients authenticate through getIdentityByToken. Cluster creation creates a machine user and token for Coreflow.

Typical consumers use @serve.zone/api:

import { CloudlyApiClient } from '@serve.zone/api';

const client = new CloudlyApiClient({
  registerAs: 'admin-tool',
  cloudlyUrl: 'https://cloudly.example.com',
});

await client.start();
const identity = await client.loginWithUsernameAndPassword('admin', 'change-me');
const clusters = await client.cluster.getClusters();

Machine clients such as Coreflow authenticate with getIdentityByToken. Auth-sensitive requests and WebSocket identity tags transport only the JWT-shaped IIdentityCredential; Cloudly verifies that credential and reconstructs the authoritative identity fields from persisted user data. The issued JWT never outlives its source token, and a cluster machine user is accepted only while it belongs to exactly one persisted cluster. The WebSocket tag lets Cloudly push configuration to already-connected Coreflow instances instead of opening inbound connections to cluster nodes.

Cluster Flow

The implemented cluster flow is intentionally simple:

1. An admin creates a Cloudly cluster record.
2. Cloudly creates a machine user with a long-lived cluster token.
3. Coreflow starts on a Docker Swarm manager node with CLOUDLY_URL and JUMPCODE.
4. Coreflow authenticates to Cloudly and requests the cluster configuration payload.
5. Cloudly returns cluster data, workload services, platform bindings, provider configs, and optional external gateway configuration.
6. Coreflow reconciles Docker networks, services, volumes, platform bindings, backups, and routing. Cloudly exposes the enrolled-recipient and sealed-material APIs for Secrets v2, but end-to-end delivery remains gated on the matching Coreflow/Spark release and enrollment.

When service, platform, or gateway settings change, Cloudly pushes updated config to connected Coreflow clients where supported.

Jump Codes for Existing Systems

Admins can generate a short-lived, single-use Jump Code for a cluster. The dashboard displays a command in this form:

curl -fsSL 'https://cloudly.example.com/jump/<code>' | sudo bash

The public /jump/<code> URL renders a browser landing page for humans and a shell bootstrap script for curl/CLI clients. The script installs the required host tooling, claims the code through POST /jump/v1/claim, receives the cluster runtime token, and starts Spark in coreflow-node mode. The long-lived cluster token is never displayed in the dashboard command.

Jump Codes expire by default after 30 minutes and are consumed on first successful claim.

Registry and Deploy-On-Push

Cloudly serves an OCI registry under /v2 through CloudlyRegistryManager. The registry uses configured S3 storage and issues OCI tokens from Cloudly authentication state.

For Cloudly-managed services, getServiceRegistryTarget() creates stable registry targets like:

<cloudly-host>/workloads/<service-name>-<service-id-prefix>:<tag>

Registry push hooks record tag/digest metadata on the linked image and service. Legacy services may still use automatic reconciliation unless deployOnPush is explicitly false. The immutable deployment API requires deployOnPush: false and never promotes latest or another mutable reference.

Registry token requests use HTTP Basic credentials against Cloudly users. User passwords and unexpired user tokens are accepted. Admins retain general write access. API machine users can obtain a write-capable registry token, but every manifest PUT is separately authorized against one active deployment operation, persisted service grant, exact repository, and exact non-latest release tag.

Safe Immutable Deployment Flow

An administrator uses configureServiceDeploymentMachineUser to attach a persisted grant to an existing service or one exact future service slot. The response confirms the grant but never returns or rotates token material. Routine deployment requests then authenticate as that API machine user; they do not require global administrator authority.

The typed workflow is:

1. getDeploymentPreflight evaluates the requested reserve, promote, or route phase from persisted permissions and current state.
2. reserveServiceDeployment durably reserves the service slot, namespace, registry repository, exact release tag, and routes. Replaying the same actor/idempotency key/request returns the original operation; changing the request fails closed.
3. For greenfield mode, Cloudly creates an immutable provisional service with deployOnPush: false, a placement hold, no public domains, a service image, and canonical empty Secrets v2 configuration. Existing-service mode verifies ownership and the current rollout-generation fence.
4. The deployer pushes a multi-platform OCI index to the returned repository and exact release tag. Cloudly records the root index digest and server-created evidence bound to the operation, actor, host, repository, tag, digest, and media type.
5. promoteServiceImageRelease accepts only that trusted release and persists a digest-pinned rollout with a fresh generation. A conflicting active rollout or stale generation is rejected.
6. getServiceDeploymentStatus reports rollout identity, counts, health, and per-task observed/reported digest evidence. Route promotion remains fenced until every required replica is healthy and verifies the exact digest.
7. promoteServiceDeploymentRoute publishes the requested route, resolves only public addresses, performs an SNI HTTPS request with certificate validation, rejects redirects, and verifies the configured readiness status. Failed route or TLS checks restore the previous route state and leave structured retryable diagnostics.

An explicit routes[].proxied boolean is persisted during route promotion and overrides prior provider metadata for the hostname. Historical operations that omit it preserve an existing proxy setting when one is available.

retryServiceDeployment and cleanupServiceDeployment are revision-fenced. Cleanup removes only operation-owned provisional resources and claims; normal service deletion also releases stable deployment claims.

BaseOS and CoreBuild

Cloudly can manage BaseOS nodes and image builds:

• BaseOS devices register through POST /baseos/v1/nodes/register and heartbeat through POST /baseos/v1/nodes/heartbeat.
• A configured encrypted BASEOS_JOIN_TOKEN system secret accepts generic device enrollment.
• BaseOS image builds create one-time provisioning tokens that are embedded in generated images.
• Cloudly selects a CoreBuild worker based on /corebuild/v1/capabilities and sends the build to /corebuild/v1/jobs/baseos-image.
• Supported build kinds are ubuntu-iso and balena-raw; Raspberry Pi builds use balena-raw.
• Supported architecture values are amd64, arm64, and rpi.
• Completed artifacts are stored in the configured S3 bucket and served through short-lived /baseos/v1/images/:buildId/download URLs.

CoreBuild worker routing uses the public corebuildWorkerUrl or corebuildWorkerUrls settings. A shared credential is stored under the encrypted COREBUILD_WORKER_TOKEN system-secret key; per-worker URL/token entries are stored as JSON under the encrypted COREBUILD_WORKERS key.

Backups and Corestore

Cloudly owns backup records and user-facing backup/restore requests. Coreflow executes the cluster-local work, and Corestore snapshots volumes, database resources, object storage resources, and archive objects.

The backup path includes:

• createServiceBackup and restoreServiceBackup typed requests for admins.
• executeServiceBackup and executeServiceRestore requests from Cloudly to Coreflow.
• Corestore volume/resource snapshot and restore endpoints behind Coreflow.
• Optional archive replication through prepareBackupReplication, uploadBackupArchiveObject, completeBackupReplication, getBackupArchiveManifest, and downloadBackupArchiveObject.
• Optional scheduled backup-all-services task when CLOUDLY_BACKUP_CRON is set.

Manual createServiceBackup requests require clusterId when Cloudly manages multiple clusters; it may be omitted only when exactly one cluster exists. Coreflow completes remote archive replication. Cloudly validates archive object size and SHA-256 checksums, writes a manifest, records target metadata, and marks completed backups as replicated. Restores read the manifest and objects back through the configured target writer.

Task Automation

Cloudly registers a TaskBuffer-backed task manager. The API and dashboard can list tasks, trigger enabled tasks manually, inspect execution logs/metrics, and request cancellation for running tasks. Disabled tasks cannot run manually or on a schedule.

Predefined tasks currently include:

Task	Status	Purpose
cloudflare-domain-sync	Enabled	Imports and updates domains from configured Cloudflare zones.
dns-sync	Disabled	Iterates DNS entries marked as external; provider sync is currently a placeholder.
cert-renewal	Disabled	Checks activated domains for certificate renewal; renewal logic is currently a placeholder.
cleanup	Disabled	Removes old task executions and contains placeholders for log/image cleanup.
health-check	Disabled	Iterates deployments and records health metrics; runtime health checks are currently placeholders.
resource-report	Disabled	Generates node resource metrics; values are currently placeholders until runtime metrics are wired in.
db-maintenance	Disabled	Maintenance shell for database optimization tasks.
security-scan	Disabled	Security scan shell for exposed ports, image freshness, and weak configuration checks.
docker-cleanup	Disabled	Docker cleanup shell for containers, images, volumes, and networks.
backup-all-services	Conditional	Registered by the backup manager and enabled only when CLOUDLY_BACKUP_CRON is set.

External Gateway Integration

Cloudly can integrate with a dcrouter gateway when the public gateway settings and encrypted gateway system secret are configured. The current integration syncs externally available domains into Cloudly, syncs public service routes directly to dcrouter, and fetches or reprovisions certificates through dcrouter. Coreflow consumes the resulting Cloudly service configuration and keeps cluster-local Coretraffic routing focused on internal workload traffic.

Service mail can also be configured on a service. Each address can enable inbound mail through a dcrouter smtpForward target to a published service port and outbound mail through a dcrouter-managed SMTP credential. Cloudly stores only public credential metadata in service.data.mail; one-time SMTP secrets are written as service-owned Secrets v2 entries with sanitized address-token variables such as MAIL_PLATFORMTEST_GATED_ONE_SMTP_* plus default SMTP_* aliases for the selected default sender.

Development

Common commands:

pnpm install
pnpm build
pnpm test
pnpm run build:docker
pnpm run release:docker
pnpm run docs

Important paths:

Path	Purpose
ts/index.ts	CLI/runtime entry point exporting runCli, Cloudly, and ICloudlyConfig.
ts/classes.cloudly.ts	Main service coordinator and startup order.
ts/classes.server.ts	API/dashboard server, registry bridge, and BaseOS HTTP routes.
ts/manager.*	Domain managers for auth, clusters, services, images, registry, platform, backups, BaseOS, and more.
ts/connector.*	External system connectors for MongoDB, Cloudflare, and Let's Encrypt.
ts_web/	Browser dashboard web components.
ts_cliclient/	Published @serve.zone/cli submodule.

Accuracy Notes

The package metadata and settings schema include fields for several cloud providers. The code paths currently exercised in this repository are Cloudflare for ACME DNS-01 and domain sync, Hetzner for selected node/bare-metal provisioning paths, S3-compatible storage, SMB/S3 backup archive targets, MongoDB/SmartData, CoreBuild, Coreflow, Corestore, and optional dcrouter integration. Several provider connection tests and predefined tasks are configuration checks or implementation shells; verify provider-specific behavior in the relevant manager before relying on it operationally.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.