@serve.zone/cloudly

Cloudly is the serve.zone control plane: a TypeScript service and browser dashboard that stores desired infrastructure state, authenticates humans and machines, coordinates clusters, serves an OCI registry, manages workload metadata, and pushes runtime configuration to connected node components.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Cloudly is the place where serve.zone operators describe what should run. It does not directly run every workload itself. Instead, it keeps the authoritative desired state in MongoDB and exposes TypedRequest/TypedSocket APIs so runtime components can reconcile that state where the containers actually live.

The current runtime pattern is reverse-connect:

browser / CLI / SDK
  -> Cloudly HTTP + TypedSocket API
      -> MongoDB-backed managers
      -> S3-backed image and artifact storage
      <- Coreflow cluster agents connect outward
          -> Docker Swarm reconciliation
          -> Coretraffic routing updates
          -> Corestore platform resources and backups

What Cloudly Manages

Cloudly currently coordinates these areas:

• Authentication and identity: human admin login, JWT identities, machine tokens, and cluster identities.
• Clusters: desired cluster records and machine users used by Coreflow to authenticate back to Cloudly.
• Services: workload definitions, image references, domains, ports, scale factors, value-free secret configuration, service mail addresses, volumes, and deployment metadata.
• Deployments: deployment records, node placement metadata, health/resource fields, restart/scale API stubs, and DNS activation/deactivation hooks.
• Immutable deployment operations: scoped deployer grants, durable reservations, exact OCI release evidence, rollout fences, runtime digest status, route promotion, and HTTPS readiness evidence.
• Images and registries: image metadata, S3-backed image storage, external registry records, and an embedded OCI registry mounted at /v2.
• Secrets: encrypted Secret/SecretVersion material, organization SecretSets, service attachments, lifecycle state, revision fences, retention metadata, and value-free resolution previews.
• Domains, DNS, and mail: domain records, DNS entries, optional domain sync from a dcrouter external gateway, and service mail binding reconciliation.
• Platform bindings: capabilities such as database, objectstorage, pushnotification, logging, backup, and RPC-style platform services. Web Push is provider-managed; storage capabilities remain Corestore-managed.
• Backups: backup records, service backup/restore requests, scheduled backup tasks, and archive replication handshakes with Coreflow/Corestore.
• BaseOS: managed BaseOS node registration, heartbeat handling, desired-state response, image build tracking, and image download URLs.
• CoreBuild workers: selection of external build workers for BaseOS ISO and balena raw-image artifact generation.
• Tasks: TaskBuffer-backed operational tasks with execution history, metrics, logs, manual triggers, cancellation, and cron schedules.
• Node and bare-metal inventory: Hetzner-backed node creation paths and bare-metal metadata records where configured.
• Dashboard: a web component UI rendered from ts_web. Secrets are service-first: a service detail carries a Secrets tab with Values, Resolved and Rollout, backed by an organization-wide secret index at /secrets/secrets and shared SecretSets at /secrets/secretsets. Values are write-only everywhere; an operator-copy value is generated in the browser and revealed once inside its unsaved dialog, because no API returns a secret value.

Runtime Components

Component	Role
Cloudly	Main service coordinator. Creates connectors and managers, then starts the API server.
CloudlyServer	TypedServer/TypedSocket HTTP server, dashboard static server, OCI registry HTTP bridge, and BaseOS HTTP endpoints.
MongodbConnector	SmartData persistence layer for Cloudly records.
CloudlyCoreflowManager	Authenticates Coreflow, returns cluster config payloads, and pushes config updates to connected Coreflow clients.
CloudlyJumpManager	Creates short-lived Jump Codes for onboarding existing systems into clusters.
CloudlyRegistryManager	Embedded OCI registry backed by configured S3 storage, including exact-tag release evidence and immutable rollout promotion.
DeploymentOperationManager	Durable existing-service and greenfield deployment operations, resource/route claims, retry/cleanup, rollout status, and public HTTPS verification.
CloudlyBaseOsManager	BaseOS registration, heartbeat, image build orchestration, worker selection, and artifact downloads.
CloudlyBackupManager	Service backup/restore orchestration and remote archive object replication.
CloudlyTaskManager	Registers predefined and runtime tasks, tracks task executions, schedules cron jobs, and exposes task APIs.
CloudlySettingsManager	Stores an allowlisted public settings DTO in MongoDB and refreshes gateway/Coreflow state after relevant changes. Credential material is not part of the settings response.
CloudlyFleetCutoverManager	Owns authenticated fleet census/scope classification, non-ready prepared volume-handoff intent, isolated replacement-ingress designation, and the node-owned Spark legacy retirement receipts.
CloudlyClusterIngressManager	Designates the ingress of a cluster born on Pallet, authenticates the elected ingress, and publishes revisioned whole-cluster public route tables with revocation-first withdrawal.

Private Node Enrollment Qualification

The backend-private NodeEnrollmentStore consumes Interfaces 28.1's shared Spark/Pallet enrollment contract. It is not registered with runtime routes yet: both node-local identity owners and the replacement installer must qualify before the old Jump claim writer and endpoint are retired in the coordinated cutover. This source capability does not mean deployed workers use Pallet.

One owned SmartData transaction consumes an unexpired Jump, creates its node and cluster snapshot, issues independent Spark and Pallet hash authorities, and writes an exact combined receipt. Existing-node adoption instead fences the current Spark generation/session, rotates Spark, requires Pallet absence, and preserves the existing node. Cloudly never generates or stores either new plaintext bearer.

Bootstrap proof authorizes only the first commit. Lost acknowledgements recover the same complete proposal with its pending Spark bearer, only while both issued credential generations/hashes remain active and the node exists. Pallet reporter reconnect may advance the session epoch without invalidating that receipt. Revocation, rotation, changed proposals, expired/consumed or duplicate-hash Jump records, and missing nodes fail closed. A private node enrollmentFence nonce ensures a real transactional write against concurrent deletion even with a frozen clock; it is not a credential, generation, or public node field.

Fleet Cutover Preparation and Cluster Ingress

Cloudly registers getFleetCutoverScope, classifyFleetCutoverScope, prepareFleetCutoverVolumeHandoffs, and prepareFleetCutoverIngress. Prepared volume handoffs are desired, non-ready filesystem intents derived from current Service volume authority and authenticated node mount observations. They do not claim provider readiness or physical adoption. The current aggregate uses immutable node-bound pages and a compare-and-swap pointer, so incomplete or superseded staging cannot become current and exact replay does not renew authority. Docker volume preparation joins each node-local physical mount to the manager's logical service mount through complete one-to-one correlation evidence. Cloudly keeps the physical mount identity in the handoff and refuses observations that omit, duplicate, add, forge, or mismatch a correlation.

Ingress preparation designates a held replacement Service with no public host ports and creates its platform-managed cluster bearer. The cluster-ingress owner then authenticates registerClusterIngress, serves getClusterRouteTable, and pushes complete revisioned public route tables. Missing current authority produces an empty withdrawal before certificate or route publication; secret material, certificates, and route tables are not persisted in its durable records. A route whose hostname the table cannot carry, such as a wildcard, whose certificate cannot be obtained, or whose source conflicts with its runtime policy or with another service's route is left out of the table while the other routes are still pushed, and is logged with its name, cluster and failure; a hostname a deployment route claim assigns is served for its owner alone, and one two services declare without such a claim for neither, while a service that cannot be a route source, such as one being deleted, leaves out only its own routes. Only route sources that cannot be read withdraw the whole table; that withdrawal, and a refresh or sweep that fails, is logged with its failure.

A cluster born on Pallet never ran Swarm, so it has no cutover scope, census or target set to rest an ingress on. designateClusterIngress (admin) designates one existing service as its ingress (IClusterIngressDesignation): the request is a compare-and-swap against the cluster's current Pallet runtime, the service's exact runtime spec revision and the current designation, and it mints the platform-managed bearer SERVEZONE_CLUSTER_INGRESS_BEARER (scope platform:cluster-ingress:<clusterId>) in the same transaction. Designating the same service again rotates the bearer under a new generation; designating another service revokes the previous one's bearer with the change. A replayed mutationId answers the recorded result, and one reused for another request is refused mutation-reused. getClusterIngressDesignation reads it. A cluster holds one kind of designation: a cluster the fleet cutover designated is refused cutover-designation-present, and prepareFleetCutoverIngress refuses a cluster designated here. The ingress may register from the nodes its service's spec places it on.

Such an ingress dials the workloads themselves rather than published host ports: a public domain routes to every ready execution at its workload lease address and the domain's target port. Each node's signed network projection grants the ingress workload a one-way flow to exactly that protocol and port of each target lease (workloadIngress, authorized by the designation), on the node of the ingress lease and on the node of the target alike, and a port is routed only once every ingress lease's grant is signed on both. A public port mapping is not an ingress route on such a cluster: it stays the workload's own publication on its node's uplink. Losing the designation, or a designation that no longer names the cluster's current runtime, withdraws every grant.

A Swarm-era Spark states what it is replacing on the observation it already posts. The optional fleetCutoverLegacyRetirement member is admitted in every observation state, because it describes the host rather than its Docker daemon, and it names no node: Cloudly binds it to the reporter it authenticated. Each accepted statement advances two node-owned records together -- the mode refusal and the configuration retirement -- under one named observation, because the Coreflow retirement evidence that cites both requires them to name the same accepted observation. A changed configuration is judged only by the contract's successor rule: one that does not advance is refused under that rule's own name and nothing is written. A restatement of the accepted pair renews nothing, and a new bundle that restates the same configuration is a new revision of both receipts, because the mode refusal now names that bundle. A bundle at or below the newest published Spark that carries no refusal at all is refused legacy-retirement-bundle-below-minimum, which is the one part of this the contract leaves to the server. Assembly joins both receipts through the contract and requires the observation they name to still be fresh, because an omitted member advances nothing and a node that stopped refusing must stop looking retired.

Every one of those answers travels on that observation's own accepted receipt, as fleetCutoverLegacyRetirement: accepted at the generation both records advanced to, restated for the replay that renewed nothing, or refused by the contract's own name and never retryable, because resending one exact statement cannot change what the records say. A refused statement leaves the observation accepted -- a node's liveness and census are true whatever its released runtime states -- and the member is present exactly when the observation stated one, which the contract binds in both directions. An exact observation replay answers the receipt that acceptance wrote rather than judging the statement a second time. Where an evidence member is refused for this body and every resend of it, the observation itself is rejected evidence-refused and nothing is recorded; a condition a later attempt can pass -- a credential this attempt could not fence, a concurrent mutation -- stays acceptance-race. Answering the member raises Cloudly's sparkSwarmNode minimum peer version to 32.18.0, because a receipt is read exact-key too and an older Spark would refuse one carrying a member it does not know.

Cloudly intentionally does not register advanceFleetCutoverPhase or getFleetCutoverIngressAuthority. Phase advancement still lacks authenticated Pallet-held, Corestore-adoption, Coreflow-retirement, legacy-ingress-absence, port-transfer, and publication-readback owner evidence. These paths remain fail-closed until those producers exist; caller-supplied proof references are not accepted as substitutes.

Run the real NoSQLDB qualification with:

pnpm exec tstest test/test.nodeenrollment.node.ts --verbose --logfile --timeout 120

Development IdP Qualification

test/test.devidp-qualification.node.ts qualifies the platform OIDC login against the real development identity provider from @idp.global/devidp instead of a fake issuer. One run proves the whole organization path end to end: the authorization request carries the organization login scopes, the seeded owner persona completes an authorization-code login headlessly, Cloudly issues its own federated identity through the login handoff, the owner picker resolves the seeded organization from live UserInfo claims, organization owner authority is proven by live introspection and denied for a foreign organization, and a persona without an organization membership is refused an identity.

The qualification is opt-in because it starts containers: it needs a reachable Docker engine, a usable kernel keyring and several minutes on a cold image. Without the variable every step is skipped with its reason, so the ordinary suite stays independent of Docker.

CLOUDLY_DEVIDP_QUALIFICATION=true pnpm exec tstest test/test.devidp-qualification.node.ts --verbose --timeout 600

Variable	Purpose
CLOUDLY_DEVIDP_QUALIFICATION	Exactly true runs the qualification. Unset skips it. Any other set value (TRUE, 1, yes) fails the run instead of skipping silently, so a mistyped opt-in cannot look like a pass.
CLOUDLY_DEVIDP_APP_IMAGE	Qualifies a different idp.global app image instead of the one @idp.global/devidp pins. A reference containing @sha256: is a pinned published image and is pulled; a reference without a digest is a locally built candidate and is never pulled.

The default path is the proven one: @idp.global/devidp 0.3.0 pins the released idp.global/app 15.0.0 image index, whose development relying-party client allows the organizations and roles scopes and seeds one organization, its owner persona membership and the app connection that maps the owner role to application roles. With the variable unset every assertion above runs against that image. CLOUDLY_DEVIDP_APP_IMAGE exists to qualify a candidate image — a locally built app or a newer published index — before it becomes the pinned default. An image without that fixture does not allow the organizations and roles scopes, so the run fails at the authorization request with an invalid_scope diagnostic naming the image instead of skipping or weakening its assertions.

The qualification owns the kernel keyring namespace cloudly-qualification and releases it through DevIdpFoundation at both ends of the run — once before it starts anything, so an interrupted earlier run cannot leave material behind, and once after every Cloudly manager, the database engine and the IdP containers have stopped — proving the namespace absent each time. One committed namespace holds ten keyring entries and about 4.7 KB of the 20,000-byte per-user quota, so a run leaves the keyring exactly as it found it.

The run trusts the development IdP's local certificate authority for the duration of the test process only and restores the previous trust store afterwards; it never installs trust into the operating system or a browser.

Configuration

Cloudly uses @push.rocks/smartconfig AppData with environment mappings and ephemeral storage. Runtime and bootstrap configuration stays in memory; startup does not create or update an AppData file. The runtime entry point loads .nogit/environment values through @push.rocks/qenv, and embedded callers can override public ICloudlyConfig values by constructing new Cloudly(config) programmatically. The secret keyring and WorkloadInit approval paths are deployment-local environment variables; neither is part of the public constructor DTO.

The names listed under required: in qenv.yml are resolved while the server entry point constructs its Qenv, before any manager runs. A missing one refuses the start with QenvMissingRequiredEnvVarsError, naming every variable that was not found in the process environment, the env file or the Docker secrets. That list is exactly the set whose absence already ends startup further in, so the refusal moves the same failure to the front and names it; a variable the server defaults or guards is deliberately not enforced. The secrets CLI (cloudly secrets …) and the backup-maintenance CLI dispatch before the server entry point and are not gated by it, and the API CLI client never enforces a list it did not write.

Refused at start when absent:

Variable	Purpose
SERVEZONE_ENVIRONMENT	ACME/runtime environment, currently production or integration.
SERVEZONE_PUBLIC_ORIGIN	Canonical externally reachable Cloudly origin, including protocol and optional public port.
SERVEZONE_SSLMODE	external (TLS is terminated in front of Cloudly, by Onebox or dcrouter) or none. Cloudly holds no certificate of its own: letsencrypt is refused at start.
SERVEZONE_SECRET_KEYRING_PATH	Path to the mounted Secrets v2 keyring. Startup fails before secret model and manager initialization when it is absent or invalid; earlier non-secret startup migrations may already have run.
MONGODB_URL	MongoDB connection URL used by SmartData.
S3_ENDPOINT	S3-compatible endpoint for registry, images, and artifacts.
S3_ACCESSKEY	S3 access key.
S3_SECRETKEY	S3 secret key.
S3_BUCKET	S3 bucket name. The registry manager refuses to start without it and the image manager writes its init object into it.

Expected from a complete deployment, but tolerated when absent:

Variable	Purpose
SERVEZONE_LISTENPORT	Internal port the Cloudly TypedServer binds to. Defaults to 3000, which the container image also sets.
SERVEZONE_ADMINACCOUNT	First-run admin bootstrap in username:password format. Without it no admin is bootstrapped.
SERVEZONE_WORKLOADINIT_APPROVAL_PATH	Path to the WorkloadInit approval file. Absent means no approval file is read.
MONGODB_NAME	MongoDB database name. Optional for SmartData, which uses the database the URL names when it is unset.
MONGODB_USER	MongoDB username. Substituted into the URL only when set.
MONGODB_PASS	MongoDB password. Substituted into the URL only when set.
S3_PORT	S3 endpoint port. Defaulted where it is read.
S3_USESSL	Boolean SSL flag for the S3 endpoint. Defaulted where it is read.

Common optional public settings are stored through the Cloudly settings manager:

Setting	Purpose
deploymentArchiveRetentionDays	Retention period for archived deployments.
backupExternalTarget / backupNodeCacheKeepDays	Value-free backup target descriptor and node-cache retention.
corebuildWorkerUrl / corebuildWorkerUrls	One or more CoreBuild worker HTTP(S) URLs.
dcrouterGatewayUrl	External gateway URL. The matching credential is encrypted system-secret material, not a public setting.
dcrouterGatewayClientId	Required stable Cloudly gateway client ID used for route, DNS, certificate, domain, mail, and Web Push calls; there is no cluster-ID fallback.
dcrouterTargetHost / dcrouterTargetPort	Optional target address that dcrouter should forward workload traffic to.
dcrouterTargetHostsByNode	Optional map for pinned service route targets by swarm node name. A service a Pallet runtime spec owns ignores both: its routes target the uplink address its spec's one pinned node reported.
dcrouterMailSubmissionHost / dcrouterMailSubmissionPort / dcrouterMailSubmissionTlsMode	SMTP submission routing used when Cloudly provisions service-owned mail secrets.
dcrouterMailForwardTargetHost / dcrouterMailForwardTargetHostsByNode	Gateway-reachable SMTP forward target for inbound app mail, optionally overridden per pinned node.

Credentials for these integrations are stored as encrypted system-owned Secrets v2 values under stable keys such as HETZNER_TOKEN, BACKUP_EXTERNAL_TARGET_CREDENTIALS, DCROUTER_GATEWAY_API_TOKEN, BASEOS_JOIN_TOKEN, COREBUILD_WORKER_TOKEN, and COREBUILD_WORKERS. They are not serialized into the public settings document or returned by settings APIs.

Service Web Push

Set service.data.webPush.enabled to true, or declare pushnotification in an immutable deployment's requiredCapabilities, to request browser Web Push for a service. Cloudly first verifies that its enrolled dcrouter gateway credential has the exact readWebPush and manageWebPush capabilities and that dcrouter advertises binding, delivery, cancellation, and VAPID-rotation support.

After dcrouter creates a service-owned binding, Cloudly authenticates the one-time application credential before publishing three service-owned Secrets v2 entries:

Workload variable	Purpose
WEB_PUSH_TYPED_URL	dcrouter base URL used by the typed Web Push client.
WEB_PUSH_API_CREDENTIAL_ID	Service-scoped application credential ID.
WEB_PUSH_API_CREDENTIAL_SECRET	Service-scoped application credential secret.

The dcrouter gateway control credential never enters a workload. Cloudly stores only public VAPID metadata in the platform binding; the VAPID private key remains encrypted inside dcrouter. pushnotification does not create a Corestore namespace or resource.

Provider reconciliation is service-scoped and CAS-fenced. A crash before the first workload secret publication may replace that unpublished provider generation. Once a generation is published, missing secrets or credential drift fail closed without rotating or replacing provider state. Disabling Web Push or deleting the service removes the provider binding and managed application credentials before local ownership state is removed.

Optional runtime environment variables:

Variable	Purpose
SERVEZONE_INSTALL_DEMO_DATA	Runs the destructive demo data installer when set to true.
SERVEZONE_WORKLOADINIT_APPROVAL_PATH	Private WorkloadInit release-approval artifact used for Coreflow secret-runtime registration. Cloudly can start without it; after target and recipient authority are ready, registration returns workloadinit-unconfigured until an exact active approval verifies.
CLOUDLY_BACKUP_CRON	Enables the scheduled backup-all-services task with the supplied cron expression.
CLOUDLY_BACKUP_KEEP_LAST	Number of completed/failed backups to retain per service; defaults to 24.
CLOUDLY_BASEOS_IMAGE_CLEANUP_INTERVAL_MS	BaseOS image artifact cleanup interval; defaults to 12 hours.

The cache tier always uses Cloudly's canonical runtime s3Descriptor. Configure an optional long-term tier with the value-free backupExternalTarget setting. S3 and SMB credentials must be stored only in the encrypted BACKUP_EXTERNAL_TARGET_CREDENTIALS system secret; missing, malformed, or revoked credentials invalidate the cached external writer and fail closed. NFS targets use only their value-free path descriptor and do not read a credential secret.

For an offline S3 backup-target credential cutover, stop and fence every Cloudly and backup writer first. The bounded Cloudly maintenance command starts the real config, database, migration, secret and settings foundation without starting HTTP, gateway enrollment or scheduled managers. Inspect its value-free revision and descriptor with node cli.js maintenance backup-external-target-status. Then provide a private JSON file containing only accessKey and accessSecret:

node cli.js maintenance backup-external-target \
  --operation-id backup-cutover-1 --created-by operator:cutover \
  --expected-secret-revision 4 \
  --expected-source-endpoint onebox-smartstorage \
  --target-endpoint onebox-corestore \
  --credentials-file /run/secrets/backup-target-next.json

The file must meet the same owner, mode and no-symlink checks as other private Cloudly inputs. Credential bytes never appear in arguments, environment values or JSON output. The exact operation ID, source endpoint and system-secret revision fence the request. Cloudly persists a prepared phase, uses the idempotent SystemSecret mutation, then updates the value-free S3 endpoint and records complete. After interruption, rerun the same request and protected credential file while writers remain fenced; a mismatched replay or descriptor drift fails closed. The output contains only the operation ID, target revision, endpoint, bucket and completion status.

CoreMail Control Plane

Cloudly is the control client for the cluster's CoreMail workload. CoreMail fronts dcrouter as its single gateway peer; hosted apps submit mail to CoreMail instead of dcrouter, and CoreMail reports per-service mail statistics back.

CoreMail is itself a Cloudly-managed service, so Cloudly composes its runtime environment. Everything below is off unless coreMailEnabled is true; with it false the cluster stays on today's dcrouter-direct mail path, untouched.

Settings

Key	Shape
coreMailEnabled	boolean cluster switch
coreMailServiceId	the Cloudly service running CoreMail
coreMailControlEndpointUrl	https://host/socket — the control socket
coreMailTransferOrigin	https://host — bare origin, no trailing slash
coreMailGatewayEndpointUrl	wss://host/ — bare origin with trailing slash
coreMailControlCredentialId / coreMailControlCredentialVersion	control credential identity
coreMailSmtpHost / coreMailSmtpPort / coreMailSmtpTlsMode	submission listener the SMTP_* variables point at
coreMailDefaultMessagesPerMinute / ...PerDay / coreMailDefaultMaxPendingInbound	per-binding quotas, clamped to the published contract ceilings

The two URL shapes differ in exactly one character and are not interchangeable: the interfaces normalizer canonicalises endpointUrl with url.toString() (trailing slash) and coreMailTransferOrigin with url.origin (no slash). Cloudly rejects the wrong shape at edit time rather than letting CoreMail refuse the desired state later.

coreMailConfigEpoch is server-only and never writable from the wire. It is the monotonic desired-state epoch; CoreMail refuses any preparation that does not strictly exceed what it has applied, so a caller-supplied value could strand reconciliation permanently.

Secrets

Four system secrets back the control plane:

• COREMAIL_CONTROL_CREDENTIAL_SECRET — Cloudly's own control-session credential, minted by Cloudly on the first reconcile. CoreMail receives only the matching verifier, in COREMAIL_CONTROL_BOOTSTRAP.
• COREMAIL_GATEWAY_CREDENTIAL_SECRET — the credential CoreMail uses to authenticate to dcrouter. Cloudly delivers the plaintext to CoreMail's environment and hands dcrouter only the verifier.
• COREMAIL_SMTP_TLS_CERTIFICATE_PEM and COREMAIL_SMTP_TLS_PRIVATE_KEY_PEM — operator-supplied in v1. Automatic issuance for the submission listener is a later workstream; until it ships the operator provisions and rotates this material. Absent PEMs simply mean the listener is not provisioned, which is not a reconcile failure.

Per-service material is delivered as managed secrets with managementSource: 'coremail' and scope coremail:<serviceId> — deliberately distinct from mail:<serviceId> so the two controllers cannot overwrite each other while both environment sets coexist.

The delivered names are exactly the ones @serve.zone/platformclient reads, so a Cloudly-provisioned service and an Onebox-provisioned one are identical to the workload. The unscoped group covers the default outbound address: MAIL_COREMAIL_URL, MAIL_COREMAIL_BINDING_ID, MAIL_COREMAIL_CREDENTIAL_ID, MAIL_COREMAIL_CREDENTIAL_VERSION, MAIL_COREMAIL_CREDENTIAL_SECRET and MAIL_FROM, plus SMTP_* when a submission listener is configured — with the SMTP username the binding id and the password the same credential secret.

Every enabled address additionally receives a complete scoped group: MAIL_<TOKEN>_FROM, all five MAIL_<TOKEN>_COREMAIL_* names and the scoped MAIL_<TOKEN>_SMTP_* set, mirroring how the dcrouter path delivers per-address variables today. platformclient takes a scoped group wholesale when one is present and otherwise uses the unscoped group only if MAIL_FROM equals the requested address, so a service with more than one address can only send from all of them if each carries a full group. The service still has exactly one binding; the scoped groups repeat its values under each address token.

Both authorities Cloudly owns keep their public half in a Cloudly-owned CoreMailControlCredential record. Verifier hashing is salted, so recomputing it would emit a different string on every pass and rewrite the bootstrap environment and dcrouter's stored peer for no change; storing the emitted verifier makes an unchanged reconcile a genuine no-op.

Credentials rotate with a 15-minute overlap: the previous credential is carried as retiring with an acceptUntil deadline so an already-connected workload is not disconnected the moment a new epoch lands.

Migration

Each service moves in three reversible phases, at most one step per reconcile:

1. dual-write — the CoreMail binding and environment are delivered while inbound still lands on the dcrouter-direct target and MAIL_TYPED_URL still resolves. Nothing in the application has changed.
2. inbound cutover — inbound is routed at CoreMail. Both environment sets remain, so a rollback is a single binding flip.
3. CoreMail only — the dcrouter-direct environment and credential are retired.

The phase is never held in memory: it is derived on every read from durable evidence — the binding in service.data.mail.coreMail, whether the CoreMail environment is delivered, the address targetType persisted at cutover, and whether the mail:<serviceId> scoped environment still exists. A Cloudly restart therefore cannot walk a migrated service backwards.

Step 2 waits until CoreMail's own reconciliation status reports the binding active; Cloudly having sent a desired state is not evidence that CoreMail applied it. Step 3 additionally waits until CoreMail has reported real delivered or acknowledged traffic for that service. A binding that stops being active walks a migrated service back one phase so the fallback is restored before inbound moves. Setting coreMailEnabled to false returns every service to the dcrouter-direct path immediately.

Operator requests

• getCoreMailControlStatus — applied epoch, digest, CoreMail's reconciliation status and the last error. enabled: false with every other field absent means no CoreMail workload is configured.
• getCoreMailServiceMailStatistics — per-service, per-UTC-day counters.
• listServiceMailTargets — every service with mail configured, CoreMail-fronted or not, so what remains on the dcrouter-direct path is visible.

Starting Cloudly

Install and build with pnpm:

pnpm install
pnpm build
pnpm start

Cloudly evolves persisted data through a @push.rocks/smartmigration chain that lives entirely in ts_migration/ and runs before the app serves. The chain carries its own data version, deliberately separate from the package version, and a SmartData-backed ledger records how far a database has advanced, so each boot applies only the outstanding steps under a lock. That ledger is named cloudly and is stored as the SmartdataEasyStore document smartmigration:cloudly. Only one instance migrates at a time, so an instance that dies while holding the lock makes every later boot fail with LOCK_TIMEOUT until the recorded lock expires. Confirm the old instance has stopped and wait for the ledger's expiresAt before retrying; do not manually clear or replace the lock. These migrations are forward-only: an applied step is never replayed, and downgrading the data version is refused.

This release carries steps from data version 0.10.0 onwards; every step below that was released, applied everywhere and then deleted. The floor is declared as oldestSupportedVersion, so a ledger below 0.10.0 fails startup by name with LEDGER_BELOW_OLDEST_SUPPORTED_VERSION on the first ledger read — before the lock is taken and before anything is written — instead of silently running the truncated remainder. A database in that state must first run the last release that still contained those steps. A database with no ledger at all is judged by its data: an empty one is stamped at the chain target, and a populated one is taken to sit at the floor and migrates forward from there. A database last migrated by a Cloudly before 25.2.0 never wrote a ledger — that release replaced the one-shot completion flags with this one — so it is taken to sit at the floor on the evidence of its collections alone, and the deleted steps are not replayed on it. Upgrade such a database through a release that writes the ledger before this one.

released-runtime-state-precondition (0.10.0 to 0.11.0) is the first step of the chain, because it reads and refuses and changes nothing: when it refuses, no step of this release has written either, and the ledger is left at the version the released Cloudly still starts on. It counts the state that only a node or a pod could have produced — the node-signed enrollment acknowledgements in cloudly_node_enrollment_receipts_v1, the node observations in cloudly_secret_runtime_node_observations_v2, the protection receipts in cloudly_runtime_network_protection_receipts_v1 and cloudly_runtime_network_protection_receipt_revisions_v1, the assignments in cloudly_runtime_assignments_v1 that carry a node observation or a terminal receipt, and the registrations in cloudly_runtime_sessions_v1 that named no node process. Those are the names the released Cloudly wrote them under, which is what a database still has at this point in the chain: the step that takes the version token off every collection name runs later, and this precondition counts whichever of the two names the database actually carries. None of it can be re-signed or re-derived from what Cloudly stores, so a single such row fails startup by the name released-runtime-state-present, naming every collection that counted, rather than letting a later step rewrite the database around it. The refusal carries the per-collection counts and the reason each class is bound on the error's counts; the message itself stays short enough that the migration ledger and the boot failure carry it in full.

cluster-runtime-phase (0.11.0 to 0.12.0) then states the runtime every existing cluster already runs on.

Secrets v2 is complete. The separately authorized offline cutover that produced it -- its CLI subcommands, its engine, and the startup gate that re-verified its completion receipt on every boot -- is removed in this release, and the data migration step retire-secret-cutover (0.12.0 to 0.13.0) retires what it left behind.

The step verifies that authority one last time: the cutover state must be completed, the production execution binding it names must be the one stored beside it, the authenticated scratch rehearsal receipt that binding embeds must carry the baseline the cutover ran against, and the completion receipt's MAC must recompute under the KEK it names from the mounted keyring. An authority that does not verify fails startup by the name secret-cutover-retirement-blocked with the reference that failed, and drops nothing.

Once it verifies, the step writes one provenance row into cloudly_secret_cutover_archive -- the run, environment and Cloudly version that completed the cutover, when it completed, the KEK and the receipt MAC that authorized it, the digests of the execution binding, the rehearsal receipt and the completion payload, the baseline digests that receipt covers, and the name, kind, row count and content digest of every namespace it is about to drop -- and then drops cloudly_secret_v2_cutover_state (the completion receipt and the execution binding) and the settlement-only cloudly_legacy_deployment_settlement_attempt_v1 and cloudly_legacy_deployment_settlement_execution_evidence_v1 collections: their provenance is kept, their evidence is not. It drops the empty SecretGroup and SecretBundle tombstone views over cloudly_legacy_secret_groups_v1_frozen and cloudly_legacy_secret_bundles_v1_frozen if a database still carries them; a database at the migration floor does not, because 25.2.0's own cleanup already dropped both views and both drained collections. A namespace that is already gone is already retired: it is neither digested nor named in the archive. A SecretGroup or SecretBundle that is present and is not that empty view over that frozen collection is somebody else's data, and is refused rather than dropped -- on a resumed run as well as on the first one. The archive row is written before the first drop, so a run that finds it finishes the drops instead of verifying an authority it has already consumed and archived.

A boot that meets a refusal reports secret-cutover-retirement-blocked: <reference>., and the reference names what failed:

• cutover-state-missing -- cloudly_secret_v2_cutover_state holds no secret-v2-cutover-v1 document.
• cutover-not-completed -- that document exists, but its phase is not completed.
• cutover-state-invalid -- a field of the cutover state is missing or does not have the shape the cutover wrote.
• completion-invalid -- the completion block or the receipt inside it is missing or malformed.
• execution-binding-missing, execution-binding-purpose, execution-binding-run -- the standalone execution binding is absent, is not the production binding, or names a different run than the state does.
• execution-binding-mismatch -- the stored binding is not the binding the completion receipt covers, so archiving it would attest an authority that never authorized this cutover.
• rehearsal-receipt-invalid, rehearsal-baseline -- the rehearsal receipt the binding embeds is malformed, or it rehearsed a different baseline than the one the cutover ran against.
• completion-receipt-key -- the mounted keyring has no KEK under the id the receipt names, which is what a premature KEK rotation looks like.
• completion-receipt -- the receipt's MAC does not recompute over the state it claims to cover.
• canonical-number -- a document that has to be authenticated or digested holds a number that has no canonical form (NaN or an infinity).
• SecretGroup-not-a-cutover-tombstone, SecretBundle-not-a-cutover-tombstone -- that namespace is present and is not the cutover's empty view over its frozen collection.
• <namespace>-unexpected-namespace-type -- the catalog reports one of the retirable names as neither a collection nor a view.
• archive-write -- the archive row was not inserted, so nothing is dropped.
• archive-record, archive-provenance, archive-dropped -- the archive row a run reads or writes is not a complete archive record. On a resumed run that row is the only proof the authority gate has already passed, so anything else under its id refuses instead of standing in for it.

unversioned-collection-names (0.13.0 to 0.14.0) then takes the version token off every collection a released Cloudly created. The 49 pairs are frozen in the step itself and are exactly the explicit collection names the 27.x releases declared; each new name is the old one without its trailing _v1 or _v2, and collections added after 27.5.0 never carried a token. The step renames the collection rather than copying it, so the documents, their _ids and their indexes are the ones that were already there, and the models this release reads them with prepare against the renamed collection on the same boot.

The whole plan is decided against one catalog read before anything moves, so a refusal leaves the database exactly as it found it. A boot that meets one reports collection-rename-blocked: <reference>.:

• <name>-unexpected-namespace-type -- either name of a pair holds something that is not an ordinary collection, so it is somebody else's namespace.
• <old name>-both-names-present -- both names hold a collection at once, which a rename cannot produce: the source stops existing the moment it moves. The step does not choose which of the two is the database.
• <old name>-document-count-after-rename -- a collection that moved does not hold the rows it held before the move.

Each pair moves on its own, and a pair that is already at its new name is not moved again, so a boot interrupted between two renames -- or between the last rename and the ledger stamp -- finishes on the next one. Nothing is left at the old names, not even an empty view: a released Cloudly started against a database this step has migrated already fails its own migration planning with TARGET_NOT_REACHABLE, before it opens a handler or a listener.

secret-material-encryption-context (0.14.0 to 0.15.0) then re-encrypts every stored secret material under the encryption context this release authenticates with. The authenticated data loses its shape member and its profile token: it is {profile: 'aes-256-gcm+wrapped-dek', purpose, secretId, secretVersionId, organizationId, owner, key, environment}, and the stored envelope states the same profile and carries no schemaVersion. Every context value, the algorithm, the wrapped-DEK construction and the fingerprint rule are the ones the released Cloudly used, so the step rebinds material and never rotates a key: each document is re-encrypted under the KEK it already names, with a fresh DEK and fresh nonces.

There is no rollback to 27.x after this step. Every ciphertext in the database is then bound to the new context, and a released binary computes the old one, so it authenticates nothing it reads. A control plane that has to go back restores the checkpoint taken before the upgrade and starts the released binary against that; the converted database is not a database 27.x can serve.

The step covers cloudly_secret_versions.encryptedMaterial, cloudly_system_secret_candidates.encryptedMaterial -- together with the stagedKekId and stagedMaterialFingerprint that restate it -- cloudly_federated_session.encryptedToken and cloudly_runtime_network_signers.encryptedPrivateKey. A version whose material was purged and a revoked signer hold none, and the step invents none. It runs after the rename above, so it walks those collections under the names this release reads them by.

One context value is derived rather than stored: a federated session's secretVersionId is a digest over its binding, and this release derives it under an unsuffixed domain. A released envelope is therefore bound to the versioned digest, so that site states a released context of its own -- the released spelling exists only inside the step -- and the envelope is opened under the released digest and re-sealed under the one the runtime derives.

The signing key's context is derived too, from the authority the signer row carries, and that authority still states its released shape while this step runs: it is read through the same released reader the rest of the chain uses, which takes the shape member off and leaves the two members the context takes -- the signer's identifier and its authority digest -- exactly as the row states them. Both of those move later in the chain, and contract-documents re-seals the key under them; until then the envelope stays bound to what the released Cloudly wrote.

One transaction covers one chunk of 64 documents and the step-owned progress collection cloudly_secret_material_encryption_cursor that records how far each collection has come, so a boot interrupted between two chunks resumes at the chunk it committed and re-encrypts nothing twice. The progress collection is dropped when the last collection is through; a database that holds no secret material never creates it and never loads the keyring. Which context a document was written under is read from the profile the document states -- never from a trial decryption -- and every re-encrypted envelope is decrypted again, through the reader the runtime uses, before the transaction that replaces the material commits.

A boot that meets something it cannot convert reports secret-material-encryption-blocked: <collection>:<document>-<reference>. and commits nothing of the chunk that refused, so the material it names is still readable under the context it was written with:

(A document whose _id is not an identifier is refused as <collection>-document-identity, and a cursor namespace that is not a plain collection as <cursor collection>-unexpected-namespace-type.)

• -material-shape -- the stored envelope is not one a released Cloudly wrote.
• -unexpected-profile -- the envelope states neither the released context nor this one.
• -kek-unavailable -- the mounted keyring carries no KEK under the id the envelope names.
• -material-not-readable -- the envelope does not open under the released context.
• -context-unavailable, -secret-metadata-missing -- the context the material is bound to cannot be rebuilt from what the database holds.
• -verification -- the re-encrypted envelope did not decrypt back to the same bytes.
• -not-rewritten, -<field>-mismatch, -cursor-not-advancing -- a document changed under the step, a field that restates the envelope disagreed with it, or a committed chunk did not move the collection forward.

Keep every KEK that protects stored material in the mounted keyring while this step runs: it converts what each document names, so a rotation that drops a KEK makes the step refuse by name.

unversioned-derived-identifiers (0.15.0 to 0.16.0) then takes the version token off every identifier a released Cloudly derived and stored, so the identifier the code composes and the one the database holds are the same string again. It rewrites the two singleton row ids (cloudly-runtime-controller, cloudly-runtime-network-protected) and every hashed id and content digest of these families, together with every member of every other row that names one:

Family	Row it identifies	Members that name it
node credential authority	cloudly_node_credential_authority	cloudly_node_credential_mutations.authorityId
node credential mutation	cloudly_node_credential_mutations	--
service-machine credential	cloudly_service_machine_authority	its own credentialId and userId, User.id, cloudly_service_machine_mutations.result.credentialId and .result.userId, cloudly_secret_metadata.managementScope
private membership, private alias	cloudly_private_membership, cloudly_private_alias	--
private mutation receipt	cloudly_private_mutation (id and requestDigest)	--
protected revision, network pool	cloudly_runtime_network_protected_revisions, …_pools	--
network node scope	cloudly_runtime_network_protection_receipts, …_projections, …_trust_deliveries, …_dns_leases	--
protection-receipt revision	cloudly_runtime_network_protection_receipt_revisions	--
network attempt	cloudly_runtime_network_workloads.attemptKey	--
allocation eligibility	cloudly_runtime_network_allocation_eligibility	cloudly_runtime_network_handoffs.eligibilityId, cloudly_runtime_network_workloads.eligibilityId
managed VPN, service policy	cloudly_runtime_managed_vpn, cloudly_runtime_network_service_policies	--
runtime slot	cloudly_runtime_slots	cloudly_runtime_assignments.slotId
runtime config	cloudly_runtime_configs	--
network signer	cloudly_runtime_network_signers	its own authority.id and authority.previous.id, cloudly_runtime_network_signing_revisions.authority.id and .authority.previous.id, cloudly_runtime_network_projections.signed.authority.id, cloudly_runtime_network_trust_deliveries.acknowledged.id and .pending.id
network signing revision	cloudly_runtime_network_signing_revisions	--

Each family is one transaction: its rows and every reference to them move together, so no row and no reference ever disagree, and an interrupted run leaves that family exactly as it found it. Each row's new identifier is derived by the composer the runtime itself looks the row up by, from the inputs the row already stores, and the released one is recomputed from the retired domain, which exists only inside the step. A row that already carries this release's identifier is passed over, so a second run writes nothing and a resumed run finishes what it did not reach; the rows are the progress record, and the step keeps none of its own. The step runs after the re-encryption above because it rewrites no member any encryption context is bound to.

A boot that meets a row it cannot move reports derived-identifier-blocked: <collection>.<member>:<identifier>-<reference>. and commits nothing of the family that refused:

• -identifier-shape -- the member that has to carry the identifier holds something else.
• -unexpected-identifier -- the stored identifier is neither the released derivation nor this one, so the step cannot say what the row is.
• -inputs-not-derivable -- the row's own stored inputs do not compose a valid identifier.
• -not-rewritten, -references-not-rewritten -- a row or a set of references changed under the step, so what it counted is not what it moved.

The step reads every released body through ts_migration/released-contract-shape.ts, because every reader of this release refuses the shape member a released body still carries. That module states the released shape rule once -- a released body is this release's own snapshot with the shape member in front, which is what every released snapshot built -- so a family that hashes a whole body (the allocation eligibility over its session bindings, the private mutation receipt over its request) reproduces the released identifier from the released bytes rather than from a stripped approximation of them. The step derives this release's identifier first and asks for the released one only for a row that has not moved, so a row already in this release's shape never needs a released reader at all.

The signing families are derived from the controller row rather than from anything the signer row holds: a controller statement is written once and never replaced, so a database holds exactly one signer row. A signer row with no controller row to name refuses as cloudly_runtime_controller-missing. Moving the signer identifier also moves the AAD the signing key's envelope is sealed under, so the step that recomputes the signing-authority digest re-seals that key under both the moved identifier and the recomputed digest.

Two groups of released bytes deliberately keep their version token here. The mutation receipts of the human and service-machine credential families hash a mutationId that was never stored, and their requestDigest covers material only the caller held -- the human one under a runtime HMAC key -- so neither can be recomputed, and the service deletion receipt hashes a binding that is likewise not stored. No reader re-derives any of them: a pre-32 mutation id is not found and is judged a new mutation, which its own generation fence then refuses as a conflict rather than replaying it. The network projection body's own id and the handoff lease's conntrack label name values inside a body a stored digest covers, so they move in the same write as that digest rather than here.

contract-documents (0.16.0 to 0.17.0) then rewrites every stored document into the contract shape this release reads. Every released contract body stated a schemaVersion that no reader of this release admits, and every digest taken over such a body changed twice over: the member left the payload, and the digest domain lost its version token. The step takes the member off and recomputes the digest with the composer the runtime itself verifies with:

Family	Collection	Digests recomputed
service-machine authority, service-machine mutation	cloudly_service_machine_authority, …_mutations	--
private network, private membership	cloudly_private_network, cloudly_private_membership	--
private mutation	cloudly_private_mutation	requestDigest over the request in the same write
runtime config	cloudly_runtime_configs	the container invocation, then the config that covers it
runtime assignment	cloudly_runtime_assignments	every revision, bound to the revision before it; the workload's config reference and the predecessor attempt's last revision
runtime session	cloudly_runtime_sessions	--
resolved secret manifest	cloudly_resolved_secret_manifests	manifest.digest
secret-version purge operation	cloudly_secret_version_purge_operations	operation.intentDigest
secret runtime recipients, cluster targets	cloudly_secret_runtime_recipients, …_cluster_targets	the ready target set's targetSetDigest
corestore publication grant	cloudly_corestore_publication_grants	grant.bindingRequestDigest over the binding request the same row stores
corestore publication receipt	cloudly_corestore_publication_receipts	both copies of the grant follow the grant row the receipt names
network protected revision, network protected	cloudly_runtime_network_protected_revisions, cloudly_runtime_network_protected	the protected authority per generation, each bound to the generation before it
network pool	cloudly_runtime_network_pools	the reference to the protected authority follows the revision it names
network handoff	cloudly_runtime_network_handoffs	the lease, over the authority reference it follows and the conntrack label this release derives
network workload	cloudly_runtime_network_workloads	the lease, and the reservation's own reference to the handoff lease
network signing revision, network signer	cloudly_runtime_network_signing_revisions, cloudly_runtime_network_signers	the signing authority per generation; the signer's private key is re-sealed in the same write
cluster, cluster node	Cluster, ClusterNode	--
platform binding	PlatformBinding	the retention evidence's intentSha256
service, deployment operation, adoption operation, adoption rollout	Service, ServiceDeploymentOperation, ServiceDeploymentAdoptionOperation, ServiceDeploymentAdoptionRolloutOperation	the container invocation each one carries

Each family is one transaction, and a document is compared with what its own recomposition produces: the rewrite and the no-op are the same rule, so a second run writes nothing and a run interrupted between two families resumes at the next one. What the comparison reads is what a value holds, so a member the step does not compose -- a TTL date, for one -- is carried through as it was stored rather than rewritten or flattened.

Before the first write the step counts the classes of document it is known not to be able to move and refuses on one: the six the precondition at the top of the chain counts, and an assignment whose record-level predecessor names a row the database no longer holds, whose digest is that row's last revision. Those counts are all that is proven up front. A document that refuses for its own shape is found by the family that reads it, after earlier families have committed -- which is safe rather than partial, because the ledger stays at 0.16.0, every family is a fixed point of its own recomposition, and the next boot counts again and resumes at the family that refused. A boot that meets one reports contract-documents-blocked: <collection>.<member>:<reason>, naming the body that refused, with -released-runtime-state-<count>, -node-digested-eligibility-<count>, -predecessor-row-missing, -body-shape, -unexpected-shape-statement, -not-readable, -intent-missing, -workload-config-missing, -workload-authority-not-recomputable, -grant-operation-mismatch, -grant-row-not-recomposed, -reference-shape, -previous-shape, -protected-revision-missing, -signing-revision-missing, -handoff-row-missing, -conntrack-label-unexpected, -signer-scope-mismatch, -context-unavailable, -kek-unavailable, -released-context-unreadable, -reseal-not-readable or -not-rewritten. An allocation eligibility proof is one of the counted classes: its identity is a hash over the protected-authority reference this step re-digests and over the node-digested protection receipts it names, and the store writes one only when every egress authority has such a receipt, which the precondition at the top of the chain already refuses.

Five stores start empty and are emptied only after every family has committed, each with its count logged: cloudly_federated_session, cloudly_secret_runtime_enrollment_challenges, cloudly_secret_runtime_report_replay, cloudly_runtime_network_trust_deliveries and cloudly_runtime_network_projections. A projection is compiled state: the node producer composes the next one from the protected authority and tolerates no previous journal, so its previous, retiredLeases and retiredHandoffs history is dropped rather than rewritten.

Both copies a corestore publication receipt keeps of its grant follow the grant row the receipt names, which the grant family has already recomputed and committed: the replay path compares a stored receipt's copy with the grant a retrying caller presents, so a copy left on the released digest would make every stored receipt disagree with the grant it names. A receipt whose grant has already been reaped by its TTL keeps what was written, because the value it would follow is gone and a replay needs that row anyway. The ingress admission binding always keeps what was written: it covers the request bytes a caller sealed an envelope over, and Cloudly stores neither the envelope nor that request, so nothing here can reproduce it. A caller that retries a publication accepted before this upgrade under the mutation id it used then is answered REPLAY_CONFLICT instead of the stored receipt -- the secret versions that publication created were written before the upgrade and are not lost by it.

The network authority, lease and signer subtree moves in the same step, because one of its digests is what an encrypted private key is bound to. The protected network authority is recomputed from its first generation up -- a generation's digest covers the reference it carries to the generation before it -- and every reference to it follows the revision row it names: the singleton head, the pools, and both kinds of lease. A handoff lease also carries a conntrack label derived from its own identity under a domain this release unsuffixed, so that label moves in the same write as the digest that covers it, and a workload reservation's reference to its handoff follows the lease the handoff family recomposed and committed.

The runtime network signing key is re-sealed in the same write as the authority it belongs to. networkSigningContext names the signer's identifier as the envelope's secretId and that authority's digest as its secretVersionId; the identifier moved in unversioned-derived-identifiers and the digest moves here, so the step opens the envelope under the released pair, seals it under the moved pair with a fresh DEK and fresh nonces, opens it again and compares it with the plaintext it started from before the row is written. An envelope that already opens under the moved pair is kept exactly as it is, which is what makes a second run write nothing; one that opens under neither refuses by name with nothing written for that signer. The step takes the operator keyring for this, and loads it only when there is a sealed key to move. The projections emptied above are recompiled at generation 1 from the moved authority with the re-sealed key, so that rewrite leaves nothing to re-sign. Both halves of this rewrite -- the contract bodies and the network subtree a sealed key is bound to -- are one step of one release: no build ever runs against a database whose signing key is sealed to an identifier or a digest the code no longer computes.

runtime-session-cluster-binding (0.17.0 to 0.18.0) adds the cluster owner to every retained Pallet session binding. The step derives that owner only from the current ClusterNode when a persistent Pallet credential authority for the same node predates the session and still covers its credential generation and session epoch. A missing node or authority, a cluster mismatch, or an allocation-eligibility copy that differs from its durable session refuses with runtime-session-cluster-binding-blocked before any write. The step then recomputes each allocation eligibility identifier and moves the handoff and workload references that name it in the same transaction as the session rows. Already enriched rows are compared by the same rules and retained, so replay writes nothing. The step never reconstructs a deleted node's cluster from a replacement node and never deletes historical packet or lease authority to make an upgrade pass.

cluster-relay-registration-credential (0.19.0 to 0.20.0) removes every relay registration that was stored before a registration named the authority it was accepted under. Those rows can name no credential and none was ever recorded for them to name, and minting one inside a migration would be Cloudly inventing evidence about a socket it never saw. A stored registration is not authority — it is what a relay said on a socket that may be long gone — so the row is dropped and any live relay registers again within its own reconnect cadence, which re-creates it with the authority that relay really holds; until then the row's listen address, version, sequence and vpnHub selection are absent. Rows that already name one are checked by the very assertion that admits them into the database and retained, a row of any other shape refuses the step by name before anything is removed, and replaying the completed step removes nothing.

service-runtime-targets (0.18.0 to 0.19.0) converts the released singleton runtime spec and status rows into private service/cluster records in the same collections. It derives the composite id from the released spec's service and cluster, joins a status to exactly one spec, and changes only the private id plus serviceId and clusterId; the public projection and revision remain byte equivalent. Mixed singleton/target shapes, duplicate targets and orphaned or ambiguous statuses refuse before the transaction writes anything. Replaying the completed step retains every row.

secrets keyring-generate, secrets keyring-check, secrets keyring-convert, secrets export-cluster-relay-authorization and secrets set-corestore-control-token are the only secrets subcommands; every other one is answered with USAGE_INVALID. The subcommand was named secrets-v2 before this release. Onebox's cutover coordinator is the one live consumer that still spawns node cli.js secrets-v2 ... into a maintenance container built from the Cloudly image, including the keyring-generate and keyring-check pair; against a 32 image those calls fall through to a normal Cloudly runtime boot instead of the CLI, so Onebox must invoke secrets. The immutable Cloudly image still dispatches the subcommand before importing or constructing the normal Cloudly runtime: the keyring commands never load it, and export-cluster-relay-authorization and set-corestore-control-token load it only when they run, to boot the foundation they read and write through. Every invocation suppresses command-time stdout and stderr and then emits one bounded JSON object followed by a newline on stdout, with exit status 0 only when that object reports ok: true.

The keyring commands and the WorkloadInit runtime approval loader accept private JSON inputs only through a lexically normalized absolute path. This applies to SERVEZONE_SECRET_KEYRING_PATH and SERVEZONE_WORKLOADINIT_APPROVAL_PATH. The existing parent directory must be owned by the process user and not writable by group or other users. The input must be a regular, non-symlink file owned by the process user, have no group or other permissions, have exactly one hard link, and be no larger than 64 KiB. Normal Cloudly startup reads the configured keyring through the runtime loader; it does not apply these CLI-specific ownership, link-count, mode, and size checks. The WorkloadInit approval loader does apply the strict rules and the 64 KiB limit. Keyring output uses an atomic, no-clobber private writer with mode 0600: it holds the verified directory descriptor, writes a random same-directory mode 0600 temporary file, fsyncs, and publishes without clobbering. It reconciles only old temporary files from dead writers and never follows an output or temporary-file symlink. --kek-id and --recipient-key-id are optional safe identifiers.

node cli.js secrets keyring-generate \
  --output /run/cloudly-secrets/keyring.json
export SERVEZONE_SECRET_KEYRING_PATH=/run/cloudly-secrets/keyring.json
node cli.js secrets keyring-check

The keyring file this release reads carries activeKekId, keks and ingress and no shape member. A file a released Cloudly wrote states schemaVersion: 1, and this release does not load it: convert it once, offline, while Cloudly is stopped, and mount the converted file.

node cli.js secrets keyring-convert \
  --input /run/cloudly-secrets/keyring.json \
  --output /run/cloudly-secrets/keyring.converted.json

The conversion carries the key material across untouched -- it is a conversion, not a rotation -- publishes a fresh mode 0600 file without clobbering an existing one, leaves the mounted file exactly as it was, and reports the identity keyring-check reports and no key material. It refuses rather than repairing: a file it has already produced is answered with KEYRING_CONVERSION_REFUSED and the reference already-converted, and anything that is not a released keyring file with the reference not-a-released-keyring-file (a file whose content is not a JSON object with the reference not-a-keyring-file). Neither writes anything, so the mounted keyring is still the one the running release loads.

secrets export-cluster-relay-authorization reads the database. It hands a cluster relay's bearer to the operator who deploys that relay; the procedure is under Cluster Relay Credential Delivery. Run it where the keyring lives, inside the Cloudly container, with that container's environment:

node cli.js secrets export-cluster-relay-authorization \
  --cluster-id <clusterId> \
  --expected-generation <generation> \
  --output /root/relay-export/relay-authorization

It boots only the persistence and secret foundation, without the server, and migrates nothing: a database at any other data version than the one this build expects is refused with DATA_VERSION_MISMATCH before a model or the keyring is touched, and a foundation that does not start answers the secret manager's code (for example SECRET_KEYRING_INVALID) or CLOUDLY_START_FAILED. It then reads exactly one secret version: the version of the Cloudly-system secret SERVEZONE_CLUSTER_RELAY_AUTHORIZATION that the cluster's relay credential delivered at the named generation, and only while it is still that secret's active version. The decrypted bytes are proved against the credential's hash of the bearer that generation issued, re-read after the decryption, so a rotation that commits meanwhile is refused instead of exported. Every other refusal is RELAY_AUTHORIZATION_EXPORT_REFUSED with a reference:

Reference	Meaning
relay-credential-absent	No relay credential is minted for the cluster.
relay-credential-generation-mismatch	The credential is at another generation than --expected-generation, before the read or after it.
relay-authorization-missing	The version the credential delivered is no longer an active stored version.
relay-authorization-ambiguous	The secret's active version, or its decrypted bytes, are not what the credential issued; the command picks neither.

--output follows the private-file rules above: a lexically normalized absolute path in an existing directory owned by the invoking user and not writable by group or other users. An existing output, a symbolic link included, is refused with OUTPUT_EXISTS before the database is opened, and refused again rather than replaced when the file is published. The bearer is written as a new regular file owned by the invoking user, mode 0600, one hard link, holding the bearer's bytes and no trailing newline, through the same no-clobber private writer as the keyring output: a same-directory temporary file created with O_EXCL and hard-linked into place, so the output holds the whole bearer or does not exist. The database connection is closed and the keyring wiped before the file is written, so a failed teardown leaves no file. The command prints metadata only:

{"ok":true,"command":"export-cluster-relay-authorization","clusterId":"<clusterId>","generation":1,"byteLength":147,"sha256":"<hex>"}

sha256 is what sha256sum prints for the file. It is also the hash Cloudly keeps of the bearer, which tells nobody more than the file does: whoever reads the answer holds the file. The bearer itself never appears in the answer, in a log or in any typed request.

secrets set-corestore-control-token --stdin stores the Corestore control token (CORESTORE_API_TOKEN) as platform-managed material of both Corestore provider configs (cloudly-corestore-database and cloudly-corestore-objectstorage) in one transaction. The elected relay of a cluster reads it sealed to the secret recipient it enrolled, through getCorestoreControlCredentialMaterial, and calls its nodes' Corestore control API with it; no workload is ever delivered it, and no typed request or setting writes it. The command reads the token from stdin and from nowhere else -- never from an argument, which would sit in the process list and the shell history, and never from the environment -- so pipe it into the running Cloudly container:

read -rs CORESTORE_API_TOKEN   # pasted, not echoed
printf '%s' "$CORESTORE_API_TOKEN" \
  | docker exec -i <cloudly-container> node /app/cli.js secrets set-corestore-control-token --stdin
unset CORESTORE_API_TOKEN

docker exec needs -i to hand stdin over; without it the command reads nothing. One line terminator after the token is not part of it. The input is judged against the Corestore credential contract before the database is opened, so a stored token is always one the relay can be handed. The command boots the same foundation as the export, with the same DATA_VERSION_MISMATCH, secret-manager and CLOUDLY_START_FAILED refusals. A stored token is replaced only with --replace, and either both providers hold the new token or neither does. Every other refusal is CORESTORE_CONTROL_TOKEN_REFUSED with a reference, and none of them carries the token:

Reference	Meaning
token-input-terminal	stdin is a terminal, which would echo a token typed into it.
token-input-empty	stdin carried nothing; docker exec without -i hands over nothing.
token-input-whitespace	stdin carried whitespace only.
token-input-not-printable	A byte is not printable ASCII, or whitespace sits inside the token.
token-input-oversized	stdin carried more than 4096 bytes.
token-invalid	The contract refuses the token's length or characters.
token-present	A token is stored and the command did not say --replace.
provider-unavailable	A Corestore provider config is missing or disabled; Cloudly writes both when its server starts, so a database no Cloudly server ever started on has none yet.

The answer names the key and the provider configs it was stored under, and never a byte of the token:

{"ok":true,"command":"set-corestore-control-token","key":"CORESTORE_API_TOKEN","providerConfigIds":["cloudly-corestore-database","cloudly-corestore-objectstorage"]}

A running Cloudly reads the token on the next relay request, so nothing restarts for it. set-cloudflare-token is gone with Cloudly's Cloudflare integration: it answers USAGE_INVALID.

Keep the KEK that signed the completion receipt in the deployment keyring until the retirement step has run on every database; a rotation that drops it makes the retirement refuse. After the step has run, the archive row is the only thing that names that authority, and the keyring is needed only for the encrypted system material it owns.

Cloudly releases publish the semantic-version image code.foss.global/serve.zone/cloudly:<version> and, in the same gitzone release (gitzone 7 and later), the @serve.zone/cli npm package from ts_cliclient/ at that same version. The root package stays private and is never published to npm. The mutable latest image tag is a legacy manual channel and is not changed by the release workflow.

Publishing a newer Cloudly image does not promote the repository-owned App Store channel automatically. servezone.appstore.json stays pinned to the 15.0.1 image digest; moving it is a separate, verified change and is not part of any release that merely publishes a newer image.

Run the TypeScript entry point during development:

pnpm run startTs

Start from code when embedding the control plane in another Node.js process:

The embedding process must set SERVEZONE_SECRET_KEYRING_PATH before every normal start(); it is intentionally not part of the public constructor DTO. The keyring is mandatory even when no workload secret delivery is currently expected because it owns encrypted system material and, until the cutover retirement has run, authenticates the completion receipt that step archives. Set SERVEZONE_WORKLOADINIT_APPROVAL_PATH as well before enabling Coreflow secret-runtime registration.

import { Cloudly } from '@serve.zone/cloudly';

const cloudly = new Cloudly({
  environment: 'production',
  publicOrigin: 'https://cloudly.example.com',
  listenPort: '3000',
  sslMode: 'external',
  servezoneAdminaccount: 'admin:change-me',
  mongoDescriptor: {
    mongoDbUrl: process.env.MONGODB_URL,
    mongoDbName: 'cloudly',
    mongoDbUser: process.env.MONGODB_USER,
    mongoDbPass: process.env.MONGODB_PASS,
  },
  s3Descriptor: {
    endpoint: process.env.S3_ENDPOINT,
    accessKey: process.env.S3_ACCESSKEY,
    accessSecret: process.env.S3_SECRETKEY,
    bucketName: process.env.S3_BUCKET,
    port: process.env.S3_PORT,
    useSsl: true,
  },
});

await cloudly.start();

Set SERVEZONE_INSTALL_DEMO_DATA=true only when you intentionally want the demo data installer to run. The code labels that path destructive.

API Model

Cloudly exposes a single composed TypedRouter. Managers add their own typed handlers to the main router, and CloudlyServer exposes that router through the HTTP/WebSocket server.

Cloudly also exposes an admin-JWT authenticated read-only MCP endpoint at /mcp. The MCP tools return safe summaries for clusters, services, deployments, domains, and nodes without machine tokens, SSH keys, secret values, provider zone IDs, or deployment logs.

Secrets v2 Safety Foundation

Cloudly persists secret material only as encrypted SecretVersion records. A mounted keyring supplies the active KEK and X25519 ingress recipient; startup fails closed before secret model and manager initialization when that keyring cannot be validated or authenticated completion verification fails. Earlier non-secret startup migrations may already have run. Completion verification and runtime preparation use the same loaded keyring instance, whose sole ownership is transferred to the secret manager only after verification. Admin create and rotation requests accept generated material or recipient-sealed input, and API responses contain metadata only.

Service-owned secrets use { kind: 'service', serviceId }. Shared material uses organization-owned SecretSets attached through service.data.secretConfiguration.secretSetAttachments. Generic service writes cannot mutate secret configuration or deployment authority, and unknown wire fields are rejected before persistence. Even an unchanged attachment request revalidates every referenced SecretSet and fails closed when one is missing, retired, or owned by another organization. Mail and Web Push use exact management scopes and revision-fenced service ownership; service deletion revokes service-owned material without deleting organization SecretSets. Retiring a SecretSet atomically detaches every current service consumer and publishes each post-detachment deployment manifest in the same transaction; deletion remains blocked while any attachment survives.

Secret and attachment mutations publish one immutable resolved manifest for every current target cluster, preserving accepted rollout history while dropping stale cluster states. A valid service that has not received an immutable image deployment yet is an explicit zero-manifest target, so its secret configuration can be prepared before deployment. Exact mutation replay never republishes manifests, but it retries the post-commit Coreflow config push when the persisted SecretVersion says the original mutation published runtime state. Coreflow broadcast delivery is concurrency-bounded and uses native TypedSocket request deadlines; overlapping broadcast requests coalesce into one active pass and at most one trailing pass. Copied plaintext bytes are wiped before any post-commit network dispatch begins.

Cloudly exposes 14 admin-only Secrets v2 handlers for secret and SecretSet lifecycle, attachments, resolution previews, and purge preflight. getSecretIngressRecipient additionally allows an exact-peer-bound registered Coreflow Secrets v2 runtime and returns only active public recipient metadata. Physical purge execution remains disabled because no erasure worker is authorized. The control plane also implements recipient enrollment, Spark swarm observation intake, target-authority reconciliation, live Coreflow registration, resolved-manifest rollover, recipient-sealed material delivery, deployment report replay protection, and fenced Corestore credential publication. These runtime contracts are supplied by @serve.zone/interfaces 32.7.1 and @serve.zone/api 32.0.0. End-to-end workload delivery is not claimed until the matching Coreflow and Spark integrations are released, configured, and enrolled. Spark observation replay returns the persisted exact receipt before age checks. Every non-replay observation, including the first report of a replacement reporter session, must advance the node-wide persisted observedAt high-water.

Every normal Cloudly startup requires the mounted keyring named by SERVEZONE_SECRET_KEYRING_PATH; there is no keyless mode or empty-secret exception. Startup validates the keyring and runs the data migration chain -- which, while the cutover retirement is still outstanding, authenticates the completion receipt with it -- before secret models and managers initialize, then transfers sole keyring ownership to the secret manager for runtime use and shutdown wiping.

Coreflow runtime registration additionally requires the private artifact named by SERVEZONE_WORKLOADINIT_APPROVAL_PATH. Cloudly parses an exact active or revoked schema and cryptographically verifies the detached statement and bundle, public key, registry evidence, trusted signer and revocation policy, expected version, image-index digest, and platforms through @serve.zone/workloadinit. Missing, malformed, or otherwise unverifiable approval material returns workloadinit-unconfigured; an explicit revoked artifact, revoked trusted signer, or failed revocation/timeline policy returns workloadinit-revoked. Cloudly reloads the approval while validating a live registered session. A generation, digest, or status change invalidates that session and requires Coreflow to register again.

Protocol Offers per Session Kind

The contract two peers speak is the @serve.zone/interfaces release they were built against, exchanged in the registration body itself: there is no protocol number beside it and no shape version anywhere in the package. ts/protocol.ts holds Cloudly's side of that exchange — one IProtocolOffer per session kind Cloudly is the server of (cloudlyClient, palletRuntime, clusterRelay, sparkNode, sparkSwarmNode), each built by protocol.createProtocolOffer from the installed release with its own minimum peer version, so raising what one kind of peer must speak is one named change for that kind alone and never a number an offer claims for a contract this build does not carry. Every accepted registration answers with the offer of the kind it opened, and the Spark routes state it on every body they answer.

Negotiation is the first thing every carrier does with a body, before its own validator, before the bearer or the JWT is looked at, and before anything is written: the contract validators are exact-key, so a peer of another major that was shape-checked first would be answered as malformed instead of being told which contract it met. For registerPalletRuntimeSession that order is also what keeps a refused node's packet access: the withdrawal happens inside the registration the refusal precedes.

A refused peer is answered by name, never by a collapsed denial. On the three TypedRequest carriers — registerCloudlyClientSession, registerPalletRuntimeSession and registerClusterRelay — the answer is a TypedResponseError whose text is describeProtocolRefusal and whose errorData is the IProtocolRefusal itself, so the peer reads which of the two builds an operator has to upgrade and backs off for protocol.refusedOfferRetryIntervalMs. The five Spark routes are plain HTTPS POSTs that carry no typed error, so their answer is HTTP 422 with the IProtocolRefusal as the whole body, which is what sparkNodeHeartbeatContract.protocolRefusalStatus and sparkSwarmNodeContracts.protocolRefusalStatus state.

A body that carries no readable offer is malformed, not incompatible: nothing was negotiated, so there is no refusal to name and nothing to back off from. The typed carriers collapse it into their registration denial and the Spark routes answer 400, exactly as they answer any other unreadable body. A body that does not name its node -- nodeId and nodeToken are the two members all five Spark routes read it by, and each must be a non-empty string -- is malformed for the same reason and gets the same answer, judged right after the offer so that a statement naming no node never reaches the credential store.

A refusal costs the peer nothing it already holds. No session record, no identity tag, no session epoch, no credential mutation, no relay registration sequence and no undispatched announcement: each of them happens after the negotiation, so a refused node keeps its workloads and its packet access and a refused relay keeps its custody and its stored registration.

Two carriers state an offer outside a registration: getDeploymentPreflight, the read that opens a deployment, and getServiceById, the read that opens an adoption. A deployer states its offer on those two and on no mutating deployment carrier, because a reservation's requestDigest covers its whole request body and an offer inside one would tie an in-flight deployment to whichever @serve.zone/interfaces release the deployer had installed. There the member is optional: a request that states none is served exactly as it always was, with nothing negotiated and nothing logged, and a stated offer is judged against the cloudlyClient minimum -- before the preflight's validator, before the bearer and before any service is read -- and refused by name in the same shape the registrations answer a refusal in. A stated offer Cloudly cannot read is not a refusal, because nothing was negotiated: the preflight names it as one INVALID_REQUEST blocker of its own report and getServiceById, which has no validator of its own, answers it as a malformed offer, so a deployer that tried to state a contract is never served as one that stated none.

The node-runtime routing read requires the Spark contract in @serve.zone/interfaces 32.11.0. Roll out a Spark build carrying 32.11.0 before this Cloudly build: Cloudly's sparkNode minimum is 32.11.0 so an older Spark peer is refused at its protocol-bearing heartbeat before it depends on the new read. This raise is scoped to sparkNode; palletRuntime remains at 32.10.1 and the retiring sparkSwarmNode routes remain at 32.0.0.

Spark Node Runtime Routing Read

getNodeRuntimeBinding accepts only the node ID and its current Spark bearer. Cloudly snapshots that exact body before authentication yields, then derives the complete response itself: canonical public Cloudly origin, current node and cluster, persisted runtime controller and namespace, validated cluster relay, current cluster phase and generation, and the Spark credential generation and session epoch that authenticated the answer. The caller cannot submit a cluster, relay, controller, namespace, phase, generation, or workload grant.

The read is one SmartData transaction. It real-write fences the credential, controller, node, exact cluster data, and exact runtime record, so credential rotation or revocation, node movement or deletion, relay changes, phase changes, controller replacement, and cluster deletion cannot produce a mixed answer. A transaction retry may return only the exact response its first attempt observed; if the routing identity changed, the read rejects. Every malformed, unauthenticated, stale, missing, corrupt, or raced request receives the same Node runtime binding rejected. typed error without a bearer or record-specific reason.

The returned binding is routing identity only. It creates no runtime session, does not change a cluster phase, and grants no workload, network, secret, or phase-transition authority. Pallet must still open a fresh authenticated runtime session before it can receive controller-owned work.

Pallet Runtime Sessions per Node

registerPalletRuntimeSession binds one node session to the exact live socket that carried it. A physical peer may carry one node session or many: a node can connect directly, and a cluster relay forwards the runtime session methods of every node it serves over one socket. Registrations, bindings, peer scopes and current-peer selection are therefore keyed by (peer, node). Cloudly still hashes and fences each node's own bearer and its own credential authority, so a carrier peer never becomes an authority for the nodes it forwards.

Each node on a peer holds its own session binding. One node registering, replaying, moving to another peer or being revoked never disturbs the other sessions on the same peer. A node's session may move to another peer at any time: the newest session epoch wins, the old peer's copy stops authorizing, and every protected effect remains fenced by the persisted credential authority rather than by socket identity. A departed peer loses every node session it carried, including every managed VPN member admitted through it, and leaves sessions held through other peers untouched.

A registration is identified by the controller process, the carrying peer, the node, its credential generation and the node process that registers it. Pallet names that process with nodeInstanceNonce, the canonical lowercase UUID it generates once per process start and sends on every registration. Since @serve.zone/interfaces 32.0.0 the member is part of the registration shape itself, so a body that names no node process has no contract to be read under and is refused before any rule about carriers applies: one relay socket carries a whole cluster and could otherwise not tell a node restart from a replay. A peer holding a Cloudly identity that no longer verifies — an expired cluster machine credential, say — is refused by name with carrier-identity-unverified until its identity verifies again. The nonce authenticates nothing; the node's own bearer still does.

The binding also records the node's clusterId from the ClusterNode fenced by the registration transaction. Current work, reporter allocation reads, resolved-authority reads and historical session recovery all fence the node again and require that same owner. Cloudly therefore refuses a session when the node is absent or its owner differs, rather than allowing a retained session or eligibility proof to cross clusters. Pallet runtime protocol admission requires @serve.zone/interfaces 32.10.1 or later, the first contract in this major that carries this binding.

Every registration also states resolvedAuthorities: the sorted, duplicate-free list of what the registering build can resolve for a workload — network, secrets, storage — which Cloudly stores on the session record. It states a capability of that build, not a permission, and it is deliberately not part of the registration identity: a changed build is a new process and therefore a new nonce, so a replay of one registration that states different authorities is refused with conflict rather than quietly widening or narrowing what that session may be handed. The service runtime producer reads the list back from the node's current session inside its own admit transaction and refuses node-authority-unsupported when the workload it is about to seal needs an authority the node did not resolve, instead of handing a node work it would silently park. Cloudly seals no network, secret or storage authority until those producers exist, so the workloads it admits today need none.

A sealed run workload also states where its node fetches the image bytes. A cluster is outbound-only, so nothing inside it dials Cloudly's registry directly: when the cluster has a relay origin published, the workload carries pullEndpoint — that origin as host[:port] — and the node pulls through its own relay. registryHost is never rewritten: it stays the publication identity the image release is bound to, and the digest pins the bytes, so the endpoint is untrusted for content and buildRuntimeAssignmentImageReference gives the same digest either way. A cluster whose relay an operator has not enabled states no endpoint and pulls from the publishing registry.

A relay carries one cluster, and only the nodes of that cluster. Cloudly resolves the cluster machine identity on the carrying socket to exactly one cluster for every registration it forwards, and the registration transaction requires the node that registration names to belong to that cluster, read from the node document the very same transaction fences. A node of another cluster is refused by name with relay-node-foreign, and a carrier whose cluster Cloudly can no longer name — because that cluster was deleted, say — is refused with carrier-identity-unverified rather than taken for a carrier that speaks only for itself. Both registration transactions apply the rule, so a refusal happens before the node's old packet authority is withdrawn and consumes nothing at all: no session, no session epoch, no moved node fence. A directly connected node is unrestricted as before: its own socket carries nothing but itself.

A changed nonce, or a changed credential generation, is a new registration: a new binding, a new session epoch, and the node's previous session fenced. An unchanged pair is a replay and returns the exact historical binding. A node restarting behind a live relay is therefore a new registration, and bearer rotation behind a relay needs no reconnect of the carrier: the node registers again with its unchanged nonce and its new generation on the same live socket, and the other nodes that socket carries keep their sessions. The carrying peer stays part of the identity because a committed session names the exact carrier it was registered on, so a registration that arrives over another socket is a new registration, exactly as it was before a node could name its process.

One protected server tag name marks a peer as a live Pallet session carrier for server enumeration, however many nodes it carries; TypedSocket caps server tag names per connection, so a carrier cannot mint one per node. That tag confers no authority and its payload only echoes the most recent registration.

Controller pushes are bounded per node, not per controller or per peer: at most eight in-flight assignment deliveries per node, and the node-keyed network delivery lane keeps at most one in-flight network delivery per node while later callers join the pending one. A single node reaches exactly the same ceiling it reached before, and one busy node can no longer exhaust the delivery budget of the other nodes on its peer.

A relay socket must sustain the delivery of every node it carries: each node keeps its own eight in-flight assignment deliveries and its own single network delivery, so one relay peer carrying N nodes can see 8 * N + N controller pushes on one socket. The socket is not what bounds that. With a realistic service configuration each push is roughly 40 KiB of canonical JSON, far inside both the contract's 320 KiB canonical limit and the 2 MiB frame the server accepts, and sixteen nodes each filling their own bound over one socket are delivered and acknowledged inside the 10 s each push is given. What bounds it is Cloudly's protected transaction layer: every runtime transaction takes a real write on the single controller record and on the node's own credential authority, so concurrent deliveries serialize and their optimistic retries cost quadratically. Measured on the runtime socket fixture, one delivery settles in 50 ms, eight concurrent in 0.9 s, thirty-two in 9 s, sixty-four in 32 s, and one hundred and twenty-eight exceed the database driver's 120 s transaction deadline and fail with a write conflict. Cloudly's own delivery pass never produces that burst — it is one joined lane walking one bounded page, one push at a time — so what protects a relay socket is the per-node ceiling together with that serial pass, and a caller that fans delivery out across a whole cluster at once must respect the same limit.

Runtime Service Authority

Every service-scoped runtime, network and credential write advances that service's runtime authority fence inside the transaction that commits it. Runtime and network callers name the intent explicitly, and the service credential paths use the admit default. The final removed observation of a relocated slot is the one accepted report that leaves the service fence untouched, because a replacement may already own the slot.

admit authorizes new work, and a deleting service refuses it: admitting or delivering a run revision (which includes issuing its registry pull credential), reserving a workload lease, staging network policy, setting private-network membership, and creating, rotating, revoking or exercising a service credential. settle carries execution that was already admitted to its end, and a deleting service accepts it: admitting or delivering a stop or removal revision, accepting terminal receipts and observations, quarantining a stopped workload lease, retiring the service credential during deletion, and every input a network compilation takes from the service (its executions and their observations, its reserved workloads, its network policy, and preparing its private-network membership, which may create an empty membership or detach networks that are retiring). A deleting service therefore still drains: Pallet reports on what it runs, its stop and removal revisions reach the node, and nodes that share a network with it keep compiling. This holds while the delete runs and after a failed delete released its lease, because the fence pins the service's loaded lifecycle state together with its lease: the live lease while one is held, and no lease once it is released. A run revision of a deleting service is never delivered.

Each accepted observation stores observationAcceptedAt, read from Cloudly's own wall clock (the runtime manager's now option) rather than the worker's observedAt. An exact replay keeps the earlier value.

Signed Pallet Network Delivery

The runtime manager prepares a dedicated Ed25519 signing authority at startup. Its private key is encrypted through SecretManager with controller-specific identity and environment-bound authenticated metadata. Restart preserves the authority; rotation and revocation require an explicit controller operation. Public revision history remains available for disconnected workers. Rotating replaces the encrypted private key, and revocation removes it.

RuntimeNetworkProjectionStore retains one complete signed intent per node in SmartData/NoSQLDB. A producer must fence allocations, memberships and current assignment observations in the controller transaction before admitting it. Admission verifies the signature, current signing authority and exact projection predecessor. An unacknowledged projection prevents a newer generation from replacing it. Re-signing unchanged contents under the current rotated authority requires a new admission ACK.

The joined runtime delivery lane sends the exact pending signing revision before its projection over the registered physical Pallet socket. Pending trust is persisted before transmission, so lost ACKs and reconnects replay the same key and then walk every later rotation. Replies must identify the exact sent key or projection. Credential revocation, reconnect, controller change and shutdown invalidate late replies. The server permits 2 MiB WebSocket frames; the shared projection contract limits complete canonical state to 896 KiB.

These ACKs prove admission only. They never establish readiness, renew the signed DNS deadline, clear packet withdrawals, or release addresses, handoffs or aliases. Activation barriers and Pallet realization remain required before this delivery foundation can activate workloads.

Network Reservation Ledger

RuntimeNetworkAllocationStore retains explicit protected-network revisions, pool cursors and complete handoff/workload reservations through SmartData. It uses the runtime controller's caller-owned transaction, including real node and service fences. A producer reserves the workload lease and admits its matching assignment in that same transaction. Startup prepares the store without choosing network settings or reserving addresses.

Before reserving either kind of lease, the store requires a current native protection receipt from every affected egress owner, including offline workers. The authenticated reportRuntimeNetworkProtection socket handler retains exact receipt history. A receipt names the worker's complete joined native protection journal; projection admission ACKs cannot substitute for it. The allocation transaction fences the current credential/session, receipt and node for every owner and retains the complete proof with each reservation. A reconnect, credential change, protection update or null native barrier blocks new allocations until the complete current proof is available. Historical outbox replay never restores eligibility.

Protection updates require the exact predecessor and retain existing pool definitions, protected coverage and offline egress authorities. Add a disjoint pool to expand capacity. Dedicated transit links reserve a /31; workload links reserve a /30 with two ordinary usable peers. Allocation cursors advance without reuse, and unique subnet, assignment and attempt indexes protect retained reservations. Lost replies replay the same immutable reservation.

Quarantine retains the full lease and its original protection/handoff bindings across restarts. A handoff cannot enter quarantine while it still has reserved workloads. Neither quarantine nor a newer protection revision makes addresses available again. Protection is required before reservation; activating reserved endpoints additionally requires native application. The native receipt producer and CNI application remain unfinished. The store does not issue application receipts, provide a generic fence override or release quarantine.

Complete Network Compilation

RuntimeNetworkCompiler reads reserved local executions, complete visible private memberships and their executions, and still-reserved endpoints from the previous projection. It includes declared names with no replicas and members with no aliases. Removing a local view retains live remote leases; explicit assignment stop intent can quarantine a lease. Retiring networks are detached through the membership owner, with revision and alias changes committed in the same transaction. Empty initial memberships are persisted as revision one.

Readiness requires the exact lease-bound run assignment, an accepted ready observation, and its current durable reporter session. A reconnect preserves old observation provenance but removes its readiness until a new observation arrives. Private packet grants do not depend on DNS readiness. Explicit service egress policy is persisted separately with revision CAS; missing policy blocks that service's compilation. DNS forwarding permission never grants internet or platform packet access.

Router-origin egress has its own explicit router.resolverIds and router.platformEndpointIds selections in the trusted compilation request. Both arrays are required, sorted and unique; empty arrays grant nothing. Selected IDs resolve against the transaction's current protected inventory and require the exact selected local handoff. A resolver grants its declared address and port over TCP and UDP; a platform endpoint grants only its declared tuple. Neither workload egress nor DNS forwarding policy supplies these selections. The signed projection retains full router-source and destination withdrawals across admission ACKs, including handoff replacement and endpoint tuple changes. An outstanding withdrawal blocks regrant until native application proves denial.

On a cluster whose ingress was designated for a Pallet-born cluster, the compilation adds workloadIngress: one grant from each ingress lease to each lease of every other service holding a slot in the cluster, on each TCP target port its public domains route to, where either lease runs on the compiled node. Both leases become endpoints of the projection even without a shared private network. A grant the successor drops is withdrawn explicitly and carried like every other withdrawal, and a projection that never granted such a flow omits the member and keeps its canonical bytes. hostPlatformEndpointIds names the protected authority's platform endpoints whose address the compiled node's own host carries: the listener and the managed VPN hub its cluster's relay registered from that node, and every address the node's current session reported for its host as hostAddresses beside its protection receipt. A plan's platform endpoint shares the relay listener's address when it runs beside the relay, and a host without a relay — a node with a Corestore beside it on one machine — names its own through hostAddresses; the node then delivers a workload's flow to such an endpoint locally. It grants nothing. hostAddresses describes the host as the current session sees it now, so a current report states them on a replay of its receipt too, a historical report states nothing, and a current report without them leaves the node's host carrying none (cloudly_runtime_network_node_host_addresses holds one record per node).

Compilation fences every contributing owner, signs locally under the current encrypted signing authority, and admits the complete projection in one controller transaction. Concurrent changes refresh the whole transaction or reject it. It carries exact withdrawals and retired material across admission ACKs; a regrant that conflicts with an outstanding withdrawal is rejected until the native application protocol can prove denial. No address or alias becomes reusable.

The runtime manager's trusted controller methods are stageNetworkProtection, stageNetworkPolicy, reserveNetworkHandoff, and stageNetworkSelection. No caller supplies an ad-hoc compilation request: compilation takes the node's staged selection, which names an exact handoff (or null for protection-only bootstrap) and explicit router selections, together with a controller-issued DNS window of at most 15 minutes. Startup seeds neither pools, policies nor selections. These methods join shutdown and are not worker RPCs. The controller-issued DNS window does not qualify a worker clock. Deployment admission, native application and Pallet-aware service deletion coordination remain prerequisites for live activation.

Node Scope

RuntimeNodeScope (ts/manager.runtime/classes.runtimenodescope.ts) is the one definition of the nodes Pallet runs, for every producer that needs it. A node is an egress member when it has an active Pallet credential, a current session, and its cluster's runtime phase is switching or pallet. Requiring a session keeps a node that never registers out of the controller-wide eligibility set; a node that registered and went offline keeps its session and stays a member. A member is assignable when its architecture states a container platform (x86_64 selects linux/amd64 and aarch64 selects linux/arm64); a member whose architecture states neither is listed with no platform and is not assignable.

Membership is read inside the caller's controller transaction: credentials page eight at a time and the cursor is the credential cursor, so a page may hold fewer members than credentials. Reading a node's current session joins that node's credential and document to the transaction. Each page reads the phase of each of its clusters once and fences the clusters Pallet owns, so a phase that moves before the caller commits takes the listing with it. A cluster Cloudly holds no phase for refuses the listing with cluster-runtime-missing; that refusal is controller-wide and not per node, so while one cluster states no phase nothing is produced for any node, and the cluster is named in the log once per process (cluster <id> has no runtime phase; network production is refused). A credential that outlived the node it was issued for names no cluster, so it names no member instead of refusing the listing, and retiring it belongs to the node's lifecycle. A listed node stays a hint that its producer re-fences; membership is never authority.

Address Plan

cloudly_runtime_network_address_plan holds one IRuntimeNetworkAddressPlan (id: 'address-plan'): the prefixes, pools, resolvers (at most 16), platformEndpoints and vpn an administrator gives the runtime network, under a revision compare-and-set. It is the one input every protected authority is composed from; startup writes no plan, and until an administrator sets one the runtime network is unconfigured.

setRuntimeNetworkAddressPlan({ identity, expectedRevision, plan }) and getRuntimeNetworkAddressPlan({ identity }) are admin-guarded. expectedRevision is the revision the caller read, or null when no plan exists; anything else is refused as conflict. The contract validates the plan (snapshotRuntimeNetworkAddressPlan) by building the authority its own sets describe, so no plan can compose an authority the protected-authority contract refuses, and a plan whose two VPN prefixes — vpn.controlPrefix for the tunnel's control addresses and vpn.hubPrefix for the underlay every cluster hub binds inside — are not reserved inside the protected space it states is already refused there. The hub endpoints themselves are no administrator's input: each cluster's relay reports the one it bound and the composition turns it into platform endpoints. The contract also decides the change (admitRuntimeNetworkAddressPlanChange): leases and the managed VPN outlive a plan revision, so every pool stays exactly as it was, the protected space only grows, and the VPN never moves. Equal content is not a write: the stored revision stands and the document is left alone.

Every refusal is a TypedResponseError carrying its name, and keeps the error that caused it for Cloudly's own diagnostics:

• conflict — the caller stated a revision that is not the stored one.
• plan-invalid — the contract refused the plan's structure without naming a rule of its own (overlapping pools, an endpoint outside the protected prefixes, a control prefix longer than /30, a duplicate endpoint tuple).
• plan-vpn-immutable, plan-pool-removed, plan-prefix-shrunk — the contract's three change refusals.
• plan-vpn-foreign — the plan states a managed VPN other than the one this controller already runs. The configuration is written once and never moves, so a plan that states a different control or hub prefix would leave the running VPN outside the address space it must sit in.
• plan-resolver-referenced — a resolver the change removes or alters is still named by a router selection (by id) or by a private network's external DNS upstreams (by address and port).
• plan-endpoint-referenced — a platform endpoint the change removes or alters is still named by a router selection or an authorized service policy.

The plan is also where Cloudly's dcrouter gateway client takes its DNS record allowance (allowedDnsRecordAddresses) from: vpn.hubPrefix and every platform endpoint address as a host prefix, which are the two places a cluster relay binds its listener. A committed change re-derives it, and a changed allowance makes the held gateway credential's policy stale, so the gateway settings ask for a re-bootstrap, which is how dcrouter learns it. A plan whose allowance would exceed the gateway contract's 64 prefixes is refused plan-invalid.

mutatePrivateNetwork states the upstream rule from the other side: a private network may only forward to an upstream the plan states as a resolver, matched by address and port, and anything else is refused as upstream-unprotected. A network that denies external resolution names no upstream and reads no plan. The plan document is fenced inside the private-network transaction, so a plan write that removes the resolver a mutation relies on conflicts instead of stranding it. The private network manager therefore starts after the runtime manager, which owns the plan store.

Protection Production

The protection producer composes the one protected authority this controller publishes, id: 'runtime-network-protected', from the address plan and the nodes RuntimeNodeScope states. The plan is the whole address space; the producer adds only what a plan cannot state: the authority's identity, its revision chain, and its egress authorities, which are the current ones plus every node now in scope. Each cluster's selected hub is added as a UDP platform endpoint by the contract's own composeRuntimeNetworkProtectedAuthority, so the authority a plan describes is composed in exactly one place.

One pass reports one outcome:

• unconfigured — no plan. Nothing is read further and nothing is written.
• no-members — a plan, but no node is a member yet; a protected authority grants egress to at least one node, so none can be composed.
• unchanged — composing the plan and the current egress at the current point of the chain reproduces the current authority byte for byte, digest included.
• protection-propagating — the named current egress nodes hold no projection head, acknowledged or still pending, citing the current authority.
• staged — the successor was staged.

Egress only grows: a node that goes offline, loses its credential or leaves a Pallet-phase cluster keeps its grant until an independent fence can retire it, which is also what the reservation ledger enforces. Every join of one pass goes into one bump, because a bump clears every managed VPN member: staging one authority per joining node would drop every tunnel once per node. A successor is staged only once every current egress node's head cites the current authority, so the fleet is never asked to apply two generations at once.

The pass plans first and only a real change re-decides inside the managed VPN withdrawal, so an unchanged fleet keeps its tunnels. Planning is not a read-only transaction: it stages nothing and touches no projection unless the content differs, but it takes every fence its reads need — the controller record, the protected authority and the phase of each cluster it pages, and on a differing pass every current egress node and its projection head. The propagating gate is exactly that differing-pass read: every fence is a write, so a steady window leaves every projection exactly as its node acknowledged it. The authority is composed from the plan's own sets plus one UDP platform endpoint per cluster hub, read inside the deciding transaction: a hub that moves is therefore a successor of its own, and a hub that only re-keys is not, because a public key is not part of the authority. After a pass that states a current protected authority the managed VPN is configured from the plan, once per process: a replay is safe, and the configuration never moves afterwards, so a later pass has nothing left to configure.

Two bounds come with that. The contract admits at most 256 egress authorities, but a differing pass rewrites every current egress node's projection document inside one transaction, so the fleet size this gate actually carries is far below that number; a gate that pages those reads, or that compares the stored revision instead of replacing each document, is what would raise it. And a grant outlives the node that holds it: both the gate and the staging fence require that node's document, so deleting a node the current protected authority grants egress to would wedge every later differing pass. Node deletion therefore refuses it — getNodeDeletionImpact states the blocker node holds a runtime network egress grant beside the heartbeat and deployment blockers, and deleteNodeById refuses on it — until an independent fence can retire a grant.

LevelTriggeredSweep schedules the pass on the same period, hint delay and floor as network production, and every committed change hints all three producers. produceNetworkProtection() and inspectNetworkProtection(), for administration and tests, join the pass in flight or run one at once.

Node Network Production

The node network producer gives every node RuntimeNodeScope states the two things its own traffic needs before anything can leave it: one reserved transit handoff, and the selection the network compiler reads. Both derive from durable state alone. The node's router incarnation, stored in cloudly_runtime_router_incarnations as {id: <nodeId>, incarnation}, is written at 1 inside the transaction that reserves the first handoff, and the handoff's id and the router incarnation its leases cite are sha256 over the domain, the node and that counter — so a controller that restarts between reserving and staging recomputes the same id, and the reservation replays instead of allocating a second transit range.

One pass over one node reports one outcome:

• unconfigured — no address plan, or no protected authority yet. Nothing else is read and nothing is written.
• egress-pending — the current protected authority does not grant this node egress.
• protection-receipts-pending — the named egress owners hold no current protection receipt or reporter session, so no reservation on this controller is eligible. The owners are read before the attempt rather than caught from it, because a reservation refuses an ineligible fleet without saying who owes what.
• transit-pool-missing — the address space states no transit pool to allocate from.
• handoff-quarantined — this incarnation's handoff was quarantined; only a reincarnation gives the router a fresh one.
• unchanged — the staged selection already names the live handoff and the current router destinations, so the pass leaves the node's own documents exactly as they stand.
• staged — the selection was written at its next revision.

The selection names every resolver of the current protected authority and the hub of the node's own cluster. The resolvers come from the authority rather than from the plan they came from, because a selection may only name what the current authority states: a plan revision that adds a resolver before the protection producer has staged its successor would otherwise be refused. The hub's endpoint id is derived by the contract from the hub the cluster's relay advertised (runtimeNetworkVpnHubEndpoints), so the selection, the composition that sealed the authority and the credential binder all name it identically — without it in the selection no node is ever admitted as a managed VPN member. A node may name only its own cluster's hub, because every inter-node packet stays inside the cluster; a cluster whose relay advertised no hub, or one the plan's hubPrefix does not cover, leaves its nodes with their handoff and resolvers and no VPN membership at all. A staging pass fences the address plan, so a plan write that drops a resolver the selection names conflicts with it instead of stranding it.

getRuntimeNetworkNodeReadiness answers whether one node's runtime network is ready for workloads and every condition it still misses, in the order the producer meets them: egress-pending, protection-receipts-pending with its owners, handoff-pending, selection-pending, projection-pending and vpn-member-pending. Ready is exactly "nothing is missing": the node is in egress, holds a reserved handoff, its selection names that handoff, its acknowledged projection head carries it, and this controller process holds a managed VPN member for it. Readiness is a readout, never a step of the sweep — reading the acknowledged head fences the projection row, and a pass that asked it for every node would rewrite every projection on every period — and the receipts of the fleet are read only while the node holds no handoff, because current receipts gate the reservation and nothing else. Membership is process state, so a Cloudly restart withdraws readiness until each node re-fetches its managed VPN credential.

reincarnateRuntimeNetworkRouter is the one lever over a node's router: it advances the incarnation against the one the caller read, and the next pass reserves the successor handoff and stages the selection naming it. It invalidates what the previous handoff carried — the next compilation quarantines it and retires it in the node's projection, the managed VPN member falls with that projection, and readiness returns to handoff-pending until the pass has run. Its refusals reach the administrator by name: conflict for an expectation that is not the stored incarnation, router-incarnation-missing for a node that never held a handoff, and router-handoff-in-use while a workload lease still cites the current handoff, because the next compilation could not quarantine a handoff a lease holds and that node would never compile again.

Three bounds come with the producer. A node retains at most 256 handoff leases, so that is how often its router may be reincarnated for the lifetime of its document, and no path releases one before an independent fence exists. Every node's selection names every resolver the plan states, so a plan write that removes or changes a resolver is refused as plan-resolver-referenced while any node holds a selection; removing one is a maintenance path, not a plan edit. And a steady pass is not a free pass: each node runs in its own transaction, which fences the controller record, and the only reader of the protected authority transitions that singleton, so one sweep over N nodes rewrites both documents N times even when no node changed. Correctness holds — the transaction driver retries the transient conflict, and a producer that may write from that reading holds the fence it takes — but the write rate of an idle fleet scales with its size, and lowering it means one protection reading shared by the whole pass.

LevelTriggeredSweep schedules this pass on the same timing as the other two producers. Every committed change hints it, and a protection pass that staged a successor hints it as well, because that successor is what a node's first reservation waits for. In the other direction only a staged selection is a committed change, and network production is the one producer waiting for it, so that is the only hint a pass sends: a pass that changed nothing never asks the fleet to sweep again. produceNodeNetworks() and inspectNodeNetworkProduction() join the pass in flight or run one at once.

Service Runtime Spec

IServiceRuntimeSpec is what Cloudly runs for one service on one Pallet cluster: the cluster, the placement (every-node or a sorted list of pinned node ids), the replica count per node, a hold, an execution profile and a network mode. It is stored complete in cloudly_service_runtime_specs, one document per service/cluster pair, and never defaulted at read time. Its private persisted id derives from that pair, while its public id remains the service id. It lives beside the service document because a service update merges caller data and because a hold a cutover writes must not race an operator's edit of the same document. Its revision is the compare-and-swap: setServiceRuntimeSpec takes expectedRevision, where null states that the caller read no spec for that exact cluster, and every accepted write advances that pair's revision by one and replaces every field — nothing is carried over from the revision it replaces. A revision read from another cluster can never satisfy the compare-and-swap.

A spec write runs under the service's lifecycle lease, which serializes it against a service update or deletion across Cloudly processes, and inside that lease one transaction fences the cluster's runtime phase, so a phase that moves before the commit takes the write with it. Refusals reach the caller by name: service-not-found, service-deleting, service-organization-missing, conflict, spec-node-not-in-cluster for a pinned node that belongs to another cluster, spec-readiness-target-undeclared for a TCP or HTTP probe naming a port the service does not declare, cluster-runtime-phase-refused for a cluster still in swarm, whose placement is Coreflow's, and cluster-runtime-missing for a cluster id Cloudly holds no phase for at all. A spec is accepted in phases draining, switching and pallet. A numeric expectedRevision against an absent service/cluster pair is a conflict, as is null against that pair when it already exists. getServiceRuntimeSpecForCluster reads one exact pair and getServiceRuntimeSpecs pages every pair in cluster order with its matching status. The released getServiceRuntimeSpec scalar answers zero or one target and refuses two or more as service-runtime-spec-ambiguous; it never chooses one. The assignment producer alone writes the pair-local verdict in cloudly_service_runtime_status, so it reads as null until that target is decided. removeServiceRuntimeSpecForCluster derives and fences the current phase, revision, hold, terminal slot evidence and release inside its transaction; it removes only that held pair. Deleting a service removes every target and verdict inside its deletion.

createService writes the default spec only where the cluster leaves no choice: the one cluster Cloudly holds a phase for is a cluster Pallet owns (switching or pallet). With no cluster, with more than one, or with any Coreflow-owned cluster in the fleet, a new service starts with no spec and the operator states one with setServiceRuntimeSpec, or the cutover writes it. That write fences the cluster's phase exactly as an administrator's does, so a cluster that leaves the Pallet-owned phases while the service is being created refuses the creation by name rather than binding the service to a cluster Coreflow took back. The default states placement: every-node, hold: false, network: isolated, cpuMillis: null, runAs: null (the image's own OCI user), readonlyRootfs: false, and:

• replicasPerNode is the service's scaleFactor when that is an integer within the contract's bound of 64, and 1 otherwise — a Swarm scale factor outside the Pallet bound states nothing about a Pallet replica count.
• memoryBytes is resources.memorySizeLimitMB in bytes, or 512 MiB when the service states no limit. A stated limit the runtime contract cannot express refuses the creation as service-runtime-spec-memory-invalid; nothing is substituted silently.
• readiness is a TCP probe on the service's first declared target port when that port's transport is TCP-mapped (http, https, ssh or tcp) and its name is one a runtime config can name, and a process probe otherwise, including when the service declares no target port at all. Probe timing is 1s initial delay, 30s startup grace, 5s interval, 2s timeout, two successes and three failures.

An attached network states publicEgress and the platform endpoints a workload may reach. Cloudly supplies no default for either: the contract requires an attached spec to state publicEgress explicitly, and a write that omits it is refused rather than completed. Admission requires an explicit service network policy that matches both fields, the selected handoff for the placed node, its current protected egress authority, and a free workload /30. The allocation owner chooses the lowest-id current workload pool with capacity; the service spec cannot select a pool. Reservation and the exact assignment admission share one controller transaction. A run or stop admission requests network compilation, which projects or quarantines the retained lease. Missing policy, mismatched policy, missing workload pools and exhausted workload pools have distinct service refusal reasons.

Once a service has any spec, those specs own its placement and replica counts: a generic updateService that states placement or scaleFactor is refused as service-runtime-spec-owns-placement, whether or not it repeats the stored value. An update that states neither reads no spec at all. The ports stay the service document's, but a spec's probe names one of them, so an update stating targetPorts or ports that would leave the stored probe naming an undeclared port is refused as spec-readiness-target-undeclared — the same answer the spec write gives.

Service Runtime Production

The assignment producer is the writer that makes every runtime target of a service true. One pass covers all of that service's targets and has exactly three kinds of step: one transaction that reads everything and writes nothing, the build that resolves what the pinned release executes, and one transaction per slot that actually acts. A service whose desire already stands costs the first of those and nothing else.

Desire. A spec desires nothing while it holds the service, while the service is being deleted, or while its cluster is not in phase pallet — switching prepares the network, it does not run workloads. Otherwise the desire is selectReplicaPlacement over the nodes RuntimeNodeScope calls assignable: every-node takes every node of the spec's cluster whose reported architecture states a container platform, and nodes takes exactly the pinned ones that are assignable. Each node carries replicasPerNode replicas named <nodeId>.<index> from zero, so a replica never moves node: an offline node keeps its replicas, and only retirement, credential revocation or a spec change moves them. A pinned node that is momentarily not a member leaves its replica unplaced and the service converging; only a desire that is empty while the spec wants at least one replica is refused, as no-eligible-node.

Object-storage environment. A target runs with the service's own environment and the names of every enabled object-storage binding of the service that is not tombstoned. Cloudly places each binding first, by the rule the relay's placements follow: on the node its legacy adoption names, otherwise on the one node the spec pins. A binding that lands on no node of the target's cluster — a spec on every node or on several, or a node that is not the cluster's — makes the target wait as bindings-pending. The rest is the contract's composeObjectStorageServiceEnvironment over each binding and its node's corestoreObjectStorage (see Object-storage endpoint): the nine endpoint names, and the eleven retention names once retention is proven. A binding that declares retention without evidence waits as objectstorage-retention-pending; a node that states no valid endpoint is refused objectstorage-endpoint-missing, a binding the contract cannot compose from objectstorage-binding-invalid, and a name the service or another binding already sets objectstorage-environment-conflict. A refusal outranks every wait, and among waits the lowest binding id answers. A target that waits or is refused here decides nothing: no run and no stop, so a replica already running keeps running while its target reports why. The composed environment replaces the service's own in the desired key and in the built config, where the image's defaults stay beneath it.

Key. desiredAttemptKey is sha256 over the target cluster id, rollout id and generation, the pinned image digest, the spec revision, the composed environment (so a moved object-storage endpoint is a drift), the node's platform and the network mode — database values only, so a commit can recompute it under its own fences. desiredAttemptId binds that key to the removal the attempt succeeds, and the assignment id derives from the attempt id, which is what makes drift detectable without storing the key: recomputing the attempt id with the incumbent's own stored predecessor yields the incumbent's attempt id exactly while the desire has not changed. The network lease is deliberately excluded, because its own id derives from the assignment id.

Transition. decideSlotTransition is pure and exhaustive, and every case is named:

case	the slot	the step
admit-run-first	no slot, and a desire	run, generation 1, no predecessor
admit-run-successor	removed with both receipts, and a desire	run, generation 1, citing the removal
settled	removed with both receipts, no desire	nothing
running	at the desired attempt, not failed	nothing
drift	at another attempt than the desire	stop, generation 2
undesired	running with no desire	stop, generation 2
failed-elapsed	failed, backoff elapsed	stop, generation 2
failed-waiting	failed, backoff pending	wait, failure-backoff
stop-receipt-pending	stopped without its receipt	wait
remove	stopped with its receipt	remove, generation 3
remove-receipt-pending	removed without its second receipt	wait
replacement-in-flight	drifted while another slot is replacing	wait

The failure backoff is min(10 s · 2^n, 10 min) measured from observationAcceptedAt, Cloudly's own clock when it accepted the observation, with n the number of consecutive earlier attempts of that slot that ran this exact desire — the restarts the producer itself already made. Those ancestors are read from the attempt chain, at most eight back, and the walk stops at the first one that ran something else.

A service replaces one slot at a time: while any slot sits between its stop and a new run that has been observed ready, another slot's drift waits. The rule binds drift only. An empty desire — a hold, a deletion, a cluster that stopped running the service — stops every slot at once, because nothing will become ready and a cutover depends on the whole service draining.

Build. Every step that has a desire behind it is built before anything is committed, and a refusal is local to the target whose desire could not be built: stopping a replica because its desire changed or failed is the first step of replacing it, and a replacement this runtime cannot build must not begin. That is what makes a refusal a verdict about new work and never the reason a running slot was retired. The steps that carry no desire are the other half of the same rule and a refusal never holds them back: the stop of a replica the service no longer wants and the removal that follows any stop are committed while the build is refused, so a scale-down, a hold or a deletion finishes even while the registry cannot answer. resolveRuntimeImageMaterial answers what the pinned release executes — registry I/O, so it runs outside every transaction and once per pass — and buildRuntimeConfig seals the rest from values alone: the image's environment with the composed environment over it, the image's working directory or /, the profile's resource and identity policy, and the service's declared ports, where udp stays udp and http, https, ssh and tcp become tcp. The config id is config-<sha256> over its own content at generation 1, so equal content is the same document. A service declaring more target ports than a config carries has no refusal name in the contract yet, so its pass fails with that count and the sweep retains it by service id, rather than borrowing a name that would say something untrue; the ports-too-many refusal arrives with the next interfaces minor.

Commit. Each acting slot gets one transaction that decides the same slot again from freshly read inputs and refuses conflict unless it reaches the same case and, for a run, the same desired key: a spec, a service, a rollout, a phase or a membership that moved under the pass invalidates the step rather than committing it. Only then does it fence the cluster phase, require the node's runtime network to be ready — else the service waits as node-network-pending —, require the desired secret manifest to carry the published credential of every enabled Corestore binding of the service — else it waits as bindings-pending, so a first start never runs without its database credential, or as legacy-adoption-pending while the relay reports such a binding refused until its legacy resource is adopted (see Corestore Legacy Resource Adoption) — and hand the sealed revision to admit, which fences the service document's whole data, the node and the slot. Readiness is asked on a committing pass alone, because reading the acknowledged projection head rewrites that node's projection row.

An attached run seals its reserved network lease in the admission transaction. A run with a current resolved secret manifest seals its independently hashed secret authority and retains every pinned version in that same transaction. The node must have resolved each authority it needs. Cloudly refuses a missing or invalid secret plan as secrets-unsupported; a node without the secrets capability is refused as node-authority-unsupported, and so is a node without storage for a service that declares volumes.

Local storage claims. A declared volume runs as a node-local local storage claim (IRuntimeLocalStorageClaim), one per volume and node, sealed into the run's workload.storage. A claim is one node-local directory mounted ReadWriteOnce, so a volume whose driver is neither absent nor local is refused volume-driver-unsupported, driver options are volume-options-unsupported, a volume shared by source, one mount path declared twice or a spec with more than one replica per node is volume-shared-unsupported, a placement that does not pin exactly one node is volume-placement-unsupported, and more volumes than an assignment seals (64) is volumes-unsupported. A volume's name is Coreflow's Docker name and backup its snapshot scheduling; neither names anything on a Pallet node.

The claim volumeId is volume- followed by the SHA-256 of ["cloudly-service-volume", mountPath]: a service names its volumes by mount path, so a volume whose mount path changes is a new volume with a new claim, and the old claim is retained on its node. The claim id is derived from the controller's kind and id (not its epoch), the organization, cluster, node, runtime namespace, service and volumeId, so it is stable across every generation. The first generation takes its initialization and initialOwnership from the volume's stored initialization through deriveServiceVolumeClaimInitialization under the spec's runAs: a volume without one is refused volume-initialization-unknown, and a run-as ownership under runAs: null volume-ownership-unknown, never created empty. Every later generation keeps what the first fixed, whatever the initialization or runAs says by then; only the mount (readOnly) and the controller move, and a mount change replaces the replica. The claims are prepared, sealed into the run, bound with bindRuntimeLocalStorageClaimToAssignment and stored in the admission transaction, so no claim is stored for a refused run and no run references an unstored claim. Cloudly never issues a purge: the reclaim policy is retain.

Every claim is replayed to its node over the node's session (applyRuntimeLocalStorageClaim) on each delivery pass, before the assignments of that pass, and the answer for the claim's current generation is recorded in cloudly_runtime_local_storage_claims. While a claim of a target answers import-required, the target waits as storage-import-pending: its run is admitted and waits on the node until the operator's offline import lands, and the next replay that answers ready moves the verdict on.

A Pallet node enrolls a public X25519 recipient through a one-time possession challenge on its current authenticated runtime session. Its private key stays in process memory; reconnection requires a new enrollment generation before Cloudly issues fresh material. For an admitted secret-bearing run, Cloudly checks the current assignment, node session, recipient generation, accepted manifest and indefinite version holds before encrypting the pinned values to that recipient. The response is bound to the full run assignment and carries only sealed envelopes. An environment-delivered entry also requires the currently approved WorkloadInit artifact; Cloudly never places a secret value in runtime config, an assignment, or a log. The holds end only when the same run's joined native remove receipt is accepted, in the receipt transaction. Material issuance stops when a manifest leaves the current desired or previous-acceptance window, even while an old run remains retained for safe cleanup.

Verdict. cloudly_service_runtime_status holds one verdict per service/cluster target about that target's spec revision, and it is written only when it changed, so changedAt says when the verdict last moved rather than when it was last confirmed. converged means every desired replica runs its desired attempt and has been observed ready and nothing is pending; converging is everything else that is neither waiting nor refused; waiting names node-network-pending, conflict, failure-backoff, storage-import-pending, bindings-pending, legacy-adoption-pending or objectstorage-retention-pending; refused names cluster-runtime-missing, organization-missing, image-plan-missing, platform-unsupported, secrets-unsupported, volumes-unsupported, port-protocol-unsupported, readiness-target-undeclared, environment-too-large, no-eligible-node, replica-limit, node-authority-unsupported, one of the four volume-*-unsupported names above, volume-initialization-unknown, volume-ownership-unknown, objectstorage-endpoint-missing, objectstorage-binding-invalid or objectstorage-environment-conflict. A service with no spec and no slots is produced as unconfigured and writes no row at all.

Volume initialization. IServiceVolume.initialization states how a volume's first claim starts (empty with an owner and mode, or import of an existing Docker volume's bytes). setServiceVolumeInitialization { serviceId, mountPath, expected, initialization } is its only write: expected is the value the caller read (null for none), the stored value again is answered unchanged, and the refusals are IServiceVolumeInitializationErrorData: volume-not-found, conflict, volume-initialization-invalid, and volume-initialization-bound while a bound claim of the volume exists on any node. The claim read and the write share one transaction with the service document that the claim-issuing admission fences, so the two cannot pass each other. createService and updateService refuse a volume that states an initialization (SERVICE_VOLUME_INITIALIZATION_SERVER_MANAGED) unless it is exactly the value already stored for the same mount path — what a read-modify-write round trip sends back, in any key order — and keep every stored initialization with the volume of the same mount path; a removed volume takes its initialization with it, and two volumes at one mount path are refused SERVICE_WRITE_INPUT_INVALID.

The sweep is the fourth LevelTriggeredSweep, on a 15-second period rather than the network producers' 30, because it is what an operator waits for. It pages target rows but emits each unique service once, then visits first and then every service that still holds slots, which is how a service whose spec was removed is still visited by the pass that has to settle its slots; those pages seek in service order, so a service whose slots straddle a page boundary is still a candidate exactly once per sweep. Every committed runtime change hints it through afterRuntimeChange, and a spec write hints it directly through requestServiceRuntimeSweep(). produceServiceRuntimes() and inspectServiceRuntimeProduction() join the pass in flight or run one at once, and listServiceSlots(serviceId) reads one service's slots without taking a single fence, bounded at one slot past the replica limit, so a service holding more of them is refused as replica-limit rather than read without end.

Two bounds come with it. A pass reads the node scope for every service it visits, so an idle fleet's read rate scales with services times nodes, and each transaction fences the controller record and the phase of every cluster it took members from, exactly as the network producers do. And until the two-phase deletion lands, a service that ever held a slot cannot be deleted: deleteServiceById refuses service-runtime-active while any slot row exists, before it takes the lifecycle lease, because taking that lease marks a service deleting for good and a refusal inside it would strand the service.

Service Runtime Views

What the producer admitted is served back through two reads that write nothing, take no fence and derive nothing. getServiceRuntimeAssignments({ serviceId }) answers one service's replica slots in replica order. getClusterRuntimeAssignments({ clusterId, cursor, limit }) answers one cluster's slots a page at a time in slot order; limit runs from 1 to serviceRuntimeContract.maximumAssignmentPageSize (256) and nextCursor is the slot id the page ended at. A cluster carries every node's slots of every service, so every slot states the cluster it runs on — taken at admission from the node that admission already fences — and the page seeks that cluster's slots alone: a page holds exactly limit rows unless the cluster holds fewer.

Both answer the same row, projected in serviceruntimeview.schema.ts from the stored rows alone: serviceId, replicaId, nodeId, the current assignment reference, the disposition, terminalReceiptCount, the assignment record's own observation (phase, ready, observedAt, sequence) and delivery. Delivery is delivered once the node's own evidence cites the exact revision that stands — an observation that names it, or the terminal receipt that does, since receipts join in order — and undelivered until then, because Cloudly keeps no acknowledgement journal: a revision the node admitted but has not reported on reads as undelivered.

The service read passes the same inspectable-service guard as getServiceRuntimeSpec, so an administrator or the service's own organization may read it and nobody else. The cluster read is an administrator request, like getClusterRuntime. A page size or a cursor the contract does not admit is refused by name, as assignment-page-limit or assignment-cursor-invalid.

The dashboard renders those rows as a runtime tab on the service, node and cluster details; a node's tab is its cluster's page filtered to that node. A cluster tab walks at most eight pages of 128 rows, and a reading that ends with the cursor still open says so in its heading rather than showing the rows it kept as if they were the whole cluster. It refreshes on pushServiceRuntimeChanged, which carries only a service id — the UI reads the views again — and travels on the serviceRuntime UI live channel, so a UI holding none of those views open is never woken by it. The push is emitted from exactly one place: the service runtime pass announces when what an administrator would read moved. Every committed runtime change already hints that pass, so its own admissions, an accepted observation, a joined receipt and a spec write are all covered by it, and a repeated report or a steady service announces nothing at all.

Service Deletion and Node Retirement

deleteServiceById is the first of two phases. It takes the service lifecycle lease with delete intent, which marks the service deleting durably — every later admission is refused by the service's own runtime fence, and only settlement is still accepted — and then removes everything Cloudly owns that nothing running depends on: the external gateway routes and the DNS entries, web push, mail and CoreMail bindings, deployments, platform bindings and their Corestore delivery, backups, the registry repository, OIDC, the service-owned image and the deployment claims. Coreflow's reconciliation of the service is quiesced whenever any configured cluster remains Coreflow-owned, including a legacy cluster for which that service has no Pallet target row. It skips the fence only when every current cluster is Pallet-owned. Each phase asks that once, while the spec that answers it still exists, and holds the quiesce for the whole phase, so the row deletion inside the second one never asks again for itself.

What Pallet runs is retired by the service runtime pass, which desires nothing for a deleting service, so its slots walk run, stop and removal against the node's own receipts. The second phase belongs to the retirement finalizer, a level-triggered sweep over every service a deletion left deleting. Every pass runs the first phase again before it reads anything: a request that failed midway through it left the service deleting with routes, DNS or bindings still standing, and every step of that phase removes what exists, so the re-run is what keeps a deletion from finishing over its own leftovers. Then it reads what the runtime still owes and names the wait it is in:

• slot-settlement-pending — a replica slot has not reached its removal with both terminal receipts joined;
• lease-quarantine-pending — a workload lease of the service is still reserved, which the network compiler quarantines when it sees the stop intent;
• projection-acknowledgement-pending — a node the service reached has no acknowledged projection head, or the head it holds still carries one of the service's endpoints. The nodes asked are the ones its slots stand on together with every node a workload lease of the service names, quarantined leases included, so a replica that moved keeps the node it left in the wait until that node says it dropped what it carried.

Only once nothing is owed does the finalizer take a fresh delete lease and finish the service: it revokes the service-owned secrets and the platform credential, deletes the network policy and the private-network membership that authorized its workloads, releases the replica slots together with the attempts and the runtime configs only they cited, removes the runtime spec and verdict, and runs the fenced service.delete() against the runtime generation the release just advanced. A slot release refuses a slot the node has not removed, so a half-finished deletion is impossible; a service nothing ever ran for needs no second pass and is finished by the request that started it. The sweep is level-triggered, so a deletion interrupted by a failure, a restart or a slow node is finished by the next pass rather than by a retry somebody has to issue.

deleteNodeById refuses a node that still holds runtime state, and getNodeDeletionImpact names each blocker: the replica slots that run on it, a reserved workload lease or transit handoff, each active spark or pallet credential, and the egress grant the protected authority holds for it — a grant only grows until an independent fence can retire it, and deleting the node it names would refuse every later protection pass. The node document and its removal from every cluster snapshot are written in one node-fenced transaction, so an enrollment, a heartbeat or a runtime write that lands meanwhile conflicts instead of leaving a relay dialing a node that is gone. Credentials are revoked through revokeNodeCredential before an enrolled node is deleted.

Network Production

stageNetworkSelection({ nodeId, revision, handoff, router }, expectedRevision) retains the durable per-node compilation input in cloudly_runtime_network_selections under a revision CAS, fenced against the node. A selection that could never compile is refused: router choices without a handoff (invalid), a handoff that is not the node's current reservation, or a resolver or platform endpoint the current protected authority does not name (inactive). A node without a staged selection compiles the protection-only bootstrap: no handoff and no router egress. Neither inventory presence nor policy stands in for a selection.

RuntimeNetworkProducer is the production trigger for compilation. It pages the nodes in scope and produces each one; the scheduling around that pass is LevelTriggeredSweep (ts/manager.runtime/classes.leveltriggeredsweep.ts), which owns the period, the hint delay, the floor and the single timer described below. Every sweep visits every member in pages of eight, so correctness never depends on a hint arriving, and restarts and other controller instances are covered because membership is read from the database.

These committed changes ask for a sweep without naming a node: session registration, assignment observations and terminal receipts, network protection receipts, stageNetworkProtection, stageNetworkPolicy, setServicePublishedPorts, reserveNetworkHandoff, stageNetworkSelection, setRuntimeNetworkAddressPlan, mutatePrivateNetwork, setServicePrivateNetworks, node enrollment (which activates a Pallet credential) and the rotation or revocation of a Pallet credential. Each of them reaches both producers, because a membership change may move a projection and the protected authority alike. Anything else, such as service deletion, is picked up by the periodic sweep. One sweep runs at a time. The next one starts once hints have been quiet for one second, since every hint restarts that wait, or 30 seconds after the previous sweep ended, whichever comes first. No timer-driven sweep starts sooner than five seconds after the previous one ended, and a hint arriving mid-sweep earns exactly one follow-up sweep. These waits are measured on the monotonic clock, so a wall-clock step neither stalls nor spins production. produceNetworks(), for administration and tests, joins the sweep in flight or runs one at once without waiting out the five seconds.

Each node is produced plan first. Planning reports unconfigured (no protected authority), awaiting-acknowledgement (the head is unacknowledged and signed by the current authority), unchanged, or a change. It commits only idempotent safety writes that never advance the chain: fences (an absent private-network membership is materialized empty so it can be fenced), stop-intent workload quarantine, replaced-handoff quarantine and the detachment of retiring private networks. They stay committed when the commit that follows fails, and the next commit rebuilds on top of them. Only a change takes the managed VPN node lane and recompiles inside the committing transaction, so an unchanged node never has its packet authority withdrawn. A network is unchanged when the candidate matches the acknowledged head in every field except chain position, digest and DNS window and the head carries the current signing authority; a rotated authority therefore signs a successor of the same network. An unacknowledged head signed by a rotated authority could never be delivered, so the current authority re-signs it unchanged at its own generation. Either way the new key is delivered before the projection. Evidence follows the same rule as the network: an endpoint the node already holds keeps the observation it already applied unless its readiness flipped, so a repeated probe is not a network change and does not restart a peer's data plane. A compiled projection is pushed immediately, and the 5-second delivery lane remains the retry path. inspectNetworkProduction() returns a copy of the last sweep's per-outcome counts with up to eight named per-node failures. Shutdown cancels the timer, joins the sweep in flight and abandons its remaining nodes; nothing is admitted after stop.

The DNS window is not network state: an otherwise unchanged network is not re-signed for a new window, because sameNetworkApplication ignores dns. Windows start on a ten-minute grid and last the contract maximum of 15 minutes, so nodes compiled in one slot share one window. A controller window never moves backwards: a successor states the slot of the later of the controller's wall clock and the window the node already holds, which is its acknowledged renewal's or else its head's, so a clock stepped back issues that slot again instead of a successor the projection contract refuses or a window earlier than the node's own. A clock that ran ahead issued windows that are not valid yet; once it is corrected, successors keep that slot until the clock reaches it, and nodes wait out the not-yet-valid window. A node whose network stays unchanged keeps its DNS through signed renewals instead (see DNS Lease Renewals).

DNS Lease Renewals

A renewal extends one node's DNS window under its exact acknowledged projection and changes nothing the node applied, so a steady network keeps its DNS without a new projection restarting its data plane. Its sequence is the slot index, issuedAt / 600000, and its window is that slot's: it opens at the slot's start and lasts the contract maximum of 15 minutes. Every field follows from the node scope, the projection reference and the slot, and Ed25519 signatures are deterministic, so signing a slot again yields the byte-identical envelope and a retry never puts two windows under one sequence. A renewal is signed only by the authority revision that signed its projection, inside the transaction that fences the signer against rotation; after a rotation, only a newly signed projection continues DNS.

RuntimeNetworkDnsLeaseStore keeps one row per node in cloudly_runtime_network_dns_leases: the projection its renewals extend, which is none until the node acknowledges its first, the latest signed and the latest acknowledged sequence, and dnsAuthorityHorizon, the latest window end Cloudly ever signed for the node under any projection. The horizon never decreases; nothing tied to the node's names may be released before it. It is raised in the transaction that admits a projection, so a successor whose acknowledgement is lost still counts — the node may already hold it. The row is adopted in the transaction that records a projection's acknowledgement, so no acknowledged projection exists without its lease, and adoption ends every renewal of the previous projection. prepareRenewal then:

• refuses superseded unless the node's delivery head is acknowledged and is the lease's projection;
• picks the slot of the later of the controller clock and the node's current window, which is its acknowledged renewal's or else the projection's, so a clock stepped back never states an earlier window;
• refuses beyond-horizon when that window would end more than the contract's four-hour escrow horizon after the clock. A window a node can take now ends within 15 minutes, so today this names a controller clock that fell far behind a window it already signed;
• asks the contract's own admitRuntimeNetworkDnsLeaseRenewal, on the controller clock, whether the node could admit the renewal now. The projection's own slot, the slot already acknowledged and a clock stepped back therefore renew nothing and change nothing;
• signs it and advances the latest signed sequence and the horizon by compare-and-set, never backwards.

acknowledge records a renewal the node admitted. It takes only the lease's own renewal of its projection, at a sequence no later than the latest signed one, refuses a renewal of a superseded projection as superseded, and a late acknowledgement never moves the lease back.

prepareRun states the same slot and signs the whole run of consecutive slots a relay may hold from it, under one signer fence, and raises the latest signed sequence and the horizon to its last window by compare-and-set. The run's length is never written down: dnsLeaseRunSlots takes slots while the contract's escrow allows them — at most maximumEscrowedRenewals, at most maximumEscrowHorizonMs from the first window's start to the last one's end, and no window ending more than that span after the controller clock — which on the ten-minute grid is 23 slots, just under four hours. The slots are consecutive because a window covers its own slot and half of the next, so a skipped slot would leave five minutes uncovered. A clock that fell behind a window already signed shortens the run; one behind it by more than the escrow horizon refuses it beyond-horizon, as it refuses a single renewal.

dispatchNetwork adds a third step after the projection ACK or replay: prepare the renewal, push applyRuntimeNetworkDnsLeaseRenewal through the same push helper, bind the ACK to exactly the renewal that was sent, and record it. It resolves { projection, renewal, escrow }, where renewal is { status: 'delivered', renewal }, { status: 'not-due' } or { status: 'failed', error }, so a failed renewal never hides a delivered projection. A renewal becomes deliverable when its slot opens and must land before the previous window ends five minutes later. The delivery loop visits eight projection rows per pass and starts each pass five seconds after the previous one ended, so it reaches every node within those five minutes for up to 400 nodes while each pass finishes within a second. Beyond that, windows lapse and fail closed until the node's next visit.

A node whose delivery fails, because a push timed out or was refused or the node has no session, is skipped by the loop for one second, doubling with each further failure to at most 60 seconds on the monotonic clock, and the pass counts it as deferred. A node that times out therefore costs one 10-second push timeout per wait instead of one per pass. A successful delivery or a new session registration for the node ends the wait, and shutdown forgets every wait. Direct dispatchNetwork calls and the push after a compile do not wait, but every dispatch outcome starts, extends or ends the node's wait. A Pallet node that does not accept renewals refuses each push and keeps its acknowledged projection; its DNS lapses closed.

The run a cluster relay holds

A cluster is outbound-only, so a node that loses Cloudly loses the only signer of its DNS window. The fourth dispatch step therefore hands the node's cluster relay the run it may release while it has no Cloudly session: Cloudly mints the run, pushes holdRuntimeNetworkDnsLeaseRenewals with the contract's own request snapshot, and binds the answer to the exact run that was sent. escrow reports { status: 'held', cluster, held }, { status: 'retained', cluster }, { status: 'no-relay' }, { status: 'deferred', retryAt } or { status: 'failed', error }.

Only a node whose current session is carried by its cluster's elected relay is signed a run at all, which ClusterManager.readElectedRelayCluster decides through the one election every dispatch into a cluster uses. A directly connected node, and a node whose carrying peer is not its cluster's elected relay — it registered none, none is live, or two sockets claim the cluster — are handed nothing and signed nothing, so their revocation latency stays one window. A node without a session never reaches this step at all: the dispatch refuses it before the renewal. A relay-carried node's dnsAuthorityHorizon instead sits up to four hours ahead, which is exactly the contract's revocationLatency: a detached cluster keeps DNS authority until its last held window ends. The run's first renewal is the renewal the node is handed directly in the step before, byte for byte, so a node may hold that window from Cloudly and from its relay and both are one statement.

What the run would state is read and compared with what the relay was last handed in the same transaction that mints it — readRun follows exactly the rules prepareRun follows and signs and writes nothing — so a pass that would repeat itself answers retained at the cost of one fenced read transaction: no signature and no lease write. What it is compared against is the memory-only record RelayRunHandovers keys by the relay peer, the node's reporterSessionId, the projection reference and the run's first and last slot. Those are the four cases in which the relay itself gives a run up: the node registers another session, the node admits a superseding projection (which a signer rotation reaches through the successor it forces), another socket becomes the relay, and the last window of the run ends. The slot rolls over every ten minutes, long before that last window, so the key changes — and the run is minted and handed over again — long before the relay reaches the end of what it holds. A fifth case is Cloudly's own and changes none of them: a registration Cloudly refuses with relay-cluster-not-enabled, relay-node-foreign, relay-sequence-not-advancing or relay-credential-generation-stale tells that relay it is not the one Cloudly dispatches to, and it drops every run it holds on the same socket, under the same session, for the same projection and the same slot. Cloudly issues that refusal, so Cloudly forgets what it recorded for that peer: the runtime manager subscribes to ClusterManager.subscribeRelayUndispatched while it runs, and the next pass hands every run of that relay over again. A hand-over already in flight when that refusal arrives is recorded as nothing at all, because a pass records what it sent only against the forget count it read before it minted. A restarted controller remembers no hand-over at all and hands every run over once more, mints the identical bytes, and the relay takes it as the run it already holds.

The step is last and answers for itself, so neither the projection nor the renewal before it can be delayed or failed by a relay. The escrow names six refusals — escrow-run-invalid, escrow-node-not-carried, escrow-run-foreign-session, escrow-run-stale, escrow-run-unordered and escrow-cloudly-only — and words each of them with that name first. Each is an answer from a working socket: the run does not satisfy the contract, the relay carries the node no longer, the run arrived under a session the node has replaced, the relay holds a run reaching at least as far, it cannot order this projection against the one it holds, or the hold reached its node listener instead of its own Cloudly session. Only those six leave the relay no wait, so the next pass hands over again as soon as what caused one of them changed; they are the contract's names, held in dnsLeaseEscrowRefusals until @serve.zone/interfaces states them beside holdRuntimeNetworkDnsLeaseRenewals. Every other answer makes that relay wait: the texts typedrequest words for a method the relay does not serve and for a handler that threw, an answer that does not name the run that was sent, and a socket that does not answer at all. The wait is one second, doubling to at most 60 seconds, on the same backoff the nodes use but keyed by the relay peer: one unreachable relay then costs one wait instead of one 10-second timeout per node it carries, and it is released when that socket closes or when a hand-over lands. Nothing is read, signed or sent for a relay while it waits — the pass reports deferred with the monotonic reading the wait ends at — so a broken relay never advances the horizon of the nodes it carries.

Managed VPN — the cluster's own hub

Cloudly runs no VPN server. Each cluster's relay is that cluster's packet relay: every node of the cluster dials its relay's managed hub, and no inter-node packet leaves the cluster or reaches Cloudly's process. Cloudly keeps every authority — who is a member, which control address they hold, which key their credential carries — and states one complete network per cluster, which that cluster's hub applies.

configureManagedVpn({ controlPrefix, hubPrefix }) is a trusted controller operation that stores the plan's immutable vpn and the append-only node control-address reservations through SmartData. controlPrefix is the private IPv4 prefix control addresses are allocated from; hubPrefix is the protected prefix every cluster hub must bind inside. Both must be covered by the protected union, disjoint from workload and transit pools and from the resolvers, and disjoint from each other — control addresses live inside the tunnel and hub endpoints on the underlay. The network and gateway addresses remain reserved; disconnects, expiry and restarts never recycle a node address. Later protected-authority updates must preserve this configuration. Startup without configuration runs no reconciliation pass at all.

A relay advertises the hub it bound on its own registration (registerClusterRelay, relay.vpnHub = { publicKey, address, quicPort }), and the registration is replaced whole: a relay that restarts re-keys its cluster's hub, and one that comes back serving none withdraws it rather than leaving the keys of a process that is gone standing. The registration is judged by the contract's own validator, so a hub Cloudly would refuse on the wire is never persisted; whether the plan covers the address it bound is decided where the hub is selected, so one misconfigured relay cannot fail the authority every other cluster is composed into.

getRuntimeManagedVpnCredential requires the exact current physical Pallet socket, complete session binding, current authenticated projection and acknowledged signing authority. That projection must explicitly select its own cluster's hub for router egress. The response carries transport: 'quic' — the one transport a managed hub serves — the hub endpoint the node dials, the authority id, the hub's public key and a short-lived client key pair. It stays process-local; never log, persist or embed it in projections. Exact current retries retain their keys, and a relay that re-keyed its hub costs exactly its own cluster's members their credential. The credential lease cannot outlive the node's effective DNS window: the window of the renewal it acknowledged for its current projection, or else the projection's own.

One network per cluster is compiled from the same durable state (compileClusterVpnNetworks) and pushed to that cluster's relay over its already registered session (applyClusterVpnNetwork); a relay whose hub has just started pulls what Cloudly states instead of waiting (getClusterVpnNetwork). A network is the whole authority its hub serves, never a patch, so an absent member is a withdrawn one and a cluster whose hub holds no member is stated an empty network. It carries only exact local workload /32 prefixes, each with its own policy domain, and a grant requires both node projections to contain the same lease material and directed permission. Missing peers, absent reverse permission and control, public or platform destinations confer no grant. Two answers are refusals rather than networks, and Cloudly states them by name: relay-vpn-hub-absent for a cluster with no hub to select, and grant-cross-cluster-unsupported for a grant whose two workloads sit on two clusters — hubs are per cluster, so such a grant has no path through any one of them, and hub federation is a later lane.

The revision each cluster is stated a network at is durable (clusterNetworks on the managed VPN record). A hub's daemon outlives Cloudly's process, so a counter kept only in memory would restart at one while that hub still holds a higher revision, and every push after a restart would be refused as stale. It rises only when a cluster's content moves; a cluster whose relay is unreachable is retried with the same revision and spends no counter, and the relay's refusals (vpn-network-authority-foreign, vpn-network-revision-stale) are logged by name against the cluster that stated them.

One serialized owner checks current source authority before and after it states a network. Registration, human Pallet credential rotation/revocation and projection replacement withdraw the affected member before the mutation; protected-configuration changes withdraw every member. Physical close immediately closes future credential admission and queues joined withdrawal. A one-second reconciliation pass also detects out-of-band source changes and moves each member's expiry to its effective window under the key pair the node already holds, so a renewed node keeps its membership past its projection's window and an exact retry returns the same keys with the renewed expiry. The pass judges every member and the network it compiles on one clock reading, so a window that has ended at that reading costs the member its place, one ending mid-pass costs it on the next pass, and neither stops the pass. A pass that cannot read its state withdraws every member rather than stating stale grants, and the expiry every member carries into the network is the backstop for a pass that cannot run at all: a hub stops relaying for a member whose window ended without being told to.

Database acceptance is not proof of instantaneous packet withdrawal on disconnected workers, and a hub that accepted a network does not prove Pallet TUN installation, workload ACTIVE state or deployment readiness.

Private Network Desired State

ts/manager.privatenetwork/ owns organization DNS namespaces, private network definitions, placement-independent service attachments, alias reservations and historical mutation receipts through SmartData. Names use the immutable keys and strict metadata contracts published by @serve.zone/interfaces. Display-name changes preserve fully qualified names. Alias uniqueness is enforced per network; the same alias can be held independently in another network.

The seven versioned network RPCs require a signed Cloudly identity bound to a server-held IDP login session. Every transaction attempt obtains fresh confidential introspection through @idp.global/sdk/server and requires the requested canonical organization's owner role. Cloudly infrastructure administrators, local password logins and ordinary hosted-app OIDC logins confer no organization authority.

Mutations require an exact expected revision and retained mutation ID. The same SmartData transaction fences the exact login session, human credential generation and immutable issuer/subject binding. Expiry is checked before commit, independently of asynchronous TTL cleanup. Service membership writes also fence current service ownership and every referenced network against retirement. A failed receipt write rolls back the entire mutation. Historical replay returns its original metadata and cannot restore current attachment or access authority.

Detaching an alias marks its reservation as retiring. Network retirement blocks new attachments and remains pending. This foundation provides no completion or identity-reuse operation: the node grant ledger, explicit denial/fencing evidence and DNS quarantine must be integrated before reservations can be released. Authoritative endpoint production and Pallet packet/DNS application remain under development. The private-network store creates no public DNS records, ingress routes, host ports or runtime effects.

getServicePublishedPorts and setServicePublishedPorts are registered by the same manager and reuse the same organization-owner authorization, but their durable state is the runtime network service policy, not this store. They authorize exact host ports for one service: Service.data.publishedPorts remains the operator's request, while the policy document is the authorization of record, and every entry the compiler writes into the signed runtime network projection carries the exact policy revision and digest that authorized it. Port ranges and an explicit hostIp are refused; an omitted host address means the node uplink, which the runtime enforces. Host ports are unique per node, but placement is resolved when the projection is compiled, so a duplicate claim fails compilation rather than authorization.

The Networks view is available to ordinary organization users after generic Cloudly identity registration; their dashboard exposes only this view. Existing administrator views remain available to administrators, alongside Networks. The organization picker obtains current owner memberships through the SDK's subject-bound UserInfo client using the encrypted server-held token. These summaries are display context, not authorization. Service candidates are paginated within the selected organization, exclude deleting services, and require fresh owner introspection on every transaction attempt.

Owners can create or edit network names and external DNS policies, retire networks, and manage service attachments, aliases and the default short-name network. A double-click opens details; retirement requires explicit confirmation. Unconfirmed saves retain their exact mutation ID and body for manual retry. Successful replies trigger current-state reads, so a historical receipt never replaces a newer revision. Changing identity or organization clears editors, pending edits and lists, and suppresses older responses.

The existing SERVEZONE_PLATFORM_OIDC_* configuration selects the trusted issuer and confidential client. An IDP issuer must advertise openid, profile, organizations, roles and its canonical /oauth/introspect and /oauth/userinfo endpoints. Cloudly requests those four scopes and uses audience-mapped app_roles for its local user/admin role; IDP's global roles and token organization snapshots grant no network permissions. Configure the IDP application connection's role mappings accordingly. Incomplete organization-capable discovery fails closed. Existing hosted-app issuers continue using openid and their app-instance claim contract.

IDP access tokens remain encrypted in cloudly_federated_session using the Secrets v2 keyring and envelope engine. Authenticated context binds all immutable session metadata; only an opaque session ID enters the Cloudly JWT. Session expiry is bounded by the access token, ID token and seven-day local maximum. Reauthenticate after expiry; this flow does not request or persist refresh tokens. Keep referenced KEKs available for unexpired sessions. Local credential revocation, deleted sessions and changed issuer/subject bindings invalidate the associated identities. Shutdown drains network requests and admitted session creation before closing the secret manager and database. The IDP provider must be released and qualified before these RPCs can establish live production authority.

Hosted-App OIDC

The Access view lets administrators enable or disable platform OIDC for hosted apps whose App Store version declares a valid platformOidc capability, and assign the declared app roles to Cloudly human users. Cloudly exposes the matching getHostedAppAccessConfiguration, setHostedAppRoleAssignment, and setHostedAppPlatformOidc admin requests. Browser authorization handoff uses the shared getHostedAppOidcAuthorization, completeHostedAppOidcAuthorization, and cancelHostedAppOidcAuthorization requests from @serve.zone/interfaces 32.7.1.

Cloudly owns its hosted-app signing JWK as an encrypted system Secrets v2 value. Each enabled app gets a generated service-owned client-secret record with launcher-environment delivery metadata for its declared variable. Cloudly includes that entry in resolved manifests and can seal its material to the enrolled cluster recipient; actual workload injection still depends on the downstream runtime rollout described above. Registration records and API responses contain only value-free metadata. Enable, disable, restart recovery, and service cleanup are fenced to the exact service lifecycle and Secrets v2 management scope. The app must have a canonical HTTPS domain, and template upgrades across an OIDC contract change require OIDC to be disabled first.

On first startup, Cloudly bootstraps the first human admin from SERVEZONE_ADMINACCOUNT. Human clients authenticate through adminLoginWithUsernameAndPassword; machine clients authenticate through getIdentityByToken. Cluster creation creates a machine user and token for Coreflow.

Human Credential Administration

Human passwords use salted scrypt verifiers in a private SmartData authority, separate from mutable User records. A User record can never carry a password: save() refuses one. Existing human sessions without a credential generation must authenticate again after upgrading. Machine identities and their token flow remain supported.

Use the shared requests.admin.IReq_Admin_GetHumanCredential contract with getHumanCredential to inspect userId, generation, passwordEnabled, and updatedAt. mutateHumanCredential takes the target user ID, expected generation, a stable mutation ID, and one operation: rotatePassword with a new password, revokePassword, or revokeSessions. Both methods require a human administrator who authenticated within the last five minutes. New passwords require at least 20 characters and at most 1,024 UTF-8 bytes. Never log request bodies.

Every successful mutation advances the generation and invalidates older human UI/API JWTs and OCI registry tokens. Password revocation disables password login; a new trusted OIDC login remains possible. Updating SERVEZONE_ADMINACCOUNT does not rotate an existing user's password. Bootstrap creates the user and authority in one transaction, and does not overwrite existing human users.

For a lost response, authenticate again and repeat the same mutation ID and exact request. The actor-bound receipt returns the original metadata with replayed: true, without reapplying the mutation or returning a password. That metadata is historical; inspect current authority separately. A changed request under the same mutation ID conflicts. Actor authorization, target generation, authority update, and receipt persist within one SmartData transaction.

Typical consumers use @serve.zone/api:

import { CloudlyApiClient } from '@serve.zone/api';

const client = new CloudlyApiClient({
  registerAs: 'admin-tool',
  cloudlyUrl: 'https://cloudly.example.com',
});

await client.start();
const identity = await client.loginWithUsernameAndPassword('admin', 'change-me');
const clusters = await client.cluster.getClusters();

Machine clients such as Coreflow authenticate with getIdentityByToken. Auth-sensitive requests and WebSocket identity tags transport only the JWT-shaped IIdentityCredential; Cloudly verifies that credential and reconstructs the authoritative identity fields from persisted user data. The issued JWT never outlives its source token, and a cluster machine user is accepted only while it belongs to exactly one persisted cluster. The WebSocket tag lets Cloudly push configuration to already-connected Coreflow instances instead of opening inbound connections to cluster nodes.

Cluster Flow

The implemented cluster flow is intentionally simple:

1. An admin creates a Cloudly cluster record.
2. Cloudly writes that record, the machine user its long-lived cluster token belongs to, and the cluster's runtime phase in one transaction. A cluster created from here on states pallet, so steps 3 to 6 are the path of the clusters that predate that record — the ones the cluster-runtime-phase migration states swarm — and nothing in this release moves a new cluster to swarm.
3. Coreflow starts on a Docker Swarm manager node with CLOUDLY_URL and JUMPCODE.
4. Coreflow authenticates to Cloudly and requests the cluster configuration payload.
5. Cloudly returns cluster data, workload services, platform bindings, provider configs, and optional external gateway configuration.
6. Coreflow reconciles Docker networks, services, volumes, platform bindings, backups, and routing. Cloudly exposes the enrolled-recipient and sealed-material APIs for Secrets v2, but end-to-end delivery remains gated on the matching Coreflow/Spark release and enrollment.

When service, platform, or gateway settings change, Cloudly pushes updated config to connected Coreflow clients where supported.

Cluster Runtime Phase

Which runtime owns a cluster's workloads is a record of its own, cloudly_cluster_runtimes, keyed by the cluster id: swarm is Coreflow's, draining stops every service while its volumes stay, switching lets the cluster's nodes join the runtime network while nothing runs, and pallet is Pallet's alone. It is stored beside the cluster document because a cluster update merges caller data, and no caller may move a phase with it. Every phase write advances a generation, so a caller that read a phase states which reading it acted on, and runtime work joins the record through fenceClusterRuntime, which refuses a phase it does not serve and conflicts when the phase moves before the caller commits.

A cluster created from here on states pallet, in the same transaction as the cluster and its machine user; every cluster that predates the record is stated swarm once by the cluster-runtime-phase migration, and a phase that already moved is never restated. Cloudly holds no phase for a cluster it never stated one for and refuses by name with cluster-runtime-missing rather than assuming Coreflow owns it.

getClusterRuntime serves that record to an administrator together with the blockers Cloudly can actually read today: a relay nobody enabled (relay-not-enabled) or that never registered (relay-not-registered), a node without an active Pallet credential (node-without-pallet-credential) or whose architecture is not one Pallet runs (node-arch-unsupported), and a service the cluster shares with another cluster (service-multi-cluster-scope) or that holds a platform binding (platform-binding-present). Only a service whose immutable rollout targets this cluster is inspected, because that rollout is the one cluster scope a service states today, while the config push is fleet-wide: a platform-bound service without such a rollout runs in the cluster too and is still not named. An empty list states that nothing Cloudly can read blocks the cutover, never that nothing does.

Withdrawing a cluster relay is a Coreflow-era operation, so disableClusterRelay is refused outside swarm, as is repairing an immutable rollout's placement: placement is what Coreflow schedules against. Deleting a cluster removes what is only its own — the accepted relay registration, the machine identity its relay authenticates with, and its phase — in one transaction, and is refused while nodes still belong to it or a relay still carries it on a live socket. The user is removed only when it is the cluster machine user Cloudly minted with the cluster; a cluster document that names anything else — a human administrator, another cluster's identity — refuses the whole deletion with cluster-user-not-machine instead of deleting that user. Cloudly owns that pointer, so updateCluster never writes it: a request that echoes the stored id is accepted and changes nothing, one that names a different user is refused, and the key is dropped from the merge either way.

Cluster Relay

A cluster may hold one relay: the single outbound socket that carries the runtime sessions of every Pallet node in that cluster. Cloudly owns the name it is reached under. It mints https://relay.<clusterId>.<zone>:8443 — the port the relay binds its own TLS listener on, named explicitly because an origin drops only :443 — where <zone> is the most specific DNS zone Cloudly manages that holds Cloudly's own public hostname. The Domain documents are that authority, not a public-suffix list: they are exactly the zones an administrator registered with Cloudly, so a name derived any other way could be one dcrouter holds neither an A record nor a certificate for. A Cloudly whose public hostname sits in no managed zone is refused by name instead of minting an origin nothing can back. Every minted block is validated with the contract's own validateClusterRelay before it is stored.

Because the origin is Cloudly's, updateCluster refuses any request that carries a relay key at all, including one that only echoes the block it was served. Enabling and disabling are their own admin requests, enableClusterRelay and disableClusterRelay, authorized like every other cluster request: enable mints and stores the block once Cloudly proved it can back the name (see Cluster Relay Names), disable removes it, and both push the cluster config to the relays already connected, so a relay learns the change without reconnecting. A cluster update is pushed the same way. In the dashboard a cluster shows its relay as off until an operator enables it, then enabled, awaiting relay registration until a relay reports itself in, and afterwards what that relay registered — registered · <listenAddress>:<listenPort> · <version> · since <acceptedAt>, or registered, socket gone when no socket carries it at this instant. The row and the detail actions flip it, and both refresh the registrations along with the clusters.

A relay reports itself in with registerClusterRelay over its own socket, after that socket authenticated with registerCloudlyClientSession. The cluster it speaks for is the one its verified machine identity names and is never a field it sends; the node it runs on must be a node of that cluster and must still exist when the write commits; and a cluster whose relay no operator enabled has no origin to register into, so the registration is refused by name (relay-identity-unverified, relay-node-foreign, relay-cluster-not-enabled). The response carries the origin Cloudly published, so a relay can check the certificate it holds against the name its nodes are given, and beside it two references it always carries: what the acceptance was made under, and what it produced. credential is the {id, generation, digest} of the authority Cloudly authenticated the socket as, read from Cloudly's own records and never from the body; for a relay that still holds its cluster's shared pre-relay authority it is that authority's own record, which Cloudly writes the first time a relay names it and removes with the cluster. acceptedRegistration is the reference of the durable registration record, which now carries its own identity beside the cluster it is keyed by, a generation that advances by one per acceptance, and the contract's digest over it, so a citation of a superseded registration names the same record and fails on its generation and digest. A relay that also serves its cluster's managed VPN hub reports the endpoint it bound on that same registration (relay.vpnHub), and the registration is replaced whole, so a relay that restarts re-keys its cluster's hub and one that comes back serving none withdraws it.

The accepted registration is durable, one document per cluster in cloudly_cluster_relay_registrations, so one cluster having one relay is a database rule rather than a convention, and a Cloudly restart does not forget which relay a cluster has. What that database cannot hold is whether the relay is connected right now: liveness lives in an in-memory registry keyed weakly by peer, which confers no authority and is retired when the socket closes, leaving the registration itself untouched. The sequence a relay counts is checked only within the socket that sent it, where it must strictly advance — across sockets it says nothing, because a restarted relay counts from zero again, so there the newer accepted registration wins and acceptedAt, which is Cloudly's own and never moves backwards, is the "since" an operator sees. A refused registration does not consume that sequence: a relay whose registration was refused, for a node that does not belong to its cluster or for any other named reason, may send the very same body again once the cause is gone. Only an accepted one advances the socket's count, so a retry that repeats an accepted sequence is refused.

Beside those registrations, cloudly_cluster_relay_legacy_credential holds one document per cluster for the shared pre-relay authority: the machine identity the cluster was created with, recorded the first time a relay registers under it or the first time its relay credential is minted, so an acceptance cites a record rather than nothing. It carries no bearer and no hash of one, because the bearer lives on the cluster's user document and nowhere else. It is written absent-only, so a relay that registers again never re-records it, and it is removed with the cluster rather than with any registration made under it.

A cluster's relay authenticates with its own generation-bearing credential instead of that shared authority. mutateClusterRelayCredential takes the contract's mutation from a freshly authenticated administrator: ensure requires an absent record and rotate the exact current generation, never a minimum, and the administrator is fenced again inside the transaction that writes. The bearer is delivered in that same transaction as a new immutable encrypted version of the Cloudly-system secret SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, owned by { kind: 'system', systemId: 'cluster-relay:<cluster>' } under the management scope system:cluster-relay:<cluster>, so a credential Cloudly accepted is always one the relay can read and a delivery that failed leaves no authority claiming a bearer nobody holds. A mint requires that secret to be absent and a rotation names the exact active version the authority records. Cloudly answers metadata only: cloudly_cluster_relay_credential_authority keeps the hash of the one bearer each generation issued and the public projection builds the contract's members without it, so no record, receipt or answer of this chain carries a bearer. No request returns the bearer either: an operator exports it out of band with cloudly secrets export-cluster-relay-authorization and hands it to the relay (see Cluster Relay Credential Delivery). A retained mutationId answers the accepted result again with replayed: true and renews nothing — not the generation, not the delivery, not the acceptance time — while the same id over a changed body is refused conflict; those receipts live in cloudly_cluster_relay_credential_mutations and are removed with their cluster, as the authority and the shared record are.

The relay exchanges that bearer for a session the way every machine does, with getIdentityByToken. The session is the cluster's own machine identity — exactly what the shared authority authenticates, so a relay changes one bearer and keeps every call it makes — and the JWT carries the generation it was exchanged for. That generation is re-read on every request the session is presented with, so rotation has no overlap: the previous generation stops authenticating at commit rather than at an expiry. A socket whose generation has been replaced is told apart from one Cloudly never verified and is refused relay-credential-generation-stale before its registration body is judged, which ends that relay's custody; a restart under the bearer of the current generation is what resumes it. The authority a registration is accepted under is also fenced inside the acceptance itself, so a rotation committing between a socket's verification and that write makes one of the two lose instead of leaving a registration that cites a generation nobody holds.

getClusterRelayCredential reads the whole chain for an administrator — the relay credential, the shared record and the current accepted registration, each null while no record names one. Liveness is not part of it: that is a fact of a socket, which getClusterRelayRegistrations answers.

retireLegacyClusterRelayCredential is the last step of that replacement: it revokes the shared authority once something else authenticates the relay. The request carries compare-and-swap inputs only — the exact expectedLegacyGeneration, the replacement credential reference and the acceptedRegistration reference an operator observed — and Cloudly dereferences every one of them against its own records inside the transaction that writes, with the administrator fenced there as a credential mutation fences one. Five conditions hold together, and there is no phase or fleet condition beside them: the shared record is this cluster's current one at the named generation and still accepted (legacy-credential-absent, legacy-credential-already-retired, conflict), the named replacement is this cluster's relay credential at its current generation (relay-credential-absent, legacy-retirement-replacement-not-current), the named accepted registration is the current one and was made under that very replacement (legacy-retirement-registration-not-current), and one verified live socket carries the cluster's relay under exactly that credential at the instant of acceptance (legacy-retirement-relay-not-live). A registration made under the shared authority names that record rather than the replacement, so a relay that never moved can never justify retiring the authority it still holds. Acceptance writes the one-shot invalidation in cloudly_cluster_relay_legacy_credential_invalidation — the shared record at the generation before the revocation, the replacement, the registration, the administrator and the acceptance time — and advances the shared record to state: 'revoked' naming it back, so neither digest covers the other's. A retained mutationId answers that receipt again with replayed: true, the same id over a changed body is refused conflict, and any other retirement of a revoked record is refused legacy-credential-already-retired: nothing is revoked twice. From that commit on, a socket that still holds the shared bearer is refused legacy-credential-already-retired instead of registering under it, and the receipt is removed with its cluster like every other record of this chain. Revoking the bearer itself stays an operator's action; what Cloudly owns is that it authenticates nothing here.

An administrator reads those registrations with getClusterRelayRegistrations, authorized by the same admin guard as every other cluster request, and narrowed to named clusters with clusterIds or left to answer for every cluster whose relay ever registered. The answer is the contract's own read shape — the cluster, the node, the listen address and port, the relay version, the sequence it counted, Cloudly's acceptedAt — plus live, which is whether a connection the server still enumerates holds the Cloudly identity it was verified with and registered as that cluster's relay in this process. live is a liveness fact and nothing more: it grants no authority, it is not part of what was registered, and a registration whose socket is gone stays exactly as valid. The shape is validated with the contract's validateClusterRelayRegistration before it is served, and the stored record is checked against that very same validator, so Cloudly can never persist a registration it would later refuse to serve.

Cluster Relay Credential Delivery

The relay reads its bearer from the launch variable SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, which its qenv resolves from the process environment or from the Docker secret file /run/secrets/SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, byte for byte. Cloudly holds the bearer only as the encrypted version the credential names, so an operator carries it across as a file that exists only for the handover:

1. Mint the credential with mutateClusterRelayCredential ({ clusterId, operation: 'ensure', expectedGeneration: null, mutationId }, keeping the mutationId for a retry) and read the generation getClusterRelayCredential answers: 1 after a mint.

2. Export it inside the running Cloudly container, into a directory only root can read:

   docker exec <cloudly-container> mkdir -m 0700 /root/relay-export
   docker exec <cloudly-container> node /app/cli.js secrets export-cluster-relay-authorization \
     --cluster-id <clusterId> --expected-generation 1 \
     --output /root/relay-export/relay-authorization
   

   Keep the answered sha256, and check every copy of the file against it with sha256sum before using that copy.

3. Carry the file to the Swarm manager of the relay's cluster without widening its mode (docker cp out of the container into a root-only directory, then scp -p), and create the Docker secret there from the file:

   docker secret create <secret-name> - < relay-authorization
   

   The relay's service mounts it with --secret source=<secret-name>,target=SERVEZONE_CLUSTER_RELAY_AUTHORIZATION.

4. Remove every copy of the file -- inside the Cloudly container, on the Cloudly host and on the manager -- with shred -u, or with rm where shred is unavailable.

A rotation (mutateClusterRelayCredential with operation: 'rotate' and the exact current expectedGeneration) is delivered the same way, exported with --expected-generation set to the new generation. Docker secrets are immutable, so the new bearer becomes a second secret under a new name, swapped in with docker service update --secret-rm <old-secret-name> --secret-add source=<new-secret-name>,target=SERVEZONE_CLUSTER_RELAY_AUTHORIZATION <relay-service>, which restarts the relay under it; the old secret is removed afterwards. Rotation has no overlap: from its commit until that restart the relay is refused relay-credential-generation-stale, so the export and the service update follow the rotation directly.

Cluster Relay Names

dcrouter holds every relay name Cloudly mints: its A record and its exact certificate. Cloudly is a dcrouter gateway client (see the gateway settings) and holds each name under the gateway-client hostname ownership { appId: 'cluster-relay:<clusterId>', hostname: <relay name> }, through the gateway-client hostname requests syncGatewayClientDnsRecord, getGatewayClientCertificate and releaseGatewayClientCertificate. Cloudly holds no DNS provider credential and issues no certificate itself. enableClusterRelay proves that dcrouter advertises client hostname records and exact issuance (dns.clientRecords, certificates.exactIssuance) before it stores the block, and refuses by name otherwise, leaving the cluster unchanged: relay-gateway-unconfigured or relay-gateway-unsupported.

Certificate. A relay asks for its certificate with getCertificateForDomain when it starts. A name that is exactly an enabled relay's name is answered with dcrouter's certificate for that ownership and never through the gateway's route certificates. Its key is one cluster's: it goes to an administrator or to that cluster's own machine identity, and any other cluster identity is refused relay-certificate-not-owned. dcrouter issues the certificate by DNS-01 in the relay name's zone; while it is issuing, the relay is refused relay-certificate-pending and asks again. Asking is also what renews a certificate past its renew-by date, so the relay's request is answered by dcrouter directly and never from a copy Cloudly kept. Every other name in the relay namespace -- relay.<label>.<zone> with <zone> a zone Cloudly publishes in, whose relay is disabled or was never enabled, or whose cluster does not exist -- is refused relay-name-reserved, whoever asks. The generic certificate path refuses such a name for every caller, a service route's included: a certificate issued there would still be valid once a relay carries the name. A name of that shape in a zone Cloudly does not publish in is outside the namespace and is served as any other name. The wildcard *.<label>.<zone> covers relay.<label>.<zone> and is refused relay-name-reserved the same way, whether that relay is enabled or not; *.<zone> and deeper wildcards cover no relay name of <zone>. Such a name is also never attached to a service: createService, updateService, reserveServiceDeployment and promoteServiceDeploymentRoute refuse a hostname in the relay namespace, or a wildcard that covers one, that the service does not hold yet with RELAY_NAME_RESERVED, before they write anything. Every other certificate Cloudly hands out is dcrouter's as well; a Cloudly without a gateway has none to hand out.

A record. Once a relay registered, dcrouter publishes the relay name at the listenAddress of the accepted registration, with a 60-second TTL, because the nodes dial the relay itself. It is written after the acceptance commits and never inside it, and the registration's answer does not wait on dcrouter. The record follows the relay when a later registration moves it; a relay whose socket closed keeps its name; the record is withdrawn and the certificate released once disableClusterRelay removes the block or the cluster is deleted, and a relay name that changed is released under its old ownership before the new one is held. dcrouter publishes only an address inside the gateway client's allowedDnsRecordAddresses, which Cloudly derives from the runtime network address plan (see Runtime network address plan), and never overwrites or adopts a record it does not manage.

cloudly_cluster_relay_hostnames keeps one document per cluster whose relay is enabled: the name, the address and the reference of the registration it carries (both null before a relay registered, when the certificate alone is held), the state (pending, published, failed or withdrawing), the refusal of the last failed attempt and the attempts since the last success. The document is written before dcrouter is asked, and every dcrouter call is idempotent under the ownership, so an attempt that dies half-way is finished by the next one. A failed attempt is noted by its refusal -- relay-gateway-unconfigured, relay-gateway-unsupported, relay-dns-zone-unmanaged, relay-dns-address-not-allowed, relay-hostname-conflict, relay-gateway-failed or conflict -- and a pass over every relay retries it within five minutes, as it does at every start; the same pass asks for every certificate, so dcrouter renews it.

The data-version step retire-cluster-relay-acme (0.20.0 to 0.21.0) drops the collections Cloudly kept while it issued and published relay names itself (cloudly_cluster_relay_acme_accounts, cloudly_cluster_relay_certificate_issuances, cloudly_cluster_relay_certificates, cloudly_cluster_relay_dns_records). An A record Cloudly had written at Cloudflare is not removed by it: dcrouter refuses a relay name another writer holds, so the step logs each such name as a warning for the operator to remove at the provider.

Node Identifiers

A node id is drawn from nanoid's URL alphabet and is judged everywhere Cloudly stores or reads one by the node vocabulary ^[A-Za-z0-9_-]{1,128}$, which is what the credential authority, the session binding and the Spark wire all state and which admits a leading - or _. The canonical vocabulary ^[A-Za-z0-9][A-Za-z0-9:._-]{0,199}$ belongs to organization, service, cluster, rollout and namespace ids and to the content ids derived from them; a node id is never judged by it, so a node whose stored id begins with a separator is still the node its credential, its session, its network pass and the deletion impact an operator reads all name. Composed ids that take a node id state which part it is, so the id itself is unchanged while each part is judged by its own rule.

New node ids are minted through createClusterNodeId(), which keeps the first character alphanumeric and judges the draw against both vocabularies where the identifier is made. That is what the Interfaces contracts carrying a node id in the assignment and runtime network domains require — IRuntimeAssignment.nodeId and replicaId, the network node scope, and the egress owners, leases and receipts of the protected authority all read it as a canonical identifier — so a node already stored with a leading separator is refused a workload and a runtime network by the contract itself, and only an Interfaces release admitting the node vocabulary there changes that.

Node Credential Authority

NodeManager owns independent spark and pallet credential authorities outside the mutable node inventory. New secure tokens use 32 cryptographic random bytes. All Spark authentication uses this authority; a Spark token cannot authorize Pallet. Heartbeat state, action delivery/results and Swarm observation acceptance commit under the same credential-generation fence as their database effects. Rotation or revocation racing those transactions prevents stale effects from committing. This does not retract an action already delivered before revocation.

Nodes without a Spark hash remain unenrolled, and existing credentials are not automatically rotated.

getNodeConfig, getNodeCredential, rotateNodeCredential and revokeNodeCredential use the published Interfaces 28 contracts and require a verified infrastructure administrator. Config returns only the public node DTO; credential results contain value-free metadata. Mutations require exact generation and session-epoch preconditions plus an immutable mutation ID. Rotation accepts a SHA-256 hash of pending node-generated material, never returns plaintext, and requires the node to retain that material durably before requesting activation. Receipts bind the verified actor and exact request. Exact replay returns the historical result without reactivating replaced or revoked authority.

This is a source integration, not a completed Pallet rollout. Pallet identity bootstrap, peer-bound sessions, atomic Jump claim/replay, isolated migration rehearsal and staged rotation of existing credentials remain deployment gates. Infrastructure administrator permissions are distinct from workload-organization RBAC; these methods do not establish organization permissions.

Spark Host Heartbeats

Two Spark runtimes report to Cloudly, and they are two runtimes rather than an old and a new one. The node runtime posts sparkNodeHeartbeatContract.endpoint (POST /spark/nodes/heartbeat); a Swarm-era worker in coreflow-node mode posts the four sparkSwarmNodeContracts endpoints (POST /spark/swarm-nodes/{heartbeat,metrics-sample,action-result,swarm-observation}). The Swarm-era family retires with coreflow, not with this release. Every one of those routes reads at most the maxRequestBytes its own contract states, within ten seconds, and every request and response body carries protocol; the request and response shapes are owned by @serve.zone/interfaces, so Cloudly and Spark no longer carry private copies of them.

The four sparkSwarmNodeContracts bodies are admitted by the contract's own validators -- validateSparkSwarmNodeHeartbeatRequest, validateSparkMetricsSampleRequest, validateSparkActionResultRequest and validateSparkSwarmObservationRequest -- so a node's body is judged by the exact-key rules it was written by rather than by a second set of member checks here. The order is the one every Spark route keeps: bounded read, the offer, the node the body names, the contract's validator, then the credential store. A refused body is answered 400 naming the member the contract refused -- the observation route answers its own invalid request, because an observation is digested as a whole -- before the node behind it is authenticated and without reaching the credential store. acceptSparkHeartbeat, acceptSparkMetricsSample and acceptSparkActionResult take unknown and run the same validators, so an in-process caller is held to exactly what the route admits.

The node heartbeat body accepts the published host report from the independently enrolled Spark credential. It contains Linux host metrics, bundle identity and the explicit local-ready/workload-unverified classification. It never leases operator actions or supplies Docker, Swarm or container-count observations.

The current Spark generation and session epoch fence a single SmartData transaction containing node liveness, the separate data.sparkNodeReport, and the cluster's public node snapshot. The response is a secret-free historical acceptance receipt. Rotation or revocation racing this write prevents all of those effects from committing. Shutdown closes admission and joins accepted operations before storage closes. Invalid requests return bounded responses without reflecting credentials or database errors.

The node and cluster views display the report and its acceptance time. Local readiness never means workloads are ready. A node that reports through the Swarm-era family clears the node-runtime presentation record, so a Swarm worker's own report is what that node presents. This source integration does not activate production workers.

Node Onboarding with Jump Codes

A fresh host becomes a node of a cluster by enrolling itself with a single-use Jump Code. One bootstrap script serves that flow, and two routes render it:

• /jump/<code> for a host an admin onboards by hand. The dashboard displays the command

  curl -fsSL 'https://cloudly.example.com/jump/<code>' | sudo bash
  

  and the same URL answers a browser with a landing page and a curl/CLI client with the script (/jump/<code>/setup.sh always answers the script).
• /curlfresh/setup.sh?jumpcode=<code> for an auto-provisioned server. Hetzner provisioning mints one single-use code per server and passes exactly this URL in the server's cloud-init document, so one code never bootstraps two hosts. No cluster token travels with a provisioned server.

The script runs under set -euo pipefail as root. It installs curl and ca-certificates, resolves the newest published Spark v<major>.x release of Cloudly's own major line from the Gitea releases API — paging until it finds one, refusing by name when that major publishes none — and hands that exact tag to install.sh --node-bundle --version <tag>, which verifies the asset against its SHA256SUMS.txt. The node bundle carries Pallet, so nothing installs Docker: a node of this line runs its workloads through Pallet's own container runtime.

Enrollment is one JSON document on stdin, and the Jump Code is its bearer:

{"operation":"jump","cloudlyOrigin":"https://cloudly.example.com","hostname":"node-a","bearer":"<16-character Jump Code>"}

printf is a bash builtin, so that document exists only in the pipe spark enrollnode reads; the code never enters a process argument list, an environment variable or a file. The hostname comes from hostname -s, lowercased, and a host whose name is not one lowercase DNS label is refused by name before it enrolls. Cloudly's script states one assumption about enrollnode: it exits non-zero when it refuses an enrollment, and writes its SPARK_ENROLLMENT record to stdout when it succeeds. The script acts on that exit status alone and never parses the record. A successful enrollment is what creates the node record, its Spark and Pallet credentials and its replay receipt, in one transaction inside Cloudly — provisioning pre-creates nothing, so the fleet only ever holds records of hosts that actually came online. The script then runs spark installnodeservice, spark activateguard and systemctl enable --now smartdaemon_spark.service.

Rendering a bootstrap script requires an https SERVEZONE_PUBLIC_ORIGIN, because node enrollment accepts no other scheme. Without it the landing page, the Jump script route, the CurlFresh route and Hetzner provisioning all refuse by name instead of serving a script that cannot enroll, or minting a code that could not be used.

An auto-provisioned server also carries its hardware identity in its code. Provisioning allocates the BareMetal record's id before it asks the provider for a server, mints that server's code naming that record, and writes the record under exactly that id once the provider answered. The enrollment that consumes such a code assigns its new node to that record in the same transaction, and a record that already belongs to a different node refuses the claim instead of being overwritten. A host can enroll before the provider's answer is written, so a claim that finds no record yet is not a failure: provisioning notes on the code when it wrote the record, and that same write assigns whatever node the code has meanwhile created. Codes an admin mints name no hardware and assign none, and Jump documents written before this link existed are never backfilled.

Jump Codes expire by default after 30 minutes — at most 24 hours when a caller asks for more — and are consumed by the first successful enrollment. A code that is unknown, expired or already used is answered 404 jump code is invalid, expired, or already used, never a script.

POST /jump/claim and Spark's claimjump command remain for the unconverted Swarm hosts that still run a Coreflow daemon: that path consumes the same single-use Jump Code, creates the node record itself and answers with the cluster runtime token those hosts need. No rendered script calls it any more. It retires with the Swarm generation, once the fleet is converted.

Registry and Deploy-On-Push

Cloudly serves an OCI registry under /v2 through CloudlyRegistryManager. The registry uses configured S3 storage and issues OCI tokens from Cloudly authentication state.

For Cloudly-managed services, getServiceRegistryTarget() creates stable registry targets like:

<cloudly-host>/workloads/<service-name>-<service-id-prefix>:<tag>

Registry push hooks record tag/digest metadata on the linked image and service. Legacy services may still use automatic reconciliation unless deployOnPush is explicitly false. The immutable deployment API requires deployOnPush: false and never promotes latest or another mutable reference.

Registry token requests authenticate human passwords against the private credential authority and accept unexpired machine-user tokens. Human OCI sessions carry the generation captured during password verification; Cloudly's SmartRegistry auth provider checks current authority on validation and authorization. Admins retain general write access. API machine users can obtain a write-capable registry token, but every manifest PUT is separately authorized against one active deployment operation, persisted service grant, exact repository, and exact non-latest release tag.

Safe Immutable Deployment Flow

An administrator uses configureServiceDeploymentMachineUser to attach a persisted grant to an existing service or one exact future service slot. The response confirms the grant but never returns or rotates token material. Routine deployment requests then authenticate as that API machine user; they do not require global administrator authority.

The typed workflow is:

1. getDeploymentPreflight evaluates the requested reserve, promote, or route phase from persisted permissions and current state.
2. reserveServiceDeployment durably reserves the service slot, namespace, registry repository, exact release tag, and routes. Replaying the same actor/idempotency key/request returns the original operation; changing the request fails closed.
3. For greenfield mode, Cloudly creates an immutable provisional service with deployOnPush: false, a placement hold, no public domains, a service image, and canonical empty Secrets v2 configuration. Existing-service mode verifies ownership and the current rollout-generation fence.
4. The deployer pushes a multi-platform OCI index to the returned repository and exact release tag. Cloudly records the root index digest and server-created evidence bound to the operation, actor, host, repository, tag, digest, and media type.
5. promoteServiceImageRelease accepts only that trusted release and persists a digest-pinned rollout with a fresh generation. A conflicting active rollout or stale generation is rejected.
6. getServiceDeploymentStatus reports rollout identity, counts, health, and per-task observed/reported digest evidence. Route promotion remains fenced until every required replica is healthy and verifies the exact digest.
7. promoteServiceDeploymentRoute publishes the requested route, resolves only public addresses, performs an SNI HTTPS request with certificate validation, rejects redirects, and verifies the configured readiness status. Failed route or TLS checks restore the previous route state and leave structured retryable diagnostics.

An explicit routes[].proxied boolean is persisted during route promotion and overrides prior provider metadata for the hostname. Historical operations that omit it preserve an existing proxy setting when one is available.

retryServiceDeployment and cleanupServiceDeployment are revision-fenced. Cleanup removes only operation-owned provisional resources and claims; normal service deletion also releases stable deployment claims.

An immutable rollout is scoped by every runtime that owns one of its targets. A service Coreflow owns is scoped by the Swarm identities its online nodes report, and proved by a connected Coreflow whose capability tag names each target node. A Pallet node reports an architecture and carries no swarmNodeId, and no Coreflow answers for its cluster. Each spec owns its scope only while its cluster is in a Pallet-owned phase (switching or pallet); a draining cluster is still Coreflow's until the switch. A service with a Pallet-owned spec is Pallet's alone: no Swarm node of another cluster joins its plan, whatever node records that cluster still keeps (Swarm node records are never retired by liveness), and no Coreflow is asked to vouch for it. A Pallet target takes its cluster and replica count from its spec, its target node names from the nodes of that cluster whose reported architecture states a container platform (x86_64 selects linux/amd64, aarch64 selects linux/arm64), and is refused as image-plan-spec-held while the spec holds the service, as image-plan-node-platform-unknown when no node of the cluster states a platform, and as replica-limit when nodes times replicas per node exceeds the contract's 64. A pinned placement narrows the scope to exactly the nodes it pins, and a pinned node that is no longer a node of that cluster with a platform — retired, moved, or no longer reporting — refuses the plan as spec-node-not-in-cluster rather than quietly planning the rest. The Swarm placement.hold is not read on this path: it belongs to the Coreflow runtime and the cutover leaves it set forever, so a spec-owned service's hold is spec.hold. Whether a target node currently holds a credential and a session is the assignment producer's question, not the plan's: a plan is durable desired state and must not change because a node reconnected.

The Coreflow capability tag is replaced there by the release's own platform evidence. A Coreflow-owned rollout seals its invocation over both platforms, because its plan names hostnames and states no architecture; a spec-owned rollout seals it over exactly the platforms its target nodes report, so a release built for one architecture is a legitimate release for a cluster of that architecture. A platform the release carries no exact evidence for refuses the plan as image-plan-platform-unevidenced, naming the nodes it would otherwise have admitted work their image cannot execute. Adoption stays a Coreflow operation: it takes custody of a deployment Coreflow is already running, so a service with a runtime spec is refused as adoption-runtime-spec-present.

BaseOS and CoreBuild

Cloudly can manage BaseOS nodes and image builds:

• BaseOS devices register through POST /baseos/v1/nodes/register and heartbeat through POST /baseos/v1/nodes/heartbeat.
• A configured encrypted BASEOS_JOIN_TOKEN system secret accepts generic device enrollment.
• BaseOS image builds create one-time provisioning tokens that are embedded in generated images.
• Cloudly selects a CoreBuild worker based on /corebuild/v1/capabilities and sends the build to /corebuild/v1/jobs/baseos-image.
• Supported build kinds are ubuntu-iso and balena-raw; Raspberry Pi builds use balena-raw.
• Supported architecture values are amd64, arm64, and rpi.
• Completed artifacts are stored in the configured S3 bucket and served through short-lived /baseos/v1/images/:buildId/download URLs.

CoreBuild worker routing uses the public corebuildWorkerUrl or corebuildWorkerUrls settings. A shared credential is stored under the encrypted COREBUILD_WORKER_TOKEN system-secret key; per-worker URL/token entries are stored as JSON under the encrypted COREBUILD_WORKERS key.

Cluster-Scoped Dispatch

Everything Cloudly sends into a cluster goes through the one socket that cluster's relay holds. ClusterManager.findLiveRelayConnection(clusterId) is the single election: a socket carries a relay when the server still enumerates it, it holds the Cloudly identity tag it was verified with, and this process accepted its registration. No live socket is relay-not-connected, and two live sockets for one cluster is relay-ambiguous — never a silent choice between two processes that both believe they serve the cluster.

Node-scoped work names its node: fireRelayRequestForNode(nodeId, method, payload) resolves the node's cluster from the node document, elects that cluster's relay, and puts the node id in the request body. A node Cloudly does not hold, and a cluster whose relay is not connected, are refused before anything is sent. The Swarm-era election — a connection tagged with the node hostnames it served — is gone, together with the readiness probe that waited on it: a cluster relay authors no tags, so those lookups had no author and every node-scoped call through them failed after a full timeout.

Strict config delivery is per cluster for the same reason. prepareClusterConfigDeliveryToRequiredClusters elects each required cluster's relay before the caller's durable mutation, and the returned delivery step pushes on those exact sockets and re-verifies each identity, so a socket that no longer speaks for the cluster it was elected for delivers nothing.

Fleet-wide Coreflow work is scoped by phase. The deployment view, the config broadcast, the capability reads and the workspace, logs and shell paths all select their sockets from the clusters whose phase is swarm or draining; a cluster that moved on keeps a socket, because its relay authenticates with the same cluster machine identity, and asking it to serve Coreflow requests would be answered by a runtime that no longer owns the cluster. A cluster Cloudly holds no phase for is not asked either, and is named in the log once per Cloudly process, so a socket that answers nothing is not a silent drop. Each socket's cluster identity is proved once, where the socket is admitted, and every fan-out reads it from there. Identifying an inbound report's connection is unchanged: evidence is checked against whoever proved a cluster identity, not against a phase.

Three Swarm-era paths have no Pallet equivalent yet and say so by name instead of pretending: per-node traffic statistics and resource usage answer node-telemetry-unavailable, service migration is refused at startMigration with service-migration-unavailable because the data move needs the per-node Corestore endpoint, and base-service reconciliation through Coreflow is gone entirely — base services are Pallet system workloads delivered as runtime assignments, so reconcileBaseServices keeps its name and states that, and reconcileNodeBaseServices is removed. Deployment route promotion no longer asks Coreflow to prove its routing; it still makes every promoted hostname certificate-ready before the routes are declared promoted, and a hostname whose certificate is not ready fails the promotion with DEPLOYMENT_CERTIFICATE_NOT_READY.

Cluster Platform Desired State

getClusterPlatformDesiredState answers one cluster's relay with the capabilities, provider configurations and platform bindings that cluster may act on. It carries no identity of its own: the cluster is the one the verified machine identity on the asking socket names, and that socket must also be the one the relay election named. Both facts come from the cluster manager's shared requireElectedRelayConnection, so this handler refuses with the same names every other relay-facing request does — relay-identity-unverified for a socket without a verified cluster machine identity, relay-not-connected for a socket this process holds no accepted relay registration for, and relay-ambiguous when two live sockets claim the same cluster.

The binding list is derived from the Corestore publication grants rather than from the bindings themselves. The relay refuses an ungranted binding with binding-not-granted, so serving a placement Cloudly did not grant would be Cloudly asking for work it did not authorize; deriving one list from the grant computation makes grant-before-serve hold by construction. A binding the grant computation dropped — incomplete authority, a disabled provider, or its own grant cap — is therefore absent from the answer, and the reason is logged with the binding id rather than left to look like a binding that does not exist.

Each remaining binding carries the node its Corestore resources live on. Corestore is node-local, so a binding lives on the one node its service's runtime spec pins in this cluster. The spec states that before any workload runs and keeps stating it while the service is held, so a database is placed before its service first starts -- which is what lets its credential reach the first start -- and stays placed while the service runs nowhere. A service without a spec for this cluster is omitted as binding-unplaced, and one whose spec runs on every node or on several as binding-placement-ambiguous, both by name in the log. Placing a database on "whichever node answered" is precisely what the relay replaced. A binding that adopts a legacy Corestore resource is placed on the node its corestoreLegacyAdoption names, whatever the spec pins, because that node's Corestore holds the data; a node outside the cluster leaves it binding-unplaced. The answer is sorted by node and then by binding id, so two reads compare directly.

The cluster config a relay is pushed carries the publication grants of exactly these placed bindings and no others. A relay retains every binding a node's Corestore holds that it holds a grant for but finds placed on no node, and a retained binding is a permanent tombstone, so a grant travels only with its placement.

The relay is also the one party that reads the Corestore control token (getCorestoreControlCredentialMaterial) and publishes binding credentials (publishCorestoreCredentialMaterial): both are answered to the socket the relay election names and to nobody else, the request's identity must be that relay's cluster, and the material is sealed to the recipient the relay enrolled. The token itself is stored with secrets set-corestore-control-token --stdin. The elected relay also reads Cloudly's secret ingress recipient (getSecretIngressRecipient), which it seals the published material to; a socket that registered a Docker-era Coreflow secret runtime reads it as that runtime, and every other cluster socket must be its cluster's elected relay.

updatePlatformBindingStatus stores the statusReason the responsible cluster reports with a status (legacy-adoption-pending with provisioning, legacy-adoption-conflict and legacy-adoption-mismatch with failed, checked by validatePlatformBindingStatusReason) and clears it on a report without one; an administrator reports no reason. upsertPlatformBinding refuses corestoreLegacyAdoption and statusReason beside objectstorageBucketName: all three are server-managed.

Corestore Endpoints

Corestore is node-local: the database and the bucket of a workload live on the node the workload runs on. While every cluster ran one coreflow per node, "this node" was whoever answered; one relay serving a whole cluster has no such implicit answer, and the cluster-local alias every component used while Docker Swarm resolved it is exactly what stops resolving. Cloudly therefore holds the endpoint per node, in the cluster node record, and the relay reads it.

setClusterNodeCorestoreEndpoint { nodeId; endpoint | null } is the administrator surface, guarded like every other node administration request. The origin is checked by the contract's own validateClusterNodeCorestore before anything is written — a canonical http: or https: origin with no path, query, fragment, userinfo or trailing slash — and the write takes the node's own real-write fence inside its transaction, so a node deleted while an administrator is setting its endpoint conflicts instead of leaving an endpoint behind for a node that is gone. endpoint: null clears it, which states the same fact as never having set one: this node runs no Corestore the relay may reach, and an inventory for it is answered CORESTORE_ENDPOINT_UNKNOWN by the relay rather than by guessing a hostname.

Every refusal is named: Node write refused: inactive for a node Cloudly no longer holds, invalid_input for an identifier no node can have, and conflict when the node or its cluster changed under the fence — a routine heartbeat is enough — so an administrator reads what to do next instead of an internal error.

The endpoint says where, never with what. The bearer control token that authorizes every Corestore call stays with getCorestoreControlCredentialMaterial, sealed to the recipient the relay enrolled, and no credential travels in the node document. What a relay reads is the cluster document, so the node write carries the cluster snapshot with it in the same transaction — the two land together or not at all, exactly as an accepted heartbeat writes them — and the cluster configuration is pushed afterwards, so every connected relay learns the change without reconnecting. The node detail view shows the current endpoint and edits it through the same request.

Object-storage endpoint

A workload dials its bucket at a different origin than the relay dials the control API, so a node states it separately in corestoreObjectStorage { endpoint, region }. setClusterNodeCorestoreObjectStorageEndpoint { nodeId; objectStorage | null } sets it under the same admin guard, fence, snapshot and refusals as the control endpoint, after the contract's validatePlatformObjectStorageEndpoint admits it (a canonical http: or https: origin and a lowercase region). The two requests never touch each other's field. objectStorage: null clears it: no object-storage binding placed on this node can be given its environment, and nothing derives a port, scheme or region in its place. No relay reads this field, so nothing is pushed; the service runtime producer composes from it, so the write asks it to decide again. The node detail view shows it and edits it through the same request; both fields empty clear it.

Corestore Inventory

collectCorestoreInventoryReport asks each node's own cluster relay what that node's Corestore holds, with getClusterCorestoreInventory { nodeId }: one relay serves a whole cluster, so the node an inventory describes is stated in the request and answered in the inventory rather than derived from whichever connection replied. The expected nodes are the cluster's own node records, by node id — a greenfield proof covers every node of the operation's target clusters, and every registered node while an operation names no cluster yet.

The node set is deliberately not filtered by reported status: a node whose Corestore cannot be proven blank blocks the proof by name, and a stale node record is deleted rather than skipped, because a node Cloudly still holds is a node a workload could land on.

Every node keeps its own diagnostic: a cluster with no live relay, an RPC that timed out or was refused, an answer the contract refuses, an answer about another node, and a node whose Corestore is unreachable are five different facts. A reachable inventory with an empty service list is the blank-namespace proof; an unreachable one proves nothing and never reads as "nothing provisioned". The greenfield preflight consumes exactly that: it refuses by name when any target node's probe is not ok, instead of swallowing the failure into an empty inventory.

Corestore Legacy Resource Adoption

A Corestore converted from a release before 32 still holds the databases and buckets /resources/provision created per service, and refuses a credential binding beside such a resource until an operator authorizes that exact binding to adopt it, offline.

getCorestoreLegacyResources { clusterId, nodeId } (administrators) reads one node's inventory through its cluster's relay with getClusterCorestoreLegacyResources { nodeId } and answers it with inventorySha256 (computeClusterCorestoreLegacyResourceInventorySha256, null for an unreachable Corestore). The node must belong to the cluster; a relay that is not connected or does not answer is legacy-inventory-unreachable, and an answer about another node or of a shape the contract refuses is an error, never an empty inventory.

adoptCorestoreLegacyResources { clusterId, nodeId, expectedInventorySha256, resources } reads the inventory again, refuses legacy-inventory-changed unless it still has the digest the administrator saw, and hands the selection to the adoption step in ts_migration/corestore-legacy-adoption.ts, which is the only writer of a binding carrying a legacy bucket name and retention intent. In one transaction it creates each adopting binding — enabled, on the canonical Corestore provider and management scope, with corestoreLegacyAdoption { nodeId, resourceName, inventorySha256 }, and for a bucket the legacy objectstorageBucketName and the legacy retention intent exactly — checked with validatePlatformBindingCorestoreLegacyAdoption, or creates none. A resource whose adopting binding exists already is answered unchanged. The refusals are ICorestoreLegacyAdoptionErrorData: legacy-inventory-unreachable (the only retryable one), legacy-inventory-changed, legacy-resource-absent, legacy-resource-blocked, legacy-resource-authorized-elsewhere, service-absent and binding-exists. The answer carries the bindings and the plan (createCorestoreLegacyResourceAdoptionPlan), exactly the file Corestore's offline legacy-resource-adoption migration reads, and the cluster config is pushed so the relay places the bindings on nodeId. Until the operator runs that migration, the relay reports each binding provisioning with legacy-adoption-pending, and its service waits as legacy-adoption-pending.

The relay handler getClusterCorestoreLegacyResources belongs to Coreflow 33; a relay without it answers the read as legacy-inventory-unreachable.

Backups and Corestore

Cloudly owns backup records and user-facing backup/restore requests. Coreflow executes the cluster-local work, and Corestore snapshots volumes, database resources, object storage resources, and archive objects.

The backup path includes:

• createServiceBackup and restoreServiceBackup typed requests for admins.
• backupClusterService and restoreClusterService requests from Cloudly to the cluster relay, each naming the node whose Corestore holds the data.
• backupServiceDatabaseClosure and restoreServiceDatabaseClosure requests from the cluster relay, which stream a service's database closure to Cloudly and back (see Database Closure Streams).
• Corestore volume/resource snapshot and restore endpoints behind the relay.
• Optional archive replication through prepareBackupReplication, uploadBackupArchiveObject, completeBackupReplication, getBackupArchiveManifest, and downloadBackupArchiveObject.
• Optional scheduled backup-all-services task when CLOUDLY_BACKUP_CRON is set.

Corestore data is node-local, so a backup or a restore must name the node whose Corestore holds it: a request that names none is refused by name (backup-node-required) instead of being sent to whichever socket of the cluster answered first. The named node is resolved to its cluster node id at the request boundary — a name that matches no node, or more than one, is refused with node-unresolved rather than reaching an arbitrary node's data — and the dispatch then travels over the relay of that node's cluster. Archive pruning is per node the same way, through pruneClusterNodeArchive, and stays best-effort: a node whose relay is gone keeps its cache until the next pass.

Manual createServiceBackup requests require clusterId when Cloudly manages multiple clusters; it may be omitted only when exactly one cluster exists. In the cluster-scoped flow Cloudly authors the replication record itself, from the database closure it accepted, and the relay no longer does (see Database Closure Streams). The archive replication requests still validate archive object size and SHA-256 checksums, write a manifest, record target metadata, and mark completed backups as replicated. Restores read the manifest and objects back through the configured target writer.

Database Closure Streams

A cluster is outbound-only, so Cloudly never reads a cluster's Corestore: both directions of a database closure are relay-initiated. backupServiceDatabaseClosure carries one service's closure from the relay to Cloudly as a VirtualStream, and restoreServiceDatabaseClosure serves it back. Neither request carries an identity — the cluster is the one the asking socket authenticated as and was elected relay for, through the same requireElectedRelayConnection the desired-state request uses — and both name the node, because Corestore is node-local.

Every refusal of these two requests reaches the relay by name as a TypedResponseError. Its error.text is Cluster relay operation rejected: <code>. for a code of the relay vocabulary and Closure stream refused: <code> for a code of the closure vocabulary:

Vocabulary	Code	Meaning
relay	invalid	The request has no live request context, or the asking socket changed while it was verified.
relay	relay-identity-unverified	The asking socket carries no verified cluster machine identity.
relay	relay-not-connected	The asking socket is not its cluster's elected relay.
relay	relay-ambiguous	More than one live socket claims the relay's cluster, so Cloudly serves none of them.
relay	relay-node-foreign	The named node is not a node of the relay's cluster, or does not exist.
closure	closure-request-invalid	The backup, service or node id is malformed.
closure	closure-backup-unknown	No such backup for this cluster and service; another cluster's backup reads the same, so no relay can probe backup ids.
closure	closure-not-awaited	The backup is not waiting for a closure: it already holds one, one is being received, or the backup is no longer running.
closure	closure-descriptor-invalid	The descriptor fails the contract's own validateCorestoreDatabaseClosureDescriptor.
closure	closure-descriptor-foreign	The descriptor's receipt names another backup or service than the backup it is announced for or held under.
closure	closure-not-held	A restore asked for a closure Cloudly does not hold, or whose manifest fails its checksum.
closure	closure-unavailable	Cloudly cannot take or serve closures right now: the cache tier cannot be opened or cannot create exact-length objects, a held manifest cannot be read, or the backup manager is stopping.
closure	closure-size-mismatch, closure-digest-mismatch	Never an answer: bytes that disagree with the descriptor fail the stream itself, and the backup records the code.

The code lives in ts/manager.backup/closurestreams.ts (ClosureStreamServer: the two handlers, stream lifecycles, the cache-tier store and the stream authorizations) and ts/manager.backup/closurerecords.ts (ClosureBackupRecords: which backup a closure belongs to and what storing it means for that backup's record). The backup manager wires them and only decides the backup's state.

Cloudly stores the closure while it streams: the bytes pass a counting, hashing transform straight into fastPutStreamExact at the exact length the descriptor announced, so the acceptance after the last byte is immediate and fits the relay's 120 s acceptance window. A closure whose length or digest disagrees fails inside that transform before its end reaches the store, and an exact upload publishes nothing whose source did not end cleanly, so a wrong closure is never stored at all. One backup takes one closure: the backup id is reserved before its record is read, so a concurrent or later announcement is refused closure-not-awaited and can never replace or remove a closure the backup holds.

The contract's descriptor validator proves shape and self-consistency only, so Cloudly also binds the descriptor to its backup: the receipt's backupId must be the announced backup, and its sourceServiceId that backup's service. Cloudly checks this before anything is stored, and again before it serves a stored descriptor back, because a closure of another service held under this backup would restore that service's database into this one.

The backup manager opens and probes the cache tier when it starts. A relay-minted stream expires unless it is opened within the transport's 10-second capability deadline, and opening and probing a cache tier can take longer than that, so the first closure after a start does not wait for it. A failed open is logged and forgotten; the next announcement opens the tier again.

The backup record is the commit point. Cloudly stores the closure object, writes the closure manifest (<backup prefix>/closure/manifest.json), and then commits the record with a conditional update that only a backup still running without a replication record takes; the stream is accepted only after that commit, so the receipt the relay reads is never ahead of the record an operator reads, and an acceptance lost after the commit leaves Cloudly holding a closure it recorded, which it logs. A commit the record does not take removes the manifest and the object again. A record write that reported an error may still have applied, so Cloudly reads the record back: a record that refers to the closure manifest keeps everything and the stream is accepted, and one that does not has the manifest and the object removed. When that read fails too, the outcome is in doubt: the stream is refused and everything is kept, because removing a closure a committed record refers to would lose the backup. Deleting a backup that holds no replication record removes its prefix on the configured cache tier, so whatever no record took goes with the backup.

On a restore the relay asks Cloudly first, before any node-local step, so a backup Cloudly holds no closure for is refused closure-not-held from a manifest lookup that opens no stream at all — that ordering is what keeps a node from being half restored. The descriptor travels with the stream because Corestore opens its restore from it, and Cloudly declares the stream's integrity from the same descriptor. The served stream's authorization binds only on the asking socket and is revalidated on every framed operation, so it ends when that socket goes away or the backup manager stops.

Cloudly is the replication target of the cluster-scoped backup flow, so it authors the replication record itself from the closure it accepted; the relay's answer to backupClusterService decides no replication record. Admission answers before the bytes move, so after the relay answers Cloudly first waits for that backup's own closure transfer to settle (never for another backup's). An accepted closure makes the backup replicated with a record of one object of exactly the closure's size, and a manifest that states the closure, its node and its descriptor and lists no archive objects — volumes and object storage have no streamed contract yet, so nothing claims they were replicated. A backup that asked for replication, produced a database snapshot and holds no closure is failed with database closure not replicated: <observed>, where <observed> is only what Cloudly saw:

• no closure was received for this backup — Cloudly admitted no announcement for this waiting backup: either the relay announced none, or Cloudly refused one before it resolved the backup, which Cloudly logs under the backup id once it has parsed the announcement, as described below;
• closure-descriptor-invalid: <reasons>, closure-descriptor-foreign: <the backup and service the receipt names> or closure-unavailable: <cause> — Cloudly refused the announcement after it had resolved the waiting backup, for example because the cache tier failed its exact-upload probe, and logged the cause under the backup id;
• closure-stream-reset — the stream ended before the closure was stored. The transport resets with a constant text, so the relay's reason stays in the relay's log under the same backup id; a reset Cloudly caused because its storage failed is logged by Cloudly under that id;
• closure-size-mismatch or closure-digest-mismatch — the bytes were not the announced ones;
• closure-unavailable: <cause> — the closure was stored, but the record did not take it and it was removed;
• closure-unavailable: its commit could not be read back — the commit's outcome is in doubt, and the closure is kept as described above.

A refusal made before the waiting backup is resolved records nothing, because no backup has been proven to be the one waiting: every refusal of the relay vocabulary, closure-request-invalid, closure-backup-unknown, closure-not-awaited, and closure-unavailable for a manager that was already stopping. Once Cloudly has parsed an announcement it logs every refusal of it under the backup id the announcement named, so closure-backup-unknown, closure-not-awaited and relay-node-foreign leave a trace. A refusal made before the parse — invalid, relay-identity-unverified, relay-not-connected or relay-ambiguous for the asking socket, closure-request-invalid, or a manager that was already stopping — is not logged under a backup id, because none was parsed. Only a backup that is still running takes a failure record, so a backup that already finished keeps its own reason, and a later attempt that stores the closure clears what an earlier one recorded.

A backup replicated as a streamed closure stays on the cache tier: moving one to the external tier would read it whole into memory, and no streamed external-tier contract exists yet.

Stopping the backup manager refuses new transfers, waits for admissions still deciding, and drains the admitted transfers — a received closure may still record what it stored, because the archive writers close only after the drain. Nothing opens the cache tier once the stop began, so the stop closes the one store there is. A restarted manager takes closures again.

The server installs one VirtualStream authorization adapter over several owners (ts/virtualstreamdispatcher.ts): the image manager owns image download authorizations, the backup manager owns closure ones, and each owner binds only what it minted. An authorization nobody claims is refused stream-authorization-unclaimed rather than as one feature's denial, and one claimed by two owners is refused stream-authorization-ambiguous, naming both, instead of being guessed.

Task Automation

Cloudly registers a TaskBuffer-backed task manager. The API and dashboard can list tasks, trigger enabled tasks manually, inspect execution logs/metrics, and request cancellation for running tasks. Disabled tasks cannot run manually or on a schedule.

Predefined tasks currently include:

Task	Status	Purpose
dns-sync	Disabled	Iterates DNS entries marked as external; provider sync is currently a placeholder.
cert-renewal	Disabled	Checks activated domains for certificate renewal; renewal logic is currently a placeholder.
cleanup	Disabled	Removes old task executions and contains placeholders for log/image cleanup.
health-check	Disabled	Iterates deployments and records health metrics; runtime health checks are currently placeholders.
resource-report	Disabled	Generates node resource metrics; values are currently placeholders until runtime metrics are wired in.
db-maintenance	Disabled	Maintenance shell for database optimization tasks.
security-scan	Disabled	Security scan shell for exposed ports, image freshness, and weak configuration checks.
docker-cleanup	Disabled	Docker cleanup shell for containers, images, volumes, and networks.
backup-all-services	Conditional	Registered by the backup manager and enabled only when CLOUDLY_BACKUP_CRON is set.

External Gateway Integration

Cloudly can integrate with a dcrouter gateway when the public gateway settings and encrypted gateway system secret are configured. The current integration syncs externally available domains into Cloudly, syncs public service routes directly to dcrouter, and fetches or reprovisions certificates through dcrouter. Coreflow consumes the resulting Cloudly service configuration and keeps cluster-local Coretraffic routing focused on internal workload traffic. A service a Pallet runtime spec owns is routed to the uplink address the one node its spec pins reported, where that cluster's ingress and its own published ports answer; a Pallet spec that pins no single node, or a pinned node that reported no uplink, refuses the sync by name. The gateway client also holds Cloudly's cluster relay names (see Cluster Relay Names).

Service mail can also be configured on a service. Each address can enable inbound mail through a dcrouter smtpForward target to a published service port and outbound mail through a dcrouter-managed SMTP credential. Cloudly stores only public credential metadata in service.data.mail; one-time SMTP secrets are written as service-owned Secrets v2 entries with sanitized address-token variables such as MAIL_PLATFORMTEST_GATED_ONE_SMTP_* plus default SMTP_* aliases for the selected default sender.

Development

Common commands:

pnpm install
pnpm build
pnpm test
pnpm run build:docker
pnpm run release:docker
pnpm run docs

Important paths:

Path	Purpose
ts/index.ts	CLI/runtime entry point exporting runCli, Cloudly, and ICloudlyConfig.
ts/classes.cloudly.ts	Main service coordinator and startup order.
ts/classes.server.ts	API/dashboard server, registry bridge, and BaseOS HTTP routes.
ts/manager.*	Domain managers for auth, clusters, services, images, registry, platform, backups, BaseOS, and more.
ts/connector.*	The MongoDB connector.
ts_web/	Browser dashboard web components.
ts_cliclient/	@serve.zone/cli submodule, published with every Cloudly release at the Cloudly version.

Accuracy Notes

The package metadata and settings schema include fields for several cloud providers. The code paths currently exercised in this repository are Hetzner for selected node/bare-metal provisioning paths, S3-compatible storage, SMB/S3 backup archive targets, MongoDB/SmartData, CoreBuild, Coreflow, Corestore, and dcrouter, which holds every DNS record and certificate Cloudly needs. Several provider connection tests and predefined tasks are configuration checks or implementation shells; verify provider-specific behavior in the relevant manager before relying on it operationally.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.