@serve.zone/cloudly

Cloudly is the serve.zone control plane: a TypeScript service and browser dashboard that stores desired infrastructure state, authenticates humans and machines, coordinates clusters, serves an OCI registry, manages workload metadata, and pushes runtime configuration to connected node components.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Why It Exists

Cloudly is the place where serve.zone operators describe what should run. It does not directly run every workload itself. Instead, it keeps the authoritative desired state in MongoDB and exposes TypedRequest/TypedSocket APIs so runtime components can reconcile that state where the containers actually live.

The current runtime pattern is reverse-connect:

browser / CLI / SDK
  -> Cloudly HTTP + TypedSocket API
      -> MongoDB-backed managers
      -> S3-backed image and artifact storage
      <- Coreflow cluster agents connect outward
          -> Docker Swarm reconciliation
          -> Coretraffic routing updates
          -> Corestore platform resources and backups

What Cloudly Manages

Cloudly currently coordinates these areas:

• Authentication and identity: human admin login, JWT identities, machine tokens, and cluster identities.
• Clusters: desired cluster records and machine users used by Coreflow to authenticate back to Cloudly.
• Services: workload definitions, image references, domains, ports, scale factors, value-free secret configuration, service mail addresses, volumes, and deployment metadata.
• Deployments: deployment records, node placement metadata, health/resource fields, restart/scale API stubs, and DNS activation/deactivation hooks. A record mirrors a runtime task a Coreflow currently reports: a report deletes the records of its service that it no longer names (within the reporter's node scope), and a service deletion removes all of them in one write. No archived history is kept, so getDeployments answers the current records whether or not it is asked to includeArchived, no record carries archivedAt, and the dashboard's Deployments view lists the current deployments only.
• Immutable deployment operations: scoped deployer grants, durable reservations, exact OCI release evidence, rollout fences, runtime digest status, route promotion, and HTTPS readiness evidence.
• Images and registries: image metadata, S3-backed image storage, external registry records, and an embedded OCI registry mounted at /v2.
• Secrets: encrypted Secret/SecretVersion material, organization SecretSets, service attachments, lifecycle state, revision fences, retention metadata, and value-free resolution previews.
• Domains, DNS, and mail: domain records, DNS entries, optional domain sync from a dcrouter external gateway, and service mail binding reconciliation.
• Platform bindings: capabilities such as database, objectstorage, pushnotification, logging, backup, and RPC-style platform services. Web Push is provider-managed; storage capabilities remain Corestore-managed.
• Backups: backup records, service backup/restore requests, scheduled backup tasks, and archive replication handshakes with Coreflow/Corestore.
• BaseOS: managed BaseOS node registration, heartbeat handling, desired-state response, image build tracking, and image download URLs.
• CoreBuild workers: selection of external build workers for BaseOS ISO and balena raw-image artifact generation.
• Tasks: TaskBuffer-backed operational tasks with execution history, metrics, logs, manual triggers, cancellation, and cron schedules.
• Node and bare-metal inventory: Hetzner-backed node creation paths and bare-metal metadata records where configured.
• Dashboard: a web component UI rendered from ts_web. Secrets are service-first: a service detail carries a Secrets tab with Values, Resolved and Rollout, backed by an organization-wide secret index at /secrets/secrets and shared SecretSets at /secrets/secretsets. Values are write-only everywhere; an operator-copy value is generated in the browser and revealed once inside its unsaved dialog, because no API returns a secret value.

Runtime Components

Component	Role
Cloudly	Main service coordinator. Creates connectors and managers, then starts the API server.
CloudlyServer	TypedServer/TypedSocket HTTP server, dashboard static server, OCI registry HTTP bridge, and BaseOS HTTP endpoints.
MongodbConnector	SmartData persistence layer for Cloudly records.
CloudlyCoreflowManager	Authenticates Coreflow, returns cluster config payloads, and pushes config updates to connected Coreflow clients.
CloudlyJumpManager	Creates short-lived Jump Codes for onboarding existing systems into clusters.
CloudlyRegistryManager	Embedded OCI registry backed by configured S3 storage, including exact-tag release evidence and immutable rollout promotion.
DeploymentOperationManager	Durable existing-service and greenfield deployment operations, resource/route claims, retry/cleanup, rollout status, and public HTTPS verification.
CloudlyBaseOsManager	BaseOS registration, heartbeat, image build orchestration, worker selection, and artifact downloads.
CloudlyBackupManager	Service backup/restore orchestration and remote archive object replication.
CloudlyTaskManager	Registers predefined and runtime tasks, tracks task executions, schedules cron jobs, and exposes task APIs.
CloudlySettingsManager	Stores an allowlisted public settings DTO in MongoDB and refreshes gateway/Coreflow state after relevant changes. Credential material is not part of the settings response.
CloudlyFleetCutoverManager	Owns authenticated fleet census/scope classification, non-ready prepared volume-handoff intent, isolated replacement-ingress designation, and the node-owned Spark legacy retirement receipts. It keeps of each node's accepted Spark observations only the rows a reader reaches: the current and previous sequence of the node's current reporter session, which the census reads, and the rows an accepted census names, which a prepared handoff reads; the transaction that accepts an observation removes the rest.
CloudlyClusterIngressManager	Designates the ingress of a cluster born on Pallet, authenticates the elected ingress, and publishes revisioned whole-cluster route tables (non-public hostnames restricted to dcrouter's forwarding sources), withdrawing exactly the revoked routes first.

Private Node Enrollment Qualification

The backend-private NodeEnrollmentStore consumes Interfaces 28.1's shared Spark/Pallet enrollment contract. It is not registered with runtime routes yet: both node-local identity owners and the replacement installer must qualify before the old Jump claim writer and endpoint are retired in the coordinated cutover. This source capability does not mean deployed workers use Pallet.

One owned SmartData transaction consumes an unexpired Jump, creates its node and cluster snapshot, issues independent Spark and Pallet hash authorities, and writes an exact combined receipt. Existing-node adoption instead fences the current Spark generation/session, rotates Spark, requires Pallet absence, and preserves the existing node. Cloudly never generates or stores either new plaintext bearer.

Bootstrap proof authorizes only the first commit. Lost acknowledgements recover the same complete proposal with its pending Spark bearer, only while both issued credential generations/hashes remain active and the node exists. Pallet reporter reconnect may advance the session epoch without invalidating that receipt. Revocation, rotation, changed proposals, expired/consumed or duplicate-hash Jump records, and missing nodes fail closed. A private node enrollmentFence nonce ensures a real transactional write against concurrent deletion even with a frozen clock; it is not a credential, generation, or public node field.

Fleet Cutover Preparation and Cluster Ingress

Cloudly registers getFleetCutoverScope, classifyFleetCutoverScope, prepareFleetCutoverVolumeHandoffs, and prepareFleetCutoverIngress. Prepared volume handoffs are desired, non-ready filesystem intents derived from current Service volume authority and authenticated node mount observations. They do not claim provider readiness or physical adoption. The current aggregate uses immutable node-bound pages and a compare-and-swap pointer, so incomplete or superseded staging cannot become current and exact replay does not renew authority. Docker volume preparation joins each node-local physical mount to the manager's logical service mount through complete one-to-one correlation evidence. Cloudly keeps the physical mount identity in the handoff and refuses observations that omit, duplicate, add, forge, or mismatch a correlation.

Ingress preparation designates a held replacement Service with no public host ports and creates its platform-managed cluster bearer. The cluster-ingress owner then authenticates registerClusterIngress, serves getClusterRouteTable, and pushes complete revisioned route tables. A revoked route is withdrawn at once, before any certificate is fetched, by a table that keeps every route still authorized; a revoked designation withdraws all of them. Secret material, certificates, and route tables are not persisted in its durable records, so after a start Cloudly's first table of a cluster is the full one, never an empty one: until it is issued, the relay's read is answered { table: null } and the relay keeps the table it holds. A route endpoint is a node instance's statement under its credential generation, so a node that only registers again (after a Cloudly restart or a relay re-registration) changes no route, while a restarted Pallet, a new credential generation or a deleted node withdraws its routes at once. A route whose hostname the table cannot carry, such as a wildcard, whose certificate cannot be obtained, or whose source conflicts with its runtime policy or with another service's route is left out of the table while the other routes are still pushed, and is logged with its name, cluster and failure; a hostname a deployment route claim assigns is served for its owner alone, and one two services declare without such a claim for neither, while a service that cannot be a route source, such as one being deleted, leaves out only its own routes. Only route sources that cannot be read withdraw the whole table — within 180 s of a start, while Cloudly holds no table of the cluster yet, they keep the relay's table instead, and the clusters still waiting at the deadline are named in one line; that withdrawal, and a refresh or sweep that fails, is logged with its failure.

Non-public hostnames. dcrouter forwards every domain of a service to the node, whatever its source profile, and checks the profile per request; the node ingress must therefore serve a non-public hostname to dcrouter alone. A domain whose sourceProfileRef is not PUBLIC compiles into a route whose sourceAllowList is the setting dcrouterForwardSourceAddresses, the addresses dcrouter's forwarded connections arrive from at a node; the ingress refuses every other source for that hostname. The allowlist is part of the route's identity, so a change of the setting revokes the route and reissues it, and a write of the setting refreshes every cluster's table at once. Such a domain is left out by name, while public routes are still served, with one of three reasons:

Reason	Fix
no gateway forwarding source is configured (dcrouterForwardSourceAddresses), so the ingress cannot restrict this non-public hostname to the gateway	Set dcrouterForwardSourceAddresses.
the cluster ingress cannot enforce a source allowlist: its registration does not state the route-source-allowlist capability	Run a CoreTraffic that registers clusterIngressCapabilities.routeSourceAllowList under the current designation.
the cluster relay would refuse a route table carrying a source allowlist: it speaks @serve.zone/interfaces <version>, below 32.55.0	Upgrade the cluster's Coreflow relay.

The relay's release is the interfacesVersion of the offer its live registration was admitted with, kept with the connection and never stored (clusterRouteTableMemberMinimums in ts/protocol.ts); while no relay is connected nothing is withheld, because nothing is sent until a relay registers and states its release. Public port mappings stay public-only.

A cluster born on Pallet never ran Swarm, so it has no cutover scope, census or target set to rest an ingress on. designateClusterIngress (admin) designates one existing service as its ingress (IClusterIngressDesignation): the request is a compare-and-swap against the cluster's current Pallet runtime, the service's exact runtime spec revision and the current designation, and it mints the platform-managed bearer SERVEZONE_CLUSTER_INGRESS_BEARER (scope platform:cluster-ingress:<clusterId>) in the same transaction. Designating the same service again rotates the bearer under a new generation; designating another service revokes the previous one's bearer with the change. A current bearer that is no longer active, such as one revoked with its service's deletion, is not revoked again and does not block a new designation: the same service recovers its revoked bearer secret with a fresh version (the revoked version stays revoked), and another service takes over as on any move. A service whose own bearer was revoked earlier, by a move away, recovers it the same way when it is designated again. Every other fence of the request still holds. A replayed mutationId answers the recorded result, and one reused for another request is refused mutation-reused. getClusterIngressDesignation reads it. A cluster holds one kind of designation: a cluster the fleet cutover designated is refused cutover-designation-present, and prepareFleetCutoverIngress refuses a cluster designated here. The ingress may register from the nodes its service's spec places it on.

Ingress environment. While a designation of either kind names a service its cluster's ingress, every replica of that service runs with the contract's clusterIngressEnvironmentNames: the bearer through its secret manifest, and SERVEZONE_CLUSTER_RELAY_ORIGIN (the cluster's exact IClusterRelay.origin) and SERVEZONE_CLUSTER_NODE_ID (the node the replica runs on) added to its environment by composeClusterIngressEnvironment. They follow from the designation existing, not from it naming the current runtime; a service no designation names gets neither. A cluster that states no valid relay refuses its ingress cluster-relay-missing (an administrator enables the relay with enableClusterRelay), and an ingress service whose own environment sets one of the three names is refused cluster-ingress-environment-conflict. Both values are part of the desired key, so a relay origin that moves replaces the ingress replicas.

Such an ingress dials the workloads themselves rather than published host ports: a domain routes to every ready execution at its workload lease address and the domain's target port. Each node's signed network projection grants the ingress workload a one-way flow to exactly that protocol and port of each target lease (workloadIngress, authorized by the designation), on the node of the ingress lease and on the node of the target alike, and a port is routed only once every ingress lease's grant is signed on both. A public port mapping is not an ingress route on such a cluster: it stays the workload's own publication on its node's uplink. Losing the designation, or a designation that no longer names the cluster's current runtime, withdraws every grant.

The ingress workload itself is reached from the internet through its own published ports, which its organization owner authorizes with setServicePublishedPorts like any other service's. For CoreTraffic in cluster-ingress mode that is exactly:

Protocol	Host port (node uplink)	Target port	Purpose
tcp	80	7999	HTTP
tcp	443	8000	HTTPS
udp	443	8000	HTTP/3 (QUIC)

CoreTraffic's status port 3000 is not published: it stays inside the runtime network.

registerClusterIngress answers every refusal except a protocol mismatch as the contract's named IClusterIngressRegistrationRefusal (cluster-ingress-registration-refused, in the error's data): not-ready while Cloudly cannot judge registrations (starting, stopping, no environment, or an internal failure), registration-invalid, relay-not-connected for a relay that is not the cluster's elected one, ingress-not-designated when the cluster holds no ingress designation of either kind (a designation that cannot be read is not-ready, and is logged), node-mismatch, assignment-not-running when the designated service has no single ready assignment on that node under its current session, manifest and bearer, bearer-refused, and conflict when the designation, assignment or session moved while it was judged. retryAfterMs is 5 s for not-ready, relay-not-connected and assignment-not-running, 1 s for conflict and 60 s for the rest. Neither the text nor the data quotes the bearer.

A Swarm-era Spark states what it is replacing on the observation it already posts. The optional fleetCutoverLegacyRetirement member is admitted in every observation state, because it describes the host rather than its Docker daemon, and it names no node: Cloudly binds it to the reporter it authenticated. Each accepted statement advances two node-owned records together -- the mode refusal and the configuration retirement -- under one named observation, because the Coreflow retirement evidence that cites both requires them to name the same accepted observation. A changed configuration is judged only by the contract's successor rule: one that does not advance is refused under that rule's own name and nothing is written. A restatement of the accepted pair renews nothing, and a new bundle that restates the same configuration is a new revision of both receipts, because the mode refusal now names that bundle. A bundle at or below the newest published Spark that carries no refusal at all is refused legacy-retirement-bundle-below-minimum, which is the one part of this the contract leaves to the server. Assembly joins both receipts through the contract and requires the observation they name to still be fresh, because an omitted member advances nothing and a node that stopped refusing must stop looking retired.

Every one of those answers travels on that observation's own accepted receipt, as fleetCutoverLegacyRetirement: accepted at the generation both records advanced to, restated for the replay that renewed nothing, or refused by the contract's own name and never retryable, because resending one exact statement cannot change what the records say. A refused statement leaves the observation accepted -- a node's liveness and census are true whatever its released runtime states -- and the member is present exactly when the observation stated one, which the contract binds in both directions. An exact observation replay answers the receipt that acceptance wrote rather than judging the statement a second time. Where an evidence member is refused for this body and every resend of it, the observation itself is rejected evidence-refused and nothing is recorded; a condition a later attempt can pass -- a credential this attempt could not fence, a concurrent mutation -- stays acceptance-race. Answering the member raises Cloudly's sparkSwarmNode minimum peer version to 32.18.0, because a receipt is read exact-key too and an older Spark would refuse one carrying a member it does not know.

Cloudly intentionally does not register advanceFleetCutoverPhase or getFleetCutoverIngressAuthority. Phase advancement still lacks authenticated Pallet-held, Corestore-adoption, Coreflow-retirement, legacy-ingress-absence, port-transfer, and publication-readback owner evidence. These paths remain fail-closed until those producers exist; caller-supplied proof references are not accepted as substitutes.

Run the real NoSQLDB qualification with:

pnpm exec tstest test/test.nodeenrollment.node.ts --verbose --logfile --timeout 120

Development IdP Qualification

test/test.devidp-qualification.node.ts qualifies the platform OIDC login against the real development identity provider from @idp.global/devidp instead of a fake issuer. One run proves the whole organization path end to end: the authorization request carries the organization login scopes, the seeded owner persona completes an authorization-code login headlessly, Cloudly issues its own federated identity through the login handoff, the owner picker resolves the seeded organization from live UserInfo claims, organization owner authority is proven by live introspection and denied for a foreign organization, and a persona without an organization membership is refused an identity.

The qualification is opt-in because it starts containers: it needs a reachable Docker engine, a usable kernel keyring and several minutes on a cold image. Without the variable every step is skipped with its reason, so the ordinary suite stays independent of Docker.

CLOUDLY_DEVIDP_QUALIFICATION=true pnpm exec tstest test/test.devidp-qualification.node.ts --verbose --timeout 600

Variable	Purpose
CLOUDLY_DEVIDP_QUALIFICATION	Exactly true runs the qualification. Unset skips it. Any other set value (TRUE, 1, yes) fails the run instead of skipping silently, so a mistyped opt-in cannot look like a pass.
CLOUDLY_DEVIDP_APP_IMAGE	Qualifies a different idp.global app image instead of the one @idp.global/devidp pins. A reference containing @sha256: is a pinned published image and is pulled; a reference without a digest is a locally built candidate and is never pulled.

The default path is the proven one: @idp.global/devidp 0.3.0 pins the released idp.global/app 15.0.0 image index, whose development relying-party client allows the organizations and roles scopes and seeds one organization, its owner persona membership and the app connection that maps the owner role to application roles. With the variable unset every assertion above runs against that image. CLOUDLY_DEVIDP_APP_IMAGE exists to qualify a candidate image — a locally built app or a newer published index — before it becomes the pinned default. An image without that fixture does not allow the organizations and roles scopes, so the run fails at the authorization request with an invalid_scope diagnostic naming the image instead of skipping or weakening its assertions.

The qualification owns the kernel keyring namespace cloudly-qualification and releases it through DevIdpFoundation at both ends of the run — once before it starts anything, so an interrupted earlier run cannot leave material behind, and once after every Cloudly manager, the database engine and the IdP containers have stopped — proving the namespace absent each time. One committed namespace holds ten keyring entries and about 4.7 KB of the 20,000-byte per-user quota, so a run leaves the keyring exactly as it found it.

The run trusts the development IdP's local certificate authority for the duration of the test process only and restores the previous trust store afterwards; it never installs trust into the operating system or a browser.

Configuration

Cloudly uses @push.rocks/smartconfig AppData with environment mappings and ephemeral storage. Runtime and bootstrap configuration stays in memory; startup does not create or update an AppData file. The runtime entry point loads .nogit/environment values through @push.rocks/qenv, and embedded callers can override public ICloudlyConfig values by constructing new Cloudly(config) programmatically. The secret keyring and WorkloadInit approval paths are deployment-local environment variables; neither is part of the public constructor DTO.

The names listed under required: in qenv.yml are resolved while the server entry point constructs its Qenv, before any manager runs. A missing one refuses the start with QenvMissingRequiredEnvVarsError, naming every variable that was not found in the process environment, the env file or the Docker secrets. That list is exactly the set whose absence already ends startup further in, so the refusal moves the same failure to the front and names it; a variable the server defaults or guards is deliberately not enforced. The secrets CLI (cloudly secrets …) and the backup-maintenance CLI dispatch before the server entry point and are not gated by it, and the API CLI client never enforces a list it did not write.

Refused at start when absent:

Variable	Purpose
SERVEZONE_ENVIRONMENT	ACME/runtime environment, currently production or integration.
SERVEZONE_PUBLIC_ORIGIN	Canonical externally reachable Cloudly origin, including protocol and optional public port.
SERVEZONE_SSLMODE	external (TLS is terminated in front of Cloudly, by Onebox or dcrouter) or none. Cloudly holds no certificate of its own: letsencrypt is refused at start.
SERVEZONE_SECRET_KEYRING_PATH	Path to the mounted Secrets v2 keyring. Startup fails before secret model and manager initialization when it is absent or invalid; earlier non-secret startup migrations may already have run.
MONGODB_URL	MongoDB connection URL used by SmartData.
S3_ENDPOINT	S3-compatible endpoint for registry, images, and artifacts.
S3_ACCESSKEY	S3 access key.
S3_SECRETKEY	S3 secret key.
S3_BUCKET	S3 bucket name. The registry manager refuses to start without it and the image manager writes its init object into it.

Expected from a complete deployment, but tolerated when absent:

Variable	Purpose
SERVEZONE_LISTENPORT	Internal port the Cloudly TypedServer binds to. Defaults to 3000, which the container image also sets.
SERVEZONE_ADMINACCOUNT	First-run admin bootstrap in username:password format. Without it no admin is bootstrapped.
SERVEZONE_WORKLOADINIT_APPROVAL_PATH	Path to the WorkloadInit approval file. Absent means no approval file is read.
MONGODB_NAME	MongoDB database name. Optional for SmartData, which uses the database the URL names when it is unset.
MONGODB_USER	MongoDB username. Substituted into the URL only when set.
MONGODB_PASS	MongoDB password. Substituted into the URL only when set.
S3_PORT	S3 endpoint port. Defaulted where it is read.
S3_USESSL	Boolean SSL flag for the S3 endpoint. Defaulted where it is read.

Common optional public settings are stored through the Cloudly settings manager:

Setting	Purpose
deploymentArchiveRetentionDays	Has no effect: Cloudly keeps no archived deployments, and the dashboard does not offer it. Still accepted and stored because ICloudlySettings declares it.
backupExternalTarget / backupNodeCacheKeepDays	Value-free backup target descriptor and node-cache retention.
corebuildWorkerUrl / corebuildWorkerUrls	One or more CoreBuild worker HTTP(S) URLs.
dcrouterGatewayUrl	External gateway URL. The matching credential is encrypted system-secret material, not a public setting; it enters only through Gateway Credential Bootstrap.
dcrouterGatewayClientId	Required stable Cloudly gateway client ID used for route, DNS, certificate, domain, mail, and Web Push calls; there is no cluster-ID fallback.
dcrouterTargetHost / dcrouterTargetPort	Optional target address that dcrouter should forward workload traffic to.
dcrouterTargetHostsByNode	Optional map for pinned service route targets by swarm node name. A service a Pallet runtime spec owns ignores both: its routes target the uplink address its spec's one pinned node reported.
dcrouterForwardSourceAddresses	The source addresses dcrouter's forwarded connections arrive from at a node, as canonical prefixes such as ["192.0.2.10/32"] (read on the dcrouter host with ip route get <node uplink>, field src). A node ingress serves non-public hostnames to these sources only; while it is absent none is served. Judged by validateClusterRouteSourceAllowList; not a gateway credential setting, so a write applies at once.
dcrouterMailSubmissionHost / dcrouterMailSubmissionPort / dcrouterMailSubmissionTlsMode	SMTP submission routing used when Cloudly provisions service-owned mail secrets.
dcrouterMailForwardTargetHost / dcrouterMailForwardTargetHostsByNode	Gateway-reachable SMTP forward target for inbound app mail, optionally overridden per pinned node.

Credentials for these integrations are stored as encrypted system-owned Secrets v2 values under stable keys such as HETZNER_TOKEN, BACKUP_EXTERNAL_TARGET_CREDENTIALS, DCROUTER_GATEWAY_API_TOKEN, BASEOS_JOIN_TOKEN, COREBUILD_WORKER_TOKEN, and COREBUILD_WORKERS. They are not serialized into the public settings document or returned by settings APIs. DCROUTER_GATEWAY_API_TOKEN is Cloudly's own least-privilege gateway-client credential, which Cloudly provisions itself; nobody supplies it directly.

Service Web Push

Set service.data.webPush.enabled to true, or declare pushnotification in an immutable deployment's requiredCapabilities, to request browser Web Push for a service. Cloudly first verifies that its enrolled dcrouter gateway credential has the exact readWebPush and manageWebPush capabilities and that dcrouter advertises binding, delivery, cancellation, and VAPID-rotation support.

After dcrouter creates a service-owned binding, Cloudly authenticates the one-time application credential before publishing three service-owned Secrets v2 entries:

Workload variable	Purpose
WEB_PUSH_TYPED_URL	dcrouter base URL used by the typed Web Push client.
WEB_PUSH_API_CREDENTIAL_ID	Service-scoped application credential ID.
WEB_PUSH_API_CREDENTIAL_SECRET	Service-scoped application credential secret.

The dcrouter gateway control credential never enters a workload. Cloudly stores only public VAPID metadata in the platform binding; the VAPID private key remains encrypted inside dcrouter. pushnotification does not create a Corestore namespace or resource.

Provider reconciliation is service-scoped and CAS-fenced. A crash before the first workload secret publication may replace that unpublished provider generation. Once a generation is published, missing secrets or credential drift fail closed without rotating or replacing provider state. Disabling Web Push or deleting the service removes the provider binding and managed application credentials before local ownership state is removed.

Optional runtime environment variables:

Variable	Purpose
SERVEZONE_INSTALL_DEMO_DATA	Runs the destructive demo data installer when set to true.
SERVEZONE_WORKLOADINIT_APPROVAL_PATH	Private WorkloadInit release-approval artifact used for Coreflow secret-runtime registration. Cloudly can start without it; after target and recipient authority are ready, registration returns workloadinit-unconfigured until an exact active approval verifies.
CLOUDLY_BACKUP_CRON	Enables the scheduled backup-all-services task with the supplied cron expression.
CLOUDLY_BACKUP_KEEP_LAST	Number of completed/failed backups to retain per service; defaults to 24.
CLOUDLY_BASEOS_IMAGE_CLEANUP_INTERVAL_MS	BaseOS image artifact cleanup interval; defaults to 12 hours.

The cache tier always uses Cloudly's canonical runtime s3Descriptor. Configure an optional long-term tier with the value-free backupExternalTarget setting. S3 and SMB credentials must be stored only in the encrypted BACKUP_EXTERNAL_TARGET_CREDENTIALS system secret; missing, malformed, or revoked credentials invalidate the cached external writer and fail closed. NFS targets use only their value-free path descriptor and do not read a credential secret.

For an offline S3 backup-target credential cutover, stop and fence every Cloudly and backup writer first. The bounded Cloudly maintenance command starts the real config, database, migration, secret and settings foundation without starting HTTP, gateway enrollment or scheduled managers. Inspect its value-free revision and descriptor with node cli.js maintenance backup-external-target-status. Then provide a private JSON file containing only accessKey and accessSecret:

node cli.js maintenance backup-external-target \
  --operation-id backup-cutover-1 --created-by operator:cutover \
  --expected-secret-revision 4 \
  --expected-source-endpoint onebox-smartstorage \
  --target-endpoint onebox-corestore \
  --credentials-file /run/secrets/backup-target-next.json

The file must meet the same owner, mode and no-symlink checks as other private Cloudly inputs. Credential bytes never appear in arguments, environment values or JSON output. The exact operation ID, source endpoint and system-secret revision fence the request. Cloudly persists a prepared phase, uses the idempotent SystemSecret mutation, then updates the value-free S3 endpoint and records complete. After interruption, rerun the same request and protected credential file while writers remain fenced; a mismatched replay or descriptor drift fails closed. The output contains only the operation ID, target revision, endpoint, bucket and completion status.

CoreMail Control Plane

Cloudly is the control client for the cluster's CoreMail workload. CoreMail fronts dcrouter as its single gateway peer; hosted apps submit mail to CoreMail instead of dcrouter, and CoreMail reports per-service mail statistics back.

CoreMail is itself a Cloudly-managed service, so Cloudly composes its runtime environment. Everything below is off unless coreMailEnabled is true; with it false the cluster stays on today's dcrouter-direct mail path, untouched.

Settings

Key	Shape
coreMailEnabled	boolean cluster switch
coreMailServiceId	the Cloudly service running CoreMail
coreMailControlEndpointUrl	https://host/socket — the control socket
coreMailTransferOrigin	https://host — bare origin, no trailing slash
coreMailGatewayEndpointUrl	wss://host/ — bare origin with trailing slash
coreMailControlCredentialId / coreMailControlCredentialVersion	control credential identity
coreMailSmtpHost / coreMailSmtpPort / coreMailSmtpTlsMode	submission listener the SMTP_* variables point at
coreMailDefaultMessagesPerMinute / ...PerDay / coreMailDefaultMaxPendingInbound	per-binding quotas, clamped to the published contract ceilings

The two URL shapes differ in exactly one character and are not interchangeable: the interfaces normalizer canonicalises endpointUrl with url.toString() (trailing slash) and coreMailTransferOrigin with url.origin (no slash). Cloudly rejects the wrong shape at edit time rather than letting CoreMail refuse the desired state later.

coreMailConfigEpoch is server-only and never writable from the wire. It is the monotonic desired-state epoch; CoreMail refuses any preparation that does not strictly exceed what it has applied, so a caller-supplied value could strand reconciliation permanently.

Secrets

Four system secrets back the control plane:

• COREMAIL_CONTROL_CREDENTIAL_SECRET — Cloudly's own control-session credential, minted by Cloudly on the first reconcile. CoreMail receives only the matching verifier, in COREMAIL_CONTROL_BOOTSTRAP.
• COREMAIL_GATEWAY_CREDENTIAL_SECRET — the credential CoreMail uses to authenticate to dcrouter. Cloudly delivers the plaintext to CoreMail's environment and hands dcrouter only the verifier.
• COREMAIL_SMTP_TLS_CERTIFICATE_PEM and COREMAIL_SMTP_TLS_PRIVATE_KEY_PEM — operator-supplied in v1. Automatic issuance for the submission listener is a later workstream; until it ships the operator provisions and rotates this material. Absent PEMs simply mean the listener is not provisioned, which is not a reconcile failure.

Per-service material is delivered as managed secrets with managementSource: 'coremail' and scope coremail:<serviceId> — deliberately distinct from mail:<serviceId> so the two controllers cannot overwrite each other while both environment sets coexist.

The delivered names are exactly the ones @serve.zone/platformclient reads, so a Cloudly-provisioned service and an Onebox-provisioned one are identical to the workload. The unscoped group covers the default outbound address: MAIL_COREMAIL_URL, MAIL_COREMAIL_BINDING_ID, MAIL_COREMAIL_CREDENTIAL_ID, MAIL_COREMAIL_CREDENTIAL_VERSION, MAIL_COREMAIL_CREDENTIAL_SECRET and MAIL_FROM, plus SMTP_* when a submission listener is configured — with the SMTP username the binding id and the password the same credential secret.

Every enabled address additionally receives a complete scoped group: MAIL_<TOKEN>_FROM, all five MAIL_<TOKEN>_COREMAIL_* names and the scoped MAIL_<TOKEN>_SMTP_* set, mirroring how the dcrouter path delivers per-address variables today. platformclient takes a scoped group wholesale when one is present and otherwise uses the unscoped group only if MAIL_FROM equals the requested address, so a service with more than one address can only send from all of them if each carries a full group. The service still has exactly one binding; the scoped groups repeat its values under each address token.

Both authorities Cloudly owns keep their public half in a Cloudly-owned CoreMailControlCredential record. Verifier hashing is salted, so recomputing it would emit a different string on every pass and rewrite the bootstrap environment and dcrouter's stored peer for no change; storing the emitted verifier makes an unchanged reconcile a genuine no-op.

Credentials rotate with a 15-minute overlap: the previous credential is carried as retiring with an acceptUntil deadline so an already-connected workload is not disconnected the moment a new epoch lands.

Migration

Each service moves in three reversible phases, at most one step per reconcile:

1. dual-write — the CoreMail binding and environment are delivered while inbound still lands on the dcrouter-direct target and MAIL_TYPED_URL still resolves. Nothing in the application has changed.
2. inbound cutover — inbound is routed at CoreMail. Both environment sets remain, so a rollback is a single binding flip.
3. CoreMail only — the dcrouter-direct environment and credential are retired.

The phase is never held in memory: it is derived on every read from durable evidence — the binding in service.data.mail.coreMail, whether the CoreMail environment is delivered, the address targetType persisted at cutover, and whether the mail:<serviceId> scoped environment still exists. A Cloudly restart therefore cannot walk a migrated service backwards.

Step 2 waits until CoreMail's own reconciliation status reports the binding active; Cloudly having sent a desired state is not evidence that CoreMail applied it. Step 3 additionally waits until CoreMail has reported real delivered or acknowledged traffic for that service. A binding that stops being active walks a migrated service back one phase so the fallback is restored before inbound moves. Setting coreMailEnabled to false returns every service to the dcrouter-direct path immediately.

Operator requests

• getCoreMailControlStatus — applied epoch, digest, CoreMail's reconciliation status and the last error. enabled: false with every other field absent means no CoreMail workload is configured.
• getCoreMailServiceMailStatistics — per-service, per-UTC-day counters.
• listServiceMailTargets — every service with mail configured, CoreMail-fronted or not, so what remains on the dcrouter-direct path is visible.

Starting Cloudly

Install and build with pnpm:

pnpm install
pnpm build
pnpm start

Cloudly evolves persisted data through a @push.rocks/smartmigration chain that lives entirely in ts_migration/ and runs before the app serves. The chain carries its own data version, deliberately separate from the package version, and a SmartData-backed ledger records how far a database has advanced, so each boot applies only the outstanding steps under a lock. That ledger is named cloudly and is stored as the SmartdataEasyStore document smartmigration:cloudly. Only one instance migrates at a time, so an instance that dies while holding the lock makes every later boot fail with LOCK_TIMEOUT until the recorded lock expires. Confirm the old instance has stopped and wait for the ledger's expiresAt before retrying; do not manually clear or replace the lock. These migrations are forward-only: an applied step is never replayed, and downgrading the data version is refused.

This release carries steps from data version 0.10.0 onwards; every step below that was released, applied everywhere and then deleted. The floor is declared as oldestSupportedVersion, so a ledger below 0.10.0 fails startup by name with LEDGER_BELOW_OLDEST_SUPPORTED_VERSION on the first ledger read — before the lock is taken and before anything is written — instead of silently running the truncated remainder. A database in that state must first run the last release that still contained those steps. A database with no ledger at all is judged by its data: an empty one is stamped at the chain target, and a populated one is taken to sit at the floor and migrates forward from there. A database last migrated by a Cloudly before 25.2.0 never wrote a ledger — that release replaced the one-shot completion flags with this one — so it is taken to sit at the floor on the evidence of its collections alone, and the deleted steps are not replayed on it. Upgrade such a database through a release that writes the ledger before this one.

released-runtime-state-precondition (0.10.0 to 0.11.0) is the first step of the chain, because it reads and refuses and changes nothing: when it refuses, no step of this release has written either, and the ledger is left at the version the released Cloudly still starts on. It counts the state that only a node or a pod could have produced — the node-signed enrollment acknowledgements in cloudly_node_enrollment_receipts_v1, the node observations in cloudly_secret_runtime_node_observations_v2, the protection receipts in cloudly_runtime_network_protection_receipts_v1 and cloudly_runtime_network_protection_receipt_revisions_v1, the assignments in cloudly_runtime_assignments_v1 that carry a node observation or a terminal receipt, and the registrations in cloudly_runtime_sessions_v1 that named no node process. Those are the names the released Cloudly wrote them under, which is what a database still has at this point in the chain: the step that takes the version token off every collection name runs later, and this precondition counts whichever of the two names the database actually carries. None of it can be re-signed or re-derived from what Cloudly stores, so a single such row fails startup by the name released-runtime-state-present, naming every collection that counted, rather than letting a later step rewrite the database around it. The refusal carries the per-collection counts and the reason each class is bound on the error's counts; the message itself stays short enough that the migration ledger and the boot failure carry it in full.

cluster-runtime-phase (0.11.0 to 0.12.0) then states the runtime every existing cluster already runs on.

Secrets v2 is complete. The separately authorized offline cutover that produced it -- its CLI subcommands, its engine, and the startup gate that re-verified its completion receipt on every boot -- is removed in this release, and the data migration step retire-secret-cutover (0.12.0 to 0.13.0) retires what it left behind.

The step verifies that authority one last time: the cutover state must be completed, the production execution binding it names must be the one stored beside it, the authenticated scratch rehearsal receipt that binding embeds must carry the baseline the cutover ran against, and the completion receipt's MAC must recompute under the KEK it names from the mounted keyring. An authority that does not verify fails startup by the name secret-cutover-retirement-blocked with the reference that failed, and drops nothing.

Once it verifies, the step writes one provenance row into cloudly_secret_cutover_archive -- the run, environment and Cloudly version that completed the cutover, when it completed, the KEK and the receipt MAC that authorized it, the digests of the execution binding, the rehearsal receipt and the completion payload, the baseline digests that receipt covers, and the name, kind, row count and content digest of every namespace it is about to drop -- and then drops cloudly_secret_v2_cutover_state (the completion receipt and the execution binding) and the settlement-only cloudly_legacy_deployment_settlement_attempt_v1 and cloudly_legacy_deployment_settlement_execution_evidence_v1 collections: their provenance is kept, their evidence is not. It drops the empty SecretGroup and SecretBundle tombstone views over cloudly_legacy_secret_groups_v1_frozen and cloudly_legacy_secret_bundles_v1_frozen if a database still carries them; a database at the migration floor does not, because 25.2.0's own cleanup already dropped both views and both drained collections. A namespace that is already gone is already retired: it is neither digested nor named in the archive. A SecretGroup or SecretBundle that is present and is not that empty view over that frozen collection is somebody else's data, and is refused rather than dropped -- on a resumed run as well as on the first one. The archive row is written before the first drop, so a run that finds it finishes the drops instead of verifying an authority it has already consumed and archived.

A boot that meets a refusal reports secret-cutover-retirement-blocked: <reference>., and the reference names what failed:

• cutover-state-missing -- cloudly_secret_v2_cutover_state holds no secret-v2-cutover-v1 document.
• cutover-not-completed -- that document exists, but its phase is not completed.
• cutover-state-invalid -- a field of the cutover state is missing or does not have the shape the cutover wrote.
• completion-invalid -- the completion block or the receipt inside it is missing or malformed.
• execution-binding-missing, execution-binding-purpose, execution-binding-run -- the standalone execution binding is absent, is not the production binding, or names a different run than the state does.
• execution-binding-mismatch -- the stored binding is not the binding the completion receipt covers, so archiving it would attest an authority that never authorized this cutover.
• rehearsal-receipt-invalid, rehearsal-baseline -- the rehearsal receipt the binding embeds is malformed, or it rehearsed a different baseline than the one the cutover ran against.
• completion-receipt-key -- the mounted keyring has no KEK under the id the receipt names, which is what a premature KEK rotation looks like.
• completion-receipt -- the receipt's MAC does not recompute over the state it claims to cover.
• canonical-number -- a document that has to be authenticated or digested holds a number that has no canonical form (NaN or an infinity).
• SecretGroup-not-a-cutover-tombstone, SecretBundle-not-a-cutover-tombstone -- that namespace is present and is not the cutover's empty view over its frozen collection.
• <namespace>-unexpected-namespace-type -- the catalog reports one of the retirable names as neither a collection nor a view.
• archive-write -- the archive row was not inserted, so nothing is dropped.
• archive-record, archive-provenance, archive-dropped -- the archive row a run reads or writes is not a complete archive record. On a resumed run that row is the only proof the authority gate has already passed, so anything else under its id refuses instead of standing in for it.

unversioned-collection-names (0.13.0 to 0.14.0) then takes the version token off every collection a released Cloudly created. The 49 pairs are frozen in the step itself and are exactly the explicit collection names the 27.x releases declared; each new name is the old one without its trailing _v1 or _v2, and collections added after 27.5.0 never carried a token. The step renames the collection rather than copying it, so the documents, their _ids and their indexes are the ones that were already there, and the models this release reads them with prepare against the renamed collection on the same boot.

The whole plan is decided against one catalog read before anything moves, so a refusal leaves the database exactly as it found it. A boot that meets one reports collection-rename-blocked: <reference>.:

• <name>-unexpected-namespace-type -- either name of a pair holds something that is not an ordinary collection, so it is somebody else's namespace.
• <old name>-both-names-present -- both names hold a collection at once, which a rename cannot produce: the source stops existing the moment it moves. The step does not choose which of the two is the database.
• <old name>-document-count-after-rename -- a collection that moved does not hold the rows it held before the move.

Each pair moves on its own, and a pair that is already at its new name is not moved again, so a boot interrupted between two renames -- or between the last rename and the ledger stamp -- finishes on the next one. Nothing is left at the old names, not even an empty view: a released Cloudly started against a database this step has migrated already fails its own migration planning with TARGET_NOT_REACHABLE, before it opens a handler or a listener.

secret-material-encryption-context (0.14.0 to 0.15.0) then re-encrypts every stored secret material under the encryption context this release authenticates with. The authenticated data loses its shape member and its profile token: it is {profile: 'aes-256-gcm+wrapped-dek', purpose, secretId, secretVersionId, organizationId, owner, key, environment}, and the stored envelope states the same profile and carries no schemaVersion. Every context value, the algorithm, the wrapped-DEK construction and the fingerprint rule are the ones the released Cloudly used, so the step rebinds material and never rotates a key: each document is re-encrypted under the KEK it already names, with a fresh DEK and fresh nonces.

There is no rollback to 27.x after this step. Every ciphertext in the database is then bound to the new context, and a released binary computes the old one, so it authenticates nothing it reads. A control plane that has to go back restores the checkpoint taken before the upgrade and starts the released binary against that; the converted database is not a database 27.x can serve.

The step covers cloudly_secret_versions.encryptedMaterial, cloudly_system_secret_candidates.encryptedMaterial -- together with the stagedKekId and stagedMaterialFingerprint that restate it -- cloudly_federated_session.encryptedToken and cloudly_runtime_network_signers.encryptedPrivateKey. A version whose material was purged and a revoked signer hold none, and the step invents none. It runs after the rename above, so it walks those collections under the names this release reads them by.

One context value is derived rather than stored: a federated session's secretVersionId is a digest over its binding, and this release derives it under an unsuffixed domain. A released envelope is therefore bound to the versioned digest, so that site states a released context of its own -- the released spelling exists only inside the step -- and the envelope is opened under the released digest and re-sealed under the one the runtime derives.

The signing key's context is derived too, from the authority the signer row carries, and that authority still states its released shape while this step runs: it is read through the same released reader the rest of the chain uses, which takes the shape member off and leaves the two members the context takes -- the signer's identifier and its authority digest -- exactly as the row states them. Both of those move later in the chain, and contract-documents re-seals the key under them; until then the envelope stays bound to what the released Cloudly wrote.

One transaction covers one chunk of 64 documents and the step-owned progress collection cloudly_secret_material_encryption_cursor that records how far each collection has come, so a boot interrupted between two chunks resumes at the chunk it committed and re-encrypts nothing twice. The progress collection is dropped when the last collection is through; a database that holds no secret material never creates it and never loads the keyring. Which context a document was written under is read from the profile the document states -- never from a trial decryption -- and every re-encrypted envelope is decrypted again, through the reader the runtime uses, before the transaction that replaces the material commits.

A boot that meets something it cannot convert reports secret-material-encryption-blocked: <collection>:<document>-<reference>. and commits nothing of the chunk that refused, so the material it names is still readable under the context it was written with:

(A document whose _id is not an identifier is refused as <collection>-document-identity, and a cursor namespace that is not a plain collection as <cursor collection>-unexpected-namespace-type.)

• -material-shape -- the stored envelope is not one a released Cloudly wrote.
• -unexpected-profile -- the envelope states neither the released context nor this one.
• -kek-unavailable -- the mounted keyring carries no KEK under the id the envelope names.
• -material-not-readable -- the envelope does not open under the released context.
• -context-unavailable, -secret-metadata-missing -- the context the material is bound to cannot be rebuilt from what the database holds.
• -verification -- the re-encrypted envelope did not decrypt back to the same bytes.
• -not-rewritten, -<field>-mismatch, -cursor-not-advancing -- a document changed under the step, a field that restates the envelope disagreed with it, or a committed chunk did not move the collection forward.

Keep every KEK that protects stored material in the mounted keyring while this step runs: it converts what each document names, so a rotation that drops a KEK makes the step refuse by name.

unversioned-derived-identifiers (0.15.0 to 0.16.0) then takes the version token off every identifier a released Cloudly derived and stored, so the identifier the code composes and the one the database holds are the same string again. It rewrites the two singleton row ids (cloudly-runtime-controller, cloudly-runtime-network-protected) and every hashed id and content digest of these families, together with every member of every other row that names one:

Family	Row it identifies	Members that name it
node credential authority	cloudly_node_credential_authority	cloudly_node_credential_mutations.authorityId
node credential mutation	cloudly_node_credential_mutations	--
service-machine credential	cloudly_service_machine_authority	its own credentialId and userId, User.id, cloudly_service_machine_mutations.result.credentialId and .result.userId, cloudly_secret_metadata.managementScope
private membership, private alias	cloudly_private_membership, cloudly_private_alias	--
private mutation receipt	cloudly_private_mutation (id and requestDigest)	--
protected revision, network pool	cloudly_runtime_network_protected_revisions, …_pools	--
network node scope	cloudly_runtime_network_protection_receipts, …_projections, …_trust_deliveries, …_dns_leases	--
protection-receipt revision	cloudly_runtime_network_protection_receipt_revisions	--
network attempt	cloudly_runtime_network_workloads.attemptKey	--
allocation eligibility	cloudly_runtime_network_allocation_eligibility	cloudly_runtime_network_handoffs.eligibilityId, cloudly_runtime_network_workloads.eligibilityId
managed VPN, service policy	cloudly_runtime_managed_vpn, cloudly_runtime_network_service_policies	--
runtime slot	cloudly_runtime_slots	cloudly_runtime_assignments.slotId
runtime config	cloudly_runtime_configs	--
network signer	cloudly_runtime_network_signers	its own authority.id and authority.previous.id, cloudly_runtime_network_signing_revisions.authority.id and .authority.previous.id, cloudly_runtime_network_projections.signed.authority.id, cloudly_runtime_network_trust_deliveries.acknowledged.id and .pending.id
network signing revision	cloudly_runtime_network_signing_revisions	--

Each family is one transaction: its rows and every reference to them move together, so no row and no reference ever disagree, and an interrupted run leaves that family exactly as it found it. Each row's new identifier is derived by the composer the runtime itself looks the row up by, from the inputs the row already stores, and the released one is recomputed from the retired domain, which exists only inside the step. A row that already carries this release's identifier is passed over, so a second run writes nothing and a resumed run finishes what it did not reach; the rows are the progress record, and the step keeps none of its own. The step runs after the re-encryption above because it rewrites no member any encryption context is bound to.

A boot that meets a row it cannot move reports derived-identifier-blocked: <collection>.<member>:<identifier>-<reference>. and commits nothing of the family that refused:

• -identifier-shape -- the member that has to carry the identifier holds something else.
• -unexpected-identifier -- the stored identifier is neither the released derivation nor this one, so the step cannot say what the row is.
• -inputs-not-derivable -- the row's own stored inputs do not compose a valid identifier.
• -not-rewritten, -references-not-rewritten -- a row or a set of references changed under the step, so what it counted is not what it moved.

The step reads every released body through ts_migration/released-contract-shape.ts, because every reader of this release refuses the shape member a released body still carries. That module states the released shape rule once -- a released body is this release's own snapshot with the shape member in front, which is what every released snapshot built -- so a family that hashes a whole body (the allocation eligibility over its session bindings, the private mutation receipt over its request) reproduces the released identifier from the released bytes rather than from a stripped approximation of them. The step derives this release's identifier first and asks for the released one only for a row that has not moved, so a row already in this release's shape never needs a released reader at all.

The signing families are derived from the controller row rather than from anything the signer row holds: a controller statement is written once and never replaced, so a database holds exactly one signer row. A signer row with no controller row to name refuses as cloudly_runtime_controller-missing. Moving the signer identifier also moves the AAD the signing key's envelope is sealed under, so the step that recomputes the signing-authority digest re-seals that key under both the moved identifier and the recomputed digest.

Two groups of released bytes deliberately keep their version token here. The mutation receipts of the human and service-machine credential families hash a mutationId that was never stored, and their requestDigest covers material only the caller held -- the human one under a runtime HMAC key -- so neither can be recomputed, and the service deletion receipt hashes a binding that is likewise not stored. No reader re-derives any of them: a pre-32 mutation id is not found and is judged a new mutation, which its own generation fence then refuses as a conflict rather than replaying it. The network projection body's own id and the handoff lease's conntrack label name values inside a body a stored digest covers, so they move in the same write as that digest rather than here.

contract-documents (0.16.0 to 0.17.0) then rewrites every stored document into the contract shape this release reads. Every released contract body stated a schemaVersion that no reader of this release admits, and every digest taken over such a body changed twice over: the member left the payload, and the digest domain lost its version token. The step takes the member off and recomputes the digest with the composer the runtime itself verifies with:

Family	Collection	Digests recomputed
service-machine authority, service-machine mutation	cloudly_service_machine_authority, …_mutations	--
private network, private membership	cloudly_private_network, cloudly_private_membership	--
private mutation	cloudly_private_mutation	requestDigest over the request in the same write
runtime config	cloudly_runtime_configs	the container invocation, then the config that covers it
runtime assignment	cloudly_runtime_assignments	every revision, bound to the revision before it; the workload's config reference and the predecessor attempt's last revision
runtime session	cloudly_runtime_sessions	--
resolved secret manifest	cloudly_resolved_secret_manifests	manifest.digest
secret-version purge operation	cloudly_secret_version_purge_operations	operation.intentDigest
secret runtime recipients, cluster targets	cloudly_secret_runtime_recipients, …_cluster_targets	the ready target set's targetSetDigest
corestore publication grant	cloudly_corestore_publication_grants	grant.bindingRequestDigest over the binding request the same row stores
corestore publication receipt	cloudly_corestore_publication_receipts	both copies of the grant follow the grant row the receipt names
network protected revision, network protected	cloudly_runtime_network_protected_revisions, cloudly_runtime_network_protected	the protected authority per generation, each bound to the generation before it
network pool	cloudly_runtime_network_pools	the reference to the protected authority follows the revision it names
network handoff	cloudly_runtime_network_handoffs	the lease, over the authority reference it follows and the conntrack label this release derives
network workload	cloudly_runtime_network_workloads	the lease, and the reservation's own reference to the handoff lease
network signing revision, network signer	cloudly_runtime_network_signing_revisions, cloudly_runtime_network_signers	the signing authority per generation; the signer's private key is re-sealed in the same write
cluster, cluster node	Cluster, ClusterNode	--
platform binding	PlatformBinding	the retention evidence's intentSha256
service, deployment operation, adoption operation, adoption rollout	Service, ServiceDeploymentOperation, ServiceDeploymentAdoptionOperation, ServiceDeploymentAdoptionRolloutOperation	the container invocation each one carries

Each family is one transaction, and a document is compared with what its own recomposition produces: the rewrite and the no-op are the same rule, so a second run writes nothing and a run interrupted between two families resumes at the next one. What the comparison reads is what a value holds, so a member the step does not compose -- a TTL date, for one -- is carried through as it was stored rather than rewritten or flattened.

Before the first write the step counts the classes of document it is known not to be able to move and refuses on one: the six the precondition at the top of the chain counts, and an assignment whose record-level predecessor names a row the database no longer holds, whose digest is that row's last revision. Those counts are all that is proven up front. A document that refuses for its own shape is found by the family that reads it, after earlier families have committed -- which is safe rather than partial, because the ledger stays at 0.16.0, every family is a fixed point of its own recomposition, and the next boot counts again and resumes at the family that refused. A boot that meets one reports contract-documents-blocked: <collection>.<member>:<reason>, naming the body that refused, with -released-runtime-state-<count>, -node-digested-eligibility-<count>, -predecessor-row-missing, -body-shape, -unexpected-shape-statement, -not-readable, -intent-missing, -workload-config-missing, -workload-authority-not-recomputable, -grant-operation-mismatch, -grant-row-not-recomposed, -reference-shape, -previous-shape, -protected-revision-missing, -signing-revision-missing, -handoff-row-missing, -conntrack-label-unexpected, -signer-scope-mismatch, -context-unavailable, -kek-unavailable, -released-context-unreadable, -reseal-not-readable or -not-rewritten. An allocation eligibility proof is one of the counted classes: its identity is a hash over the protected-authority reference this step re-digests and over the node-digested protection receipts it names, and the store writes one only when every egress authority has such a receipt, which the precondition at the top of the chain already refuses.

Five stores start empty and are emptied only after every family has committed, each with its count logged: cloudly_federated_session, cloudly_secret_runtime_enrollment_challenges, cloudly_secret_runtime_report_replay, cloudly_runtime_network_trust_deliveries and cloudly_runtime_network_projections. A projection is compiled state: the node producer composes the next one from the protected authority and tolerates no previous journal, so its previous, retiredLeases and retiredHandoffs history is dropped rather than rewritten.

Both copies a corestore publication receipt keeps of its grant follow the grant row the receipt names, which the grant family has already recomputed and committed: the replay path compares a stored receipt's copy with the grant a retrying caller presents, so a copy left on the released digest would make every stored receipt disagree with the grant it names. A receipt whose grant has already been reaped by its TTL keeps what was written, because the value it would follow is gone and a replay needs that row anyway. The ingress admission binding always keeps what was written: it covers the request bytes a caller sealed an envelope over, and Cloudly stores neither the envelope nor that request, so nothing here can reproduce it. A caller that retries a publication accepted before this upgrade under the mutation id it used then is answered REPLAY_CONFLICT instead of the stored receipt -- the secret versions that publication created were written before the upgrade and are not lost by it.

The network authority, lease and signer subtree moves in the same step, because one of its digests is what an encrypted private key is bound to. The protected network authority is recomputed from its first generation up -- a generation's digest covers the reference it carries to the generation before it -- and every reference to it follows the revision row it names: the singleton head, the pools, and both kinds of lease. A handoff lease also carries a conntrack label derived from its own identity under a domain this release unsuffixed, so that label moves in the same write as the digest that covers it, and a workload reservation's reference to its handoff follows the lease the handoff family recomposed and committed.

The runtime network signing key is re-sealed in the same write as the authority it belongs to. networkSigningContext names the signer's identifier as the envelope's secretId and that authority's digest as its secretVersionId; the identifier moved in unversioned-derived-identifiers and the digest moves here, so the step opens the envelope under the released pair, seals it under the moved pair with a fresh DEK and fresh nonces, opens it again and compares it with the plaintext it started from before the row is written. An envelope that already opens under the moved pair is kept exactly as it is, which is what makes a second run write nothing; one that opens under neither refuses by name with nothing written for that signer. The step takes the operator keyring for this, and loads it only when there is a sealed key to move. The projections emptied above are recompiled at generation 1 from the moved authority with the re-sealed key, so that rewrite leaves nothing to re-sign. Both halves of this rewrite -- the contract bodies and the network subtree a sealed key is bound to -- are one step of one release: no build ever runs against a database whose signing key is sealed to an identifier or a digest the code no longer computes.

runtime-session-cluster-binding (0.17.0 to 0.18.0) adds the cluster owner to every retained Pallet session binding. The step derives that owner only from the current ClusterNode when a persistent Pallet credential authority for the same node predates the session and still covers its credential generation and session epoch. A missing node or authority, a cluster mismatch, or an allocation-eligibility copy that differs from its durable session refuses with runtime-session-cluster-binding-blocked before any write. The step then recomputes each allocation eligibility identifier and moves the handoff and workload references that name it in the same transaction as the session rows. Already enriched rows are compared by the same rules and retained, so replay writes nothing. The step never reconstructs a deleted node's cluster from a replacement node and never deletes historical packet or lease authority to make an upgrade pass.

cluster-relay-registration-credential (0.19.0 to 0.20.0) removes every relay registration that was stored before a registration named the authority it was accepted under. Those rows can name no credential and none was ever recorded for them to name, and minting one inside a migration would be Cloudly inventing evidence about a socket it never saw. A stored registration is not authority — it is what a relay said on a socket that may be long gone — so the row is dropped and any live relay registers again within its own reconnect cadence, which re-creates it with the authority that relay really holds; until then the row's listen address, version, sequence and vpnHub selection are absent. Rows that already name one are checked by the very assertion that admits them into the database and retained, a row of any other shape refuses the step by name before anything is removed, and replaying the completed step removes nothing.

service-runtime-targets (0.18.0 to 0.19.0) converts the released singleton runtime spec and status rows into private service/cluster records in the same collections. It derives the composite id from the released spec's service and cluster, joins a status to exactly one spec, and changes only the private id plus serviceId and clusterId; the public projection and revision remain byte equivalent. Mixed singleton/target shapes, duplicate targets and orphaned or ambiguous statuses refuse before the transaction writes anything. Replaying the completed step retains every row.

secrets keyring-generate, secrets keyring-check, secrets keyring-convert, secrets export-cluster-relay-authorization and secrets set-corestore-control-token are the only secrets subcommands; every other one is answered with USAGE_INVALID. The subcommand was named secrets-v2 before this release. Onebox's cutover coordinator is the one live consumer that still spawns node cli.js secrets-v2 ... into a maintenance container built from the Cloudly image, including the keyring-generate and keyring-check pair; against a 32 image those calls fall through to a normal Cloudly runtime boot instead of the CLI, so Onebox must invoke secrets. The immutable Cloudly image still dispatches the subcommand before importing or constructing the normal Cloudly runtime: the keyring commands never load it, and export-cluster-relay-authorization and set-corestore-control-token load it only when they run, to boot the foundation they read and write through. Every invocation suppresses command-time stdout and stderr and then emits one bounded JSON object followed by a newline on stdout, with exit status 0 only when that object reports ok: true.

The keyring commands and the WorkloadInit runtime approval loader accept private JSON inputs only through a lexically normalized absolute path. This applies to SERVEZONE_SECRET_KEYRING_PATH and SERVEZONE_WORKLOADINIT_APPROVAL_PATH. The existing parent directory must be owned by the process user and not writable by group or other users. The input must be a regular, non-symlink file owned by the process user, have no group or other permissions, have exactly one hard link, and be no larger than 64 KiB. Normal Cloudly startup reads the configured keyring through the runtime loader; it does not apply these CLI-specific ownership, link-count, mode, and size checks. The WorkloadInit approval loader does apply the strict rules and the 64 KiB limit. Keyring output uses an atomic, no-clobber private writer with mode 0600: it holds the verified directory descriptor, writes a random same-directory mode 0600 temporary file, fsyncs, and publishes without clobbering. It reconciles only old temporary files from dead writers and never follows an output or temporary-file symlink. --kek-id and --recipient-key-id are optional safe identifiers.

node cli.js secrets keyring-generate \
  --output /run/cloudly-secrets/keyring.json
export SERVEZONE_SECRET_KEYRING_PATH=/run/cloudly-secrets/keyring.json
node cli.js secrets keyring-check

The keyring file this release reads carries activeKekId, keks and ingress and no shape member. A file a released Cloudly wrote states schemaVersion: 1, and this release does not load it: convert it once, offline, while Cloudly is stopped, and mount the converted file.

node cli.js secrets keyring-convert \
  --input /run/cloudly-secrets/keyring.json \
  --output /run/cloudly-secrets/keyring.converted.json

The conversion carries the key material across untouched -- it is a conversion, not a rotation -- publishes a fresh mode 0600 file without clobbering an existing one, leaves the mounted file exactly as it was, and reports the identity keyring-check reports and no key material. It refuses rather than repairing: a file it has already produced is answered with KEYRING_CONVERSION_REFUSED and the reference already-converted, and anything that is not a released keyring file with the reference not-a-released-keyring-file (a file whose content is not a JSON object with the reference not-a-keyring-file). Neither writes anything, so the mounted keyring is still the one the running release loads.

secrets export-cluster-relay-authorization reads the database. It hands a cluster relay's bearer to the operator who deploys that relay; the procedure is under Cluster Relay Credential Delivery. The usual path needs no access to Cloudly's process at all: servezone relay export-authorization asks the running Cloudly over its API. This command is the offline alternative for a Cloudly that is not serving, and runs where the keyring lives, inside the Cloudly container, with that container's environment:

node cli.js secrets export-cluster-relay-authorization \
  --cluster-id <clusterId> \
  --expected-generation <generation> \
  --output /root/relay-export/relay-authorization

It boots only the persistence and secret foundation, without the server, and migrates nothing: a database at any other data version than the one this build expects is refused with DATA_VERSION_MISMATCH before a model or the keyring is touched, and a foundation that does not start answers the secret manager's code (for example SECRET_KEYRING_INVALID) or CLOUDLY_START_FAILED. It then reads exactly one secret version: the version of the Cloudly-system secret SERVEZONE_CLUSTER_RELAY_AUTHORIZATION that the cluster's relay credential delivered at the named generation, and only while it is still that secret's active version. The decrypted bytes are proved against the credential's hash of the bearer that generation issued, re-read after the decryption, so a rotation that commits meanwhile is refused instead of exported. Every other refusal is RELAY_AUTHORIZATION_EXPORT_REFUSED with a reference:

Reference	Meaning
relay-credential-absent	No relay credential is minted for the cluster.
relay-credential-generation-mismatch	The credential is at another generation than --expected-generation, before the read or after it.
relay-authorization-missing	The version the credential delivered is no longer an active stored version.
relay-authorization-ambiguous	The secret's active version, or its decrypted bytes, are not what the credential issued; the command picks neither.

--output follows the private-file rules above: a lexically normalized absolute path in an existing directory owned by the invoking user and not writable by group or other users. An existing output, a symbolic link included, is refused with OUTPUT_EXISTS before the database is opened, and refused again rather than replaced when the file is published. The bearer is written as a new regular file owned by the invoking user, mode 0600, one hard link, holding the bearer's bytes and no trailing newline, through the same no-clobber private writer as the keyring output: a same-directory temporary file created with O_EXCL and hard-linked into place, so the output holds the whole bearer or does not exist. The database connection is closed and the keyring wiped before the file is written, so a failed teardown leaves no file. The command prints metadata only:

{"ok":true,"command":"export-cluster-relay-authorization","clusterId":"<clusterId>","generation":1,"byteLength":147,"sha256":"<hex>"}

sha256 is what sha256sum prints for the file. It is also the hash Cloudly keeps of the bearer, which tells nobody more than the file does: whoever reads the answer holds the file. The bearer itself never appears in the answer, in a log or in any typed request.

An administrator sets the Corestore control token (CORESTORE_API_TOKEN) through the typed request setCorestoreControlToken { identity, token, expected } (Interfaces 32.42 requests.corestorecontrol), and reads its state with getCorestoreControlTokenState { identity }. Cloudly stores the token as platform-managed material of both Corestore provider configs (cloudly-corestore-database and cloudly-corestore-objectstorage) in one transaction. The elected relay of a cluster reads it sealed to the secret recipient it enrolled, through getCorestoreControlCredentialMaterial, and calls its nodes' Corestore control API with it; no workload is ever delivered it.

expected is the state the administrator last read -- { state: 'absent' } or { state: 'present', revision } -- and never a minimum. The transaction that stores the token fences the administrator (a freshly signed-in administrator, re-read at write time exactly as a relay credential mutation is), reads the state, refuses an expectation that is not exactly that state, stores the token under both providers, advances the revision by one and writes a value-free receipt (id, revision, via, actorId, setAt) into cloudly_corestore_control_token_receipts; the revision lives in cloudly_corestore_control_token_state. Either all of it commits or nothing does. The answer is the new state and its receipt, never the token. Refusals are the contract's corestoreControlTokenSetRefusals: token-invalid, token-present, token-absent, token-revision-mismatch and provider-unavailable; every other failure, an unauthorized or stale administrator included, is Corestore control token request denied.. The state read answers the latest 4096 receipts, oldest first. servezone corestore control-token-state and servezone corestore set-control-token --stdin --expect absent|<revision> (@serve.zone/cli, see ts_cliclient/readme.md) wrap both requests and read the token from stdin only.

The in-container command stays for an operator who has no administrator session yet. secrets set-corestore-control-token --stdin stores the token through the same transaction, so it advances the same revision and writes a receipt via: 'container-cli' naming operator:secrets-cli. It reads the token from stdin and from nowhere else -- never from an argument, which would sit in the process list and the shell history, and never from the environment -- so pipe it into the running Cloudly container:

read -rs CORESTORE_API_TOKEN   # pasted, not echoed
printf '%s' "$CORESTORE_API_TOKEN" \
  | docker exec -i <cloudly-container> node /app/cli.js secrets set-corestore-control-token --stdin
unset CORESTORE_API_TOKEN

docker exec needs -i to hand stdin over; without it the command reads nothing. One line terminator after the token is not part of it. The input is judged against the Corestore credential contract before the database is opened, so a stored token is always one the relay can be handed. The command boots the same foundation as the export, with the same DATA_VERSION_MISMATCH, secret-manager and CLOUDLY_START_FAILED refusals. A stored token is replaced only with --replace, which takes the next revision whatever revision is stored, and either both providers hold the new token or neither does. Every other refusal is CORESTORE_CONTROL_TOKEN_REFUSED with a reference, and none of them carries the token:

Reference	Meaning
token-input-terminal	stdin is a terminal, which would echo a token typed into it.
token-input-empty	stdin carried nothing; docker exec without -i hands over nothing.
token-input-whitespace	stdin carried whitespace only.
token-input-not-printable	A byte is not printable ASCII, or whitespace sits inside the token.
token-input-oversized	stdin carried more than 4096 bytes.
token-invalid	The contract refuses the token's length or characters.
token-present	A token is stored and the command did not say --replace.
provider-unavailable	A Corestore provider config is missing or disabled; Cloudly writes both when its server starts, so a database no Cloudly server ever started on has none yet.

The answer names the key and the provider configs it was stored under, the revision it stored and its receipt, and never a byte of the token:

{"ok":true,"command":"set-corestore-control-token","key":"CORESTORE_API_TOKEN","providerConfigIds":["cloudly-corestore-database","cloudly-corestore-objectstorage"],"state":{"state":"present","revision":1},"receipt":{"id":"corestore-control-token-set-1","revision":1,"via":"container-cli","actorId":"operator:secrets-cli","setAt":1790000000000}}

The data-version step corestore-control-token-state (0.23.0 to 0.24.0) records a token stored before this release as revision 1 without a receipt: no actor or time of that set was recorded, and the contract admits exactly that one revision without one. A token counts as stored while either provider config holds an active one; a database without a token stays absent, and a rerun writes nothing.

The data-version step runtime-log-attempts (0.24.0 to 0.25.0) writes the log record of every slot's current attempt issued before this release, so its batches bind; its issuedAt is the time the step ran. A rerun writes nothing, and a database without slots is left untouched.

The data-version step cluster-node-hostname-intent (0.25.0 to 0.26.0) gives every node that reports through the Pallet-era node runtime (data.sparkNodeReport) the hostname intent of exactly the hostname it reports, at generation 1 and named by proposeClusterNodeName (cloudly-worker-ottilie is named ottilie, cloudlyworkergrasberg keeps its own name), and writes it into the cluster snapshot in the same transaction. It renames nothing. A node whose reported hostname is not one label or proposes no name, a node that only ever reported through the Swarm-era family, and a node whose cluster is gone get no intent; a node that already holds one keeps it, so a rerun writes nothing, and a database without nodes is left untouched.

The data-version step deployment-operation-route-evidence (0.26.0 to 0.27.0) records routeVerification: [] on every succeeded deployment operation that declares no route and recorded no route evidence, and moves its revision. A routeless operation now records that empty evidence in the transition that succeeds it; one that succeeded before carried none, so its status answered evidence the operation did not record. Operations in any other state, routed operations and operations that already record evidence are left as they are, so a rerun writes nothing.

The data-version step service-secret-target-predecessor (0.27.0 to 0.28.0) removes the secret target state a deleted service left behind. A service's target state is its secret revision authority, advanced in lockstep with its row's secretConfiguration.secretsRevision; deletion used to keep it, so a service recreated under the same id started at revision 0 against its predecessor's revision and every secret written to it was refused as SECRET_TARGET_REVISION_CONFLICT. Service rows carry no creation time, so the step decides from that invariant, and only for a state no secret of the service is still active under: a state whose service has no row in its organization, and a state above revision 0 whose row is at revision 0, which can only be its predecessor's. A state in lockstep with its row, a disagreement at any other row revision and a state with an active secret are left as they are; each removal is conditioned on the revisions it read, so a rerun, or a database without target states, removes nothing.

The data-version step route-claim-absent-owners (0.28.0 to 0.29.0) removes the stored nulls of absent owners from every route claim: a null wildcard owner is unset, and an exact owner list whose owners hold a null operationId or predecessorServiceId is written again without it. Nothing else of a claim changes and its revision moves. Each update is conditioned on the revision it read; a claim a concurrent writer moved is read again and its write planned from the fresh claim, and one that still cannot be written after three attempts fails the step by its id, so the data version stays at 0.28.0. Every reader and writer reads a claim through one shape reader, readDeploymentRouteClaimData, which leaves an owner stated as undefined out and refuses anything else that is not the claim's shape as DEPLOYMENT_ROUTE_CLAIM_MALFORMED. It reads and writes through the driver before any model is prepared; a rerun writes nothing.

The data-version step bounded-runtime-history (0.29.0 to 0.30.0) removes the two histories Cloudly kept without retention. Every deployment record the archive kept goes, one marked archiveState: 'archived' and one carrying archivedAt; the active records lose the archiveState discriminator, and its index archiveState_1 is dropped, because the deployment model no longer declares it and an undeclared index makes the model refuse its collection. Of the accepted Spark observations it keeps, per node, the rows of the sequence its observation state names and the one before it under the state's reporter session, plus every row an accepted census names; the rest are removed. It reads and writes through the driver before any model is prepared, and a rerun, or a database without either collection, changes nothing.

A running Cloudly reads the token on the next relay request, so nothing restarts for it. set-cloudflare-token is gone with Cloudly's Cloudflare integration: it answers USAGE_INVALID.

Keep the KEK that signed the completion receipt in the deployment keyring until the retirement step has run on every database; a rotation that drops it makes the retirement refuse. After the step has run, the archive row is the only thing that names that authority, and the keyring is needed only for the encrypted system material it owns.

Cloudly releases publish the semantic-version image code.foss.global/serve.zone/cloudly:<version> and, in the same gitzone release (gitzone 7 and later), the @serve.zone/cli npm package from ts_cliclient/ at that same version. The root package stays private and is never published to npm. The mutable latest image tag is a legacy manual channel and is not changed by the release workflow.

Run the TypeScript entry point during development:

pnpm run startTs

Start from code when embedding the control plane in another Node.js process:

The embedding process must set SERVEZONE_SECRET_KEYRING_PATH before every normal start(); it is intentionally not part of the public constructor DTO. The keyring is mandatory even when no workload secret delivery is currently expected because it owns encrypted system material and, until the cutover retirement has run, authenticates the completion receipt that step archives. Set SERVEZONE_WORKLOADINIT_APPROVAL_PATH as well before enabling Coreflow secret-runtime registration.

import { Cloudly } from '@serve.zone/cloudly';

const cloudly = new Cloudly({
  environment: 'production',
  publicOrigin: 'https://cloudly.example.com',
  listenPort: '3000',
  sslMode: 'external',
  servezoneAdminaccount: 'admin:change-me',
  mongoDescriptor: {
    mongoDbUrl: process.env.MONGODB_URL,
    mongoDbName: 'cloudly',
    mongoDbUser: process.env.MONGODB_USER,
    mongoDbPass: process.env.MONGODB_PASS,
  },
  s3Descriptor: {
    endpoint: process.env.S3_ENDPOINT,
    accessKey: process.env.S3_ACCESSKEY,
    accessSecret: process.env.S3_SECRETKEY,
    bucketName: process.env.S3_BUCKET,
    port: process.env.S3_PORT,
    useSsl: true,
  },
});

await cloudly.start();

Set SERVEZONE_INSTALL_DEMO_DATA=true only when you intentionally want the demo data installer to run. The code labels that path destructive.

API Model

Cloudly exposes a single composed TypedRouter. Managers add their own typed handlers to the main router, and CloudlyServer exposes that router through the HTTP/WebSocket server.

Cloudly also exposes an admin-JWT authenticated read-only MCP endpoint at /mcp. The MCP tools return safe summaries for clusters, services, deployments, domains, and nodes without machine tokens, SSH keys, secret values, provider zone IDs, or deployment logs.

Secrets v2 Safety Foundation

Cloudly persists secret material only as encrypted SecretVersion records. A mounted keyring supplies the active KEK and X25519 ingress recipient; startup fails closed before secret model and manager initialization when that keyring cannot be validated or authenticated completion verification fails. Earlier non-secret startup migrations may already have run. Completion verification and runtime preparation use the same loaded keyring instance, whose sole ownership is transferred to the secret manager only after verification. Admin create and rotation requests accept generated material or recipient-sealed input, and API responses contain metadata only.

Service-owned secrets use { kind: 'service', serviceId }. Shared material uses organization-owned SecretSets attached through service.data.secretConfiguration.secretSetAttachments. Generic service writes cannot mutate secret configuration or deployment authority, and unknown wire fields are rejected before persistence. Even an unchanged attachment request revalidates every referenced SecretSet and fails closed when one is missing, retired, or owned by another organization. Mail and Web Push use exact management scopes and revision-fenced service ownership; service deletion revokes service-owned material without deleting organization SecretSets. Retiring a SecretSet atomically detaches every current service consumer and publishes each post-detachment deployment manifest in the same transaction; deletion remains blocked while any attachment survives.

Secret and attachment mutations publish one immutable resolved manifest for every current target cluster, preserving accepted rollout history while dropping stale cluster states. A valid service that has not received an immutable image deployment yet is an explicit zero-manifest target, so its secret configuration can be prepared before deployment. Exact mutation replay never republishes manifests, but it retries the post-commit Coreflow config push when the persisted SecretVersion says the original mutation published runtime state. Coreflow broadcast delivery is concurrency-bounded and uses native TypedSocket request deadlines; overlapping broadcast requests coalesce into one active pass and at most one trailing pass. Copied plaintext bytes are wiped before any post-commit network dispatch begins.

Cloudly exposes 14 admin-only Secrets v2 handlers for secret and SecretSet lifecycle, attachments, resolution previews, and purge preflight. getSecretIngressRecipient additionally allows an exact-peer-bound registered Coreflow Secrets v2 runtime and returns only active public recipient metadata. Physical purge execution remains disabled because no erasure worker is authorized. The control plane also implements recipient enrollment, Spark swarm observation intake, target-authority reconciliation, live Coreflow registration, resolved-manifest rollover, recipient-sealed material delivery, deployment report replay protection, and fenced Corestore credential publication. These runtime contracts are supplied by @serve.zone/interfaces 32.7.1 and @serve.zone/api 32.0.0. End-to-end workload delivery is not claimed until the matching Coreflow and Spark integrations are released, configured, and enrolled. Spark observation replay returns the persisted exact receipt before age checks. Every non-replay observation, including the first report of a replacement reporter session, must advance the node-wide persisted observedAt high-water.

Every normal Cloudly startup requires the mounted keyring named by SERVEZONE_SECRET_KEYRING_PATH; there is no keyless mode or empty-secret exception. Startup validates the keyring and runs the data migration chain -- which, while the cutover retirement is still outstanding, authenticates the completion receipt with it -- before secret models and managers initialize, then transfers sole keyring ownership to the secret manager for runtime use and shutdown wiping.

Coreflow runtime registration additionally requires the private artifact named by SERVEZONE_WORKLOADINIT_APPROVAL_PATH. Cloudly parses an exact active or revoked schema and cryptographically verifies the detached statement and bundle, public key, registry evidence, trusted signer and revocation policy, expected version, image-index digest, and platforms through @serve.zone/workloadinit. Missing, malformed, or otherwise unverifiable approval material returns workloadinit-unconfigured; an explicit revoked artifact, revoked trusted signer, or failed revocation/timeline policy returns workloadinit-revoked. Cloudly reloads the approval while validating a live registered session. A generation, digest, or status change invalidates that session and requires Coreflow to register again. Every reader — registration, a live session's validation, runtime admission and Pallet material issuance — reloads the file under the same strict rules and judges its state and timeline at its own time; only the cryptographic and registry verification of an active artifact is remembered, once per exact artifact content, because it fetches the release's registry graph (thirteen sequential requests). A changed, revoked or removed artifact is seen by the next read, and a verification that failed is never remembered.

Protocol Offers per Session Kind

The contract two peers speak is the @serve.zone/interfaces release they were built against, exchanged in the registration body itself: there is no protocol number beside it and no shape version anywhere in the package. ts/protocol.ts holds Cloudly's side of that exchange — one IProtocolOffer per session kind Cloudly is the server of (cloudlyClient, palletRuntime, clusterRelay, sparkNode, sparkSwarmNode), each built by protocol.createProtocolOffer from the installed release with its own minimum peer version, so raising what one kind of peer must speak is one named change for that kind alone and never a number an offer claims for a contract this build does not carry. Every accepted registration answers with the offer of the kind it opened, and the Spark routes state it on every body they answer.

Negotiation is the first thing every carrier does with a body, before its own validator, before the bearer or the JWT is looked at, and before anything is written: the contract validators are exact-key, so a peer of another major that was shape-checked first would be answered as malformed instead of being told which contract it met. For registerPalletRuntimeSession that order is also what keeps a refused node's packet access: the withdrawal happens inside the registration the refusal precedes.

A refused peer is answered by name, never by a collapsed denial. On the three TypedRequest carriers — registerCloudlyClientSession, registerPalletRuntimeSession and registerClusterRelay — the answer is a TypedResponseError whose text is describeProtocolRefusal and whose errorData is the IProtocolRefusal itself, so the peer reads which of the two builds an operator has to upgrade and backs off for protocol.refusedOfferRetryIntervalMs. The five Spark routes are plain HTTPS POSTs that carry no typed error, so their answer is HTTP 422 with the IProtocolRefusal as the whole body, which is what sparkNodeHeartbeatContract.protocolRefusalStatus and sparkSwarmNodeContracts.protocolRefusalStatus state.

A body that carries no readable offer is malformed, not incompatible: nothing was negotiated, so there is no refusal to name and nothing to back off from. The typed carriers collapse it into their registration denial and the Spark routes answer 400, exactly as they answer any other unreadable body. A body that does not name its node -- nodeId and nodeToken are the two members all five Spark routes read it by, and each must be a non-empty string -- is malformed for the same reason and gets the same answer, judged right after the offer so that a statement naming no node never reaches the credential store.

A refusal costs the peer nothing it already holds. No session record, no identity tag, no session epoch, no credential mutation, no relay registration sequence and no undispatched announcement: each of them happens after the negotiation, so a refused node keeps its workloads and its packet access and a refused relay keeps its custody and its stored registration.

Two carriers state an offer outside a registration: getDeploymentPreflight, the read that opens a deployment, and getServiceById, the read that opens an adoption. A deployer states its offer on those two and on no mutating deployment carrier, because a reservation's requestDigest covers its whole request body and an offer inside one would tie an in-flight deployment to whichever @serve.zone/interfaces release the deployer had installed. There the member is optional: a request that states none is served exactly as it always was, with nothing negotiated and nothing logged, and a stated offer is judged against the cloudlyClient minimum -- before the preflight's validator, before the bearer and before any service is read -- and refused by name in the same shape the registrations answer a refusal in. A stated offer Cloudly cannot read is not a refusal, because nothing was negotiated: the preflight names it as one INVALID_REQUEST blocker of its own report and getServiceById, which has no validator of its own, answers it as a malformed offer, so a deployer that tried to state a contract is never served as one that stated none.

The node-runtime routing read requires the Spark contract in @serve.zone/interfaces 32.11.0. Roll out a Spark build carrying 32.11.0 before this Cloudly build: Cloudly's sparkNode minimum is 32.11.0 so an older Spark peer is refused at its protocol-bearing heartbeat before it depends on the new read. This raise is scoped to sparkNode; palletRuntime remains at 32.10.1 and the retiring sparkSwarmNode routes remain at 32.0.0.

Spark Node Runtime Routing Read

getNodeRuntimeBinding accepts only the node ID and its current Spark bearer. Cloudly snapshots that exact body before authentication yields, then derives the complete response itself: canonical public Cloudly origin, current node and cluster, persisted runtime controller and namespace, validated cluster relay, current cluster phase and generation, and the Spark credential generation and session epoch that authenticated the answer. The caller cannot submit a cluster, relay, controller, namespace, phase, generation, or workload grant.

The read is one SmartData transaction. It real-write fences the credential, node, exact cluster data, and exact runtime record, and reads the controller record, which a controller identity takeover fences through every node's credential record (see "Protected transactions, probes and request budgets"), so credential rotation or revocation, node movement or deletion, relay changes, phase changes, controller replacement, and cluster deletion cannot produce a mixed answer. A transaction retry may return only the exact response its first attempt observed; if the routing identity changed, the read rejects. Every malformed, unauthenticated, stale, missing, corrupt, or raced request receives the same Node runtime binding rejected. typed error without a bearer or record-specific reason.

The returned binding is routing identity only. It creates no runtime session, does not change a cluster phase, and grants no workload, network, secret, or phase-transition authority. Pallet must still open a fresh authenticated runtime session before it can receive controller-owned work.

Pallet Runtime Sessions per Node

registerPalletRuntimeSession binds one node session to the exact live socket that carried it. A physical peer may carry one node session or many: a node can connect directly, and a cluster relay forwards the runtime session methods of every node it serves over one socket. Registrations, bindings, peer scopes and current-peer selection are therefore keyed by (peer, node). Cloudly still hashes and fences each node's own bearer and its own credential authority, so a carrier peer never becomes an authority for the nodes it forwards.

Each node on a peer holds its own session binding. One node registering, replaying, moving to another peer or being revoked never disturbs the other sessions on the same peer. A node's session may move to another peer at any time: the newest session epoch wins, the old peer's copy stops authorizing, and every protected effect remains fenced by the persisted credential authority rather than by socket identity. A departed peer loses every node session it carried, including every managed VPN member admitted through it, and leaves sessions held through other peers untouched.

A registration is identified by the controller process, the carrying peer, the node, its credential generation and the node process that registers it. Pallet names that process with nodeInstanceNonce, the canonical lowercase UUID it generates once per process start and sends on every registration. Since @serve.zone/interfaces 32.0.0 the member is part of the registration shape itself, so a body that names no node process has no contract to be read under and is refused before any rule about carriers applies: one relay socket carries a whole cluster and could otherwise not tell a node restart from a replay. A peer holding a Cloudly identity that no longer verifies — an expired cluster machine credential, say — is refused by name with carrier-identity-unverified until its identity verifies again. The nonce authenticates nothing; the node's own bearer still does.

The binding also records the node's clusterId from the ClusterNode fenced by the registration transaction. Current work, reporter allocation reads, resolved-authority reads and historical session recovery all fence the node again and require that same owner. Cloudly therefore refuses a session when the node is absent or its owner differs, rather than allowing a retained session or eligibility proof to cross clusters. Pallet runtime protocol admission requires @serve.zone/interfaces 32.10.1 or later, the first contract in this major that carries this binding.

Every registration also states resolvedAuthorities: the sorted, duplicate-free list of what the registering build can resolve for a workload — network, secrets, storage — which Cloudly stores on the session record. It states a capability of that build, not a permission, and it is deliberately not part of the registration identity: a changed build is a new process and therefore a new nonce, so a replay of one registration that states different authorities is refused with conflict rather than quietly widening or narrowing what that session may be handed. The service runtime producer reads the list back from the node's current session inside its own admit transaction and refuses node-authority-unsupported when the workload it is about to seal needs an authority the node did not resolve, instead of handing a node work it would silently park. Cloudly seals no network, secret or storage authority until those producers exist, so the workloads it admits today need none.

The session record also stores the registration's protocol offer. A projection member that an earlier contract's exact-key reader refuses is not a session minimum — the node keeps its session and every projection that does not carry the member — so the network compiler reads the release the node's current session speaks and withholds only a projection that carries such a member: published-port ranges and symmetric entries need 32.31.0 (palletProjectionMemberMinimums in ts/protocol.ts), and a successor that drops the withdrawals its head carries needs 32.50.0, the release whose node passes its own application receipt to its admission. A session recorded before Cloudly stored the offer holds protocol: null (the data-version step runtime-session-protocol-offer, 0.21.0 to 0.22.0), which is judged below every member's minimum until the node registers again. So the rollout order is Pallet first: a service may publish a range or a symmetric port before its nodes are upgraded, and each node's projection is signed for it once it registers from a build that reads them.

A sealed run workload also states where its node fetches the image bytes. A cluster is outbound-only, so nothing inside it dials Cloudly's registry directly: when the cluster has a relay origin published, the workload carries pullEndpoint — that origin as host[:port] — and the node pulls through its own relay. registryHost is never rewritten: it stays the publication identity the image release is bound to, and the digest pins the bytes, so the endpoint is untrusted for content and buildRuntimeAssignmentImageReference gives the same digest either way. A cluster whose relay an operator has not enabled states no endpoint and pulls from the publishing registry.

A relay carries one cluster, and only the nodes of that cluster. Cloudly resolves the cluster machine identity on the carrying socket to exactly one cluster for every registration it forwards, and the registration transaction requires the node that registration names to belong to that cluster, read from the node document the very same transaction fences. A node of another cluster is refused by name with relay-node-foreign, and a carrier whose cluster Cloudly can no longer name — because that cluster was deleted, say — is refused with carrier-identity-unverified rather than taken for a carrier that speaks only for itself. Both registration transactions apply the rule, so a refusal happens before the node's old packet authority is withdrawn and consumes nothing at all: no session, no session epoch, no moved node fence. A directly connected node is unrestricted as before: its own socket carries nothing but itself.

A changed nonce, or a changed credential generation, is a new registration: a new binding, a new session epoch, and the node's previous session fenced. An unchanged pair is a replay and returns the exact historical binding. A node restarting behind a live relay is therefore a new registration, and bearer rotation behind a relay needs no reconnect of the carrier: the node registers again with its unchanged nonce and its new generation on the same live socket, and the other nodes that socket carries keep their sessions. The carrying peer stays part of the identity because a committed session names the exact carrier it was registered on, so a registration that arrives over another socket is a new registration, exactly as it was before a node could name its process.

One protected server tag name marks a peer as a live Pallet session carrier for server enumeration, however many nodes it carries; TypedSocket caps server tag names per connection, so a carrier cannot mint one per node. That tag confers no authority and its payload only echoes the most recent registration.

Controller pushes are bounded per node, not per controller or per peer: at most eight in-flight assignment deliveries per node, and the node-keyed network delivery lane keeps at most one in-flight network delivery per node while later callers join the pending one. A single node reaches exactly the same ceiling it reached before, and one busy node can no longer exhaust the delivery budget of the other nodes on its peer.

A relay socket must sustain the delivery of every node it carries: each node keeps its own eight in-flight assignment deliveries and its own single network delivery, so one relay peer carrying N nodes can see 8 * N + N controller pushes on one socket. The socket is not what bounds that. With a realistic service configuration each push is roughly 40 KiB of canonical JSON, far inside both the contract's 320 KiB canonical limit and the 2 MiB frame the server accepts, and sixteen nodes each filling their own bound over one socket are delivered and acknowledged inside the 10 s each push is given. What bounds it is Cloudly's protected transaction layer: a node's transactions write that node's own records, so concurrent deliveries to one node serialize and their optimistic retries cost quadratically, while deliveries to different nodes do not conflict at all. Cloudly's own delivery pass never produces such a burst — it is one joined lane walking one bounded page, one push at a time — so what protects a relay socket is the per-node ceiling together with that serial pass, and a caller that fans delivery out to one node at once must respect the same limit.

The delivery pass walks every desired assignment on every period, but a session is pushed each assignment revision once. A node keeps what it acknowledged in its durable ledger, and a session never outlives the node process that registered it: Pallet draws a fresh nodeInstanceNonce at every process start, so a restarted node always registers a new session. A revision the node's current session acknowledged is therefore answered from that acknowledgement, after the same proof that the revision is still current, and is not pushed again (AcknowledgedAssignments, keyed by node with the peer and session it was acknowledged on). A new revision, a new session on the same or another peer, and a restarted manager push it again. One edge is left to the operator: a node whose database is wiped or restored while its Pallet process keeps running keeps its session, so it is pushed its assignments again only once it reconnects. Before, every assignment was pushed again about every 6.5 s and each push cost the node a full admission (lab 2026-10-03, Pallet 36.1.0). Local storage claims are still replayed on every pass: their answer carries the volume's state.

What Cloudly remembers is bounded by what is still desired: each remembered revision is stamped with the delivery cycle that last proved it current, and once a pass completes the scan of every desired slot, every revision that cycle did not prove — a retired attempt (each deploy attempt has its own assignment id), a node gone offline, a replaced session — is forgotten. Forgetting one can only cost one push again. inspectDelivery().acknowledged counts the revisions remembered.

Every dispatch names its outcome (AssignmentDispatchOutcomes), because the pass only counts it and the runtime view's delivery reads the node's own reports: a revision the node accepted but has not yet reported on reads undelivered exactly like one that never reached it. The reason is a closed word: accepted, replay or historical is the node's answer and acknowledged an answer from the session's acknowledgement; otherwise the step that failed — node-busy (the node's limit of pushes in flight), node-not-connected, session-moved, intent-not-current, request-unbound, node-refused (the node or its relay answered with a refusal), push-failed (no answer within the push timeout, or a failed carrier) or acknowledgement-unbound. Cloudly logs an outcome when it changes for that assignment, a failure with its cause chain's classes and codes only, never a request or an answer, and inspectAssignmentDispatch(assignmentId) reads the last one; outcomes are bounded and forgotten like the acknowledgements. A refused or unanswered push is pushed again on the next pass under the same session.

A node report Cloudly refuses — reportRuntimeAssignmentTerminalReceipt, reportRuntimeAssignmentObservation, reportRuntimeAssignmentLogs or reportRuntimeNetworkProtection — keeps its fixed error text and carries { code: 'runtime-session-report-refused', reason } as the error's data. The reason names the check that refused it in words only: for a protection receipt reporter-session-unknown, request-binding-rejected, protected-authority-unknown, receipt-replay-conflict, endpoint-replay-conflict, uplink-replay-conflict, receipt-admission-rejected, receipt-retirement-rejected or a …-write-conflict; for a log batch log-attempt-unkept (an attempt Cloudly never issued or whose logs it let go) or log-capture-ended (a new number after the capture's final batch); for any report request-invalid, not-current, session-superseded, controller-unavailable, state-conflict and the other session refusals by code — among them the retryable controller-deadline, a request that did not settle within its budget — and unclassified for a failure that is not a refusal (a database or transport failure). Cloudly logs each refusal as a warning — <method> from node <nodeId> refused: <reason> (<cause chain>), each link's class with a runtime refusal's code and reason or a foreign error's first message line cut at 160 characters — once per node, report and distinct reason until that report is next accepted, since a node retries a refused report about once a second.

Protected transactions, probes and request budgets

The database is first-writer-wins: a transaction that writes a record another open transaction has written, or that was published after its snapshot, fails and is retried by the driver after a short backoff. A record every transaction writes therefore serializes all of them, so Cloudly fences only what a transaction's decision depends on, and takes a fence only where it commits something.

• A node's request writes its node's records only. runCurrent (the Pallet observation, terminal, network and credential reports and registration) real-write fences the node's credential record, which serializes it with rotation, revocation and re-registration, and the node document, which serializes it with deletion and movement. It reads the controller record and refuses an identity other than the one this process holds, but never writes it, so requests of different nodes never conflict.
• A log batch appends without a fence. appendCurrent (withAppendTransaction) commits a log batch's ledger rows (batch, capture, attempt, bounds, the node's discard record) after the same identity and session checks, in one snapshot, but fenceRead writes no fence in it: a batch never conflicts with a producer pass or another report of its node, only with writers of its own capture and attempt rows. Its decision (decide, and the not-kept answer) runs as a probe. The price, accepted: the append is proven current at its snapshot only, so a revocation or supersession that commits while it is in flight does not abort it. A batch is keyed by assignment, capture and sequence, so a resend after a refused or unknown commit lands once.
• The controller identity is stated once and fenced through the nodes. The record is never replaced; a new identity is stated only where none is stored (a fresh database, or one whose record was removed), and the transaction that states it also writes every node's credential record of both purposes. A node transaction still running under the old identity therefore either committed before the takeover, or conflicts with it and, retried, reads the new identity and refuses. A producer's transaction (runController) still writes the controller record, which keeps producers serialized among themselves and against a takeover; no node request writes it, so a producer never makes a node request retry.
• A pass that decides nothing commits nothing. A probe (withProbeTransaction, probeCurrent, probeController) reads one snapshot, takes the same identity and session checks, writes no fence (fenceRead is the one owner of that rule) and is rolled back once it answered, so a write made in one by mistake is never published. The delivery listing, a delivery's reads before and after its push (trust, head, renewal due, relay run, assignment and claim proof), every producer's plan, the service runtime pass's decision and verdict comparison, the node network pass and the managed VPN reads are probes; only a step that changes something runs as a transaction, which reads and fences everything again. A steady fleet therefore writes nothing on its delivery period and its sweeps, where every pass rewrote the controller record, the protected authority and the managed VPN state before.
• Every node request has a budget. Its transactions share 6 s (requestBudgetMs), below the 8 s a cluster relay waits for a forwarded request: each runs under the client's transaction deadline (timeoutMS) for what is left, and under a signal that the request's cancellation and the controller's shutdown abort. A request whose transactions do not settle in time, or that the relay cancelled, is refused deadline — the report refusal reason controller-deadline — and the node retries it unchanged; a stopping controller answers controller-unavailable. A request is answered when its budget ends even if its work ignores the signal; that work drains under the runtime manager and counts as critical until it settles. A registration carries, beside its fixed text Pallet runtime session registration denied., the @serve.zone/interfaces runtime session refusal { code: 'runtime-session-report-refused', reason } for every momentary cause of Cloudly's own: controller-deadline for a spent budget, controller-unavailable for everything else that is no verdict on the node — the database unreachable, a write conflict, a credential being applied, a controller not ready or stopping, a failure nothing names. A verdict keeps the fixed text alone: an inactive or revoked credential, a deleted, retiring (node-retiring) or foreign node, an unverifiable carrier, a replay that states another registration (registration-replay-mismatch), a moved controller identity, an invalid body (registrationRefusalOf). Each registration refusal is logged once per node and reason, value-free, until the node registers. A request waiting for the managed VPN lane leaves it as soon as its budget ends.
• A request's signal lives as long as its work. Between two pieces of a request's work its budget is a deadline and nothing else. Each piece (RuntimeRequestBudget.run) owns one abort controller with one listener on the request's signal, one on the controller's lifetime signal and one timer, and removes all three when the piece is answered, however it ends. Nothing a request leaves behind hangs off the lifetime signal: on the Node.js release the image runs (24.13.0), composing each request's signal with AbortSignal.any off the lifetime signal kept about 37 MiB per 200,000 requests, and with a timeout signal beside it about 425 MiB, after every timer had fired.

Measured on the database engine production runs (@lossless.org/nosqldb 10.6.1, first-writer-wins), in test.runtimefencecontention: thirty-two nodes reporting at once beside two producers committing back to back settle in about 0.2 s; with every transaction writing the controller record, not one of the thirty-two requests settled within 120 s.

Workload Logs

Cloudly keeps the logs Pallet nodes report with reportRuntimeAssignmentLogs. Each attempt gets its log record in the transaction that issues it, holding its run revision and its config's log policy, so its batches bind after the slot released the assignment record too. A batch already held, by its capture's high-water mark in the ledger or in the queue, is answered replay. A new batch is queued with its payload in one cheap transaction and answered accepted; the drain commits its ledger row and keeps the payload inline beside it, in a collection of its own (cloudly_runtime_log_inline_payloads), so it reads at once. A gap the node did not account for is recorded; numbers it skipped behind a not-kept loss are not a gap. A batch's discardedCaptures is summed against the node and session once per accepted batch, never on a replay.

Payloads are stored as segments in Cloudly's own bucket (s3Descriptor): one object per capture, runtime-logs/<serviceId>/<assignmentId>/<captureId>/<first>-<last>.jsonl, holding each batch's canonical JSON on a line of its own. The drain writes a capture's inline payloads as a segment when the oldest was committed 5 minutes ago, when they fill 1 MiB (a larger batch is a segment of its own), when the capture's final batch arrived, or, largest capture first, when all inline payloads pass 16 MiB; past 32 MiB it commits nothing more until segments are written, so the queue's bounds refuse new reports. Only after the write are the rows moved onto the segment, so a payload is always in the database or in a written segment, and a stop loses none: the next process writes what is due. At about one batch a second per capture this is one object per capture every five minutes instead of about 300. Rows committed before segments name their own object, runtime-logs/<serviceId>/<assignmentId>/<captureId>/<sequence>.json, and are read and removed as before until they age out. Everything below runtime-logs/ is workload logs, which the service template declares data a backup leaves out.

Kept logs are bounded per attempt to 8 MiB of line text and 8192 batches, and per slot to the newest 8 attempts; each batch lives for its config's logs.retentionMs, and at least 30 days. The oldest batches go first, and the capture's truncation states them (batch-cap, byte-cap, retention, with exact batches, bytes and entries), never a silent drop. A released attempt keeps its logs until the later of its issue, its last accepted batch and its release, plus that retention; an attempt its slot still holds keeps its record however quiet it is. One sweep drops expired batches and attempts, removes each dropped row, a row's own object before the row and a segment once no row names it, and removes objects nothing holds once they are an hour old, re-reading each one's age right before it removes it. A Cloudly configured without object storage keeps no workload logs and answers every batch not-kept.

Reading

An operator reads kept logs through two service requests, authorized like getServiceRuntimeAssignments (an administrator, or a deployment identity of the service's organization). getServiceRuntimeLogAttempts { serviceId, cursor, limit } pages the service's kept attempts, newest issued first and ties in assignment id order. An attempt its slot still holds states expiresAt null; a released one states when its logs are dropped whole.

getRuntimeAssignmentLogs { serviceId, assignmentId, cursor, limit } pages one attempt's records in delivery order: captures in the order their first batch was accepted, each by sequence. A capture's truncation opens it, each kept batch is stated as the node sent it, and a gap states Cloudly's own losses: numbers recorded missing and kept rows whose payload object or segment is gone. Adjacent losses are one gap, stated once the batch after it is found or the capture ends, so a page never ends inside a loss and the next page never opens with its rest. A page ends before limit records, the contract's 4 096 entries or its 2 MiB, and continues at <capture ordinal>.<last sequence delivered>. The ledger is read in transactions and the payloads outside them: a batch a bound drops while a page is read is stated by the truncation that dropped it, never as a gap. Both reads refuse attempt-not-found, cursor-invalid and limit-invalid by name, and every answer is checked against the contract before it leaves.

The service page's Logs tab shows the newest attempt, read when the tab is shown: stderr lines as errors, stdout lines as info, and every loss the node stated, gap and truncation as a warning. It reads at most 64 pages, oldest first, and says so when the attempt holds more.

Runtime Service Authority

Every service-scoped runtime, network and credential write advances that service's runtime authority fence inside the transaction that commits it. Runtime and network callers name the intent explicitly, and the service credential paths use the admit default. The final removed observation of a relocated slot is the one accepted report that leaves the service fence untouched, because a replacement may already own the slot.

admit authorizes new work, and a deleting service refuses it: admitting or delivering a run revision (which includes issuing its registry pull credential), reserving a workload lease, staging network policy, setting private-network membership, and creating, rotating, revoking or exercising a service credential. settle carries execution that was already admitted to its end, and a deleting service accepts it: admitting or delivering a stop or removal revision, accepting terminal receipts and observations, quarantining a stopped workload lease, retiring the service credential during deletion, and every input a network compilation takes from the service (its executions and their observations, its reserved workloads, its network policy, and preparing its private-network membership, which may create an empty membership or detach networks that are retiring). A deleting service therefore still drains: Pallet reports on what it runs, its stop and removal revisions reach the node, and nodes that share a network with it keep compiling. This holds while the delete runs and after a failed delete released its lease, because the fence pins the service's loaded lifecycle state together with its lease: the live lease while one is held, and no lease once it is released. A run revision of a deleting service is never delivered.

Each accepted observation stores observationAcceptedAt, read from Cloudly's own wall clock (the runtime manager's now option) rather than the worker's observedAt. An exact replay keeps the earlier value.

Signed Pallet Network Delivery

The runtime manager prepares a dedicated Ed25519 signing authority at startup. Its private key is encrypted through SecretManager with controller-specific identity and environment-bound authenticated metadata. Restart preserves the authority; rotation and revocation require an explicit controller operation. Public revision history remains available for disconnected workers. Rotating replaces the encrypted private key, and revocation removes it.

RuntimeNetworkProjectionStore retains one complete signed intent per node in SmartData/NoSQLDB. A producer must fence allocations, memberships and current assignment observations in the controller transaction before admitting it. Admission verifies the signature, current signing authority and exact projection predecessor. An unacknowledged projection prevents a newer generation from replacing it. The compiler also waits for the exact head's application receipt before composing a successor: storing a projection alone cannot advance its generation. Changes received meanwhile coalesce into the next compilation from the current durable owners. Re-signing the same generation under a rotated authority remains possible while application is pending and requires a new admission ACK.

The joined runtime delivery lane sends the exact pending signing revision before its projection over the registered physical Pallet socket. Pending trust is persisted before transmission, so lost ACKs and reconnects replay the same key and then walk every later rotation. Replies must identify the exact sent key or projection. Credential revocation, reconnect, controller change and shutdown invalidate late replies. The server permits 2 MiB WebSocket frames; the shared projection contract limits complete canonical state to 896 KiB.

These ACKs prove admission only. They never establish readiness, renew the signed DNS deadline, clear packet withdrawals, or release addresses, handoffs or aliases. Activation barriers and Pallet realization remain required before this delivery foundation can activate workloads.

Projection Application Receipts

A projection carries every packet-grant withdrawal of its predecessor, and may not grant a withdrawn flow again, until the node proves it applied the head that carries them. That proof is the node's projection application receipt (IRuntimeNetworkProjectionApplicationReceipt, @serve.zone/interfaces 32.50.0): its durable statement, minted from its own packet journal, that every packet table it holds was composed from one admitted projection, or that it holds none. Pallet 36.3.0 and later report each receipt over the node's current session (reportRuntimeNetworkProjectionApplication).

RuntimeNetworkProjectionStore.acceptApplicationReceipt binds the report to the current session and to the reporter history entry the receipt names (current, or historical for a receipt minted under an earlier session of the node), and the receipt to the node's receipt chain and its projection head (admitRuntimeNetworkProjectionApplicationReceipt): its digest, its scope, the exact predecessor receipt, and a named projection that is the head or an earlier generation of it. It keeps the latest receipt per node (cloudly_runtime_network_application_receipts); the exact latest receipt again is answered replay. A refusal names its check in the report's error payload and in Cloudly's log: reporter-session-unknown, request-binding-rejected, projection-head-unknown, receipt-admission-rejected or receipt-write-conflict.

Both admitters of a successor — the compiler composing it and admit committing it, in one transaction — ask one owner for its appliedPrevious (readAppliedPrevious): the head itself while the latest receipt names exactly it (selectRuntimeNetworkProjectionAppliedPrevious) and the node's current session speaks 32.50.0 or later (palletProjectionMemberMinimums.applicationReceipts), else nothing. The compiler reports awaiting-application, with the held head's ID, generation and digest, until this proof is available. Repeated waits for the same head log one warning. A promptly applying node can advance on the next pass without an additional timer; an accepted application report requests that pass. With it, the successor drops the head's withdrawals and withdraws only what the head grants and it does not, so a withdrawn workload, router or cluster ingress grant is granted again; without it the entire current projection stays held. A receipt that names an older projection than the head advances the chain and releases nothing. A node that sends no receipt — Pallet 36.2.1 and earlier — cannot advance beyond its first projection, and neither can a session that speaks Interfaces below 32.50.0. An accepted receipt asks the network producer for a pass, which signs the changes held behind the head. When dropping the head's withdrawals would be the successor's only change, no successor is signed: the receipt already proves those withdrawals applied, and a generation that changes nothing would only make the node apply again, superseding every attachment it is preparing meanwhile.

Upgrade nodes to Pallet 36.3.0 or later, speaking Interfaces 32.50.0 or later, before rolling out application pacing. A receipt with nativeJournal: null proves every packet table was released and also permits advancement after a failed application; failure, absence, a storage ACK or a reconnect alone does not. The receipt wire contract is unchanged. A Pallet that sends receipts does so to any controller whose offer states @serve.zone/interfaces 32.50.0 or later, which includes a Cloudly released before this handler: that Cloudly answers that the method does not exist, and the node records the refusal against its receipt unit alone, keeps the receipt in its outbox and goes on reporting everything else, while every withdrawal stays carried. Its receipt is delivered once Cloudly is upgraded.

Network Reservation Ledger

RuntimeNetworkAllocationStore retains explicit protected-network revisions, pool cursors and complete handoff/workload reservations through SmartData. It uses the runtime controller's caller-owned transaction, including real node and service fences. A producer reserves the workload lease and admits its matching assignment in that same transaction. Startup prepares the store without choosing network settings or reserving addresses.

Before reserving either kind of lease, the store requires a current native protection receipt from every affected egress owner, including offline workers. The authenticated reportRuntimeNetworkProtection socket handler retains exact receipt history. A receipt names the worker's complete joined native protection journal; projection admission ACKs cannot substitute for it. The allocation transaction fences the current credential/session, receipt and node for every owner and retains the complete proof with each reservation. A reconnect, credential change, protection update or null native barrier blocks new allocations until the complete current proof is available. Historical outbox replay never restores eligibility.

A node built on @serve.zone/interfaces 32.44.0 or later reports a null native barrier only in answer to the retirement request of its projection (runtimeNetworkProtectionContract.withdrawal): its allocation-pool guard outlives the node process and is restored at boot before networking, so a clean stop, a crash, a reboot or being offline keeps its last receipt current, and a stopped node no longer blocks allocation for the whole fleet. A node of an earlier release still reports its null barrier when it stops, and Cloudly still accepts it. While a node's projection head carries the request, each of its receipts also passes bindRuntimeNetworkProtectionReceiptRetirement with the same predecessor as its admission: the null answer always passes, a positive receipt only while it could predate the request — no authority newer than the request's and no null predecessor — and anything else is refused as receipt-retirement-rejected. The head is read without a fence, the newest one whether acknowledged or not: the request reaches a node only in a head that committed first.

Protection updates require the exact predecessor and retain existing pool definitions, protected coverage and offline egress authorities. Add a disjoint pool to expand capacity. Dedicated transit links reserve a /31; workload links reserve a /30 with two ordinary usable peers. Allocation cursors advance without reuse, and unique subnet, assignment and attempt indexes protect retained reservations. Lost replies replay the same immutable reservation.

Quarantine retains the full lease and its original protection/handoff bindings across restarts. A handoff cannot enter quarantine while it still has reserved workloads. Neither quarantine nor a newer protection revision makes addresses available again. Protection is required before reservation; activating reserved endpoints additionally requires native application. The native receipt producer and CNI application remain unfinished. The store does not issue application receipts, provide a generic fence override or release quarantine.

Complete Network Compilation

RuntimeNetworkCompiler reads reserved local executions, complete visible private memberships and their executions, and still-reserved endpoints from the previous projection. It includes declared names with no replicas and members with no aliases. Removing a local view retains live remote leases; explicit assignment stop intent quarantines a lease once the node's joined stop receipt is accepted. Until then the stopping lease stays an endpoint that is never ready: its run may still hold the address, and its node's ACTIVE generation still carries the attachment, so the lease is never withdrawn from under a run that has not stopped. A handoff replacement therefore waits for that receipt as well. Retiring networks are detached through the membership owner, with revision and alias changes committed in the same transaction. Empty initial memberships are persisted as revision one.

Readiness requires the exact lease-bound run assignment, an accepted ready observation, and its current durable reporter session. A reconnect preserves old observation provenance but removes its readiness until a new observation arrives. Private packet grants do not depend on DNS readiness. Explicit service egress policy is persisted separately with revision CAS; missing policy blocks that service's compilation. DNS forwarding permission never grants internet or platform packet access.

Router-origin egress has its own explicit router.resolverIds and router.platformEndpointIds selections in the trusted compilation request. Both arrays are required, sorted and unique; empty arrays grant nothing. Selected IDs resolve against the transaction's current protected inventory and require the exact selected local handoff. A resolver grants its declared address and port over TCP and UDP; a platform endpoint grants only its declared tuple. Neither workload egress nor DNS forwarding policy supplies these selections. The signed projection retains full router-source and destination withdrawals across admission ACKs, including handoff replacement and endpoint tuple changes. An outstanding withdrawal blocks regrant until the node's application receipt names the head that carries it (Projection Application Receipts).

On a cluster whose ingress was designated for a Pallet-born cluster, the compilation adds workloadIngress: one grant from each ingress lease to each lease of every other service holding a slot in the cluster, on each TCP target port its public domains route to, where either lease runs on the compiled node. Both leases become endpoints of the projection even without a shared private network. A grant the successor drops is withdrawn explicitly. A flow wanted again while the head still withdraws it stays held, and the route that wants it names the hold. The whole projection waits for the node's application receipt of that head; its successor coalesces pending changes, drops those withdrawals and can grant the flow again. A projection that never granted such a flow omits the member and keeps its canonical bytes. hostPlatformEndpointIds names the protected authority's platform endpoints whose address the compiled node's own host carries: the listener and the managed VPN hub its cluster's relay registered from that node, and every address the node's current session reported for its host as hostAddresses beside its protection receipt. A plan's platform endpoint shares the relay listener's address when it runs beside the relay, and a host without a relay — a node with a Corestore beside it on one machine — names its own through hostAddresses; the node then delivers a workload's flow to such an endpoint locally. It grants nothing. hostAddresses describes the host as the current session sees it now, so a current report states them on a replay of its receipt too, a historical report states nothing, and a current report without them leaves the node's host carrying none (cloudly_runtime_network_node_host_addresses holds one record per node).

Compilation fences every contributing owner, signs locally under the current encrypted signing authority, and admits the complete projection in one controller transaction. Concurrent changes refresh the whole transaction or reject it. It carries exact withdrawals and retired material across admission ACKs; a regrant that conflicts with an outstanding withdrawal is rejected (withdrawn-grant-reissued) until the node's application receipt names the head that carries it, after which the head's withdrawals are dropped. Retired leases and handoffs stay carried. No address or alias becomes reusable.

The runtime manager's trusted controller methods are stageNetworkProtection, stageServiceNetworkEgress, stageServicePublishedPorts, reserveNetworkHandoff, and stageNetworkSelection; the two policy methods back the administrator's egress and the owner's published-port RPCs, each writing its half of one service policy document. No caller supplies an ad-hoc compilation request: compilation takes the node's staged selection, which names an exact handoff (or null for protection-only bootstrap) and explicit router selections, together with a controller-issued DNS window of at most 15 minutes. Startup seeds neither pools, policies nor selections. These methods join shutdown and are not worker RPCs. The controller-issued DNS window does not qualify a worker clock. Deployment admission, native application and Pallet-aware service deletion coordination remain prerequisites for live activation.

Node Scope

RuntimeNodeScope (ts/manager.runtime/classes.runtimenodescope.ts) is the one definition of the nodes Pallet runs, for every producer that needs it. A node is an egress member when it has an active Pallet credential, a current session, was not admitted to retire, and its cluster's runtime phase is switching or pallet. Requiring a session keeps a node that never registers out of the controller-wide eligibility set; a node that registered and went offline keeps its session and stays a member. A node is out of scope from the moment its retirement is admitted (see Node Retirement), so no producer counts it again; a node still queued to retire stays a member but is listed with no platform, so nothing new is placed on it. The network compiler alone also pages a node admitted to retire as withdrawn whose projections ask it to withdraw its protection (listProjectionPage, see Node Retirement), until its credentials are revoked; no other producer sees it. A member is assignable when its architecture states a container platform (x86_64 selects linux/amd64 and aarch64 selects linux/arm64); a member whose architecture states neither is listed with no platform and is not assignable.

Membership is read inside the caller's controller transaction: credentials page eight at a time and the cursor is the credential cursor, so a page may hold fewer members than credentials. Reading a node's current session joins that node's credential and document to the transaction. Each page reads the phase of each of its clusters once and fences the clusters Pallet owns, so a phase that moves before the caller commits takes the listing with it. A cluster Cloudly holds no phase for refuses the listing with cluster-runtime-missing; that refusal is controller-wide and not per node, so while one cluster states no phase nothing is produced for any node, and the cluster is named in the log once per process (cluster <id> has no runtime phase; network production is refused). A credential that outlived the node it was issued for names no cluster, so it names no member instead of refusing the listing, and retiring it belongs to the node's lifecycle. A listed node stays a hint that its producer re-fences; membership is never authority.

Address Plan

cloudly_runtime_network_address_plan holds one IRuntimeNetworkAddressPlan (id: 'address-plan'): the prefixes, pools, resolvers (at most 16), platformEndpoints and vpn an administrator gives the runtime network, under a revision compare-and-set. It is the one input every protected authority is composed from; startup writes no plan, and until an administrator sets one the runtime network is unconfigured.

setRuntimeNetworkAddressPlan({ identity, expectedRevision, plan }) and getRuntimeNetworkAddressPlan({ identity }) are admin-guarded. expectedRevision is the revision the caller read, or null when no plan exists; anything else is refused as conflict. The contract validates the plan (snapshotRuntimeNetworkAddressPlan) by building the authority its own sets describe, so no plan can compose an authority the protected-authority contract refuses, and a plan whose two VPN prefixes — vpn.controlPrefix for the tunnel's control addresses and vpn.hubPrefix for the underlay every cluster hub binds inside — are not reserved inside the protected space it states is already refused there. The hub endpoints themselves are no administrator's input: each cluster's relay reports the one it bound and the composition turns it into platform endpoints. The contract also decides the change (admitRuntimeNetworkAddressPlanChange): leases and the managed VPN outlive a plan revision, so every pool stays exactly as it was, the protected space only grows, and the VPN never moves. Equal content is not a write: the stored revision stands and the document is left alone.

Every refusal is a TypedResponseError carrying its name, and keeps the error that caused it for Cloudly's own diagnostics:

• conflict — the caller stated a revision that is not the stored one.
• plan-invalid — the contract refused the plan's structure without naming a rule of its own (overlapping pools, an endpoint outside the protected prefixes, a control prefix longer than /30, a duplicate endpoint tuple).
• plan-unsorted — a list is out of its canonical order: prefixes strictly ascending by prefix text, pools, resolvers and platformEndpoints by id, in string order. The refusal names the list as its reference (Runtime network address plan rejected: plan-unsorted: pools.), and NetworkAddressPlanError.reference carries it.
• plan-vpn-immutable, plan-pool-removed, plan-prefix-shrunk — the contract's three change refusals.
• plan-vpn-foreign — the plan states a managed VPN other than the one this controller already runs. The configuration is written once and never moves, so a plan that states a different control or hub prefix would leave the running VPN outside the address space it must sit in.
• plan-resolver-referenced — a resolver the change removes or alters is still named by a router selection (by id) or by a private network's external DNS upstreams (by address and port).
• plan-endpoint-referenced — a platform endpoint the change removes or alters is still named by a router selection or an authorized service policy.

The plan is also where Cloudly's dcrouter gateway client takes its DNS record allowance (allowedDnsRecordAddresses) from: vpn.hubPrefix and every platform endpoint address as a host prefix, which are the two places a cluster relay binds its listener. A committed change re-derives it, and a changed allowance makes the held gateway credential's policy stale: the running runtime credential stops serving and the enrollment turns admin-bootstrap-required until an administrator bootstraps the gateway client again, which is how dcrouter learns the new allowance (see Gateway Credential Bootstrap). A plan whose allowance would exceed the gateway contract's 64 prefixes is refused plan-invalid.

mutatePrivateNetwork states the upstream rule from the other side: a private network may only forward to an upstream the plan states as a resolver, matched by address and port, and anything else is refused as upstream-unprotected. A network that denies external resolution names no upstream and reads no plan. The plan document is fenced inside the private-network transaction, so a plan write that removes the resolver a mutation relies on conflicts instead of stranding it. The private network manager therefore starts after the runtime manager, which owns the plan store.

Protection Production

The protection producer composes the one protected authority this controller publishes, id: 'runtime-network-protected', from the address plan and the nodes RuntimeNodeScope states. The plan is the whole address space; the producer adds only what a plan cannot state: the authority's identity, its revision chain, and its egress authorities, which are the current ones plus every node now in scope. Each cluster's selected hub is added as a UDP platform endpoint by the contract's own composeRuntimeNetworkProtectedAuthority, so the authority a plan describes is composed in exactly one place.

One pass reports one outcome:

• unconfigured — no plan. Nothing is read further and nothing is written.
• no-members — a plan, but no node is a member yet; a protected authority grants egress to at least one node, so none can be composed.
• unchanged — composing the plan and the current egress at the current point of the chain reproduces the current authority byte for byte, digest included.
• protection-propagating — the named current egress nodes hold no projection head, acknowledged or still pending, citing the current authority.
• staged — the successor was staged.

A node that goes offline, loses its credential or leaves a Pallet-phase cluster keeps its grant: an owner leaves the authority only behind its own fence, which its retirement records (see Node Retirement), and the reservation ledger enforces the same. The pass drops owners whose fences are recorded against exactly the current authority, at most sixteen per step: the admitted withdrawn group whole, then lost owners in owner order, provided those fences agree and clear exactly the owners the step keeps less the lost owners it holds as exemptions; otherwise it drops nobody and looks again on the next pass. A node the scope lists joins only while it has no fence record, read by its id, so an owner a step fenced never joins again, and the propagating gate does not wait for a retiring owner, which nothing compiles again. A join only adds, so its successor continues the current authority (continuesRuntimeNetworkProtection) and every managed VPN member keeps its key pair and hub session; a step that removes or changes anything — the first authority, a moved hub, a changed pool — withdraws every member. Every join of one pass still goes into one step, because each step is a barrier every egress owner reports a receipt for. A successor is staged only once every current egress node's head cites the current authority, so the fleet is never asked to apply two generations at once, and a projection lags the current authority by at most one step.

The pass plans first and only a real change enters the managed VPN lane (advanceProtection), which builds the candidate again there and withdraws the members only when it does not continue the current authority, so an unchanged fleet and a join both keep their tunnels. Planning is a probe: it reads one snapshot, writes no fence and commits nothing, so a steady fleet costs no write at all. The staging that acts on a differing plan builds it again in its own transaction and takes every fence its reads need — the controller record, the protected authority and the phase of each cluster it pages, and every current egress node and its projection head. The authority is composed from the plan's own sets plus one UDP platform endpoint per cluster hub, read inside the deciding transaction: a hub that moves is therefore a successor of its own, and a hub that only re-keys is not, because a public key is not part of the authority. After a pass that states a current protected authority the managed VPN is configured from the plan, once per process: a replay is safe, and the configuration never moves afterwards, so a later pass has nothing left to configure.

Two bounds come with that. The contract admits at most 256 egress authorities, but a differing pass rewrites every current egress node's projection document inside one transaction, so the fleet size this gate actually carries is far below that number; a gate that pages those reads, or that compares the stored revision instead of replacing each document, is what would raise it. And a grant outlives the node that holds it: both the gate and the staging fence require that node's document, so deleting a node the current protected authority grants egress to would wedge every later differing pass. Node deletion therefore refuses it — getNodeDeletionImpact states the blocker node holds a runtime network egress grant beside the heartbeat and deployment blockers, and deleteNodeById refuses on it — until the node's retirement fenced it and a protection step dropped it.

LevelTriggeredSweep schedules the pass on the same period, hint delay and floor as network production, and every committed change hints all three producers. produceNetworkProtection() and inspectNetworkProtection(), for administration and tests, join the pass in flight or run one at once.

Node Network Production

The node network producer gives every node RuntimeNodeScope states the two things its own traffic needs before anything can leave it: one reserved transit handoff, and the selection the network compiler reads. Both derive from durable state alone. The node's router incarnation, stored in cloudly_runtime_router_incarnations as {id: <nodeId>, incarnation}, is written at 1 inside the transaction that reserves the first handoff, and the handoff's id and the router incarnation its leases cite are sha256 over the domain, the node and that counter — so a controller that restarts between reserving and staging recomputes the same id, and the reservation replays instead of allocating a second transit range.

One pass over one node reports one outcome:

• unconfigured — no address plan, or no protected authority yet. Nothing else is read and nothing is written.
• egress-pending — the current protected authority does not grant this node egress.
• protection-receipts-pending — the named egress owners hold no current protection receipt or reporter session, so no reservation on this controller is eligible. The owners are read before the attempt rather than caught from it, because a reservation refuses an ineligible fleet without saying who owes what.
• transit-pool-missing — the address space states no transit pool to allocate from.
• handoff-quarantined — this incarnation's handoff was quarantined; only a reincarnation gives the router a fresh one.
• unchanged — the staged selection already names the live handoff and the current router destinations, so the pass leaves the node's own documents exactly as they stand.
• staged — the selection was written at its next revision.

The selection names every resolver of the current protected authority and the hub of the node's own cluster. The resolvers come from the authority rather than from the plan they came from, because a selection may only name what the current authority states: a plan revision that adds a resolver before the protection producer has staged its successor would otherwise be refused. The hub's endpoint id is derived by the contract from the hub the cluster's relay advertised (runtimeNetworkVpnHubEndpoints), so the selection, the composition that sealed the authority and the credential binder all name it identically — without it in the selection no node is ever admitted as a managed VPN member. A node may name only its own cluster's hub, because every inter-node packet stays inside the cluster; a cluster whose relay advertised no hub, or one the plan's hubPrefix does not cover, leaves its nodes with their handoff and resolvers and no VPN membership at all. A staging pass fences the address plan, so a plan write that drops a resolver the selection names conflicts with it instead of stranding it.

getRuntimeNetworkNodeReadiness answers whether one node's runtime network is ready for workloads and every condition it still misses: first liveness, node-observation-stale with lastObservedAt when no accepted report within clusterNodeObservationFreshnessMs (180 s) shows the node online, then in the order the producer meets them: egress-pending, protection-receipts-pending with its owners, handoff-pending, selection-pending, projection-pending and vpn-member-pending. Ready is exactly "nothing is missing": the node is observed running, is in egress, holds a reserved handoff, its selection names that handoff, its acknowledged projection head carries it, and this controller process holds a managed VPN member for it. Readiness is a readout, never a step of the sweep — reading the acknowledged head fences the projection row, and a pass that asked it for every node would rewrite every projection on every period — and the receipts of the fleet are read only while the node holds no handoff, because current receipts gate the reservation and nothing else. Membership is process state, so a Cloudly restart withdraws readiness until each node re-fetches its managed VPN credential.

reincarnateRuntimeNetworkRouter is the one lever over a node's router: it advances the incarnation against the one the caller read, and the next pass reserves the successor handoff and stages the selection naming it. It invalidates what the previous handoff carried — the next compilation quarantines it and retires it in the node's projection, the managed VPN member falls with that projection, and readiness returns to handoff-pending until the pass has run. Its refusals reach the administrator by name: conflict for an expectation that is not the stored incarnation, router-incarnation-missing for a node that never held a handoff, and router-handoff-in-use while a workload lease still cites the current handoff, because the next compilation could not quarantine a handoff a lease holds and that node would never compile again.

Three bounds come with the producer. A node retains at most 256 handoff leases, so that is how often its router may be reincarnated for the lifetime of its document, and no path releases one before an independent fence exists. Every node's selection names every resolver the plan states, so a plan write that removes or changes a resolver is refused as plan-resolver-referenced while any node holds a selection; removing one is a maintenance path, not a plan edit. A steady pass writes nothing: each node's pass runs first as a probe, and only a pass that would reserve a handoff or stage a selection runs again as a transaction, which fences the controller record, the protected authority and the node as before.

LevelTriggeredSweep schedules this pass on the same timing as the other two producers. Every committed change hints it, and a protection pass that staged a successor hints it as well, because that successor is what a node's first reservation waits for. In the other direction only a staged selection is a committed change, and network production is the one producer waiting for it, so that is the only hint a pass sends: a pass that changed nothing never asks the fleet to sweep again. produceNodeNetworks() and inspectNodeNetworkProduction() join the pass in flight or run one at once.

Service Runtime Spec

IServiceRuntimeSpec is what Cloudly runs for one service on one Pallet cluster: the cluster, the placement (every-node or a sorted list of pinned node ids), the replica count per node, a hold, an execution profile and a network mode. It is stored complete in cloudly_service_runtime_specs, one document per service/cluster pair, and never defaulted at read time. Its private persisted id derives from that pair, while its public id remains the service id. It lives beside the service document because a service update merges caller data and because a hold a cutover writes must not race an operator's edit of the same document. Its revision is the compare-and-swap: setServiceRuntimeSpec takes expectedRevision, where null states that the caller read no spec for that exact cluster, and every accepted write advances that pair's revision by one and replaces every field — nothing is carried over from the revision it replaces. A revision read from another cluster can never satisfy the compare-and-swap.

A spec write runs under the service's lifecycle lease, which serializes it against a service update or deletion across Cloudly processes, and inside that lease one transaction fences the cluster's runtime phase, so a phase that moves before the commit takes the write with it. Refusals reach the caller by name: service-not-found, service-deleting, service-organization-missing, conflict, spec-node-not-in-cluster for a pinned node that belongs to another cluster, spec-readiness-target-undeclared for a TCP or HTTP probe naming a port the service does not declare, cluster-runtime-phase-refused for a cluster still in swarm, whose placement is Coreflow's, and cluster-runtime-missing for a cluster id Cloudly holds no phase for at all. A spec is accepted in phases draining, switching and pallet. A numeric expectedRevision against an absent service/cluster pair is a conflict, as is null against that pair when it already exists. getServiceRuntimeSpecForCluster reads one exact pair and getServiceRuntimeSpecs pages every pair in cluster order with its matching status. The released getServiceRuntimeSpec scalar answers zero or one target and refuses two or more as service-runtime-spec-ambiguous; it never chooses one. The assignment producer alone writes the pair-local verdict in cloudly_service_runtime_status, so it reads as null until that target is decided. removeServiceRuntimeSpecForCluster derives and fences the current phase, revision, hold, terminal slot evidence and release inside its transaction; it removes only that held pair. Deleting a service removes every target and verdict inside its deletion.

createService writes the default spec only where the cluster leaves no choice: the one cluster Cloudly holds a phase for is a cluster Pallet owns (switching or pallet). With no cluster, with more than one, or with any Coreflow-owned cluster in the fleet, a new service starts with no spec and the operator states one with setServiceRuntimeSpec, or the cutover writes it. That write fences the cluster's phase exactly as an administrator's does, so a cluster that leaves the Pallet-owned phases while the service is being created refuses the creation by name rather than binding the service to a cluster Coreflow took back. The default states placement: every-node, hold: false, network: isolated, cpuMillis: null, runAs: null (the image's own OCI user), readonlyRootfs: false, and:

• replicasPerNode is the service's scaleFactor when that is an integer within the contract's bound of 64, and 1 otherwise — a Swarm scale factor outside the Pallet bound states nothing about a Pallet replica count.
• memoryBytes is resources.memorySizeLimitMB in bytes, or 512 MiB when the service states no limit. A stated limit the runtime contract cannot express refuses the creation as service-runtime-spec-memory-invalid; nothing is substituted silently.
• readiness is a process probe, whatever target ports the service declares: the default network is isolated, and an isolated workload has no address the node can dial, so a TCP or HTTP probe could never pass and Pallet refuses one (unsupported-readiness). An administrator who attaches the service states a port probe in the same spec write. Probe timing is 1s initial delay, 30s startup grace, 5s interval, 2s timeout, two successes and three failures.

An attached network states publicEgress and the platform endpoints a workload may reach. Cloudly supplies no default for either: the contract requires an attached spec to state publicEgress explicitly, and a write that omits it is refused rather than completed. Admission requires an explicit service network policy that matches both fields — the service's egress, written by an administrator with setServiceNetworkEgress — the selected handoff for the placed node, its current protected egress authority, and a free workload /30. The allocation owner chooses the lowest-id current workload pool with capacity; the service spec cannot select a pool. Reservation and the exact assignment admission share one controller transaction. A run or stop admission requests network compilation, which projects or quarantines the retained lease. Missing policy, mismatched policy, missing workload pools and exhausted workload pools have distinct service refusal reasons.

Once a service has any spec, those specs own its placement and replica counts: a generic updateService that states placement or scaleFactor is refused as service-runtime-spec-owns-placement, whether or not it repeats the stored value. An update that states neither reads no spec at all. The ports stay the service document's, but a spec's probe names one of them, so an update stating targetPorts or ports that would leave the stored probe naming an undeclared port is refused as spec-readiness-target-undeclared — the same answer the spec write gives.

Service Runtime Production

The assignment producer is the writer that makes every runtime target of a service true. One pass covers all of that service's targets and has exactly three kinds of step: one transaction that reads everything and writes nothing, the build that resolves what the pinned release executes, and one transaction per slot that actually acts. A service whose desire already stands costs the first of those and nothing else.

Desire. A spec desires nothing while it holds the service, while the service is being deleted, or while its cluster is not in phase pallet — switching prepares the network, it does not run workloads. Otherwise the desire is selectReplicaPlacement over the nodes RuntimeNodeScope calls assignable: every-node takes every node of the spec's cluster whose reported architecture states a container platform, and nodes takes exactly the pinned ones that are assignable. Each node carries replicasPerNode replicas named <nodeId>.<index> from zero, so a replica never moves node: an offline node keeps its replicas, and only retirement, credential revocation or a spec change moves them. A pinned node that is momentarily not a member leaves its replica unplaced and the service converging; only a desire that is empty while the spec wants at least one replica is refused, as no-eligible-node.

Object-storage environment. A target runs with the service's own environment and the names of every enabled object-storage binding of the service that is not tombstoned. Cloudly places each binding first, by the rule the relay's placements follow: on the node its legacy adoption names, otherwise on the one node the spec pins. A binding that lands on no node of the target's cluster — a spec on every node or on several, or a node that is not the cluster's — makes the target wait as bindings-pending. The rest is the contract's composeObjectStorageServiceEnvironment over each binding and its node's corestoreObjectStorage (see Object-storage endpoint): the nine endpoint names, and the eleven retention names once retention is proven. A binding that declares retention without evidence waits as objectstorage-retention-pending; a node that states no valid endpoint is refused objectstorage-endpoint-missing, a binding the contract cannot compose from objectstorage-binding-invalid, and a name the service or another binding already sets objectstorage-environment-conflict. An enabled database binding placed on a node of the cluster that states no valid corestoreDatabase (see Database endpoint) is refused database-endpoint-missing: it is never granted, so its credential would never arrive. A binding whose node states a valid endpoint the target's network does not select is refused database-endpoint-unselected or objectstorage-endpoint-unselected, by the contract's platform.findUnselectedCorestoreEndpoints: a workload reaches a Corestore endpoint only when its attached network selects a platform endpoint with that address, tcp and that port (the scheme's default for an object-storage origin that states none), and an isolated network selects none, so the workload is refused rather than started against a connection that stays dark. A refusal outranks every wait, and among waits the lowest binding id answers. A target that waits or is refused here decides nothing: no run and no stop, so a replica already running keeps running while its target reports why. The composed environment replaces the service's own in the desired key and in the built config, where the image's defaults stay beneath it.

Key. desiredAttemptKey is sha256 over the target cluster id, rollout id and generation, the pinned image digest, the spec revision, the composed environment (so a moved object-storage endpoint is a drift), the node's platform and the network mode — database values only, so a commit can recompute it under its own fences. desiredAttemptId binds that key to the removal the attempt succeeds, and the assignment id derives from the attempt id, which is what makes drift detectable without storing the key: recomputing the attempt id with the incumbent's own stored predecessor yields the incumbent's attempt id exactly while the desire has not changed. The network lease is deliberately excluded, because its own id derives from the assignment id.

Transition. decideSlotTransition is pure and exhaustive, and every case is named:

case	the slot	the step
admit-run-first	no slot, and a desire	run, generation 1, no predecessor
admit-run-successor	removed with both receipts, and a desire	run, generation 1, citing the removal
settled	removed with both receipts, no desire	nothing
running	at the desired attempt, not failed	nothing
drift	at another attempt than the desire	stop, generation 2
undesired	running with no desire	stop, generation 2
failed-elapsed	failed, backoff elapsed	stop, generation 2
failed-waiting	failed, backoff pending	wait, failure-backoff
stop-receipt-pending	stopped without its receipt	wait
remove	stopped with its receipt	remove, generation 3
remove-receipt-pending	removed without its second receipt	wait
replacement-in-flight	drifted while another slot is replacing	wait

The failure backoff is min(10 s · 2^n, 10 min) measured from observationAcceptedAt, Cloudly's own clock when it accepted the observation, with n the number of consecutive earlier attempts of that slot that ran this exact desire — the restarts the producer itself already made. Those ancestors are read from the attempt chain, at most eight back, and the walk stops at the first one that ran something else.

A service replaces one slot at a time: while any slot sits between its stop and a new run that has been observed ready, another slot's drift waits. The rule binds drift only. An empty desire — a hold, a deletion, a cluster that stopped running the service — stops every slot at once, because nothing will become ready and a cutover depends on the whole service draining.

Build. Every step that has a desire behind it is built before anything is committed, and a refusal is local to the target whose desire could not be built: stopping a replica because its desire changed or failed is the first step of replacing it, and a replacement this runtime cannot build must not begin. That is what makes a refusal a verdict about new work and never the reason a running slot was retired. The steps that carry no desire are the other half of the same rule and a refusal never holds them back: the stop of a replica the service no longer wants and the removal that follows any stop are committed while the build is refused, so a scale-down, a hold or a deletion finishes even while the registry cannot answer. resolveRuntimeImageMaterial answers what the pinned release executes — registry I/O, so it runs outside every transaction and once per pass — and buildRuntimeConfig seals the rest from values alone: the image's environment with the composed environment over it, the image's working directory or /, the profile's resource and identity policy, and the service's declared ports, where udp stays udp and http, https, ssh and tcp become tcp. The config id is config-<sha256> over its own content at generation 1, so equal content is the same document. A service declaring more target ports than a config carries has no refusal name in the contract yet, so its pass fails with that count and the sweep retains it by service id, rather than borrowing a name that would say something untrue; the ports-too-many refusal arrives with the next interfaces minor.

Commit. Each acting slot gets one transaction that decides the same slot again from freshly read inputs and refuses conflict unless it reaches the same case and, for a run, the same desired key: a spec, a service, a rollout, a phase or a membership that moved under the pass invalidates the step rather than committing it. Only then does it fence the cluster phase, require the node's runtime network to be ready — else the service waits as node-network-pending —, require the desired secret manifest to carry the published credential of every enabled Corestore binding of the service — else it waits as bindings-pending, so a first start never runs without its database credential, or as legacy-adoption-pending while the relay reports such a binding refused until its legacy resource is adopted (see Corestore Legacy Resource Adoption) — and hand the sealed revision to admit, which fences the service document's whole data, the node and the slot. Readiness is asked on a committing pass alone, because reading the acknowledged projection head rewrites that node's projection row.

An attached run seals its reserved network lease in the admission transaction. A run with a current resolved secret manifest seals its independently hashed secret authority and retains every pinned version in that same transaction. The node must have resolved each authority it needs. Cloudly refuses a missing or invalid secret plan as secrets-unsupported; a node without the secrets capability is refused as node-authority-unsupported, and so is a node without storage for a service that declares volumes.

Local storage claims. A declared volume runs as a node-local local storage claim (IRuntimeLocalStorageClaim), one per volume and node, sealed into the run's workload.storage. A claim is one node-local directory mounted ReadWriteOnce, so a volume whose driver is neither absent nor local is refused volume-driver-unsupported, driver options are volume-options-unsupported, a volume shared by source, one mount path declared twice or a spec with more than one replica per node is volume-shared-unsupported, a placement that does not pin exactly one node is volume-placement-unsupported, and more volumes than an assignment seals (64) is volumes-unsupported. A volume's name is Coreflow's Docker name and backup its snapshot scheduling; neither names anything on a Pallet node.

The claim volumeId is volume- followed by the SHA-256 of ["cloudly-service-volume", mountPath]: a service names its volumes by mount path, so a volume whose mount path changes is a new volume with a new claim, and the old claim is retained on its node. The claim id is derived from the controller's kind and id (not its epoch), the organization, cluster, node, runtime namespace, service and volumeId, so it is stable across every generation. The first generation takes its initialization and initialOwnership from the volume's stored initialization through deriveServiceVolumeClaimInitialization under the spec's runAs: a volume without one is refused volume-initialization-unknown, and a run-as ownership under runAs: null volume-ownership-unknown, never created empty. Every later generation keeps what the first fixed, whatever the initialization or runAs says by then; only the mount (readOnly) and the controller move, and a mount change replaces the replica. The claims are prepared, sealed into the run, bound with bindRuntimeLocalStorageClaimToAssignment and stored in the admission transaction, so no claim is stored for a refused run and no run references an unstored claim. Cloudly never issues a purge: the reclaim policy is retain.

Every claim is replayed to its node over the node's session (applyRuntimeLocalStorageClaim) on each delivery pass, before the assignments of that pass, and the answer for the claim's current generation is recorded in cloudly_runtime_local_storage_claims. While a claim of a target answers import-required, the target waits as storage-import-pending: its run is admitted and waits on the node until the operator's offline import lands, and the next replay that answers ready moves the verdict on.

An administrator reads the claims of one cluster, optionally of one node, with getRuntimeLocalStorageClaims { identity; clusterId; nodeId?; cursor?; limit? }: at most 200 summaries per page (runtimeLocalStorageClaimContract.maximumSummaryPageSize, and fewer when limit asks), in strictly ascending claimId, each the contract's summarizeRuntimeLocalStorageClaim of the stored claim — claim id, generation, cluster, node, service, volume id, mount path, source kind, initialization, lifecycle and the node's state (null until the node acknowledged that generation). No source beyond its kind, and so no credential reference, leaves Cloudly through it. nextCursor is the last claim id of a page that has a successor. The claimId is the name an operator places an import volume's bytes under (import/<claimId>) before pallet-control storage-import.

A Pallet node enrolls a public X25519 recipient through a one-time possession challenge on its current authenticated runtime session. Its private key stays in process memory, so the recipient is active for every session the same node process registers (the registration's nodeInstanceNonce): a reconnect, a credential rotation or a Cloudly restart keeps it, and a restarted node process holds no key and enrolls a new generation before Cloudly issues it fresh material. For an admitted secret-bearing run, Cloudly checks the current assignment, node session, recipient generation, accepted manifest and indefinite version holds before encrypting the pinned values to that recipient. The response is bound to the full run assignment and carries only sealed envelopes. An environment-delivered entry also requires the currently approved WorkloadInit artifact; Cloudly never places a secret value in runtime config, an assignment, or a log. A refused material request is answered with its fixed text and the payload { code: 'runtime-session-report-refused', reason }, where reason names the check that refused it in words only (for example workloadinit-approval-unavailable, secret-manifest-not-accepted or secret-retention-absent; the vocabulary is TRuntimeRefusalReason), and Cloudly logs the node, the reason and the bounded cause chain once per change of reason. The holds end only when the same run's joined native remove receipt is accepted, in the receipt transaction. Material issuance stops when a manifest leaves the current desired or previous-acceptance window, even while an old run remains retained for safe cleanup.

Verdict. cloudly_service_runtime_status holds one verdict per service/cluster target about that target's spec revision, and it is written only when it changed, so changedAt says when the verdict last moved rather than when it was last confirmed. converged means every desired replica runs its desired attempt and has been observed ready and nothing is pending; converging is everything else that is neither waiting nor refused; waiting names node-network-pending, conflict, failure-backoff, storage-import-pending, bindings-pending, legacy-adoption-pending or objectstorage-retention-pending; refused names cluster-runtime-missing, organization-missing, image-plan-missing, platform-unsupported, secrets-unsupported, volumes-unsupported, port-protocol-unsupported, readiness-target-undeclared, environment-too-large, no-eligible-node, replica-limit, node-authority-unsupported, one of the four volume-*-unsupported names above, volume-initialization-unknown, volume-ownership-unknown, objectstorage-endpoint-missing, objectstorage-binding-invalid, objectstorage-environment-conflict, objectstorage-endpoint-unselected, database-endpoint-missing, database-endpoint-unselected, cluster-relay-missing or cluster-ingress-environment-conflict (see Ingress environment). image-plan-missing also names a retained rollout plan the current runtime cannot execute: one that breaks validateImmutableImageDeploymentPlan, such as a Swarm-era plan without a sealed containerInvocation. Only a new promotion replaces it, and its existing-service reserve preflight answers GO with the RECOVERY_REDEPLOY_OVER_FAILED_RUNTIME warning while the current runtime cannot be attested and no deployment operation is in flight. A service with no spec and no slots is produced as unconfigured and writes no row at all.

Volume initialization. IServiceVolume.initialization states how a volume's first claim starts (empty with an owner and mode, or import of an existing Docker volume's bytes). setServiceVolumeInitialization { serviceId, mountPath, expected, initialization } is its only write: expected is the value the caller read (null for none), the stored value again is answered unchanged, and the refusals are IServiceVolumeInitializationErrorData: volume-not-found, conflict, volume-initialization-invalid, and volume-initialization-bound while a bound claim of the volume exists on any node. The claim read and the write share one transaction with the service document that the claim-issuing admission fences, so the two cannot pass each other. createService and updateService refuse a volume that states an initialization (SERVICE_VOLUME_INITIALIZATION_SERVER_MANAGED) unless it is exactly the value already stored for the same mount path — what a read-modify-write round trip sends back, in any key order — and keep every stored initialization with the volume of the same mount path; a removed volume takes its initialization with it, and two volumes at one mount path are refused SERVICE_WRITE_INPUT_INVALID.

The sweep is the fourth LevelTriggeredSweep, on a 15-second period rather than the network producers' 30, because it is what an operator waits for. It pages target rows but emits each unique service once, then visits first and then every service that still holds slots, which is how a service whose spec was removed is still visited by the pass that has to settle its slots; those pages seek in service order, so a service whose slots straddle a page boundary is still a candidate exactly once per sweep. Every committed runtime change hints it through afterRuntimeChange, and a spec write hints it directly through requestServiceRuntimeSweep(). produceServiceRuntimes() and inspectServiceRuntimeProduction() join the pass in flight or run one at once, and listServiceSlots(serviceId) reads one service's slots without taking a single fence, bounded at one slot past the replica limit, so a service holding more of them is refused as replica-limit rather than read without end.

Two bounds come with it. A pass reads the node scope for every service it visits, so an idle fleet's read rate scales with services times nodes. The decision and the verdict comparison are probes that write nothing; each commit fences the controller record and the phase of every cluster it took members from, exactly as the network producers do. And until the two-phase deletion lands, a service that ever held a slot cannot be deleted: deleteServiceById refuses service-runtime-active while any slot row exists, before it takes the lifecycle lease, because taking that lease marks a service deleting for good and a refusal inside it would strand the service.

Rollout evidence. On a rollout Pallet executes — every cluster its plan targets is in phase pallet (isPalletExecutedRollout) — the assignment producer is the rollout's reporter, because it alone knows which runs belong to it: a replica counts only while its incumbent runs exactly the current desire (runsDesiredAttempt), whose key carries the rollout id, its generation and the image digest, so a run of an earlier rollout never vouches for a later one. Each pass evaluates one reporter per target cluster (evaluateRuntimeRolloutReporter, reporter id pallet/<clusterId>) over the plan's target nodes at the plan's replicas per node: a run admitted or observed but not ready is starting, a ready run is healthy, a failed run is failed and a stopped one unhealthy; a run is verified when the image digest its node observed and the digest its assignment was sealed to are both the requested digest, and either one naming another digest is a mismatch. The shortfall precedence (decideImmutableRolloutShortfall), the aggregation and the latch on a terminal failure are the ones Coreflow reports go through, so the rollout succeeds, fails or keeps rolling out by the same rules and the startup grace applies alike. The pass writes imageRolloutStatus only when its verdict moved, compare-and-set on the rollout identity, the status it read and the route fence, and never replaces a status that describes another rollout or one it has no reporter for: a pass whose desired replicas stand on none of the plan's target nodes — a plan scoped before its cluster moved to Pallet names Swarm hostnames — leaves the last verdict standing. getServiceDeploymentStatus reads such a rollout's runtime tasks from readRolloutRuntimeTasks, a fresh read-only plan of the same producer that states every observed run of exactly that rollout, instead of asking Coreflow; a rollout with any target cluster still switching, draining or swarm stays Coreflow's.

Service Runtime Views

What the producer admitted is served back through two reads that write nothing, take no fence and derive nothing. getServiceRuntimeAssignments({ serviceId }) answers one service's replica slots in replica order. getClusterRuntimeAssignments({ clusterId, cursor, limit }) answers one cluster's slots a page at a time in slot order; limit runs from 1 to serviceRuntimeContract.maximumAssignmentPageSize (256) and nextCursor is the slot id the page ended at. A cluster carries every node's slots of every service, so every slot states the cluster it runs on — taken at admission from the node that admission already fences — and the page seeks that cluster's slots alone: a page holds exactly limit rows unless the cluster holds fewer.

Both answer the same row, projected in serviceruntimeview.schema.ts from the stored rows alone: serviceId, replicaId, nodeId, the current assignment reference, the disposition, terminalReceiptCount, the assignment record's own observation (phase, ready, observedAt, sequence, and the cause a node stated for a failed run) and delivery. Delivery is delivered once the node's own evidence cites the exact revision that stands — an observation that names it, or the terminal receipt that does, since receipts join in order — and undelivered until then, because Cloudly keeps no acknowledgement journal: a revision the node admitted but has not reported on reads as undelivered.

The service read passes the same inspectable-service guard as getServiceRuntimeSpec, so an administrator or the service's own organization may read it and nobody else. The cluster read is an administrator request, like getClusterRuntime. A page size or a cursor the contract does not admit is refused by name, as assignment-page-limit or assignment-cursor-invalid.

The dashboard renders those rows as a runtime tab on the service, node and cluster details; a node's tab is its cluster's page filtered to that node. A failed run's cause is shown in its node's words — the image pull's refusal reason and gRPC status, the exit code, "runtime absent" or "network epoch ended" — and a failed run whose node named no cause shows none. A cause never changes what Cloudly does: every failed run is replaced on the same backoff. A cluster tab walks at most eight pages of 128 rows, and a reading that ends with the cursor still open says so in its heading rather than showing the rows it kept as if they were the whole cluster. It refreshes on pushServiceRuntimeChanged, which carries only a service id — the UI reads the views again — and travels on the serviceRuntime UI live channel, so a UI holding none of those views open is never woken by it. The push is emitted from exactly one place: the service runtime pass announces when what an administrator would read moved. Every committed runtime change already hints that pass, so its own admissions, an accepted observation, a joined receipt and a spec write are all covered by it, and a repeated report or a steady service announces nothing at all.

Service Deletion and Node Retirement

deleteServiceById refuses a service that the current ingress designation of any cluster, of either kind, names as SERVICE_IS_CLUSTER_INGRESS, naming the clusters, before anything is touched: deleting it would revoke the bearer the designation still names and leave the cluster with an ingress that cannot register. Move the designation to another service first. The check runs before the deletion takes its lease, so a designation that commits in between can still leave a cluster designating a deleted service; re-designating the cluster recovers it (see the recovery of a retired bearer under the Pallet-born ingress designation).

The deletion's last step deletes the service row together with the service's secret target state, its secret revision authority, in one transaction and only after every secret the service owned was revoked. A service created under the same id later therefore starts at secret revision 0 like any new one, and recovers a revoked secret of its predecessor through revocation recovery; the revoked secrets stay as audit and purge history.

deleteServiceById is the first of two phases. It takes the service lifecycle lease with delete intent, which marks the service deleting durably — every later admission is refused by the service's own runtime fence, and only settlement is still accepted — and then removes everything Cloudly owns that nothing running depends on: the external gateway routes and the DNS entries, web push, mail and CoreMail bindings, deployments, platform bindings and their Corestore delivery, backups, the registry repository, OIDC, the service-owned image and the deployment claims. A retired Corestore binding is delivered strictly to the clusters its service's immutable deployment plan targets, the only clusters Cloudly ever granted it for; a malformed plan, or one that names no cluster, refuses the deletion as CORESTORE_CLEANUP_TARGET_AUTHORITY_REQUIRED. A service without a plan was never granted a binding, so no Corestore holds one of it and its bindings are retired with the ordinary config push. It can still own a legacy Corestore resource, though, and nothing retires that, so each of its live Corestore bindings must first show that its capability passed to a successor (see Corestore Legacy Resource Adoption); otherwise the deletion is refused under the same code. Coreflow's reconciliation of the service is quiesced whenever any configured cluster remains Coreflow-owned, including a legacy cluster for which that service has no Pallet target row. It skips the fence only when every current cluster is Pallet-owned. Each phase asks that once, while the spec that answers it still exists, and holds the quiesce for the whole phase, so the row deletion inside the second one never asks again for itself.

What Pallet runs is retired by the service runtime pass, which desires nothing for a deleting service, so its slots walk run, stop and removal against the node's own receipts. The second phase belongs to the retirement finalizer, a level-triggered sweep over every service a deletion left deleting. Every pass runs the first phase again before it reads anything: a request that failed midway through it left the service deleting with routes, DNS or bindings still standing, and every step of that phase removes what exists, so the re-run is what keeps a deletion from finishing over its own leftovers. Then it reads what the runtime still owes and names the wait it is in:

• slot-settlement-pending — a replica slot has not reached its removal with both terminal receipts joined;
• lease-quarantine-pending — a workload lease of the service is still reserved, which the network compiler quarantines when it sees the stop intent;
• projection-acknowledgement-pending — a node the service reached has no acknowledged projection head, or the head it holds still carries one of the service's endpoints. The nodes asked are the ones its slots stand on together with every node a workload lease of the service names, quarantined leases included, so a replica that moved keeps the node it left in the wait until that node says it dropped what it carried.

Only once nothing is owed does the finalizer take a fresh delete lease and finish the service: it revokes the service-owned secrets and the platform credential, deletes the network policy and the private-network membership that authorized its workloads, releases the replica slots together with the attempts and the runtime configs only they cited, removes the runtime spec and verdict, and runs the fenced service.delete() against the runtime generation the release just advanced. A slot release refuses a slot the node has not removed, so a half-finished deletion is impossible; a service nothing ever ran for needs no second pass and is finished by the request that started it. The sweep is level-triggered, so a deletion interrupted by a failure, a restart or a slow node is finished by the next pass rather than by a retry somebody has to issue.

deleteNodeById refuses a node that still holds runtime state, and getNodeDeletionImpact names each blocker: the replica slots that run on it, a reserved workload lease or transit handoff, each active spark or pallet credential, the egress grant the protected authority holds for it — deleting the node it names would refuse every later protection pass — and a retirement that has not finished. The node document, its removal from every cluster snapshot and its retirement row are written in one node-fenced transaction, so an enrollment, a heartbeat or a runtime write that lands meanwhile conflicts instead of leaving a relay dialing a node that is gone. An enrolled node is retired before it is deleted.

Node Retirement

retireClusterNode { identity, nodeId, expectedGeneration, disposition? } retires one node from its cluster's Pallet runtime. It requires a verified infrastructure administrator, fences the node and its cluster's phase record, and writes the node's retirement row (INodeRuntimeRetirement, collection cloudly_node_runtime_retirements) as retiring; nothing is taken away by writing it. A request that states no disposition asks for withdrawn; lost is taken only when stated and escalates a withdrawn row still retiring. Refusals are the contract's nodeRuntimeRetirementRefusals, answered by name: node-missing, cluster-runtime-not-pallet (no phase record, or a phase other than pallet), conflict (the cluster runtime generation moved), node-already-retired, and node-still-reporting for a lost request about a node whose last accepted report is younger than clusterNodeObservationFreshnessMs.

One protection step drops at most sixteen egress owners (maximumFencesPerStep). Beside the row Cloudly keeps the node's admission (cloudly_node_runtime_retirement_admissions). A lost node will never acknowledge a head again, so nothing may wait for it: it is admitted at once, by the request that states it — an escalation of a queued owner included — outside any group. Withdrawn egress owners leave in groups: the request queues one, and the first sweep pass that finds no withdrawn group in flight — no withdrawn owner of the current authority admitted and still retiring — admits the next group of up to sixteen in one transaction, in the order they were queued. A queued owner is an ordinary owner until then: in scope (but not assignable), compiled, its credentials active, able to register and acknowledging heads, and counted among the owners every fence clears; its pass answers retirement-queued. A node no protected authority names is admitted at once. The admission is durable, so a restarted controller resumes the same group, and it is written against each admitted node's document fence, so a registration racing it conflicts: the attempt that read the node before the other committed aborts, and only its retry, on the committed state, may commit. A queued owner that stops before it acknowledges a head free of the group's endpoints would hold the group; stated lost, it is admitted, its own fence makes it the group's exemption, and the group drains.

From its admission on, the node is out of every producer's scope and a registration of it is refused. A level-triggered retirement sweep, hinted with every other producer, moves each admitted node through the contract's egress fence, each step its own transaction, and names the wait it ends in:

• slot-settlement-pending — a withdrawn node still runs a slot its own removal receipts have not settled; the service runtime pass stops and removes everything on it.
• protection-withdrawal-pending — a withdrawn egress owner has not reported its null protection withdrawal. From its admission on, the network compiler signs the request into its projections (see below), and its Pallet answers with the null barrier; a Pallet of an earlier release is never asked and reports it when it stops. Either null completes the step; the pass does not wait for the node to acknowledge a head carrying the request, which an earlier release never would. The fence re-reads the withdrawal in the transaction that records it; a node that reported a barrier again before its credentials went is never fenced as withdrawn, and leaves as lost.
• then, for either disposition: both credentials are revoked for good in the node's managed VPN lane (a purpose never issued is recorded revoked, so it can never be issued later, and a rotation of a retiring node's credential is refused); every reserved workload lease and then every handoff of the node is quarantined; and the node's slots are released — the settled ones of a withdrawn node, every one of a lost node, each abandoned slot audited in cloudly_runtime_slot_abandonments with the administrator who stated the loss, and the secret versions its run retained released as its joined removal would have.
• lease-ledger-over-bound — the node holds more than 256 handoffs or 512 workload leases, more than one fence can name; an operator has to look.
• packet-window-open — the latest DNS or managed VPN window ever signed for the node (the DNS lease journal's authority horizon) has not ended yet.
• projection-acknowledgement-pending — a remaining egress owner has not acknowledged a projection head that carries none of the node's endpoints.
• protection-step-pending — the fence (IRuntimeNetworkEgressRetirement) is recorded against the exact current protected authority, in cloudly_runtime_network_egress_retirements beside the authority, where it outlives the node document; the protection pass drops the node behind it.

A fence clears exactly the owners of its authority that are not admitted to leave, so no admitted owner waits for another's head. The step that consumes it drops the admitted withdrawn group whole — nothing else can keep those owners once they stop acknowledging — and fills up with lost fences; every other lost owner of the authority is held as an exemption and leaves in a later step, sixteen at a time. A fence binds the exact authority it was recorded against, so each pass records a waiting owner's fence again against the authority the last step left. The staging reads the fence record of every owner either side of the step names, one read by id each: it consumes the fence of every dropped owner, reads and fences — but does not consume — the lost fence of every kept owner those fences do not clear, and passes every owner either side names that an earlier step fenced as the retired-owner history. It checks the whole step with bindRuntimeNetworkProtectedAuthorityRetirement(previous, next, fences, retiredOwners, exemptions) and marks the consumed fences retired in the same transaction; a drop without a fence, a kept owner neither cleared nor exempt and a return of a retired owner are all refused. Once no protected authority names the node — or when it never was an egress owner — its row is retired, and the node can be deleted.

The request is IRuntimeNetworkProjection.retirement ({ disposition: 'withdrawn', protectedAuthority }). The compiler adds it to a node's next projection once the node is asked (RuntimeSessionStore.isAskedToWithdraw): its retirement is withdrawn, still retiring and admitted, the current protected authority names it as an egress owner, and its current session speaks @serve.zone/interfaces 32.44.0 or later (palletProjectionMemberMinimums.retirement). The request names the projection's own authority, and every later projection carries it unchanged, however far the authority moves on, as the contract latches it. A queued owner is not asked: it is an ordinary owner until its group is admitted, and a withdrawal would leave it unable to report the receipts every later step needs from it. A node of an earlier release is not asked either, because its exact-key reader refuses a projection carrying the request; it withdraws when it stops, and the projections it already has stay as they are. The network producer pages the nodes it compiles through listProjectionPage, which adds every asked node to the members, so an asked node keeps receiving projections until its credentials are revoked.

produceNodeRetirements() and inspectNodeRetirement() run or read the sweep for administration and tests. The drop is a protection step that does not only add, so it withdraws every managed VPN member, as the contract states for every retirement.

Node Hostnames

Cloudly owns the static hostname a node should carry, as data.hostnameIntent (name, hostname, generation, requestedAt, requestedBy), and Spark applies it on the host. The default scheme is cloudly-worker-<name>: the name is the operator's short name for the node, kept beside the hostname so a renamed node can always be reset to the default.

• setClusterNodeHostname { identity, nodeId, expectedGeneration, hostname } renames a node. The node keeps the name of its intent; a node without one takes the name proposeClusterNodeName(hostname) proposes (foo names it foo, cloudly-worker-grasberg names it grasberg).
• resetClusterNodeHostname { identity, nodeId, expectedGeneration, name? } sets cloudly-worker-<name>, from the stated name or else the name of the node's intent.

Both require a verified infrastructure administrator, whose user id becomes requestedBy. expectedGeneration is the generation of the intent the caller read (0 without one); the node fence, the cluster's phase record, the intent and the cluster snapshot are written in one transaction, so two changes in one cluster serialize and one hostname is never given to two of its nodes. Cloudly answers the node at once; an intent the node has not converged to may always be replaced. Refusals are the contract's clusterNodeHostnameRefusals, answered by name and changing nothing: node-missing; hostname-invalid (not one lowercase RFC 1123 label); name-invalid (no name the scheme can carry, at most 48 characters, or a reset that names none for a node without an intent); hostname-in-use (another node of the cluster reports that hostname or is meant to carry it); hostname-unchanged; conflict (a stale expectedGeneration, or a write that raced this one); and node-runtime-unsupported (a node without a Pallet-era report, or in a cluster whose phase is not pallet). Only the Pallet-era report is read, never the Swarm-era sparkRuntimeInfo. node-hostname-referenced is never answered: gateway routes and inbound mail of a Pallet service reach the uplink of the node its runtime spec pins by id (resolvePalletServiceTargetHost), so nothing of a Pallet cluster is keyed on a node's hostname and a rename moves no traffic.

The node has converged when its report states the intent's hostname. The nodes view offers "Rename hostname" and "Reset hostname to default" for a node on the Pallet-era runtime, shows the intended hostname marked pending until the node reports it, or with the reason Spark's last apply failed; the clusters view, the deployment and service node columns and the overview attention list name a node by its intended hostname too.

Network Production

stageNetworkSelection({ nodeId, revision, handoff, router }, expectedRevision) retains the durable per-node compilation input in cloudly_runtime_network_selections under a revision CAS, fenced against the node. A selection that could never compile is refused: router choices without a handoff (invalid), a handoff that is not the node's current reservation, or a resolver or platform endpoint the current protected authority does not name (inactive). A node without a staged selection compiles the protection-only bootstrap: no handoff and no router egress. Neither inventory presence nor policy stands in for a selection.

RuntimeNetworkProducer is the production trigger for compilation. It pages the nodes in scope and produces each one; the scheduling around that pass is LevelTriggeredSweep (ts/manager.runtime/classes.leveltriggeredsweep.ts), which owns the period, the hint delay, the floor and the single timer described below. Every sweep visits every member in pages of eight, so correctness never depends on a hint arriving, and restarts and other controller instances are covered because membership is read from the database.

These committed changes ask for a sweep without naming a node: session registration, assignment observations and terminal receipts, network protection receipts, stageNetworkProtection, setServiceNetworkEgress, setServicePublishedPorts, reserveNetworkHandoff, stageNetworkSelection, setRuntimeNetworkAddressPlan, mutatePrivateNetwork, setServicePrivateNetworks, node enrollment (which activates a Pallet credential) and the rotation or revocation of a Pallet credential. Each of them reaches both producers, because a membership change may move a projection and the protected authority alike. Anything else, such as service deletion, is picked up by the periodic sweep. One sweep runs at a time. The next one starts once hints have been quiet for one second, since every hint restarts that wait, or 30 seconds after the previous sweep ended, whichever comes first. No timer-driven sweep starts sooner than five seconds after the previous one ended, and a hint arriving mid-sweep earns exactly one follow-up sweep. These waits are measured on the monotonic clock, so a wall-clock step neither stalls nor spins production. produceNetworks(), for administration and tests, joins the sweep in flight or runs one at once without waiting out the five seconds.

Each node is produced plan first. Planning reports unconfigured (no protected authority), awaiting-acknowledgement (the head is unacknowledged and signed by the current authority), awaiting-application (the acknowledged head has no qualifying application receipt), unchanged, a named refusal, or a change. Pending application returns before composing new material or quarantining it. Planning commits only idempotent safety writes that never advance the chain: fences (an absent private-network membership is materialized empty so it can be fenced), stop-intent workload quarantine, replaced-handoff quarantine and the detachment of retiring private networks. They stay committed when the commit that follows fails, and the next commit rebuilds on top of them. Only a change takes the managed VPN node lane and recompiles inside the committing transaction, so an unchanged node never has its packet authority withdrawn. Inside the lane the change is planned once more and its candidate judged against the node's managed VPN member: a candidate that continues the membership (continuesRuntimeManagedVpnMembership — the same projection lineage, protected authority and handoff), which is every workload, readiness, egress, policy or selection change, keeps the member, its key pair and its place in the hub's network, and the pass after the commit states the node with its new prefixes and grants. Only a candidate of another membership, such as a fresh handoff, withdraws the member before the commit, exactly as an authority mutation does. A network is unchanged when the candidate matches the acknowledged head in every field except chain position, digest and DNS window and the head carries the current signing authority; after application a rotated authority therefore signs a successor of the same network. An unacknowledged or unapplied head signed by a rotated authority could never be delivered, so the current authority re-signs it unchanged at its own generation. Either way the new key is delivered before the projection. Evidence follows the same rule as the network: an endpoint the node already holds keeps the observation it already applied unless its readiness flipped, so a repeated probe is not a network change and does not restart a peer's data plane. A compiled projection is pushed immediately, and the 5-second delivery lane remains the retry path. inspectNetworkProduction() returns a copy of the last sweep's per-outcome counts with up to eight named per-node failures. A paged producer reports its own failures, whoever started the sweep (PagedProducerSweep over ProducerFailureReports): a node whose production throws is logged as runtime network projection of node <id> failed after <ms> ms: <cause> once per change of cause and … produced again: <outcome> once it compiles again, and inspectNetworkProductionFailure(nodeId) reads the failure, its time and how long the attempt ran until then. The cause is value-free: the failing step (NetworkProductionError [plan|membership|compile]) and the check that refused, such as network-lease-handoff-not-selected, network-execution-replaced-unsettled or network-projection-rejected (the vocabulary is TRuntimeRefusalReason). The service runtime producer reports the same way (service runtime of service <id> failed after …, inspectServiceRuntimeProductionFailure(serviceId)), and so do the node network, node retirement and service retirement sweeps; the protection and relay name sweeps, which are single passes, report a failed pass. Shutdown cancels the timer, joins the sweep in flight and abandons its remaining nodes; nothing is admitted after stop, and an abandoned sweep is not reported.

The DNS window is not network state: an otherwise unchanged network is not re-signed for a new window, because sameNetworkApplication ignores dns. Windows start on a ten-minute grid and last the contract maximum of 15 minutes, so nodes compiled in one slot share one window. A controller window never moves backwards: a successor states the slot of the later of the controller's wall clock and the window the node already holds, which is its acknowledged renewal's or else its head's, so a clock stepped back issues that slot again instead of a successor the projection contract refuses or a window earlier than the node's own. A clock that ran ahead issued windows that are not valid yet; once it is corrected, successors keep that slot until the clock reaches it, and nodes wait out the not-yet-valid window. A node whose network stays unchanged keeps its DNS through signed renewals instead (see DNS Lease Renewals).

DNS Lease Renewals

A renewal extends one node's DNS window under its exact acknowledged projection and changes nothing the node applied, so a steady network keeps its DNS without a new projection restarting its data plane. Its sequence is the slot index, issuedAt / 600000, and its window is that slot's: it opens at the slot's start and lasts the contract maximum of 15 minutes. Every field follows from the node scope, the projection reference and the slot, and Ed25519 signatures are deterministic, so signing a slot again yields the byte-identical envelope and a retry never puts two windows under one sequence. A renewal is signed only by the authority revision that signed its projection, inside the transaction that fences the signer against rotation; after a rotation, only a newly signed projection continues DNS.

RuntimeNetworkDnsLeaseStore keeps one row per node in cloudly_runtime_network_dns_leases: the projection its renewals extend, which is none until the node acknowledges its first, the latest signed and the latest acknowledged sequence, and dnsAuthorityHorizon, the latest window end Cloudly ever signed for the node under any projection. The horizon never decreases; nothing tied to the node's names may be released before it. It is raised in the transaction that admits a projection, so a successor whose acknowledgement is lost still counts — the node may already hold it. The row is adopted in the transaction that records a projection's acknowledgement, so no acknowledged projection exists without its lease, and adoption ends every renewal of the previous projection. prepareRenewal then:

• refuses superseded unless the node's delivery head is acknowledged and is the lease's projection;
• picks the slot of the later of the controller clock and the node's current window, which is its acknowledged renewal's or else the projection's, so a clock stepped back never states an earlier window;
• refuses beyond-horizon when that window would end more than the contract's four-hour escrow horizon after the clock. A window a node can take now ends within 15 minutes, so today this names a controller clock that fell far behind a window it already signed;
• asks the contract's own admitRuntimeNetworkDnsLeaseRenewal, on the controller clock, whether the node could admit the renewal now. The projection's own slot, the slot already acknowledged and a clock stepped back therefore renew nothing and change nothing;
• signs it and advances the latest signed sequence and the horizon by compare-and-set, never backwards.

acknowledge records a renewal the node admitted. It takes only the lease's own renewal of its projection, at a sequence no later than the latest signed one, refuses a renewal of a superseded projection as superseded, and a late acknowledgement never moves the lease back.

prepareRun states the same slot and signs the whole run of consecutive slots a relay may hold from it, under one signer fence, and raises the latest signed sequence and the horizon to its last window by compare-and-set. The run's length is never written down: dnsLeaseRunSlots takes slots while the contract's escrow allows them — at most maximumEscrowedRenewals, at most maximumEscrowHorizonMs from the first window's start to the last one's end, and no window ending more than that span after the controller clock — which on the ten-minute grid is 23 slots, just under four hours. The slots are consecutive because a window covers its own slot and half of the next, so a skipped slot would leave five minutes uncovered. A clock that fell behind a window already signed shortens the run; one behind it by more than the escrow horizon refuses it beyond-horizon, as it refuses a single renewal.

dispatchNetwork adds a third step after the projection ACK or replay: prepare the renewal, push applyRuntimeNetworkDnsLeaseRenewal through the same push helper, bind the ACK to exactly the renewal that was sent, and record it. It resolves { projection, renewal, escrow }, where renewal is { status: 'delivered', renewal }, { status: 'not-due' } or { status: 'failed', error }, so a failed renewal never hides a delivered projection. A renewal becomes deliverable when its slot opens and must land before the previous window ends five minutes later. The delivery loop visits eight projection rows per pass and starts each pass five seconds after the previous one ended, so it reaches every node within those five minutes for up to 400 nodes while each pass finishes within a second. Beyond that, windows lapse and fail closed until the node's next visit.

A node whose delivery fails, because a push timed out or was refused or the node has no session, is skipped by the loop for one second, doubling with each further failure to at most 60 seconds on the monotonic clock, and the pass counts it as deferred. A node that times out therefore costs one 10-second push timeout per wait instead of one per pass. A successful delivery or a new session registration for the node ends the wait, and shutdown forgets every wait. Direct dispatchNetwork calls and the push after a compile do not wait, but every dispatch outcome starts, extends or ends the node's wait. A Pallet node that does not accept renewals refuses each push and keeps its acknowledged projection; its DNS lapses closed.

The run a cluster relay holds

A cluster is outbound-only, so a node that loses Cloudly loses the only signer of its DNS window. The fourth dispatch step therefore hands the node's cluster relay the run it may release while it has no Cloudly session: Cloudly mints the run, pushes holdRuntimeNetworkDnsLeaseRenewals with the contract's own request snapshot, and binds the answer to the exact run that was sent. escrow reports { status: 'held', cluster, held }, { status: 'retained', cluster }, { status: 'no-relay' }, { status: 'deferred', retryAt } or { status: 'failed', error }.

Only a node whose current session is carried by its cluster's elected relay is signed a run at all, which ClusterManager.readElectedRelayCluster decides through the one election every dispatch into a cluster uses. A directly connected node, and a node whose carrying peer is not its cluster's elected relay — it registered none, none is live, or two sockets claim the cluster — are handed nothing and signed nothing, so their revocation latency stays one window. A node without a session never reaches this step at all: the dispatch refuses it before the renewal. A relay-carried node's dnsAuthorityHorizon instead sits up to four hours ahead, which is exactly the contract's revocationLatency: a detached cluster keeps DNS authority until its last held window ends. The run's first renewal is the renewal the node is handed directly in the step before, byte for byte, so a node may hold that window from Cloudly and from its relay and both are one statement.

What the run would state is read and compared with what the relay was last handed in the same transaction that mints it — readRun follows exactly the rules prepareRun follows and signs and writes nothing — so a pass that would repeat itself answers retained at the cost of one fenced read transaction: no signature and no lease write. What it is compared against is the memory-only record RelayRunHandovers keys by the relay peer, the node's reporterSessionId, the projection reference and the run's first and last slot. Those are the four cases in which the relay itself gives a run up: the node registers another session, the node admits a superseding projection (which a signer rotation reaches through the successor it forces), another socket becomes the relay, and the last window of the run ends. The slot rolls over every ten minutes, long before that last window, so the key changes — and the run is minted and handed over again — long before the relay reaches the end of what it holds. A fifth case is Cloudly's own and changes none of them: a registration Cloudly refuses with relay-cluster-not-enabled, relay-node-foreign, relay-node-unknown, relay-sequence-not-advancing or relay-credential-generation-stale tells that relay it is not the one Cloudly dispatches to, and it drops every run it holds on the same socket, under the same session, for the same projection and the same slot. Cloudly issues that refusal, so Cloudly forgets what it recorded for that peer: the runtime manager subscribes to ClusterManager.subscribeRelayUndispatched while it runs, and the next pass hands every run of that relay over again. A hand-over already in flight when that refusal arrives is recorded as nothing at all, because a pass records what it sent only against the forget count it read before it minted. A restarted controller remembers no hand-over at all and hands every run over once more, mints the identical bytes, and the relay takes it as the run it already holds.

The step is last and answers for itself, so neither the projection nor the renewal before it can be delayed or failed by a relay. The escrow names six refusals — escrow-run-invalid, escrow-node-not-carried, escrow-run-foreign-session, escrow-run-stale, escrow-run-unordered and escrow-cloudly-only — and words each of them with that name first. Each is an answer from a working socket: the run does not satisfy the contract, the relay carries the node no longer, the run arrived under a session the node has replaced, the relay holds a run reaching at least as far, it cannot order this projection against the one it holds, or the hold reached its node listener instead of its own Cloudly session. Only those six leave the relay no wait, so the next pass hands over again as soon as what caused one of them changed; they are the contract's names, held in dnsLeaseEscrowRefusals until @serve.zone/interfaces states them beside holdRuntimeNetworkDnsLeaseRenewals. Every other answer makes that relay wait: the texts typedrequest words for a method the relay does not serve and for a handler that threw, an answer that does not name the run that was sent, and a socket that does not answer at all. The wait is one second, doubling to at most 60 seconds, on the same backoff the nodes use but keyed by the relay peer: one unreachable relay then costs one wait instead of one 10-second timeout per node it carries, and it is released when that socket closes or when a hand-over lands. Nothing is read, signed or sent for a relay while it waits — the pass reports deferred with the monotonic reading the wait ends at — so a broken relay never advances the horizon of the nodes it carries.

Managed VPN — the cluster's own hub

Cloudly runs no VPN server. Each cluster's relay is that cluster's packet relay: every node of the cluster dials its relay's managed hub, and no inter-node packet leaves the cluster or reaches Cloudly's process. Cloudly keeps every authority — who is a member, which control address they hold, which key their credential carries — and states one complete network per cluster, which that cluster's hub applies.

configureManagedVpn({ controlPrefix, hubPrefix }) is a trusted controller operation that stores the plan's immutable vpn and the append-only node control-address reservations through SmartData. controlPrefix is the private IPv4 prefix control addresses are allocated from; hubPrefix is the protected prefix every cluster hub must bind inside. Both must be covered by the protected union, disjoint from workload and transit pools and from the resolvers, and disjoint from each other — control addresses live inside the tunnel and hub endpoints on the underlay. The network and gateway addresses remain reserved; disconnects, expiry and restarts never recycle a node address. Later protected-authority updates must preserve this configuration. Startup without configuration runs no reconciliation pass at all.

A relay advertises the hub it bound on its own registration (registerClusterRelay, relay.vpnHub = { publicKey, address, quicPort }), and the registration is replaced whole: a relay that restarts re-keys its cluster's hub, and one that comes back serving none withdraws it rather than leaving the keys of a process that is gone standing. The registration is judged by the contract's own validator, so a hub Cloudly would refuse on the wire is never persisted; whether the plan covers the address it bound is decided where the hub is selected, so one misconfigured relay cannot fail the authority every other cluster is composed into.

getRuntimeManagedVpnCredential requires the exact current physical Pallet socket, complete session binding, current authenticated projection and acknowledged signing authority. That projection must explicitly select its own cluster's hub for router egress. The response carries transport: 'quic' — the one transport a managed hub serves — the hub endpoint the node dials, the authority id, the hub's public key and a short-lived client key pair. It stays process-local; never log, persist or embed it in projections. Exact current retries retain their keys, and a relay that re-keyed its hub costs exactly its own cluster's members their credential. The credential lease cannot outlive the node's effective DNS window: the window of the renewal it acknowledged for its current projection, or else the projection's own.

A key pair belongs to the node's membership, not to one projection. Every workload change signs a new projection, so each reconciliation pass moves a member onto the node's current projection with bindGetRuntimeManagedVpnCredentialSuccessor: a successor of the same membership keeps the session, the hub and the key pair, and only the credential's projection and expiry move. A request naming the successor is answered with that same key pair. A protection step that only adds (a node joining the egress, a cluster hub joining the platform endpoints) keeps the membership: until its node producer signs the projection citing the new authority, a member's projection citing the direct predecessor still stands, and the next projection moves the member onto the new authority. The member ends — and the node's next credential carries a new key pair — on a registration, a protection step that does not only add, a lost session, a relay that re-keyed its hub, a projection of another membership (a fresh handoff), or a DNS window that lapsed. A request that still names a projection the node no longer holds is refused and costs the member nothing. A refused request is answered value-free, Runtime managed VPN credential request denied. with the reason word of the report refusals (runtime-session-report-refused), and Cloudly logs the node and the cause once per reason: the error's name, message and stack and those of its causes, credentials redacted (describeErrorCause).

One clock reading judges every member of a pass, and it is taken once every member has been read: a window is issued from an earlier reading of the same clock, so a renewal of the next slot that the node acknowledged while the pass ran is never judged as not yet open.

One network per cluster is compiled from the same durable state (compileClusterVpnNetworks) and pushed to that cluster's relay over its already registered session (applyClusterVpnNetwork); a relay whose hub has just started pulls what Cloudly states instead of waiting (getClusterVpnNetwork). A network is the whole authority its hub serves, never a patch, so an absent member is a withdrawn one and a cluster whose hub holds no member is stated an empty network. It carries only exact local workload /32 prefixes, each with its own policy domain, and a grant requires both node projections to contain the same lease material and directed permission. Missing peers, absent reverse permission and control, public or platform destinations confer no grant. Two answers are refusals rather than networks, and Cloudly states them by name: relay-vpn-hub-absent for a cluster with no hub to select, and grant-cross-cluster-unsupported for a grant whose two workloads sit on two clusters — hubs are per cluster, so such a grant has no path through any one of them, and hub federation is a later lane.

The revision each cluster is stated a network at is durable (clusterNetworks on the managed VPN record). A hub's daemon outlives Cloudly's process, so a counter kept only in memory would restart at one while that hub still holds a higher revision, and every push after a restart would be refused as stale. It rises only when a cluster's content moves; a cluster whose relay is unreachable is retried with the same revision and spends no counter, and the relay's refusals (vpn-network-authority-foreign, vpn-network-revision-stale) are logged by name against the cluster that stated them.

Each cluster's network is pushed by a deliverer of its own, beside the owner's lane and never inside it: a relay that answers slowly or not at all holds up the next push of its own cluster and nothing else — no credential, no pass and no other cluster waits on it. A cluster has at most one push in flight, and a network stated meanwhile replaces the pending one, so a hub is only ever handed the latest revision. A push that fails, and a hub that answers anything but idle (applying, failed), is logged once per cause (managed VPN network revision … was not applied by cluster …: <cause>; retrying with backoff) and retried after a wait that starts at 1 s and doubles to 60 s; a newly stated revision is pushed at once.

One serialized owner checks current source authority before and after it states a network. Registration, human Pallet credential rotation/revocation and projection replacement withdraw the affected member before the mutation; protected-configuration changes withdraw every member. Physical close immediately closes future credential admission and queues joined withdrawal. A reconciliation pass runs on every change the owner learns of — a credential, a registration, a lost peer, a projection or protection step, a renewal or projection a node acknowledged, and a relay registration, which may have moved or re-keyed its cluster's hub — and moves each member's expiry to its effective window under the key pair the node already holds, so a renewed node keeps its membership past its projection's window and an exact retry returns the same keys with the renewed expiry. A pass also runs when the earliest member's window ends (no sooner than a second away), and a safety pass every 60 s detects source changes made outside every one of those paths, such as a database edit: such a change is noticed within a minute, where a pass ran every second before. Every read of a pass is a probe, so a pass that finds nothing to change writes nothing. The pass judges every member and the network it compiles on one clock reading, so a window that has ended at that reading costs the member its place, one ending mid-pass costs it on the next pass, and neither stops the pass. A pass that cannot read its state withdraws every member rather than stating stale grants, and the expiry every member carries into the network is the backstop for a pass that cannot run at all: a hub stops relaying for a member whose window ended without being told to.

Database acceptance is not proof of instantaneous packet withdrawal on disconnected workers, and a hub that accepted a network does not prove Pallet TUN installation, workload ACTIVE state or deployment readiness.

Private Network Desired State

ts/manager.privatenetwork/ owns organization DNS namespaces, private network definitions, placement-independent service attachments, alias reservations and historical mutation receipts through SmartData. Names use the immutable keys and strict metadata contracts published by @serve.zone/interfaces. Display-name changes preserve fully qualified names. Alias uniqueness is enforced per network; the same alias can be held independently in another network.

The seven versioned network RPCs require a signed Cloudly identity bound to a server-held IDP login session. Every transaction attempt obtains fresh confidential introspection through @idp.global/sdk/server and requires the requested canonical organization's owner role. Cloudly infrastructure administrators, local password logins and ordinary hosted-app OIDC logins confer no organization authority.

Mutations require an exact expected revision and retained mutation ID. The same SmartData transaction fences the exact login session, human credential generation and immutable issuer/subject binding. Expiry is checked before commit, independently of asynchronous TTL cleanup. Service membership writes also fence current service ownership and every referenced network against retirement. A failed receipt write rolls back the entire mutation. Historical replay returns its original metadata and cannot restore current attachment or access authority.

Detaching an alias marks its reservation as retiring. Network retirement blocks new attachments and remains pending. This foundation provides no completion or identity-reuse operation: the node grant ledger, explicit denial/fencing evidence and DNS quarantine must be integrated before reservations can be released. Authoritative endpoint production and Pallet packet/DNS application remain under development. The private-network store creates no public DNS records, ingress routes, host ports or runtime effects.

getServicePublishedPorts and setServicePublishedPorts are registered by the same manager and reuse the same organization-owner authorization for every service of an idp.global organization, but their durable state is the runtime network service policy, not this store. They authorize host ports for one service: Service.data.publishedPorts remains the operator's request, while the policy document is the authorization of record, and every entry the compiler writes into the signed runtime network projection carries the exact policy revision and digest that authorized it. An entry is one port (hostPort to targetPort) or a contiguous range (hostPort to hostPortEnd, each host port to the same port inside the workload, so targetPort equals hostPort), and symmetric: true makes the flows the workload originates from a published port leave the node from that same host port, so a peer such as a SIP trunk sees one address and port in both directions. Entries of one protocol never overlap and together cover at most 1024 ports in at most 64 entries (runtimeNetworkPublishedPortContract). An App Store range onto other inside ports (targetPortEnd, publishedPort*) and an explicit hostIp are refused with an actionable message; an omitted host address means the node uplink, which the runtime enforces. Ports 49152-65535 are the band the handoff leases translate outbound flows to (runtimeNetworkHandoffSnatPortRange) and are refused published-port-reserved; a symmetric entry rides the service's public egress grant and is refused published-port-symmetric-without-egress while the document denies it. A symmetric entry owns its inside ports: another entry of its protocol publishing to any of them, symmetric or not, is refused published-port-symmetric-inside-overlap with both entries named (findRuntimeNetworkPublishedPortInsideOverlap), since the flows the workload opens from those ports would otherwise have two published sources. After the owner is authorized, a policy refusal reaches the caller as Service network policy rejected: <refusal>. rather than the generic denial. Host ports are unique per node, but placement is resolved when the projection is compiled, so two entries claiming one port of one node are refused there: the compiler signs nothing for that node, reports published-port-conflict and logs the two services once. A range covering 29460 or the media block of a call router is therefore expressed as one entry, and a cluster ingress port mapping or an inbound mail target inside a range is authorized by it.

A service that belongs to no idp.global organization — a cluster ingress, a platform call router, any service whose organization id is namespaced with : such as legacy:Service:<id> or Organization:<id> — has no organization owner who could authorize its ports. An infrastructure administrator designates such a service platform-owned with designatePlatformOwnedService (interfaces IServicePlatformOwnership), withdraws it with undesignatePlatformOwnedService and reads it with getServicePlatformOwnership; all three require a verified administrator, and the designation is stored on the service document field-wise by ServicePlatformOwnership, so no service save carries it and a deleted service takes it along. From then on getServicePublishedPorts and setServicePublishedPorts for that service require a verified administrator, the identity check of setServiceNetworkEgress, and no organization owner; for every other service they keep the organization-owner rule above, and an administrator does not substitute for the owner. A caller without the authority the service requires is answered the one denial text whichever path it took: the designation is read inside the manager's admitted operation, after the credential's shape and the admission bound, and a failed read is that denial too. A designation counts only while it names the service's current organization. The authority is read again inside the transaction that stages the write, so a designation change committed before that transaction refuses the write as the policy write's conflict (Service network policy rejected: conflict., as a stale expectedRevision is). Only a platform organization's service is designated: interfaces isPlatformOrganizationId is true exactly for a canonical identifier that carries :, and false for an idp.global nanoid, whichever character it starts with (- and _ included), and for anything that is not a canonical identifier, such as an empty or whitespace id. Every other id is refused organization-not-platform before the service is read; the refusals are the closed servicePlatformOwnershipRefusals: service-unknown, service-deleting, organization-not-platform and conflict. Withdrawing a designation leaves the published ports as they are with no writer: the service is back under the organization-owner rule, and a platform organization has no idp.global owner, so nobody reads or writes its ports until an administrator designates it again.

Every handoff lease reserved from 33.4.0 on translates outbound flows only into 49152-65535, for TCP and UDP, so no publishable port is ever also a translated source port. A lease is immutable: one reserved earlier keeps its 1024-65535 ranges until the node's router is re-created (reincarnateRuntimeNetworkRouter) and a new lease replaces it, and Pallet refuses a publication that overlaps its lease's ranges by name meanwhile.

getServiceNetworkEgress and setServiceNetworkEgress read and write the other half of the same document: whether the service's attached workloads reach the public internet (publicEgress) and which platform endpoints of the protected authority they reach (platformEndpointIds, sorted and unique). Platform endpoints are platform authority, so only a verified administrator reads or writes egress, as only an administrator writes the runtime spec whose attached network must state the same values; the runtime manager registers both. One document per service holds both halves under one revision, and each write carries the other half forward unchanged. The first accepted write of either (expectedRevision: null) creates it — a published-port write with the explicit deny-all egress { publicEgress: false, platformEndpointIds: [] }, an egress write with no published ports — and from then on both reads return it. A retry of the exact body answers the stored document with replayed: true, and every accepted write asks the network producers for a sweep, so each node's projection recompiles its endpoints with the new values. The refusals are answered as Service network policy rejected: <refusal>.: invalid (a body the contract refuses), conflict (a revision that is not the stored one, or a write that lost to a concurrent one), service-unknown, service-deleting (a service whose deletion has begun), protection-missing (no protected authority yet), platform-endpoint-unknown (an id the current protected authority does not hold), and the two published-port rules, which hold whichever half a write sets: published-port-reserved, and published-port-symmetric-without-egress, so clearing public egress under a symmetric port is refused as adding one without it is. An attached spec whose document still carries a reserved or egress-less symmetric port, written before these rules, is refused the same names by the service runtime producer. An address-plan write that removes or moves a platform endpoint a service egress selects is refused plan-endpoint-referenced. The service detail of the dashboard shows the authorized egress beside the published ports, each range and symmetric entry as authorized.

The Networks view is available to ordinary organization users after generic Cloudly identity registration; their dashboard exposes only this view. Existing administrator views remain available to administrators, alongside Networks. The organization picker obtains current owner memberships through the SDK's subject-bound UserInfo client using the encrypted server-held token. These summaries are display context, not authorization. Service candidates are paginated within the selected organization, exclude deleting services, and require fresh owner introspection on every transaction attempt.

Owners can create or edit network names and external DNS policies, retire networks, and manage service attachments, aliases and the default short-name network. A double-click opens details; retirement requires explicit confirmation. Unconfirmed saves retain their exact mutation ID and body for manual retry. Successful replies trigger current-state reads, so a historical receipt never replaces a newer revision. Changing identity or organization clears editors, pending edits and lists, and suppresses older responses.

The existing SERVEZONE_PLATFORM_OIDC_* configuration selects the trusted issuer and confidential client. An IDP issuer must advertise openid, profile, organizations, roles and its canonical /oauth/introspect and /oauth/userinfo endpoints. Cloudly requests those four scopes and uses audience-mapped app_roles for its local user/admin role; IDP's global roles and token organization snapshots grant no network permissions. Configure the IDP application connection's role mappings accordingly. Incomplete organization-capable discovery fails closed. Existing hosted-app issuers continue using openid and their app-instance claim contract.

IDP access tokens remain encrypted in cloudly_federated_session using the Secrets v2 keyring and envelope engine. Authenticated context binds all immutable session metadata; only an opaque session ID enters the Cloudly JWT. Session expiry is bounded by the access token, ID token and seven-day local maximum. Reauthenticate after expiry; this flow does not request or persist refresh tokens. Keep referenced KEKs available for unexpired sessions. Local credential revocation, deleted sessions and changed issuer/subject bindings invalidate the associated identities. Shutdown drains network requests and admitted session creation before closing the secret manager and database. The IDP provider must be released and qualified before these RPCs can establish live production authority.

App Store Catalog

Cloudly bundles no App Store data. It loads the signed catalog of @serve.zone/appstore at runtime and picks up new apps and versions without a Cloudly release:

• Never a startup blocker. Starting Cloudly only schedules the catalog work. The last-known-good catalog is restored from NoSQLDB and verified again against the signing keys this release trusts (ts/manager.appstore/appstore.trustedkeys.ts), then the registries are asked in the background: every 15 minutes, and after a failure with a backoff from 1 to 30 minutes. The catalog takes no part in boot, readiness or /status.
• Verified before use. A candidate is activated only with a valid signature by a trusted key, a matching digest, the supported contract and a revision greater than the active one; a published version can never be removed, rewritten or reinstated. A refused or failed load keeps the active catalog.
• Last-known-good in NoSQLDB. The active envelope is stored in cloudly_appstore_catalog (the pointer) and cloudly_appstore_catalog_chunks (the envelope text in chunks, named by its sha256), never on the filesystem. The pointer only ever moves to a greater revision; when another Cloudly on the same database already stored a greater one, this one verifies that catalog and activates it instead of the older one it loaded. A failed cleanup of replaced chunks is logged and left to the next write.
• Registries are a setting. appStoreCatalogRegistries lists https npm registries in order; empty means npmjs.
• Status and refusals. getAppStoreCatalogStatus, refreshAppStoreCatalog and importAppStoreCatalog (an operator-carried signed envelope, for an offline Cloudly) answer the catalog status, also shown on the App Store view. While no verified catalog is active, exactly these requests are refused with APPSTORE_CATALOG_UNAVAILABLE (retryable: true) and its cause: getAppStoreConfig (the version config and app details an install reads), getUpgradeableAppStoreServices, getAppStoreUpgradePreview, upgradeAppStoreService and startAppStoreServiceUpgrade. getAppStoreTemplates answers an empty app list with the status, and a hosted app's upgrade status is unknown with the unavailable reason as its blocker. Nothing else is refused; service writes work as before.
• Template snapshots. A service deployed from the App Store carries the snapshot of the version it runs (appStoreTemplate). Upgrades write it. Creating a service, or an update that moves it to another appTemplateId or appTemplateVersion, adopts it at once from the active catalog; an update that moves it drops the old one first. Snapshot adoption on every confirmation of the active catalog gives every remaining service whose version that catalog lists its snapshot, so a service written while no catalog was active gets it once one is. Adoption never fails a service write. Runtime paths, such as hosted-app OIDC, read the snapshot and never the live catalog; a service without one is named as such per app.

Hosted-App OIDC

The Access view lets administrators enable or disable platform OIDC for hosted apps whose App Store version declares a valid platformOidc capability, and assign the declared app roles to Cloudly human users. Cloudly exposes the matching getHostedAppAccessConfiguration, setHostedAppRoleAssignment, and setHostedAppPlatformOidc admin requests. Browser authorization handoff uses the shared getHostedAppOidcAuthorization, completeHostedAppOidcAuthorization, and cancelHostedAppOidcAuthorization requests from @serve.zone/interfaces 32.7.1.

Cloudly owns its hosted-app signing JWK as an encrypted system Secrets v2 value. Each enabled app gets a generated service-owned client-secret record with launcher-environment delivery metadata for its declared variable. Cloudly includes that entry in resolved manifests and can seal its material to the enrolled cluster recipient; actual workload injection still depends on the downstream runtime rollout described above. Registration records and API responses contain only value-free metadata. Enable, disable, restart recovery, and service cleanup are fenced to the exact service lifecycle and Secrets v2 management scope. The app must have a canonical HTTPS domain, and template upgrades across an OIDC contract change require OIDC to be disabled first.

On first startup, Cloudly bootstraps the first human admin from SERVEZONE_ADMINACCOUNT. Human clients authenticate through adminLoginWithUsernameAndPassword; machine clients authenticate through getIdentityByToken. Cluster creation creates a machine user and token for Coreflow.

Human Credential Administration

Human passwords use salted scrypt verifiers in a private SmartData authority, separate from mutable User records. A User record can never carry a password: save() refuses one. Existing human sessions without a credential generation must authenticate again after upgrading. Machine identities and their token flow remain supported.

Use the shared requests.admin.IReq_Admin_GetHumanCredential contract with getHumanCredential to inspect userId, generation, passwordEnabled, and updatedAt. mutateHumanCredential takes the target user ID, expected generation, a stable mutation ID, and one operation: rotatePassword with a new password, revokePassword, or revokeSessions. Both methods require a human administrator who authenticated within the last five minutes. New passwords require at least 20 characters and at most 1,024 UTF-8 bytes. Never log request bodies.

Every successful mutation advances the generation and invalidates older human UI/API JWTs and OCI registry tokens. Password revocation disables password login; a new trusted OIDC login remains possible. Updating SERVEZONE_ADMINACCOUNT does not rotate an existing user's password. Bootstrap creates the user and authority in one transaction, and does not overwrite existing human users.

For a lost response, authenticate again and repeat the same mutation ID and exact request. The actor-bound receipt returns the original metadata with replayed: true, without reapplying the mutation or returning a password. That metadata is historical; inspect current authority separately. A changed request under the same mutation ID conflicts. Actor authorization, target generation, authority update, and receipt persist within one SmartData transaction.

Typical consumers use @serve.zone/api:

import { CloudlyApiClient } from '@serve.zone/api';

const client = new CloudlyApiClient({
  registerAs: 'admin-tool',
  cloudlyUrl: 'https://cloudly.example.com',
});

await client.start();
const identity = await client.loginWithUsernameAndPassword('admin', 'change-me');
const clusters = await client.cluster.getClusters();

Machine clients such as Coreflow authenticate with getIdentityByToken. Auth-sensitive requests and WebSocket identity tags transport only the JWT-shaped IIdentityCredential; Cloudly verifies that credential and reconstructs the authoritative identity fields from persisted user data. The issued JWT never outlives its source token, and a cluster machine user is accepted only while it belongs to exactly one persisted cluster. The JWT of every other machine user names the SHA-256 of the token it was exchanged for, and Cloudly verifies it only while the user still holds that unexpired token, so removing a token — rotate-token, revoke-token — ends every session opened with it at once. A registry bearer issued for a machine login is bound to its token the same way. A machine JWT or registry bearer that names no token is refused, and its holder exchanges its token again. A cluster identity is ended by its relay credential's generation or by the cluster's deletion, and a service workload identity by its service credential's generation, so neither names a token. The WebSocket tag lets Cloudly push configuration to already-connected Coreflow instances instead of opening inbound connections to cluster nodes.

Cluster Flow

The implemented cluster flow is intentionally simple:

1. An admin creates a Cloudly cluster record.
2. Cloudly writes that record, the machine user its long-lived cluster token belongs to, and the cluster's runtime phase in one transaction. A cluster created from here on states pallet, so steps 3 to 6 are the path of the clusters that predate that record — the ones the cluster-runtime-phase migration states swarm — and nothing in this release moves a new cluster to swarm.
3. Coreflow starts on a Docker Swarm manager node with CLOUDLY_URL and JUMPCODE.
4. Coreflow authenticates to Cloudly and requests the cluster configuration payload.
5. Cloudly returns cluster data, workload services, platform bindings, provider configs, and optional external gateway configuration.
6. Coreflow reconciles Docker networks, services, volumes, platform bindings, backups, and routing. Cloudly exposes the enrolled-recipient and sealed-material APIs for Secrets v2, but end-to-end delivery remains gated on the matching Coreflow/Spark release and enrollment.

When service, platform, or gateway settings change, Cloudly pushes updated config to connected Coreflow clients where supported.

Cluster Runtime Phase

Which runtime owns a cluster's workloads is a record of its own, cloudly_cluster_runtimes, keyed by the cluster id: swarm is Coreflow's, draining stops every service while its volumes stay, switching lets the cluster's nodes join the runtime network while nothing runs, and pallet is Pallet's alone. It is stored beside the cluster document because a cluster update merges caller data, and no caller may move a phase with it. Every phase write advances a generation, so a caller that read a phase states which reading it acted on, and runtime work joins the record through fenceClusterRuntime, which refuses a phase it does not serve and conflicts when the phase moves before the caller commits.

A cluster created from here on states pallet, in the same transaction as the cluster and its machine user; every cluster that predates the record is stated swarm once by the cluster-runtime-phase migration, and a phase that already moved is never restated. Cloudly holds no phase for a cluster it never stated one for and refuses by name with cluster-runtime-missing rather than assuming Coreflow owns it.

getClusterRuntime serves that record to an administrator together with the blockers Cloudly can actually read today: a relay nobody enabled (relay-not-enabled) or that never registered (relay-not-registered), a node without an active Pallet credential (node-without-pallet-credential) or whose architecture is not one Pallet runs (node-arch-unsupported), and a service the cluster shares with another cluster (service-multi-cluster-scope) or that holds a platform binding (platform-binding-present). Only a service whose immutable rollout targets this cluster is inspected, because that rollout is the one cluster scope a service states today, while the config push is fleet-wide: a platform-bound service without such a rollout runs in the cluster too and is still not named. An empty list states that nothing Cloudly can read blocks the cutover, never that nothing does.

Withdrawing a cluster relay is a Coreflow-era operation, so disableClusterRelay is refused outside swarm, as is repairing an immutable rollout's placement: placement is what Coreflow schedules against. Deleting a cluster removes what is only its own — the accepted relay registration, the machine identity its relay authenticates with, and its phase — in one transaction, and is refused while nodes still belong to it or a relay still carries it on a live socket. The user is removed only when it is the cluster machine user Cloudly minted with the cluster; a cluster document that names anything else — a human administrator, another cluster's identity — refuses the whole deletion with cluster-user-not-machine instead of deleting that user. Cloudly owns that pointer, so updateCluster never writes it: a request that echoes the stored id is accepted and changes nothing, one that names a different user is refused, and the key is dropped from the merge either way.

Cluster Relay

A cluster may hold one relay: the single outbound socket that carries the runtime sessions of every Pallet node in that cluster. Cloudly owns the name it is reached under. It mints https://relay.<clusterId>.<zone>:8443 — the port the relay binds its own TLS listener on, named explicitly because an origin drops only :443 — where <zone> is the most specific DNS zone Cloudly manages that holds Cloudly's own public hostname. The Domain documents are that authority, not a public-suffix list: they are exactly the zones an administrator registered with Cloudly, so a name derived any other way could be one dcrouter holds neither an A record nor a certificate for. A Cloudly whose public hostname sits in no managed zone is refused by name instead of minting an origin nothing can back. Every minted block is validated with the contract's own validateClusterRelay before it is stored.

Because the origin is Cloudly's, updateCluster refuses any request that carries a relay key at all, including one that only echoes the block it was served. Enabling and disabling are their own admin requests, enableClusterRelay and disableClusterRelay, authorized like every other cluster request: enable mints and stores the block once Cloudly proved it can back the name (see Cluster Relay Names), disable removes it, and both push the cluster config to the relays already connected, so a relay learns the change without reconnecting. A cluster update is pushed the same way. In the dashboard a cluster shows its relay as off until an operator enables it, then enabled, awaiting relay registration until a relay reports itself in, and afterwards what that relay registered — registered · <listenAddress>:<listenPort> · <version> · since <acceptedAt>, or registered, socket gone when no socket carries it at this instant. The row and the detail actions flip it, and both refresh the registrations along with the clusters.

A relay reports itself in with registerClusterRelay over its own socket, after that socket authenticated with registerCloudlyClientSession. The cluster it speaks for is the one its verified machine identity names and is never a field it sends; the node it runs on must be a node of that cluster and must still exist when the write commits; and a cluster whose relay no operator enabled has no origin to register into, so the registration is refused by name (relay-identity-unverified, relay-node-foreign for a node of another cluster, relay-node-unknown for a node Cloudly does not hold, relay-cluster-not-enabled). The response carries the origin Cloudly published, so a relay can check the certificate it holds against the name its nodes are given, and beside it two references it always carries: what the acceptance was made under, and what it produced. credential is the {id, generation, digest} of the authority Cloudly authenticated the socket as, read from Cloudly's own records and never from the body; for a relay that still holds its cluster's shared pre-relay authority it is that authority's own record, which Cloudly writes the first time a relay names it and removes with the cluster. acceptedRegistration is the reference of the durable registration record, which now carries its own identity beside the cluster it is keyed by, a generation that advances by one per acceptance, and the contract's digest over it, so a citation of a superseded registration names the same record and fails on its generation and digest. A relay that also serves its cluster's managed VPN hub reports the endpoint it bound on that same registration (relay.vpnHub), and the registration is replaced whole, so a relay that restarts re-keys its cluster's hub and one that comes back serving none withdraws it.

The accepted registration is durable, one document per cluster in cloudly_cluster_relay_registrations, so one cluster having one relay is a database rule rather than a convention, and a Cloudly restart does not forget which relay a cluster has. What that database cannot hold is whether the relay is connected right now: liveness lives in an in-memory registry keyed weakly by peer, which confers no authority and is retired when the socket closes, leaving the registration itself untouched. The sequence a relay counts is checked only within the socket that sent it, where it must strictly advance — across sockets it says nothing, because a restarted relay counts from zero again, so there the newer accepted registration wins and acceptedAt, which is Cloudly's own and never moves backwards, is the "since" an operator sees. A refused registration does not consume that sequence: a relay whose registration was refused, for a node that does not belong to its cluster or for any other named reason, may send the very same body again once the cause is gone. Only an accepted one advances the socket's count, so a retry that repeats an accepted sequence is refused.

Beside those registrations, cloudly_cluster_relay_legacy_credential holds one document per cluster for the shared pre-relay authority: the machine identity the cluster was created with, recorded the first time a relay registers under it or the first time its relay credential is minted, so an acceptance cites a record rather than nothing. It carries no bearer and no hash of one, because the bearer lives on the cluster's user document and nowhere else. It is written absent-only, so a relay that registers again never re-records it, and it is removed with the cluster rather than with any registration made under it.

A cluster's relay authenticates with its own generation-bearing credential instead of that shared authority. mutateClusterRelayCredential takes the contract's mutation from a freshly authenticated administrator: ensure requires an absent record and rotate the exact current generation, never a minimum, and the administrator is fenced again inside the transaction that writes. The bearer is delivered in that same transaction as a new immutable encrypted version of the Cloudly-system secret SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, owned by { kind: 'system', systemId: 'cluster-relay:<cluster>' } under the management scope system:cluster-relay:<cluster>, so a credential Cloudly accepted is always one the relay can read and a delivery that failed leaves no authority claiming a bearer nobody holds. A mint requires that secret to be absent and a rotation names the exact active version the authority records. Cloudly answers metadata only: cloudly_cluster_relay_credential_authority keeps the hash of the one bearer each generation issued and the public projection builds the contract's members without it, so no record, receipt or answer of this chain carries a bearer. No request returns the bearer either: an operator exports it out of band with cloudly secrets export-cluster-relay-authorization and hands it to the relay (see Cluster Relay Credential Delivery). A retained mutationId answers the accepted result again with replayed: true and renews nothing — not the generation, not the delivery, not the acceptance time — while the same id over a changed body is refused conflict; those receipts live in cloudly_cluster_relay_credential_mutations and are removed with their cluster, as the authority and the shared record are.

The relay exchanges that bearer for a session the way every machine does, with getIdentityByToken. The session is the cluster's own machine identity — exactly what the shared authority authenticates, so a relay changes one bearer and keeps every call it makes — and the JWT carries the generation it was exchanged for. That generation is re-read on every request the session is presented with, so rotation has no overlap: the previous generation stops authenticating at commit rather than at an expiry. A socket whose generation has been replaced is told apart from one Cloudly never verified and is refused relay-credential-generation-stale before its registration body is judged, which ends that relay's custody; a restart under the bearer of the current generation is what resumes it. The authority a registration is accepted under is also fenced inside the acceptance itself, so a rotation committing between a socket's verification and that write makes one of the two lose instead of leaving a registration that cites a generation nobody holds.

getClusterRelayCredential reads the whole chain for an administrator — the relay credential, the shared record and the current accepted registration, each null while no record names one. Liveness is not part of it: that is a fact of a socket, which getClusterRelayRegistrations answers.

retireLegacyClusterRelayCredential is the last step of that replacement: it revokes the shared authority once something else authenticates the relay. The request carries compare-and-swap inputs only — the exact expectedLegacyGeneration, the replacement credential reference and the acceptedRegistration reference an operator observed — and Cloudly dereferences every one of them against its own records inside the transaction that writes, with the administrator fenced there as a credential mutation fences one. Five conditions hold together, and there is no phase or fleet condition beside them: the shared record is this cluster's current one at the named generation and still accepted (legacy-credential-absent, legacy-credential-already-retired, conflict), the named replacement is this cluster's relay credential at its current generation (relay-credential-absent, legacy-retirement-replacement-not-current), the named accepted registration is the current one and was made under that very replacement (legacy-retirement-registration-not-current), and one verified live socket carries the cluster's relay under exactly that credential at the instant of acceptance (legacy-retirement-relay-not-live). A registration made under the shared authority names that record rather than the replacement, so a relay that never moved can never justify retiring the authority it still holds. Acceptance writes the one-shot invalidation in cloudly_cluster_relay_legacy_credential_invalidation — the shared record at the generation before the revocation, the replacement, the registration, the administrator and the acceptance time — and advances the shared record to state: 'revoked' naming it back, so neither digest covers the other's. A retained mutationId answers that receipt again with replayed: true, the same id over a changed body is refused conflict, and any other retirement of a revoked record is refused legacy-credential-already-retired: nothing is revoked twice. From that commit on, a socket that still holds the shared bearer is refused legacy-credential-already-retired instead of registering under it, and the receipt is removed with its cluster like every other record of this chain. Revoking the bearer itself stays an operator's action; what Cloudly owns is that it authenticates nothing here.

An administrator reads those registrations with getClusterRelayRegistrations, authorized by the same admin guard as every other cluster request, and narrowed to named clusters with clusterIds or left to answer for every cluster whose relay ever registered. The answer is the contract's own read shape — the cluster, the node, the listen address and port, the relay version, the sequence it counted, Cloudly's acceptedAt — plus live, which is whether a connection the server still enumerates holds the Cloudly identity it was verified with and registered as that cluster's relay in this process. live is a liveness fact and nothing more: it grants no authority, it is not part of what was registered, and a registration whose socket is gone stays exactly as valid. The shape is validated with the contract's validateClusterRelayRegistration before it is served, and the stored record is checked against that very same validator, so Cloudly can never persist a registration it would later refuse to serve.

Cluster Relay Credential Delivery

The relay reads its bearer from the launch variable SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, which its qenv resolves from the process environment or from the Docker secret file /run/secrets/SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, byte for byte (a Spark host service seals the same bytes with set-credential). Cloudly holds the bearer only as the encrypted version the credential names, so an operator carries it across as a file that exists only for the handover:

1. Mint the credential with mutateClusterRelayCredential ({ clusterId, operation: 'ensure', expectedGeneration: null, mutationId }, keeping the mutationId for a retry) and read the generation getClusterRelayCredential answers: 1 after a mint.

2. Export it over the API, signed in as an administrator within the last five minutes, into a directory only you can write:

   install -d -m 0700 /root/relay-export
   CLOUDLY_URL=https://cloudly.example.com CLOUDLY_USERNAME=admin CLOUDLY_PASSWORD=... \
     servezone relay export-authorization \
     --cluster-id <clusterId> --expected-generation 1 \
     --output /root/relay-export/relay-authorization
   

   The CLI sends exportClusterRelayAuthorization through @serve.zone/api. Cloudly reads the version the credential delivered at exactly that generation, proves it against the hash that generation issued, writes an audit receipt in a transaction that fences the administrator and the credential at that generation, and only then answers the bearer. The CLI checks the answer against its request (data.bindClusterRelayAuthorizationExport) and writes it with @push.rocks/smartfs as a new file, mode 0600 whatever the umask, exclusive (an existing entry, a symbolic link included, is refused as output-exists and left untouched) and atomic (the whole bearer or no file). The directory must already exist, resolve to itself (no symbolic link on its path), be owned by you and be writable by nobody else, otherwise it is refused (output-directory-missing, output-directory-not-private) before Cloudly is asked, so nothing is fetched and no receipt is written. The bearer is never printed; stdout carries only {"ok":true,"command":"relay export-authorization","clusterId":…,"generation":…,"outputPath":…,"byteLength":…,"sha256":…,"receiptId":…}. Keep the sha256, and check every copy of the file against it with sha256sum before using that copy. Without a reachable Cloudly API, the offline node cli.js secrets export-cluster-relay-authorization inside the Cloudly container writes the same bytes (see the secrets commands above).

3. Carry the file to the host of the relay's cluster without widening its mode (scp -p), and hand it to the relay there: docker secret create <secret-name> - < relay-authorization for a Swarm relay, whose service mounts it with --secret source=<secret-name>,target=SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, or Spark's set-credential for a host-service relay.

4. Remove every copy of the file with shred -u, or with rm where shred is unavailable.

Refusals are named by data.clusterRelayAuthorizationExportRefusals: relay-credential-absent, relay-credential-generation-mismatch (another generation than --expected-generation, before the read or after it), relay-authorization-missing and relay-authorization-ambiguous. Every other failure, an unauthorized or not freshly signed-in caller included, is Cluster relay authorization export denied., and none of them carries the bearer. Every answered export leaves a receipt (IClusterRelayAuthorizationExportReceipt: the administrator, the exact credential reference and the time, never the bearer or a hash), stored in cloudly_cluster_relay_authorization_exports; getClusterRelayAuthorizationExports { identity, clusterId } answers a cluster's receipts, oldest first, to an administrator, and a cluster's deletion removes them with its other relay authority records. A repeated export of one generation is a second receipt. Nothing on this path logs the bearer: the handler logs nothing, and the typed request libraries log method names and connection events only, never a request or response body.

A rotation (mutateClusterRelayCredential with operation: 'rotate' and the exact current expectedGeneration) is delivered the same way, exported with --expected-generation set to the new generation. Docker secrets are immutable, so the new bearer becomes a second secret under a new name, swapped in with docker service update --secret-rm <old-secret-name> --secret-add source=<new-secret-name>,target=SERVEZONE_CLUSTER_RELAY_AUTHORIZATION <relay-service>, which restarts the relay under it; the old secret is removed afterwards. Rotation has no overlap: from its commit until that restart the relay is refused relay-credential-generation-stale, so the export and the service update follow the rotation directly.

Cluster Relay Names

dcrouter holds every relay name Cloudly mints: its A record and its exact certificate. Cloudly is a dcrouter gateway client (see the gateway settings) and holds each name under the gateway-client hostname ownership { appId: 'cluster-relay:<clusterId>', hostname: <relay name> }, through the gateway-client hostname requests syncGatewayClientDnsRecord, getGatewayClientCertificate and releaseGatewayClientCertificate. Cloudly holds no DNS provider credential and issues no certificate itself. enableClusterRelay proves that dcrouter advertises client hostname records and exact issuance (dns.clientRecords, certificates.exactIssuance) before it stores the block, and refuses by name otherwise, leaving the cluster unchanged: relay-gateway-unconfigured or relay-gateway-unsupported.

Certificate. A relay asks for its certificate with getCertificateForDomain when it starts. A name that is exactly an enabled relay's name is answered with dcrouter's certificate for that ownership and never through the gateway's route certificates. Its key is one cluster's: it goes to an administrator or to that cluster's own machine identity, and any other cluster identity is refused relay-certificate-not-owned. dcrouter issues the certificate by DNS-01 in the relay name's zone; while it is issuing, the relay is refused relay-certificate-pending and asks again. Asking is also what renews a certificate past its renew-by date, so the relay's request is answered by dcrouter directly and never from a copy Cloudly kept. Every other name in the relay namespace -- relay.<label>.<zone> with <zone> a zone Cloudly publishes in, whose relay is disabled or was never enabled, or whose cluster does not exist -- is refused relay-name-reserved, whoever asks. The generic certificate path refuses such a name for every caller, a service route's included: a certificate issued there would still be valid once a relay carries the name. A name of that shape in a zone Cloudly does not publish in is outside the namespace and is served as any other name. The wildcard *.<label>.<zone> covers relay.<label>.<zone> and is refused relay-name-reserved the same way, whether that relay is enabled or not; *.<zone> and deeper wildcards cover no relay name of <zone>. Such a name is also never attached to a service: createService, updateService, reserveServiceDeployment and promoteServiceDeploymentRoute refuse a hostname in the relay namespace, or a wildcard that covers one, that the service does not hold yet with RELAY_NAME_RESERVED, before they write anything. Every other certificate Cloudly hands out is dcrouter's as well; a Cloudly without a gateway has none to hand out.

A record. Once a relay registered, dcrouter publishes the relay name at the listenAddress of the accepted registration, with a 60-second TTL, because the nodes dial the relay itself. It is written after the acceptance commits and never inside it, and the registration's answer does not wait on dcrouter. The record follows the relay when a later registration moves it; a relay whose socket closed keeps its name; the record is withdrawn and the certificate released once disableClusterRelay removes the block or the cluster is deleted, and a relay name that changed is released under its old ownership before the new one is held. dcrouter publishes only an address inside the gateway client's allowedDnsRecordAddresses, which Cloudly derives from the runtime network address plan (see Runtime network address plan), and never overwrites or adopts a record it does not manage.

cloudly_cluster_relay_hostnames keeps one document per cluster whose relay is enabled: the name, the address and the reference of the registration it carries (both null before a relay registered, when the certificate alone is held), the state (pending, published, failed or withdrawing), the refusal of the last failed attempt and the attempts since the last success. The document is written before dcrouter is asked, and every dcrouter call is idempotent under the ownership, so an attempt that dies half-way is finished by the next one. A failed attempt is noted by its refusal -- relay-gateway-unconfigured, relay-gateway-unsupported, relay-dns-zone-unmanaged, relay-dns-address-not-allowed, relay-hostname-conflict, relay-gateway-failed or conflict -- and a pass over every relay retries it within five minutes, as it does at every start; the same pass asks for every certificate, so dcrouter renews it.

The data-version step retire-cluster-relay-acme (0.20.0 to 0.21.0) drops the collections Cloudly kept while it issued and published relay names itself (cloudly_cluster_relay_acme_accounts, cloudly_cluster_relay_certificate_issuances, cloudly_cluster_relay_certificates, cloudly_cluster_relay_dns_records). An A record Cloudly had written at Cloudflare is not removed by it: dcrouter refuses a relay name another writer holds, so the step logs each such name as a warning for the operator to remove at the provider.

Node Identifiers

A node id is drawn from nanoid's URL alphabet and is judged everywhere Cloudly stores or reads one by the node vocabulary ^[A-Za-z0-9_-]{1,128}$, which is what the credential authority, the session binding and the Spark wire all state and which admits a leading - or _. The canonical vocabulary ^[A-Za-z0-9][A-Za-z0-9:._-]{0,199}$ belongs to organization, service, cluster, rollout and namespace ids and to the content ids derived from them; a node id is never judged by it, so a node whose stored id begins with a separator is still the node its credential, its session, its network pass and the deletion impact an operator reads all name. Composed ids that take a node id state which part it is, so the id itself is unchanged while each part is judged by its own rule.

New node ids are minted through createClusterNodeId(), which keeps the first character alphanumeric and judges the draw against both vocabularies where the identifier is made. That is what the Interfaces contracts carrying a node id in the assignment and runtime network domains require — IRuntimeAssignment.nodeId and replicaId, the network node scope, and the egress owners, leases and receipts of the protected authority all read it as a canonical identifier — so a node already stored with a leading separator is refused a workload and a runtime network by the contract itself, and only an Interfaces release admitting the node vocabulary there changes that.

Node Credential Authority

NodeManager owns independent spark and pallet credential authorities outside the mutable node inventory. New secure tokens use 32 cryptographic random bytes. All Spark authentication uses this authority; a Spark token cannot authorize Pallet. Heartbeat state, action delivery/results and Swarm observation acceptance commit under the same credential-generation fence as their database effects. Rotation or revocation racing those transactions prevents stale effects from committing. This does not retract an action already delivered before revocation.

Nodes without a Spark hash remain unenrolled, and existing credentials are not automatically rotated.

getNodeConfig, getNodeCredential, rotateNodeCredential and revokeNodeCredential use the published Interfaces 28 contracts and require a verified infrastructure administrator. Config returns only the public node DTO; credential results contain value-free metadata. Mutations require exact generation and session-epoch preconditions plus an immutable mutation ID. Rotation accepts a SHA-256 hash of pending node-generated material, never returns plaintext, and requires the node to retain that material durably before requesting activation. Receipts bind the verified actor and exact request. Exact replay returns the historical result without reactivating replaced or revoked authority.

This is a source integration, not a completed Pallet rollout. Pallet identity bootstrap, peer-bound sessions, atomic Jump claim/replay, isolated migration rehearsal and staged rotation of existing credentials remain deployment gates. Infrastructure administrator permissions are distinct from workload-organization RBAC; these methods do not establish organization permissions.

Spark Host Heartbeats

Two Spark runtimes report to Cloudly, and they are two runtimes rather than an old and a new one. The node runtime posts sparkNodeHeartbeatContract.endpoint (POST /spark/nodes/heartbeat); a Swarm-era worker in coreflow-node mode posts the four sparkSwarmNodeContracts endpoints (POST /spark/swarm-nodes/{heartbeat,metrics-sample,action-result,swarm-observation}). The Swarm-era family retires with coreflow, not with this release. Every one of those routes reads at most the maxRequestBytes its own contract states, within ten seconds, and every request and response body carries protocol; the request and response shapes are owned by @serve.zone/interfaces, so Cloudly and Spark no longer carry private copies of them.

The four sparkSwarmNodeContracts bodies are admitted by the contract's own validators -- validateSparkSwarmNodeHeartbeatRequest, validateSparkMetricsSampleRequest, validateSparkActionResultRequest and validateSparkSwarmObservationRequest -- so a node's body is judged by the exact-key rules it was written by rather than by a second set of member checks here. The order is the one every Spark route keeps: bounded read, the offer, the node the body names, the contract's validator, then the credential store. A refused body is answered 400 naming the member the contract refused -- the observation route answers its own invalid request, because an observation is digested as a whole -- before the node behind it is authenticated and without reaching the credential store. acceptSparkHeartbeat, acceptSparkMetricsSample and acceptSparkActionResult take unknown and run the same validators, so an in-process caller is held to exactly what the route admits.

The node heartbeat body accepts the published host report from the independently enrolled Spark credential. It contains Linux host metrics, bundle identity and the explicit local-ready/workload-unverified classification. It never leases operator actions or supplies Docker, Swarm or container-count observations.

Spark reports only while its node runtime runs, so an accepted report stands for clusterNodeObservationFreshnessMs (180 s, three of Spark's default 60 s intervals) and no longer. Every node read (getNodes, getNodeById, UI pushes, the MCP summary) answers offline for an online node whose lastHealthCheck is older; the stored record keeps what the last report said, and the next accepted report reads online again. Scheduled service updates and the node deletion blocker judge a node running by the same bound.

The current Spark generation and session epoch fence a single SmartData transaction containing node liveness, the separate data.sparkNodeReport, and the cluster's public node snapshot. The response is a secret-free historical acceptance receipt. Rotation or revocation racing this write prevents all of those effects from committing. Shutdown closes admission and joins accepted operations before storage closes. Invalid requests return bounded responses without reflecting credentials or database errors.

The node runtime reads the hostname Cloudly wants it to carry at sparkNodeHostContract.endpoint (POST /spark/nodes/host, { nodeId, nodeToken, protocol }, 1024 bytes each way). It is authenticated by the node's Spark credential and judged in the heartbeat's order and answered the heartbeat's way on refusal (protocolRefusalStatus for an offer, 400, 401, 409 or 503 with { accepted: false }, never echoing the token, always cache-control: no-store). The answer is ISparkNodeDesiredHost: the intent's hostname and generation, or hostname: null at generation 0 while Cloudly holds no intent, and Spark then leaves the host's hostname alone. The outcome of Spark's last apply travels on the next heartbeat as report.hostApply (generation, hostname, applied or failed with a reason) and is stored with the report; whether the node carries the hostname is report.runtime.hostname, not that outcome.

The node and cluster views display the report and its acceptance time. Local readiness never means workloads are ready. A node that reports through the Swarm-era family clears the node-runtime presentation record, so a Swarm worker's own report is what that node presents. This source integration does not activate production workers.

Node Onboarding with Jump Codes

A fresh host becomes a node of a cluster by enrolling itself with a single-use Jump Code. One bootstrap script serves that flow, and two routes render it:

• /jump/<code> for a host an admin onboards by hand. The dashboard displays the command

  curl -fsSL 'https://cloudly.example.com/jump/<code>' | sudo bash
  

  and the same URL answers a browser with a landing page and a curl/CLI client with the script (/jump/<code>/setup.sh always answers the script).
• /curlfresh/setup.sh?jumpcode=<code> for an auto-provisioned server. Hetzner provisioning mints one single-use code per server and passes exactly this URL in the server's cloud-init document, so one code never bootstraps two hosts. No cluster token travels with a provisioned server.

The script runs under set -euo pipefail as root. It installs curl and ca-certificates, resolves the newest published Spark v<major>.x release of Cloudly's own major line from the Gitea releases API — paging until it finds one, refusing by name when that major publishes none — and hands that exact tag to install.sh --node-bundle --version <tag>, which verifies the asset against its SHA256SUMS.txt. The node bundle carries Pallet, so nothing installs Docker: a node of this line runs its workloads through Pallet's own container runtime.

Enrollment is one JSON document on stdin, and the Jump Code is its bearer:

{"operation":"jump","cloudlyOrigin":"https://cloudly.example.com","hostname":"node-a","bearer":"<16-character Jump Code>"}

printf is a bash builtin, so that document exists only in the pipe spark enrollnode reads; the code never enters a process argument list, an environment variable or a file. The hostname comes from hostname -s, lowercased, and a host whose name is not one lowercase DNS label is refused by name before it enrolls. Cloudly's script states one assumption about enrollnode: it exits non-zero when it refuses an enrollment, and writes its SPARK_ENROLLMENT record to stdout when it succeeds. The script acts on that exit status alone and never parses the record. A successful enrollment is what creates the node record, its Spark and Pallet credentials and its replay receipt, in one transaction inside Cloudly — provisioning pre-creates nothing, so the fleet only ever holds records of hosts that actually came online. The script then runs spark installnodeservice, spark activateguard and systemctl enable --now smartdaemon_spark.service.

createNodeJumpCommand may state nodeName. Its script then runs hostnamectl set-hostname cloudly-worker-<nodeName> after installing the bundle and before it reads the hostname, so the host enrolls under its default name, and the enrolled node's hostname intent is that default. A name the scheme cannot carry is refused as name-invalid; a name whose default a node of the cluster reports or is meant to carry, or that an unused code for the cluster already names, is refused as hostname-in-use. Mints are not serialized, so the claim refuses a duplicate again in its own transaction against the cluster snapshot, as an enrollment refusal. Without nodeName the host keeps its hostname, and its node's intent records that hostname under the name it proposes, or none when it proposes none. That observed hostname is not checked against the cluster: it renames nothing, so cloned hosts that all enrolled as debian keep that shared intent until an administrator renames them one by one.

Rendering a bootstrap script requires an https SERVEZONE_PUBLIC_ORIGIN, because node enrollment accepts no other scheme. Without it the landing page, the Jump script route, the CurlFresh route and Hetzner provisioning all refuse by name instead of serving a script that cannot enroll, or minting a code that could not be used.

An auto-provisioned server also carries its hardware identity in its code. Provisioning allocates the BareMetal record's id before it asks the provider for a server, mints that server's code naming that record, and writes the record under exactly that id once the provider answered. The enrollment that consumes such a code assigns its new node to that record in the same transaction, and a record that already belongs to a different node refuses the claim instead of being overwritten. A host can enroll before the provider's answer is written, so a claim that finds no record yet is not a failure: provisioning notes on the code when it wrote the record, and that same write assigns whatever node the code has meanwhile created. Codes an admin mints name no hardware and assign none, and Jump documents written before this link existed are never backfilled.

Jump Codes expire by default after 30 minutes — at most 24 hours when a caller asks for more — and are consumed by the first successful enrollment. A code that is unknown, expired or already used is answered 404 jump code is invalid, expired, or already used, never a script.

POST /jump/claim and Spark's claimjump command remain for the unconverted Swarm hosts that still run a Coreflow daemon: that path consumes the same single-use Jump Code, creates the node record itself and answers with the cluster runtime token those hosts need. No rendered script calls it any more. It retires with the Swarm generation, once the fleet is converted.

Registry and Deploy-On-Push

Cloudly serves an OCI registry under /v2 through CloudlyRegistryManager. The registry uses configured S3 storage and issues OCI tokens from Cloudly authentication state.

For Cloudly-managed services, getServiceRegistryTarget() creates stable registry targets like:

<cloudly-host>/workloads/<service-name>-<service-id-prefix>:<tag>

Registry push hooks record tag/digest metadata on the linked image and service. Legacy services may still use automatic reconciliation unless deployOnPush is explicitly false. The immutable deployment API requires deployOnPush: false and never promotes latest or another mutable reference.

Registry token requests authenticate human passwords against the private credential authority and accept unexpired machine-user tokens. Human OCI sessions carry the generation captured during password verification; Cloudly's SmartRegistry auth provider checks current authority on validation and authorization. Admins retain general write access, except to a release tag a deployment operation reserved: that manifest PUT is refused with 409 release-tag-reserved naming the operation, because an admin digest there would leave the operation unable to record its own. API machine users can obtain a write-capable registry token, but every manifest PUT is separately authorized against one active deployment operation, persisted service grant, exact repository, and exact non-latest release tag.

Registry storage retention

Cloudly keeps its registry's storage bounded (CloudlyRegistryRetention, policy registryRetentionPolicy in ts/manager.registry/registryretention.ts). One pass runs 3 minutes after start, then every 24 hours, or every hour while a pass left work behind or a step failed. Each pass logs one summary line: releases pruned, the most releases one service has within 7 days (no service named, so the line can be exported; it tells whether the age rule needs a count cap), tags removed, manifests and blobs deleted with their bytes, why blobs were skipped (naming the repositories that could not be judged), unreadable manifests, unmapped repositories, failures, and whether work remains.

A release row is recorded per repository root and belongs to the service that recorded the root first; only that service can promote it or roll back to it. The rules that name a release by what runs are judged across every service mapped to the repository (by the repository its pushes are recorded for, its recorded target or its plan). A release is kept when:

• any service's current plan names it (imageDeployment.releaseId) or its repository root (requestedDigest);
• any service that is not immutable (immutableImageRequired) runs a tag (imageVersion) that names its repository root in a repository the service maps to (Coreflow pulls such a service by tag);
• a deployment operation still needs its id or its bound root: one that is not terminal, or one that failed retryably less than 7 days ago;
• an unfinished rollout operation (preparing), adoption image promotion (preparing) or adoption rollout (prepared, rolling-out) names it;
• it is one of its service's 3 most recently promoted accepted releases besides the current one (keepPromotedReleases), the rollback targets rollbackServiceImageRelease accepts; for a service that is not immutable, the 3 most recently observed;
• it was observed or promoted less than 7 days ago (minimumReleaseAgeMs);
• another kept release records the same repository root, or its repository's tags could not be read.

Every other release is pruned, oldest first and at most 200 per pass, one repository at a time: the tags in its repository that name its root digest (except a tag a mapped service runs), the image versions of its release, then its row. Tags go before rows, so a row a stopped pass left without its tags is pruned by the next pass, while no tag outlives its row. The release tag's reservation claim of the operation that pushed it stays, so a pruned version still answers VERSION_COLLISION. A release whose service row is gone is pruned unless a rule across services keeps it; a service being deleted keeps everything until its deletion retires it.

Manifests and blobs are judged from a listing of the whole registry. A manifest is kept when a tag, a release, a current plan, the root a deployment operation that has not finished bound, or an unconsumed legacy attestation reaches it, or when it was written or used less than 24 hours ago (orphanObjectGraceMs), together with everything it references; any other manifest is deleted with its media type sidecar. A blob no kept manifest references is deleted once it was written and last used more than 24 hours ago. Requests that read or push a blob or a manifest record the digests they used (registryRequestDigests), so a client that found a blob present keeps it for that window. Blobs are collected only when every repository could be judged, every deletable manifest is gone, and the process has served for an hour (minimumUptimeForBlobsMs), because a client may find a blob present before a restart and push the manifest naming it after; a manifest PUT that names a blob the registry no longer holds is refused MANIFEST_BLOB_UNKNOWN, so a client pushes it again. At most 20,000 objects are deleted per pass. Nothing depends on what an earlier pass deleted, so a pass that stops at any point is completed by the next. A repository whose tags or manifests cannot be read is named in the summary and skipped, and no blob is collected that pass (blobs are shared by every repository); the others are swept. Cloudly has no operator alert channel yet, so that condition shows only as the summary's warning line.

Every deletion runs under the registry mutation lock every write request takes, in holds that take no new item once 2 seconds passed (lockHoldMs) and always complete at least one. The lock's chain is first in, first out, so the pushes that queued during a hold go before the next one. Each hold judges its items again under the lock: release holds from their repository's facts across every service, manifest holds from what protects their repositories now. Every path that tags, records, plans, binds or attests a root does so under the same lock (a deployment operation binds its pushed root inside the manifest PUT's hold), so a hold sees every binding made before it, and a binding made after it inspects the image and refuses when part of it is gone.

Repositories that no service, release, deployment operation or legacy attestation names (an administrator's push to an unmapped repository) are listed in the summary and kept, together with what their tags reach.

A service deletion and a greenfield cleanup delete only what the service's repository owns: its manifests, then its tags document, in time-bounded holds of the same lock, without reading any other repository and without deleting any blob; the blob sweep is the only collector of blobs, and it collects the ones nothing references any more after their grace window. Each hold checks again that no other service maps to the repository; the first one also refuses while a legacy import of the service is pending (an import that starts later reads its whole source before it copies, so it copies all of it between two holds or refuses). A greenfield cleanup retires the provisional service's registry state only while the operation's reservation holds the repository, and no deletion removes a repository another service maps to by its recorded target, its plan or its derived name.

Safe Immutable Deployment Flow

A deployment runs as a deployment machine user: an API machine user with a bearer token and the grants an administrator attaches to it. An administrator creates one with mutateDeploymentMachineUser (create { username, expiresAt }), or in the dashboard under Access → Deployment machine users. The username follows isDeploymentMachineUsername (lowercase, at most 64 characters, no : or /), is also the registry login name, and must not be held by any other user; pallet-registry- names belong to the registry's runtime credentials and are refused as taken. The token is answered exactly once, on the first answer to create or rotate-token, and expires at most one year after the mutation. A retry of the same mutationId by the same administrator answers the same user with token: null; the same mutationId with another body or from another administrator is refused DEPLOYMENT_MACHINE_USER_MUTATION_CONFLICT. rotate-token { userId, expiresAt } replaces every token of the user with one new token, and revoke-token { userId } removes every token while the grants stay; either ends every identity and registry bearer opened with a removed token at once. getDeploymentMachineUsers lists every deployment machine user with its token expiries and grants and never a token value. A human, an administrator, a cluster identity or a service workload identity is not a deployment machine user and is refused DEPLOYMENT_MACHINE_USER_INVALID; every refusal carries { code, retryable: false }. The retained answer of each accepted mutation lives in cloudly_deployment_machine_user_mutations, without the token.

An administrator uses configureServiceDeploymentMachineUser to attach a persisted grant to an existing service or one exact future service slot. The response confirms the grant but never returns or rotates token material. Routine deployment requests then authenticate as that API machine user; they do not require global administrator authority.

Only a deployment machine user reserves a deployment. reserveServiceDeployment refuses any other verified actor, an administrator included, with DEPLOYMENT_MACHINE_IDENTITY_REQUIRED before anything is reserved, and the reserve-phase getDeploymentPreflight reports the same code as a blocker. Only the reserving machine user's registry push is bound to its operation and records the trusted release evidence promotion requires, so an operation any other actor reserved could never be promoted.

The typed workflow is:

1. getDeploymentPreflight evaluates the requested reserve, promote, or route phase from persisted permissions and current state. It reads the operation and the service first and everything else at once, collects the Corestore inventory only when the request needs a persistent namespace (for a greenfield request from every node of the Pallet clusters, for an existing service from its plan's clusters' nodes), and answers within 25 seconds: past that it answers the retryable DEPLOYMENT_PREFLIGHT_UNAVAILABLE, and a preflight slower than 5 seconds logs one warning with its per-step durations.
2. reserveServiceDeployment durably reserves the service slot, namespace, registry repository, exact release tag, and routes. Replaying the same actor/idempotency key/request returns the original operation; changing the request fails closed. A release tag the service registry already holds a digest for — an admin push outside any operation, or an earlier operation's — is refused with DEPLOYMENT_RELEASE_TAG_OCCUPIED before anything is reserved, and the reserve-phase preflight reports it as VERSION_COLLISION in both modes: an operation learns its image only from its own manifest PUT, which an occupied immutable tag never takes (another digest is refused, and a registry client skips the PUT of a digest the tag already names), so the operation would wait in awaiting-image forever. Reserve a new version instead.
3. For greenfield mode, Cloudly creates an immutable provisional service with deployOnPush: false, a placement hold, no public domains, a service image, and canonical empty Secrets v2 configuration. Existing-service mode verifies ownership and the current rollout-generation fence.
4. The deployer pushes a multi-platform OCI index to the returned repository and exact release tag, logging in to the registry with its username and token. Cloudly records the root index digest and server-created evidence bound to the operation, actor, host, repository, tag, digest, and media type.
5. promoteServiceImageRelease accepts only that trusted release and persists a digest-pinned rollout with a fresh generation. A conflicting active rollout or stale generation is rejected. A greenfield service is placed only by a runtime spec on a Pallet cluster: without one its promotion is refused GREENFIELD_PALLET_RUNTIME_SPEC_REQUIRED, never placed by Coreflow, so state the spec with setServiceRuntimeSpec first.
6. getServiceDeploymentStatus reports rollout identity, counts, health, and per-task observed/reported digest evidence. Route promotion remains fenced until every required replica is healthy and verifies the exact digest. The call also advances the operation: verified runtime evidence moves it to ready-for-route (or to succeeded when it has no routes), and a failed rollout or an exhausted startup grace fails it. readServiceDeploymentStatus takes the same request, the same identity and the same deployment:status capability, and answers the same status from the same evidence, but writes nothing; use it to inspect an operation.
7. promoteServiceDeploymentRoute publishes the requested route, resolves only public addresses, performs an SNI HTTPS request with certificate validation, rejects redirects, and verifies the configured readiness status. Each attempt opens its own TLS connection and never resumes a cached TLS session, so the evidence records the subject alternative names of the certificate the route presented; a connection that exposes no certificate (TLS_PEER_CERTIFICATE_UNAVAILABLE) or a certificate without subject alternative names (TLS_SUBJECT_ALT_NAMES_MISSING) fails at the tls stage. Failed route or TLS checks restore the previous route state and leave structured retryable diagnostics.

A required database or objectstorage capability is evidenced by the service's one enabled Corestore binding of that capability at ready. Corestore's /resources inventory lists /resources/provision resources alone, never a binding's, so it is not the evidence. A binding becomes ready in the transaction that publishes the credential material Corestore delivered for it, or through a status report of its cluster or an administrator (updatePlatformBindingStatus), so ready alone does not prove that material was published. The runtime guarantees it: Cloudly starts no replica of a service before the credential of every enabled Corestore binding reaches its secret manifest, and route promotion requires every replica healthy.

A held greenfield service is issued no credential grant, so its binding's credential is published only once its first promotion releases the hold. That first promote preflight (the service is held and has no rollout yet) therefore accepts the binding while it is requested or provisioning, and reports PLATFORM_BINDING_READINESS_DEFERRED. A binding that adopts a legacy resource must in addition be listed on its node's Corestore legacy inventory as authorized for exactly that binding, read live through the relay; otherwise Corestore would give the successor a fresh, empty resource. The hold stays: activating an adoption re-derives the database password and replaces the bucket secret, so a grant before promotion would cut off the still-running legacy owner. The route preflight still requires the binding ready.

An explicit routes[].proxied boolean is persisted during route promotion and overrides prior provider metadata for the hostname. Historical operations that omit it preserve an existing proxy setting when one is available.

retryServiceDeployment and cleanupServiceDeployment are revision-fenced. Cleanup removes only operation-owned provisional resources and claims; normal service deletion also releases stable deployment claims.

An immutable rollout is scoped by every runtime that owns one of its targets. A service Coreflow owns is scoped by the Swarm identities its online nodes report, and proved by a connected Coreflow whose capability tag names each target node. A Pallet node reports an architecture and carries no swarmNodeId, and no Coreflow answers for its cluster. Each spec owns its scope only while its cluster is in a Pallet-owned phase (switching or pallet); a draining cluster is still Coreflow's until the switch. A service with a Pallet-owned spec is Pallet's alone: no Swarm node of another cluster joins its plan, whatever node records that cluster still keeps (Swarm node records are never retired by liveness), and no Coreflow is asked to vouch for it. A Pallet target takes its cluster and replica count from its spec, its target node names from the nodes of that cluster whose reported architecture states a container platform (x86_64 selects linux/amd64, aarch64 selects linux/arm64), and is refused as image-plan-spec-held while the spec holds the service, as image-plan-node-platform-unknown when no node of the cluster states a platform, and as replica-limit when nodes times replicas per node exceeds the contract's 64. A pinned placement narrows the scope to exactly the nodes it pins, and a pinned node that is no longer a node of that cluster with a platform — retired, moved, or no longer reporting — refuses the plan as spec-node-not-in-cluster rather than quietly planning the rest. The Swarm placement.hold is not read on this path: it belongs to the Coreflow runtime and the cutover leaves it set forever, so a spec-owned service's hold is spec.hold. Whether a target node currently holds a credential and a session is the assignment producer's question, not the plan's: a plan is durable desired state and must not change because a node reconnected.

The Coreflow capability tag is replaced there by the release's own platform evidence. A Coreflow-owned rollout seals its invocation over both platforms, because its plan names hostnames and states no architecture; a spec-owned rollout seals it over exactly the platforms its target nodes report, so a release built for one architecture is a legitimate release for a cluster of that architecture. A platform the release carries no exact evidence for refuses the plan as image-plan-platform-unevidenced, naming the nodes it would otherwise have admitted work their image cannot execute. Adoption stays a Coreflow operation: it takes custody of a deployment Coreflow is already running, so a service with a runtime spec is refused as adoption-runtime-spec-present. Promotion and the promote-phase preflight judge a release by the same rule: its index must carry a runnable manifest for every platform of the service's rollout target, both for a Coreflow scope and exactly the target nodes' platforms for a spec-owned one.

Legacy image import

A legacy deployOnPush service without deployable source moves to a greenfield successor that runs the exact image it ran, without a build:

1. While the legacy replicas still run, an administrator reads the legacy Swarm service and its containers on the hosts that run them and calls captureLegacyServiceRuntimeAttestation with the legacy service id, that hostObservation and, optionally, the expected digest. Legacy hosts run no fleet Coreflow session and no Spark, so the observation is the runtime evidence; it must have been read at most 15 minutes before the capture (2 minutes of clock skew the other way). Cloudly takes the one root every observed replica runs from the service's own repository, requires it whole in its registry and a complete runnable linux/amd64 or linux/arm64 image (one platform manifest, or an index of them), and judges the observation against it: exactly the desired replicas of a replicated Swarm service, each running and healthy or without a healthcheck, each container's image id the root digest on the containerd image store or a platform image configuration digest of the root on the classic store. The legacy tag need not still name the root. It stores the evidence, the observation included, once per legacy service and root (getLegacyServiceRuntimeAttestations lists it). Incomplete evidence is refused and nothing is stored; once the legacy runtime is stopped there is no way to capture it. Until a release consumed the attestation or the legacy service is retired, the registry retention keeps the attested root, whether or not a tag still names it.

2. The successor's deployment machine user, holding an organization-service-slot grant with deployment:import-legacy besides the greenfield capabilities, reserves a greenfield deployment naming legacyImport (legacy service, attestation, asserted root digest) instead of a source revision. Its declared target platforms must be exactly the attested ones. Like every reservation, it is refused as REGISTRY_REPOSITORY_MISMATCH when the repository its name and id derive already belongs to another service, such as the legacy service itself.

   The reservation takes over the legacy service's exact and wildcard route claims in the import's organization; a claim of any other service, or of the legacy service in another organization, still refuses it as ROUTE_UNAVAILABLE, naming the hostname and the owning service. The legacy service keeps serving each hostname until the successor declares it, so reserving cuts no live route. Cleaning up or rolling back the import before promotion hands each taken claim back to the legacy service while that service still exists and is not being deleted; otherwise the claim is released and the hostname falls free. A hand-back that lands while the legacy service is being deleted is withdrawn again, so no claim outlives the service it names.

3. Instead of pushing, it calls importServiceDeploymentLegacyImage. Cloudly copies the attested image from its own registry into the reserved repository and tag: an index as it is, a single platform manifest under a one-descriptor index of the manifest's own family, so the platform manifest keeps the digest the legacy runtime ran. The release records evidence of source cloudly-legacy-import naming the legacy service, the attestation and the source root. No registry push is accepted for such an operation. An index root is copied with every manifest it references, attestation manifests included; a missing manifest or blob refuses the import as LEGACY_IMPORT_SOURCE_MISMATCH before anything is written.

4. Promotion, rollout and route verification continue as for any greenfield deployment: the successor needs a runtime spec on a Pallet cluster before it is promoted. A single-platform image promotes only onto a target whose nodes run that platform, such as a Pallet spec pinned to such a node. Retiring the legacy service afterwards leaves the imported release whole, because its blobs are referenced from the successor's repository.

deleteServiceById refuses a legacy service as SERVICE_LEGACY_IMPORT_PENDING (retryable) while a legacy-import operation that names one of its attestations is still reserving, reserved, provisioning or awaiting-image. Such an import still reads the attested image from the legacy repository, and the deletion removes that repository. The check runs before the service is marked deleting, so a refused legacy service keeps running. It runs again under the delete lease, inside the registry's mutation serialization that deletes the repository; an import reads and copies its source under that same serialization, so one that started after the check finds the repository gone and refuses as LEGACY_IMPORT_SOURCE_MISMATCH instead of copying from a repository being deleted. When the deletion proceeds, it deletes the attestations no release consumed, because without the repository they can never be imported. It keeps every attestation that a release's cloudly-legacy-import evidence names: that is the release's provenance, and the successor's remaining deployment phases judge the import against it again.

tsdeploy import-legacy drives steps 2 to 4.

BaseOS and CoreBuild

Cloudly can manage BaseOS nodes and image builds:

• BaseOS devices register through POST /baseos/v1/nodes/register and heartbeat through POST /baseos/v1/nodes/heartbeat.
• A configured encrypted BASEOS_JOIN_TOKEN system secret accepts generic device enrollment.
• BaseOS image builds create one-time provisioning tokens that are embedded in generated images.
• Cloudly selects a CoreBuild worker based on /corebuild/v1/capabilities and sends the build to /corebuild/v1/jobs/baseos-image.
• Supported build kinds are ubuntu-iso and balena-raw; Raspberry Pi builds use balena-raw.
• Supported architecture values are amd64, arm64, and rpi.
• Completed artifacts are stored in the configured S3 bucket and served through short-lived /baseos/v1/images/:buildId/download URLs.

CoreBuild worker routing uses the public corebuildWorkerUrl or corebuildWorkerUrls settings. A shared credential is stored under the encrypted COREBUILD_WORKER_TOKEN system-secret key; per-worker URL/token entries are stored as JSON under the encrypted COREBUILD_WORKERS key.

Cluster-Scoped Dispatch

Everything Cloudly sends into a cluster goes through the one socket that cluster's relay holds. ClusterManager.findLiveRelayConnection(clusterId) is the single election: a socket carries a relay when the server still enumerates it, it holds the Cloudly identity tag it was verified with, and this process accepted its registration. No live socket is relay-not-connected, and two live sockets for one cluster is relay-ambiguous — never a silent choice between two processes that both believe they serve the cluster.

Node-scoped work names its node: fireRelayRequestForNode(nodeId, method, payload) resolves the node's cluster from the node document, elects that cluster's relay, and puts the node id in the request body. A node Cloudly does not hold, and a cluster whose relay is not connected, are refused before anything is sent. The Swarm-era election — a connection tagged with the node hostnames it served — is gone, together with the readiness probe that waited on it: a cluster relay authors no tags, so those lookups had no author and every node-scoped call through them failed after a full timeout.

Strict config delivery is per cluster for the same reason. prepareClusterConfigDeliveryToRequiredClusters elects each required cluster's relay before the caller's durable mutation, and the returned delivery step pushes on those exact sockets and re-verifies each identity, so a socket that no longer speaks for the cluster it was elected for delivers nothing.

Fleet-wide Coreflow work is scoped by phase. The deployment view, the capability reads and the workspace, logs and shell paths all select their sockets from the clusters whose phase is swarm or draining; a cluster that moved on keeps a socket, because its relay authenticates with the same cluster machine identity, and asking it to serve Coreflow requests would be answered by a runtime that no longer owns the cluster. A cluster Cloudly holds no phase for is not asked either, and is named in the log once per Cloudly process, so a socket that answers nothing is not a silent drop. The config broadcast is not Coreflow work: it reaches those Coreflows and, whatever the phase, the elected relay of every cluster (listElectedRelayConnections, the election of findLiveRelayConnection for every cluster at once, without an ambiguous one), each socket once. A relay adopts every push — the Corestore endpoints it dials, the platform bindings and grants it reconciles — so a node endpoint written while a Pallet cluster's relay is connected reaches it without a reconnect. Each socket's cluster identity is proved once, where the socket is admitted, and every fan-out reads it from there. Identifying an inbound report's connection is unchanged: evidence is checked against whoever proved a cluster identity, not against a phase.

Three Swarm-era paths have no Pallet equivalent yet and say so by name instead of pretending: per-node traffic statistics and resource usage answer node-telemetry-unavailable, service migration is refused at startMigration with service-migration-unavailable because the data move needs the per-node Corestore endpoint, and base-service reconciliation through Coreflow is gone entirely — base services are Pallet system workloads delivered as runtime assignments, so reconcileBaseServices keeps its name and states that, and reconcileNodeBaseServices is removed. Deployment route promotion no longer asks Coreflow to prove its routing; it still makes every promoted hostname certificate-ready before the routes are declared promoted, and a hostname whose certificate is not ready fails the promotion with DEPLOYMENT_CERTIFICATE_NOT_READY.

Cluster Platform Desired State

getClusterPlatformDesiredState answers one cluster's relay with the capabilities, provider configurations and platform bindings that cluster may act on. It carries no identity of its own: the cluster is the one the verified machine identity on the asking socket names, and that socket must also be the one the relay election named. Both facts come from the cluster manager's shared requireElectedRelayConnection, so this handler refuses with the same names every other relay-facing request does — relay-identity-unverified for a socket without a verified cluster machine identity, relay-not-connected for a socket this process holds no accepted relay registration for, and relay-ambiguous when two live sockets claim the same cluster.

The binding list is derived from the Corestore publication grants rather than from the bindings themselves. The relay refuses an ungranted binding with binding-not-granted, so serving a placement Cloudly did not grant would be Cloudly asking for work it did not authorize; deriving one list from the grant computation makes grant-before-serve hold by construction. A binding the grant computation dropped — incomplete authority, a disabled provider, or its own grant cap — is therefore absent from the answer, and the reason is logged with the binding id rather than left to look like a binding that does not exist.

Each remaining binding carries the node its Corestore resources live on. Corestore is node-local, so a binding lives on the one node its service's runtime spec pins in this cluster. The spec states that before any workload runs and keeps stating it while the service is held, so a database is placed before its service first starts -- which is what lets its credential reach the first start -- and stays placed while the service runs nowhere. A service without a spec for this cluster is omitted as binding-unplaced, and one whose spec runs on every node or on several as binding-placement-ambiguous, both by name in the log. Placing a database on "whichever node answered" is precisely what the relay replaced. A binding that adopts a legacy Corestore resource is placed on the node its corestoreLegacyAdoption names, whatever the spec pins, because that node's Corestore holds the data; a node outside the cluster leaves it binding-unplaced. The answer is sorted by node and then by binding id, so two reads compare directly.

The cluster config a relay is pushed carries the publication grants of exactly these placed bindings and no others. A relay retains every binding a node's Corestore holds that it holds a grant for but finds placed on no node, and a retained binding is a permanent tombstone, so a grant travels only with its placement.

The relay is also the one party that reads the Corestore control token (getCorestoreControlCredentialMaterial) and publishes binding credentials (publishCorestoreCredentialMaterial): both are answered to the socket the relay election names and to nobody else, the request's identity must be that relay's cluster, and the material is sealed to the recipient the relay enrolled. The token itself is set by an administrator with setCorestoreControlToken, or with secrets set-corestore-control-token --stdin inside Cloudly's container (see the secrets CLI section). The elected relay also reads Cloudly's secret ingress recipient (getSecretIngressRecipient), which it seals the published material to; a socket that registered a Docker-era Coreflow secret runtime reads it as that runtime, and every other cluster socket must be its cluster's elected relay.

updatePlatformBindingStatus stores the statusReason the responsible cluster reports with a status (legacy-adoption-pending with provisioning, legacy-adoption-conflict and legacy-adoption-mismatch with failed, checked by validatePlatformBindingStatusReason) and clears it on a report without one; an administrator reports no reason. upsertPlatformBinding refuses corestoreLegacyAdoption and statusReason beside objectstorageBucketName: all three are server-managed.

Corestore Endpoints

Corestore is node-local: the database and the bucket of a workload live on the node the workload runs on. While every cluster ran one coreflow per node, "this node" was whoever answered; one relay serving a whole cluster has no such implicit answer, and the cluster-local alias every component used while Docker Swarm resolved it is exactly what stops resolving. Cloudly therefore holds the endpoint per node, in the cluster node record, and the relay reads it.

setClusterNodeCorestoreEndpoint { nodeId; endpoint | null } is the administrator surface, guarded like every other node administration request. The origin is checked by the contract's own validateClusterNodeCorestore before anything is written — a canonical http: or https: origin with no path, query, fragment, userinfo or trailing slash — and the write takes the node's own real-write fence inside its transaction, so a node deleted while an administrator is setting its endpoint conflicts instead of leaving an endpoint behind for a node that is gone. endpoint: null clears it, which states the same fact as never having set one: this node runs no Corestore the relay may reach, and an inventory for it is answered CORESTORE_ENDPOINT_UNKNOWN by the relay rather than by guessing a hostname.

Every refusal is named: Node write refused: inactive for a node Cloudly no longer holds, invalid_input for an identifier no node can have, and conflict when the node or its cluster changed under the fence — a routine heartbeat is enough — so an administrator reads what to do next instead of an internal error.

The endpoint says where, never with what. The bearer control token that authorizes every Corestore call stays with getCorestoreControlCredentialMaterial, sealed to the recipient the relay enrolled, and no credential travels in the node document. What a relay reads is the cluster document, so the node write carries the cluster snapshot with it in the same transaction — the two land together or not at all, exactly as an accepted heartbeat writes them — and the cluster configuration is pushed afterwards, so every connected relay learns the change without reconnecting. The node detail view shows the current endpoint and edits it through the same request.

Object-storage endpoint

A workload dials its bucket at a different origin than the relay dials the control API, so a node states it separately in corestoreObjectStorage { endpoint, region }. setClusterNodeCorestoreObjectStorageEndpoint { nodeId; objectStorage | null } sets it under the same admin guard, fence, snapshot and refusals as the control endpoint, after the contract's validatePlatformObjectStorageEndpoint admits it (a canonical http: or https: origin and a lowercase region). The two requests never touch each other's field. objectStorage: null clears it: no object-storage binding placed on this node can be given its environment, and nothing derives a port, scheme or region in its place. No relay reads this field, so nothing is pushed; the service runtime producer composes from it, so the write asks it to decide again. The node detail view shows it and edits it through the same request; both fields empty clear it.

Database endpoint

A database credential embeds its password, so its host cannot be composed into a workload's public environment afterwards: it is addressed once, when Corestore produces it. A node therefore states where a workload on it reaches its Corestore database in corestoreDatabase { host, port }, and setClusterNodeCorestoreDatabaseEndpoint { nodeId; database | null } sets it under the same admin guard, fence, snapshot and refusals as the other two endpoints, after the contract's validatePlatformDatabaseEndpoint admits it (an IPv4 literal, an IPv6 literal without brackets or a lowercase DNS name, and a port from 1 to 65535). The three requests never touch each other's field.

Every database binding placed on the node — on the node its legacy adoption names, otherwise on the one node its service's runtime spec pins in the cluster, the rule the relay's placements follow — is granted for this endpoint: it is the endpoint of the binding request, and so part of the digest the grant covers. The relay states it to Corestore, Corestore addresses the eleven MONGODB_* and MONGO_* values to it, and Cloudly refuses published material that is addressed anywhere else (validateCorestoreDatabaseCredentialMaterialEndpoint), so a workload is never handed Corestore's own public host. A node without the endpoint gets no grant for its database bindings, and their services are refused database-endpoint-missing. A moved endpoint changes the request of every database binding on the node; a binding whose grant was already consumed is granted again for the moved request, and Corestore re-addresses the same database, user and password. The write pushes the cluster configuration, because the relay reads the endpoint and the grants are computed from it, and asks the service runtime producer to decide again. The node detail view shows it and edits it through the same request; both fields empty clear it.

Upgrading. Roll out Cloudly 33.1.0 first, then Coreflow 33.1.0, then Corestore 33.0.0. Set each node's corestoreDatabase with setClusterNodeCorestoreDatabaseEndpoint before workloads with database bindings are to receive credentials again: until then their bindings get no grant. A binding Corestore recorded before this release gets no material until it is granted again with an endpoint, which happens by itself once its node's endpoint is set.

A workload reaches the endpoint only through the platform endpoints its runtime spec grants (network.platformEndpointIds, matched by its network policy): an attached service with a database binding lists the platform endpoint of its node's Corestore database there, exactly as it lists the one of its object storage.

Corestore Inventory

collectCorestoreInventoryReport asks each node's own cluster relay what that node's Corestore holds, with getClusterCorestoreInventory { nodeId }: one relay serves a whole cluster, so the node an inventory describes is stated in the request and answered in the inventory rather than derived from whichever connection replied. The expected nodes are the cluster's own node records, by node id. A greenfield proof, a legacy import's included, covers every node of every cluster in a Pallet phase (switching or pallet): a greenfield service is placed only by a Pallet runtime spec, which does not exist yet when its namespace is reserved, so any Pallet cluster is where it could land and a Coreflow-owned cluster is not. The reserve preflight and the reservation read this one set (resolveGreenfieldNamespaceTargetNodeIds).

Within that set, nodes are deliberately not filtered by reported status or relay state: a node whose Corestore cannot be proven blank blocks the proof by name, and a stale node record is deleted rather than skipped, because a Pallet node Cloudly still holds is a node a workload could land on.

Every node keeps its own diagnostic: a cluster with no live relay, an RPC that timed out or was refused, an answer the contract refuses, an answer about another node, and a node whose Corestore is unreachable are five different facts. A reachable inventory with an empty service list is the blank-namespace proof; an unreachable one proves nothing and never reads as "nothing provisioned". The greenfield preflight consumes exactly that: it refuses by name when any target node's probe is not ok, instead of swallowing the failure into an empty inventory.

Corestore Legacy Resource Adoption

A Corestore converted from a release before 32 still holds the databases and buckets /resources/provision created per service, and refuses a credential binding beside such a resource until an operator authorizes that exact binding to adopt it, offline.

getCorestoreLegacyResources { clusterId, nodeId } (administrators) reads one node's inventory through its cluster's relay with getClusterCorestoreLegacyResources { nodeId } and answers it with inventorySha256 (computeClusterCorestoreLegacyResourceInventorySha256, null for an unreachable Corestore). The node must belong to the cluster; a relay that is not connected or does not answer is legacy-inventory-unreachable, and an answer about another node or of a shape the contract refuses is an error, never an empty inventory.

adoptCorestoreLegacyResources { clusterId, nodeId, expectedInventorySha256, resources } reads the inventory again, refuses legacy-inventory-changed unless it still has the digest the administrator saw, and hands the selection to the adoption step in ts_migration/corestore-legacy-adoption.ts, which is the only writer of a binding carrying a legacy bucket name and retention intent. In one transaction it creates each adopting binding — enabled, on the canonical Corestore provider and management scope, with corestoreLegacyAdoption { nodeId, resourceName, inventorySha256 }, and for a bucket the legacy objectstorageBucketName and the legacy retention intent exactly — checked with validatePlatformBindingCorestoreLegacyAdoption, or creates none. A resource whose adopting binding exists already is answered unchanged.

A legacy owner adopting its own resource may hold a Corestore binding of the capability already, created by the generic path before its Corestore was converted. That binding adopts the resource in place when it is enabled, not tombstoned, on the canonical Corestore provider, adopts nothing yet, and passes validatePlatformBindingCorestoreLegacyAdoption with the adoption: a bucket binding names the legacy bucket and its retention intent exactly, a database binding carries neither. The adoption is then written onto it in the same transaction, as a compare-and-set on everything that judgement read (PlatformBinding.markCorestoreLegacyAdoption; a binding that changed meanwhile is judged again or refused), its status returns to requested, and the plan names its id. Any other binding the adopting service holds refuses binding-exists, a successor's always.

A selection entry may name successorServiceId: a fresh service that replaces the legacy owner, created by its greenfield reserve, takes over the owner's database or bucket. The binding is then created on the successor, with corestoreLegacyAdoption.legacyServiceId naming the owner, and the plan entry carries the successor. The successor must exist, belong to the owner's organization, be an immutable-deployment service (deployOnPush: false, immutableImageRequired: true), own no legacy resource of the capability and hold no other binding of it. Adoption therefore runs between the successor's reserve and its promotion. Once Corestore activates the successor's binding, the legacy owner's credentials stop working and its manifest entry is gone.

Once the successor holds the resource, the legacy owner can be deleted. deleteServiceById tombstones the owner's own Corestore bindings, which its legacy resource kept from ever starting, only when every CorestoreLegacyAdoptionClaim naming the owner for that capability names a live binding of its successor adopting exactly the claimed resource. That binding's node must also answer, through the relay, one of two things: the resource is authorized for exactly that binding, or the owner no longer holds it because activating the binding removed the owner's entry. A claim whose binding is gone, an adoption still waiting for the offline migration, an unreachable Corestore and an owner that adopted its resource itself all keep the owner. Deleting the owner never touches the successor's binding, its claim or the adopted resource.

A legacy resource is adopted by one service only: a binding of another service that Cloudly holds as the resource's adoption refuses the selection. Every created binding also writes the resource's CorestoreLegacyAdoptionClaim, an id derived from node, capability and resource name, in the same transaction (so does a binding adopted in place), so of two concurrent adoptions of one resource one commits and the other rolls back whole. Every new adoption is logged with its binding, both services, node and inventory digest.

The refusals are ICorestoreLegacyAdoptionErrorData: legacy-inventory-unreachable (the only retryable one), legacy-inventory-changed, legacy-resource-absent, legacy-resource-blocked, legacy-resource-authorized-elsewhere, legacy-resource-adopted-elsewhere, service-absent (the owner or the successor), binding-exists, successor-organization-mismatch, successor-not-greenfield and successor-owns-legacy-resource. The answer carries the bindings and the plan (createCorestoreLegacyResourceAdoptionPlan), exactly the file Corestore's offline legacy-resource-adoption migration reads, and the cluster config is pushed so the relay places the bindings on nodeId. Until the operator runs that migration, the relay reports each binding provisioning with legacy-adoption-pending, and its service waits as legacy-adoption-pending.

The relay handler getClusterCorestoreLegacyResources belongs to Coreflow 33; a relay without it answers the read as legacy-inventory-unreachable.

Backups and Corestore

Cloudly owns backup records and user-facing backup/restore requests. Coreflow executes the cluster-local work, and Corestore snapshots volumes, database resources, object storage resources, and archive objects.

The backup path includes:

• createServiceBackup and restoreServiceBackup typed requests for admins.
• backupClusterService and restoreClusterService requests from Cloudly to the cluster relay, each naming the node whose Corestore holds the data.
• backupServiceDatabaseClosure and restoreServiceDatabaseClosure requests from the cluster relay, which stream a service's database closure to Cloudly and back (see Database Closure Streams).
• Corestore volume/resource snapshot and restore endpoints behind the relay.
• Optional archive replication through prepareBackupReplication, uploadBackupArchiveObject, completeBackupReplication, getBackupArchiveManifest, and downloadBackupArchiveObject.
• Optional scheduled backup-all-services task when CLOUDLY_BACKUP_CRON is set.

Corestore data is node-local, so a backup or a restore must name the node whose Corestore holds it: a request that names none is refused by name (backup-node-required) instead of being sent to whichever socket of the cluster answered first. The named node is resolved to its cluster node id at the request boundary — a name that matches no node, or more than one, is refused with node-unresolved rather than reaching an arbitrary node's data — and the dispatch then travels over the relay of that node's cluster. Archive pruning is per node the same way, through pruneClusterNodeArchive, and stays best-effort: a node whose relay is gone keeps its cache until the next pass.

Manual createServiceBackup requests require clusterId when Cloudly manages multiple clusters; it may be omitted only when exactly one cluster exists. In the cluster-scoped flow Cloudly authors the replication record itself, from the database closure it accepted, and the relay no longer does (see Database Closure Streams). The archive replication requests still validate archive object size and SHA-256 checksums, write a manifest, record target metadata, and mark completed backups as replicated. Restores read the manifest and objects back through the configured target writer.

Database Closure Streams

A cluster is outbound-only, so Cloudly never reads a cluster's Corestore: both directions of a database closure are relay-initiated. backupServiceDatabaseClosure carries one service's closure from the relay to Cloudly as a VirtualStream, and restoreServiceDatabaseClosure serves it back. Neither request carries an identity — the cluster is the one the asking socket authenticated as and was elected relay for, through the same requireElectedRelayConnection the desired-state request uses — and both name the node, because Corestore is node-local.

Every refusal of these two requests reaches the relay by name as a TypedResponseError. Its error.text is Cluster relay operation rejected: <code>. for a code of the relay vocabulary and Closure stream refused: <code> for a code of the closure vocabulary:

Vocabulary	Code	Meaning
relay	invalid	The request has no live request context, or the asking socket changed while it was verified.
relay	relay-identity-unverified	The asking socket carries no verified cluster machine identity.
relay	relay-not-connected	The asking socket is not its cluster's elected relay.
relay	relay-ambiguous	More than one live socket claims the relay's cluster, so Cloudly serves none of them.
relay	relay-node-foreign	The named node is a node of another cluster.
relay	relay-node-unknown	The named node does not exist.
closure	closure-request-invalid	The backup, service or node id is malformed.
closure	closure-backup-unknown	No such backup for this cluster and service; another cluster's backup reads the same, so no relay can probe backup ids.
closure	closure-not-awaited	The backup is not waiting for a closure: it already holds one, one is being received, or the backup is no longer running.
closure	closure-descriptor-invalid	The descriptor fails the contract's own validateCorestoreDatabaseClosureDescriptor.
closure	closure-descriptor-foreign	The descriptor's receipt names another backup or service than the backup it is announced for or held under.
closure	closure-not-held	A restore asked for a closure Cloudly does not hold, or whose manifest fails its checksum.
closure	closure-unavailable	Cloudly cannot take or serve closures right now: the cache tier cannot be opened or cannot create exact-length objects, a held manifest cannot be read, or the backup manager is stopping.
closure	closure-size-mismatch, closure-digest-mismatch	Never an answer: bytes that disagree with the descriptor fail the stream itself, and the backup records the code.

The code lives in ts/manager.backup/closurestreams.ts (ClosureStreamServer: the two handlers, stream lifecycles, the cache-tier store and the stream authorizations) and ts/manager.backup/closurerecords.ts (ClosureBackupRecords: which backup a closure belongs to and what storing it means for that backup's record). The backup manager wires them and only decides the backup's state.

Cloudly stores the closure while it streams: the bytes pass a counting, hashing transform straight into fastPutStreamExact at the exact length the descriptor announced, so the acceptance after the last byte is immediate and fits the relay's 120 s acceptance window. A closure whose length or digest disagrees fails inside that transform before its end reaches the store, and an exact upload publishes nothing whose source did not end cleanly, so a wrong closure is never stored at all. One backup takes one closure: the backup id is reserved before its record is read, so a concurrent or later announcement is refused closure-not-awaited and can never replace or remove a closure the backup holds.

The contract's descriptor validator proves shape and self-consistency only, so Cloudly also binds the descriptor to its backup: the receipt's backupId must be the announced backup, and its sourceServiceId that backup's service. Cloudly checks this before anything is stored, and again before it serves a stored descriptor back, because a closure of another service held under this backup would restore that service's database into this one.

The backup manager opens and probes the cache tier when it starts. A relay-minted stream expires unless it is opened within the transport's 10-second capability deadline, and opening and probing a cache tier can take longer than that, so the first closure after a start does not wait for it. A failed open is logged and forgotten; the next announcement opens the tier again.

The backup record is the commit point. Cloudly stores the closure object, writes the closure manifest (<backup prefix>/closure/manifest.json), and then commits the record with a conditional update that only a backup still running without a replication record takes; the stream is accepted only after that commit, so the receipt the relay reads is never ahead of the record an operator reads, and an acceptance lost after the commit leaves Cloudly holding a closure it recorded, which it logs. A commit the record does not take removes the manifest and the object again. A record write that reported an error may still have applied, so Cloudly reads the record back: a record that refers to the closure manifest keeps everything and the stream is accepted, and one that does not has the manifest and the object removed. When that read fails too, the outcome is in doubt: the stream is refused and everything is kept, because removing a closure a committed record refers to would lose the backup. Deleting a backup that holds no replication record removes its prefix on the configured cache tier, so whatever no record took goes with the backup.

On a restore the relay asks Cloudly first, before any node-local step, so a backup Cloudly holds no closure for is refused closure-not-held from a manifest lookup that opens no stream at all — that ordering is what keeps a node from being half restored. The descriptor travels with the stream because Corestore opens its restore from it, and Cloudly declares the stream's integrity from the same descriptor. The served stream's authorization binds only on the asking socket and is revalidated on every framed operation, so it ends when that socket goes away or the backup manager stops.

Cloudly is the replication target of the cluster-scoped backup flow, so it authors the replication record itself from the closure it accepted; the relay's answer to backupClusterService decides no replication record. Admission answers before the bytes move, so after the relay answers Cloudly first waits for that backup's own closure transfer to settle (never for another backup's). An accepted closure makes the backup replicated with a record of one object of exactly the closure's size, and a manifest that states the closure, its node and its descriptor and lists no archive objects — volumes and object storage have no streamed contract yet, so nothing claims they were replicated. A backup that asked for replication, produced a database snapshot and holds no closure is failed with database closure not replicated: <observed>, where <observed> is only what Cloudly saw:

• no closure was received for this backup — Cloudly admitted no announcement for this waiting backup: either the relay announced none, or Cloudly refused one before it resolved the backup, which Cloudly logs under the backup id once it has parsed the announcement, as described below;
• closure-descriptor-invalid: <reasons>, closure-descriptor-foreign: <the backup and service the receipt names> or closure-unavailable: <cause> — Cloudly refused the announcement after it had resolved the waiting backup, for example because the cache tier failed its exact-upload probe, and logged the cause under the backup id;
• closure-stream-reset — the stream ended before the closure was stored. The transport resets with a constant text, so the relay's reason stays in the relay's log under the same backup id; a reset Cloudly caused because its storage failed is logged by Cloudly under that id;
• closure-size-mismatch or closure-digest-mismatch — the bytes were not the announced ones;
• closure-unavailable: <cause> — the closure was stored, but the record did not take it and it was removed;
• closure-unavailable: its commit could not be read back — the commit's outcome is in doubt, and the closure is kept as described above.

A refusal made before the waiting backup is resolved records nothing, because no backup has been proven to be the one waiting: every refusal of the relay vocabulary, closure-request-invalid, closure-backup-unknown, closure-not-awaited, and closure-unavailable for a manager that was already stopping. Once Cloudly has parsed an announcement it logs every refusal of it under the backup id the announcement named, so closure-backup-unknown, closure-not-awaited, relay-node-foreign and relay-node-unknown leave a trace. A refusal made before the parse — invalid, relay-identity-unverified, relay-not-connected or relay-ambiguous for the asking socket, closure-request-invalid, or a manager that was already stopping — is not logged under a backup id, because none was parsed. Only a backup that is still running takes a failure record, so a backup that already finished keeps its own reason, and a later attempt that stores the closure clears what an earlier one recorded.

A backup replicated as a streamed closure stays on the cache tier: moving one to the external tier would read it whole into memory, and no streamed external-tier contract exists yet.

Stopping the backup manager refuses new transfers, waits for admissions still deciding, and drains the admitted transfers — a received closure may still record what it stored, because the archive writers close only after the drain. Nothing opens the cache tier once the stop began, so the stop closes the one store there is. A restarted manager takes closures again.

The server installs one VirtualStream authorization adapter over several owners (ts/virtualstreamdispatcher.ts): the image manager owns image download authorizations, the backup manager owns closure ones, and each owner binds only what it minted. An authorization nobody claims is refused stream-authorization-unclaimed rather than as one feature's denial, and one claimed by two owners is refused stream-authorization-ambiguous, naming both, instead of being guessed.

Task Automation

Cloudly registers a TaskBuffer-backed task manager. The API and dashboard can list tasks, trigger enabled tasks manually, inspect execution logs/metrics, and request cancellation for running tasks. Disabled tasks cannot run manually or on a schedule.

Predefined tasks currently include:

Task	Status	Purpose
dns-sync	Disabled	Iterates DNS entries marked as external; provider sync is currently a placeholder.
cert-renewal	Disabled	Checks activated domains for certificate renewal; renewal logic is currently a placeholder.
cleanup	Disabled	Removes old task executions and contains placeholders for log/image cleanup.
resource-report	Disabled	Generates node resource metrics; values are currently placeholders until runtime metrics are wired in.
db-maintenance	Disabled	Maintenance shell for database optimization tasks.
security-scan	Disabled	Security scan shell for exposed ports, image freshness, and weak configuration checks.
docker-cleanup	Disabled	Docker cleanup shell for containers, images, volumes, and networks.
backup-all-services	Conditional	Registered by the backup manager and enabled only when CLOUDLY_BACKUP_CRON is set.

External Gateway Integration

Cloudly can integrate with a dcrouter gateway when the public gateway settings and encrypted gateway system secret are configured. The current integration syncs externally available domains into Cloudly, syncs public service routes directly to dcrouter, and fetches or reprovisions certificates through dcrouter. Coreflow consumes the resulting Cloudly service configuration and keeps cluster-local Coretraffic routing focused on internal workload traffic. A service a Pallet runtime spec owns is routed to the uplink address the one node its spec pins reported, where that cluster's ingress and its own published ports answer; a Pallet spec that pins no single node, or a pinned node that reported no uplink, refuses the sync by name. The gateway client also holds Cloudly's cluster relay names (see Cluster Relay Names).

Service mail can also be configured on a service. Each address can enable inbound mail through a dcrouter smtpForward target to a published service port and outbound mail through a dcrouter-managed SMTP credential. Cloudly stores only public credential metadata in service.data.mail; one-time SMTP secrets are written as service-owned Secrets v2 entries with sanitized address-token variables such as MAIL_PLATFORMTEST_GATED_ONE_SMTP_* plus default SMTP_* aliases for the selected default sender.

Gateway Credential Bootstrap

Cloudly enrolls itself at dcrouter as one least-privilege gateway client, and the only way a gateway credential enters it is a dcrouter admin token an administrator hands in once, with the admin request bootstrapExternalGatewayCredential (@serve.zone/interfaces 32.29, requests.settings). The request names the target it means, expectedGatewayUrl and expectedGatewayClientId as getSettings answers them, and carries the token sealed to Cloudly's active secret ingress recipient under data.createGatewayBootstrapEnvelopeContext(target), so the token opens only for a request naming that same target. Cloudly compares the target with its own gateway settings before the token leaves it, asks dcrouter whether the token is an admin token, and provisions, finalizes and stores its own gateway-client credential under the policy its settings and the runtime network address plan state now. The admin token serves that one call; Cloudly never stores or logs it.

The answer is the enrollment that results, data.IExternalGatewayEnrollmentState: ready once the new credential is Cloudly's runtime authority, and the gateway legs (domains, routes, mail, CoreMail, Web Push, the Coreflow push) run again without a restart. When dcrouter issued the candidate but cannot be reached to finalize it, the answer is degraded with finalizationPending: true, and Cloudly's own retry finalizes it. A refusal leaves the held credential unchanged, and one decided before provisioning leaves the enrollment state as it was too. It is one of data.externalGatewayBootstrapRefusals, answered as External gateway bootstrap refused: <refusal>.:

• gateway-unconfigured — no dcrouterGatewayUrl or dcrouterGatewayClientId is set.
• gateway-target-mismatch — the named target is not the configured one.
• bootstrap-token-invalid — the request or its envelope does not hold for the active recipient and the named target, or what it opens to is not 1 to 1024 bytes of visible ASCII.
• bootstrap-token-rejected — dcrouter does not accept the token at all.
• bootstrap-token-not-admin — dcrouter accepts the token, but its role is not admin.
• bootstrap-candidate-pending — a candidate from an earlier provisioning still awaits finalization; the retry finalizes it, then bootstrap again if it is still needed.
• gateway-unreachable — dcrouter could not be reached before provisioning; repeat the call.
• bootstrap-provisioning-refused — dcrouter refused provisioning, or the credential it issued failed Cloudly's validation.

getExternalGatewayEnrollmentState({ identity }) answers the current enrollment. Both requests are administrator-only; reason is Cloudly's own text, at most 4096 characters, and names the request to use whenever the state is admin-bootstrap-required. @serve.zone/api (32.1+) sends both from its settings group, and the servezone CLI of @serve.zone/cli wraps them as servezone gateway bootstrap and servezone gateway status (see ts_cliclient/readme.md); the token is read from stdin only. Cloudly's own secrets CLI does not carry the request: it runs beside the database rather than as an administrator.

A gateway settings update (updateSettings) is validated first: settings the gateway rules refuse outright, such as a client id without a URL, are refused and nothing is stored. Otherwise it depends on what Cloudly holds:

• No gateway credential at all (no stored credential and no candidate in flight): the settings are stored as given and the enrollment turns admin-bootstrap-required.
• A credential that carries the new settings: Cloudly reconciles with it and stays ready.
• A credential that cannot carry them, because the gateway client's policy or target changed: the settings are staged. The running gateway client keeps serving on its current settings and credential, getSettings answers the staged settings, and the enrollment turns admin-bootstrap-required with a reason naming each staged key, from and to (Staged gateway settings change dcrouterTargetHost "a" -> "b". ...). The next bootstrap provisions against the staged settings, and its expectedGatewayUrl and expectedGatewayClientId are the staged ones; on success it switches settings and credential together, and a refusal leaves the change staged. A later update replaces the staged change, and an update that states the running settings again discards it (the reason lists them; an empty string clears a key that was unset).

For the first enrollment, a gateway settings change, and whenever the enrollment answers admin-bootstrap-required (after the runtime network address plan moved the gateway client's DNS record allowance, for instance):

1. Commit the runtime network address plan (setRuntimeNetworkAddressPlan).
2. For a first enrollment or a settings change, store the gateway settings with updateSettings (a first enrollment needs dcrouterGatewayUrl and dcrouterGatewayClientId); the enrollment answers admin-bootstrap-required, and a change stays staged until step 4.
3. Mint a short-lived admin token at dcrouter.
4. Hand it to Cloudly: servezone gateway bootstrap --gateway-url <dcrouterGatewayUrl> --client-id <dcrouterGatewayClientId> < token-file.
5. Run servezone gateway status until it answers ready.
6. Revoke the admin token at dcrouter.
7. Enable the cluster relay (enableClusterRelay).

Development

Common commands:

pnpm install
pnpm build
pnpm test
pnpm run build:docker
pnpm run release:docker
pnpm run docs

Important paths:

Path	Purpose
ts/index.ts	CLI/runtime entry point exporting runCli, Cloudly, and ICloudlyConfig.
ts/classes.cloudly.ts	Main service coordinator and startup order.
ts/classes.server.ts	API/dashboard server, registry bridge, and BaseOS HTTP routes.
ts/manager.*	Domain managers for auth, clusters, services, images, registry, platform, backups, BaseOS, and more.
ts/connector.*	The MongoDB connector.
ts_web/	Browser dashboard web components.
ts_cliclient/	@serve.zone/cli submodule, published with every Cloudly release at the Cloudly version.

Accuracy Notes

The package metadata and settings schema include fields for several cloud providers. The code paths currently exercised in this repository are Hetzner for selected node/bare-metal provisioning paths, S3-compatible storage, SMB/S3 backup archive targets, MongoDB/SmartData, CoreBuild, Coreflow, Corestore, and dcrouter, which holds every DNS record and certificate Cloudly needs. Several provider connection tests and predefined tasks are configuration checks or implementation shells; verify provider-specific behavior in the relevant manager before relying on it operationally.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.