@serve.zone/spark

Spark is a Deno-powered server management agent for serve.zone hosts. It installs as a system daemon, activates Docker Swarm, schedules host/service maintenance tasks, and provides the bootstrap profiles currently used by Cloudly and Coreflow node deployments.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Current Role

Spark is intentionally small and operational. It is not a general-purpose configuration management framework; it is the serve.zone node-side utility that knows how to run itself as a daemon and keep selected Docker services moving.

The current implementation:

• Installs and updates a smartdaemon_spark systemd service through @push.rocks/smartdaemon.
• Runs in an explicit mode: cloudly or coreflow-node.
• Activates Docker Swarm through @apiclient.xyz/docker when daemon mode starts.
• Collects host metrics for CPU, memory, disk, and Docker container counts.
• Sends Cloudly heartbeats when cloudlyUrl, nodeId, and nodeToken are configured.
• Sends authenticated Swarm observation v2 reports after accepted Cloudly heartbeats.
• Schedules recurring tasks with @push.rocks/taskbuffer for Spark updates and managed Docker service updates.
• Optionally schedules host package updates when --enableHostUpdates=true has been configured.

Spark also owns a callable Swarm observation v2 producer. Every Swarm member captures a bounded, coherent local container and physical-mount census between matching before/after container sets. Active managers additionally capture the same bounded /services?status=true, /tasks, and /nodes view, including canonical service-spec, constraint, inventory, and mount-correlation identities. Incomplete or changing census branches are omitted instead of claiming evidence that Docker did not provide. Worker observations never invent the manager-only Swarm cluster ID. Heartbeat transport owns a separate private reporter session, validates bound Cloudly receipts, retries only the exact pending request after ambiguous or retryable delivery, and quarantines terminally rejected sessions until process restart.

Installation

A serve.zone node runs the complete Linux node bundle. Install it from the release's own installer with one exact version:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version>

That mode is described under Complete node bootstrap; a host that still runs the released Swarm daemon is converted in the order under Cutover to the node runtime.

Released standalone executable

The default installer mode installs the released Swarm-era executable and nothing else. It remains the line for a host that has not been converted yet, and it installs no node bundle, no node service and no enrolled identity.

Install a released binary:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash

Pin to the newest release within a supported major channel:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --major 1

Verify a selected release asset without changing the host:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | bash -s -- --version <version> --verify-only

The installer downloads SHA256SUMS.txt and verifies the exact platform binary before stopping services or replacing an existing installation. Release builds currently target Linux x64/ARM64, macOS x64/ARM64, and Windows x64.

On systemd hosts, upgrades preserve smartdaemon_spark.service and its enablement. Only a previously active daemon is stopped and restarted; inactive or failed daemons remain stopped. An existing noncanonical spark.service, uncertain service state, or a command symlink pointing at another installation stops the installer before binary replacement. Resolve that ownership explicitly before retrying; the installer does not delete or adopt another service.

The verified executable is staged on the installation filesystem and renamed over the old binary. Other installation contents and configuration are left alone. An existing executable is retained at the backup path printed during installation. A failed restart returns a nonzero exit status with the new binary still installed; inspect the daemon before choosing recovery. Automatic binary rollback is not attempted because a started version may have migrated state. A completed bare installation prints the node-bootstrap order for the version it just installed; the released line has no daemon-installation command to advertise.

Complete node bootstrap

The node integration after 1.12.0 adds --node-bundle and installnodebundle; use a verified release containing these commands. On a Linux systemd host, run the release's installer with one exact version:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version>

The transition from 32.0.1 must use the qualified 32.1.1 installer with the exact 32.1.1 version in both the download URL and --version. Spark 32.0.1 requires the legacy manifest digest in the editable release body, so its installed binary cannot resolve a bodyless journaled release itself. The 32.1.1 installer verifies the standalone 32.1.1 binary and uses that binary to fetch and verify the complete bundle. The 32.1.0 tag and frozen journal bytes are retained, but its release is a permanently withdrawn draft because its compiled payload included checkout-local dependency state; never install or resume it. Later journaled releases use the normal bundle client and do not repeat the private direct-upload exception used to recover 32.0.1.

This mode runs the verified standalone executable from disposable staging to download, verify and select the complete native bundle at /opt/spark/current. It preserves the old /opt/spark/spark executable, global CLI link, unit files, mask, enablement and node data. Existing root-owned 0755 code roots retain their mode; new roots and owned release/seal directories use 0700.

The canonical service must be masked and quiescent, or absent, with no alternate spark.service, drop-ins or stale manager configuration. If NeedDaemonReload is set, explicitly reload systemd and re-prove this admission before retrying. Bootstrap never reloads, unmasks, enables or starts a service. It requires a pinned version and rejects --major, --install-dir and --verify-only.

A verified external executable can also run installnodebundle X.Y.Z directly. The SPARK_NODE_BUNDLE result records the selected version, commit and manifest digest. On a host that already has a selected bundle the same command, and the same installer invocation, updates it instead; see Updating a node bundle. Bootstrap itself still refuses with selection_changed only when a competing installer selects a bundle while it downloads. Successful bootstrap means code is selected. Credential enrollment through enrollnode and definition installation through installnodeservice remain explicit steps, and production activation requires the separate rollout acceptance.

Cutover to the node runtime

A host that still runs the released Swarm daemon is converted in this order, one step at a time, each step verified before the next is run:

systemctl mask --now smartdaemon_spark.service
curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version>
/opt/spark/current/spark enrollnode   # {"operation":"adopt-existing","hostname":"<hostname>"} on stdin
/opt/spark/current/spark installnodeservice --forwarding-mode docker-shared   # or exclusive; see Host forwarding mode
systemctl unmask smartdaemon_spark.service
/opt/spark/current/spark activateguard
systemctl start smartdaemon_spark.service

What each step proves, and what it never does:

1. Mask and stop. The operator's own step, and the only one before the start that changes what runs. It leaves the released daemon unable to be started by anything else while the host converts, which the Spark steps below require: bootstrap and adoption admit this host's canonical unit only masked and quiescent, service installation requires it quiescent, and guard activation admits no active Spark runtime. Docker services on the host keep running; masking Spark removes nothing it created.
2. Bootstrap the bundle. Proves that the exact release's complete native bundle is verified and selected at /opt/spark/current, recorded by version, source commit and manifest digest. It preserves the old /opt/spark/spark executable, the CLI link, unit files, the mask, enablement and node data, and it never reloads, unmasks, enables or starts a service.
3. Adopt the existing node. Proves that this host holds its own current Spark and Pallet credentials in its protected stores. It requires the canonical unit masked and fully stopped, reads the released configuration file exactly once through the versioned migration, and leaves that file's bytes unchanged. It changes no service enablement and starts no old runtime. Adoption enrolls a new Spark credential generation for the same node, after which the released daemon's old credential is rejected.
4. Install the node service. Proves that the units which will boot this host are exactly the canonical definitions, by digest: the node unit that executes /opt/spark/current/spark runnode --forwarding-mode <mode> for the host's stated forwarding mode, and Pallet's containerd unit smartdaemon_spark-containerd.service that executes /opt/spark/current/pallet/pallet-control containerd-serve, requires and starts after the guard unit, and is wanted by and ordered before the node unit. It preserves masks, enablement and previous generated definitions, and it never starts, stops, enables or unmasks anything.
5. Unmask, then activate the guard. The unmask is the operator's own and starts nothing; it is what makes the just-installed node unit admissible to activation. Activation admits that unit only loaded, enabled or disabled, fully stopped, without drop-ins or a pending reload, and requiring and ordering itself after the guard — a masked unit is none of these. An already enabled unit may start on reboot once unmasked, so do not leave a reboot gap here. Before a guard exists, activation first proves the node's runtime routing binding: Pallet's network acquisition dials the relay that binding names and has no other route, so activation reads the binding from Cloudly once and binds it inside Pallet's enrollment child, the same proof the node runtime makes on every start. Cloudly answers the binding only once the node's cluster relay is live, so convert a host only after that. Activation then proves that the retained allocation-pool guard is commissioned, its boot unit installed and enabled, and both dependency edges to networkd present. An active Spark runtime, an alternate network manager, a pending provider reload or an administrator drop-in prevents activation. It publishes no positive network barrier and releases no allocation.
6. Start. The operator's last step, and the second one that changes what runs. Systemd selects the installed local-vendor definition; verify the exact process and its readiness immediately. Starting the node unit first starts the containerd unit it wants: Pallet's containerd-serve verifies the bundle's pallet-containerd/ release, starts Pallet's own containerd on /run/pallet/containerd/containerd.sock with Pallet's complete configuration, imports the bundled pause image and reports ready only once the CRI runtime answers; the node runtime is its only client. Docker and its own containerd keep running beside it, untouched — Pallet refuses their sockets by path and by file identity. The start reads back the runtime routing binding that guard activation bound, offline. See Foreground node runtime.

A host that runs the cluster relay or Corestore installs them as host services between steps 4 and 5, with the node unit still stopped: hostservice configure, set-credential and install for each, then hostservice activate relay (Cloudly answers the routing binding that guard activation proves only once the relay is live) and hostservice activate corestore. A relay or Corestore still running under Docker holds the same S endpoints, so stop it first; see Host services.

None of the Spark commands above stops, removes or adopts a running service, and none of them deletes the released configuration source. The three systemctl steps are the operator's own, and only the mask and the start change what runs.

A host converted under Spark 33 reinstalls its units. Spark 34 adds Pallet's containerd unit and changes the node unit's bytes (it now wants and starts after that unit). Until installnodeservice runs again, such a host's installed node unit is not the canonical definition, so its boot identity reads as unconverted and asdaemon no longer refuses by name. After selecting the Spark 34 bundle, stop the node unit and rerun /opt/spark/current/spark installnodeservice --forwarding-mode <mode>, then activateguard and the start as above. A host enrolled fresh under Spark 34 needs nothing extra.

The host services release changes the node unit's bytes again: it now wants and starts after the relay and Corestore units. updatenode replaces a changed node definition while the unit is stopped, so a node updated with it keeps the canonical definition; a node whose bundle was selected another way reruns installnodeservice with the node unit stopped.

Spark 35 changes them once more: the command line states the host's forwarding mode (runnode --forwarding-mode <mode>) and the unit keeps Pallet's router namespace in its file descriptor store. A unit of Spark 34 states no mode, so the first update to Spark 35 needs it once: --forwarding-mode docker-shared on a host that runs Docker (see Host forwarding mode).

After the cutover, spark asdaemon refuses to start on this host by name (released_mode_retired). The refusal is decided from this host's own boot path — a verified bundle together with an installed unit whose bytes are the canonical runnode definition of either forwarding mode — and it happens before the released configuration is written and before any managed service is pushed, so no Coreflow owner session is opened and no Docker secret is created. Nothing running is stopped or removed by the refusal: the command only declines to own the host.

The released configuration source is retired separately, and only after adoption has consumed its credential. The node's own record states it retained-inert at generation one and removed at generation two, after the exact imported bytes are unlinked; a replay advances nothing and a removed source can never become a retained one. See Store migrations; no command invokes that retirement yet.

Spark supervises the relay; the relay credential still comes from its holder. The released owning mode wrote the Coreflow relay's launch environment as a Docker secret containing JUMPCODE. The current relay authenticates with SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, which a relay running as a Spark host service reads from a systemd credential (see Host services). Spark never fetches that bearer: its holder exports it from Cloudly (cloudly secrets export-cluster-relay-authorization) and hands it to spark hostservice set-credential relay SERVEZONE_CLUSTER_RELAY_AUTHORIZATION on stdin. Converting a node neither moves, restarts nor revokes a relay already running on it.

Cloudly is upgraded first, always. Publishing a Spark release is the fleet rollout: a node with a running released daemon checks for a newer release every ten minutes and installs it, with no channel pin and no per-node gate (updateSpark under Daemon Behavior). Central Cloudly must therefore consume the @serve.zone/interfaces release a Spark build requires, and be deployed, before that Spark release is published. Spark offers its two session kinds under separate minimums for exactly this reason, and the Swarm minimum is raised only in the release that starts requiring the newer Cloudly; a Cloudly that is older then refuses the Swarm offer for every self-updating node at once. See Protocol handshake.

Host services

A node bundle also carries the platform services a fleet host runs beside Pallet, which Spark installs and supervises as systemd units:

Service	Unit	Component (node recipe)	Bundle directory
relay	smartdaemon_spark-relay.service	coreflow: Gitea TsPack coreflow-relay of serve.zone/coreflow	coreflow/
corestore	smartdaemon_spark-corestore.service	corestore: Gitea TsPack corestore-host of serve.zone/corestore	corestore/

The build requires binary/node-build.json to pin each by version, source commit and manifest SHA-256 under coreflow and corestore, exactly like Pallet, and refuses a recipe without both (Node recipe must pin the coreflow and corestore host service releases); the node notices record both manifest digests and are regenerated with every pin change (scripts/node-notices.ts). This release pins Coreflow 33.4.6 (source f4ec44d, manifest e4e24c7a…) and Corestore 34.1.1 (source 9c726c2, manifest 57f5edc2…). The build verifies each release before it packs it: every file the component declares with its mode, no other executable, and every *.tsrust-build.json naming its own sibling executable, built clean for the target. A node bundle of this release carries both; a selected bundle of an earlier release carries neither and is still read, so a node updates from it. Neither is started by an update that installs it.

What each component must carry (ts/bundle/hostservice.inventory.ts):

Component	Files
coreflow	coreflow (the relay, 0755), qenv.yml, coreflow-vpn (0755) with coreflow-vpn.tsrust-build.json, coreflow.third-party-notices.txt and .json, license, and the smartvpn notices under notices/
corestore	corestore (0755), its engines corestore-nosqldb, corestore-objectstorage and corestore-containerarchive (0755) each with its .tsrust-build.json, third-party-notices.txt and .json, license.md, and each engine's notices under notices/

Corestore selects each engine by its exact path beside its executable and refuses to start when one is missing or unusable, so the bundle carries all three or the release is not installable.

The operator's commands, one service at a time:

/opt/spark/current/spark hostservice configure relay < relay.json
/opt/spark/current/spark hostservice set-credential relay SERVEZONE_CLUSTER_RELAY_AUTHORIZATION < bearer
/opt/spark/current/spark hostservice install relay
/opt/spark/current/spark hostservice activate relay
/opt/spark/current/spark hostservice status relay

• configure stores the service's whole configuration from one JSON document on stdin, {"expectedRevision": null | <revision read>, "configuration": {...}}, in Spark's store (spark_host_services). The first write names no revision and every later one the revision it read; a stale writer is refused (conflict).
  • relay: listenerAddress (S, an IP literal), listenerPort (the port the relay origin names, 8443), vpnHub ({ "address": <unicast IPv4>, "quicPort": <port> } or null). The relay's COREFLOW_NODE_ID and CLOUDLY_URL are this node's enrollment, never configured.
  • corestore: dataDirectory, bindAddress (S), publicHost, controlPort, s3Port, dbPort, and region, dbRootUser, clusterId, nodeName, objectStoragePoolsJson (null for Corestore's default). Every value but the bind address is part of the root's identity and must be exactly what the root was created with. A root a container created at /data/corestore is renamed there on its own filesystem before the first start, as Corestore's readme ("Keeping the root identity") describes; no bind mount is involved.
• set-credential seals one credential from stdin with systemd-creds encrypt --with-key=host into /var/lib/serve.zone/spark-credentials/<service>/<name>.cred (root, 0700 directories, 0600 files) and replaces it atomically. The value must be exactly what the service reads, byte for byte (see Credentials below); nothing is trimmed. It is never an argument, an environment variable, a unit line or output. The relay reads SERVEZONE_CLUSTER_RELAY_AUTHORIZATION, Corestore corestore-control-token. A running service reads a new value at its next start: activate it.
• install renders the unit from the configuration, this node's complete enrollment and the sealed credentials, and installs it like every Spark unit: masks and enablement preserved, the unit admitted only while quiescent, nothing started. The relay unit runs /opt/spark/current/coreflow/coreflow (Restart=always); the Corestore unit runs /opt/spark/current/corestore/corestore (Restart=on-failure, TimeoutStopSec=120, CORESTORE_VOLUME_PLUGIN=disabled, CORESTORE_API_TOKEN_FILE naming the credential's path under /run/credentials/). The relay is Type=exec: it sends no readiness, so Spark proves it (below). Corestore is Type=notify with NotifyAccess=main and TimeoutStartSec=180s: it sends READY=1 once GET /health answers 200. While a data-proportional start step (the first 34.x start's volume archive conversion) makes progress, through its copy and its index build, it extends the start timeout by 30 seconds and reports the progress as its status text (converting volume archive: N of M files (x of y GiB), then building volume archive index: <read>/<total> segments read, <written> written, <verified> verified); once a conversion that extended the deadline finishes, it gives the rest of the start a fresh 180 seconds. The base 180 seconds cover the start up to the conversion's first progress report. Both are KillMode=mixed, root (every release directory is private to root), ordered after the guard, the network and, for Corestore, the relay, and before the node unit, which wants both. Neither requires the guard: a guard failure on this host must not take the cluster's relay away from every other node.
• activate enables the installed unit, starts it (restarting a running one), and returns once it is proved healthy. A unit with a drop-in, a pending reload or another definition file, or one that does not start as its contract states (type, notify access, start timeout: startup_mismatch), is refused before any effect: these checks run before the unit is enabled, stopped or started. The stop and the start are enqueued without waiting for the job (systemctl --no-block) and followed through the unit's state, so neither is cut off by the systemctl client's command timeout. While Corestore starts, each new status text it reports is printed to stderr as SPARK_HOST_SERVICE_PROGRESS {"service":"corestore","statusText":"…"}. Spark sets the start no deadline of its own: systemd's start timeout, which Corestore extends only while its progress moves, ends a start that hangs (start_timed_out); any other failed start is unit_failed with systemd's result (exit-code, signal, …); both carry the last status text. Once systemd reports the service running, it is proved healthy: the relay when a process of its unit holds a TCP listener bound exactly to S:port within 60 seconds (a wildcard listener or another process's does not count), Corestore when GET http://S:controlPort/health answers 200 within 10 seconds.
• status reports the unit's state, its status text and result, the component its process runs and the one the selection carries, and a condition from one observation, without waiting: healthy or unhealthy for a running service, starting while systemd starts it (a converting Corestore, with its progress as the status text), stopping, stopped, and failed for a failed unit or a failed start systemd will retry.

configure and install open Spark's store, which the node runtime holds while it runs, so they refuse with node_running until the node unit is stopped; set-credential, activate and status do not need it. At boot the order is guard → relay → Corestore → containerd → node; the node needs the relay only for its first routing binding, which it keeps offline.

Updates. updatenode restarts a running host service whose process runs another component than the new release carries, as activate does, after the switch and before the node starts, relay first, each proved healthy before the next stops. Restarting Corestore is an outage of every workload's data path on the node, so it needs --restart-host-services; without it such an update is refused (host_service_restart_required) before anything stops. --at-reboot restarts nothing and the boot starts each on the new release; the two flags exclude each other. Corestore's three engines join the engine rules below: none is ever downgraded. Corestore 34's first start converts the volume archive (<dataDirectory>/volume-archive) to the index-generation format (containerarchive 0.13 in 34.0.1) and keeps the old one as volume-archive.pre-34; it refuses to start while an isolated restore, backup import or export package still holds an archive, and needs the archive's size free on the data filesystem (else it refuses before any migration step, startup failed: corestore-migration-archive-space-insufficient, which activate reports as unit_failed). A stop during the conversion ends it at a file boundary and the next start resumes it. An update never rewrites a host service's unit: rerun hostservice install when a release changes its definition.

Credentials. set-credential admits a value exactly as its service reads it and seals those bytes unchanged: the relay bearer is exactly what cloudly secrets export-cluster-relay-authorization writes (clusterRelayToken_<64 hex>_<64 hex>, no newline); Corestore's control token is UTF-8 without whitespace, 32 to 4096 bytes. Sealing always uses systemd-creds encrypt --with-key=host: the host key (/var/lib/systemd/credential.secret, root-only) and never a TPM, so a cleared TPM or a disk moved to another machine cannot lock a node out of its relay or Corestore. Whoever can read that key can read the credentials, exactly as whoever can read the root-only credential directory.

Not yet.

• Rotating the relay bearer after central Cloudly runs without Docker has no export path: Cloudly's export is an in-process CLI and the kit reaches it by a container exec, which a Pallet-run Cloudly does not offer. That is Cloudly's and Onebox's to provide.
• The node runtime's Docker census (the Swarm observation and legacy-retirement statement runnode posts) stays until the fleet has left Swarm (after W9), because Cloudly still reads those observations during the cutover.
• The relay runs as root. It needs no privilege, but every release directory is private to root; a dedicated user with read access to the relay's component only is a hardening follow-up.
• Corestore's engine rule compares bundles only. The first host-service Corestore on a root that a Docker-run Corestore wrote has no selected engine to compare against, and a stopped NoSQLDB root does not state which engine version last wrote it (its stopped inspection reports identity, lineage and digests, no engine version), so Spark cannot refuse an engine older than the root's before it starts. The older engine refuses such a root itself at startup (NoSQLDB admits only formats it supports), so the failure is closed but late: pin a Corestore whose engine is at least the one the Docker runtime ran.

Updating a node bundle

A node that already has a bundle selected is updated with the new release's own installer, the same command that bootstrapped it:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version>
curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version> --at-reboot
curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version> --restart-host-services
curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version> --at-reboot --forwarding-mode docker-shared

The installer verifies the release's standalone executable and runs its installnodebundle, which updates when a selection exists. spark updatenode <version> [--at-reboot | --restart-host-services] [--forwarding-mode <mode>], run by that release's own executable, is the update-only form; it never bootstraps. Only the release being installed performs its own update, so the unit definitions, drain protocol and engine rules it applies are the target's: an executable of another version refuses with updater_mismatch and names the installer command to run. The update holds the installation lock from the verified download to the final readiness check and prints one SPARK_NODE_UPDATE JSON record.

A small release request (the release and tag API answers, the TsPack manifest and the checksum list) that times out, loses its connection or gets a 5xx answer is attempted again, up to three attempts with their own 30 s deadlines and pauses of 1 s and 3 s; the archive has one attempt within 10 minutes. Nothing after the download is repeated. A request that keeps failing ends the update before any effect and names itself, for example Spark release download rejected: timeout (checksums request, 3 attempts).

What happens depends on the node unit, which the update never enables, unmasks or reloads beyond a definition it replaces:

• Running and ready. The node runtime is drained through its private control socket, the unit is stopped, current switches to the new release, a changed node definition is replaced, a containerd running another runtime release is restarted from the new selection (its shims and workloads keep running), and the unit is started; the new process must report ready for exactly the new bundle, within the node's whole start budget of 712 seconds (a start may fence a retained Pallet generation first). outcome is restarted and runtimeReady is true. The new process must be the one the update started: if it fails within that window and the service manager restarts it, the update reports update_incomplete with the new release selected; inspect the unit's journal for the failure.
• Stopped, failed or masked. current switches and a changed definition is replaced; nothing is started or unmasked. outcome is selected. A node unit the service manager is about to restart after a failure is neither, and is refused (unsafe_service).
• --at-reboot. The running node is drained and stopped and current switches; nothing is started and containerd is not restarted. The enabled unit starts the new release at the next boot, which the operator or Cloudly performs. outcome is reboot_pending. Until that reboot the new release must not run against the kernel state the old engines left: the update writes a reboot marker (see below) before it switches, and while it exists runnode, guard-recover and activateguard refuse to start any Pallet engine (reboot_pending: reboot the node). A plain systemctl start smartdaemon_spark.service therefore fails by name instead of starting the new release. Rerunning the same update keeps the marker; only the reboot clears it.
• Already selected. Nothing runs; outcome is unchanged.

On systemd 254 and later a stopped Spark 35 node unit whose descriptor store still holds Pallet's router namespace reads inactive/dead-resources-pinned instead of inactive/dead; that is a stop like any other (SmartDaemon's systemdUnitQuiescence() decides, for every unit Spark admits as stopped). An update keeps the store, through the definition replacement too, so the next start of the node adopts the namespace; only setforwardingmode and the uninstaller release it.

The node unit states the host's forwarding mode, and an update keeps it: the mode is the one the unit's file states, or, for a unit of an earlier release that states none (or a masked unit whose command line the manager has not loaded), the one --forwarding-mode names. Without it such an update is refused before the download (forwarding_mode_required); stating a mode other than the unit's is refused too (forwarding_mode_change), because a mode changes only with the node stopped (spark setforwardingmode, see Host forwarding mode). A unit the manager runs with another command line than its file states (a drop-in, a pending reload) is unit_definition_changed.

Pallet's control is an engine as well: Pallet 36 journals records Pallet 35 cannot read, so an update never selects an older Pallet, nor other bytes of the same Pallet version (engine_downgrade, engine pallet), read from the bundle's sealed pallet/control-build.json.

Spark 34.3.2 and older installed the node unit with Restart=no, so a failed node runtime stayed down. The first update from one of them replaces that definition (definitionsReplaced names smartdaemon_spark.service), and the node is restarted after a failure from then on (see Stable systemd boot entry).

Every release stays in /opt/spark/releases; the record's previous names the one that was selected. Returning to it is another update to that version with its own installer, admitted only while every engine rule below holds. A node that ran Pallet 35.2.0 or later may have journaled an epoch withdrawal, which Pallet 35.1.1 and older refuse as owner_failed, so returning such a node to Spark 34.2.3 or older needs its Pallet journal reset, not just the older bundle. Likewise, a node that ran Pallet 35.4.3 or later may have stored a run operation as failed (a refused image pull), which Pallet 35.4.2 and older refuse as invalid, so returning such a node to Spark 34.3.4 or older needs its Pallet assignment store reset as well. A node on a bundle carrying Pallet 36 never returns in place to one carrying Pallet 35.x (Spark 34.x): its Pallet and its packet engine (SmartNFTables 4.3.0 or later) would both move backwards, which the update refuses (engine_downgrade). A node that ran Pallet 36.1.0 or later (Spark 35.0.3 and later) may have journaled an in-place transition that detached a workload attachment, which Pallet 36.0.3 and older refuse (the node process fails at start), so returning such a node to Spark 35.0.2 or older needs its Pallet journal reset, not just the older bundle. A node that ran Pallet 36.3.2 or later (Spark 35.1.1 and later) carries typed nftables sets once Pallet reconciled a table, which SmartNFTables 4.4.x refuses, so it returns to Pallet 36.3.1 or older only after a reboot; one whose Docker contribution journal recorded a transition-release end ({ kind: 'transition' }) is refused by Pallet 36.3.1 and older, so it cannot return to them in place until two later generations were raised. Releases older than 34.3.0 carry no host services: their bundles have no coreflow/ or corestore/ directory, and their installer does not know the host units. Before returning a node with an installed relay or Corestore host service to such a release, stop and disable smartdaemon_spark-relay.service and smartdaemon_spark-corestore.service (systemctl disable --now); otherwise their next start runs a path under /opt/spark/current/ that no longer exists and fails. Stopping them ends that node's cluster relay session and its Corestore, so the relay and Corestore must run elsewhere, or be accepted as down, for as long as the node stays on the older release.

The update refuses, before it stops anything, with these codes (SparkNodeBundleUpdateError, each message naming its remedy):

Code	Meaning and remedy
invalid_options	The command takes one exact version and optionally one of --at-reboot and --restart-host-services.
host_service_restart_required	A running Corestore executes another component than this release carries, and restarting it is an outage of the node's data path: rerun with --restart-host-services, or with --at-reboot and reboot.
busy	This process is already updating. A second installer is refused by the installation lock.
no_selection	No bundle is selected: bootstrap without --at-reboot.
updater_mismatch	The executable is another release: run the target release's installer.
unsafe_service	The node unit is neither running nor fully stopped (pending job, a restart the service manager scheduled after a failed node runtime, leftover processes, unsupported kill mode): stop smartdaemon_spark.service, then rerun. A failing node is restarted again and again and never settles.
node_starting	The node runtime has not reported ready (or its control socket is not bound yet): wait for readiness, or stop the unit, then rerun.
node_draining	The runtime is draining or drained, as an interrupted update leaves it: stop the unit, then rerun.
engine_downgrade	The identity store engine (spark-smartdb), Pallet's store engine (pallet/pallet-smartdb), one of Corestore's engines (corestore/corestore-nosqldb, corestore/corestore-objectstorage, corestore/corestore-containerarchive) or the packet engine (pallet/pallet-guard) would change to anything but a strictly newer version of the same project, read from each engine's sealed tsrust build provenance, whose binarySha256 must name the verified executable. Engines are never downgraded; a bad engine is recovered by restoring the node's stores.
engine_unverifiable	The selected release's provenance does not name one of those engines, so it is known by its bytes only; the same bytes pass, other bytes cannot be ordered. Update first to a release carrying the same engine bytes with provenance. (Every released 34.x bundle carries valid provenance for all three.)
reboot_required	The packet engine's bytes change and a Pallet engine has run in this boot — the node runtime, or only the guard unit on a node whose runtime never started — so the packet state it retains in the kernel cannot be released by the new engine: rerun with --at-reboot, then reboot.
unit_definition_changed	The release changes the guard definition (never replaced by an update), or the containerd definition while that unit still holds processes or workloads (mask the node unit, reboot, rerun, then unmask and start), or the node definition is kept but the service manager loads it with other settings than its restart contract (a drop-in or pending reload: remove it or run systemctl daemon-reload, then rerun).
selection_changed	Another installer changed current during the update: inspect it, then rerun.
invalid_bundle	A bundle's engine build provenance is missing or does not name its verified engine.

After it has started, a failure is reported as one of:

Code	State left behind
update_aborted	The failure came before the switch: the previous release is still selected. A node that was running is stopped if needed, started again on it and proved ready; detail.restored is true, or false when that did not succeed (start the unit and verify it) — unless the reboot-pending marker of an at-reboot update remains, in which case no restore is attempted and every node-runtime and guard start is refused with reboot_pending until the node reboots (reboot it). This includes cancellation.
update_incomplete	The new release is selected, or the switch cannot be excluded (the selection could not be read back after it was attempted, or a restore found another selection than the one the update started from). Nothing is started and nothing rolls back: a started release may already have migrated state. Inspect /opt/spark/current and the unit, fix the cause, then start the unit or rerun.

Both per-boot facts live in /run/serve.zone/spark-boot, which /run (a tmpfs) starts empty on every boot, as empty mode-0600 files owned by one module (ts/node/boot.markers.ts):

• pallet-ran is written immediately before any Pallet engine starts: every guard oneshot (guard-recover at boot, guard-commission and network-acquire during activateguard) and the node runtime's Pallet runtime. Pallet's enrollment modes, which enrollnode and the runtime binding proof run, apply no packet or guard state and write none.
• reboot-pending is written by an at-reboot update before it switches and removed again only when that update fails before switching.

For reboot_required a Pallet engine has run in this boot when any of these holds; released bundles write no pallet-ran, so the first update from one relies on the other three:

• the node unit is running;
• pallet-ran exists;
• the node service's lock anchor /run/serve.zone/spark-node/owner.lock exists (runnode creates it before Pallet's runtime starts and never removes it);
• the guard unit smartdaemon_spark-guard.service is loaded and either not inactive/dead (a oneshot with RemainAfterExit=yes stays active/exited after running, failed after a failure) or enabled (an enabled guard is required by network-pre.target and networkd, so every boot starts it).

The test is conservative: any of these counts as retained packet state, whether or not an ACTIVE generation exists. On a commissioned node the guard runs every boot, so a packet engine change always needs --at-reboot there. Moving a node from a bundle carrying Pallet 34.x (SmartNFTables 2.6.0 guard) to one carrying Pallet 35.0.0 or later (SmartNFTables 4.0.0) is such a change: Pallet 35's engine compiles a different packet graph and cannot release tables the 2.6.0 engine applied in the same boot, so update those nodes with --at-reboot and reboot them. Pallet 35.0.0 and 35.1.0 carry the same packet engine, so moving a node between them needs no reboot. Pallet 35.1.1 carries SmartNFTables 4.0.1, a new packet engine, so moving a node from Pallet 35.1.0 or older to it needs --at-reboot and a reboot as well. Pallet 35.2.0 carries the same packet engine as 35.1.1, so moving a node from Pallet 35.1.1 to it needs no reboot; from Pallet 35.1.0 or older it still does. Pallet 35.3.1, 35.4.0, 35.4.1, 35.4.2 and 35.4.3 carry the same identity, store and packet engines as 35.2.0, so moving a node from Pallet 35.1.1 or later to any of them needs no reboot either. Pallet 36.0.1 (Spark 35.0.0), 36.0.2 (Spark 35.0.1 and 35.0.2), 36.1.0 (Spark 35.0.3), 36.1.1 (Spark 35.0.4), 36.1.2 (Spark 35.0.5), 36.1.4 (Spark 35.0.6), 36.1.5 (Spark 35.0.7), 36.1.6 (Spark 35.0.8), 36.1.8 (Spark 35.0.9) and 36.2.0 (Spark 35.0.10) carry SmartNFTables 4.3.0, a new packet engine, with the same NoSQLDB 10.5.1 store engine: moving a node from any Pallet 35.x bundle (Spark 34.x) to any of them is an at-reboot update. Spark refuses it in place on a node whose Pallet engine ran in the current boot (reboot_required); install it with --at-reboot (and, for a node unit of Spark 34.x, which states no forwarding mode, --forwarding-mode <mode>), then reboot. Pallet 36.0.2, 36.1.0, 36.1.1, 36.1.2, 36.1.4, 36.1.5, 36.1.6, 36.1.8 and 36.2.0 carry the same store and packet engines as 36.0.1; moving back between them is refused (engine_downgrade). Pallet 36.0.0 to 36.1.1 release the router namespace's handoff set when they stop, while the node unit's descriptor store keeps the namespace, so a Pallet started after them in the same boot fails (Pallet 36.1.2: handoffset.reconcile.member_vanished) until a reboot empties the store. Spark therefore refuses every in-place update of a node whose selected bundle carries one of them (Spark 35.0.0 to 35.0.4) once a Pallet engine ran in the current boot (reboot_required, engine pallet), whatever Pallet the target carries; install it with --at-reboot, then reboot. Pallet 36.1.2 and later hand the set over at their stop, so a node on them moves forward in place. Pallet 36.1.3 and later journal a network generation discarded before activation as discarded, which Pallet 36.1.2 and earlier refuse to read: do not move a node that wrote one back below 36.1.3 in place. Pallet 36.1.6 and later journal the detachment of an attachment before its links are deleted as an amendment row with detaches: true, which Pallet 36.1.5 and earlier refuse to read: do not move a node that wrote one back below 36.1.6 in place. Pallet 36.1.8 writes no record that 36.1.6 cannot read: the assignment ids its scans read are kept in memory only, and the guard pass a newer admission supersedes (guard.propose.superseded) commits nothing. A node on Spark 35.0.5 or later (Pallet 36.1.2 or later) therefore moves to Spark 35.0.9 in place, and moving it back to 35.0.8 is refused like any older Pallet (engine_downgrade). Pallet 36.2.0 writes no record that 36.1.8 cannot read either: the resolver it states for a workload's sandbox is derived from the run's existing network preparation when the run starts and is not stored. A node on Spark 35.0.5 or later moves to Spark 35.0.10 in place, and moving it back to 35.0.9 is refused (engine_downgrade). A workload sandbox created by Pallet 36.1.8 or earlier keeps the host's copied /etc/resolv.conf until its workload is recreated (a new attempt or a redeploy). Pallet 36.2.1 (Spark 35.0.11 and later) carries SmartNFTables 4.4.0, a new packet engine whose allocation-pool guard admits the node's published ports, with the same NoSQLDB 10.5.1 store engine; it reads the guard rows 36.2.0 and earlier wrote. Moving a node from any earlier bundle to it is an at-reboot update: Spark refuses it in place once a Pallet engine ran in the current boot (reboot_required, engine packet); install it with --at-reboot, then reboot. Moving it back to Spark 35.0.10 or earlier is refused (engine_downgrade). Pallet 36.3.1 (the Spark releases after 35.0.11) carries the same NoSQLDB 10.5.1 store engine and SmartNFTables 4.4.0 packet engine bytes as 36.2.1, so a node on Spark 35.0.11 moves to it in place; from Spark 35.0.10 or earlier it is still an at-reboot update. Its projection application receipts live in two new collections of Pallet's store and need no migration. Moving a node back to Spark 35.0.11 is refused (engine_downgrade). Pallet 36.3.2 (the Spark releases after 35.1.0) carries SmartNFTables 4.6.0, a new packet engine, with the same NoSQLDB 10.5.1 store engine; it starts a docker-shared node whose stopped generation left an unconfirmed DOCKER-USER contribution step. Moving a node from any earlier bundle to it is an at-reboot update: Spark refuses it in place once a Pallet engine ran in the current boot (reboot_required, engine packet); install it with --at-reboot, then reboot. Moving it back to Spark 35.1.0 or earlier is refused (engine_downgrade). Pallet 36.3.4 (Spark 35.1.2) carries the same SmartNFTables 4.6.0 packet engine and NoSQLDB 10.5.1 store engine bytes as 36.3.2, so a node on Spark 35.1.1 moves to it in place; from Spark 35.1.0 or earlier it is still an at-reboot update. It keeps a node's router namespace across a restart only after a clean stop that journaled the generation DOWN, so it does not adopt the namespace an exclusive node kept under Pallet 36.3.2 or earlier (whose stop left the ACTIVE generation UP): that epoch ends, its runs fail as network-epoch-ended and Cloudly replaces them. A docker-shared node, whose earlier stop already withdrew the generation, keeps its namespace and its workloads. Moving a node back to Spark 35.1.1 is refused (engine_downgrade). Pallet 36.3.5 (Spark 35.1.3) carries the same SmartNFTables 4.6.0 packet engine and NoSQLDB 10.5.1 store engine bytes as 36.3.4, so a node on Spark 35.1.1 or later moves to it in place; from Spark 35.1.0 or earlier it is still an at-reboot update. It refreshes an unbound run's network preparation before each CNI ADD and keeps a run pending that is still unbound after its bounded sandbox refusals; it activates a new generation only after the released attachments it reconciles first are durably absent. Moving a node back to Spark 35.1.2 is refused (engine_downgrade). Pallet 36.3.6 (Spark 35.1.4) carries the same packet and store engine bytes as 36.3.5, so a node on Spark 35.1.1 or later moves to it in place; from Spark 35.1.0 or earlier it is still an at-reboot update. It starts over a journaled workload whose sandbox namespace is gone (no live holder means destroyed), keeps network passes running while an assignment page is busy, extends a held generation under its own application and retries an amendment that lost to a newer admission. Moving a node back to Spark 35.1.3 is refused (engine_downgrade). Pallet 36.3.7 (Spark 35.1.5) carries the same packet and store engine bytes as 36.3.6, so a node on Spark 35.1.1 or later moves to it in place; from Spark 35.1.0 or earlier it is still an at-reboot update. It refuses a single workload ADD whose sandbox is gone instead of losing the network coordinator, rolls back a pair bound to a refused sandbox, answers every CNI invocation it cannot take with its reason, and keeps the node running when the store refuses a scan read or a DNS refresh. Moving a node back to Spark 35.1.4 is refused (engine_downgrade). Pallet 36.3.8 (Spark 35.1.6) carries the same SmartNFTables 4.6.0 packet engine bytes as 36.3.7 and moves its store engine from NoSQLDB 10.5.1 to 10.9.1. Spark's identity engine is that same binary, so Spark's own identity store and its NoSQLDB client move to 10.9.1 with it. Both store engines move to a strictly newer version and the packet engine is unchanged, so a node on Spark 35.1.1 or later moves to it in place; from Spark 35.1.0 or earlier it is still an at-reboot update. Its workload attachments (CNI ADD) no longer slow down as a node accumulates workloads and in-place transitions. The node runtime's first start converts every ACTIVE journal in Pallet's store (resumable), and Pallet 36.3.7 and earlier refuse a converted store, so a node cannot be moved back in place after that start: back up the node's stores (/var/lib/serve.zone/spark, /var/lib/serve.zone/pallet and /var/lib/serve.zone/pallet-storage) with the node unit stopped before the update. Moving a node back to Spark 35.1.5 is refused (engine_downgrade); recover from a bad update by restoring those stores. Pallet 36.3.10 (Spark 35.1.7) moves the packet engine (pallet-guard, SmartNFTables 4.6.0 to 4.6.1) and the store engine (NoSQLDB 10.9.1 to 10.9.2). Spark's identity engine is that same binary, so Spark's own identity store and its NoSQLDB client move to 10.9.2 with it. The packet engine changes, so every node moves to it with --at-reboot and a reboot; the plain form refuses reboot_required. A stop or removal whose runtime call failed is driven again in the same boot instead of holding the node's native lane, a lost packet engine costs the generation's coordinator instead of the node (active.packets.lost, once per raised generation), and SmartNFTables retries the nftables generation race that Pallet's pool and host owners share: a step it refuses after proving it changed nothing fails only that apply or release (packet.apply.refused, packet.release.refused) and keeps the packet owner. Unsited attachment refusals name their native code. A node still on Pallet 36.3.7 or earlier converts its ACTIVE journals at the first start exactly as for 36.3.8, so back up the node's stores with the node unit stopped before the update. Moving a node back to Spark 35.1.6 is refused (engine_downgrade). Pallet 36.3.11 (Spark 35.1.8) carries the same packet engine (SmartNFTables 4.6.1) and store engine (NoSQLDB 10.9.2) bytes as 36.3.10, so a node on Spark 35.1.7 moves to it in place; from Spark 35.1.6 or earlier it is an at-reboot update, as for 36.3.10. Its first-start conversion of ACTIVE journals (pallet-active-amendment-rows) reads the journals by _id instead of by position. On a large journal store, Pallet 36.3.10's positional paging read every journal it skipped, and the admitting transaction outlived the store engine's 60 s transaction lifetime, so every start failed with noderuntime.start < SmartdataPersistenceError:unsupported_operation; journals an earlier attempt converted stay converted. A refused Docker contribution step fails only that workload ADD instead of stopping the node (docker.reconcile.refused, with the engine's code), and a Docker forwarding owner lost for any other reason (active.docker.lost) costs the generation, not the node. Moving a node back to Spark 35.1.7 is refused (engine_downgrade).

Pallet 36.3.12 (the Spark releases after 35.1.8) carries the same packet engine (SmartNFTables 4.6.1), store engine (NoSQLDB 10.9.2) and VPN bytes as 36.3.11, so a node on Spark 35.1.8 moves to it in place. Its router namespace inspections take turns: two inspections asked together used to refuse the second, which the node read as its namespace lost, so it retired its network (handoffset.lost) and the node runtime failed. Logs go to Cloudly on their own delivery lane, so no log batch holds back a stop, removal, observation or receipt, and a stop refused while the node's last observation was unacknowledged is admitted on Cloudly's next try. A long-lived ACTIVE generation is replaced in place by a checkpoint before it reaches its per-generation bound instead of being withdrawn. Pallet 36.3.11 and earlier refuse a store that holds a checkpoint, and moving a node back to Spark 35.1.8 is refused (engine_downgrade).

Requirements

• Linux with systemd for daemon operation.
• For the node unit, systemd 254 or newer: it keeps Pallet's router network namespace in its file descriptor store with FileDescriptorStorePreserve=yes (Pallet's readme, "Service unit contract"), a setting older managers do not know. SmartDaemon reads the running manager's version (systemctl show --property=Version) before it installs the node unit, reads a unit's loaded settings or releases a descriptor store, and Spark names its refusal before any effect: systemd_too_old (the message states the version found and the major required; the SmartDaemon error is the cause) or systemd_version_unknown (a version Spark cannot read). installnodeservice, installnodebundle/updatenode (before the download), setforwardingmode and activateguard (with an installed node or containerd unit, before any Pallet process starts) refuse such a host, and so do bundle activation and the host service commands that read a unit's loaded settings.
• For the node runtime and its allocation-pool guard, Linux 6.9 or newer with the nftables filter and NAT modules, as Pallet's packet engine requires (Pallet's readme, "Host kernel requirements"). Pallet 35.1.0 and later names a kernel below that floor unsupported_kernel; Spark reports it as the cause pallet_process unsupported_kernel (enrollnode, runnode, activateguard) or Spark guard process failed: unsupported_kernel. (activateguard, guard-recover), followed by that remedy.
• Docker for service and Swarm management.
• Root privileges for daemon installation, Docker maintenance, and package updates.
• Deno only when running from source.

macOS and Windows binaries are built for CLI/library availability, but the operational daemon paths are Linux/systemd oriented.

Quick Start

The commands in this section are the released Swarm-era line and apply to a host that has not been converted. On a converted host spark asdaemon refuses by name; see Cutover to the node runtime.

An unconverted host already carries the smartdaemon_spark.service unit written by the release that installed it. spark claimjump is the only command that still writes one, as part of Cloudly Jump setup; there is no separate daemon-installation command.

Run daemon mode with a profile:

sudo spark asdaemon --mode cloudly

or:

sudo spark asdaemon --mode coreflow-node

The selected mode is persisted in a user-home npmextra key/value store under the servezone_spark identity. Later spark asdaemon calls can reuse the stored mode when no --mode flag is provided.

Follow daemon logs:

sudo spark logs

CLI Reference

spark <command> [options]

Command	Purpose
claimjump --claimUrl <url> --jumpCode <code>	Claim a Cloudly Jump code, persist the returned node/Coreflow credentials, and install/start the daemon. Also accepts CLAIM_URL and JUMP_CODE.
enrollnode	Enroll separate Spark and Pallet credentials from one JSON document on stdin, using an installed verified node bundle. Does not activate the daemon.
runnode --forwarding-mode <docker-shared|exclusive>	Run the selected compiled node bundle in the foreground using existing enrolled identity and settings; the node unit states the host's forwarding mode.
installnodebundle <version> [--at-reboot|--restart-host-services] [--forwarding-mode <mode>]	Bootstrap the complete node bundle on a host without one, or update the selected bundle (see Updating a node bundle). install.sh --node-bundle runs it from the verified release executable.
updatenode <version> [--at-reboot|--restart-host-services] [--forwarding-mode <mode>]	Update the selected node bundle with this release; never bootstraps. The mode is needed once, for a node unit that states none.
hostservice <command> <relay|corestore>	Configure, set-credential <name>, install, activate or inspect a host service; see Host services.
installnodeservice --forwarding-mode <docker-shared|exclusive>	Install the canonical systemd definition for the verified current node bundle and the host's forwarding mode, preserving masks and enablement.
setforwardingmode <docker-shared|exclusive>	State the host's forwarding mode on the stopped node unit and release its descriptor store.
activateguard	Prove the node's runtime routing binding, then commission or recover the retained allocation-pool guard, install its boot unit, and verify persistent networkd dependencies.
guard-recover	Recover the existing retained guard from the selected bundle; the systemd boot entry uses this command.
upgrade	Install a newer Spark release when available and restart the daemon.
asdaemon --mode cloudly	Run the daemon loop with the Cloudly profile. Refused by name on a host whose boot path is the node runtime.
asdaemon --mode coreflow-node	Run the daemon loop with the Coreflow node profile. Refused by name on a host whose boot path is the node runtime.
metrics	Print a JSON host metrics snapshot.
logs	Follow journalctl -u smartdaemon_spark -f.

The command comes first. spark alone, spark --help or -h lists the commands with their usage on stdout and exits 0, and spark --version or -v prints the version; anything else that is not one of the commands above stops before any command runs, with exit code 1 and Error: Spark CLI rejected: unknown_command. (or invalid_arguments for an option in place of the command) on stderr, naming the executable's version. An executable older than a command, for example a 34.1.0 executable asked for updatenode, therefore refuses instead of doing nothing. Each command validates its own arguments.

claimjump, upgrade and asdaemon remain in the binary for hosts that have not been converted. They are no longer the advertised path: a completed bare installation prints the node-bootstrap order instead.

installdaemon, updatedaemon and prune were removed. Both daemon writers wrote a unit whose ExecStart is spark asdaemon under the canonical name the node runtime's own runnode unit uses, so on a converted host they would clobber that unit with a process that now refuses to start; neither is part of the conversion order, which masks the released unit before the bundle is installed. prune was a destructive Docker sweep outside the ownership protocol with no place in the node runtime.

Pallet credential enrollment

spark enrollnode is the external credential bootstrap for a verified, selected Linux node bundle under /opt/spark. The bare-binary installer above does not provide that bundle. Run the command as root with one JSON document on stdin followed by EOF; it accepts no command options, credential environment variables, executable paths, or caller-supplied writer-stop claims. Input is limited to 16 KiB and must finish within 30 seconds.

For an existing node, the input is:

{"operation":"adopt-existing","hostname":"worker.example"}

The command requires smartdaemon_spark.service to be masked and fully stopped, with no pending job, main/control PID, or remaining cgroup. A noncanonical spark.service must be absent. It verifies that state before the one-time import, again before enrollment, and after cleanup. It reads only the canonical root daemon's /root/.npmextra/kv/servezone_spark.json; the versioned migration preserves the source bytes and records the original identity and digest in private NoSQLDB state. The command never changes service enablement or starts an old runtime.

A fresh node instead uses this input, supplying the actual Jump code through stdin:

{"operation":"jump","cloudlyOrigin":"https://cloudly.example","hostname":"worker.example","bearer":"<16-character Jump code>"}

Fresh enrollment also permits an absent canonical Spark unit. Cloudly communication uses only the configured HTTPS origin. Spark and Pallet generate and retain separate credentials in their own protected NoSQLDB stores; Spark receives no Pallet bearer. Explicit provisioning creates missing platform data parents, while ordinary resume only recreates disposable IPC directories. Enrollment needs no cluster relay: the Pallet runtime routing binding is read and bound by activateguard, before Pallet's network acquisition needs it, and proven again on every node runtime start (see Foreground node runtime).

Resume a persisted attempt without repeating credentials or reading the legacy file:

{"operation":"resume"}

After the coordinator confirms both current local identities and joins the Pallet child and Spark database cleanup, stdout contains one SPARK_ENROLLMENT record followed by JSON with credentialEnrollment: "complete", runtimeReady: false, and a secret-free enrollment summary. Other dependency lifecycle log lines can precede that record. This result does not establish workload or remote-session readiness. A lost response or outcome_unconfirmed must be reconciled through the saved attempt; no second identity or legacy bootstrap is substituted. If failure preceded initial store provisioning, repeat the exact original input to provision it.

A failure names what failed underneath, in a closed vocabulary that never carries a URL, Jump code, bearer or remote text:

• cloudly_unreachable (cloudly unreachable: <cause>) — the Cloudly request that failed, and its protocol recovery attempt when one was made, could not connect, so Cloudly never received it. The cause is the local resolver, socket or certificate verifier result: dns_unresolved, connection_refused, network_unreachable, connect_failed, tls_untrusted_issuer, tls_name_mismatch, tls_certificate_expired, tls_certificate_not_yet_valid, tls_certificate_invalid or tls_handshake_failed, followed by the remedy. The saved attempt is kept; run the same input again once the cause is fixed.
• outcome_unconfirmed (<owner> <code>) — the request may have reached Cloudly, or a local owner failed around it (for example cloudly timeout, cloudly rejected, pallet conflict, identity unavailable). Reconcile it by running the same input, or {"operation":"resume"}, again.

TLS trust

Every Spark process verifies TLS peers — Cloudly, the release origin — against the Mozilla root bundle compiled into Deno and the operating system trust store. A node that must trust a private or internal certificate authority trusts it the way the host's other services do: install the CA certificate into the operating system trust store (on Debian and Ubuntu, copy it to /usr/local/share/ca-certificates/<name>.crt and run update-ca-certificates; on RHEL-family hosts, /etc/pki/ca-trust/source/anchors/ and update-ca-trust). The store is read when a Spark process opens its first TLS connection, so the next command and the next node runtime start pick it up; no unit drop-in, environment file or Spark setting is involved, and certificate verification is never disabled.

The Spark entry point selects Deno's stores itself (DENO_TLS_CA_STORE=mozilla,system) before any command runs, because a compiled Deno executable otherwise trusts the Mozilla bundle alone. An operator who sets DENO_TLS_CA_STORE explicitly keeps that choice, provided it lists only mozilla and system; any other value, including an empty one, stops the command with invalid_ca_store rather than failing every later connection. Pallet's control executable (34.0.3 and later) applies the same selection and the same refusal in every mode. Spark starts it with a cleared environment, so Spark's own value never reaches it, and names a refusal it reports invalid_ca_store; the containerd unit, which systemd starts, takes the manager's environment, so a DefaultEnvironment store selection must follow the same rule. So does the Coreflow relay unit, which sets no store: Coreflow 33.4.3 and later applies mozilla,system itself with the same refusal, so an invalid DefaultEnvironment value would restart-loop the relay.

Daemon Behavior

Spark.daemonStart() starts the node reporting and reconciliation subsystems before enabling recurring tasks. A failed initial managed-service reconciliation stops already-started subsystems in reverse order instead of leaving a partial daemon running. The asdaemon command handles SIGINT and SIGTERM by disabling actions, terminating active action process groups, stopping tasks and service reconciliation, releasing the Coreflow session, and stopping heartbeat reporting before exit. Shutdown aborts active pinned image pulls and retains the Coreflow owner session until reconciliation settles. A pull that ignores cancellation retains the owner session until that reconciliation eventually settles. Separately, subprocess cleanup that cannot prove exit retains exact process tracking and quarantines the affected lifecycle restart while the daemon continues its ordered shutdown.

• SparkServicesManager.start() activates Docker Swarm and, for Coreflow nodes, acquires the host-wide reconciliation lock.
• SparkTaskManager.start() schedules recurring maintenance tasks.
• SparkHeartbeatManager.start() sends node metrics to Cloudly when node credentials are configured.
• SparkMetricsStreamer.start() streams lightweight metrics and, after initial service reconciliation completes, executes Cloudly systemUpgrade, reboot, and serveZoneServiceUpdate node actions when node credentials are configured. Spark executes an action only after Cloudly accepts its running claim, requires Cloudly to accept the terminal result, reports non-owner service-update skips as failures, and disables further actions if terminalization cannot be proven.

Heartbeat metric branches, host updates, self-updates, and action processes are cancelled and drained during shutdown. Metric commands are timeout-bounded and escalate from SIGTERM to SIGKILL; action process groups, including detached descendants, must terminate before lifecycle restart is allowed.

Scheduled tasks:

Task	Schedule	Action
updateServices	Daily at midnight in cloudly mode	Checks generic managed Docker services for image and published-port drift. Coreflow-node updates are initial-startup and Cloudly-action driven with complete authority-bound shape proof.
updateSpark	Every 10 minutes	Checks for a newer Spark release and reloads the daemon after upgrade.
updateHost	Daily at midnight when enabled	Runs apt update/upgrade/autoremove/autoclean.

The managed service list is populated by the selected mode before daemon startup. Service updates use version-pinned Docker image pulls, Docker secrets, and declared published port mappings.

Base service images

serve.zone images publish released version tags only, so Spark states the exact version of every base service it installs. binary/base-service-images.json pins each one by registryHost, repository, version and the repository digest that version resolved to; this release installs Coreflow 32.2.0 and Cloudly 32.0.0. Upgrading a cluster to a newer base service is an explicit change of that manifest and a new Spark release, never a moving tag.

A managed-service descriptor is refused by name when it leaves the installed version open: a reference without a tag, a mutable tag such as latest, a digest in place of the version, or a digest that is not a canonical repository digest. The pinned tag is pulled and verified against the pinned digest, so a registry that moved the tag fails the pull instead of silently installing another build, and the created Swarm service carries the digest-pinned reference (code.foss.global/serve.zone/coreflow@sha256:...).

Reconciliation therefore decides on that digest: a running service is replaced when its image is not exactly the pinned digest — including a service still bound to the version tag or to a different digest of the same repository. The image version label is descriptive only. A Coreflow service an earlier Spark created from a mutable tag is still adopted, because adoption proves ownership of the pinned repository and the following reconciliation moves it onto the pinned digest.

In coreflow-node mode, Spark records the durable Coreflow owner in Swarm Spec labels through version-index CAS. The matching node label is only the scheduling projection. Coreflow is configured for one replica on that manager with no published ports, the Docker socket and /var/lib/serve.zone/coreflow mounted writable, and a root-owned secret.json mounted with mode 0400. Its szncorechat network must be a non-ingress, non-internal, non-attachable, IPv4-only Swarm overlay with the exact coreflow alias. Coreflow may add at most one alias-free sznwebgateway attachment and one read-only /sys/fs/cgroup:/host/cgroup mount. New services are created at zero replicas, proven against the complete managed shape, and only then scaled to one desired replica. The state directory and spark-owner.json marker are root-owned with modes 0700 and 0600. The retained host lock at /var/lib/serve.zone/.spark-coreflow-owner.lock is also root-owned with mode 0600. Spark does not migrate ownership automatically when the recorded node disappears or loses manager status. Within managed reconciliation, only the current owner may create, stop, remove, or replace the Coreflow service and secret. Each owner Spark process uses a new opaque Coreflow config revision, so an owner daemon restart rotates the Coreflow secret and replaces the service once. A wrong same-name szncorechat network fails closed instead of being replaced.

Protocol handshake

Spark states the @serve.zone/interfaces release it was built against on every body it posts to Cloudly, and the oldest Cloudly release it accepts. It opens two session kinds, because it runs two runtimes rather than two versions of one, and each names its own minimum:

Session kind	Runtime	Routes	Oldest peer
sparkNode	pallet node (runnode)	POST /spark/nodes/{heartbeat,host}	32.52.0
sparkSwarmNode	Swarm-era coreflow-node daemon	POST /spark/swarm-nodes/{heartbeat,metrics-sample,action-result,swarm-observation}	32.18.0

These routes carry no typed error, so Cloudly answers a refused offer with HTTP 422 and an IProtocolRefusal body, judged before the node is authenticated. Spark tears nothing down when that happens: it keeps its credentials, its bundle, its timers and any pending Swarm observation, logs the refusal once, and offers again after the shared five-minute refusal cadence. The node runtime reports the state protocol-incompatible while it holds one. An accepted answer clears the refusal and returns the host to its configured interval. A refused node action is left with Cloudly, which still holds it and delivers it again, rather than disabling this node's actions.

Cloudly heartbeat requires a Spark node token. spark claimjump is the Spark-side claim path for Cloudly Jump setup: it claims the Jump code over HTTP, validates the response, persists cloudlyUrl, the Coreflow jumpcode, nodeId, and nodeToken, and installs the daemon. Existing installations without those values continue to run service maintenance without reporting node telemetry.

Programmatic Usage

Spark exports Spark and SparkSwarmObservationManager from mod.ts:

import { Spark } from './mod.ts';

const spark = new Spark();
await spark.daemonStart();

// Stop timers, action processes, tasks, reconciliation, and heartbeat reporting.
await spark.daemonStop();

The public instance exposes:

Property	Purpose
smartdaemon	systemd service integration.
sparkConfig	persisted mode/config key-value store.
sparkTaskManager	taskbuffer scheduler and built-in maintenance tasks.
sparkUpdateManager	Docker Swarm activation and managed service update logic.
sparkMetricsCollector	Host metrics collection for local output and Cloudly heartbeat payloads.
sparkHeartbeatManager	Cloudly heartbeat and authenticated Swarm observation client.
sparkMetricsStreamer	Cloudly metrics stream and guarded node-action executor.
sparkSelfUpdater	Bounded binary and checksum download/update lifecycle.
sparkSwarmObservationManager	Callable, serialized read-only Swarm observation producer.
coreflowConfigRevision	Per-process opaque revision used for owner-managed Coreflow secret rotation.
sparkInfo	package metadata lookup.

Collect one local observation without using the heartbeat transport session:

const observation = await spark.sparkSwarmObservationManager.collectObservation();

Direct SparkSwarmObservationManager construction requires an IDockerSwarmReadClient; the optional reporter session, initial sequence, and clock inputs support lifecycle ownership and deterministic tests. Collection accepts an optional AbortSignal, bounds Docker reads, and keeps timestamps strictly increasing within its reporter session.

Development

Run from source:

deno task dev

Quality and build tasks:

deno task check
deno task lint
deno task fmt
deno task test
deno task compile

The package scripts intentionally defer to Deno tasks; pnpm build only reports that no Node build is needed.

The source checkout pins pnpm and disables install lifecycle scripts through pnpm-workspace.yaml. Its npm wrapper postinstall downloads an already published Spark binary, so running it during source release preparation would depend on the release being built. This source-only setting is not included in the npm wrapper. Frozen Deno commands install the runtime imports, including TsPack (which the node also uses to verify and extract bundles), from deno.json and deno.lock. Build tooling resolves in a graph of its own: TsDeno is pinned in deno.tools.json and locked in deno.tools.lock, so a tooling update can never bind a peer dependency or add a package in the runtime graph, and never reaches the shipped executables. Run TsDeno from the project root with --config deno.tools.json; it compiles mod.ts against the runtime deno.json and deno.lock. The tooling config sits in the project root because Deno reads .npmrc from the config file's directory.

Both locks are registry-neutral: their npm entries carry no tarball URL. Deno writes one only when it resolved a package from a registry other than npmjs, and from Deno 2.9.7 it refuses a lock whose tarball origin is not the configured registry. The tracked .npmrc therefore sets registry=https://registry.npmjs.org/; Deno and pnpm both read it from the project root and prefer it over ~/.npmrc, so a lock refreshed on a machine whose home configuration names a mirror still resolves from npmjs. Deno's NPM_CONFIG_REGISTRY environment variable overrides the project .npmrc: the graph resolution shared by the release build and the notice generator refuses to run while it names another registry, and refuses a lock that carries any tarball field; every scripts/build-node.ts compile checks deno.tools.lock the same way before it starts TsDeno.

Refresh the locks with env -u NPM_CONFIG_REGISTRY deno install (runtime) and env -u NPM_CONFIG_REGISTRY deno install --config deno.tools.json (tooling), then confirm each with deno install --frozen (plus --config deno.tools.json) and regenerate the node notices after a runtime lock change. A refresh keeps the peer bindings already in a lock: Deno re-evaluates an optional peer only for a package it resolves again. test/nodedependencies.test.ts therefore refuses a runtime lock with an entry its own imports do not reach, or with an optional peer that nothing else in it needs.

Release tags use GitZone's journaled Gitea-assets target. The release build runs with the pinned Deno compiler (2.9.7, denoVersion in binary/node-build.json, which must be the compiler of the pinned Pallet control executable) and scripts/build-node.ts --release --gitea-manifest, which creates both Linux node archives, the five standalone CLI assets, the installer and legal files in one verified TsPack set. Its strict schema-1 attachment manifest also includes the TsPack manifest and checksum list. All five executables are compiled from one owner-verified private DENO_DIR with the frozen lock, --node-modules-dir=none and --exclude-unused-npm; checkout node_modules is never a release input. Every compile runs through @git.zone/tsdeno (pinned in deno.tools.json), which leaves out npm files the target provably cannot load and prints a size report. Each executable's compile target in .smartconfig.json (@git.zone/tsdeno) also drops NoSQLDB's packaged engines and debug server and UI, which Spark never loads because it starts NoSQLDB only with the bundle's verified engine; deno task compile builds the same targets. The same section pins the Deno release (denoVersion, which the release build requires to equal the node recipe's) and gives each executable a size floor and budget and a smoke check. TsDeno refuses any other Deno, fails a binary below its floor (Deno 2.9 can exit 0 with a fraction of the binary when npm downloads fail) or above its budget, and on the host's own target runs it once as spark --version in an empty environment, where it must exit 0 and print its version; binaries for other targets skip the smoke check. The release build hands TsDeno its own running compiler by path. With Deno 2.9.7 the Linux executables are about 280 MB (439 MB before the trim, with Deno 2.9.4) and the macOS and Windows ones 232–242 MB (from about 410 MB). Notice generation resolves that same isolated graph, and the exact sorted npm package identities and their digest are recorded in the notices and versioned build receipts. The builder requires those inventories to match before compilation, rechecks the graph, source commit and lock after all targets finish, and removes the private dependency cache before TsPack seals the artifacts. GitZone freezes those exact bytes in its schema-4 release journal before Git or Gitea mutation, publishes them through the configured target, and verifies every public attachment anonymously. If publication is interrupted after the journal is written, gitzone release resume reuses the retained bytes without rebuilding. The tag workflow has no publication credentials or build authority; it verifies the completed public release, its source commit, full asset set, checksums and both Linux architectures. It polls for at most one hour only while the release is absent or its canonical attachment set is still incomplete, permits at most three consecutive transport failures, and rejects a complete invalid release immediately.

Spark's documents are owned by @lossless.org/client/nosqldb, the persistence client that vendors the SmartData document implementation; the compiled qualification report states the exact upstream baseline it was built from.

On Linux, deno task test also compiles and runs an isolated NoSQLDB identity-engine qualification. It uses a verified published native executable and disposable SmartData records to check exact engine selection, exclusive ownership and transaction recovery after engine termination. It also recovers the new private Spark identity store's pending and acknowledged credentials after separate engine terminations. This is a migration prerequisite; the Spark daemon still uses its existing configuration store. A successful local test does not qualify production cutover, host power-loss recovery or another CPU architecture.

The backend-private ts/identity/ module durably prepares Spark-only credentials, binds an exact outbound request digest, and activates only an exactly matching acknowledgement. Immutable receipts support replay without restoring superseded credentials. The module is not wired into the daemon or exported as an enrollment API: daemon startup integration, configuration activation, sidecar installation/self-update and the atomic Cloudly enrollment exchange are still required. Pallet owns a separate store and credential.

Private enrollment helpers now consume the published Interfaces 32.18.0 contract. prepareAdoption requires imported Spark authority and authenticated Cloudly state; it records the source fingerprint, original proof hash and exact generation/session fence without activating either credential. The caller must authenticate that state over HTTPS using the exact imported bearer. Generic prepare/bind/acknowledge calls cannot bypass this adoption path. Shared binding checks Spark's own prepared hash against the complete Spark/Pallet proposal, and activation requires an exact shared acknowledgement. The immutable receipt retains adoption provenance.

readEnrollmentRecovery is a private coordination capability, not a status or IPC API. It returns only Spark's bearer for the exact bound pending proposal or its still-current acknowledged identity, allowing recovery across partial activation. Changed proposals, unbound material and superseded credentials are rejected. It never returns Pallet's bearer or the original imported proof.

The private ts/enrollment/ coordinator now persists an immutable operation intent before external I/O, followed by the complete prepared proposal, both-owner binding, authenticated Cloudly acknowledgement and confirmed local completion. Restart resumes that same intent and credentials. A failed initial exchange permits one protocol-defined pending-Spark recovery request; an unconfirmed result stays bound and never creates replacement credentials. Saved acknowledgements bypass network enrollment and resume the independent Pallet-then-Spark activation.

Its owner ports are private integration boundaries. Status contains only stage, enrollment ID, Cloudly origin and node ID. Completed enrollment is historical installation evidence, not proof of current remote authority or workload health. Source tests use a real native Spark store and test-only external-owner stubs; the compiled fixture also recovers bound, acknowledged and complete coordination state after native engine termination. Daemon integration remains required before production cutover.

SparkEnrollmentHttpTransport implements the Cloudly port using a fixed canonical HTTPS origin and TypedRequest 8.0.4. Both RPCs use exact published contracts; no redirect, cache, implicit retry or stream transport is enabled. One request deadline covers response consumption, request/response bodies are capped at 32/16 KiB, and cancellation drains the actual fetch/body owner before returning. Credential RPC hooks are disabled and errors omit remote text, URLs, nested causes and arbitrary abort reasons. Shared response validation still proves exact node/proposal binding. Native TLS tests use a disposable CA and server certificate; production uses normal certificate verification. The adapter is not yet connected to daemon startup, and Cloudly's new enrollment handlers are still not exposed.

SparkEnrollmentPalletTransport implements the local owner port against published Interfaces 32.18.0 requests.pallet methods. It accepts trusted installer metadata, defaults to UID 0 and /run/serve.zone/pallet/control.sock, and requires a protected mode-0700 parent and mode-0600 socket. It rejects symlinks, unsafe ancestors and changed socket/parent inodes before sending enrollment data. No database access, plaintext Pallet bearer, TCP fallback or permissions repair is available.

Each TypedRequest uses one bounded length-prefixed JSON frame followed by request write-half-close, with 32/16 KiB request/response caps. The client requires the full reply and EOF, strict UTF-8, matching envelope identity and shared proposal/phase/ assigned-node bindings. Hooks, implicit retries, caches and stream transports are disabled. One default 10-second deadline aborts socket I/O; the actual connection closure and pending path checks remain joined before return. Once a peer has half-closed its request, the server still needs its own deadline to bound a lost client while no response is being written. Endpoint protection assumes trusted root/installer ownership, not protection against a hostile root process.

Real Unix tests cover all three methods, Jump/adoption bindings, malformed frames, endpoint ownership/replacement, pre-await input capture, cancellation, deadlines and caller-directed lost-response replay. Both adapters remain private and are not yet connected to the running daemon or production bootstrap.

SparkIdentityRuntime owns the private native database lifecycle using NoSQLDB 10.9.2 fileStorageStartup: 'current-format-only'. It verifies the selected engine digest and protected paths; NoSQLDB admits current layouts read-only and acquires its native file-root lease before storage writes. No Spark advisory lock or local format detector remains. Each start uses a separate mode-0700 disposable IPC directory so a rejected competitor cannot remove the active owner's socket.

Shutdown drains accepted callbacks, closes SmartData, then stops the engine. A failed database close retains the native lease for cleanup retry. Startup and shutdown deadlines bound caller waits without abandoning actual work or allowing restart before cleanup settles. Ordinary startup refuses missing storage instead of creating an empty replacement; explicit first installation alone provisions it. Strict legacy/mixed/staging rejection is tested without storage mutation. The compiled test uses this owner directly and verifies the exact 10.9.2 static musl engine provenance; production bootstrap wiring remains separate.

test/native/qualify-identity-upgrade.ts qualifies a disposable store written by the NoSQLDB 10.5.1 engine that Spark 34.0.0 to 35.1.5 shipped through 10.9.2 startup, writes and a second restart. Supply the two published static musl engine paths in that order (the dist_rust/rustdb_linux_<arch>_musl executables of @lossless.org/nosqldb 10.5.1 and 10.9.2, which Pallet ships byte-identical as pallet-smartdb); the fixture checks their independently pinned SHA-256 values. It preserves active identity, an unfinished rotation and settings. Downgrade requires a pre-upgrade backup.

test/native/qualify-released-store-readthrough.ts qualifies one document implementation against another over a store the other one wrote. Run it as record <seededStoreRoot> <dumpPath> on the implementation that owns the store's bytes and as verify <seededStoreRoot> <dumpPath> on the implementation under qualification; the dump is the canonical byte image of every document the released-store conversion leaves behind. The supplied store is consumed, because the engine binds a store to the path it was created under: seed a fresh store at the same path for each run. The ledger contributes ids only, since its applied timestamp is wall-clock.

SparkPalletProcess owns one verified foreground Pallet control process through Deno's native process API. Trusted bundle metadata supplies its protected absolute path and SHA-256. A bounded file verifier permits the larger compiled control while preserving the separate 256 MiB NoSQLDB engine limit. Its default enrollment mode selects enrollment-serve or explicitly requested enrollment-provision. Trusted mode: 'runtime' selects runtime-serve with pallet.node.process.v1 readiness and rejects provisioning; the two modes use separate child lifetimes.

The owner admits its scoped callback after Pallet's versioned readiness event. Parent cancellation, invalid process output or child exit closes admission and cancels the callback's signal; the callback must drain its admitted work before returning. Stdin carries only the parent lifetime. Shutdown closes it for graceful EOF, then escalates to SIGTERM/SIGKILL after bounded grace periods if needed. The owner waits for actual OS exit and output drainage; unconfirmed exit retains the lifetime instead of permitting a restart. Forced termination is a failure, and the owner never starts a replacement child. The runtime's start allows 600 seconds by default, Pallet's documented node start limit (Pallet's readme, "Foreground node process"; palletNodeStartTimeoutSeconds in @serve.zone/pallet-bundle): a start fences what the previous lifetime retained before ready, so a start after a crashed runtime withdraws the retained ACTIVE generation first, and Pallet's per-request deadlines add up to 469 seconds on the longest path that does not grow with the node's workloads. Each retained workload attachment adds 5 seconds, so a trusted caller may allow up to an hour, Pallet's own maximum. The runtime's shutdown allows 330 seconds: Pallet bounds one native operation at five minutes, then the database joins (Pallet's "Service unit contract"). Enrollment allows 15 seconds for each. The node unit is Type=exec, so no TimeoutStartSec= cuts that start short; its TimeoutStopSec=360 covers a stop's 330 seconds. An update or a rollback that restarts the node waits for ready just as long: it allows the node's whole start, 712 seconds (sparkNodeReadinessBudgetMs), which is the identity runtime's 10-second start, the 72-second runtime-binding proof (the enrollment child's 15-second start and stop, its three 10-second Pallet requests and one 10-second Cloudly request, and two 1-second termination graces), the runtime's 600-second start, and 30 seconds for the local work between them. When Spark's own start deadline ends the child, the failure is startup_timeout, with what the stopped child failed with as its cause, for example runtime_failed (pallet_process startup_timeout: owner_failed), or runtime_failed (pallet_process startup_timeout: shutdown_failed) when the child also missed its stop, had to be signalled and reported no failure of its own. Runtime output is drained with bounded chunks and 4 KiB stdout lines; enrollment additionally retains its 1 MiB cumulative output limit. Static errors prevent child diagnostics from becoming public error text.

A failed lifetime names what ended it apart from what a later start can pass:

Failure	Cause	Repeats on the next start
artifact_rejected	Verification refused the executable: its SHA-256 differs from the bundle's, or the file or a parent directory is unsafe (type, owner, mode, link count, or changed while it was read).	Yes
launch_failed	Spark could not read or start an executable it has not refused: an I/O or resource error such as EAGAIN, ENOMEM or EMFILE from opening, reading or spawning it.	No
protocol_invalid	Pallet wrote output outside its control protocol: a malformed or unexpected PALLET_CONTROL line, a failed reason outside the protocol, or a line or output over its bound.	Yes
output_interrupted	Reading the child's output failed, or the output ended inside a line or a character, and no exit or stop explains it. A child killed while writing is owner_failed, the failure its unrequested exit names.	No

deno test --allow-all test/palletprocess.test.ts

The focused tests cover protected artifact validation, readiness, provisioning selection, cancellation, callback failure, malformed output, startup timeout, forced cleanup and runtime log drainage beyond the enrollment lifetime limit. The node recipe pins the published Pallet 36.3.12 release and its NoSQLDB 10.9.2 engine. Before compiling Spark, packaging verifies its control record, source provenance and Pallet's own containerd CRI socket /run/pallet/containerd/containerd.sock (a build naming the host's shared /run/containerd/containerd.sock, which on a Docker CE host is Docker's daemon, is refused), and the exact 95-file control inventory: 11 native roles, 24 clock files, the five pallet-containerd/ files (containerd, its runc shim, runc, the pause image archive and their manifest, whose digest the control record carries as containerdManifestSha256), 38 SmartDNS notices, 3 CoreDNS notices, and 14 SmartVPN notices. Every declared file must match the sealed archive's size, hash, and mode. The whole upstream legal tree is included under pallet/: third-party-notices.md indexes it, native-notices/ holds the executor's generated native notices (every file named by native-notices/manifest.json), and notices/containerd/ and sources/containerd/ hold the containerd distribution's license texts and corresponding sources, which notices/containerd/manifest.json lists completely. Packaging and notice generation both refuse a missing, changed or unlisted file in either inventory. SmartNFTables 4.6.1's guard binary and build provenance are independently pinned for each architecture, and both verify its complete tsrust.native-notices inventory under pallet/notices/smartnftables/ against the pinned manifest digest. Installation requires the runtime, guard, DNS, VPN, clock and containerd executables with their provenance and notice manifests. It admits the sealed empty pallet-clock/gnutls.config and rejects undeclared executables. Runtime supervision is not yet wired into the external service activation flow or self-updating. runnode composes it with the ordinary local lifetime described below. The released asdaemon command still uses the legacy service managers on a host that has not been converted, and refuses to start once this host's boot path is the node runtime.

Foreground node runtime

Run the selected compiled executable with runnode --forwarding-mode <docker-shared|exclusive> and no other argument; the node unit does (see Host forwarding mode). The command verifies the installed whole bundle and requires its selected Spark path to match the running executable. A bare download, stale selected version, or deno run mod.ts runnode cannot start node ownership.

/opt/spark/current/spark runnode --forwarding-mode docker-shared

SparkNodeRuntime reopens existing Spark identity and settings and requires a complete persisted enrollment. Startup then runs in a fixed order:

1. Inside the Pallet enrollment child, the only Pallet process that serves its control socket, reconfirm both current local identities.
2. In the same child, prove that Pallet holds this node's runtime routing binding. A binding Pallet already holds is read back offline, so a bound node restarts without Cloudly; activateguard binds it on a converted host before the first start. A node without one reads its current binding from Cloudly (getNodeRuntimeBinding, authenticated with the node's Spark credential) and binds it once. activateguard runs this same proof.
3. Join the enrollment child, then start the Pallet runtime, which refuses to start without a binding.

Cloudly answers the binding only once the node's cluster relay is live. A refused or unreachable answer, an answer that does not bind to this node, and a binding Pallet refuses each stop startup by name (runtime_binding_unavailable, runtime_binding_invalid, runtime_binding_conflict) together with the Cloudly or Pallet cause behind it, for example runtime_binding_unavailable (cloudly unreachable: tls_untrusted_issuer) followed by the remedy, with no retry inside the process, no direct-to-Cloudly fallback and no runtime start. Any other startup failure is runtime_failed with the owner and code that ended it; a kernel Pallet's packet engine refuses is runtime_failed (pallet_process unsupported_kernel) followed by the remedy (see Requirements). Pallet holds the binding immutably: a moved relay needs a rebind capability that Pallet and the interfaces contract do not offer yet. Missing state or settings fail; ordinary startup never provisions replacement data, imports legacy configuration or resumes an incomplete remote enrollment. Only disposable IPC parents are recreated.

A failed runnode prints its error on standard error and exits with a status the node unit acts on (see Stable systemd boot entry):

Exit status	Failures	Service manager
0	A stop by SIGINT or SIGTERM whose cleanup completed.	Not restarted.
78	Refusals every start repeats until an operator acts on the host: reboot_pending (reboot), invalid_input (a missing or unknown forwarding mode among them), bundle_missing, executable_mismatch (start the selected bundle's executable), enrollment_unconfirmed, runtime_binding_conflict, runtime_binding_invalid whose cause is Pallet's malformed answer, and runtime_failed whose cause is pallet_process unsupported_kernel, forwarding_mode_refused (the host does not meet the stated mode), invalid_ca_store, artifact_rejected or protocol_invalid.	Not restarted; the unit stays failed.
1	Every other failure: runtime_failed with any other cause (owner_failed among them, as an outage of Cloudly beyond the tunnel lease ends the runtime, and launch_failed or output_interrupted, an I/O or resource failure starting Pallet or reading its output), runtime_binding_unavailable, runtime_binding_invalid whose cause is Cloudly's answer (a response cut short while Cloudly restarts reads as malformed), identity_unavailable, shutdown_failed, and a control socket another process owns.	Restarted after 5 seconds.

Whatever a pallet-control child writes to its standard error (in any mode Spark runs it: runtime-serve, enrollment, guard and network acquisition) reaches Spark's own standard error, and so the unit's journal, as Pallet <mode> stderr: <line>, with control characters escaped, each line cut at 512 characters and at most 64 lines per minute; the lines beyond that are counted in one N further line(s) not shown line. Pallet writes no diagnostics of its own there, so these lines are chiefly its runtime's report of an error Pallet did not handle, the one cause an owner_failed cannot name.

The command emits SPARK_NODE records with protocol spark.node.process. The ready event reports the selected bundle, secret-free enrollment summary, settings snapshot, localRuntimeReady: true and workloadReadiness: "unverified". SIGINT or SIGTERM closes admission, joins reporting and any host update, and waits for Pallet and Spark storage cleanup before emitting stopped. Failed or forced cleanup remains an error even when a stop signal initiated shutdown. Loss of either owner cancels admitted work; no replacement process starts within the same lifetime.

After local readiness, the runtime collects CPU, memory and root-filesystem observations directly from the host and sends the node heartbeat to POST /spark/nodes/heartbeat at the exact enrolled HTTPS origin. Every attempt reads the current private Spark credential and requires a receipt for that node and credential generation. Requests and responses are limited to 8192 and 2048 bytes; the ten-second deadline includes reading the response. Redirects, cookies, response caching and immediate retries are disabled. Completed attempts wait the persisted heartbeatIntervalMs before the next sample. That interval is at most 50 seconds: Cloudly counts a node offline once its last accepted report is older than clusterNodeObservationFreshnessMs (180 seconds, @serve.zone/interfaces), and three passes of one interval plus one ten-second exchange must fit into it, so two failed passes in a row never take a running node offline. The bound is derived from the published constant; a longer setting is refused with heartbeat_interval_exceeds_freshness, and the migration heartbeat-interval-within-freshness shortened a longer stored one to 50 seconds once (the released configuration and a released store are projected the same way as they are read). The default is 30 seconds. Cloudly unavailability does not block local readiness; a receipt records accepted host observations and does not establish workload readiness or deliver operator actions. A Cloudly that refuses this build's protocol offer is reported as protocol-incompatible and re-offered on the refusal cadence, without dropping local readiness. A Cloudly that cannot be connected to at all is reported as unreachable with its cause and remedy (Spark host reporting: unreachable (tls_untrusted_issuer). …), apart from unavailable, which is an exchange that failed after its request may have been sent; a new cause is reported as a new state. Diagnostic state changes go to stderr and contain no credentials or remote response text.

Cloudly owns the node's static hostname (default cloudly-worker-<name>). After each accepted heartbeat the runtime reads the desired hostname from POST /spark/nodes/host with the same credential, origin, deadline and refusal handling (1024-byte request and response bounds). When Cloudly holds an intent whose hostname differs from the host's, Spark rewrites the first 127.0.1.1 line of /etc/hosts to name it (a synced temporary file under the file's mode and owner, renamed over it in /etc, the directory then synced). The line is added after 127.0.0.1 when absent, further 127.0.1.1 lines are dropped, and every other line is left as it was. Spark then runs /usr/bin/hostnamectl set-hostname <hostname>. Any failure after the hosts file was read ends with the mapping converged back to the hostname the host actually carries, and the attempt fails. A host that already carries the hostname keeps a 127.0.1.1 line that names it and has one naming another host repaired; a value that is not one lowercase RFC 1123 label is refused without a command. The next heartbeat states the outcome as hostApply (applied, or failed with a short reason), held in memory only, and runtime.hostname is read on every pass. Each new outcome is logged as Spark hostname: ….

When the persisted hostUpdatesEnabled setting is true, a separate joined owner runs /usr/bin/apt-get update, upgrade -y, autoremove -y and autoclean -y at the next host-local midnight, in that order and without overlapping runs. There is no immediate startup run or replay of missed offline schedules. A failed command stops that day's sequence. Cancellation terminates and joins the process group, allowing up to five minutes before forced termination; an unproven cleanup remains a runtime shutdown error. This owner does not replace the selected Spark bundle. Settings are read once per runtime lifetime, so changed settings take effect after a controlled drain and restart.

runnode now wraps the runtime in SparkNodeService. It holds a private Unix HTTP control socket at /run/serve.zone/spark-node/control.sock, with a root-owned mode-0700 parent, mode-0600 socket and retained empty IPC lock anchor. This process lock is acquired before starting data owners and remains held after they drain. A competing service cannot replace the listener or start another runtime. Stale socket retirement requires the lock, a refused connection and matching inode and ownership checks. Native HTTP owns final socket cleanup; privileged operators must not replace its pathname while the service is running.

SparkNodeControlClient reads bounded spark.node.control status containing the actual PID, random process-instance ID, phase and exact bundle version, source and manifest. A drain command must match the observed instance. It closes admission and reports draining until both data owners have joined; only then does it report drained. The process and control socket remain alive until SIGINT/SIGTERM, allowing an external installer to prove drainage before stopping the systemd unit. Repeated drains of that instance are idempotent. Losing the caller does not cancel an accepted drain, and failed cleanup never produces a successful drained result. No readiness, settings, credentials or other application data are written to files.

SparkNodeServiceActivation composes this client with the existing systemd activation policy inside SparkBundleInstallation.run. It matches the old PID and selected manifest before drainage, then verifies the new PID and exact selected bundle after start. Private HTTP requests have a five-second bound; startup observation allows the node's whole start budget, 712 seconds (see the Pallet process lifetime above), and drainage 360 seconds. A rollback that restores the previous release after a failed update observes the restarted node for the same 712 seconds, whatever its stop took before. These are observation limits; a timed-out caller cannot undo or abandon server-side drainage. This path preserves masks and enablement and rejects the alternate spark.service unit.

The same activation owns Pallet's containerd unit across a bundle change. Before any effect it admits that unit only absent (a host whose units predate it) or loaded with the service settings the contract requires, as the service manager reports them with drop-ins included (unsafe_containerd otherwise), and a running containerd-serve only when its own executable is the pallet/pallet-control of a release of this installation. After the node unit has drained and stopped and the new bundle is selected, a running containerd whose release carries a different pallet-containerd/manifest.json is stopped and started again, so it runs the new selection's containerd, shim and runc, before the node unit starts; the restarted process must run from the new release with the same contract. KillMode=process and Delegate=yes keep every shim and container running across that restart, and the new containerd reattaches them. An unchanged runtime keeps running, and a stopped containerd is left for the node unit to start. The result states containerdRestarted.

Stable systemd boot entry

After installing and selecting a complete node bundle, configure its service with:

/opt/spark/current/spark installnodeservice --forwarding-mode docker-shared

The command takes the host's forwarding mode and no other argument, and requires root, a verified current bundle and a quiescent canonical service. It holds the same native installation lock used by bundle selection. SmartDaemon 2.19.0 installs /usr/local/lib/systemd/system/smartdaemon_spark.service with direct execution of /opt/spark/current/spark runnode --forwarding-mode <mode>, Type=exec, KillMode=mixed and a 360-second stop deadline. A node unit already installed for the other mode is refused (forwarding_mode_change); the mode changes only through spark setforwardingmode. A service manager older than systemd 254, or one whose version cannot be read, is refused before either unit is written (systemd_too_old, systemd_version_unknown; see Requirements). The atomic current selection is the stable boot path; the node unit explicitly requires and starts after the retained guard unit, so a failed guard also blocks node startup and its database owners.

A failed node lifetime is restarted by the service manager, never inside the process, as Pallet's node unit contract requires (Pallet's readme, "Service unit contract"):

[Unit]
StartLimitIntervalSec=0s

[Service]
Restart=on-failure
RestartSec=5s
RestartPreventExitStatus=78

RestartSec=5s is Pallet's reference node unit value. The start rate limit is off, so no DefaultStartLimitIntervalSec= of the host makes the manager give up on the node during a long outage: a node whose Cloudly stays away longer than its tunnel lease fails once per lapsed lease, and each new process fences the lapsed generation and waits for its controller. runnode exits 78 (sysexits EX_CONFIG) for a refusal every start would repeat until an operator acts, which leaves the unit failed with that message in its journal (see Foreground node runtime); every other failure exits 1 and is restarted. A stop (systemctl stop, SIGTERM) is never restarted, whatever its status. Admission reads this contract back from the service manager, drop-ins included: an update that keeps the node definition refuses a unit loaded with another restart policy, exit-status handling, kill mode, a shorter stop deadline, another descriptor store or notification access, or a command line that is not runnode --forwarding-mode with a mode (SmartDaemon reads the loaded ExecStart from the unit's D-Bus property through busctl) (unit_definition_changed), and so does activateguard (configuration_changed).

The node unit keeps Pallet's router network namespace across its restarts and stops, so a restarted runtime can adopt the namespace its workloads are attached to:

[Service]
Type=exec
NotifyAccess=all
FileDescriptorStoreMax=1
FileDescriptorStorePreserve=yes

Pallet's native, below runnode and pallet-control runtime-serve, stores the namespace descriptor through the unit's notification socket (FDSTORE=1, FDNAME=pallet-router), which needs NotifyAccess=all; systemd 255 sets NOTIFY_SOCKET for every unit whose NotifyAccess= is not none. On the next start systemd hands the stored descriptor to runnode (LISTEN_FDS, LISTEN_FDNAMES). runnode marks every passed descriptor close-on-exec before anything else runs, removes LISTEN_PID, LISTEN_FDS, LISTEN_FDNAMES and NOTIFY_SOCKET from its own environment, and hands only the pallet-router descriptor and the notification socket to Pallet's runtime: the descriptor as the child's descriptor 3 with --router-namespace-descriptor 3, the socket as NOTIFY_SOCKET beside PATH, nothing else. Pallet 36 and later receive both and the forwarding mode; a bundle with an earlier Pallet starts runtime-serve without them. A reboot empties the store; spark setforwardingmode and the uninstaller release it.

The same command installs Pallet's containerd unit, /usr/local/lib/systemd/system/smartdaemon_spark-containerd.service, to the contract Pallet's readme states: /opt/spark/current/pallet/pallet-control containerd-serve as the root main process with null stdin, Type=notify with NotifyAccess=all, KillMode=process and Delegate=yes (stopping the unit stops containerd and leaves every shim and workload running), Restart=always with a short RestartSec=, a 60-second stop deadline, LimitNOFILE=infinity, TasksMax=infinity and OOMScoreAdjust=-999, and Requires=/After= the guard unit. The node unit Wants= it and starts after it, so the node runtime, containerd's only CRI client, never starts before Pallet's containerd is ready. Spark remains the systemd main process and verifies its selected executable before opening its data owners. Graceful shutdown signals Spark first, allowing it to join Pallet and storage before the whole-cgroup failure deadline.

Installation preserves existing masks and enablement, retains previous generated definitions and verifies the reloaded manager view. It leaves an absent service disabled. It never starts, stops, enables or unmasks a unit, initializes data, changes settings or constructs the legacy daemon. An active owner, alternate spark.service, administrator shadowing definition or effective drop-in is rejected. A failed reload leaves the new definition for explicit inspection; selection and definitions are never automatically rolled back.

Successful installation emits a SPARK_NODE_SERVICE JSON record with the exact selected bundle, boot path, definition digest and changed/masked flags. Service start and enablement remain explicit operations after enrollment and rollout admission. External bundle activation supports both control-group and mixed, requires the kill policy to stay unchanged during activation, and still proves private drainage and exact new-PID readiness independently of systemd state.

Production conversion from the legacy service remains gated on rollout admission. Local readiness does not establish a Cloudly session, containerd workloads, private DNS or production takeover.

Host forwarding mode

Who owns the host's IPv4 forwarding is a fact of the host, which the operator states on the node unit and Spark passes to Pallet on every start (pallet-control runtime-serve --forwarding-mode <mode>); nothing infers it (Pallet's readme, "Host forwarding modes"):

• docker-shared: Docker runs on the host and owns forwarding, with its iptables FORWARD policy DROP. Pallet never writes forwarding and passes its flows through Docker's DOCKER-USER chain. The host provides the iptables-nft 1.8.10 or newer frontend (/usr/sbin/iptables-nft, iptables-nft-save, iptables-nft-restore, root-owned) and Docker's DOCKER-USER and DOCKER-FORWARD chains as the first two jumps of FORWARD.
• exclusive: nothing else forwards. The host runs no Docker under systemd and persists no IPv4 forwarding in sysctl configuration; Pallet switches forwarding on only while a generation is up behind its exclusive packet table.

A host that does not meet the stated mode is refused by Pallet at start; runnode exits 78 (runtime_failed (pallet_process forwarding_mode_refused)), and the node unit stays failed until the host or the mode is corrected.

Moving a host from docker-shared to exclusive (D9) is done in this order only:

systemctl stop smartdaemon_spark.service       # docker-shared: withdraws the generation; forwarding stays Docker's
/opt/spark/current/spark setforwardingmode exclusive
# remove Docker (stop, disable, purge and mask docker.socket, docker.service, containerd.service)
# remove every persisted IPv4 forwarding setting (/etc/sysctl.conf, /etc/sysctl.d, networkd IPForward=)
systemctl reboot                                # the node starts exclusive

What a stop of the node does depends on the mode the running node states (Pallet's readme, "Host forwarding modes"): a docker-shared node withdraws the generation it holds and releases its Docker forwarding contribution and host table, and never writes the host's IPv4 forwarding; an exclusive node lowers only its managed VPN tunnel, switches the host's IPv4 forwarding off and keeps the generation UP in its journal, which the next node process fences. Neither stop withdraws the node's allocation protection: from Pallet 36 a stop, a restart, a crash, a reboot and being offline all leave the node's last protection receipt current, because its allocation-pool guard stays in the kernel and its boot unit restores it before networking. Only Cloudly's retirement request withdraws it (a withdrawn retirement of the node, retireClusterNode), which the running node answers.

setforwardingmode <docker-shared|exclusive> admits only a loaded, stopped (inactive, not failed) node unit that is a definition of this release as the manager loads it; a stopped node holding its namespace reads dead-resources-pinned on systemd 254 and later and is admitted. It replaces the unit with the stated mode's definition and releases the unit's descriptor store (systemctl clean --what=fdstore), which holds the previous lifetime's router namespace, then prints one SPARK_NODE_FORWARDING_MODE record. A failed unit is refused (node_failed): systemd releases a store only of an inactive unit, so run systemctl reset-failed smartdaemon_spark.service first. Releasing the store is the change's commit, which systemd refuses for a unit that is not inactive. Once the manager accepted it the change is done, and a node started right after is left alone (the record's fileDescriptorStoreCount is then above 0, or null when the read after the release failed). A clean that fails is read back first, because a failed client may hide a release (SmartDaemon 2.18): a stopped node with an empty store counts as released, a stopped one that still holds descriptors fails store_not_released. A node started before the commit fails the change: node_started when it started before the unit file was replaced (nothing is undone, the node keeps running), node_started_restored when it started after (the node is stopped and the previous definition restored), restore_failed when that restore did not complete, and state_unknown when the unit could not be read afterwards. Rerunning it is safe. Never remove Docker from a running docker-shared node and carry on without the reboot: Docker's forwarding stays on with no owner left for its FORWARD policy.

Retained allocation-pool guard boot entry

After enrollment, while management networking is available, the privileged installer runs /opt/spark/current/spark activateguard. The command accepts no path, provider, policy or credential overrides. Its supported host profile uses the verified systemd-networkd service; an alternate network manager, pending provider reload, administrator drop-in or active Spark runtime prevents activation. An already installed node unit must also be unmasked, loaded and stopped, and require and order itself after the guard, and the service manager must load it with its restart contract (see Stable systemd boot entry; configuration_changed otherwise); rerun installnodeservice --forwarding-mode <mode> while it is stopped to update an older definition. Qualify the host's actual interface ownership before using this profile.

One bundle installation lock spans the complete operation. Before a guard unit exists, Spark first proves this node's runtime routing binding with the node runtime's own proof: it opens Spark's existing identity store, reconfirms both current identities inside Pallet's enrollment child, reads Cloudly's routing answer once when Pallet holds no binding, binds it there, and joins the child and drains the store. Pallet's acquisition dials the relay the binding names and has no other route. A node without a complete enrollment stops with enrollment_unconfirmed; a refused, unreachable or foreign answer stops with runtime_binding_unavailable, runtime_binding_invalid or runtime_binding_conflict and its cause, before acquisition, commissioning or any unit change. Spark then runs the verified Pallet control's network-acquire mode. That projection-only connection retains fresh authenticated authority without starting native owners, accepting assignments or delivering runtime reports. Spark requires the exact pallet.network.acquisition.v1 ready/stopped sequence, drained pipes and successful child exit before explicitly commissioning Pallet's durable guard lane. A failed or cancelled acquisition leaves commissioning and unit installation unstarted. A kernel Pallet's packet engine refuses stops acquisition, commissioning and recovery with Spark guard process failed: unsupported_kernel. and the remedy (see Requirements). It then installs smartdaemon_spark-guard.service as a disabled Type=oneshot unit with RemainAfterExit=yes, manually starts it, and persistently enables it. The exact definition orders recovery after local filesystems and before network-pre, networkd and shutdown; both network-pre and networkd require it. Success requires the exact file digest and fragment, no drop-ins or pending reload, completed oneshot state, and both forward and reverse dependency edges. Enabled state alone does not establish complete boot protection.

The unit executes /opt/spark/current/spark guard-recover. That launcher verifies the selected whole bundle and its own executable, then runs its authenticated Pallet control with null stdin. It uses read-only bundle inspection so it can run while the activating parent holds the installation lock. The bounded child owner requires pallet.guard.process.v1 ready/stopped events, drained pipes and successful actual exit. Cancellation or forced cleanup fails the boot prerequisite.

An existing exact unit admits offline recovery only and never repeats projection acquisition. Interrupted reload, start or partial enablement is resumed from retained state; missing guard data is never silently recommissioned. Masks and foreign definitions reject without replacement. Errors retain the guard policy and any installed dependencies. Completion emits one SPARK_GUARD record identifying the bundle and definition. This deny-only bootstrap does not publish a positive network barrier, release allocations or establish workload/DNS readiness.

test/helpers/palletenrollment.fixture.ts is an explicit cross-artifact qualification executable. Supply absolute paths for the Node executable, built Pallet control module, exact NoSQLDB engine and its independently verified SHA-256, and the built Cloudly enrollment store module:

deno run --allow-all --frozen test/helpers/palletenrollment.fixture.ts \
  /path/to/node /path/to/pallet/dist_ts/control/classes.enrollmentcontrol.js \
  /path/to/rustdb_linux_amd64_musl VERIFIED_ENGINE_SHA256 \
  /path/to/cloudly/dist_ts/manager.node/classes.nodeenrollmentstore.js

It runs real Pallet and Cloudly owners in separate Node processes and the real Spark coordinator with independent stores. Cloudly's actual enrollment/credential stores and installed TypedRequest router run behind a disposable loopback HTTPS listener; Spark uses its actual HTTPS client with an isolated test CA, not disabled TLS checks. Fresh Jump and imported-node adoption both recover a committed-but-lost TLS response and a lost local activation result. All three owners and their exact NoSQLDB engines then restart from separate file stores and converge on the same node/credentials. Assertions verify one enrollment receipt, two distinct credential authorities, the expected generations, rejection of the old Spark credential, settings continuity, and unchanged synthetic legacy bytes. The Cloudly authority stub has been removed.

This qualifies the private store/transport integration, not the production CloudlyServer route/admission lifecycle, a distributable node bundle, daemon wiring or deployed workload readiness. The isolated fixture registers no production route and introduces no local package link or application dependency on sibling sources.

The private runtime context now includes SparkSettingsStore. It stores only typed hostUpdatesEnabled and heartbeatIntervalMs settings; connection origin, node identity and credentials remain owned by enrollment and the current identity. Ordinary reads reject missing settings. Explicit fresh installation initializes the false/30000 defaults, and revision-checked full updates reject stale writers and counter exhaustion. Responses are detached/frozen and contain no credential or migration-source metadata. An ambiguous write must be reconciled by reading the saved revision; it is not automatically repeated.

The settings projection follows the legacy import, copies operational settings once and retains the exact import source digest as completion provenance. Later startup/import replay preserves user edits. Explicit import preflights any fresh-initialized settings before the import may persist authority, and snapshots installer options before those checks. Without an import, startup leaves settings absent until explicit provisioning. Tests cover concurrency, ambiguity, restart and native crash recovery. These are the new owner's APIs, not a KV compatibility adapter: the existing daemon/CLI still uses npmextra until coordinated bootstrap replacement connects the new lifecycle and removes those old consumers.

Store migrations

ts_migration/ owns every conversion of state an earlier Spark release wrote. The runner walks an ordered list of idempotent steps, refuses un-migratable state by name before any step writes, and records each completed step as one row in the spark_migrations collection through the same exact models the rest of the store uses. runSparkMigrations() returns the applied and already-applied step ids plus the outcome of the legacy import.

• released-store-conversion moves the settings, identity state, identity receipt and enrollment documents to unversioned ids, drops the settings shape version and renames the legacy source and adoption discriminants. It also drops the shape version from the five shared contract payloads those documents embed, and re-derives the node-enrollment digest, whose domain and hashed payload both changed with the release; the stored copies of that digest in the identity state and identity receipts move in the same transaction, while a digest the enrollment does not derive is preserved exactly. It writes the replacements first and removes the released documents second, so an interrupted run is finished by the next one. A migrated id that already holds a different document is refused as spark-migration-unversioned-id-occupied. The bootstrap import row keeps its released id: it is the completion record of the released npmextra import, and released migration history is immutable.
• coreflow-owner-authority converts the Coreflow owner marker and the Swarm authority labels that state which runtime owns this node. Both lose their shape version; the owner node id and authority id keep their bytes. It runs inside the existing owner fence of SparkCoreflowOwnerManager.ensureAuthority(), because that process owns the Docker Swarm authority and holds no database. The converted marker is its durable completion record. A marker that names a different owner than the Swarm authority, a partial label set, or a marker of unknown shape is refused as spark-coreflow-owner-authority-ambiguous or -unreadable. Services and secrets created by a 1.15.0 owner still carry the retired resource label until the next reconciliation replaces them.

The ts_migration/ runner runs inside that private owner's startup, after protected database readiness and before identity readiness. An explicit installer-only legacyImport option admits the owning user's npmextra JSON file only after the installer has stopped the old writer. The path must be canonical, owned by the expected UID, regular, singly linked, bounded to 1 MiB, and free of symlinks and unsafe writable ancestors. Import never invokes npmextra or repairs permissions. It hashes the original bytes, validates all known settings and stores the configuration, exact original Spark bearer, Coreflow Jump value and completion stage together in one exact SmartData document. Unknown fields, partial credentials and invalid present settings fail closed. The historical file is not modified.

Explicit replay requires the same path and exact bytes. Ordinary startup omits legacyImport and uses protected state without reopening the historical file. Missing optional settings retain the previous daemon defaults; a credential-free source never becomes an enrolled identity. Imported authority is separate from the new private identity store: no generation or acknowledgement is fabricated. The daemon/CLI configuration consumers and the authenticated adoption exchange are still unwired.

Source retirement has an owner in ts/legacy/: one record beside the import in the node's protected store states what this host has done with that source, and nothing else. It is retained-inert at generation one, and removed at generation two after the file's bytes are revalidated against the recorded digest through the same fenced read the import used and then unlinked. The unlink is fenced by its own durable counter, written before the file is touched, so an interruption between the unlink and the transition is repaired by the next call while a first attempt that finds no file refuses by name. Every transition is offered to the contract's own successor rule before it is persisted, so a replay advances nothing and a removed source can never become a retained one. The record holds a digest and two counters, never a member decoded from the file. No command invokes it yet: retirement must occur only after the coordinated Spark and Pallet activation is durable, which is also when adoption has consumed the source's credential.

deno test --allow-all test/bootstrapimport.test.ts
deno test --allow-all test/legacyretirement.test.ts

Source map:

Path	Purpose
mod.ts	CLI entry point and library export.
ts/spark.cli.ts	Command routing.
ts/spark.classes.spark.ts	Main class wiring.
ts/spark.classes.updatemanager.ts	Docker Swarm and service update manager.
ts/spark.baseserviceimages.ts	Base service image pins, refusal rules, and digest references.
binary/base-service-images.json	The pinned version and digest of every base service image.
ts/spark.classes.coreflowowner.ts	Durable Coreflow ownership, marker, lock, and node projection.
ts/spark.classes.swarmobservationmanager.ts	Swarm observation v2 producer.
ts/spark.classes.taskmanager.ts	Scheduled maintenance tasks.
ts/spark.classes.config.ts	Persisted mode/config store.
ts/identity/	Private SmartData credential state machine; not yet used by the daemon.
ts/enrollment/	Private durable Spark/Pallet enrollment coordinator and typed owner ports.
scripts/compile-all.sh	Multi-platform Deno compilation.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.