Spark is a Deno-powered server management agent for serve.zone hosts. It installs as a system daemon, activates Docker Swarm, schedules host/service maintenance tasks, and provides the bootstrap profiles currently used by Cloudly and Coreflow node deployments.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Current Role

Spark is intentionally small and operational. It is not a general-purpose configuration management framework; it is the serve.zone node-side utility that knows how to run itself as a daemon and keep selected Docker services moving.

The current implementation:

Installs and updates a smartdaemon_spark systemd service through @push.rocks/smartdaemon.
Runs in an explicit mode: cloudly or coreflow-node.
Activates Docker Swarm through @apiclient.xyz/docker when daemon mode starts.
Collects host metrics for CPU, memory, disk, and Docker container counts.
Sends Cloudly heartbeats when cloudlyUrl, nodeId, and nodeToken are configured.
Sends authenticated Swarm observation v2 reports after accepted Cloudly heartbeats.
Schedules recurring tasks with @push.rocks/taskbuffer for Spark updates and managed Docker service updates.
Optionally schedules host package updates when --enableHostUpdates=true has been configured.

Spark also owns a callable Swarm observation v2 producer. It reads local Docker membership and, on active managers, attaches a digest-verified node snapshot only when the complete manager view validates. Worker observations never invent the manager-only Swarm cluster ID. Heartbeat transport owns a separate private reporter session, validates bound Cloudly receipts, retries only the exact pending request after ambiguous or retryable delivery, and quarantines terminally rejected sessions until process restart.

Installation

Install a released binary:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash

Pin to the newest release within a supported major channel:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --major 1

Verify a selected release asset without changing the host:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | bash -s -- --version <version> --verify-only

The installer downloads SHA256SUMS.txt and verifies the exact platform binary before stopping services or replacing an existing installation. Release builds currently target Linux x64/ARM64, macOS x64/ARM64, and Windows x64.

On systemd hosts, upgrades preserve smartdaemon_spark.service and its enablement. Only a previously active daemon is stopped and restarted; inactive or failed daemons remain stopped. An existing noncanonical spark.service, uncertain service state, or a command symlink pointing at another installation stops the installer before binary replacement. Resolve that ownership explicitly before retrying; the installer does not delete or adopt another service.

The verified executable is staged on the installation filesystem and renamed over the old binary. Other installation contents and configuration are left alone. An existing executable is retained at the backup path printed during installation. A failed restart returns a nonzero exit status with the new binary still installed; inspect the daemon before choosing recovery. Automatic binary rollback is not attempted because a started version may have migrated state. The default installer mode installs only the standalone Spark executable.

Complete node bootstrap

The node integration after 1.12.0 adds --node-bundle and installnodebundle; use a verified release containing these commands. On a Linux systemd host, run the release's installer with one exact version:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --node-bundle --version <version>

This mode runs the verified standalone executable from disposable staging to download, verify and select the complete native bundle at /opt/spark/current. It preserves the old /opt/spark/spark executable, global CLI link, unit files, mask, enablement and node data. Existing root-owned 0755 code roots retain their mode; new roots and owned release/seal directories use 0700.

The canonical service must be masked and quiescent, or absent, with no alternate spark.service, drop-ins or stale manager configuration. If NeedDaemonReload is set, explicitly reload systemd and re-prove this admission before retrying. Bootstrap never reloads, unmasks, enables or starts a service. It requires a pinned version and rejects --major, --install-dir and --verify-only.

A verified external executable can also run installnodebundle X.Y.Z directly. The SPARK_NODE_BUNDLE result records the selected version, commit and manifest digest. Retry uses that exact release and rejects any different current selection; normal updates use the separate drain/activation owner. Successful bootstrap means code is selected. Credential enrollment through enrollnode and definition installation through installnodeservice remain explicit steps, and production activation requires the separate rollout acceptance.

Requirements

Linux with systemd for daemon operation.
Docker for service and Swarm management.
Root privileges for daemon installation, Docker maintenance, package updates, and prune.
Deno only when running from source.

macOS and Windows binaries are built for CLI/library availability, but the operational daemon paths are Linux/systemd oriented.

Quick Start

Install Spark as a daemon:

sudo spark installdaemon

Run daemon mode with a profile:

sudo spark asdaemon --mode cloudly

or:

sudo spark asdaemon --mode coreflow-node

The selected mode is persisted in a user-home npmextra key/value store under the servezone_spark identity. Later spark asdaemon calls can reuse the stored mode when no --mode flag is provided.

Follow daemon logs:

sudo spark logs

CLI Reference

spark <command> [options]

Command	Purpose
`installdaemon`	Create, enable, and start the Spark daemon service.
`claimjump --claimUrl <url> --jumpCode <code>`	Claim a Cloudly Jump code, persist the returned node/Coreflow credentials, and install/start the daemon. Also accepts `CLAIM_URL` and `JUMP_CODE`.
`enrollnode`	Enroll separate Spark and Pallet credentials from one JSON document on stdin, using an installed verified node bundle. Does not activate the daemon.
`runnode`	Run the selected compiled node bundle in the foreground using existing enrolled identity and settings.
`installnodeservice`	Install the canonical systemd definition for the verified current node bundle, preserving masks and enablement.
`activateguard`	Commission or recover the retained allocation-pool guard, install its boot unit, and verify persistent networkd dependencies.
`guard-recover`	Recover the existing retained guard from the selected bundle; the systemd boot entry uses this command.
`updatedaemon`	Reload the daemon service definition for the current Spark version.
`upgrade`	Install a newer Spark release when available and restart the daemon.
`asdaemon --mode cloudly`	Run the daemon loop with the Cloudly profile.
`asdaemon --mode coreflow-node`	Run the daemon loop with the Coreflow node profile.
`metrics`	Print a JSON host metrics snapshot.
`logs`	Follow `journalctl -u smartdaemon_spark -f`.
`prune`	Stop Spark, remove Docker stacks/services/secrets, remove selected networks, prune Docker, restart Docker, and restart Spark.

prune is destructive. Use it only on nodes where Spark owns the Docker runtime state or where losing all stacks, services, and secrets is intended.

Pallet credential enrollment

spark enrollnode is the external credential bootstrap for a verified, selected Linux node bundle under /opt/spark. The bare-binary installer above does not provide that bundle. Run the command as root with one JSON document on stdin followed by EOF; it accepts no command options, credential environment variables, executable paths, or caller-supplied writer-stop claims. Input is limited to 16 KiB and must finish within 30 seconds.

For an existing node, the input is:

{"operation":"adopt-existing","hostname":"worker.example"}

The command requires smartdaemon_spark.service to be masked and fully stopped, with no pending job, main/control PID, or remaining cgroup. A noncanonical spark.service must be absent. It verifies that state before the one-time import, again before enrollment, and after cleanup. It reads only the canonical root daemon's /root/.npmextra/kv/servezone_spark.json; the versioned migration preserves the source bytes and records the original identity and digest in private NoSQLDB state. The command never changes service enablement or starts an old runtime.

A fresh node instead uses this input, supplying the actual Jump code through stdin:

{"operation":"jump","cloudlyOrigin":"https://cloudly.example","hostname":"worker.example","bearer":"<16-character Jump code>"}

Fresh enrollment also permits an absent canonical Spark unit. Cloudly communication uses only the configured HTTPS origin. Spark and Pallet generate and retain separate credentials in their own protected NoSQLDB stores; Spark receives no Pallet bearer. Explicit provisioning creates missing platform data parents, while ordinary resume only recreates disposable IPC directories.

Resume a persisted attempt without repeating credentials or reading the legacy file:

{"operation":"resume"}

After the coordinator confirms both current local identities and joins the Pallet child and Spark database cleanup, stdout contains one SPARK_ENROLLMENT record followed by JSON with credentialEnrollment: "complete", runtimeReady: false, and a secret-free enrollment summary. Other dependency lifecycle log lines can precede that record. This result does not establish workload or remote-session readiness. A lost response or outcome_unconfirmed must be reconciled through the saved attempt; no second identity or legacy bootstrap is substituted. If failure preceded initial store provisioning, repeat the exact original input to provision it.

Daemon Behavior

Spark.daemonStart() starts the node reporting and reconciliation subsystems before enabling recurring tasks. A failed initial managed-service reconciliation stops already-started subsystems in reverse order instead of leaving a partial daemon running. The asdaemon command handles SIGINT and SIGTERM by disabling actions, terminating active action process groups, stopping tasks and service reconciliation, releasing the Coreflow session, and stopping heartbeat reporting before exit. Shutdown aborts active mutable image pulls and retains the Coreflow owner session until reconciliation settles. A pull that ignores cancellation retains the owner session until that reconciliation eventually settles. Separately, subprocess cleanup that cannot prove exit retains exact process tracking and quarantines the affected lifecycle restart while the daemon continues its ordered shutdown.

SparkServicesManager.start() activates Docker Swarm and, for Coreflow nodes, acquires the host-wide reconciliation lock.
SparkTaskManager.start() schedules recurring maintenance tasks.
SparkHeartbeatManager.start() sends node metrics to Cloudly when node credentials are configured.
SparkMetricsStreamer.start() streams lightweight metrics and, after initial service reconciliation completes, executes Cloudly systemUpgrade, reboot, and serveZoneServiceUpdate node actions when node credentials are configured. Spark executes an action only after Cloudly accepts its running claim, requires Cloudly to accept the terminal result, reports non-owner service-update skips as failures, and disables further actions if terminalization cannot be proven.

Heartbeat metric branches, host updates, self-updates, and action processes are cancelled and drained during shutdown. Metric commands are timeout-bounded and escalate from SIGTERM to SIGKILL; action process groups, including detached descendants, must terminate before lifecycle restart is allowed.

Scheduled tasks:

Task	Schedule	Action
`updateServices`	Daily at midnight in `cloudly` mode	Checks generic managed Docker services for image and published-port drift. Coreflow-node updates are initial-startup and Cloudly-action driven with complete authority-bound shape proof.
`updateSpark`	Every 10 minutes	Checks for a newer Spark release and reloads the daemon after upgrade.
`updateHost`	Daily at midnight when enabled	Runs apt update/upgrade/autoremove/autoclean.

The managed service list is populated by the selected mode before daemon startup. Service updates use explicitly tagged mutable Docker image pulls, Docker secrets, and declared published port mappings.

In coreflow-node mode, Spark records the durable Coreflow owner in Swarm Spec labels through version-index CAS. The matching node label is only the scheduling projection. Coreflow is configured for one replica on that manager with no published ports, the Docker socket and /var/lib/serve.zone/coreflow mounted writable, and a root-owned secret.json mounted with mode 0400. Its szncorechat network must be a non-ingress, non-internal, non-attachable, IPv4-only Swarm overlay with the exact coreflow alias. Coreflow may add at most one alias-free sznwebgateway attachment and one read-only /sys/fs/cgroup:/host/cgroup mount. New services are created at zero replicas, proven against the complete managed shape, and only then scaled to one desired replica. The state directory and spark-owner.json marker are root-owned with modes 0700 and 0600. The retained host lock at /var/lib/serve.zone/.spark-coreflow-owner.lock is also root-owned with mode 0600. Spark does not migrate ownership automatically when the recorded node disappears or loses manager status. Within managed reconciliation, only the current owner may create, stop, remove, or replace the Coreflow service and secret. Each owner Spark process uses a new opaque Coreflow config revision, so an owner daemon restart rotates the Coreflow secret and replaces the service once. A wrong same-name szncorechat network fails closed instead of being replaced. The explicitly destructive spark prune operator command is outside this ownership protocol.

Cloudly heartbeat requires a Spark node token. spark claimjump is the Spark-side claim path for Cloudly Jump setup: it claims the Jump code over HTTP, validates the response, persists cloudlyUrl, the Coreflow jumpcode, nodeId, and nodeToken, and installs the daemon. Existing installations without those values continue to run service maintenance without reporting node telemetry.

Programmatic Usage

Spark exports Spark and SparkSwarmObservationManager from mod.ts:

import { Spark } from './mod.ts';

const spark = new Spark();
await spark.daemonStart();

// Stop timers, action processes, tasks, reconciliation, and heartbeat reporting.
await spark.daemonStop();

The public instance exposes:

Property	Purpose
`smartdaemon`	systemd service integration.
`sparkConfig`	persisted mode/config key-value store.
`sparkTaskManager`	taskbuffer scheduler and built-in maintenance tasks.
`sparkUpdateManager`	Docker Swarm activation and managed service update logic.
`sparkMetricsCollector`	Host metrics collection for local output and Cloudly heartbeat payloads.
`sparkHeartbeatManager`	Cloudly heartbeat and authenticated Swarm observation client.
`sparkMetricsStreamer`	Cloudly metrics stream and guarded node-action executor.
`sparkSelfUpdater`	Bounded binary and checksum download/update lifecycle.
`sparkSwarmObservationManager`	Callable, serialized read-only Swarm observation producer.
`coreflowConfigRevision`	Per-process opaque revision used for owner-managed Coreflow secret rotation.
`sparkInfo`	package metadata lookup.

Collect one local observation without using the heartbeat transport session:

const observation = await spark.sparkSwarmObservationManager.collectObservation();

Direct SparkSwarmObservationManager construction requires an IDockerSwarmReadClient; the optional reporter session, initial sequence, and clock inputs support lifecycle ownership and deterministic tests. Collection accepts an optional AbortSignal, bounds Docker reads, and keeps timestamps strictly increasing within its reporter session.

Development

Run from source:

deno task dev

Quality and build tasks:

deno task check
deno task lint
deno task fmt
deno task test
deno task compile

The package scripts intentionally defer to Deno tasks; pnpm build only reports that no Node build is needed.

The source checkout pins pnpm and disables install lifecycle scripts through pnpm-workspace.yaml. Its npm wrapper postinstall downloads an already published Spark binary, so running it during source release preparation would depend on the release being built. This source-only setting is not included in the npm wrapper. Frozen Deno commands install the runtime imports, including TsPack, from deno.json and deno.lock.

Release tags use GitZone's managed sealed-release workflow. The pinned Deno compiler is installed with deno upgrade --force and its effective version is checked before building. scripts/build-node.ts --release creates both Linux node archives and the five standalone CLI assets in one verified TsPack set. The workflow retains that complete set before publication, downloads it into a fresh directory, and verifies --release --reuse without recompilation. Retry the original Gitea run to resume publication of the same bytes. Regenerate the managed workflow with gitzone format --only assets --write --yes after configuration changes.

On Linux, deno task test also compiles and runs an isolated NoSQLDB identity-engine qualification. It uses a verified published native executable and disposable SmartData records to check exact engine selection, exclusive ownership and transaction recovery after engine termination. It also recovers the new private Spark identity store's pending and acknowledged credentials after separate engine terminations. This is a migration prerequisite; the Spark daemon still uses its existing configuration store. A successful local test does not qualify production cutover, host power-loss recovery or another CPU architecture.

The backend-private ts/identity/ module durably prepares Spark-only credentials, binds an exact outbound request digest, and activates only an exactly matching acknowledgement. Immutable receipts support replay without restoring superseded credentials. The module is not wired into the daemon or exported as an enrollment API: daemon startup integration, configuration activation, sidecar installation/self-update and the atomic Cloudly enrollment exchange are still required. Pallet owns a separate store and credential.

Private enrollment helpers now consume the published Interfaces 28.1 contract. prepareAdoption requires imported Spark authority and authenticated Cloudly state; it records the source fingerprint, original proof hash and exact generation/session fence without activating either credential. The caller must authenticate that state over HTTPS using the exact imported bearer. Generic prepare/bind/acknowledge calls cannot bypass this adoption path. Shared binding checks Spark's own prepared hash against the complete Spark/Pallet proposal, and activation requires an exact shared acknowledgement. The immutable receipt retains adoption provenance.

readEnrollmentRecovery is a private coordination capability, not a status or IPC API. It returns only Spark's bearer for the exact bound pending proposal or its still-current acknowledged identity, allowing recovery across partial activation. Changed proposals, unbound material and superseded credentials are rejected. It never returns Pallet's bearer or the original imported proof.

The private ts/enrollment/ coordinator now persists an immutable operation intent before external I/O, followed by the complete prepared proposal, both-owner binding, authenticated Cloudly acknowledgement and confirmed local completion. Restart resumes that same intent and credentials. A failed initial exchange permits one protocol-defined pending-Spark recovery request; an unconfirmed result stays bound and never creates replacement credentials. Saved acknowledgements bypass network enrollment and resume the independent Pallet-then-Spark activation.

Its owner ports are private integration boundaries. Status contains only stage, enrollment ID, Cloudly origin and node ID. Completed enrollment is historical installation evidence, not proof of current remote authority or workload health. Source tests use a real native Spark store and test-only external-owner stubs; the compiled fixture also recovers bound, acknowledged and complete coordination state after native engine termination. Daemon integration remains required before production cutover.

SparkEnrollmentHttpTransport implements the Cloudly port using a fixed canonical HTTPS origin and TypedRequest 8.0.3. Both RPCs use exact published contracts; no redirect, cache, implicit retry or stream transport is enabled. One request deadline covers response consumption, request/response bodies are capped at 32/16 KiB, and cancellation drains the actual fetch/body owner before returning. Credential RPC hooks are disabled and errors omit remote text, URLs, nested causes and arbitrary abort reasons. Shared response validation still proves exact node/proposal binding. Native TLS tests use a disposable CA and server certificate; production uses normal certificate verification. The adapter is not yet connected to daemon startup, and Cloudly's new enrollment handlers are still not exposed.

SparkEnrollmentPalletTransport implements the local owner port against published Interfaces 28.2.0 requests.pallet methods. It accepts trusted installer metadata, defaults to UID 0 and /run/serve.zone/pallet/control.sock, and requires a protected mode-0700 parent and mode-0600 socket. It rejects symlinks, unsafe ancestors and changed socket/parent inodes before sending enrollment data. No database access, plaintext Pallet bearer, TCP fallback or permissions repair is available.

Each TypedRequest uses one bounded length-prefixed JSON frame followed by request write-half-close, with 32/16 KiB request/response caps. The client requires the full reply and EOF, strict UTF-8, matching envelope identity and shared proposal/phase/ assigned-node bindings. Hooks, implicit retries, caches and stream transports are disabled. One default 10-second deadline aborts socket I/O; the actual connection closure and pending path checks remain joined before return. Once a peer has half-closed its request, the server still needs its own deadline to bound a lost client while no response is being written. Endpoint protection assumes trusted root/installer ownership, not protection against a hostile root process.

Real Unix tests cover all three methods, Jump/adoption bindings, malformed frames, endpoint ownership/replacement, pre-await input capture, cancellation, deadlines and caller-directed lost-response replay. Both adapters remain private and are not yet connected to the running daemon or production bootstrap.

SparkIdentityRuntime owns the private native database lifecycle using NoSQLDB 10.2.0 fileStorageStartup: 'current-format-only'. It verifies the selected engine digest and protected paths; NoSQLDB admits current layouts read-only and acquires its native file-root lease before storage writes. No Spark advisory lock or local format detector remains. Each start uses a separate mode-0700 disposable IPC directory so a rejected competitor cannot remove the active owner's socket.

Shutdown drains accepted callbacks, closes SmartData, then stops the engine. A failed database close retains the native lease for cleanup retry. Startup and shutdown deadlines bound caller waits without abandoning actual work or allowing restart before cleanup settles. Ordinary startup refuses missing storage instead of creating an empty replacement; explicit first installation alone provisions it. Strict legacy/mixed/staging rejection is tested without storage mutation. The compiled test uses this owner directly and verifies the exact 10.2.0 static musl engine provenance; production bootstrap wiring remains separate.

test/native/qualify-identity-upgrade.ts qualifies a disposable NoSQLDB 9.0.0 store through 10.2.0 startup, writes and a second restart. Supply the two published static musl engine paths in that order; the fixture checks their independently pinned SHA-256 values. It preserves active identity, an unfinished rotation and settings. NoSQLDB 10 writes require version 10 or later; downgrade requires a pre-upgrade backup.

SparkPalletProcess owns one verified foreground Pallet control process through Deno's native process API. Trusted bundle metadata supplies its protected absolute path and SHA-256. A bounded file verifier permits the larger compiled control while preserving the separate 256 MiB NoSQLDB engine limit. Its default enrollment mode selects enrollment-serve or explicitly requested enrollment-provision. Trusted mode: 'runtime' selects runtime-serve with pallet.node.process.v1 readiness and rejects provisioning; the two modes use separate child lifetimes.

The owner admits its scoped callback after Pallet's versioned readiness event. Parent cancellation, invalid process output or child exit closes admission and cancels the callback's signal; the callback must drain its admitted work before returning. Stdin carries only the parent lifetime. Shutdown closes it for graceful EOF, then escalates to SIGTERM/SIGKILL after bounded grace periods if needed. The owner waits for actual OS exit and output drainage; unconfirmed exit retains the lifetime instead of permitting a restart. Forced termination is a failure, and the owner never starts a replacement child. Runtime shutdown allows 330 seconds by default for the native operation window and cleanup, versus 15 seconds for enrollment. Runtime output is drained with bounded chunks and 4 KiB stdout lines; enrollment additionally retains its 1 MiB cumulative output limit. Static errors prevent child diagnostics from becoming public error text.

deno test --allow-all test/palletprocess.test.ts

The focused tests cover protected artifact validation, readiness, provisioning selection, cancellation, callback failure, malformed output, startup timeout, forced cleanup and runtime log drainage beyond the enrollment lifetime limit. The node recipe pins the published Pallet 0.5.0 release and verifies control schema 3, its seven control/native roles, the fixed containerd CRI socket, and each file's size, hash and source provenance before compiling Spark. The whole upstream native legal tree is included under pallet/; its third-party-notices.md and notices/inventory.json index the materials. SmartNFTables 1.9.0's guard binary and build provenance are independently pinned for each architecture. Packaging and notice generation both verify its complete native notice inventory under pallet/notices/smartnftables/. Installation requires the runtime and guard executables, provenance and notice manifest and rejects undeclared executables. Runtime supervision is not yet wired into the external service activation flow or self-updating. runnode composes it with the ordinary local lifetime described below; the existing asdaemon command still uses the legacy service managers.

Foreground node runtime

Run the selected compiled executable with runnode and no additional arguments. The command verifies the installed whole bundle and requires its selected Spark path to match the running executable. A bare download, stale selected version, or deno run mod.ts runnode cannot start node ownership.

/opt/spark/current/spark runnode

SparkNodeRuntime reopens existing Spark identity and settings and requires a complete persisted enrollment. It reconfirms both current local identities through the enrollment child, joins that child, then starts the Pallet runtime. This local reentry does not need Cloudly connectivity. Missing state or settings fail; ordinary startup never provisions replacement data, imports legacy configuration or resumes an incomplete remote enrollment. Only disposable IPC parents are recreated.

The command emits SPARK_NODE records with protocol spark.node.process.v1. The ready event reports the selected bundle, secret-free enrollment summary, settings snapshot, localRuntimeReady: true and workloadReadiness: "unverified". SIGINT or SIGTERM closes admission, joins reporting and any host update, and waits for Pallet and Spark storage cleanup before emitting stopped. Failed or forced cleanup remains an error even when a stop signal initiated shutdown. Loss of either owner cancels admitted work; no replacement process starts within the same lifetime.

After local readiness, the runtime collects CPU, memory and root-filesystem observations directly from the host and sends the Interfaces 30.5 heartbeat to POST /spark/v2/nodes/heartbeat at the exact enrolled HTTPS origin. Every attempt reads the current private Spark credential and requires a receipt for that node and credential generation. Requests and responses are limited to 8192 and 2048 bytes; the ten-second deadline includes reading the response. Redirects, cookies, response caching and immediate retries are disabled. Completed attempts wait the persisted heartbeatIntervalMs before the next sample. Cloudly unavailability does not block local readiness; a receipt records accepted host observations and does not establish workload readiness or deliver operator actions. Diagnostic state changes go to stderr and contain no credentials or remote response text.

When the persisted hostUpdatesEnabled setting is true, a separate joined owner runs /usr/bin/apt-get update, upgrade -y, autoremove -y and autoclean -y at the next host-local midnight, in that order and without overlapping runs. There is no immediate startup run or replay of missed offline schedules. A failed command stops that day's sequence. Cancellation terminates and joins the process group, allowing up to five minutes before forced termination; an unproven cleanup remains a runtime shutdown error. This owner does not replace the selected Spark bundle. Settings are read once per runtime lifetime, so changed settings take effect after a controlled drain and restart.

runnode now wraps the runtime in SparkNodeService. It holds a private Unix HTTP control socket at /run/serve.zone/spark-node/control.sock, with a root-owned mode-0700 parent, mode-0600 socket and retained empty IPC lock anchor. This process lock is acquired before starting data owners and remains held after they drain. A competing service cannot replace the listener or start another runtime. Stale socket retirement requires the lock, a refused connection and matching inode and ownership checks. Native HTTP owns final socket cleanup; privileged operators must not replace its pathname while the service is running.

SparkNodeControlClient reads bounded spark.node.control.v1 status containing the actual PID, random process-instance ID, phase and exact bundle version, source and manifest. A drain command must match the observed instance. It closes admission and reports draining until both data owners have joined; only then does it report drained. The process and control socket remain alive until SIGINT/SIGTERM, allowing an external installer to prove drainage before stopping the systemd unit. Repeated drains of that instance are idempotent. Losing the caller does not cancel an accepted drain, and failed cleanup never produces a successful drained result. No readiness, settings, credentials or other application data are written to files.

SparkNodeServiceActivation composes this client with the existing systemd activation policy inside SparkBundleInstallation.run. It matches the old PID and selected manifest before drainage, then verifies the new PID and exact selected bundle after start. Private HTTP requests have a five-second bound; startup observation allows 90 seconds and drainage 360 seconds. These are observation limits; a timed-out caller cannot undo or abandon server-side drainage. This path preserves masks and enablement and rejects the alternate spark.service unit.

Stable systemd boot entry

After installing and selecting a complete node bundle, configure its service with:

/opt/spark/current/spark installnodeservice

The command takes no additional arguments and requires root, a verified current bundle and a quiescent canonical service. It holds the same native installation lock used by bundle selection. SmartDaemon 2.6.0 installs /usr/local/lib/systemd/system/smartdaemon_spark.service with direct execution of /opt/spark/current/spark runnode, Type=exec, KillMode=mixed, a 360-second stop deadline and Restart=no. The atomic current selection is the stable boot path; the node unit explicitly requires and starts after the retained guard unit, so a failed guard also blocks node startup and its database owners. Spark remains the systemd main process and verifies its selected executable before opening its data owners. Graceful shutdown signals Spark first, allowing it to join Pallet and storage before the whole-cgroup failure deadline.

Installation preserves existing masks and enablement, retains previous generated definitions and verifies the reloaded manager view. It leaves an absent service disabled. It never starts, stops, enables or unmasks a unit, initializes data, changes settings or constructs the legacy daemon. An active owner, alternate spark.service, administrator shadowing definition or effective drop-in is rejected. A failed reload leaves the new definition for explicit inspection; selection and definitions are never automatically rolled back.

Successful installation emits a SPARK_NODE_SERVICE JSON record with the exact selected bundle, boot path, definition digest and changed/masked flags. Service start and enablement remain explicit operations after enrollment and rollout admission. External bundle activation supports both control-group and mixed, requires the kill policy to stay unchanged during activation, and still proves private drainage and exact new-PID readiness independently of systemd state.

Production conversion from the legacy service remains gated on rollout admission. Local readiness does not establish a Cloudly session, containerd workloads, private DNS or production takeover.

Retained allocation-pool guard boot entry

After enrollment, while management networking is available, the privileged installer runs /opt/spark/current/spark activateguard. The command accepts no path, provider, policy or credential overrides. Its supported host profile uses the verified systemd-networkd service; an alternate network manager, pending provider reload, administrator drop-in or active Spark runtime prevents activation. An already installed node unit must also require and order itself after the guard; rerun installnodeservice while it is stopped to update an older definition. Qualify the host's actual interface ownership before using this profile.

One bundle installation lock spans the complete operation. Before a guard unit exists, Spark runs the verified Pallet control's network-acquire mode. That projection-only connection retains fresh authenticated authority without starting native owners, accepting assignments or delivering runtime reports. Spark requires the exact pallet.network.acquisition.v1 ready/stopped sequence, drained pipes and successful child exit before explicitly commissioning Pallet's durable guard lane. A failed or cancelled acquisition leaves commissioning and unit installation unstarted. It then installs smartdaemon_spark-guard.service as a disabled Type=oneshot unit with RemainAfterExit=yes, manually starts it, and persistently enables it. The exact definition orders recovery after local filesystems and before network-pre, networkd and shutdown; both network-pre and networkd require it. Success requires the exact file digest and fragment, no drop-ins or pending reload, completed oneshot state, and both forward and reverse dependency edges. Enabled state alone does not establish complete boot protection.

The unit executes /opt/spark/current/spark guard-recover. That launcher verifies the selected whole bundle and its own executable, then runs its authenticated Pallet control with null stdin. It uses read-only bundle inspection so it can run while the activating parent holds the installation lock. The bounded child owner requires pallet.guard.process.v1 ready/stopped events, drained pipes and successful actual exit. Cancellation or forced cleanup fails the boot prerequisite.

An existing exact unit admits offline recovery only and never repeats projection acquisition. Interrupted reload, start or partial enablement is resumed from retained state; missing guard data is never silently recommissioned. Masks and foreign definitions reject without replacement. Errors retain the guard policy and any installed dependencies. Completion emits one SPARK_GUARD record identifying the bundle and definition. This deny-only bootstrap does not publish a positive network barrier, release allocations or establish workload/DNS readiness.

test/helpers/palletenrollment.fixture.ts is an explicit cross-artifact qualification executable. Supply absolute paths for the Node executable, built Pallet control module, exact NoSQLDB engine and its independently verified SHA-256, and the built Cloudly enrollment store module:

deno run --allow-all --frozen test/helpers/palletenrollment.fixture.ts \
  /path/to/node /path/to/pallet/dist_ts/control/classes.enrollmentcontrol.js \
  /path/to/rustdb_linux_amd64_musl VERIFIED_ENGINE_SHA256 \
  /path/to/cloudly/dist_ts/manager.node/classes.nodeenrollmentstore.js

It runs real Pallet and Cloudly owners in separate Node processes and the real Spark coordinator with independent stores. Cloudly's actual enrollment/credential stores and installed TypedRequest router run behind a disposable loopback HTTPS listener; Spark uses its actual HTTPS client with an isolated test CA, not disabled TLS checks. Fresh Jump and imported-node adoption both recover a committed-but-lost TLS response and a lost local activation result. All three owners and their exact NoSQLDB engines then restart from separate file stores and converge on the same node/credentials. Assertions verify one enrollment receipt, two distinct credential authorities, the expected generations, rejection of the old Spark credential, settings continuity, and unchanged synthetic legacy bytes. The Cloudly authority stub has been removed.

This qualifies the private store/transport integration, not the production CloudlyServer route/admission lifecycle, a distributable node bundle, daemon wiring or deployed workload readiness. The isolated fixture registers no production route and introduces no local package link or application dependency on sibling sources.

The private runtime context now includes SparkSettingsStore. It stores only typed hostUpdatesEnabled and heartbeatIntervalMs settings; connection origin, node identity and credentials remain owned by enrollment and the current identity. Ordinary reads reject missing settings. Explicit fresh installation initializes the existing false/60000 defaults, and revision-checked full updates reject stale writers and counter exhaustion. Responses are detached/frozen and contain no credential or migration-source metadata. An ambiguous write must be reconciled by reading the saved revision; it is not automatically repeated.

Versioned v2 startup migration follows v1 import, copies operational settings once and retains the exact import source digest as completion provenance. Later startup/import replay preserves user edits. Explicit import preflights any fresh-initialized settings before v1 may persist authority, and snapshots installer options before those checks. Without an import, startup leaves settings absent until explicit provisioning. Tests cover concurrency, ambiguity, restart and native crash recovery. These are the new owner's APIs, not a KV compatibility adapter: the existing daemon/CLI still uses npmextra until coordinated bootstrap replacement connects the new lifecycle and removes those old consumers.

The versioned ts_migration/ runner now runs inside that private owner's startup, after protected database readiness and before identity readiness. An explicit installer-only legacyImport option admits the owning user's npmextra JSON file only after the installer has stopped the old writer. The path must be canonical, owned by the expected UID, regular, singly linked, bounded to 1 MiB, and free of symlinks and unsafe writable ancestors. Import never invokes npmextra or repairs permissions. It hashes the original bytes, validates all known settings and stores the configuration, exact original Spark bearer, Coreflow Jump value and completion stage together in one exact SmartData document. Unknown fields, partial credentials and invalid present settings fail closed. The historical file is not modified.

Explicit replay requires the same path and exact bytes. Ordinary startup omits legacyImport and uses protected state without reopening the historical file. Missing optional settings retain the previous daemon defaults; a credential-free source never becomes an enrolled identity. Imported authority is separate from the new private identity store: no generation or acknowledgement is fabricated. The daemon/CLI configuration consumers, authenticated adoption exchange and source retirement are still unwired. Source retirement must revalidate the original file and occur only after the coordinated Spark and Pallet activation is durable.

deno test --allow-all test/bootstrapimport.test.ts

Source map:

Path	Purpose
`mod.ts`	CLI entry point and library export.
`ts/spark.cli.ts`	Command routing.
`ts/spark.classes.spark.ts`	Main class wiring.
`ts/spark.classes.updatemanager.ts`	Docker Swarm and service update manager.
`ts/spark.classes.coreflowowner.ts`	Durable Coreflow ownership, marker, lock, and node projection.
`ts/spark.classes.swarmobservationmanager.ts`	Swarm observation v2 producer.
`ts/spark.classes.taskmanager.ts`	Scheduled maintenance tasks.
`ts/spark.classes.config.ts`	Persisted mode/config store.
`ts/identity/`	Private SmartData credential state machine; not yet used by the daemon.
`ts/enrollment/`	Private durable Spark/Pallet enrollment coordinator and typed owner ports.
`scripts/compile-all.sh`	Multi-platform Deno compilation.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

@serve.zone/spark