@serve.zone/spark

Spark is a Deno-powered server management agent for serve.zone hosts. It installs as a system daemon, activates Docker Swarm, schedules host/service maintenance tasks, and provides the bootstrap profiles currently used by Cloudly and Coreflow node deployments.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Current Role

Spark is intentionally small and operational. It is not a general-purpose configuration management framework; it is the serve.zone node-side utility that knows how to run itself as a daemon and keep selected Docker services moving.

The current implementation:

• Installs and updates a smartdaemon_spark systemd service through @push.rocks/smartdaemon.
• Runs in an explicit mode: cloudly or coreflow-node.
• Activates Docker Swarm through @apiclient.xyz/docker when daemon mode starts.
• Collects host metrics for CPU, memory, disk, and Docker container counts.
• Sends Cloudly heartbeats when cloudlyUrl, nodeId, and nodeToken are configured.
• Sends authenticated Swarm observation v2 reports after accepted Cloudly heartbeats.
• Schedules recurring tasks with @push.rocks/taskbuffer for Spark updates and managed Docker service updates.
• Optionally schedules host package updates when --enableHostUpdates=true has been configured.

Spark also owns a callable Swarm observation v2 producer. It reads local Docker membership and, on active managers, attaches a digest-verified node snapshot only when the complete manager view validates. Worker observations never invent the manager-only Swarm cluster ID. Heartbeat transport owns a separate private reporter session, validates bound Cloudly receipts, retries only the exact pending request after ambiguous or retryable delivery, and quarantines terminally rejected sessions until process restart.

Installation

Install a released binary:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash

Pin to the newest release within a supported major channel:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | sudo bash -s -- --major 1

Verify a selected release asset without changing the host:

curl -fsSL https://code.foss.global/serve.zone/spark/releases/download/<version>/install.sh | bash -s -- --version <version> --verify-only

The installer downloads SHA256SUMS.txt and verifies the exact platform binary before stopping services or replacing an existing installation. Release builds currently target Linux x64/ARM64, macOS x64/ARM64, and Windows x64.

Requirements

• Linux with systemd for daemon operation.
• Docker for service and Swarm management.
• Root privileges for daemon installation, Docker maintenance, package updates, and prune.
• Deno only when running from source.

macOS and Windows binaries are built for CLI/library availability, but the operational daemon paths are Linux/systemd oriented.

Quick Start

Install Spark as a daemon:

sudo spark installdaemon

Run daemon mode with a profile:

sudo spark asdaemon --mode cloudly

or:

sudo spark asdaemon --mode coreflow-node

The selected mode is persisted in a user-home npmextra key/value store under the servezone_spark identity. Later spark asdaemon calls can reuse the stored mode when no --mode flag is provided.

Follow daemon logs:

sudo spark logs

CLI Reference

spark <command> [options]

Command	Purpose
installdaemon	Create, enable, and start the Spark daemon service.
claimjump --claimUrl <url> --jumpCode <code>	Claim a Cloudly Jump code, persist the returned node/Coreflow credentials, and install/start the daemon. Also accepts CLAIM_URL and JUMP_CODE.
updatedaemon	Reload the daemon service definition for the current Spark version.
upgrade	Install a newer Spark release when available and restart the daemon.
asdaemon --mode cloudly	Run the daemon loop with the Cloudly profile.
asdaemon --mode coreflow-node	Run the daemon loop with the Coreflow node profile.
metrics	Print a JSON host metrics snapshot.
logs	Follow journalctl -u smartdaemon_spark -f.
prune	Stop Spark, remove Docker stacks/services/secrets, remove selected networks, prune Docker, restart Docker, and restart Spark.

prune is destructive. Use it only on nodes where Spark owns the Docker runtime state or where losing all stacks, services, and secrets is intended.

Daemon Behavior

Spark.daemonStart() starts the node reporting and reconciliation subsystems before enabling recurring tasks. A failed initial managed-service reconciliation stops already-started subsystems in reverse order instead of leaving a partial daemon running. The asdaemon command handles SIGINT and SIGTERM by disabling actions, terminating active action process groups, stopping tasks and service reconciliation, releasing the Coreflow session, and stopping heartbeat reporting before exit. Shutdown aborts active mutable image pulls and retains the Coreflow owner session until reconciliation settles. A pull that ignores cancellation retains the owner session until that reconciliation eventually settles. Separately, subprocess cleanup that cannot prove exit retains exact process tracking and quarantines the affected lifecycle restart while the daemon continues its ordered shutdown.

• SparkServicesManager.start() activates Docker Swarm and, for Coreflow nodes, acquires the host-wide reconciliation lock.
• SparkTaskManager.start() schedules recurring maintenance tasks.
• SparkHeartbeatManager.start() sends node metrics to Cloudly when node credentials are configured.
• SparkMetricsStreamer.start() streams lightweight metrics and, after initial service reconciliation completes, executes Cloudly systemUpgrade, reboot, and serveZoneServiceUpdate node actions when node credentials are configured. Spark executes an action only after Cloudly accepts its running claim, requires Cloudly to accept the terminal result, reports non-owner service-update skips as failures, and disables further actions if terminalization cannot be proven.

Heartbeat metric branches, host updates, self-updates, and action processes are cancelled and drained during shutdown. Metric commands are timeout-bounded and escalate from SIGTERM to SIGKILL; action process groups, including detached descendants, must terminate before lifecycle restart is allowed.

Scheduled tasks:

Task	Schedule	Action
updateServices	Daily at midnight in cloudly mode	Checks generic managed Docker services for image and published-port drift. Coreflow-node updates are initial-startup and Cloudly-action driven with complete authority-bound shape proof.
updateSpark	Every 10 minutes	Checks for a newer Spark release and reloads the daemon after upgrade.
updateHost	Daily at midnight when enabled	Runs apt update/upgrade/autoremove/autoclean.

The managed service list is populated by the selected mode before daemon startup. Service updates use explicitly tagged mutable Docker image pulls, Docker secrets, and declared published port mappings.

In coreflow-node mode, Spark records the durable Coreflow owner in Swarm Spec labels through version-index CAS. The matching node label is only the scheduling projection. Coreflow is configured for one replica on that manager with no published ports, the Docker socket and /var/lib/serve.zone/coreflow mounted writable, and a root-owned secret.json mounted with mode 0400. Its szncorechat network must be a non-ingress, non-internal, non-attachable, IPv4-only Swarm overlay with the exact coreflow alias. Coreflow may add at most one alias-free sznwebgateway attachment and one read-only /sys/fs/cgroup:/host/cgroup mount. New services are created at zero replicas, proven against the complete managed shape, and only then scaled to one desired replica. The state directory and spark-owner.json marker are root-owned with modes 0700 and 0600. The retained host lock at /var/lib/serve.zone/.spark-coreflow-owner.lock is also root-owned with mode 0600. Spark does not migrate ownership automatically when the recorded node disappears or loses manager status. Within managed reconciliation, only the current owner may create, stop, remove, or replace the Coreflow service and secret. Each owner Spark process uses a new opaque Coreflow config revision, so an owner daemon restart rotates the Coreflow secret and replaces the service once. A wrong same-name szncorechat network fails closed instead of being replaced. The explicitly destructive spark prune operator command is outside this ownership protocol.

Cloudly heartbeat requires a Spark node token. spark claimjump is the Spark-side claim path for Cloudly Jump setup: it claims the Jump code over HTTP, validates the response, persists cloudlyUrl, the Coreflow jumpcode, nodeId, and nodeToken, and installs the daemon. Existing installations without those values continue to run service maintenance without reporting node telemetry.

Programmatic Usage

Spark exports Spark and SparkSwarmObservationManager from mod.ts:

import { Spark } from './mod.ts';

const spark = new Spark();
await spark.daemonStart();

// Stop timers, action processes, tasks, reconciliation, and heartbeat reporting.
await spark.daemonStop();

The public instance exposes:

Property	Purpose
smartdaemon	systemd service integration.
sparkConfig	persisted mode/config key-value store.
sparkTaskManager	taskbuffer scheduler and built-in maintenance tasks.
sparkUpdateManager	Docker Swarm activation and managed service update logic.
sparkMetricsCollector	Host metrics collection for local output and Cloudly heartbeat payloads.
sparkHeartbeatManager	Cloudly heartbeat and authenticated Swarm observation client.
sparkMetricsStreamer	Cloudly metrics stream and guarded node-action executor.
sparkSelfUpdater	Bounded binary and checksum download/update lifecycle.
sparkSwarmObservationManager	Callable, serialized read-only Swarm observation producer.
coreflowConfigRevision	Per-process opaque revision used for owner-managed Coreflow secret rotation.
sparkInfo	package metadata lookup.

Collect one local observation without using the heartbeat transport session:

const observation = await spark.sparkSwarmObservationManager.collectObservation();

Direct SparkSwarmObservationManager construction requires an IDockerSwarmReadClient; the optional reporter session, initial sequence, and clock inputs support lifecycle ownership and deterministic tests. Collection accepts an optional AbortSignal, bounds Docker reads, and keeps timestamps strictly increasing within its reporter session.

Development

Run from source:

deno task dev

Quality and build tasks:

deno task check
deno task lint
deno task fmt
deno task test
deno task compile

The package scripts intentionally defer to Deno tasks; pnpm build only reports that no Node build is needed.

Source map:

Path	Purpose
mod.ts	CLI entry point and library export.
ts/spark.cli.ts	Command routing.
ts/spark.classes.spark.ts	Main class wiring.
ts/spark.classes.updatemanager.ts	Docker Swarm and service update manager.
ts/spark.classes.coreflowowner.ts	Durable Coreflow ownership, marker, lock, and node projection.
ts/spark.classes.swarmobservationmanager.ts	Swarm observation v2 producer.
ts/spark.classes.taskmanager.ts	Scheduled maintenance tasks.
ts/spark.classes.config.ts	Persisted mode/config store.
scripts/compile-all.sh	Multi-platform Deno compilation.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.