Coreflow is the serve.zone Docker Swarm reconciler. It runs inside a cluster, connects outward to Cloudly, turns Cloudly's desired service state into Docker networks, secrets, services, Corestore bindings, backups, and Coretraffic routing updates.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Runtime Position

Coreflow is the cluster-side execution engine for Cloudly. Cloudly owns the desired state; Coreflow owns local reconciliation against Docker.

Cloudly control plane
  <- Coreflow WebSocket client registered as coreflow
Coreflow on Docker Swarm manager
  -> local Docker Engine / Swarm socket
  -> internal SmartServe + TypedSocket on port 3000
  <- Coretraffic connects as coretraffic
  -> Corestore control API for database, object storage, volumes, and backups

Coreflow does not require Cloudly to dial into the cluster. It connects to Cloudly using the cluster machine token, subscribes to pushClusterConfig events through CloudlyApiClient.configUpdateSubject, and also runs a scheduled reconciliation loop.

What It Reconciles

Coreflow currently handles:

Base overlay networks: sznwebgateway for public workload traffic and szncorechat for internal platform traffic.
Base services: coretraffic, corelog, and a global corestore service.
Workload services from Cloudly service records.
Docker secrets generated from Cloudly secret bundles plus provisioned platform binding environment variables.
Internal Cloudly images, external registry images, and service-specific Cloudly registry targets.
Corestore-backed Docker volumes using Docker's VolumeDriver API.
Platform bindings for database and objectstorage through Corestore.
Routing data for Coretraffic, including certificates from dcrouter when available or Cloudly ACME as fallback.
Service backup and restore requests for volumes, databases, and object storage resources.
Optional external dcrouter route synchronization when Cloudly provides external gateway config.

Configuration

Coreflow reads runtime configuration with @push.rocks/qenv from the process environment and .nogit overlays.

Required variables:

Variable	Purpose
`CLOUDLY_URL`	Cloudly TypedSocket/HTTP endpoint.
`JUMPCODE`	Cloudly machine token for the cluster identity.

Optional variables:

Variable	Purpose
`CORESTORE_API_TOKEN`	Forwarded into the global Corestore service and used for Corestore control API calls.
`CORESTORE_CONTROL_URL`	Override for the Corestore control URL; defaults to `http://corestore:3000`.

Example .nogit/.env:

CLOUDLY_URL=https://cloudly.example.com
JUMPCODE=cluster-machine-token
CORESTORE_API_TOKEN=shared-corestore-token

Starting Coreflow

Coreflow is normally deployed as a service/container on a Docker Swarm manager node with access to the local Docker socket.

pnpm install
pnpm build
pnpm start

For local TypeScript execution:

pnpm run startTs

Programmatic startup inside this repository:

import { Coreflow } from './ts/coreflow.classes.coreflow.js';

const coreflow = new Coreflow();
await coreflow.start();

process.on('SIGTERM', async () => {
  await coreflow.stop();
});

Startup order is deliberate:

InternalServer starts SmartServe and TypedSocket on port 3000.
CloudlyConnector connects to Cloudly and authenticates with JUMPCODE.
ClusterManager reads initial cluster config and subscribes to pushed config updates.
PlatformManager reconciles platform bindings and subscribes to platform config updates.
CoreflowBackupManager registers backup/restore typed handlers.
CoreflowTaskmanager schedules the initial and recurring reconciliation chain.

Reconciliation Chain

The task manager runs a buffered chain so repeated updates collapse into one effective run:

updateBaseServices
  -> updateWorkloadServices
      -> updateTrafficRouting

The initial run starts after a 10 second delay. The base-service task is then scheduled hourly with the cron expression 0 0 * * * *.

Base Services

provisionBaseServices() ensures the cluster has the networks and base services Coreflow expects:

Resource	Details
`sznwebgateway`	Overlay network for public workload routing.
`szncorechat`	Overlay network for internal base-service communication.
`coretraffic`	Connected to both networks; Docker service maps host `80:7999` and `443:8000`.
`corelog`	Connected to `szncorechat`.
`corestore`	Global service connected to both networks, with `/var/lib/serve.zone/corestore` and `/run/docker/plugins` bind mounts.

Coreflow pulls base images from code.foss.global/serve.zone/coretraffic, code.foss.global/serve.zone/corelog, and code.foss.global/serve.zone/corestore.

Workload Services

provisionWorkloadServices() reads Cloudly services and skips records whose serviceCategory is set and not workload.

For each workload service, Coreflow:

Resolves the container image from registryTarget, internal Cloudly image storage, or an external registry location.
Authenticates to registries when credentials are available.
Provisions Corestore-backed platform bindings for database and objectstorage.
Merges provisioned environment values with the configured secret bundle.
Creates a Docker secret containing the flattened secret JSON as secret.json.
Builds Docker volume mounts from service resources.volumeMounts and first-class volumes.
Recreates the Docker service when image, secret, or volume labels indicate desired state changed.

Docker workload services are created as replicated services on sznwebgateway with a default memory limit of 1000 MB unless the service resource data specifies otherwise.

Volumes

Service records can declare first-class volumes:

volumes: [
  {
    mountPath: '/data',
    driver: 'corestore',
    backup: true,
  },
]

If a volume name is omitted, Coreflow derives a stable Docker-safe name from the service id and mount path. The generated Docker mount uses DriverConfig.Name = 'corestore' by default and forwards service metadata as driver options and labels.

Routing

updateTrafficRouting() inspects Docker services on sznwebgateway, matches them to Cloudly workload service records, resolves the first task IP currently available on the network, and builds IReverseProxyConfig[] entries from service domains and ports.

For each routed hostname, Coreflow tries to export a certificate from an external dcrouter gateway first when Cloudly provided gateway config. If that fails or no gateway is configured, it requests a certificate from Cloudly. The final route set is sent to the Coretraffic connection tagged coretraffic with the updateRouting typed request.

Custom service ports are supported through service.data.ports.custom, where each key is treated as the hostname and the value is the destination port.

Platform and Corestore Integration

PlatformManager asks Cloudly for platform desired state and handles database and objectstorage bindings through Corestore.

Default Corestore control URL:

http://corestore:3000

Provisioned values such as MONGODB_URI, S3_BUCKET, AWS_ACCESS_KEY_ID, and AWS_ENDPOINT_URL are merged into the workload secret before Docker service creation. Coreflow also reports platform binding status back to Cloudly with endpoints and credential references.

Backup and Restore

CoreflowBackupManager registers two internal typed handlers:

executeServiceBackup
executeServiceRestore

Backups provision Corestore bindings if necessary, snapshot backup-enabled service volumes, snapshot service DB/S3 resources, and optionally replicate the local Corestore archive back through Cloudly. Restores can pull missing archive objects from Cloudly, create missing Corestore volumes, restore volume snapshots, and restore DB/S3 snapshots.

Development

Common commands:

pnpm install
pnpm build
pnpm test
pnpm run watch
pnpm run build:docker

Important paths:

Path	Purpose
`ts/index.ts`	CLI startup wrapper exporting `runCli` and `stop`.
`ts/coreflow.classes.coreflow.ts`	Main coordinator and component lifecycle.
`ts/coreflow.connector.cloudlyconnector.ts`	Cloudly API connection, machine-token authentication, and config fetches.
`ts/coreflow.classes.clustermanager.ts`	Docker networks, base services, workload services, secrets, images, volumes, and routing.
`ts/coreflow.classes.platformmanager.ts`	Corestore-backed platform binding reconciliation.
`ts/coreflow.classes.backupmanager.ts`	Backup and restore execution handlers.
`ts/coreflow.connector.coretrafficconnector.ts`	TypedSocket route updates to Coretraffic.
`ts/coreflow.connector.externalgateway.ts`	dcrouter certificate export and public route sync.
`ts/coreflow.classes.taskmanager.ts`	Buffered and scheduled reconciliation tasks.
`ts/coreflow.classes.internalserver.ts`	Internal SmartServe and TypedSocket server on port `3000`.

Operational Notes

Coreflow expects to run on a Docker Swarm manager with local Docker socket access.
Workload routing currently uses the first available task IP for a Docker service.
A missing image, missing secret bundle, invalid registry credentials, or incomplete service port/domain data should fail reconciliation for that service rather than silently deploying the wrong thing.
Corestore volume data is node-local; snapshot before destructive changes or restores.
The package is marked private and is normally shipped as part of the serve.zone platform deployment flow.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md

@serve.zone/coreflow