serve.zone/coreflow
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refresh readme and legal info

Browse Source
This commit is contained in:
Jürgen Kunz
2026-05-07 20:22:12 +00:00
parent 4c9db6d115
commit 23a56a61e5
2 changed files with 117 additions and 106 deletions
Download Patch File Download Diff File Expand all files Collapse all files
package.json
+3 -3
View File
@@ -48,12 +48,12 @@
"Observability",
"TypeScript"
],
"author": "Lossless GmbH",
"author": "Task Venture Capital GmbH",
"license": "MIT",
"bugs": {
"url": "https://gitlab.com/losslessone/services/servezone/coreflow/issues"
"url": "https://community.foss.global/"
},
"homepage": "https://gitlab.com/losslessone/services/servezone/coreflow#readme",
"homepage": "https://code.foss.global/serve.zone/coreflow#readme",
"devDependencies": {
"@git.zone/tsbuild": "^4.4.0",
"@git.zone/tsdocker": "^2.2.4",
readme.md
+114 -103
View File
@@ -1,80 +1,85 @@
# @serve.zone/coreflow
Coreflow is the Docker Swarm reconciliation engine for the serve.zone platform. It runs inside a cluster, connects back to Cloudly, reads the desired cluster state, provisions the base runtime services, deploys workload services, and pushes reverse-proxy routing updates to Coretraffic.
Coreflow is the serve.zone Docker Swarm reconciler. It runs inside a cluster, connects outward to Cloudly, turns Cloudly's desired service state into Docker networks, secrets, services, Corestore bindings, backups, and Coretraffic routing updates.
## Issue Reporting and Security
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
## What Coreflow Does
## Runtime Position
Coreflow sits between Cloudly and the Docker Swarm runtime:
- Connects to Cloudly over the `@serve.zone/api` WebSocket client and registers as `coreflow`.
- Authenticates with Cloudly using the cluster jump code token.
- Reads the cluster configuration and service definitions managed by Cloudly.
- Ensures base Docker networks exist for traffic and platform communication.
- Deploys and updates base services such as `coretraffic` and `corelog`.
- Deploys workload services from Cloudly image definitions.
- Creates Docker secrets from Cloudly secret bundles and attaches them to services.
- Builds reverse proxy configs from service domains, Docker task IPs, and Cloudly certificates.
- Sends routing updates to Coretraffic through the internal TypedSocket server.
- Reconciles state initially, on Cloudly config updates, and on a scheduled hourly task.
## Runtime Model
Coreflow is not a general-purpose application framework. It is a long-running cluster component designed to be started as a service or Docker container on a Docker Swarm manager node.
Coreflow is the cluster-side execution engine for Cloudly. Cloudly owns the desired state; Coreflow owns local reconciliation against Docker.
```text
Cloudly
-> Coreflow
-> local Docker Engine / Swarm
-> Coretraffic via internal TypedSocket
Cloudly control plane
<- Coreflow WebSocket client registered as coreflow
Coreflow on Docker Swarm manager
-> local Docker Engine / Swarm socket
-> internal SmartServe + TypedSocket on port 3000
<- Coretraffic connects as coretraffic
-> Corestore control API for database, object storage, volumes, and backups
```
Coreflow never waits for Cloudly to call it. It connects outward to Cloudly, keeps the connection tagged as a `coreflow` client, and reacts to config update events from that connection.
Coreflow does not require Cloudly to dial into the cluster. It connects to Cloudly using the cluster machine token, subscribes to `pushClusterConfig` events through `CloudlyApiClient.configUpdateSubject`, and also runs a scheduled reconciliation loop.
## Requirements
## What It Reconciles
- Node.js runtime compatible with the project toolchain.
- pnpm for dependency management.
- A Docker Swarm manager with access to the local Docker socket.
- A reachable Cloudly instance.
- A valid Cloudly jump code for the target cluster.
Coreflow currently handles:
- Base overlay networks: `sznwebgateway` for public workload traffic and `szncorechat` for internal platform traffic.
- Base services: `coretraffic`, `corelog`, and a global `corestore` service.
- Workload services from Cloudly service records.
- Docker secrets generated from Cloudly secret bundles plus provisioned platform binding environment variables.
- Internal Cloudly images, external registry images, and service-specific Cloudly registry targets.
- Corestore-backed Docker volumes using Docker's VolumeDriver API.
- Platform bindings for `database` and `objectstorage` through Corestore.
- Routing data for Coretraffic, including certificates from dcrouter when available or Cloudly ACME as fallback.
- Service backup and restore requests for volumes, databases, and object storage resources.
- Optional external dcrouter route synchronization when Cloudly provides external gateway config.
## Configuration
Coreflow reads runtime configuration through `@push.rocks/qenv` from the project environment and `.nogit` overlays.
Coreflow reads runtime configuration with `@push.rocks/qenv` from the process environment and `.nogit` overlays.
Required environment variables:
Required variables:
| Variable | Purpose |
| --- | --- |
| `CLOUDLY_URL` | WebSocket/HTTP endpoint of the Cloudly control plane. |
| `JUMPCODE` | Cloudly token used to authenticate this Coreflow instance and tag the connection. |
| `CLOUDLY_URL` | Cloudly TypedSocket/HTTP endpoint. |
| `JUMPCODE` | Cloudly machine token for the cluster identity. |
Optional variables:
| Variable | Purpose |
| --- | --- |
| `CORESTORE_API_TOKEN` | Forwarded into the global Corestore service and used for Corestore control API calls. |
| `CORESTORE_CONTROL_URL` | Override for the Corestore control URL; defaults to `http://corestore:3000`. |
Example `.nogit/.env`:
```env
CLOUDLY_URL=https://cloudly.example.com
JUMPCODE=cluster-machine-token
CORESTORE_API_TOKEN=shared-corestore-token
```
## Installation
## Starting Coreflow
This package is private and normally deployed as part of the serve.zone platform image pipeline.
Coreflow is normally deployed as a service/container on a Docker Swarm manager node with access to the local Docker socket.
```sh
pnpm install
pnpm run build
pnpm build
pnpm start
```
The package also exposes the `coreflow` binary after build through `dist/cli.js`, while the repository entrypoint is `cli.js`.
For local TypeScript execution:
## Programmatic Startup
```sh
pnpm run startTs
```
The CLI path imports `runCli()` from `dist_ts/index.js`. For direct TypeScript usage inside this repository, instantiate the main class and call `start()`:
Programmatic startup inside this repository:
```ts
import { Coreflow } from './ts/coreflow.classes.coreflow.js';
@@ -87,17 +92,18 @@ process.on('SIGTERM', async () => {
});
```
`start()` initializes components in this order:
Startup order is deliberate:
1. `InternalServer` starts a SmartServe server on port `3000` with TypedSocket support.
2. `CloudlyConnector` connects to Cloudly and resolves the cluster identity.
3. `ClusterManager` reads initial Cloudly config and subscribes to config updates.
4. `PlatformManager` starts its placeholder lifecycle hook.
5. `CoreflowTaskmanager` schedules the initial and recurring reconciliation tasks.
1. `InternalServer` starts SmartServe and TypedSocket on port `3000`.
2. `CloudlyConnector` connects to Cloudly and authenticates with `JUMPCODE`.
3. `ClusterManager` reads initial cluster config and subscribes to pushed config updates.
4. `PlatformManager` reconciles platform bindings and subscribes to platform config updates.
5. `CoreflowBackupManager` registers backup/restore typed handlers.
6. `CoreflowTaskmanager` schedules the initial and recurring reconciliation chain.
## Reconciliation Flow
## Reconciliation Chain
The task manager coordinates the runtime work as a task chain:
The task manager runs a buffered chain so repeated updates collapse into one effective run:
```text
updateBaseServices
@@ -105,45 +111,41 @@ updateBaseServices
-> updateTrafficRouting
```
`updateBaseServices` ensures the base Docker networks and platform services exist:
The initial run starts after a 10 second delay. The base-service task is then scheduled hourly with the cron expression `0 0 * * * *`.
- `sznwebgateway` for public web routing.
- `szncorechat` for internal base-service communication.
- `coretraffic` attached to both networks with host ports `80` and `443` mapped to its service ports.
- `corelog` attached to the internal network.
### Base Services
`updateWorkloadServices` fetches Cloudly services, skips non-workload service categories, pulls or imports the configured Docker image, creates a Docker secret from the assigned secret bundle, and creates or replaces the Docker service when an update is required.
`provisionBaseServices()` ensures the cluster has the networks and base services Coreflow expects:
`updateTrafficRouting` inspects Docker services on the web gateway network, resolves container IPs, fetches certificates for configured domains, and sends `IReverseProxyConfig[]` updates to Coretraffic with the `updateRouting` typed request.
| Resource | Details |
| --- | --- |
| `sznwebgateway` | Overlay network for public workload routing. |
| `szncorechat` | Overlay network for internal base-service communication. |
| `coretraffic` | Connected to both networks; Docker service maps host `80:7999` and `443:8000`. |
| `corelog` | Connected to `szncorechat`. |
| `corestore` | Global service connected to both networks, with `/var/lib/serve.zone/corestore` and `/run/docker/plugins` bind mounts. |
The same base-service task is triggered when Cloudly emits a config update. After the initial delayed run, it is also scheduled hourly.
Coreflow pulls base images from `code.foss.global/serve.zone/coretraffic`, `code.foss.global/serve.zone/corelog`, and `code.foss.global/serve.zone/corestore`.
## Cloudly Integration
### Workload Services
`CloudlyConnector` wraps `CloudlyApiClient` from `@serve.zone/api`:
`provisionWorkloadServices()` reads Cloudly services and skips records whose `serviceCategory` is set and not `workload`.
```ts
this.cloudlyApiClient = new CloudlyApiClient({
registerAs: 'coreflow',
cloudlyUrl,
});
```
For each workload service, Coreflow:
After connection, Coreflow authenticates with `JUMPCODE` and requests a stateful, tagged identity. That identity is then used to fetch cluster configuration and certificates.
- Resolves the container image from `registryTarget`, internal Cloudly image storage, or an external registry location.
- Authenticates to registries when credentials are available.
- Provisions Corestore-backed platform bindings for `database` and `objectstorage`.
- Merges provisioned environment values with the configured secret bundle.
- Creates a Docker secret containing the flattened secret JSON as `secret.json`.
- Builds Docker volume mounts from service `resources.volumeMounts` and first-class `volumes`.
- Recreates the Docker service when image, secret, or volume labels indicate desired state changed.
Coreflow depends on these Cloudly-side resources being present and valid:
Docker workload services are created as replicated services on `sznwebgateway` with a default memory limit of 1000 MB unless the service resource data specifies otherwise.
- Cluster configuration for the authenticated identity.
- Service records with image, volume, resource, domain, port, and secret bundle references.
- Image records pointing either to internal Cloudly image storage or an external registry.
- Secret bundles that can be flattened into environment key/value data.
- SSL certificates for all routed domains.
### Volumes
## Corestore Volumes
Coreflow deploys `corestore` as a global base service and bind mounts `/run/docker/plugins` so Docker can discover the `corestore` VolumeDriver socket on each node.
Workload services can declare first-class volumes:
Service records can declare first-class volumes:
```ts
volumes: [
@@ -155,31 +157,36 @@ volumes: [
]
```
If `name` is omitted, Coreflow derives a stable Docker volume name from the service id and mount path. During service creation it sends a Docker volume mount with `DriverConfig.Name = 'corestore'`, plus service metadata as driver options and volume labels.
If a volume name is omitted, Coreflow derives a stable Docker-safe name from the service id and mount path. The generated Docker mount uses `DriverConfig.Name = 'corestore'` by default and forwards service metadata as driver options and labels.
Coreflow also exposes Cloudly-triggered backup handlers over its TypedSocket connection. `executeServiceBackup` snapshots corestore volumes plus provisioned smartdb/smartstorage resources, and `executeServiceRestore` restores those snapshots back into the service's corestore resources.
### Routing
## Coretraffic Integration
`updateTrafficRouting()` inspects Docker services on `sznwebgateway`, matches them to Cloudly workload service records, resolves the first task IP currently available on the network, and builds `IReverseProxyConfig[]` entries from service domains and ports.
Coreflow starts an internal SmartServe/TypedSocket server on port `3000`. Coretraffic is expected to connect to that server and tag its connection as `coretraffic`.
For each routed hostname, Coreflow tries to export a certificate from an external dcrouter gateway first when Cloudly provided gateway config. If that fails or no gateway is configured, it requests a certificate from Cloudly. The final route set is sent to the Coretraffic connection tagged `coretraffic` with the `updateRouting` typed request.
When routing changes are computed, Coreflow sends:
Custom service ports are supported through `service.data.ports.custom`, where each key is treated as the hostname and the value is the destination port.
```ts
const request = typedsocketServer.createTypedRequest('updateRouting', coretrafficConnection);
await request.fire({ reverseConfigs });
## Platform and Corestore Integration
`PlatformManager` asks Cloudly for platform desired state and handles `database` and `objectstorage` bindings through Corestore.
Default Corestore control URL:
```text
http://corestore:3000
```
Each reverse config contains destination container IPs, destination ports, hostname, and certificate material.
Provisioned values such as `MONGODB_URI`, `S3_BUCKET`, `AWS_ACCESS_KEY_ID`, and `AWS_ENDPOINT_URL` are merged into the workload secret before Docker service creation. Coreflow also reports platform binding status back to Cloudly with endpoints and credential references.
## Docker Image Handling
## Backup and Restore
Coreflow supports two image sources from Cloudly image metadata:
`CoreflowBackupManager` registers two internal typed handlers:
- Internal Cloudly images: Coreflow pulls the requested image version stream from Cloudly and imports it into Docker from a tar stream.
- External registry images: Coreflow authenticates against the configured registry, pulls the image, and updates the local Docker image.
- `executeServiceBackup`
- `executeServiceRestore`
Invalid or incomplete image location data causes reconciliation to fail for that service, which is intentional: Coreflow only deploys services with complete desired-state data.
Backups provision Corestore bindings if necessary, snapshot backup-enabled service volumes, snapshot service DB/S3 resources, and optionally replicate the local Corestore archive back through Cloudly. Restores can pull missing archive objects from Cloudly, create missing Corestore volumes, restore volume snapshots, and restore DB/S3 snapshots.
## Development
@@ -187,30 +194,34 @@ Common commands:
```sh
pnpm install
pnpm run build
pnpm build
pnpm test
pnpm run watch
pnpm run build:docker
```
Project layout:
Important paths:
| Path | Purpose |
| --- | --- |
| `ts/index.ts` | CLI startup wrapper and lifecycle entrypoints. |
| `ts/coreflow.classes.coreflow.ts` | Main coordinator class. |
| `ts/coreflow.connector.cloudlyconnector.ts` | Cloudly API connection and identity handling. |
| `ts/coreflow.classes.clustermanager.ts` | Docker network, service, secret, image, and routing reconciliation. |
| `ts/coreflow.classes.taskmanager.ts` | Buffered and scheduled reconciliation task chain. |
| `ts/coreflow.connector.coretrafficconnector.ts` | TypedSocket routing updates to Coretraffic. |
| `ts/coreflow.classes.internalserver.ts` | Internal SmartServe and TypedSocket server. |
| `ts/index.ts` | CLI startup wrapper exporting `runCli` and `stop`. |
| `ts/coreflow.classes.coreflow.ts` | Main coordinator and component lifecycle. |
| `ts/coreflow.connector.cloudlyconnector.ts` | Cloudly API connection, machine-token authentication, and config fetches. |
| `ts/coreflow.classes.clustermanager.ts` | Docker networks, base services, workload services, secrets, images, volumes, and routing. |
| `ts/coreflow.classes.platformmanager.ts` | Corestore-backed platform binding reconciliation. |
| `ts/coreflow.classes.backupmanager.ts` | Backup and restore execution handlers. |
| `ts/coreflow.connector.coretrafficconnector.ts` | TypedSocket route updates to Coretraffic. |
| `ts/coreflow.connector.externalgateway.ts` | dcrouter certificate export and public route sync. |
| `ts/coreflow.classes.taskmanager.ts` | Buffered and scheduled reconciliation tasks. |
| `ts/coreflow.classes.internalserver.ts` | Internal SmartServe and TypedSocket server on port `3000`. |
## Operational Notes
- Coreflow expects Docker access through the local Docker socket by default.
- Reconciliation removes and recreates services when the Docker service reports that it needs an update.
- Workload services must be attached to `sznwebgateway` for routing to be generated.
- The current routing logic uses the first available container IP for a service.
- `PlatformManager` provisions `database` and `objectstorage` bindings through corestore.
- Coreflow expects to run on a Docker Swarm manager with local Docker socket access.
- Workload routing currently uses the first available task IP for a Docker service.
- A missing image, missing secret bundle, invalid registry credentials, or incomplete service port/domain data should fail reconciliation for that service rather than silently deploying the wrong thing.
- Corestore volume data is node-local; snapshot before destructive changes or restores.
- The package is marked private and is normally shipped as part of the serve.zone platform deployment flow.
## License and Legal Information