serve.zone/coreflow
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore(deps): modernize coreflow tooling

Browse Source
This commit is contained in:
Jürgen Kunz
2026-04-28 12:02:22 +00:00
parent 9cce79f040
commit 6ba5e36f4f
20 changed files with 6311 additions and 6166 deletions
Download Patch File Download Diff File Expand all files Collapse all files
readme.md
+164 -294
View File
@@ -1,344 +1,214 @@
# @serve.zone/coreflow
A comprehensive solution for managing Docker and scaling applications across servers, handling tasks from service provisioning to network traffic management.
Coreflow is the Docker Swarm reconciliation engine for the serve.zone platform. It runs inside a cluster, connects back to Cloudly, reads the desired cluster state, provisions the base runtime services, deploys workload services, and pushes reverse-proxy routing updates to Coretraffic.
## Install
## Issue Reporting and Security
To install @serve.zone/coreflow, you can use npm with the following command:
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
## What Coreflow Does
Coreflow sits between Cloudly and the Docker Swarm runtime:
- Connects to Cloudly over the `@serve.zone/api` WebSocket client and registers as `coreflow`.
- Authenticates with Cloudly using the cluster jump code token.
- Reads the cluster configuration and service definitions managed by Cloudly.
- Ensures base Docker networks exist for traffic and platform communication.
- Deploys and updates base services such as `coretraffic` and `corelog`.
- Deploys workload services from Cloudly image definitions.
- Creates Docker secrets from Cloudly secret bundles and attaches them to services.
- Builds reverse proxy configs from service domains, Docker task IPs, and Cloudly certificates.
- Sends routing updates to Coretraffic through the internal TypedSocket server.
- Reconciles state initially, on Cloudly config updates, and on a scheduled hourly task.
## Runtime Model
Coreflow is not a general-purpose application framework. It is a long-running cluster component designed to be started as a service or Docker container on a Docker Swarm manager node.
```text
Cloudly
-> Coreflow
-> local Docker Engine / Swarm
-> Coretraffic via internal TypedSocket
```
Coreflow never waits for Cloudly to call it. It connects outward to Cloudly, keeps the connection tagged as a `coreflow` client, and reacts to config update events from that connection.
## Requirements
- Node.js runtime compatible with the project toolchain.
- pnpm for dependency management.
- A Docker Swarm manager with access to the local Docker socket.
- A reachable Cloudly instance.
- A valid Cloudly jump code for the target cluster.
## Configuration
Coreflow reads runtime configuration through `@push.rocks/qenv` from the project environment and `.nogit` overlays.
Required environment variables:
| Variable | Purpose |
| --- | --- |
| `CLOUDLY_URL` | WebSocket/HTTP endpoint of the Cloudly control plane. |
| `JUMPCODE` | Cloudly token used to authenticate this Coreflow instance and tag the connection. |
Example `.nogit/.env`:
```env
CLOUDLY_URL=https://cloudly.example.com
JUMPCODE=cluster-machine-token
```
## Installation
This package is private and normally deployed as part of the serve.zone platform image pipeline.
```sh
npm install @serve.zone/coreflow --save
pnpm install
pnpm run build
pnpm start
```
Given that this is a private package, make sure you have access to the required npm registry and that you are authenticated properly.
The package also exposes the `coreflow` binary after build through `dist/cli.js`, while the repository entrypoint is `cli.js`.
## Usage
## Programmatic Startup
Coreflow is designed as an advanced tool for managing Docker-based applications and services, enabling efficient scaling across servers, and handling multiple aspects of service provisioning and network traffic management. Below are examples and explanations to illustrate its capabilities and how you can leverage Coreflow in your infrastructure. Note that these examples are based on TypeScript and use ESM syntax.
The CLI path imports `runCli()` from `dist_ts/index.js`. For direct TypeScript usage inside this repository, instantiate the main class and call `start()`:
### Prerequisites
```ts
import { Coreflow } from './ts/coreflow.classes.coreflow.js';
Before you start, ensure you have Docker and Docker Swarm configured in your environment as Coreflow operates on top of these technologies. Additionally, verify that your environment variables are properly set up for accessing Coreflow's functionalities.
const coreflow = new Coreflow();
await coreflow.start();
### Setting Up Coreflow
To get started, you need to import and initialize Coreflow within your application. Here's an example of how to do this in a TypeScript module:
```typescript
import { Coreflow } from '@serve.zone/coreflow';
// Initialize Coreflow
const coreflowInstance = new Coreflow();
// Start Coreflow
await coreflowInstance.start();
// Example: Add your logic here for handling Docker events
coreflowInstance.handleDockerEvents().then(() => {
console.log('Docker events are being handled.');
});
// Stop Coreflow when done
await coreflowInstance.stop();
```
In the above example:
- The Coreflow instance is initialized.
- Coreflow is started, which internally initializes various managers and connectors.
- The method `handleDockerEvents` is used to handle Docker events.
- Finally, Coreflow is stopped gracefully.
### Configuring Service Connections
Coreflow manages applications and services, often requiring direct interactions with other services like a database, message broker, or external API. Coreflow simplifies these connections through its configuration and service discovery layers.
```typescript
// Assuming coreflowInstance is already started as per previous examples
const serviceConnection = coreflowInstance.createServiceConnection({
serviceName: 'myDatabaseService',
servicePort: 3306,
});
serviceConnection.connect().then(() => {
console.log('Successfully connected to the service');
process.on('SIGTERM', async () => {
await coreflow.stop();
});
```
### Scaling Your Application
`start()` initializes components in this order:
Coreflow excels in scaling applications across multiple servers. This involves not just replicating services, but also ensuring they are properly networked, balanced, and monitored.
1. `InternalServer` starts a SmartServe server on port `3000` with TypedSocket support.
2. `CloudlyConnector` connects to Cloudly and resolves the cluster identity.
3. `ClusterManager` reads initial Cloudly config and subscribes to config updates.
4. `PlatformManager` starts its placeholder lifecycle hook.
5. `CoreflowTaskmanager` schedules the initial and recurring reconciliation tasks.
```typescript
const scalingPolicy = {
serviceName: 'apiService',
replicaCount: 5, // Target number of replicas
maxReplicaCount: 10, // Maximum number of replicas
minReplicaCount: 2, // Minimum number of replicas
};
## Reconciliation Flow
coreflowInstance.applyScalingPolicy(scalingPolicy).then(() => {
console.log('Scaling policy applied successfully.');
The task manager coordinates the runtime work as a task chain:
```text
updateBaseServices
-> updateWorkloadServices
-> updateTrafficRouting
```
`updateBaseServices` ensures the base Docker networks and platform services exist:
- `sznwebgateway` for public web routing.
- `szncorechat` for internal base-service communication.
- `coretraffic` attached to both networks with host ports `80` and `443` mapped to its service ports.
- `corelog` attached to the internal network.
`updateWorkloadServices` fetches Cloudly services, skips non-workload service categories, pulls or imports the configured Docker image, creates a Docker secret from the assigned secret bundle, and creates or replaces the Docker service when an update is required.
`updateTrafficRouting` inspects Docker services on the web gateway network, resolves container IPs, fetches certificates for configured domains, and sends `IReverseProxyConfig[]` updates to Coretraffic with the `updateRouting` typed request.
The same base-service task is triggered when Cloudly emits a config update. After the initial delayed run, it is also scheduled hourly.
## Cloudly Integration
`CloudlyConnector` wraps `CloudlyApiClient` from `@serve.zone/api`:
```ts
this.cloudlyApiClient = new CloudlyApiClient({
registerAs: 'coreflow',
cloudlyUrl,
});
```
In the above example:
After connection, Coreflow authenticates with `JUMPCODE` and requests a stateful, tagged identity. That identity is then used to fetch cluster configuration and certificates.
- A scaling policy is defined with target, maximum, and minimum replica counts for the `apiService`.
- The `applyScalingPolicy` method of the Coreflow instance is used to apply this scaling policy.
Coreflow depends on these Cloudly-side resources being present and valid:
### Managing Network Traffic
- Cluster configuration for the authenticated identity.
- Service records with image, resource, domain, port, and secret bundle references.
- Image records pointing either to internal Cloudly image storage or an external registry.
- Secret bundles that can be flattened into environment key/value data.
- SSL certificates for all routed domains.
One of Coreflow's key features is its ability to manage network traffic, ensuring that it is efficiently distributed among various services based on load, priority, and other custom rules.
## Coretraffic Integration
```typescript
import { TrafficRule } from '@serve.zone/coreflow';
Coreflow starts an internal SmartServe/TypedSocket server on port `3000`. Coretraffic is expected to connect to that server and tag its connection as `coretraffic`.
const rule: TrafficRule = {
serviceName: 'webService',
externalPort: 80,
internalPort: 3000,
protocol: 'http',
};
When routing changes are computed, Coreflow sends:
coreflowInstance.applyTrafficRule(rule).then(() => {
console.log('Traffic rule applied successfully.');
});
```ts
const request = typedsocketServer.createTypedRequest('updateRouting', coretrafficConnection);
await request.fire({ reverseConfigs });
```
In the above example:
Each reverse config contains destination container IPs, destination ports, hostname, and certificate material.
- A traffic rule is defined for the `webService`, redirecting external traffic from port 80 to the service's internal port 3000.
- The `applyTrafficRule` method is used to enforce this rule.
## Docker Image Handling
### Continuous Deployment
Coreflow supports two image sources from Cloudly image metadata:
Coreflow integrates continuous integration and deployment processes, allowing seamless updates and rollbacks for your services:
- Internal Cloudly images: Coreflow pulls the requested image version stream from Cloudly and imports it into Docker from a tar stream.
- External registry images: Coreflow authenticates against the configured registry, pulls the image, and updates the local Docker image.
```typescript
const deploymentConfig = {
serviceName: 'userAuthService',
image: 'myregistry.com/userauthservice:latest',
updatePolicy: 'rolling', // or "recreate"
};
Invalid or incomplete image location data causes reconciliation to fail for that service, which is intentional: Coreflow only deploys services with complete desired-state data.
coreflowInstance.deployService(deploymentConfig).then(() => {
console.log('Service deployed successfully.');
});
## Development
Common commands:
```sh
pnpm install
pnpm run build
pnpm test
pnpm run watch
```
In the above example:
Project layout:
- A deployment configuration is created for the `userAuthService` using the latest image from the specified registry.
- The `deployService` method is then used to deploy the service using the specified update policy (e.g., rolling updates or recreating the service).
| Path | Purpose |
| --- | --- |
| `ts/index.ts` | CLI startup wrapper and lifecycle entrypoints. |
| `ts/coreflow.classes.coreflow.ts` | Main coordinator class. |
| `ts/coreflow.connector.cloudlyconnector.ts` | Cloudly API connection and identity handling. |
| `ts/coreflow.classes.clustermanager.ts` | Docker network, service, secret, image, and routing reconciliation. |
| `ts/coreflow.classes.taskmanager.ts` | Buffered and scheduled reconciliation task chain. |
| `ts/coreflow.connector.coretrafficconnector.ts` | TypedSocket routing updates to Coretraffic. |
| `ts/coreflow.classes.internalserver.ts` | Internal SmartServe and TypedSocket server. |
### Observability and Monitoring
## Operational Notes
To keep track of your applications' health and performance, Coreflow provides tools for logging, monitoring, and alerting.
- Coreflow expects Docker access through the local Docker socket by default.
- Reconciliation removes and recreates services when the Docker service reports that it needs an update.
- Workload services must be attached to `sznwebgateway` for routing to be generated.
- The current routing logic uses the first available container IP for a service.
- `PlatformManager` currently provides lifecycle hooks but does not reconcile platform services yet.
```typescript
coreflowInstance.monitorService('webService').on('serviceHealthUpdate', (healthStatus) => {
console.log(`Received health update for webService: ${healthStatus}`);
});
```
## License and Legal Information
In the above example:
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.
- The `monitorService` method is used to monitor the health status of the `webService`.
- When a health update event is received, it is logged to the console.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
### Detailed Example: Setting Up and Managing Coreflow
### Trademarks
Here is a detailed example that covers various features, from setup to scaling and traffic management.
This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.
#### Step 1: Initialize Coreflow
Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.
```typescript
import { Coreflow } from '@serve.zone/coreflow';
### Company Information
const coreflowInstance = new Coreflow();
Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany
async function initializeCoreflow() {
await coreflowInstance.start();
console.log('Coreflow initialized.');
await manageServices();
}
For any legal inquiries or further information, please contact us via email at hello@task.vc.
initializeCoreflow().catch((error) => {
console.error('Error initializing Coreflow:', error);
});
```
#### Step 2: Handling Docker Events
```typescript
coreflowInstance.handleDockerEvents().then(() => {
console.log('Docker events are being handled.');
});
```
#### Step 3: Configuring and Connecting to a Service
```typescript
const serviceConnection = coreflowInstance.createServiceConnection({
serviceName: 'databaseService',
servicePort: 5432,
});
serviceConnection.connect().then(() => {
console.log('Successfully connected to the database service.');
});
```
#### Step 4: Applying a Scaling Policy
```typescript
const scalingPolicy = {
serviceName: 'microserviceA',
replicaCount: 3, // Starting with 3 replicas
maxReplicaCount: 10, // Allowing up to 10 replicas
minReplicaCount: 2, // Ensuring at least 2 replicas
};
coreflowInstance.applyScalingPolicy(scalingPolicy).then(() => {
console.log('Scaling policy applied for microserviceA');
});
```
#### Step 5: Managing Network Traffic
```typescript
import { TrafficRule } from '@serve.zone/coreflow';
const trafficRules: TrafficRule[] = [
{
serviceName: 'frontendService',
externalPort: 80,
internalPort: 3000,
protocol: 'http',
},
{
serviceName: 'apiService',
externalPort: 443,
internalPort: 4000,
protocol: 'https',
},
];
Promise.all(trafficRules.map((rule) => coreflowInstance.applyTrafficRule(rule))).then(() => {
console.log('Traffic rules applied.');
});
```
#### Step 6: Deploying a Service
```typescript
const deploymentConfig = {
serviceName: 'authService',
image: 'myregistry.com/authservice:latest',
updatePolicy: 'rolling', // Performing rolling updates
};
coreflowInstance.deployService(deploymentConfig).then(() => {
console.log('AuthService deployed successfully.');
});
```
#### Step 7: Monitoring a Service
```typescript
coreflowInstance.monitorService('frontendService').on('serviceHealthUpdate', (healthStatus) => {
console.log(`Health update for frontendService: ${healthStatus}`);
});
```
### Advanced Usage: Task Scheduling and Traffic Configuration
In more complex scenarios, you might want to leverage Coreflow's ability to schedule tasks and manage traffic configurations.
#### Scheduling Tasks
Coreflow supports scheduling updates and other tasks using the `taskBuffer` API.
```typescript
import { Task } from '@push.rocks/taskbuffer';
const checkinTask = new Task({
name: 'checkin',
buffered: true,
taskFunction: async () => {
console.log('Running checkin task...');
},
});
const taskManager = coreflowInstance.taskManager;
taskManager.addAndScheduleTask(checkinTask, '0 * * * * *'); // Scheduling task to run every minute
taskManager.start().then(() => {
console.log('Task manager started.');
});
```
#### Managing Traffic Routing
Coreflow can manage complex traffic routing scenarios, such as configuring reverse proxies for different services.
```typescript
import { CoretrafficConnector } from '@serve.zone/coreflow';
// Assume coreflowInstance is already started
const coretrafficConnector = new CoretrafficConnector(coreflowInstance);
const reverseProxyConfigs = [
{
hostName: 'example.com',
destinationIp: '192.168.1.100',
destinationPort: '3000',
privateKey: '<your-private-key>',
publicKey: '<your-public-key>',
},
{
hostName: 'api.example.com',
destinationIp: '192.168.1.101',
destinationPort: '4000',
privateKey: '<your-private-key>',
publicKey: '<your-public-key>',
},
];
coretrafficConnector.setReverseConfigs(reverseProxyConfigs).then(() => {
console.log('Reverse proxy configurations applied.');
});
```
### Integrating with Cloudly
Coreflow is designed to integrate seamlessly with Cloudly, a configuration management and orchestration tool.
#### Starting the Cloudly Connector
```typescript
const cloudlyConnector = coreflowInstance.cloudlyConnector;
cloudlyConnector.start().then(() => {
console.log('Cloudly connector started.');
});
```
#### Retrieving and Applying Configurations from Cloudly
```typescript
cloudlyConnector.getConfigFromCloudly().then((config) => {
console.log('Received configuration from Cloudly:', config);
coreflowInstance.clusterManager.provisionWorkloadServices(config).then(() => {
console.log('Workload services provisioned based on Cloudly config.');
});
});
```
### Conclusion
Coreflow is a powerful and flexible tool for managing Docker-based applications, scaling services, configuring network traffic, handling continuous deployments, and ensuring observability of your infrastructure. The examples provided aim to give a comprehensive understanding of how to use Coreflow in various scenarios, ensuring it meets your DevOps and CI/CD needs.
By leveraging Coreflow's rich feature set, you can optimize your infrastructure for high availability, scalability, and efficient operation across multiple servers and environments.
undefined
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.