cloudly/readme.md

# @serve.zone/cloudly

Cloudly is the serve.zone control plane: a TypeScript service and browser dashboard that stores desired infrastructure state, authenticates humans and machines, coordinates clusters, serves an OCI registry, manages workload metadata, and pushes runtime configuration to connected node components.

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Why It Exists

Cloudly is the place where serve.zone operators describe what should run. It does not directly run every workload itself. Instead, it keeps the authoritative desired state in MongoDB and exposes TypedRequest/TypedSocket APIs so runtime components can reconcile that state where the containers actually live.

The current runtime pattern is reverse-connect:

```text
browser / CLI / SDK
  -> Cloudly HTTP + TypedSocket API
      -> MongoDB-backed managers
      -> S3-backed image and artifact storage
      <- Coreflow cluster agents connect outward
          -> Docker Swarm reconciliation
          -> Coretraffic routing updates
          -> Corestore platform resources and backups
```

## What Cloudly Manages

Cloudly currently coordinates these areas:

- **Authentication and identity**: human admin login, JWT identities, machine tokens, and cluster identities.
- **Clusters**: desired cluster records and machine users used by Coreflow to authenticate back to Cloudly.
- **Services**: workload definitions, image references, domains, ports, scale factors, secret bundles, volumes, and deployment metadata.
- **Images and registries**: image metadata, S3-backed image storage, external registry records, and an embedded OCI registry mounted at `/v2`.
- **Secrets**: secret groups and bundles that Coreflow flattens into Docker secrets for workloads.
- **Domains and DNS**: domain records, DNS entries, and optional domain sync from a dcrouter external gateway.
- **Platform bindings**: first-class `database` and `objectstorage` bindings that Coreflow provisions through Corestore.
- **Backups**: backup records, service backup/restore requests, scheduled backup tasks, and archive replication handshakes with Coreflow/Corestore.
- **BaseOS**: managed BaseOS node registration, heartbeat handling, desired-state response, image build tracking, and image download URLs.
- **CoreBuild workers**: selection of external build workers for BaseOS ISO/raw-image artifact generation.
- **Node and bare-metal inventory**: Hetzner-backed node creation paths and bare-metal metadata records where configured.
- **Dashboard**: a web component UI rendered from `ts_web` with views for clusters, services, images, secrets, domains, DNS, tasks, backups, BaseOS, and settings.

## Runtime Components

| Component | Role |
| --- | --- |
| `Cloudly` | Main service coordinator. Creates connectors and managers, then starts the API server. |
| `CloudlyServer` | TypedServer/TypedSocket HTTP server, dashboard static server, OCI registry HTTP bridge, and BaseOS HTTP endpoints. |
| `MongodbConnector` | SmartData persistence layer for Cloudly records. |
| `CloudflareConnector` | Optional Cloudflare account used by ACME DNS-01 when `cloudflareToken` is configured in settings. |
| `LetsencryptConnector` | SmartACME certificate issuance and certificate lookup. |
| `CloudlyCoreflowManager` | Authenticates Coreflow, returns cluster config payloads, and pushes config updates to connected Coreflow clients. |
| `CloudlyRegistryManager` | Embedded OCI registry backed by configured S3 storage, including deploy-on-push metadata updates. |
| `CloudlyBaseOsManager` | BaseOS registration, heartbeat, image build orchestration, worker selection, and artifact downloads. |
| `CloudlyBackupManager` | Service backup/restore orchestration and remote archive object replication. |

## Configuration

Cloudly uses `@push.rocks/npmextra` app data and environment mappings. Values can be supplied through environment variables, `.nogit` app-data files, or by constructing `new Cloudly(config)` programmatically.

Required runtime configuration:

| Variable | Purpose |
| --- | --- |
| `SERVEZONE_ENVIRONMENT` | ACME/runtime environment, currently `production` or `integration`. |
| `SERVEZONE_URL` | Public Cloudly hostname without protocol. |
| `SERVEZONE_PORT` | Public API/dashboard port as a string. |
| `SERVEZONE_SSLMODE` | `none`, `external`, or `letsencrypt`. |
| `SERVEZONE_ADMINACCOUNT` | First-run admin bootstrap in `username:password` format. |
| `MONGODB_URL` | MongoDB connection URL used by SmartData. |
| `MONGODB_NAME` | MongoDB database name. |
| `MONGODB_USER` | MongoDB username. |
| `MONGODB_PASS` | MongoDB password. |
| `S3_ENDPOINT` | S3-compatible endpoint for registry, images, and artifacts. |
| `S3_ACCESSKEY` | S3 access key. |
| `S3_SECRETKEY` | S3 secret key. |
| `S3_BUCKET` | S3 bucket name. |
| `S3_PORT` | S3 endpoint port. |
| `S3_USESSL` | Boolean SSL flag for the S3 endpoint. |

Common optional settings are stored through the Cloudly settings manager rather than direct environment variables:

| Setting | Purpose |
| --- | --- |
| `cloudflareToken` | Enables Cloudflare-backed ACME DNS-01 challenges. |
| `hetznerToken` | Enables Hetzner node and bare-metal provisioning paths. |
| `baseosJoinToken` | Allows BaseOS devices to enroll without a one-time image provisioning token. |
| `corebuildWorkersJson` | JSON array of CoreBuild worker URLs or `{ "url", "token", "id" }` objects. |
| `corebuildWorkerUrl` / `corebuildWorkerToken` | Legacy single-worker CoreBuild settings. |
| `dcrouterGatewayUrl` / `dcrouterGatewayApiToken` | Optional external gateway integration for domain and route sync. |

## Starting Cloudly

Install and build with pnpm:

```sh
pnpm install
pnpm build
pnpm start
```

Run the TypeScript entry point during development:

```sh
pnpm run startTs
```

Start from code when embedding the control plane in another Node.js process:

```ts
import { Cloudly } from '@serve.zone/cloudly';

const cloudly = new Cloudly({
  environment: 'production',
  publicUrl: 'cloudly.example.com',
  publicPort: '443',
  sslMode: 'external',
  servezoneAdminaccount: 'admin:change-me',
  mongoDescriptor: {
    mongoDbUrl: process.env.MONGODB_URL,
    mongoDbName: 'cloudly',
    mongoDbUser: process.env.MONGODB_USER,
    mongoDbPass: process.env.MONGODB_PASS,
  },
  s3Descriptor: {
    endpoint: process.env.S3_ENDPOINT,
    accessKey: process.env.S3_ACCESSKEY,
    accessSecret: process.env.S3_SECRETKEY,
    bucketName: process.env.S3_BUCKET,
    port: process.env.S3_PORT,
    useSsl: true,
  },
});

await cloudly.start();
```

Set `SERVEZONE_INSTALL_DEMO_DATA=true` only when you intentionally want the demo data installer to run. The code labels that path destructive.

## API Model

Cloudly exposes a single composed TypedRouter. Managers add their own typed handlers to the main router, and `CloudlyServer` exposes that router through the HTTP/WebSocket server.

Typical consumers use `@serve.zone/api`:

```ts
import { CloudlyApiClient } from '@serve.zone/api';

const client = new CloudlyApiClient({
  registerAs: 'admin-tool',
  cloudlyUrl: 'https://cloudly.example.com',
});

await client.start();
const identity = await client.loginWithUsernameAndPassword('admin', 'change-me');
const clusters = await client.cluster.getClusters();
```

Machine clients such as Coreflow authenticate with `getIdentityByToken`, request a stateful identity, and tag their WebSocket connection. That lets Cloudly push configuration to already-connected Coreflow instances instead of opening inbound connections to cluster nodes.

## Cluster Flow

The implemented cluster flow is intentionally simple:

1. An admin creates a Cloudly cluster record.
2. Cloudly creates a machine user with a long-lived cluster token.
3. Coreflow starts on a Docker Swarm manager node with `CLOUDLY_URL` and `JUMPCODE`.
4. Coreflow authenticates to Cloudly and requests the cluster configuration payload.
5. Cloudly returns cluster data, workload services, platform bindings, provider configs, and optional external gateway configuration.
6. Coreflow reconciles Docker networks, base services, workload services, secrets, volumes, platform bindings, backups, and routing.

When service, platform, or gateway settings change, Cloudly pushes updated config to connected Coreflow clients where supported.

## Registry and Deploy-On-Push

Cloudly serves an OCI registry under `/v2` through `CloudlyRegistryManager`. The registry uses configured S3 storage and issues OCI tokens from Cloudly authentication state.

For Cloudly-managed services, `getServiceRegistryTarget()` creates stable registry targets like:

```text
<cloudly-host>/workloads/<service-name>-<service-id>:<tag>
```

Registry push hooks record tag/digest metadata on the linked image and service. Unless `deployOnPush` is explicitly `false`, a successful push updates the service image version and asks connected Coreflow clients to reconcile.

## BaseOS and CoreBuild

Cloudly can manage BaseOS nodes and image builds:

- BaseOS devices register through `POST /baseos/v1/nodes/register` and heartbeat through `POST /baseos/v1/nodes/heartbeat`.
- A configured `baseosJoinToken` accepts generic device enrollment.
- BaseOS image builds create one-time provisioning tokens that are embedded in generated images.
- Cloudly selects a CoreBuild worker based on `/corebuild/v1/capabilities` and sends the build to `/corebuild/v1/jobs/baseos-image`.
- Completed artifacts are stored in the configured S3 bucket and served through short-lived `/baseos/v1/images/:buildId/download` URLs.

CoreBuild worker configuration can use `corebuildWorkersJson` for multiple workers or the legacy `corebuildWorkerUrl` and `corebuildWorkerToken` settings for one worker.

## Backups and Corestore

Cloudly owns backup records and user-facing backup/restore requests. Coreflow executes the cluster-local work, and Corestore snapshots volumes, database resources, object storage resources, and archive objects.

The backup path includes:

- `createServiceBackup` and `restoreServiceBackup` typed requests for admins.
- `executeServiceBackup` and `executeServiceRestore` requests from Cloudly to Coreflow.
- Corestore volume/resource snapshot and restore endpoints behind Coreflow.
- Optional archive replication through `prepareBackupReplication`, `uploadBackupArchiveObject`, `completeBackupReplication`, `getBackupArchiveManifest`, and `downloadBackupArchiveObject`.
- Optional scheduled `backup-all-services` task when `CLOUDLY_BACKUP_CRON` is set.

## External Gateway Integration

Cloudly can integrate with a dcrouter gateway when the gateway URL and API token are present in settings. The current integration syncs externally available domains into Cloudly and passes an external gateway route configuration to Coreflow. Coreflow can then ask dcrouter for certificates and synchronize public routes while still routing to cluster-local Coretraffic.

## Development

Common commands:

```sh
pnpm install
pnpm build
pnpm test
pnpm run build:docker
pnpm run release:docker
pnpm run docs
```

Important paths:

| Path | Purpose |
| --- | --- |
| `ts/index.ts` | CLI/runtime entry point exporting `runCli`, `Cloudly`, and `ICloudlyConfig`. |
| `ts/classes.cloudly.ts` | Main service coordinator and startup order. |
| `ts/classes.server.ts` | API/dashboard server, registry bridge, and BaseOS HTTP routes. |
| `ts/manager.*` | Domain managers for auth, clusters, services, images, registry, platform, backups, BaseOS, and more. |
| `ts/connector.*` | External system connectors for MongoDB, Cloudflare, and Let's Encrypt. |
| `ts_web/` | Browser dashboard web components. |
| `ts_cliclient/` | Published `@serve.zone/cli` submodule. |

## Accuracy Notes

The package metadata and settings schema include fields for several cloud providers. The code paths currently exercised in this repository are Cloudflare for ACME DNS-01, Hetzner for selected node/bare-metal provisioning paths, S3-compatible storage, MongoDB/SmartData, CoreBuild, Coreflow, Corestore, and optional dcrouter integration. Verify provider-specific behavior in the relevant manager before relying on it operationally.

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH  
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.