cloudly/readme.md

# @serve.zone/cloudly

Cloudly is the serve.zone control plane: a TypeScript service and browser dashboard that stores desired infrastructure state, authenticates humans and machines, coordinates clusters, serves an OCI registry, manages workload metadata, and pushes runtime configuration to connected node components.

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Why It Exists

Cloudly is the place where serve.zone operators describe what should run. It does not directly run every workload itself. Instead, it keeps the authoritative desired state in MongoDB and exposes TypedRequest/TypedSocket APIs so runtime components can reconcile that state where the containers actually live.

The current runtime pattern is reverse-connect:

```text
browser / CLI / SDK
  -> Cloudly HTTP + TypedSocket API
      -> MongoDB-backed managers
      -> S3-backed image and artifact storage
      <- Coreflow cluster agents connect outward
          -> Docker Swarm reconciliation
          -> Coretraffic routing updates
          -> Corestore platform resources and backups
```

## What Cloudly Manages

Cloudly currently coordinates these areas:

- **Authentication and identity**: human admin login, JWT identities, machine tokens, and cluster identities.
- **Clusters**: desired cluster records and machine users used by Coreflow to authenticate back to Cloudly.
- **Services**: workload definitions, image references, domains, ports, scale factors, secret bundles, volumes, and deployment metadata.
- **Deployments**: deployment records, node placement metadata, health/resource fields, restart/scale API stubs, and DNS activation/deactivation hooks.
- **Images and registries**: image metadata, S3-backed image storage, external registry records, and an embedded OCI registry mounted at `/v2`.
- **Secrets**: secret groups and bundles that Coreflow flattens into Docker secrets for workloads.
- **Domains and DNS**: domain records, DNS entries, and optional domain sync from a dcrouter external gateway.
- **Platform bindings**: capabilities such as `database`, `objectstorage`, `logging`, `backup`, and RPC-style platform services that Coreflow/Corestore can reconcile.
- **Backups**: backup records, service backup/restore requests, scheduled backup tasks, and archive replication handshakes with Coreflow/Corestore.
- **BaseOS**: managed BaseOS node registration, heartbeat handling, desired-state response, image build tracking, and image download URLs.
- **CoreBuild workers**: selection of external build workers for BaseOS ISO and balena raw-image artifact generation.
- **Tasks**: TaskBuffer-backed operational tasks with execution history, metrics, logs, manual triggers, cancellation, and cron schedules.
- **Node and bare-metal inventory**: Hetzner-backed node creation paths and bare-metal metadata records where configured.
- **Dashboard**: a web component UI rendered from `ts_web` with views for overview, settings, secrets, clusters, external registries, images, services, deployments, tasks, domains, DNS, mail/log/storage/database shells, backups, and BaseOS.

## Runtime Components

| Component | Role |
| --- | --- |
| `Cloudly` | Main service coordinator. Creates connectors and managers, then starts the API server. |
| `CloudlyServer` | TypedServer/TypedSocket HTTP server, dashboard static server, OCI registry HTTP bridge, and BaseOS HTTP endpoints. |
| `MongodbConnector` | SmartData persistence layer for Cloudly records. |
| `CloudflareConnector` | Optional Cloudflare account used by ACME DNS-01 when `cloudflareToken` is configured in settings. |
| `LetsencryptConnector` | SmartACME certificate issuance and certificate lookup. |
| `CloudlyCoreflowManager` | Authenticates Coreflow, returns cluster config payloads, and pushes config updates to connected Coreflow clients. |
| `CloudlyRegistryManager` | Embedded OCI registry backed by configured S3 storage, including deploy-on-push metadata updates. |
| `CloudlyBaseOsManager` | BaseOS registration, heartbeat, image build orchestration, worker selection, and artifact downloads. |
| `CloudlyBackupManager` | Service backup/restore orchestration and remote archive object replication. |
| `CloudlyTaskManager` | Registers predefined and runtime tasks, tracks task executions, schedules cron jobs, and exposes task APIs. |
| `CloudlySettingsManager` | Stores runtime settings in MongoDB, masks sensitive values for API responses, and refreshes gateway/Coreflow state after relevant changes. |

## Configuration

Cloudly uses `@push.rocks/smartconfig` `AppData` with environment mappings. The runtime entry point loads `.nogit`/environment values through `@push.rocks/qenv`, and embedded callers can override values by constructing `new Cloudly(config)` programmatically.

Required runtime configuration:

| Variable | Purpose |
| --- | --- |
| `SERVEZONE_ENVIRONMENT` | ACME/runtime environment, currently `production` or `integration`. |
| `SERVEZONE_URL` | Public Cloudly hostname without protocol. |
| `SERVEZONE_PORT` | Public API/dashboard port as a string. |
| `SERVEZONE_SSLMODE` | `none`, `external`, or `letsencrypt`. |
| `SERVEZONE_ADMINACCOUNT` | First-run admin bootstrap in `username:password` format. |
| `MONGODB_URL` | MongoDB connection URL used by SmartData. |
| `MONGODB_NAME` | MongoDB database name. |
| `MONGODB_USER` | MongoDB username. |
| `MONGODB_PASS` | MongoDB password. |
| `S3_ENDPOINT` | S3-compatible endpoint for registry, images, and artifacts. |
| `S3_ACCESSKEY` | S3 access key. |
| `S3_SECRETKEY` | S3 secret key. |
| `S3_BUCKET` | S3 bucket name. |
| `S3_PORT` | S3 endpoint port. |
| `S3_USESSL` | Boolean SSL flag for the S3 endpoint. |

Common optional settings are stored through the Cloudly settings manager rather than direct environment variables:

| Setting | Purpose |
| --- | --- |
| `cloudflareToken` | Enables Cloudflare-backed ACME DNS-01 challenges. |
| `hetznerToken` | Enables Hetzner node and bare-metal provisioning paths. |
| `baseosJoinToken` | Allows BaseOS devices to enroll without a one-time image provisioning token. |
| `corebuildWorkersJson` | JSON array of CoreBuild worker URLs or `{ "url", "token", "id" }` objects. |
| `corebuildWorkerUrl` / `corebuildWorkerToken` | Legacy single-worker CoreBuild settings. |
| `dcrouterGatewayUrl` / `dcrouterGatewayApiToken` | Optional external gateway integration for domain and route sync. |
| `dcrouterWorkHosterId` | Optional stable external gateway work hoster ID; defaults to the cluster ID. |
| `dcrouterTargetHost` / `dcrouterTargetPort` | Optional target address that dcrouter should forward workload traffic to. |

Optional runtime environment variables:

| Variable | Purpose |
| --- | --- |
| `SERVEZONE_INSTALL_DEMO_DATA` | Runs the destructive demo data installer when set to `true`. |
| `CLOUDLY_BACKUP_CRON` | Enables the scheduled `backup-all-services` task with the supplied cron expression. |
| `CLOUDLY_BACKUP_KEEP_LAST` | Number of completed/failed backups to retain per service; defaults to `24`. |
| `CLOUDLY_BACKUP_TARGET_TYPE` | Remote archive replication target, currently `s3` or `smb`. |
| `CLOUDLY_BACKUP_TARGET_PREFIX` | Remote backup path prefix; defaults to `serve.zone-backups`. |
| `CLOUDLY_BASEOS_IMAGE_CLEANUP_INTERVAL_MS` | BaseOS image artifact cleanup interval; defaults to 12 hours. |

For backup replication with `CLOUDLY_BACKUP_TARGET_TYPE=s3`, set `CLOUDLY_BACKUP_S3_ENDPOINT`, `CLOUDLY_BACKUP_S3_ACCESS_KEY`, `CLOUDLY_BACKUP_S3_SECRET_KEY`, and `CLOUDLY_BACKUP_S3_BUCKET`. Optional S3 variables are `CLOUDLY_BACKUP_S3_REGION`, `CLOUDLY_BACKUP_S3_PORT`, and `CLOUDLY_BACKUP_S3_USE_SSL`.

For backup replication with `CLOUDLY_BACKUP_TARGET_TYPE=smb`, set `CLOUDLY_BACKUP_SMB_HOST` and `CLOUDLY_BACKUP_SMB_SHARE`. Optional SMB variables are `CLOUDLY_BACKUP_SMB_PORT`, `CLOUDLY_BACKUP_SMB_USERNAME`, `CLOUDLY_BACKUP_SMB_PASSWORD`, and `CLOUDLY_BACKUP_SMB_DOMAIN`.

## Starting Cloudly

Install and build with pnpm:

```sh
pnpm install
pnpm build
pnpm start
```

Run the TypeScript entry point during development:

```sh
pnpm run startTs
```

Start from code when embedding the control plane in another Node.js process:

```ts
import { Cloudly } from '@serve.zone/cloudly';

const cloudly = new Cloudly({
  environment: 'production',
  publicUrl: 'cloudly.example.com',
  publicPort: '443',
  sslMode: 'external',
  servezoneAdminaccount: 'admin:change-me',
  mongoDescriptor: {
    mongoDbUrl: process.env.MONGODB_URL,
    mongoDbName: 'cloudly',
    mongoDbUser: process.env.MONGODB_USER,
    mongoDbPass: process.env.MONGODB_PASS,
  },
  s3Descriptor: {
    endpoint: process.env.S3_ENDPOINT,
    accessKey: process.env.S3_ACCESSKEY,
    accessSecret: process.env.S3_SECRETKEY,
    bucketName: process.env.S3_BUCKET,
    port: process.env.S3_PORT,
    useSsl: true,
  },
});

await cloudly.start();
```

Set `SERVEZONE_INSTALL_DEMO_DATA=true` only when you intentionally want the demo data installer to run. The code labels that path destructive.

## API Model

Cloudly exposes a single composed TypedRouter. Managers add their own typed handlers to the main router, and `CloudlyServer` exposes that router through the HTTP/WebSocket server.

On first startup, Cloudly bootstraps the first human admin from `SERVEZONE_ADMINACCOUNT`. Human clients authenticate through `adminLoginWithUsernameAndPassword`; machine clients authenticate through `getIdentityByToken`. Cluster creation creates a machine user and token for Coreflow.

Typical consumers use `@serve.zone/api`:

```ts
import { CloudlyApiClient } from '@serve.zone/api';

const client = new CloudlyApiClient({
  registerAs: 'admin-tool',
  cloudlyUrl: 'https://cloudly.example.com',
});

await client.start();
const identity = await client.loginWithUsernameAndPassword('admin', 'change-me');
const clusters = await client.cluster.getClusters();
```

Machine clients such as Coreflow authenticate with `getIdentityByToken`, request a stateful identity, and tag their WebSocket connection. That lets Cloudly push configuration to already-connected Coreflow instances instead of opening inbound connections to cluster nodes.

## Cluster Flow

The implemented cluster flow is intentionally simple:

1. An admin creates a Cloudly cluster record.
2. Cloudly creates a machine user with a long-lived cluster token.
3. Coreflow starts on a Docker Swarm manager node with `CLOUDLY_URL` and `JUMPCODE`.
4. Coreflow authenticates to Cloudly and requests the cluster configuration payload.
5. Cloudly returns cluster data, workload services, platform bindings, provider configs, and optional external gateway configuration.
6. Coreflow reconciles Docker networks, base services, workload services, secrets, volumes, platform bindings, backups, and routing.

When service, platform, or gateway settings change, Cloudly pushes updated config to connected Coreflow clients where supported.

## Registry and Deploy-On-Push

Cloudly serves an OCI registry under `/v2` through `CloudlyRegistryManager`. The registry uses configured S3 storage and issues OCI tokens from Cloudly authentication state.

For Cloudly-managed services, `getServiceRegistryTarget()` creates stable registry targets like:

```text
<cloudly-host>/workloads/<service-name>-<service-id-prefix>:<tag>
```

Registry push hooks record tag/digest metadata on the linked image and service. Unless `deployOnPush` is explicitly `false`, a successful push updates the service image version and asks connected Coreflow clients to reconcile.

Registry token requests use HTTP Basic credentials against Cloudly users. User passwords and unexpired user tokens are accepted; push/delete scopes require an admin user or a token with the `admin` assigned role.

## BaseOS and CoreBuild

Cloudly can manage BaseOS nodes and image builds:

- BaseOS devices register through `POST /baseos/v1/nodes/register` and heartbeat through `POST /baseos/v1/nodes/heartbeat`.
- A configured `baseosJoinToken` accepts generic device enrollment.
- BaseOS image builds create one-time provisioning tokens that are embedded in generated images.
- Cloudly selects a CoreBuild worker based on `/corebuild/v1/capabilities` and sends the build to `/corebuild/v1/jobs/baseos-image`.
- Supported build kinds are `ubuntu-iso` and `balena-raw`; Raspberry Pi builds use `balena-raw`.
- Supported architecture values are `amd64`, `arm64`, and `rpi`.
- Completed artifacts are stored in the configured S3 bucket and served through short-lived `/baseos/v1/images/:buildId/download` URLs.

CoreBuild worker configuration can use `corebuildWorkersJson` for multiple workers or the legacy `corebuildWorkerUrl` and `corebuildWorkerToken` settings for one worker.

## Backups and Corestore

Cloudly owns backup records and user-facing backup/restore requests. Coreflow executes the cluster-local work, and Corestore snapshots volumes, database resources, object storage resources, and archive objects.

The backup path includes:

- `createServiceBackup` and `restoreServiceBackup` typed requests for admins.
- `executeServiceBackup` and `executeServiceRestore` requests from Cloudly to Coreflow.
- Corestore volume/resource snapshot and restore endpoints behind Coreflow.
- Optional archive replication through `prepareBackupReplication`, `uploadBackupArchiveObject`, `completeBackupReplication`, `getBackupArchiveManifest`, and `downloadBackupArchiveObject`.
- Optional scheduled `backup-all-services` task when `CLOUDLY_BACKUP_CRON` is set.

Manual `createServiceBackup` requests expect Coreflow to complete remote archive replication. Cloudly validates archive object size and SHA-256 checksums, writes a manifest, records target metadata, and marks completed backups as `replicated`. Restores read the manifest and objects back through the configured target writer.

## Task Automation

Cloudly registers a TaskBuffer-backed task manager. The API and dashboard can list tasks, trigger tasks manually, inspect execution logs/metrics, and request cancellation for running tasks.

Predefined tasks currently include:

| Task | Purpose |
| --- | --- |
| `cloudflare-domain-sync` | Imports and updates domains from configured Cloudflare zones. |
| `dns-sync` | Iterates DNS entries marked as external; provider sync is currently a placeholder. |
| `cert-renewal` | Checks activated domains for certificate renewal; renewal logic is currently a placeholder. |
| `cleanup` | Removes old task executions and contains placeholders for log/image cleanup. |
| `health-check` | Iterates deployments and records health metrics; runtime health checks are currently placeholders. |
| `resource-report` | Generates node resource metrics; values are currently placeholders until runtime metrics are wired in. |
| `db-maintenance` | Maintenance shell for database optimization tasks. |
| `security-scan` | Security scan shell for exposed ports, image freshness, and weak configuration checks. |
| `docker-cleanup` | Docker cleanup shell for containers, images, volumes, and networks. |
| `backup-all-services` | Registered by the backup manager and enabled only when `CLOUDLY_BACKUP_CRON` is set. |

## External Gateway Integration

Cloudly can integrate with a dcrouter gateway when the gateway URL and API token are present in settings. The current integration syncs externally available domains into Cloudly and passes an external gateway route configuration to Coreflow. Coreflow can then ask dcrouter for certificates and synchronize public routes while still routing to cluster-local Coretraffic.

## Development

Common commands:

```sh
pnpm install
pnpm build
pnpm test
pnpm run build:docker
pnpm run release:docker
pnpm run docs
```

Important paths:

| Path | Purpose |
| --- | --- |
| `ts/index.ts` | CLI/runtime entry point exporting `runCli`, `Cloudly`, and `ICloudlyConfig`. |
| `ts/classes.cloudly.ts` | Main service coordinator and startup order. |
| `ts/classes.server.ts` | API/dashboard server, registry bridge, and BaseOS HTTP routes. |
| `ts/manager.*` | Domain managers for auth, clusters, services, images, registry, platform, backups, BaseOS, and more. |
| `ts/connector.*` | External system connectors for MongoDB, Cloudflare, and Let's Encrypt. |
| `ts_web/` | Browser dashboard web components. |
| `ts_cliclient/` | Published `@serve.zone/cli` submodule. |

## Accuracy Notes

The package metadata and settings schema include fields for several cloud providers. The code paths currently exercised in this repository are Cloudflare for ACME DNS-01 and domain sync, Hetzner for selected node/bare-metal provisioning paths, S3-compatible storage, SMB/S3 backup archive targets, MongoDB/SmartData, CoreBuild, Coreflow, Corestore, and optional dcrouter integration. Several provider connection tests and predefined tasks are configuration checks or implementation shells; verify provider-specific behavior in the relevant manager before relying on it operationally.

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH  
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.