smartregistry/readme.md

# @push.rocks/smartregistry

> One TypeScript registry core for OCI, npm, Maven, Cargo, Composer, PyPI, and RubyGems.

`@push.rocks/smartregistry` is a composable library for building your own multi-protocol package registry. You hand it HTTP requests, it routes them to the right protocol handler, stores artifacts in S3-compatible object storage, enforces shared auth scopes, and can proxy/cache upstream registries when content is not local.

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Why It Hits

- One registry engine, seven ecosystems.
- Shared S3-backed storage instead of protocol-specific silos.
- Shared auth, token minting, and scope checks across protocols.
- Optional upstream proxying with retries, stale caching, negative caching, and circuit breakers.
- Clean public API: `handleRequest()` in, `IResponse` out.

## What It Actually Ships

| Protocol | Paths | What works | Auth style |
| --- | --- | --- | --- |
| OCI | `/oci/*` or `/v2/*` | version check, blobs, manifests, tags, referrers, deletes | Bearer JWT |
| npm | `/npm/*` | login, publish, packuments, version metadata, tarballs, dist-tags, search, token APIs, unpublish | Bearer token |
| Maven | `/maven/*` | POM/JAR/WAR upload and download, `maven-metadata.xml`, auto-generated checksums, delete | Bearer token or Basic auth with the token as password |
| Cargo | `/cargo/*` | sparse index, `config.json`, publish, download, search, yank, unyank | plain `Authorization` token |
| Composer | `/composer/*` | `packages.json`, `p2` metadata, ZIP dists, filtered package lists, upload, version delete, package delete | Bearer token, plus Basic auth for credential-backed reads |
| PyPI | `/simple/*` and `/pypi/*` | PEP 503 HTML, PEP 691 JSON, upload, JSON metadata API, downloads, package delete, version delete | Bearer token or Basic `__token__:<token>` |
| RubyGems | `/rubygems/*` | Compact Index, gem downloads, versions/dependencies JSON, specs endpoints, upload, yank, unyank | plain `Authorization` token |

Use `oci.basePath = '/v2'` if you want native Docker/OCI client compatibility. The default `/oci` path is fine for app-level routing, but Docker expects `/v2`.

Set `pypi.registryUrl` to the host root, not `/pypi`, because the Simple API lives at `/simple/*` while uploads and JSON endpoints live under `/pypi/*`.

## Install

```bash
pnpm add @push.rocks/smartregistry
```

## Quick Start

```ts
import { SmartRegistry, type IRegistryConfig } from '@push.rocks/smartregistry';

const publicUrl = 'https://registry.example.com';

const config: IRegistryConfig = {
  storage: {
    accessKey: process.env.S3_ACCESS_KEY!,
    accessSecret: process.env.S3_ACCESS_SECRET!,
    endpoint: 's3.example.com',
    port: 443,
    useSsl: true,
    region: 'eu-central-1',
    bucketName: 'registry-artifacts',
  },
  auth: {
    jwtSecret: process.env.REGISTRY_JWT_SECRET!,
    tokenStore: 'memory',
    npmTokens: { enabled: true },
    ociTokens: {
      enabled: true,
      realm: `${publicUrl}/v2/token`,
      service: 'smartregistry',
    },
    pypiTokens: { enabled: true },
    rubygemsTokens: { enabled: true },
  },
  oci: { enabled: true, basePath: '/v2' },
  npm: { enabled: true, basePath: '/npm', registryUrl: `${publicUrl}/npm` },
  maven: { enabled: true, basePath: '/maven', registryUrl: `${publicUrl}/maven` },
  cargo: { enabled: true, basePath: '/cargo', registryUrl: `${publicUrl}/cargo` },
  composer: { enabled: true, basePath: '/composer', registryUrl: `${publicUrl}/composer` },
  pypi: { enabled: true, basePath: '/pypi', registryUrl: publicUrl },
  rubygems: { enabled: true, basePath: '/rubygems', registryUrl: `${publicUrl}/rubygems` },
};

const registry = new SmartRegistry(config);
await registry.init();

const auth = registry.getAuthManager();
const npmToken = await auth.createNpmToken('ci-bot');
const cargoToken = await auth.createCargoToken('ci-bot');
const ociToken = await auth.createOciToken('ci-bot', ['oci:repository:myorg/myapp:*']);

console.log({ npmToken, cargoToken, ociToken });
```

If you do not pass `authProvider`, the library uses `DefaultAuthProvider`, an in-memory reference implementation. That is perfect for tests and local dev, but you will usually want a real `IAuthProvider` in production.

## HTTP Integration

`SmartRegistry` is not a web framework. You own the HTTP server, request parsing, and response writing. The happy path is very small:

```ts
import { createServer, type IncomingHttpHeaders } from 'node:http';
import { Readable } from 'node:stream';

function headersToRecord(headers: IncomingHttpHeaders): Record<string, string> {
  return Object.fromEntries(
    Object.entries(headers).map(([key, value]) => [
      key,
      Array.isArray(value) ? value.join(', ') : value ?? '',
    ])
  );
}

createServer(async (req, res) => {
  const url = new URL(req.url ?? '/', 'http://localhost');
  const headers = headersToRecord(req.headers);

  const chunks: Buffer[] = [];
  for await (const chunk of req) {
    chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk));
  }
  const rawBody = Buffer.concat(chunks);

  let body: unknown = rawBody.length ? rawBody : undefined;
  if ((headers['content-type'] ?? '').includes('application/json') && rawBody.length) {
    body = JSON.parse(rawBody.toString('utf8'));
  }

  const response = await registry.handleRequest({
    method: req.method ?? 'GET',
    path: url.pathname,
    query: Object.fromEntries(url.searchParams),
    headers,
    body,
    rawBody: rawBody.length ? rawBody : undefined,
  });

  res.writeHead(response.status, response.headers);

  if (!response.body) {
    res.end();
    return;
  }

  Readable.fromWeb(response.body).pipe(res);
}).listen(3000);
```

Keep `rawBody` for OCI manifests, OCI blobs, and any other digest-sensitive request where exact bytes matter.

For PyPI uploads, parse `multipart/form-data` before calling `handleRequest()` and pass the parsed fields in `context.body`. The library expects the upload form fields, not a raw multipart buffer.

At the public API boundary, `response.body` is always a `ReadableStream<Uint8Array>`.

## Core API

| API | Why you use it |
| --- | --- |
| `new SmartRegistry(config)` | build the registry orchestrator |
| `await registry.init()` | initialize storage, auth, and enabled protocols |
| `await registry.handleRequest(context)` | route one incoming HTTP request |
| `registry.getAuthManager()` | mint, validate, revoke, and authorize tokens |
| `registry.getStorage()` | reach the shared storage abstraction directly |
| `registry.getRegistry(protocol)` | access a specific protocol handler |
| `registry.destroy()` | clean up timers and protocol resources |

The package also exports the protocol-specific registry classes, upstream classes, `RegistryStorage`, `AuthManager`, `DefaultAuthProvider`, `StaticUpstreamProvider`, `UpstreamCache`, `CircuitBreaker`, and stream helpers such as `streamToBuffer()` and `streamToJson()`.

## Configuration Reference

| Key | Purpose |
| --- | --- |
| `storage` | S3-compatible backend config. This extends `IS3Descriptor` and adds `bucketName`. |
| `auth` | shared token settings across protocols |
| `authProvider` | plug in LDAP, OAuth, OIDC, custom DB-backed auth, or anything else implementing `IAuthProvider` |
| `storageHooks` | receive before/after put/get/delete callbacks with protocol, actor, package, and version context |
| `upstreamProvider` | decide per request which upstream registries to consult |
| `oci` / `npm` / `maven` / `cargo` / `composer` / `pypi` / `rubygems` | enable protocols, set base paths, and define the public registry URL they should emit |

## Upstream Proxying

If a package, image, crate, or artifact does not exist locally, a protocol handler can resolve an upstream config on the fly and fetch it from there.

```ts
import {
  StaticUpstreamProvider,
} from '@push.rocks/smartregistry';

const upstreamProvider = new StaticUpstreamProvider({
  npm: {
    enabled: true,
    upstreams: [
      {
        id: 'npmjs',
        name: 'npmjs',
        url: 'https://registry.npmjs.org',
        priority: 1,
        enabled: true,
        auth: { type: 'none' },
      },
    ],
  },
  oci: {
    enabled: true,
    upstreams: [
      {
        id: 'dockerhub',
        name: 'dockerhub',
        url: 'https://registry-1.docker.io',
        priority: 1,
        enabled: true,
        auth: { type: 'none' },
      },
    ],
  },
});
```

Pass that provider as `upstreamProvider` in your `IRegistryConfig`.

The upstream layer supports scope rules, per-request routing, retries with backoff, circuit breakers, stale-while-revalidate caching, and negative caching for 404s.

## Custom Auth and Audit Hooks

Bring your own auth system by implementing `IAuthProvider` and passing it as `authProvider`.

```ts
const registry = new SmartRegistry({
  ...config,
  authProvider: myAuthProvider,
});
```

Use `storageHooks` when you need quota checks, audit logs, or side effects around artifact writes and deletes.

```ts
const registry = new SmartRegistry({
  ...config,
  storageHooks: {
    async beforePut(context) {
      if ((context.metadata?.size ?? 0) > 500 * 1024 * 1024) {
        return { allowed: false, reason: 'Artifact too large' };
      }
      return { allowed: true };
    },
    async afterPut(context) {
      await auditLog('storage.put', context);
    },
  },
});
```

`handleRequest()` also accepts an `actor` object. That extra context flows into storage hooks and upstream resolution, which is great for multi-tenant routing, org-aware policy checks, and audit trails.

```ts
await registry.handleRequest({
  method: 'GET',
  path: '/npm/@acme/internal-lib',
  headers: { authorization: `Bearer ${npmToken}` },
  query: {},
  actor: {
    orgId: 'acme',
    sessionId: 'sess_123',
  },
});
```

## Client Cheatsheet

### npm

```ini
registry=https://registry.example.com/npm/
//registry.example.com/npm/:_authToken=<npm-token>
```

The npm handler also implements the npm-compatible login and token endpoints, including `PUT /-/user/org.couchdb.user:<name>` and `/-/npm/v1/tokens`.

### Docker / OCI

Set `oci.basePath` to `/v2` and expose a token endpoint that returns `token`, `access_token`, and `expires_in`.

```ts
if (url.pathname === '/v2/token') {
  const token = await registry.getAuthManager().createOciToken(
    'docker-user',
    ['oci:repository:*:*'],
    3600,
  );

  res.setHeader('Content-Type', 'application/json');
  res.end(JSON.stringify({
    token,
    access_token: token,
    expires_in: 3600,
  }));
  return;
}
```

```bash
docker login registry.example.com
docker push registry.example.com/myorg/myimage:latest
docker pull registry.example.com/myorg/myimage:latest
```

### Maven

```xml
<settings>
  <servers>
    <server>
      <id>smartregistry</id>
      <username>token</username>
      <password>YOUR_MAVEN_TOKEN</password>
    </server>
  </servers>
</settings>
```

Point repositories or distribution management at `https://registry.example.com/maven`.

### Cargo

```toml
# .cargo/config.toml
[registries.smartregistry]
index = "sparse+https://registry.example.com/cargo/"
```

```toml
# .cargo/credentials.toml
[registries.smartregistry]
token = "YOUR_CARGO_TOKEN"
```

Cargo sends the token as a plain `Authorization` header, not `Bearer <token>`.

### Composer

```json
{
  "repositories": [
    {
      "type": "composer",
      "url": "https://registry.example.com/composer"
    }
  ]
}
```

```json
{
  "http-basic": {
    "registry.example.com": {
      "username": "my-user",
      "password": "my-password"
    }
  }
}
```

Composer installs can use Basic auth if your `authProvider.authenticate()` supports it. Programmatic writes use Bearer tokens cleanly.

### PyPI

```ini
[distutils]
index-servers = smartregistry

[smartregistry]
repository = https://registry.example.com/pypi
username = __token__
password = YOUR_PYPI_TOKEN
```

```bash
pip install --index-url https://registry.example.com/simple/ your-package
twine upload --repository smartregistry dist/*
```

### RubyGems

```yaml
:rubygems_api_key: YOUR_RUBYGEMS_TOKEN
```

```bash
gem push your-gem-1.0.0.gem --host https://registry.example.com/rubygems
bundle config set --global https://registry.example.com/rubygems YOUR_RUBYGEMS_TOKEN
```

## Testing and Compatibility

The repository contains protocol-level tests, cross-protocol integration tests, upstream provider tests, storage hook tests, and native-client test suites that exercise the library through real ecosystem tooling.

Native-client coverage exists for:

- Docker / OCI
- npm
- Maven (`mvn`)
- Cargo
- Composer
- PyPI (`pip` and `twine`)
- RubyGems (`gem`)

There are also integration tests for S3-compatible storage and `IS3Descriptor`-based configuration.

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./license) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH  
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.