diff --git a/changelog.md b/changelog.md index dbba547..d7c36d8 100644 --- a/changelog.md +++ b/changelog.md @@ -1,5 +1,10 @@ # Changelog +## 2026-04-16 - 2.9.1 - fix(license) +add missing MIT license file to repository + +- Adds the project license file to align the repository contents with the package metadata license declaration. + ## 2026-04-16 - 2.9.0 - feat(registry) add declarative protocol routing and request-scoped storage hook context across registries diff --git a/license b/license new file mode 100644 index 0000000..7b98c0d --- /dev/null +++ b/license @@ -0,0 +1,19 @@ +Copyright (c) 2025 Task Venture Capital GmbH (hello@task.vc) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/readme.md b/readme.md index cffefaf..30fb182 100644 --- a/readme.md +++ b/readme.md @@ -1,877 +1,409 @@ # @push.rocks/smartregistry -> ๐Ÿš€ A composable TypeScript library implementing **OCI Distribution Specification v1.1**, **NPM Registry API**, **Maven Repository**, **Cargo/crates.io Registry**, **Composer/Packagist**, **PyPI (Python Package Index)**, and **RubyGems Registry** โ€” everything you need to build a unified container and package registry in one library. +> One TypeScript registry core for OCI, npm, Maven, Cargo, Composer, PyPI, and RubyGems. + +`@push.rocks/smartregistry` is a composable library for building your own multi-protocol package registry. You hand it HTTP requests, it routes them to the right protocol handler, stores artifacts in S3-compatible object storage, enforces shared auth scopes, and can proxy/cache upstream registries when content is not local. ## Issue Reporting and Security For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly. -## โœจ Features +## Why It Hits -### ๐Ÿ”„ Multi-Protocol Support -- **OCI Distribution Spec v1.1**: Full container registry with manifest/blob operations -- **NPM Registry API**: Complete package registry with publish/install/search -- **Maven Repository**: Java/JVM artifact management with POM support -- **Cargo/crates.io Registry**: Rust crate registry with sparse HTTP protocol -- **Composer/Packagist**: PHP package registry with Composer v2 protocol -- **PyPI (Python Package Index)**: Python package registry with PEP 503/691 support -- **RubyGems Registry**: Ruby gem registry with compact index protocol +- One registry engine, seven ecosystems. +- Shared S3-backed storage instead of protocol-specific silos. +- Shared auth, token minting, and scope checks across protocols. +- Optional upstream proxying with retries, stale caching, negative caching, and circuit breakers. +- Clean public API: `handleRequest()` in, `IResponse` out. -### ๐Ÿ—๏ธ Unified Architecture -- **Composable Design**: Core infrastructure with protocol plugins โ€” enable only what you need -- **Shared Storage**: Cloud-agnostic S3-compatible backend via [@push.rocks/smartbucket](https://www.npmjs.com/package/@push.rocks/smartbucket) with standardized `IS3Descriptor` from [@tsclass/tsclass](https://www.npmjs.com/package/@tsclass/tsclass) -- **Unified Authentication**: Scope-based permissions across all protocols -- **Path-based Routing**: `/oci/*`, `/npm/*`, `/maven/*`, `/cargo/*`, `/composer/*`, `/pypi/*`, `/rubygems/*` -- **Declarative Protocol Wiring**: Protocol registration, initialization, and routing stay aligned through shared descriptors +## What It Actually Ships -### ๐Ÿ” Authentication & Authorization -- NPM UUID tokens for package operations -- OCI JWT tokens for container operations -- Protocol-specific tokens for Maven, Cargo, Composer, PyPI, and RubyGems -- Unified scope system: `npm:package:foo:write`, `oci:repository:bar:push` -- **Pluggable Auth Provider** (`IAuthProvider`): Integrate LDAP, OAuth, SSO, or any custom auth +| Protocol | Paths | What works | Auth style | +| --- | --- | --- | --- | +| OCI | `/oci/*` or `/v2/*` | version check, blobs, manifests, tags, referrers, deletes | Bearer JWT | +| npm | `/npm/*` | login, publish, packuments, version metadata, tarballs, dist-tags, search, token APIs, unpublish | Bearer token | +| Maven | `/maven/*` | POM/JAR/WAR upload and download, `maven-metadata.xml`, auto-generated checksums, delete | Bearer token or Basic auth with the token as password | +| Cargo | `/cargo/*` | sparse index, `config.json`, publish, download, search, yank, unyank | plain `Authorization` token | +| Composer | `/composer/*` | `packages.json`, `p2` metadata, ZIP dists, filtered package lists, upload, version delete, package delete | Bearer token, plus Basic auth for credential-backed reads | +| PyPI | `/simple/*` and `/pypi/*` | PEP 503 HTML, PEP 691 JSON, upload, JSON metadata API, downloads, package delete, version delete | Bearer token or Basic `__token__:` | +| RubyGems | `/rubygems/*` | Compact Index, gem downloads, versions/dependencies JSON, specs endpoints, upload, yank, unyank | plain `Authorization` token | -### ๐Ÿ“ฆ Protocol Feature Matrix +Use `oci.basePath = '/v2'` if you want native Docker/OCI client compatibility. The default `/oci` path is fine for app-level routing, but Docker expects `/v2`. -| Feature | OCI | NPM | Maven | Cargo | Composer | PyPI | RubyGems | -|---------|-----|-----|-------|-------|----------|------|----------| -| Publish/Upload | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | -| Download | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | -| Search | โ€” | โœ… | โ€” | โœ… | โ€” | โ€” | โ€” | -| Version Yank | โ€” | โ€” | โ€” | โœ… | โ€” | โ€” | โœ… | -| Metadata API | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | -| Token Auth | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | -| Checksum Verification | โœ… | โœ… | โœ… | โœ… | โ€” | โœ… | โœ… | -| Upstream Proxy | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | +Set `pypi.registryUrl` to the host root, not `/pypi`, because the Simple API lives at `/simple/*` while uploads and JSON endpoints live under `/pypi/*`. -### ๐ŸŒ Upstream Proxy & Caching -- **Multi-Upstream Support**: Configure multiple upstream registries per protocol with priority ordering -- **Scope-Based Routing**: Route specific packages/scopes to different upstreams (e.g., `@company/*` โ†’ private registry) -- **S3-Backed Cache**: Persistent caching using existing S3 storage -- **Circuit Breaker**: Automatic failover with configurable thresholds -- **Stale-While-Revalidate**: Serve cached content while refreshing in background -- **Content-Aware TTLs**: Different TTLs for immutable (tarballs) vs mutable (metadata) content - -### ๐ŸŒŠ Streaming-First Architecture -- **Web Streams API** (`ReadableStream`) โ€” cross-runtime (Node, Deno, Bun) -- **Zero-copy downloads**: Binary artifacts stream directly from S3 to the HTTP response -- **OCI upload streaming**: Chunked blob uploads stored as temp S3 objects, not accumulated in memory -- **Unified response type**: Every `response.body` is a `ReadableStream` โ€” one pattern for all consumers - -### ๐Ÿ”Œ Enterprise Extensibility -- **Storage Event Hooks** (`IStorageHooks`): Quota tracking, audit logging, virus scanning, cache invalidation -- **Request Actor Context**: Pass user/org info through requests for audit trails and rate limiting -- **Request-Scoped Hook Metadata**: Hooks receive protocol, actor, package, and version context without cross-request leakage - -## ๐Ÿ“ฅ Installation +## Install ```bash -# Using pnpm (recommended) pnpm add @push.rocks/smartregistry - -# Using npm -npm install @push.rocks/smartregistry ``` -## ๐Ÿš€ Quick Start +## Quick Start -```typescript -import { SmartRegistry, IRegistryConfig } from '@push.rocks/smartregistry'; +```ts +import { SmartRegistry, type IRegistryConfig } from '@push.rocks/smartregistry'; + +const publicUrl = 'https://registry.example.com'; const config: IRegistryConfig = { storage: { - accessKey: 'your-s3-key', - accessSecret: 'your-s3-secret', - endpoint: 's3.amazonaws.com', + accessKey: process.env.S3_ACCESS_KEY!, + accessSecret: process.env.S3_ACCESS_SECRET!, + endpoint: 's3.example.com', port: 443, useSsl: true, - region: 'us-east-1', - bucketName: 'my-registry', + region: 'eu-central-1', + bucketName: 'registry-artifacts', }, auth: { - jwtSecret: 'your-secret-key', + jwtSecret: process.env.REGISTRY_JWT_SECRET!, tokenStore: 'memory', npmTokens: { enabled: true }, ociTokens: { enabled: true, - realm: 'https://auth.example.com/token', - service: 'my-registry', + realm: `${publicUrl}/v2/token`, + service: 'smartregistry', }, + pypiTokens: { enabled: true }, + rubygemsTokens: { enabled: true }, }, - // Enable only the protocols you need - oci: { enabled: true, basePath: '/oci' }, - npm: { enabled: true, basePath: '/npm' }, - maven: { enabled: true, basePath: '/maven' }, - cargo: { enabled: true, basePath: '/cargo' }, - composer: { enabled: true, basePath: '/composer' }, - pypi: { enabled: true, basePath: '/pypi' }, - rubygems: { enabled: true, basePath: '/rubygems' }, + oci: { enabled: true, basePath: '/v2' }, + npm: { enabled: true, basePath: '/npm', registryUrl: `${publicUrl}/npm` }, + maven: { enabled: true, basePath: '/maven', registryUrl: `${publicUrl}/maven` }, + cargo: { enabled: true, basePath: '/cargo', registryUrl: `${publicUrl}/cargo` }, + composer: { enabled: true, basePath: '/composer', registryUrl: `${publicUrl}/composer` }, + pypi: { enabled: true, basePath: '/pypi', registryUrl: publicUrl }, + rubygems: { enabled: true, basePath: '/rubygems', registryUrl: `${publicUrl}/rubygems` }, }; const registry = new SmartRegistry(config); await registry.init(); -// Handle any incoming HTTP request โ€” the router does the rest -const response = await registry.handleRequest({ - method: 'GET', - path: '/npm/express', - headers: {}, - query: {}, -}); +const auth = registry.getAuthManager(); +const npmToken = await auth.createNpmToken('ci-bot'); +const cargoToken = await auth.createCargoToken('ci-bot'); +const ociToken = await auth.createOciToken('ci-bot', ['oci:repository:myorg/myapp:*']); + +console.log({ npmToken, cargoToken, ociToken }); ``` -## ๐Ÿ›๏ธ Architecture +If you do not pass `authProvider`, the library uses `DefaultAuthProvider`, an in-memory reference implementation. That is perfect for tests and local dev, but you will usually want a real `IAuthProvider` in production. -### Request Flow +## HTTP Integration -``` -HTTP Request - โ†“ -SmartRegistry (orchestrator) - โ†“ -Path-based routing - โ”œโ”€โ†’ /oci/* โ†’ OciRegistry - โ”œโ”€โ†’ /npm/* โ†’ NpmRegistry - โ”œโ”€โ†’ /maven/* โ†’ MavenRegistry - โ”œโ”€โ†’ /cargo/* โ†’ CargoRegistry - โ”œโ”€โ†’ /composer/* โ†’ ComposerRegistry - โ”œโ”€โ†’ /pypi/* โ†’ PypiRegistry - โ””โ”€โ†’ /rubygems/* โ†’ RubyGemsRegistry - โ†“ - Shared Storage & Auth - โ†“ - S3-compatible backend -``` +`SmartRegistry` is not a web framework. You own the HTTP server, request parsing, and response writing. The happy path is very small: -### Directory Structure +```ts +import { createServer, type IncomingHttpHeaders } from 'node:http'; +import { Readable } from 'node:stream'; -``` -ts/ -โ”œโ”€โ”€ core/ # Shared infrastructure -โ”‚ โ”œโ”€โ”€ classes.baseregistry.ts -โ”‚ โ”œโ”€โ”€ classes.registrystorage.ts -โ”‚ โ”œโ”€โ”€ classes.authmanager.ts -โ”‚ โ””โ”€โ”€ interfaces.core.ts -โ”œโ”€โ”€ oci/ # OCI implementation -โ”œโ”€โ”€ npm/ # NPM implementation -โ”œโ”€โ”€ maven/ # Maven implementation -โ”œโ”€โ”€ cargo/ # Cargo implementation -โ”œโ”€โ”€ composer/ # Composer implementation -โ”œโ”€โ”€ pypi/ # PyPI implementation -โ”œโ”€โ”€ rubygems/ # RubyGems implementation -โ”œโ”€โ”€ upstream/ # Upstream proxy infrastructure -โ””โ”€โ”€ classes.smartregistry.ts # Main orchestrator -``` - -## ๐Ÿ’ก Usage Examples - -### ๐Ÿณ OCI Registry (Container Images) - -```typescript -// Pull a manifest -const response = await registry.handleRequest({ - method: 'GET', - path: '/oci/library/nginx/manifests/latest', - headers: { 'Authorization': 'Bearer ' }, - query: {}, -}); - -// Push a blob (two-step upload) -const uploadInit = await registry.handleRequest({ - method: 'POST', - path: '/oci/myapp/blobs/uploads/', - headers: { 'Authorization': 'Bearer ' }, - query: {}, -}); - -const uploadId = uploadInit.headers['Docker-Upload-UUID']; - -await registry.handleRequest({ - method: 'PUT', - path: `/oci/myapp/blobs/uploads/${uploadId}`, - headers: { 'Authorization': 'Bearer ' }, - query: { digest: 'sha256:abc123...' }, - body: blobData, -}); -``` - -### ๐Ÿ“ฆ NPM Registry - -```typescript -// Get package metadata -const metadata = await registry.handleRequest({ - method: 'GET', - path: '/npm/express', - headers: {}, - query: {}, -}); - -// Publish a package -const publishResponse = await registry.handleRequest({ - method: 'PUT', - path: '/npm/my-package', - headers: { 'Authorization': 'Bearer ' }, - query: {}, - body: { - name: 'my-package', - versions: { '1.0.0': { /* version metadata */ } }, - 'dist-tags': { latest: '1.0.0' }, - _attachments: { - 'my-package-1.0.0.tgz': { - content_type: 'application/octet-stream', - data: '', - length: 12345, - }, - }, - }, -}); - -// Search packages -const search = await registry.handleRequest({ - method: 'GET', - path: '/npm/-/v1/search', - headers: {}, - query: { text: 'express', size: '20' }, -}); -``` - -Scoped package requests are supported with both encoded and unencoded paths, for example -`/npm/@scope%2fpackage` and `/npm/@scope/package`. - -### ๐Ÿฆ€ Cargo Registry (Rust Crates) - -```typescript -// Get registry config (required for Cargo sparse protocol) -const config = await registry.handleRequest({ - method: 'GET', - path: '/cargo/config.json', - headers: {}, - query: {}, -}); - -// Publish a crate (binary format: [4 bytes JSON len][JSON][4 bytes crate len][.crate]) -const publishResponse = await registry.handleRequest({ - method: 'PUT', - path: '/cargo/api/v1/crates/new', - headers: { 'Authorization': '' }, - query: {}, - body: binaryPublishData, -}); - -// Yank a version -await registry.handleRequest({ - method: 'DELETE', - path: '/cargo/api/v1/crates/my-crate/0.1.0/yank', - headers: { 'Authorization': '' }, - query: {}, -}); -``` - -**Using with Cargo CLI:** - -```toml -# .cargo/config.toml -[registries.myregistry] -index = "sparse+https://registry.example.com/cargo/" -``` - -```bash -cargo publish --registry=myregistry -cargo install --registry=myregistry my-crate -``` - -### ๐ŸŽผ Composer Registry (PHP Packages) - -```typescript -// Get repository root -const packagesJson = await registry.handleRequest({ - method: 'GET', - path: '/composer/packages.json', - headers: {}, - query: {}, -}); - -// Upload a package (ZIP with composer.json inside) -const uploadResponse = await registry.handleRequest({ - method: 'PUT', - path: '/composer/packages/vendor/package', - headers: { 'Authorization': 'Bearer ' }, - query: {}, - body: zipBuffer, -}); -``` - -**Using with Composer CLI:** - -```json -{ - "repositories": [ - { "type": "composer", "url": "https://registry.example.com/composer" } - ] +function headersToRecord(headers: IncomingHttpHeaders): Record { + return Object.fromEntries( + Object.entries(headers).map(([key, value]) => [ + key, + Array.isArray(value) ? value.join(', ') : value ?? '', + ]) + ); } + +createServer(async (req, res) => { + const url = new URL(req.url ?? '/', 'http://localhost'); + const headers = headersToRecord(req.headers); + + const chunks: Buffer[] = []; + for await (const chunk of req) { + chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); + } + const rawBody = Buffer.concat(chunks); + + let body: unknown = rawBody.length ? rawBody : undefined; + if ((headers['content-type'] ?? '').includes('application/json') && rawBody.length) { + body = JSON.parse(rawBody.toString('utf8')); + } + + const response = await registry.handleRequest({ + method: req.method ?? 'GET', + path: url.pathname, + query: Object.fromEntries(url.searchParams), + headers, + body, + rawBody: rawBody.length ? rawBody : undefined, + }); + + res.writeHead(response.status, response.headers); + + if (!response.body) { + res.end(); + return; + } + + Readable.fromWeb(response.body).pipe(res); +}).listen(3000); ``` -```bash -composer require vendor/package -``` +Keep `rawBody` for OCI manifests, OCI blobs, and any other digest-sensitive request where exact bytes matter. -### ๐Ÿ PyPI Registry (Python Packages) +For PyPI uploads, parse `multipart/form-data` before calling `handleRequest()` and pass the parsed fields in `context.body`. The library expects the upload form fields, not a raw multipart buffer. -```typescript -// Get package index (PEP 503 HTML) -const htmlIndex = await registry.handleRequest({ - method: 'GET', - path: '/simple/requests/', - headers: { 'Accept': 'text/html' }, - query: {}, -}); +At the public API boundary, `response.body` is always a `ReadableStream`. -// Get package index (PEP 691 JSON) -const jsonIndex = await registry.handleRequest({ - method: 'GET', - path: '/simple/requests/', - headers: { 'Accept': 'application/vnd.pypi.simple.v1+json' }, - query: {}, -}); +## Core API -// Upload a package -const upload = await registry.handleRequest({ - method: 'POST', - path: '/pypi/', - headers: { - 'Authorization': 'Bearer ', - 'Content-Type': 'multipart/form-data', - }, - query: {}, - body: { - ':action': 'file_upload', - protocol_version: '1', - name: 'my-package', - version: '1.0.0', - filetype: 'bdist_wheel', - content: wheelData, - filename: 'my_package-1.0.0-py3-none-any.whl', - }, -}); -``` +| API | Why you use it | +| --- | --- | +| `new SmartRegistry(config)` | build the registry orchestrator | +| `await registry.init()` | initialize storage, auth, and enabled protocols | +| `await registry.handleRequest(context)` | route one incoming HTTP request | +| `registry.getAuthManager()` | mint, validate, revoke, and authorize tokens | +| `registry.getStorage()` | reach the shared storage abstraction directly | +| `registry.getRegistry(protocol)` | access a specific protocol handler | +| `registry.destroy()` | clean up timers and protocol resources | -**Using with pip:** +The package also exports the protocol-specific registry classes, upstream classes, `RegistryStorage`, `AuthManager`, `DefaultAuthProvider`, `StaticUpstreamProvider`, `UpstreamCache`, `CircuitBreaker`, and stream helpers such as `streamToBuffer()` and `streamToJson()`. -```bash -pip install --index-url https://registry.example.com/simple/ my-package -python -m twine upload --repository-url https://registry.example.com/pypi/ dist/* -``` +## Configuration Reference -### ๐Ÿ’Ž RubyGems Registry +| Key | Purpose | +| --- | --- | +| `storage` | S3-compatible backend config. This extends `IS3Descriptor` and adds `bucketName`. | +| `auth` | shared token settings across protocols | +| `authProvider` | plug in LDAP, OAuth, OIDC, custom DB-backed auth, or anything else implementing `IAuthProvider` | +| `storageHooks` | receive before/after put/get/delete callbacks with protocol, actor, package, and version context | +| `upstreamProvider` | decide per request which upstream registries to consult | +| `oci` / `npm` / `maven` / `cargo` / `composer` / `pypi` / `rubygems` | enable protocols, set base paths, and define the public registry URL they should emit | -```typescript -// Upload a gem -const uploadGem = await registry.handleRequest({ - method: 'POST', - path: '/rubygems/api/v1/gems', - headers: { 'Authorization': '' }, - query: {}, - body: gemBuffer, -}); +## Upstream Proxying -// Get compact index -const versions = await registry.handleRequest({ - method: 'GET', - path: '/rubygems/versions', - headers: {}, - query: {}, -}); -``` +If a package, image, crate, or artifact does not exist locally, a protocol handler can resolve an upstream config on the fly and fetch it from there. -**Using with Bundler:** - -```ruby -# Gemfile -source 'https://registry.example.com/rubygems' do - gem 'my-gem' -end -``` - -```bash -gem push my-gem-1.0.0.gem --host https://registry.example.com/rubygems -bundle install -``` - -### ๐Ÿ” Authentication - -```typescript -const authManager = registry.getAuthManager(); - -// Authenticate user -const userId = await authManager.authenticate({ username: 'user', password: 'pass' }); - -// Create protocol-specific tokens -const npmToken = await authManager.createNpmToken(userId, false); -const ociToken = await authManager.createOciToken(userId, ['oci:repository:myapp:push'], 3600); -const pypiToken = await authManager.createPypiToken(userId, false); -const cargoToken = await authManager.createCargoToken(userId, false); -const composerToken = await authManager.createComposerToken(userId, false); -const rubygemsToken = await authManager.createRubyGemsToken(userId, false); - -// Validate and check permissions -const token = await authManager.validateToken(npmToken, 'npm'); -const canWrite = await authManager.authorize(token, 'npm:package:my-package', 'write'); -``` - -### ๐ŸŒ Upstream Proxy Configuration - -```typescript -import { SmartRegistry, StaticUpstreamProvider } from '@push.rocks/smartregistry'; +```ts +import { + StaticUpstreamProvider, +} from '@push.rocks/smartregistry'; const upstreamProvider = new StaticUpstreamProvider({ npm: { enabled: true, upstreams: [ { - id: 'company-private', - url: 'https://npm.internal.company.com', + id: 'npmjs', + name: 'npmjs', + url: 'https://registry.npmjs.org', priority: 1, enabled: true, - scopeRules: [{ pattern: '@company/*', action: 'include' }], - auth: { type: 'bearer', token: process.env.NPM_PRIVATE_TOKEN }, - }, - { - id: 'npmjs', - url: 'https://registry.npmjs.org', - priority: 10, - enabled: true, - scopeRules: [{ pattern: '@company/*', action: 'exclude' }], + auth: { type: 'none' }, }, ], - cache: { enabled: true, staleWhileRevalidate: true }, }, oci: { enabled: true, upstreams: [ - { id: 'dockerhub', url: 'https://registry-1.docker.io', priority: 1, enabled: true }, + { + id: 'dockerhub', + name: 'dockerhub', + url: 'https://registry-1.docker.io', + priority: 1, + enabled: true, + auth: { type: 'none' }, + }, ], }, }); - -const registry = new SmartRegistry({ - storage: { /* S3 config */ }, - auth: { /* Auth config */ }, - upstreamProvider, - npm: { enabled: true, basePath: '/npm' }, - oci: { enabled: true, basePath: '/oci' }, -}); ``` -### ๐Ÿ”Œ Custom Auth Provider +Pass that provider as `upstreamProvider` in your `IRegistryConfig`. -```typescript -import { SmartRegistry, IAuthProvider, IAuthToken, TRegistryProtocol } from '@push.rocks/smartregistry'; +The upstream layer supports scope rules, per-request routing, retries with backoff, circuit breakers, stale-while-revalidate caching, and negative caching for 404s. -class LdapAuthProvider implements IAuthProvider { - async init() { /* connect to LDAP */ } +## Custom Auth and Audit Hooks - async authenticate(credentials) { - const result = await this.ldapClient.bind(credentials.username, credentials.password); - return result.success ? credentials.username : null; - } - - async validateToken(token: string, protocol?: TRegistryProtocol): Promise { - const session = await this.sessionStore.get(token); - return session ? { userId: session.userId, scopes: session.scopes } : null; - } - - async createToken(userId: string, protocol: TRegistryProtocol, options?) { - const token = crypto.randomUUID(); - await this.sessionStore.set(token, { userId, protocol, ...options }); - return token; - } - - async revokeToken(token: string) { await this.sessionStore.delete(token); } - - async authorize(token: IAuthToken | null, resource: string, action: string) { - if (!token) return action === 'read'; - return this.checkPermissions(token.userId, resource, action); - } -} +Bring your own auth system by implementing `IAuthProvider` and passing it as `authProvider`. +```ts const registry = new SmartRegistry({ ...config, - authProvider: new LdapAuthProvider(), + authProvider: myAuthProvider, }); ``` -### ๐Ÿ“Š Storage Hooks (Quota & Audit) - -```typescript -import { SmartRegistry, IStorageHooks, IStorageHookContext } from '@push.rocks/smartregistry'; - -const storageHooks: IStorageHooks = { - async beforePut(ctx: IStorageHookContext) { - if (ctx.actor?.orgId) { - const usage = await getStorageUsage(ctx.actor.orgId); - const quota = await getQuota(ctx.actor.orgId); - if (usage + (ctx.metadata?.size || 0) > quota) { - return { allowed: false, reason: 'Storage quota exceeded' }; - } - } - return { allowed: true }; - }, - - async afterPut(ctx: IStorageHookContext) { - await auditLog.write({ - action: 'storage.put', - key: ctx.key, - protocol: ctx.protocol, - actor: ctx.actor, - timestamp: ctx.timestamp, - }); - }, - - async beforeDelete(ctx: IStorageHookContext) { - if (await isProtectedPackage(ctx.key)) { - return { allowed: false, reason: 'Cannot delete protected package' }; - } - return { allowed: true }; - }, -}; - -const registry = new SmartRegistry({ ...config, storageHooks }); -``` - -### ๐Ÿ‘ค Request Actor Context - -```typescript -// Pass actor information for audit/quota tracking -const response = await registry.handleRequest({ - method: 'PUT', - path: '/npm/my-package', - headers: { 'Authorization': 'Bearer ' }, - query: {}, - body: packageData, - actor: { - userId: 'user123', - tokenId: 'token-abc', - ip: req.ip, - userAgent: req.headers['user-agent'], - orgId: 'org-456', - }, -}); -``` - -## โš™๏ธ Configuration - -### Storage Configuration - -Extends `IS3Descriptor` from `@tsclass/tsclass`: - -```typescript -storage: { - accessKey: string; // S3 access key - accessSecret: string; // S3 secret key - endpoint: string; // S3 endpoint (e.g., 's3.amazonaws.com') - port?: number; // Default: 443 - useSsl?: boolean; // Default: true - region?: string; // AWS region - bucketName: string; // Bucket name for registry storage -} -``` - -### Authentication Configuration - -```typescript -auth: { - jwtSecret: string; - tokenStore: 'memory' | 'redis' | 'database'; - npmTokens: { enabled: boolean; defaultReadonly?: boolean }; - ociTokens: { enabled: boolean; realm: string; service: string }; - pypiTokens: { enabled: boolean }; - rubygemsTokens: { enabled: boolean }; -} -``` - -### Protocol Configuration - -Each protocol accepts: - -```typescript -{ - enabled: boolean; - basePath: string; // URL prefix, e.g. '/npm' - registryUrl?: string; // Public-facing base URL (used in generated metadata links) - features?: Record; -} -``` - -The `registryUrl` is important when the registry is served behind a reverse proxy or on a non-default port. For example, if your server is at `https://registry.example.com`, set `registryUrl: 'https://registry.example.com/npm'` for the NPM protocol so that generated metadata URLs point to the correct host. - -## ๐Ÿ“š API Reference - -### Core Classes - -#### SmartRegistry - -Main orchestrator โ€” routes requests to the appropriate protocol handler. - -| Method | Description | -|--------|-------------| -| `init()` | Initialize the registry and all enabled protocols | -| `handleRequest(context)` | Route and handle an HTTP request | -| `getStorage()` | Get the shared `RegistryStorage` instance | -| `getAuthManager()` | Get the shared `AuthManager` instance | -| `getRegistry(protocol)` | Get a specific protocol handler by name | -| `isInitialized()` | Check if the registry has been initialized | -| `destroy()` | Clean up resources | - -### Protocol Endpoints - -#### OCI Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/{name}/manifests/{ref}` | Get manifest by tag or digest | -| `PUT` | `/{name}/manifests/{ref}` | Push manifest | -| `GET` | `/{name}/blobs/{digest}` | Get blob | -| `POST` | `/{name}/blobs/uploads/` | Initiate blob upload | -| `PUT` | `/{name}/blobs/uploads/{uuid}` | Complete blob upload | -| `GET` | `/{name}/tags/list` | List tags | -| `GET` | `/{name}/referrers/{digest}` | Get referrers (OCI 1.1) | - -#### NPM Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/{package}` | Get package metadata (packument) | -| `PUT` | `/{package}` | Publish package | -| `GET` | `/{package}/-/{tarball}` | Download tarball | -| `GET` | `/-/v1/search?text=...` | Search packages | -| `PUT` | `/-/user/org.couchdb.user:{user}` | Login | -| `GET/POST/DELETE` | `/-/npm/v1/tokens` | Token management | -| `PUT` | `/-/package/{pkg}/dist-tags/{tag}` | Manage dist-tags | - -#### Maven Repository - -| Method | Path | Description | -|--------|------|-------------| -| `PUT` | `/{group}/{artifact}/{version}/{file}` | Upload artifact | -| `GET` | `/{group}/{artifact}/{version}/{file}` | Download artifact | -| `GET` | `/{group}/{artifact}/maven-metadata.xml` | Get metadata | - -#### Cargo Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/config.json` | Registry configuration | -| `GET` | `/{p1}/{p2}/{name}` | Sparse index entry | -| `PUT` | `/api/v1/crates/new` | Publish crate (binary format) | -| `GET` | `/api/v1/crates/{crate}/{version}/download` | Download .crate | -| `DELETE` | `/api/v1/crates/{crate}/{version}/yank` | Yank version | -| `PUT` | `/api/v1/crates/{crate}/{version}/unyank` | Unyank version | -| `GET` | `/api/v1/crates?q=...` | Search crates | - -#### Composer Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/packages.json` | Repository metadata | -| `GET` | `/p2/{vendor}/{package}.json` | Package version metadata | -| `GET` | `/packages/list.json` | List all packages | -| `GET` | `/dists/{vendor}/{package}/{ref}.zip` | Download package ZIP | -| `PUT` | `/packages/{vendor}/{package}` | Upload package | -| `DELETE` | `/packages/{vendor}/{package}[/{version}]` | Delete package/version | - -#### PyPI Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/simple/` | List all packages (PEP 503/691) | -| `GET` | `/simple/{package}/` | List package files | -| `POST` | `/` | Upload package (multipart) | -| `GET` | `/pypi/{package}/json` | Package metadata API | -| `GET` | `/pypi/{package}/{version}/json` | Version metadata | -| `GET` | `/packages/{package}/{filename}` | Download file | - -#### RubyGems Registry - -| Method | Path | Description | -|--------|------|-------------| -| `GET` | `/versions` | Master versions file (compact index) | -| `GET` | `/info/{gem}` | Gem info file | -| `GET` | `/names` | List all gem names | -| `POST` | `/api/v1/gems` | Upload .gem file | -| `DELETE` | `/api/v1/gems/yank` | Yank version | -| `PUT` | `/api/v1/gems/unyank` | Unyank version | -| `GET` | `/api/v1/versions/{gem}.json` | Version metadata | -| `GET` | `/gems/{gem}-{version}.gem` | Download .gem file | - -## ๐ŸŽฏ Scope Format - -Unified scope format across all protocols: - -``` -{protocol}:{type}:{name}:{action} - -Examples: - npm:package:express:read # Read express package - npm:package:*:write # Write any package - oci:repository:nginx:pull # Pull nginx image - oci:repository:*:push # Push any image - cargo:crate:serde:write # Write serde crate - composer:package:vendor/pkg:read # Read Composer package - pypi:package:requests:read # Read PyPI package - rubygems:gem:rails:write # Write RubyGems gem - {protocol}:*:*:* # Full access for a protocol -``` - -## ๐Ÿ—„๏ธ Storage Structure - -``` -bucket/ -โ”œโ”€โ”€ oci/ -โ”‚ โ”œโ”€โ”€ blobs/sha256/{hash} -โ”‚ โ”œโ”€โ”€ manifests/{repository}/{digest} -โ”‚ โ””โ”€โ”€ tags/{repository}/tags.json -โ”œโ”€โ”€ npm/ -โ”‚ โ””โ”€โ”€ packages/{name}/ -โ”‚ โ”œโ”€โ”€ index.json # Packument -โ”‚ โ””โ”€โ”€ {name}-{ver}.tgz # Tarball -โ”œโ”€โ”€ maven/ -โ”‚ โ”œโ”€โ”€ artifacts/{group}/{artifact}/{version}/ -โ”‚ โ””โ”€โ”€ metadata/{group}/{artifact}/maven-metadata.xml -โ”œโ”€โ”€ cargo/ -โ”‚ โ”œโ”€โ”€ config.json -โ”‚ โ”œโ”€โ”€ index/{p1}/{p2}/{name} # Sparse index -โ”‚ โ””โ”€โ”€ crates/{name}/{name}-{ver}.crate -โ”œโ”€โ”€ composer/ -โ”‚ โ””โ”€โ”€ packages/{vendor}/{package}/ -โ”‚ โ”œโ”€โ”€ metadata.json -โ”‚ โ””โ”€โ”€ {reference}.zip -โ”œโ”€โ”€ pypi/ -โ”‚ โ”œโ”€โ”€ simple/index.html -โ”‚ โ”œโ”€โ”€ simple/{package}/index.html -โ”‚ โ”œโ”€โ”€ packages/{package}/{filename} -โ”‚ โ””โ”€โ”€ metadata/{package}/metadata.json -โ””โ”€โ”€ rubygems/ - โ”œโ”€โ”€ versions - โ”œโ”€โ”€ info/{gemname} - โ”œโ”€โ”€ names - โ””โ”€โ”€ gems/{gemname}-{version}.gem -``` - -## ๐ŸŒŠ Streaming Architecture - -All responses from `SmartRegistry.handleRequest()` use the **Web Streams API**. The `body` field on `IResponse` is always a `ReadableStream` โ€” whether the content is a 2GB container image layer or a tiny JSON metadata response. - -### How It Works - -- **Binary downloads** (blobs, tarballs, .crate, .zip, .whl, .gem) stream directly from S3 to the response โ€” zero buffering in memory -- **JSON/metadata responses** are automatically wrapped into a `ReadableStream` at the API boundary -- **OCI chunked uploads** store each PATCH chunk as a temp S3 object instead of accumulating in memory, then stream-assemble during the final PUT with incremental SHA-256 verification - -### Stream Helpers - -```typescript -import { streamToBuffer, streamToJson, toReadableStream } from '@push.rocks/smartregistry'; - -// Consume a stream into a Buffer -const buffer = await streamToBuffer(response.body); - -// Consume a stream into parsed JSON -const data = await streamToJson(response.body); - -// Create a ReadableStream from any data type -const stream = toReadableStream({ hello: 'world' }); -``` - -### Consuming in Node.js HTTP Servers - -Since Node.js `http.ServerResponse` uses Node streams, bridge with `Readable.fromWeb()`: - -```typescript -import { Readable } from 'stream'; - -if (response.body) { - Readable.fromWeb(response.body).pipe(res); -} else { - res.end(); -} -``` - -## ๐Ÿ”Œ Integration with Express - -```typescript -import express from 'express'; -import { Readable } from 'stream'; -import { SmartRegistry } from '@push.rocks/smartregistry'; - -const app = express(); -const registry = new SmartRegistry(config); -await registry.init(); - -app.all('*', async (req, res) => { - const response = await registry.handleRequest({ - method: req.method, - path: req.path, - headers: req.headers as Record, - query: req.query as Record, - body: req.body, - }); - - res.status(response.status); - for (const [key, value] of Object.entries(response.headers)) { - res.setHeader(key, value); - } - - if (response.body) { - // All response bodies are ReadableStream โ€” pipe to HTTP response - Readable.fromWeb(response.body).pipe(res); - } else { - res.end(); - } -}); - -app.listen(5000); -``` - -## ๐Ÿงช Testing with smartstorage - -smartregistry works seamlessly with [@push.rocks/smartstorage](https://code.foss.global/push.rocks/smartstorage), a local S3-compatible server for testing โ€” no cloud credentials needed. - -```typescript -import { SmartStorage } from '@push.rocks/smartstorage'; -import { SmartRegistry } from '@push.rocks/smartregistry'; - -// Start local S3 server -const s3Server = await SmartStorage.createAndStart({ - server: { port: 3456, silent: true }, - storage: { cleanSlate: true }, -}); - -// Get S3 descriptor from the running server -const s3Descriptor = await s3Server.getStorageDescriptor(); +Use `storageHooks` when you need quota checks, audit logs, or side effects around artifact writes and deletes. +```ts const registry = new SmartRegistry({ - storage: { ...s3Descriptor, bucketName: 'my-test-registry' }, - auth: { jwtSecret: 'test', tokenStore: 'memory', npmTokens: { enabled: true } }, - npm: { enabled: true, basePath: '/npm' }, - oci: { enabled: true, basePath: '/oci' }, + ...config, + storageHooks: { + async beforePut(context) { + if ((context.metadata?.size ?? 0) > 500 * 1024 * 1024) { + return { allowed: false, reason: 'Artifact too large' }; + } + return { allowed: true }; + }, + async afterPut(context) { + await auditLog('storage.put', context); + }, + }, }); -await registry.init(); - -// ... run your tests ... -await s3Server.stop(); ``` -## ๐Ÿ› ๏ธ Development +`handleRequest()` also accepts an `actor` object. That extra context flows into storage hooks and upstream resolution, which is great for multi-tenant routing, org-aware policy checks, and audit trails. + +```ts +await registry.handleRequest({ + method: 'GET', + path: '/npm/@acme/internal-lib', + headers: { authorization: `Bearer ${npmToken}` }, + query: {}, + actor: { + orgId: 'acme', + sessionId: 'sess_123', + }, +}); +``` + +## Client Cheatsheet + +### npm + +```ini +registry=https://registry.example.com/npm/ +//registry.example.com/npm/:_authToken= +``` + +The npm handler also implements the npm-compatible login and token endpoints, including `PUT /-/user/org.couchdb.user:` and `/-/npm/v1/tokens`. + +### Docker / OCI + +Set `oci.basePath` to `/v2` and expose a token endpoint that returns `token`, `access_token`, and `expires_in`. + +```ts +if (url.pathname === '/v2/token') { + const token = await registry.getAuthManager().createOciToken( + 'docker-user', + ['oci:repository:*:*'], + 3600, + ); + + res.setHeader('Content-Type', 'application/json'); + res.end(JSON.stringify({ + token, + access_token: token, + expires_in: 3600, + })); + return; +} +``` ```bash -pnpm install # Install dependencies -pnpm run build # Build -pnpm test # Run all tests +docker login registry.example.com +docker push registry.example.com/myorg/myimage:latest +docker pull registry.example.com/myorg/myimage:latest ``` +### Maven + +```xml + + + + smartregistry + token + YOUR_MAVEN_TOKEN + + + +``` + +Point repositories or distribution management at `https://registry.example.com/maven`. + +### Cargo + +```toml +# .cargo/config.toml +[registries.smartregistry] +index = "sparse+https://registry.example.com/cargo/" +``` + +```toml +# .cargo/credentials.toml +[registries.smartregistry] +token = "YOUR_CARGO_TOKEN" +``` + +Cargo sends the token as a plain `Authorization` header, not `Bearer `. + +### Composer + +```json +{ + "repositories": [ + { + "type": "composer", + "url": "https://registry.example.com/composer" + } + ] +} +``` + +```json +{ + "http-basic": { + "registry.example.com": { + "username": "my-user", + "password": "my-password" + } + } +} +``` + +Composer installs can use Basic auth if your `authProvider.authenticate()` supports it. Programmatic writes use Bearer tokens cleanly. + +### PyPI + +```ini +[distutils] +index-servers = smartregistry + +[smartregistry] +repository = https://registry.example.com/pypi +username = __token__ +password = YOUR_PYPI_TOKEN +``` + +```bash +pip install --index-url https://registry.example.com/simple/ your-package +twine upload --repository smartregistry dist/* +``` + +### RubyGems + +```yaml +:rubygems_api_key: YOUR_RUBYGEMS_TOKEN +``` + +```bash +gem push your-gem-1.0.0.gem --host https://registry.example.com/rubygems +bundle config set --global https://registry.example.com/rubygems YOUR_RUBYGEMS_TOKEN +``` + +## Testing and Compatibility + +The repository contains protocol-level tests, cross-protocol integration tests, upstream provider tests, storage hook tests, and native-client test suites that exercise the library through real ecosystem tooling. + +Native-client coverage exists for: + +- Docker / OCI +- npm +- Maven (`mvn`) +- Cargo +- Composer +- PyPI (`pip` and `twine`) +- RubyGems (`gem`) + +There are also integration tests for S3-compatible storage and `IS3Descriptor`-based configuration. + ## License and Legal Information -This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file. +This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./license) file. **Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file. @@ -883,7 +415,7 @@ Use of these trademarks must comply with Task Venture Capital GmbH's Trademark G ### Company Information -Task Venture Capital GmbH +Task Venture Capital GmbH Registered at District Court Bremen HRB 35230 HB, Germany For any legal inquiries or further information, please contact us via email at hello@task.vc. diff --git a/ts/00_commitinfo_data.ts b/ts/00_commitinfo_data.ts index 349abe4..8b47608 100644 --- a/ts/00_commitinfo_data.ts +++ b/ts/00_commitinfo_data.ts @@ -3,6 +3,6 @@ */ export const commitinfo = { name: '@push.rocks/smartregistry', - version: '2.9.0', + version: '2.9.1', description: 'A composable TypeScript library implementing OCI, NPM, Maven, Cargo, Composer, PyPI, and RubyGems registries for building unified container and package registries' }