Files
smartregistry/readme.md

761 lines
20 KiB
Markdown

# @push.rocks/smartregistry
> 🚀 A composable TypeScript library implementing **OCI Distribution Specification v1.1**, **NPM Registry API**, **Maven Repository**, **Cargo/crates.io Registry**, and **Composer/Packagist** for building unified container and package registries.
## ✨ Features
### 🔄 Multi-Protocol Support
- **OCI Distribution Spec v1.1**: Full container registry with manifest/blob operations
- **NPM Registry API**: Complete package registry with publish/install/search
- **Maven Repository**: Java/JVM artifact management with POM support
- **Cargo/crates.io Registry**: Rust crate registry with sparse HTTP protocol
- **Composer/Packagist**: PHP package registry with Composer v2 protocol
### 🏗️ Unified Architecture
- **Composable Design**: Core infrastructure with protocol plugins
- **Shared Storage**: Cloud-agnostic S3-compatible backend ([@push.rocks/smartbucket](https://www.npmjs.com/package/@push.rocks/smartbucket))
- **Unified Authentication**: Scope-based permissions across all protocols
- **Path-based Routing**: `/oci/*` for containers, `/npm/*` for packages, `/maven/*` for Java artifacts, `/cargo/*` for Rust crates, `/composer/*` for PHP packages
### 🔐 Authentication & Authorization
- NPM UUID tokens for package operations
- OCI JWT tokens for container operations
- Unified scope system: `npm:package:foo:write`, `oci:repository:bar:push`
- Pluggable via async callbacks
### 📦 Comprehensive Feature Set
**OCI Features:**
- ✅ Pull operations (manifests, blobs)
- ✅ Push operations (chunked uploads)
- ✅ Content discovery (tags, referrers API)
- ✅ Content management (deletion)
**NPM Features:**
- ✅ Package publish/unpublish
- ✅ Package download (tarballs)
- ✅ Metadata & search
- ✅ Dist-tag management
- ✅ Token management
**Maven Features:**
- ✅ Artifact upload/download
- ✅ POM and metadata management
- ✅ Snapshot and release versions
- ✅ Checksum verification (MD5, SHA1)
**Cargo Features:**
- ✅ Crate publish (.crate files)
- ✅ Sparse HTTP protocol (modern index)
- ✅ Version yank/unyank
- ✅ Dependency resolution
- ✅ Search functionality
**Composer Features:**
- ✅ Package publish/download (ZIP format)
- ✅ Composer v2 repository API
- ✅ Package metadata (packages.json)
- ✅ Version management
- ✅ Dependency resolution
- ✅ PSR-4/PSR-0 autoloading support
## 📥 Installation
```bash
# Using npm
npm install @push.rocks/smartregistry
# Using pnpm (recommended)
pnpm add @push.rocks/smartregistry
```
## 🚀 Quick Start
```typescript
import { SmartRegistry, IRegistryConfig } from '@push.rocks/smartregistry';
const config: IRegistryConfig = {
storage: {
accessKey: 'your-s3-key',
accessSecret: 'your-s3-secret',
endpoint: 's3.amazonaws.com',
port: 443,
useSsl: true,
region: 'us-east-1',
bucketName: 'my-registry',
},
auth: {
jwtSecret: 'your-secret-key',
tokenStore: 'memory',
npmTokens: { enabled: true },
ociTokens: {
enabled: true,
realm: 'https://auth.example.com/token',
service: 'my-registry',
},
},
oci: {
enabled: true,
basePath: '/oci',
},
npm: {
enabled: true,
basePath: '/npm',
},
maven: {
enabled: true,
basePath: '/maven',
},
cargo: {
enabled: true,
basePath: '/cargo',
},
composer: {
enabled: true,
basePath: '/composer',
},
};
const registry = new SmartRegistry(config);
await registry.init();
// Handle requests
const response = await registry.handleRequest({
method: 'GET',
path: '/npm/express',
headers: {},
query: {},
});
```
## 🏛️ Architecture
### Directory Structure
```
ts/
├── core/ # Shared infrastructure
│ ├── classes.baseregistry.ts
│ ├── classes.registrystorage.ts
│ ├── classes.authmanager.ts
│ └── interfaces.core.ts
├── oci/ # OCI implementation
│ ├── classes.ociregistry.ts
│ └── interfaces.oci.ts
├── npm/ # NPM implementation
│ ├── classes.npmregistry.ts
│ └── interfaces.npm.ts
└── classes.smartregistry.ts # Main orchestrator
```
### Request Flow
```
HTTP Request
SmartRegistry (orchestrator)
Path-based routing
├─→ /oci/* → OciRegistry
└─→ /npm/* → NpmRegistry
Shared Storage & Auth
S3-compatible backend
```
## 💡 Usage Examples
### 🐳 OCI Registry (Container Images)
```typescript
// Pull an image
const response = await registry.handleRequest({
method: 'GET',
path: '/oci/v2/library/nginx/manifests/latest',
headers: {
'Authorization': 'Bearer <token>',
},
query: {},
});
// Push a blob
const uploadInit = await registry.handleRequest({
method: 'POST',
path: '/oci/v2/myapp/blobs/uploads/',
headers: { 'Authorization': 'Bearer <token>' },
query: {},
});
const uploadId = uploadInit.headers['Docker-Upload-UUID'];
await registry.handleRequest({
method: 'PUT',
path: `/oci/v2/myapp/blobs/uploads/${uploadId}`,
headers: { 'Authorization': 'Bearer <token>' },
query: { digest: 'sha256:abc123...' },
body: blobData,
});
```
### 📦 NPM Registry (Packages)
```typescript
// Install a package (get metadata)
const metadata = await registry.handleRequest({
method: 'GET',
path: '/npm/express',
headers: {},
query: {},
});
// Download tarball
const tarball = await registry.handleRequest({
method: 'GET',
path: '/npm/express/-/express-4.18.0.tgz',
headers: {},
query: {},
});
// Publish a package
const publishResponse = await registry.handleRequest({
method: 'PUT',
path: '/npm/my-package',
headers: { 'Authorization': 'Bearer <npm-token>' },
query: {},
body: {
name: 'my-package',
versions: {
'1.0.0': { /* version metadata */ },
},
'dist-tags': { latest: '1.0.0' },
_attachments: {
'my-package-1.0.0.tgz': {
content_type: 'application/octet-stream',
data: '<base64-tarball>',
length: 12345,
},
},
},
});
// Search packages
const searchResults = await registry.handleRequest({
method: 'GET',
path: '/npm/-/v1/search',
headers: {},
query: { text: 'express', size: '20' },
});
```
### 🦀 Cargo Registry (Rust Crates)
```typescript
// Get config.json (required for Cargo)
const config = await registry.handleRequest({
method: 'GET',
path: '/cargo/config.json',
headers: {},
query: {},
});
// Get index file for a crate
const index = await registry.handleRequest({
method: 'GET',
path: '/cargo/se/rd/serde', // Path based on crate name length
headers: {},
query: {},
});
// Download a crate file
const crateFile = await registry.handleRequest({
method: 'GET',
path: '/cargo/api/v1/crates/serde/1.0.0/download',
headers: {},
query: {},
});
// Publish a crate (binary format: [4 bytes JSON len][JSON][4 bytes crate len][.crate])
const publishResponse = await registry.handleRequest({
method: 'PUT',
path: '/cargo/api/v1/crates/new',
headers: { 'Authorization': '<cargo-token>' }, // No "Bearer" prefix
query: {},
body: binaryPublishData, // Length-prefixed binary format
});
// Yank a version (deprecate without deleting)
const yankResponse = await registry.handleRequest({
method: 'DELETE',
path: '/cargo/api/v1/crates/my-crate/0.1.0/yank',
headers: { 'Authorization': '<cargo-token>' },
query: {},
});
// Unyank a version
const unyankResponse = await registry.handleRequest({
method: 'PUT',
path: '/cargo/api/v1/crates/my-crate/0.1.0/unyank',
headers: { 'Authorization': '<cargo-token>' },
query: {},
});
// Search crates
const search = await registry.handleRequest({
method: 'GET',
path: '/cargo/api/v1/crates',
headers: {},
query: { q: 'serde', per_page: '10' },
});
```
**Using with Cargo CLI:**
```toml
# .cargo/config.toml
[registries.myregistry]
index = "sparse+https://registry.example.com/cargo/"
[registries.myregistry.credential-provider]
# Or use credentials directly:
# [registries.myregistry]
# token = "your-api-token"
```
```bash
# Publish to custom registry
cargo publish --registry=myregistry
# Install from custom registry
cargo install --registry=myregistry my-crate
# Search custom registry
cargo search --registry=myregistry tokio
```
### 🎼 Composer Registry (PHP Packages)
```typescript
// Get repository root (packages.json)
const packagesJson = await registry.handleRequest({
method: 'GET',
path: '/composer/packages.json',
headers: {},
query: {},
});
// Get package metadata
const metadata = await registry.handleRequest({
method: 'GET',
path: '/composer/p2/vendor/package.json',
headers: {},
query: {},
});
// Upload a package (ZIP with composer.json)
const zipBuffer = await readFile('package.zip');
const uploadResponse = await registry.handleRequest({
method: 'PUT',
path: '/composer/packages/vendor/package',
headers: { 'Authorization': `Bearer <composer-token>` },
query: {},
body: zipBuffer,
});
// Download package ZIP
const download = await registry.handleRequest({
method: 'GET',
path: '/composer/dists/vendor/package/ref123.zip',
headers: {},
query: {},
});
// List all packages
const list = await registry.handleRequest({
method: 'GET',
path: '/composer/packages/list.json',
headers: {},
query: {},
});
// Delete a specific version
const deleteVersion = await registry.handleRequest({
method: 'DELETE',
path: '/composer/packages/vendor/package/1.0.0',
headers: { 'Authorization': `Bearer <composer-token>` },
query: {},
});
```
**Using with Composer CLI:**
```json
// composer.json
{
"repositories": [
{
"type": "composer",
"url": "https://registry.example.com/composer"
}
]
}
```
```bash
# Install from custom registry
composer require vendor/package
# Update packages
composer update
```
### 🔐 Authentication
```typescript
// Get auth manager instance
const authManager = registry.getAuthManager();
// Authenticate user
const userId = await authManager.authenticate({
username: 'user',
password: 'pass',
});
// Create NPM token
const npmToken = await authManager.createNpmToken(userId, false);
// Create OCI token with scopes
const ociToken = await authManager.createOciToken(
userId,
['oci:repository:myapp:push', 'oci:repository:myapp:pull'],
3600
);
// Validate any token
const token = await authManager.validateToken(npmToken, 'npm');
// Check permissions
const canWrite = await authManager.authorize(
token,
'npm:package:my-package',
'write'
);
```
## ⚙️ Configuration
### Storage Configuration
```typescript
storage: {
accessKey: string; // S3 access key
accessSecret: string; // S3 secret key
endpoint: string; // S3 endpoint
port?: number; // Default: 443
useSsl?: boolean; // Default: true
region?: string; // Default: 'us-east-1'
bucketName: string; // Bucket name
}
```
### Authentication Configuration
```typescript
auth: {
jwtSecret: string; // Secret for signing JWTs
tokenStore: 'memory' | 'redis' | 'database';
npmTokens: {
enabled: boolean;
defaultReadonly?: boolean;
};
ociTokens: {
enabled: boolean;
realm: string; // Auth server URL
service: string; // Service name
};
}
```
### Protocol Configuration
```typescript
oci?: {
enabled: boolean;
basePath: string; // Default: '/oci'
features?: {
referrers?: boolean;
deletion?: boolean;
};
}
npm?: {
enabled: boolean;
basePath: string; // Default: '/npm'
features?: {
publish?: boolean;
unpublish?: boolean;
search?: boolean;
};
}
```
## 📚 API Reference
### Core Classes
#### SmartRegistry
Main orchestrator class that routes requests to appropriate protocol handlers.
**Methods:**
- `init()` - Initialize the registry
- `handleRequest(context)` - Handle HTTP request
- `getStorage()` - Get storage instance
- `getAuthManager()` - Get auth manager
- `getRegistry(protocol)` - Get protocol handler
#### RegistryStorage
Unified storage abstraction for both OCI and NPM content.
**OCI Methods:**
- `getOciBlob(digest)` - Get blob
- `putOciBlob(digest, data)` - Store blob
- `getOciManifest(repo, digest)` - Get manifest
- `putOciManifest(repo, digest, data, type)` - Store manifest
**NPM Methods:**
- `getNpmPackument(name)` - Get package metadata
- `putNpmPackument(name, data)` - Store package metadata
- `getNpmTarball(name, version)` - Get tarball
- `putNpmTarball(name, version, data)` - Store tarball
#### AuthManager
Unified authentication manager supporting both NPM and OCI authentication schemes.
**Methods:**
- `authenticate(credentials)` - Validate user credentials
- `createNpmToken(userId, readonly)` - Create NPM token
- `createOciToken(userId, scopes, expiresIn)` - Create OCI JWT
- `validateToken(token, protocol)` - Validate any token
- `authorize(token, resource, action)` - Check permissions
### Protocol Handlers
#### OciRegistry
OCI Distribution Specification v1.1 compliant registry.
**Endpoints:**
- `GET /v2/` - Version check
- `GET /v2/{name}/manifests/{ref}` - Get manifest
- `PUT /v2/{name}/manifests/{ref}` - Push manifest
- `GET /v2/{name}/blobs/{digest}` - Get blob
- `POST /v2/{name}/blobs/uploads/` - Initiate upload
- `PUT /v2/{name}/blobs/uploads/{uuid}` - Complete upload
- `GET /v2/{name}/tags/list` - List tags
- `GET /v2/{name}/referrers/{digest}` - Get referrers
#### NpmRegistry
NPM registry API compliant implementation.
**Endpoints:**
- `GET /{package}` - Get package metadata
- `PUT /{package}` - Publish package
- `GET /{package}/-/{tarball}` - Download tarball
- `GET /-/v1/search` - Search packages
- `PUT /-/user/org.couchdb.user:{user}` - Login
- `GET /-/npm/v1/tokens` - List tokens
- `POST /-/npm/v1/tokens` - Create token
- `PUT /-/package/{pkg}/dist-tags/{tag}` - Update tag
#### CargoRegistry
Cargo/crates.io registry with sparse HTTP protocol support.
**Endpoints:**
- `GET /config.json` - Registry configuration (sparse protocol)
- `GET /index/{path}` - Index files (hierarchical structure)
- `/1/{name}` - 1-character crate names
- `/2/{name}` - 2-character crate names
- `/3/{c}/{name}` - 3-character crate names
- `/{p1}/{p2}/{name}` - 4+ character crate names
- `PUT /api/v1/crates/new` - Publish crate (binary format)
- `GET /api/v1/crates/{crate}/{version}/download` - Download .crate file
- `DELETE /api/v1/crates/{crate}/{version}/yank` - Yank (deprecate) version
- `PUT /api/v1/crates/{crate}/{version}/unyank` - Unyank version
- `GET /api/v1/crates?q={query}` - Search crates
**Index Format:**
- Newline-delimited JSON (one line per version)
- SHA256 checksums for .crate files
- Yanked flag (keep files, mark unavailable)
#### ComposerRegistry
Composer v2 repository API compliant implementation.
**Endpoints:**
- `GET /packages.json` - Repository metadata and configuration
- `GET /p2/{vendor}/{package}.json` - Package version metadata
- `GET /p2/{vendor}/{package}~dev.json` - Dev versions metadata
- `GET /packages/list.json` - List all packages
- `GET /dists/{vendor}/{package}/{ref}.zip` - Download package ZIP
- `PUT /packages/{vendor}/{package}` - Upload package (requires auth)
- `DELETE /packages/{vendor}/{package}` - Delete entire package
- `DELETE /packages/{vendor}/{package}/{version}` - Delete specific version
**Package Format:**
- ZIP archives with composer.json in root
- SHA-1 checksums for verification
- Version normalization (1.0.0 → 1.0.0.0)
- PSR-4/PSR-0 autoloading configuration
## 🗄️ Storage Structure
```
bucket/
├── oci/
│ ├── blobs/
│ │ └── sha256/{hash}
│ ├── manifests/
│ │ └── {repository}/{digest}
│ └── tags/
│ └── {repository}/tags.json
├── npm/
│ ├── packages/
│ │ ├── {name}/
│ │ │ ├── index.json # Packument
│ │ │ └── {name}-{ver}.tgz # Tarball
│ │ └── @{scope}/{name}/
│ │ ├── index.json
│ │ └── {name}-{ver}.tgz
│ └── users/
│ └── {username}.json
├── maven/
│ ├── artifacts/
│ │ └── {group-path}/{artifact}/{version}/
│ │ ├── {artifact}-{version}.jar
│ │ ├── {artifact}-{version}.pom
│ │ └── {artifact}-{version}.{ext}
│ └── metadata/
│ └── {group-path}/{artifact}/maven-metadata.xml
├── cargo/
│ ├── config.json # Registry configuration (sparse protocol)
│ ├── index/ # Hierarchical index structure
│ │ ├── 1/{name} # 1-char crate names (e.g., "a")
│ │ ├── 2/{name} # 2-char crate names (e.g., "io")
│ │ ├── 3/{c}/{name} # 3-char crate names (e.g., "3/a/axo")
│ │ └── {p1}/{p2}/{name} # 4+ char (e.g., "se/rd/serde")
│ └── crates/
│ └── {name}/{name}-{version}.crate # Gzipped tar archives
└── composer/
└── packages/
└── {vendor}/{package}/
├── metadata.json # All versions metadata
└── {reference}.zip # Package ZIP files
```
## 🎯 Scope Format
Unified scope format across protocols:
```
{protocol}:{type}:{name}:{action}
Examples:
npm:package:express:read # Read express package
npm:package:*:write # Write any package
npm:*:*:* # Full NPM access
oci:repository:nginx:pull # Pull nginx image
oci:repository:*:push # Push any image
oci:*:*:* # Full OCI access
maven:artifact:com.example:read # Read Maven artifact
maven:artifact:*:write # Write any artifact
maven:*:*:* # Full Maven access
cargo:crate:serde:write # Write serde crate
cargo:crate:*:read # Read any crate
cargo:*:*:* # Full Cargo access
composer:package:vendor/package:read # Read Composer package
composer:package:*:write # Write any package
composer:*:*:* # Full Composer access
```
## 🔌 Integration Examples
### Express Server
```typescript
import express from 'express';
import { SmartRegistry } from '@push.rocks/smartregistry';
const app = express();
const registry = new SmartRegistry(config);
await registry.init();
app.all('*', async (req, res) => {
const response = await registry.handleRequest({
method: req.method,
path: req.path,
headers: req.headers as Record<string, string>,
query: req.query as Record<string, string>,
body: req.body,
});
res.status(response.status);
Object.entries(response.headers).forEach(([key, value]) => {
res.setHeader(key, value);
});
if (response.body) {
if (Buffer.isBuffer(response.body)) {
res.send(response.body);
} else {
res.json(response.body);
}
} else {
res.end();
}
});
app.listen(5000);
```
## 🛠️ Development
```bash
# Install dependencies
pnpm install
# Build
pnpm run build
# Test
pnpm test
```
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
### Trademarks
This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.
### Company Information
Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.