push.rocks/smartregistry
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

multi registry support v3

Browse Source
This commit is contained in:
Jürgen Kunz
2025-11-19 15:32:00 +00:00
parent e4480bff5d
commit 754ec7b7db
19 changed files with 1661 additions and 1740 deletions
Download Patch File Download Diff File Expand all files Collapse all files

516
readme.md
View File

@@ -1,17 +1,39 @@
# @push.rocks/smartregistry
A TypeScript library implementing the OCI Distribution Specification v1.1 for building container and artifact registries.
A composable TypeScript library implementing both OCI Distribution Specification v1.1 and NPM Registry API for building unified container and package registries.
## Features
- **OCI Distribution Spec v1.1 Compliant**: Implements all required and optional endpoints
- **Cloud-Agnostic Storage**: Uses @push.rocks/smartbucket for S3-compatible object storage
- **Pluggable Authentication**: Async callbacks for login and authorization
- **Bearer Token Auth**: JWT-based authentication following Docker Registry Token Authentication spec
- **Programmatic API**: Use as a library in any Node.js/TypeScript application
- **Full CRUD Operations**: Push, pull, list, and delete manifests and blobs
- **Content Discovery**: Tag listing and referrers API for artifact relationships
- **Chunked Uploads**: Support for large blob uploads with resumable sessions
### Dual Protocol Support
- **OCI Distribution Spec v1.1**: Full container registry with manifest/blob operations
- **NPM Registry API**: Complete package registry with publish/install/search
### Unified Architecture
- **Composable Design**: Core infrastructure with protocol plugins
- **Shared Storage**: Cloud-agnostic S3-compatible backend (@push.rocks/smartbucket)
- **Unified Authentication**: Scope-based permissions across both protocols
- **Path-based Routing**: `/oci/*` for containers, `/npm/*` for packages
### Authentication & Authorization
- NPM UUID tokens for package operations
- OCI JWT tokens for container operations
- Unified scope system: `npm:package:foo:write`, `oci:repository:bar:push`
- Pluggable via async callbacks
### Comprehensive Feature Set
**OCI Features:**
- ✅ Pull operations (manifests, blobs)
- ✅ Push operations (chunked uploads)
- ✅ Content discovery (tags, referrers API)
- ✅ Content management (deletion)
**NPM Features:**
- ✅ Package publish/unpublish
- ✅ Package download (tarballs)
- ✅ Metadata & search
- ✅ Dist-tag management
- ✅ Token management
## Installation
@@ -21,31 +43,14 @@ npm install @push.rocks/smartregistry
pnpm add @push.rocks/smartregistry
```
## Usage
### Basic Setup
## Quick Start
```typescript
import { SmartRegistry, IRegistryConfig, TLoginCallback, TAuthCallback } from '@push.rocks/smartregistry';
import { SmartRegistry, IRegistryConfig } from '@push.rocks/smartregistry';
// Implement login callback
const loginCallback: TLoginCallback = async (credentials) => {
// Validate credentials and return JWT token
// This should create a proper JWT with required claims
return generateJWT(credentials.username);
};
// Implement authorization callback
const authCallback: TAuthCallback = async (token, repository, action) => {
// Validate token and check permissions
const claims = verifyJWT(token);
return hasPermission(claims, repository, action);
};
// Configure registry
const config: IRegistryConfig = {
storage: {
accessKey: 'your-s3-access-key',
accessKey: 'your-s3-key',
accessSecret: 'your-s3-secret',
endpoint: 's3.amazonaws.com',
port: 443,
@@ -53,117 +58,406 @@ const config: IRegistryConfig = {
region: 'us-east-1',
bucketName: 'my-registry',
},
serviceName: 'my-registry',
tokenRealm: 'https://auth.example.com/token',
loginCallback,
authCallback,
auth: {
jwtSecret: 'your-secret-key',
tokenStore: 'memory',
npmTokens: { enabled: true },
ociTokens: {
enabled: true,
realm: 'https://auth.example.com/token',
service: 'my-registry',
},
},
oci: {
enabled: true,
basePath: '/oci',
},
npm: {
enabled: true,
basePath: '/npm',
},
};
// Create and initialize registry
const registry = new SmartRegistry(config);
await registry.init();
// Handle requests
const response = await registry.handleRequest({
method: 'GET',
path: '/npm/express',
headers: {},
query: {},
});
```
### Integration with HTTP Server
## Architecture
```typescript
import express from 'express';
### Directory Structure
const app = express();
// OCI Distribution API endpoints
app.get('/v2/', (req, res) => {
res.status(200).json({});
});
app.get('/v2/:name(*)/manifests/:reference', async (req, res) => {
const { name, reference } = req.params;
const token = req.headers.authorization?.replace('Bearer ', '');
const result = await registry.getManifest(name, reference, token);
if ('errors' in result) {
return res.status(404).json(result);
}
res.setHeader('Content-Type', result.contentType);
res.setHeader('Docker-Content-Digest', result.digest);
res.send(result.data);
});
app.get('/v2/:name(*)/blobs/:digest', async (req, res) => {
const { name, digest } = req.params;
const token = req.headers.authorization?.replace('Bearer ', '');
const result = await registry.getBlob(name, digest, token);
if ('errors' in result) {
return res.status(404).json(result);
}
res.setHeader('Content-Type', 'application/octet-stream');
res.send(result.data);
});
// ... implement other endpoints
app.listen(5000);
```
ts/
├── core/ # Shared infrastructure
│ ├── classes.baseregistry.ts
│ ├── classes.registrystorage.ts
│ ├── classes.authmanager.ts
│ └── interfaces.core.ts
├── oci/ # OCI implementation
│ ├── classes.ociregistry.ts
│ └── interfaces.oci.ts
├── npm/ # NPM implementation
│ ├── classes.npmregistry.ts
│ └── interfaces.npm.ts
└── classes.smartregistry.ts # Main orchestrator
```
### Authentication Flow
### Request Flow
```
HTTP Request
↓
SmartRegistry (orchestrator)
↓
Path-based routing
├─→ /oci/* → OciRegistry
└─→ /npm/* → NpmRegistry
↓
Shared Storage & Auth
↓
S3-compatible backend
```
## Usage Examples
### OCI Registry (Container Images)
```typescript
// Client requests without token
const challenge = registry.getAuthChallenge('library/nginx', ['pull', 'push']);
// Returns: Bearer realm="https://auth.example.com/token",service="my-registry",scope="repository:library/nginx:pull,push"
// Pull an image
const response = await registry.handleRequest({
method: 'GET',
path: '/oci/v2/library/nginx/manifests/latest',
headers: {
'Authorization': 'Bearer <token>',
},
query: {},
});
// Client authenticates
const token = await registry.login({ username: 'user', password: 'pass' });
// Push a blob
const uploadInit = await registry.handleRequest({
method: 'POST',
path: '/oci/v2/myapp/blobs/uploads/',
headers: { 'Authorization': 'Bearer <token>' },
query: {},
});
// Client uses token for subsequent requests
const manifest = await registry.getManifest('library/nginx', 'latest', token);
const uploadId = uploadInit.headers['Docker-Upload-UUID'];
await registry.handleRequest({
method: 'PUT',
path: `/oci/v2/myapp/blobs/uploads/${uploadId}`,
headers: { 'Authorization': 'Bearer <token>' },
query: { digest: 'sha256:abc123...' },
body: blobData,
});
```
### NPM Registry (Packages)
```typescript
// Install a package (get metadata)
const metadata = await registry.handleRequest({
method: 'GET',
path: '/npm/express',
headers: {},
query: {},
});
// Download tarball
const tarball = await registry.handleRequest({
method: 'GET',
path: '/npm/express/-/express-4.18.0.tgz',
headers: {},
query: {},
});
// Publish a package
const publishResponse = await registry.handleRequest({
method: 'PUT',
path: '/npm/my-package',
headers: { 'Authorization': 'Bearer <npm-token>' },
query: {},
body: {
name: 'my-package',
versions: {
'1.0.0': { /* version metadata */ },
},
'dist-tags': { latest: '1.0.0' },
_attachments: {
'my-package-1.0.0.tgz': {
content_type: 'application/octet-stream',
data: '<base64-tarball>',
length: 12345,
},
},
},
});
// Search packages
const searchResults = await registry.handleRequest({
method: 'GET',
path: '/npm/-/v1/search',
headers: {},
query: { text: 'express', size: '20' },
});
```
### Authentication
```typescript
// NPM Login
const authManager = registry.getAuthManager();
// Authenticate user
const userId = await authManager.authenticate({
username: 'user',
password: 'pass',
});
// Create NPM token
const npmToken = await authManager.createNpmToken(userId, false);
// Create OCI token with scopes
const ociToken = await authManager.createOciToken(
userId,
['oci:repository:myapp:push', 'oci:repository:myapp:pull'],
3600
);
// Validate any token
const token = await authManager.validateToken(npmToken, 'npm');
// Check permissions
const canWrite = await authManager.authorize(
token,
'npm:package:my-package',
'write'
);
```
## Configuration
### Storage Configuration
```typescript
storage: {
accessKey: string; // S3 access key
accessSecret: string; // S3 secret key
endpoint: string; // S3 endpoint
port?: number; // Default: 443
useSsl?: boolean; // Default: true
region?: string; // Default: 'us-east-1'
bucketName: string; // Bucket name
}
```
### Authentication Configuration
```typescript
auth: {
jwtSecret: string; // Secret for signing JWTs
tokenStore: 'memory' | 'redis' | 'database';
npmTokens: {
enabled: boolean;
defaultReadonly?: boolean;
};
ociTokens: {
enabled: boolean;
realm: string; // Auth server URL
service: string; // Service name
};
}
```
### Protocol Configuration
```typescript
oci?: {
enabled: boolean;
basePath: string; // Default: '/oci'
features?: {
referrers?: boolean;
deletion?: boolean;
};
}
npm?: {
enabled: boolean;
basePath: string; // Default: '/npm'
features?: {
publish?: boolean;
unpublish?: boolean;
search?: boolean;
};
}
```
## API Reference
### Pull Operations (Required)
### Core Classes
- `getManifest(repository, reference, token?)` - Download a manifest
- `headManifest(repository, reference, token?)` - Check manifest existence
- `getBlob(repository, digest, token?, range?)` - Download a blob
- `headBlob(repository, digest, token?)` - Check blob existence
#### SmartRegistry
### Push Operations
Main orchestrator class.
- `initiateUpload(repository, token, mountDigest?, fromRepository?)` - Start blob upload
- `uploadChunk(uploadId, data, contentRange, token)` - Upload blob chunk
- `completeUpload(uploadId, digest, token, finalData?)` - Finalize blob upload
- `putManifest(repository, reference, manifest, contentType, token)` - Upload manifest
**Methods:**
- `init()` - Initialize the registry
- `handleRequest(context)` - Handle HTTP request
- `getStorage()` - Get storage instance
- `getAuthManager()` - Get auth manager
- `getRegistry(protocol)` - Get protocol handler
### Content Discovery
#### RegistryStorage
- `listTags(repository, token?, pagination?)` - List all tags
- `getReferrers(repository, digest, token?, artifactType?)` - Get referencing artifacts
Unified storage abstraction.
### Content Management
**OCI Methods:**
- `getOciBlob(digest)` - Get blob
- `putOciBlob(digest, data)` - Store blob
- `getOciManifest(repo, digest)` - Get manifest
- `putOciManifest(repo, digest, data, type)` - Store manifest
- `deleteManifest(repository, digest, token)` - Delete manifest
- `deleteBlob(repository, digest, token)` - Delete blob
- `deleteTag(repository, tag, token)` - Delete tag
**NPM Methods:**
- `getNpmPackument(name)` - Get package metadata
- `putNpmPackument(name, data)` - Store package metadata
- `getNpmTarball(name, version)` - Get tarball
- `putNpmTarball(name, version, data)` - Store tarball
### Authentication
#### AuthManager
- `login(credentials)` - Get authentication token
- `getAuthChallenge(repository, actions)` - Generate WWW-Authenticate header
Unified authentication manager.
## OCI Specification Compliance
**Methods:**
- `authenticate(credentials)` - Validate user credentials
- `createNpmToken(userId, readonly)` - Create NPM token
- `createOciToken(userId, scopes, expiresIn)` - Create OCI JWT
- `validateToken(token, protocol)` - Validate any token
- `authorize(token, resource, action)` - Check permissions
This library implements:
### Protocol Handlers
- **Pull Category** (required): All manifest and blob retrieval operations
- **Push Category**: Complete blob upload workflow with chunked and monolithic modes
- **Content Discovery**: Tag listing and referrers API
- **Content Management**: Deletion operations for manifests, blobs, and tags
#### OciRegistry
**Endpoints:**
- `GET /v2/` - Version check
- `GET /v2/{name}/manifests/{ref}` - Get manifest
- `PUT /v2/{name}/manifests/{ref}` - Push manifest
- `GET /v2/{name}/blobs/{digest}` - Get blob
- `POST /v2/{name}/blobs/uploads/` - Initiate upload
- `GET /v2/{name}/tags/list` - List tags
- `GET /v2/{name}/referrers/{digest}` - Get referrers
#### NpmRegistry
**Endpoints:**
- `GET /{package}` - Get package metadata
- `PUT /{package}` - Publish package
- `GET /{package}/-/{tarball}` - Download tarball
- `GET /-/v1/search` - Search packages
- `PUT /-/user/org.couchdb.user:{user}` - Login
- `GET /-/npm/v1/tokens` - List tokens
- `POST /-/npm/v1/tokens` - Create token
- `PUT /-/package/{pkg}/dist-tags/{tag}` - Update tag
## Storage Structure
```
bucket/
├── oci/
│ ├── blobs/
│ │ └── sha256/{hash}
│ ├── manifests/
│ │ └── {repository}/{digest}
│ └── tags/
│ └── {repository}/tags.json
└── npm/
├── packages/
│ ├── {name}/
│ │ ├── index.json # Packument
│ │ └── {name}-{ver}.tgz # Tarball
│ └── @{scope}/{name}/
│ ├── index.json
│ └── {name}-{ver}.tgz
└── users/
└── {username}.json
```
## Scope Format
Unified scope format across protocols:
```
{protocol}:{type}:{name}:{action}
Examples:
npm:package:express:read # Read express package
npm:package:*:write # Write any package
npm:*:* # Full NPM access
oci:repository:nginx:pull # Pull nginx image
oci:repository:*:push # Push any image
oci:*:* # Full OCI access
```
## Integration Examples
### Express Server
```typescript
import express from 'express';
import { SmartRegistry } from '@push.rocks/smartregistry';
const app = express();
const registry = new SmartRegistry(config);
await registry.init();
app.all('*', async (req, res) => {
const response = await registry.handleRequest({
method: req.method,
path: req.path,
headers: req.headers as Record<string, string>,
query: req.query as Record<string, string>,
body: req.body,
});
res.status(response.status);
Object.entries(response.headers).forEach(([key, value]) => {
res.setHeader(key, value);
});
if (response.body) {
if (Buffer.isBuffer(response.body)) {
res.send(response.body);
} else {
res.json(response.body);
}
} else {
res.end();
}
});
app.listen(5000);
```
## Development
```bash
# Install dependencies
pnpm install
# Build
pnpm run build
# Test
pnpm test
```
## License
@@ -171,4 +465,4 @@ MIT
## Contributing
See the main repository for contribution guidelines.
Contributions welcome! Please see the repository for guidelines.