feat(bucket-tenants): add persisted bucket-scoped tenant credentials with bucket export and import APIs

This commit is contained in:
2026-05-02 11:14:15 +00:00
parent 53d663597a
commit 7f2546e041
14 changed files with 1675 additions and 117 deletions
+101 -3
View File
@@ -34,6 +34,7 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
- 🧹 **Clean slate mode** — wipe storage on startup for test isolation
- 📊 **Runtime storage stats** — cheap bucket summaries and global counts without S3 list scans
- 🔑 **Runtime credential rotation** — list and replace active auth credentials without mutating internals
- 🧩 **Bucket tenants** — provision one scoped S3 credential per bucket with restart persistence
-**Test-first design** — start/stop in milliseconds, no port conflicts
### Clustering Features
@@ -225,15 +226,76 @@ await storage.replaceCredentials([
interface IStorageCredential {
accessKeyId: string;
secretAccessKey: string;
bucketName?: string;
region?: string;
}
```
- `listCredentials()` returns the Rust core's current runtime credential set.
- `replaceCredentials()` swaps the full set atomically. On success, new requests use the new set immediately and the old credentials stop authenticating immediately.
- `replaceCredentials()` swaps the full set atomically and persists it under the storage root. On success, new requests use the new set immediately and the old credentials stop authenticating immediately.
- Requests that were already authenticated before the replacement keep running; auth is evaluated when each request starts.
- No restart is required.
- No restart is required, and runtime-created credentials survive restart unless `storage.cleanSlate` removes the storage directory.
- Replacement input must contain at least one credential, each `accessKeyId` and `secretAccessKey` must be non-empty, and `accessKeyId` values must be unique.
## Bucket Tenants
Bucket tenants are designed for platform services that need one bucket and one scoped S3 credential per app. Tenant credentials are enforced by the auth layer before the normal bucket-policy/default-auth pipeline, so a scoped credential cannot list all buckets or access another bucket even when it has a valid SigV4 signature.
```typescript
const tenant = await storage.createBucketTenant({
bucketName: 'workapp-123',
});
// Directly usable by AWS SDK v3 or env injection
const client = new S3Client({
endpoint: `http://${tenant.endpoint}:${tenant.port}`,
region: tenant.region,
credentials: {
accessKeyId: tenant.accessKeyId,
secretAccessKey: tenant.secretAccessKey,
},
forcePathStyle: true,
});
console.log(tenant.env.S3_BUCKET);
console.log(tenant.env.AWS_ACCESS_KEY_ID);
```
```typescript
await storage.rotateBucketTenantCredentials({ bucketName: 'workapp-123' });
await storage.deleteBucketTenant({ bucketName: 'workapp-123', accessKeyId: tenant.accessKeyId });
const descriptor = await storage.getBucketTenantDescriptor({ bucketName: 'workapp-123' });
const tenants = await storage.listBucketTenants();
```
- `createBucketTenant()` creates the bucket if needed and stores a scoped credential for that bucket.
- `rotateBucketTenantCredentials()` replaces the active scoped credential for the bucket and persists the new credential.
- `deleteBucketTenant({ bucketName, accessKeyId })` revokes one scoped credential and keeps the bucket.
- `deleteBucketTenant({ bucketName })` revokes scoped credentials for the bucket and deletes the bucket contents recursively.
- Tenant credentials can list, read, write, and delete objects in their assigned bucket, but cannot list all buckets, access other buckets, copy from other buckets, delete buckets, or mutate bucket policies.
- Bucket tenant APIs require `auth.enabled: true`.
## Bucket Backup/Restore
```typescript
const appBackup = await storage.exportBucket({ bucketName: 'workapp-123' });
await storage.importBucket({ bucketName: 'workapp-123-restore', source: appBackup });
```
- `exportBucket()` returns a self-contained `smartstorage.bucket.v1` JSON export with only the selected bucket's objects and object metadata.
- `importBucket()` creates the target bucket if needed and restores the exported objects into that bucket.
- Exports do not include credentials, policies, or unrelated tenant data.
## Health and Metrics APIs
```typescript
const health = await storage.getHealth();
const metrics = await storage.getMetrics();
```
- `getHealth()` reports running state, storage directory, auth enabled state, credential counts, bucket count, object count, total bytes, and cluster health.
- `getMetrics()` returns numeric counters and a Prometheus text snippet for bucket, object, byte, tenant credential, and cluster-enabled metrics.
## Runtime Stats
```typescript
@@ -577,6 +639,34 @@ Gracefully stop the server and kill the Rust process.
Create a storage bucket.
#### `createBucketTenant(options): Promise<IBucketTenantDescriptor>`
Create a bucket tenant with a generated or supplied scoped credential. Options: `{ bucketName, accessKeyId?, secretAccessKey?, region? }`.
#### `deleteBucketTenant(options): Promise<void>`
Revoke a tenant credential or delete the full tenant bucket. Options: `{ bucketName, accessKeyId? }`.
#### `rotateBucketTenantCredentials(options): Promise<IBucketTenantDescriptor>`
Replace the scoped credential for a bucket tenant. Options: `{ bucketName, accessKeyId?, secretAccessKey?, region? }`.
#### `listBucketTenants(): Promise<IBucketTenantMetadata[]>`
List scoped tenant credential metadata without returning secrets.
#### `getBucketTenantDescriptor(options): Promise<IBucketTenantDescriptor>`
Return endpoint, port, region, bucket, access key, secret key, SSL flag, legacy descriptor fields, and S3/AWS env values for the bucket tenant.
#### `exportBucket(options): Promise<IBucketExport>`
Export one bucket's objects and metadata into a `smartstorage.bucket.v1` JSON object.
#### `importBucket(options): Promise<void>`
Import a `smartstorage.bucket.v1` JSON object into the target bucket. Options: `{ bucketName, source }`.
#### `getStorageDescriptor(options?): Promise<IS3Descriptor>`
Get connection details for S3-compatible clients. Returns:
@@ -609,6 +699,14 @@ Atomically replace the active runtime credential set without restarting the serv
Read the Rust core's current cluster, drive, quorum, and repair health snapshot. Standalone mode returns `{ enabled: false }`.
#### `getHealth(): Promise<ISmartStorageHealth>`
Return running state, storage directory, auth state, credential counts, bucket count, object count, total bytes, and cluster health.
#### `getMetrics(): Promise<ISmartStorageMetrics>`
Return numeric metrics plus a Prometheus text snippet for operational scraping.
## Architecture
smartstorage uses a **hybrid Rust + TypeScript** architecture:
@@ -642,7 +740,7 @@ smartstorage uses a **hybrid Rust + TypeScript** architecture:
**Why Rust?** The original TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests.
**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `start`, `stop`, `createBucket`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, and `getClusterHealth`.
**IPC Protocol:** TypeScript communicates with the `ruststorage` binary over newline-delimited JSON via stdin/stdout. The current management commands are `start`, `stop`, `createBucket`, `createBucketTenant`, `deleteBucketTenant`, `rotateBucketTenantCredentials`, `listBucketTenants`, `getBucketTenantCredential`, `exportBucket`, `importBucket`, `getStorageStats`, `listBucketSummaries`, `listCredentials`, `replaceCredentials`, and `getClusterHealth`.
### S3-Compatible Operations