smarts3/readme.md

# @push.rocks/smarts3 🚀

A high-performance, S3-compatible local server powered by a **Rust core** with a clean TypeScript API. Drop-in replacement for AWS S3 during development and testing — no cloud, no Docker, no MinIO. Just `npm install` and go.

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## 🌟 Why smarts3?

| Feature | smarts3 | MinIO | s3rver |
|---------|---------|-------|--------|
| Install | `pnpm add` | Docker / binary | `npm install` |
| Startup time | ~20ms | seconds | ~200ms |
| Large file uploads | ✅ Streaming, zero-copy | ✅ | ❌ OOM risk |
| Range requests | ✅ Seek-based | ✅ | ❌ Full read |
| Language | Rust + TypeScript | Go | JavaScript |
| Multipart uploads | ✅ Full support | ✅ | ❌ |
| Auth | AWS v2/v4 key extraction | Full IAM | Basic |

### Core Features

- ⚡ **Rust-powered HTTP server** — hyper 1.x with streaming I/O, zero-copy, backpressure
- 🔄 **Full S3 API compatibility** — works with AWS SDK v3, SmartBucket, any S3 client
- 📂 **Filesystem-backed storage** — buckets map to directories, objects to files
- 📤 **Streaming multipart uploads** — large files without memory pressure
- 🎯 **Byte-range requests** — `seek()` directly to the requested byte offset
- 🔐 **Authentication** — AWS v2/v4 signature key extraction
- 🌐 **CORS middleware** — configurable cross-origin support
- 📊 **Structured logging** — tracing-based, error through debug levels
- 🧹 **Clean slate mode** — wipe storage on startup for test isolation
- 🧪 **Test-first design** — start/stop in milliseconds, no port conflicts

## 📦 Installation

```bash
pnpm add @push.rocks/smarts3 -D
```

> **Note:** The package ships with precompiled Rust binaries for `linux_amd64` and `linux_arm64`. No Rust toolchain needed on your machine.

## 🚀 Quick Start

```typescript
import { Smarts3 } from '@push.rocks/smarts3';

// Start a local S3 server
const s3 = await Smarts3.createAndStart({
  server: { port: 3000 },
  storage: { cleanSlate: true },
});

// Create a bucket
await s3.createBucket('my-bucket');

// Get connection details for any S3 client
const descriptor = await s3.getS3Descriptor();
// → { endpoint: 'localhost', port: 3000, accessKey: 'S3RVER', accessSecret: 'S3RVER', useSsl: false }

// When done
await s3.stop();
```

## 📖 Configuration

All config fields are optional — sensible defaults are applied automatically.

```typescript
import { Smarts3, ISmarts3Config } from '@push.rocks/smarts3';

const config: ISmarts3Config = {
  server: {
    port: 3000,              // Default: 3000
    address: '0.0.0.0',      // Default: '0.0.0.0'
    silent: false,           // Default: false
  },
  storage: {
    directory: './my-data',  // Default: .nogit/bucketsDir
    cleanSlate: false,       // Default: false — set true to wipe on start
  },
  auth: {
    enabled: false,          // Default: false
    credentials: [{
      accessKeyId: 'MY_KEY',
      secretAccessKey: 'MY_SECRET',
    }],
  },
  cors: {
    enabled: false,          // Default: false
    allowedOrigins: ['*'],
    allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'],
    allowedHeaders: ['*'],
    exposedHeaders: ['ETag', 'x-amz-request-id', 'x-amz-version-id'],
    maxAge: 86400,
    allowCredentials: false,
  },
  logging: {
    level: 'info',           // 'error' | 'warn' | 'info' | 'debug'
    format: 'text',          // 'text' | 'json'
    enabled: true,
  },
  limits: {
    maxObjectSize: 5 * 1024 * 1024 * 1024, // 5 GB
    maxMetadataSize: 2048,
    requestTimeout: 300000,  // 5 minutes
  },
  multipart: {
    expirationDays: 7,
    cleanupIntervalMinutes: 60,
  },
};

const s3 = await Smarts3.createAndStart(config);
```

### Common Configurations

**CI/CD testing** — silent, clean, fast:
```typescript
const s3 = await Smarts3.createAndStart({
  server: { port: 9999, silent: true },
  storage: { cleanSlate: true },
});
```

**Auth enabled:**
```typescript
const s3 = await Smarts3.createAndStart({
  auth: {
    enabled: true,
    credentials: [{ accessKeyId: 'test', secretAccessKey: 'test123' }],
  },
});
```

**CORS for local web dev:**
```typescript
const s3 = await Smarts3.createAndStart({
  cors: {
    enabled: true,
    allowedOrigins: ['http://localhost:5173'],
    allowCredentials: true,
  },
});
```

## 📤 Usage with AWS SDK v3

```typescript
import { S3Client, PutObjectCommand, GetObjectCommand, DeleteObjectCommand } from '@aws-sdk/client-s3';

const descriptor = await s3.getS3Descriptor();

const client = new S3Client({
  endpoint: `http://${descriptor.endpoint}:${descriptor.port}`,
  region: 'us-east-1',
  credentials: {
    accessKeyId: descriptor.accessKey,
    secretAccessKey: descriptor.accessSecret,
  },
  forcePathStyle: true,  // Required for path-style S3
});

// Upload
await client.send(new PutObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
  Body: 'Hello, S3!',
  ContentType: 'text/plain',
}));

// Download
const { Body } = await client.send(new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
}));
const content = await Body.transformToString(); // "Hello, S3!"

// Delete
await client.send(new DeleteObjectCommand({
  Bucket: 'my-bucket',
  Key: 'hello.txt',
}));
```

## 🪣 Usage with SmartBucket

```typescript
import { SmartBucket } from '@push.rocks/smartbucket';

const smartbucket = new SmartBucket(await s3.getS3Descriptor());
const bucket = await smartbucket.createBucket('my-bucket');
const dir = await bucket.getBaseDirectory();

// Upload
await dir.fastPut({ path: 'docs/readme.txt', contents: 'Hello!' });

// Download
const content = await dir.fastGet('docs/readme.txt');

// List
const files = await dir.listFiles();
```

## 📤 Multipart Uploads

For files larger than 5 MB, use multipart uploads. smarts3 handles them with **streaming I/O** — parts are written directly to disk, never buffered in memory.

```typescript
import {
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand,
} from '@aws-sdk/client-s3';

// 1. Initiate
const { UploadId } = await client.send(new CreateMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
}));

// 2. Upload parts
const parts = [];
for (let i = 0; i < chunks.length; i++) {
  const { ETag } = await client.send(new UploadPartCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    UploadId,
    PartNumber: i + 1,
    Body: chunks[i],
  }));
  parts.push({ PartNumber: i + 1, ETag });
}

// 3. Complete
await client.send(new CompleteMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
  UploadId,
  MultipartUpload: { Parts: parts },
}));
```

## 🧪 Testing Integration

```typescript
import { Smarts3 } from '@push.rocks/smarts3';
import { tap, expect } from '@git.zone/tstest/tapbundle';

let s3: Smarts3;

tap.test('setup', async () => {
  s3 = await Smarts3.createAndStart({
    server: { port: 4567, silent: true },
    storage: { cleanSlate: true },
  });
});

tap.test('should store and retrieve objects', async () => {
  await s3.createBucket('test');
  // ... your test logic using AWS SDK or SmartBucket
});

tap.test('teardown', async () => {
  await s3.stop();
});

export default tap.start();
```

## 🔧 API Reference

### `Smarts3` Class

#### `static createAndStart(config?: ISmarts3Config): Promise<Smarts3>`

Create and start a server in one call.

#### `start(): Promise<void>`

Spawn the Rust binary and start the HTTP server.

#### `stop(): Promise<void>`

Gracefully stop the server and kill the Rust process.

#### `createBucket(name: string): Promise<{ name: string }>`

Create an S3 bucket.

#### `getS3Descriptor(options?): Promise<IS3Descriptor>`

Get connection details for S3 clients. Returns:

| Field | Type | Description |
|-------|------|-------------|
| `endpoint` | `string` | Server hostname (`localhost` by default) |
| `port` | `number` | Server port |
| `accessKey` | `string` | Access key from first configured credential |
| `accessSecret` | `string` | Secret key from first configured credential |
| `useSsl` | `boolean` | Always `false` (plain HTTP) |

## 🏗️ Architecture

smarts3 uses a **hybrid Rust + TypeScript** architecture:

```
┌─────────────────────────────────┐
│  Your Code (AWS SDK, etc.)      │
│  ↕ HTTP (localhost:3000)        │
├─────────────────────────────────┤
│  rusts3 binary (Rust)           │
│  ├─ hyper 1.x HTTP server      │
│  ├─ S3 path-style routing      │
│  ├─ Streaming storage layer    │
│  ├─ Multipart manager          │
│  ├─ CORS / Auth middleware     │
│  └─ S3 XML response builder   │
├─────────────────────────────────┤
│  TypeScript (thin IPC wrapper)  │
│  ├─ Smarts3 class              │
│  ├─ RustBridge (stdin/stdout)  │
│  └─ Config & S3 descriptor    │
└─────────────────────────────────┘
```

**Why Rust?** The TypeScript implementation had critical perf issues: OOM on multipart uploads (parts buffered in memory), double stream copying, file descriptor leaks on HEAD requests, full-file reads for range requests, and no backpressure. The Rust binary solves all of these with streaming I/O, zero-copy, and direct `seek()` for range requests.

**IPC Protocol:** TypeScript spawns the `rusts3` binary with `--management` and communicates via newline-delimited JSON over stdin/stdout. Commands: `start`, `stop`, `createBucket`.

### S3 Operations Supported

| Operation | Method | Path |
|-----------|--------|------|
| ListBuckets | `GET /` | |
| CreateBucket | `PUT /{bucket}` | |
| DeleteBucket | `DELETE /{bucket}` | |
| HeadBucket | `HEAD /{bucket}` | |
| ListObjects (v1/v2) | `GET /{bucket}` | `?list-type=2` for v2 |
| PutObject | `PUT /{bucket}/{key}` | |
| GetObject | `GET /{bucket}/{key}` | Supports `Range` header |
| HeadObject | `HEAD /{bucket}/{key}` | |
| DeleteObject | `DELETE /{bucket}/{key}` | |
| CopyObject | `PUT /{bucket}/{key}` | `x-amz-copy-source` header |
| InitiateMultipartUpload | `POST /{bucket}/{key}?uploads` | |
| UploadPart | `PUT /{bucket}/{key}?partNumber&uploadId` | |
| CompleteMultipartUpload | `POST /{bucket}/{key}?uploadId` | |
| AbortMultipartUpload | `DELETE /{bucket}/{key}?uploadId` | |
| ListMultipartUploads | `GET /{bucket}?uploads` | |

### On-Disk Format

```
{storage.directory}/
  {bucket}/
    {key}._S3_object              # Object data
    {key}._S3_object.metadata.json  # Metadata (content-type, x-amz-meta-*, etc.)
    {key}._S3_object.md5          # Cached MD5 hash
  .multipart/
    {upload-id}/
      metadata.json               # Upload metadata (bucket, key, parts)
      part-1                      # Part data files
      part-2
      ...
```

## 🔗 Related Packages

- [`@push.rocks/smartbucket`](https://code.foss.global/push.rocks/smartbucket) — High-level S3 abstraction layer
- [`@push.rocks/smartrust`](https://code.foss.global/push.rocks/smartrust) — TypeScript ↔ Rust IPC bridge
- [`@git.zone/tsrust`](https://code.foss.global/git.zone/tsrust) — Rust cross-compilation for npm packages

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.