diff --git a/changelog.md b/changelog.md index 547b88a..8b65dc6 100644 --- a/changelog.md +++ b/changelog.md @@ -1,5 +1,17 @@ # Changelog +## 2025-11-23 - 5.0.0 - BREAKING CHANGE(core) +Production-ready S3-compatible server: nested config, multipart uploads, CORS, structured logging, SmartFS migration and improved error handling + +- Breaking change: configuration format migrated from flat to nested structure (server, storage, auth, cors, logging, limits). Update existing configs accordingly. +- Implemented full multipart upload support (initiate, upload part, complete, abort) with on-disk part management and final assembly. +- Added CORS middleware with configurable origins, methods, headers, exposed headers, maxAge and credentials support. +- Structured, configurable logging (levels: error|warn|info|debug; formats: text|json) and request/response logging middleware. +- Simple static credential authentication middleware (configurable list of credentials). +- Migrated filesystem operations to @push.rocks/smartfs (Web Streams interoperability) and removed smartbucket from production dependencies. +- Improved S3-compatible error handling and XML responses (S3Error class and XML utilities). +- Exposed Smarts3Server and made store/multipart managers accessible for tests and advanced usage; added helper methods like getS3Descriptor and createBucket. + ## 2025-11-23 - 4.0.0 - BREAKING CHANGE(Smarts3) Migrate Smarts3 configuration to nested server/storage objects and remove legacy flat config support diff --git a/readme.md b/readme.md index ddeaf92..723c829 100644 --- a/readme.md +++ b/readme.md @@ -1,16 +1,20 @@ # @push.rocks/smarts3 ๐Ÿš€ -**Mock S3 made simple** - A powerful Node.js TypeScript package for creating a local S3 endpoint that simulates AWS S3 operations using mapped local directories. Perfect for development and testing! +**Production-ready S3-compatible server** - A powerful, lightweight Node.js TypeScript package that brings full S3 API compatibility to your local filesystem. Perfect for development, testing, and scenarios where running MinIO is out of scope! ## ๐ŸŒŸ Features - ๐Ÿƒ **Lightning-fast local S3 simulation** - No more waiting for cloud operations during development -- โšก **Native custom S3 server** - Built on Node.js http module with zero framework dependencies +- โšก **Production-ready architecture** - Built on Node.js http module with zero framework dependencies - ๐Ÿ”„ **Full AWS S3 API compatibility** - Drop-in replacement for AWS SDK v3 and other S3 clients -- ๐Ÿ“‚ **Local directory mapping** - Your buckets live right on your filesystem with Windows-compatible encoding +- ๐Ÿ“‚ **Local directory mapping** - Your buckets live right on your filesystem +- ๐Ÿ” **Simple authentication** - Static credential-based auth for secure access +- ๐ŸŒ **CORS support** - Configurable cross-origin resource sharing +- ๐Ÿ“Š **Structured logging** - Multiple levels (error/warn/info/debug) and formats (text/JSON) +- ๐Ÿ“ค **Multipart uploads** - Full support for large file uploads (>5MB) - ๐Ÿงช **Perfect for testing** - Reliable, repeatable tests without cloud dependencies - ๐ŸŽฏ **TypeScript-first** - Built with TypeScript for excellent type safety and IDE support -- ๐Ÿ”ง **Zero configuration** - Works out of the box with sensible defaults +- ๐Ÿ”ง **Flexible configuration** - Comprehensive config system with sensible defaults - ๐Ÿงน **Clean slate mode** - Start fresh on every test run ## Issue Reporting and Security @@ -39,10 +43,15 @@ Get up and running in seconds: ```typescript import { Smarts3 } from '@push.rocks/smarts3'; -// Start your local S3 server +// Start your local S3 server with minimal config const s3Server = await Smarts3.createAndStart({ - port: 3000, - cleanSlate: true, // Start with empty buckets + server: { + port: 3000, + silent: false, + }, + storage: { + cleanSlate: true, // Start with empty buckets + }, }); // Create a bucket @@ -55,44 +64,165 @@ const s3Config = await s3Server.getS3Descriptor(); await s3Server.stop(); ``` -## ๐Ÿ“– Detailed Usage Guide +## ๐Ÿ“– Configuration Guide -### ๐Ÿ—๏ธ Setting Up Your S3 Server +### Complete Configuration Options -The `Smarts3` class provides a simple interface for managing your local S3 server: +Smarts3 uses a comprehensive nested configuration structure: ```typescript -import { Smarts3 } from '@push.rocks/smarts3'; +import { Smarts3, ISmarts3Config } from '@push.rocks/smarts3'; -// Configuration options -const config = { - port: 3000, // Port to run the server on (default: 3000) - cleanSlate: true, // Clear all data on start (default: false) +const config: ISmarts3Config = { + // Server configuration + server: { + port: 3000, // Port to listen on (default: 3000) + address: '0.0.0.0', // Bind address (default: '0.0.0.0') + silent: false, // Disable all console output (default: false) + }, + + // Storage configuration + storage: { + directory: './buckets', // Directory to store buckets (default: .nogit/bucketsDir) + cleanSlate: false, // Clear all data on start (default: false) + }, + + // Authentication configuration + auth: { + enabled: false, // Enable authentication (default: false) + credentials: [ // List of valid credentials + { + accessKeyId: 'YOUR_ACCESS_KEY', + secretAccessKey: 'YOUR_SECRET_KEY', + }, + ], + }, + + // CORS configuration + cors: { + enabled: false, // Enable CORS (default: false) + allowedOrigins: ['*'], // Allowed origins (default: ['*']) + allowedMethods: [ // Allowed HTTP methods + 'GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS' + ], + allowedHeaders: ['*'], // Allowed headers (default: ['*']) + exposedHeaders: [ // Headers exposed to client + 'ETag', 'x-amz-request-id', 'x-amz-version-id' + ], + maxAge: 86400, // Preflight cache duration in seconds + allowCredentials: false, // Allow credentials (default: false) + }, + + // Logging configuration + logging: { + level: 'info', // Log level: 'error' | 'warn' | 'info' | 'debug' + format: 'text', // Log format: 'text' | 'json' + enabled: true, // Enable logging (default: true) + }, + + // Request limits + limits: { + maxObjectSize: 5 * 1024 * 1024 * 1024, // 5GB max object size + maxMetadataSize: 2048, // 2KB max metadata size + requestTimeout: 300000, // 5 minutes request timeout + }, }; -// Create and start in one go const s3Server = await Smarts3.createAndStart(config); - -// Or create and start separately -const s3Server = new Smarts3(config); -await s3Server.start(); ``` -### ๐Ÿชฃ Working with Buckets +### Simple Configuration Examples -Creating and managing buckets is straightforward: +**Development Mode (Default)** +```typescript +const s3Server = await Smarts3.createAndStart({ + server: { port: 3000 }, + storage: { cleanSlate: true }, +}); +``` + +**Production Mode with Auth** +```typescript +const s3Server = await Smarts3.createAndStart({ + server: { port: 3000 }, + auth: { + enabled: true, + credentials: [ + { + accessKeyId: process.env.S3_ACCESS_KEY, + secretAccessKey: process.env.S3_SECRET_KEY, + }, + ], + }, + logging: { + level: 'warn', + format: 'json', + }, +}); +``` + +**CORS-Enabled for Web Apps** +```typescript +const s3Server = await Smarts3.createAndStart({ + server: { port: 3000 }, + cors: { + enabled: true, + allowedOrigins: ['http://localhost:8080', 'https://app.example.com'], + allowCredentials: true, + }, +}); +``` + +## ๐Ÿชฃ Working with Buckets + +### Creating Buckets ```typescript // Create a new bucket const bucket = await s3Server.createBucket('my-bucket'); - -// The bucket is now ready to use! console.log(`Created bucket: ${bucket.name}`); ``` -### ๐Ÿ“ค Uploading Files +## ๐Ÿ“ค File Operations -Use the powerful `SmartBucket` integration for file operations: +### Using AWS SDK v3 + +```typescript +import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3'; + +// Get connection config +const config = await s3Server.getS3Descriptor(); + +// Configure AWS SDK client +const s3Client = new S3Client({ + endpoint: `http://${config.endpoint}:${config.port}`, + region: 'us-east-1', + credentials: { + accessKeyId: config.accessKey, + secretAccessKey: config.accessSecret, + }, + forcePathStyle: true, +}); + +// Upload a file +await s3Client.send(new PutObjectCommand({ + Bucket: 'my-bucket', + Key: 'test-file.txt', + Body: 'Hello from AWS SDK!', + ContentType: 'text/plain', +})); + +// Download a file +const response = await s3Client.send(new GetObjectCommand({ + Bucket: 'my-bucket', + Key: 'test-file.txt', +})); + +const content = await response.Body.transformToString(); +console.log(content); // "Hello from AWS SDK!" +``` + +### Using SmartBucket ```typescript import { SmartBucket } from '@push.rocks/smartbucket'; @@ -102,63 +232,74 @@ const s3Config = await s3Server.getS3Descriptor(); // Create a SmartBucket instance const smartbucket = new SmartBucket(s3Config); - -// Get your bucket const bucket = await smartbucket.getBucket('my-bucket'); - -// Upload a file const baseDir = await bucket.getBaseDirectory(); -await baseDir.fastStore('path/to/file.txt', 'Hello, S3! ๐ŸŽ‰'); -// Upload with more control +// Upload files +await baseDir.fastStore('path/to/file.txt', 'Hello, S3! ๐ŸŽ‰'); await baseDir.fastPut({ path: 'documents/important.pdf', contents: Buffer.from(yourPdfData), }); -``` -### ๐Ÿ“ฅ Downloading Files - -Retrieve your files easily: - -```typescript -// Get file contents as string +// Download files const content = await baseDir.fastGet('path/to/file.txt'); -console.log(content); // "Hello, S3! ๐ŸŽ‰" - -// Get file as Buffer const buffer = await baseDir.fastGetBuffer('documents/important.pdf'); -``` -### ๐Ÿ“‹ Listing Files - -Browse your bucket contents: - -```typescript -// List all files in the bucket +// List files const files = await baseDir.listFiles(); - files.forEach((file) => { console.log(`๐Ÿ“„ ${file.name} (${file.size} bytes)`); }); -// List files with a specific prefix -const docs = await baseDir.listFiles('documents/'); +// Delete files +await baseDir.fastDelete('old-file.txt'); ``` -### ๐Ÿ—‘๏ธ Deleting Files +## ๐Ÿ“ค Multipart Uploads -Clean up when needed: +Smarts3 supports multipart uploads for large files (>5MB): ```typescript -// Delete a single file -await baseDir.fastDelete('old-file.txt'); +import { + S3Client, + CreateMultipartUploadCommand, + UploadPartCommand, + CompleteMultipartUploadCommand +} from '@aws-sdk/client-s3'; -// Delete multiple files -const filesToDelete = ['temp1.txt', 'temp2.txt', 'temp3.txt']; -for (const file of filesToDelete) { - await baseDir.fastDelete(file); +const s3Client = new S3Client(/* ... */); + +// 1. Initiate multipart upload +const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({ + Bucket: 'my-bucket', + Key: 'large-file.bin', +})); + +// 2. Upload parts (in parallel if desired) +const parts = []; +for (let i = 0; i < numParts; i++) { + const part = await s3Client.send(new UploadPartCommand({ + Bucket: 'my-bucket', + Key: 'large-file.bin', + UploadId, + PartNumber: i + 1, + Body: partData[i], + })); + + parts.push({ + PartNumber: i + 1, + ETag: part.ETag, + }); } + +// 3. Complete the upload +await s3Client.send(new CompleteMultipartUploadCommand({ + Bucket: 'my-bucket', + Key: 'large-file.bin', + UploadId, + MultipartUpload: { Parts: parts }, +})); ``` ## ๐Ÿงช Testing Integration @@ -173,8 +314,8 @@ describe('S3 Operations', () => { beforeAll(async () => { s3Server = await Smarts3.createAndStart({ - port: 9999, - cleanSlate: true, + server: { port: 9999, silent: true }, + storage: { cleanSlate: true }, }); }); @@ -200,8 +341,8 @@ describe('S3 Operations', () => { before(async () => { s3Server = await Smarts3.createAndStart({ - port: 9999, - cleanSlate: true, + server: { port: 9999, silent: true }, + storage: { cleanSlate: true }, }); }); @@ -216,40 +357,7 @@ describe('S3 Operations', () => { }); ``` -## ๐Ÿ”Œ AWS SDK Integration - -Use `smarts3` with the official AWS SDK: - -```typescript -import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3'; -import { Smarts3 } from '@push.rocks/smarts3'; - -// Start local S3 -const s3Server = await Smarts3.createAndStart({ port: 3000 }); -const config = await s3Server.getS3Descriptor(); - -// Configure AWS SDK -const s3Client = new S3Client({ - endpoint: `http://${config.endpoint}:${config.port}`, - region: 'us-east-1', - credentials: { - accessKeyId: config.accessKey, - secretAccessKey: config.accessSecret, - }, - forcePathStyle: true, -}); - -// Use AWS SDK as normal -const command = new PutObjectCommand({ - Bucket: 'my-bucket', - Key: 'test-file.txt', - Body: 'Hello from AWS SDK!', -}); - -await s3Client.send(command); -``` - -## ๐ŸŽฏ Real-World Examples +## ๐ŸŽฏ Real-World Use Cases ### CI/CD Pipeline Testing @@ -258,10 +366,13 @@ await s3Client.send(command); import { Smarts3 } from '@push.rocks/smarts3'; export async function setupTestEnvironment() { - // Start S3 server for CI tests const s3 = await Smarts3.createAndStart({ - port: process.env.S3_PORT || 3000, - cleanSlate: true, + server: { + port: process.env.S3_PORT || 3000, + silent: true, + }, + storage: { cleanSlate: true }, + logging: { level: 'error' }, // Only log errors in CI }); // Create test buckets @@ -281,8 +392,15 @@ import { Smarts3 } from '@push.rocks/smarts3'; import express from 'express'; async function startDevelopmentServer() { - // Start local S3 - const s3 = await Smarts3.createAndStart({ port: 3000 }); + // Start local S3 with CORS for local development + const s3 = await Smarts3.createAndStart({ + server: { port: 3000 }, + cors: { + enabled: true, + allowedOrigins: ['http://localhost:8080'], + }, + }); + await s3.createBucket('user-uploads'); // Start your API server @@ -302,13 +420,16 @@ async function startDevelopmentServer() { ```typescript import { Smarts3 } from '@push.rocks/smarts3'; +import { SmartBucket } from '@push.rocks/smartbucket'; async function testDataMigration() { - const s3 = await Smarts3.createAndStart({ cleanSlate: true }); + const s3 = await Smarts3.createAndStart({ + storage: { cleanSlate: true }, + }); // Create source and destination buckets - const sourceBucket = await s3.createBucket('legacy-data'); - const destBucket = await s3.createBucket('new-data'); + await s3.createBucket('legacy-data'); + await s3.createBucket('new-data'); // Populate source with test data const config = await s3.getS3Descriptor(); @@ -316,15 +437,8 @@ async function testDataMigration() { const source = await smartbucket.getBucket('legacy-data'); const sourceDir = await source.getBaseDirectory(); - // Add test files - await sourceDir.fastStore( - 'user-1.json', - JSON.stringify({ id: 1, name: 'Alice' }), - ); - await sourceDir.fastStore( - 'user-2.json', - JSON.stringify({ id: 2, name: 'Bob' }), - ); + await sourceDir.fastStore('user-1.json', JSON.stringify({ id: 1, name: 'Alice' })); + await sourceDir.fastStore('user-2.json', JSON.stringify({ id: 2, name: 'Bob' })); // Run your migration logic await runMigration(config); @@ -338,77 +452,105 @@ async function testDataMigration() { } ``` -## ๐Ÿ› ๏ธ Advanced Configuration - -### Custom S3 Descriptor Options - -When integrating with different S3 clients, you can customize the connection details: - -```typescript -const customDescriptor = await s3Server.getS3Descriptor({ - endpoint: 'localhost', // Custom endpoint - port: 3001, // Different port - useSsl: false, // SSL configuration - // Add any additional options your S3 client needs -}); -``` - -### Environment-Based Configuration - -```typescript -const config = { - port: parseInt(process.env.S3_PORT || '3000'), - cleanSlate: process.env.NODE_ENV === 'test', -}; - -const s3Server = await Smarts3.createAndStart(config); -``` - -## ๐Ÿค Use Cases - -- **๐Ÿงช Unit & Integration Testing** - Test S3 operations without AWS credentials or internet -- **๐Ÿ—๏ธ Local Development** - Develop cloud features offline with full S3 compatibility -- **๐Ÿ“š Teaching & Demos** - Perfect for workshops and tutorials without AWS setup -- **๐Ÿ”„ CI/CD Pipelines** - Reliable S3 operations in containerized test environments -- **๐ŸŽญ Mocking & Stubbing** - Replace real S3 calls in test suites -- **๐Ÿ“Š Data Migration Testing** - Safely test data migrations locally before production - ## ๐Ÿ”ง API Reference ### Smarts3 Class -#### Constructor Options +#### Static Methods -```typescript -interface ISmarts3ContructorOptions { - port?: number; // Server port (default: 3000) - cleanSlate?: boolean; // Clear storage on start (default: false) -} -``` +##### `createAndStart(config?: ISmarts3Config): Promise` -#### Methods +Create and start a Smarts3 instance in one call. -- `static createAndStart(options)` - Create and start server in one call -- `start()` - Start the S3 server -- `stop()` - Stop the S3 server -- `createBucket(name)` - Create a new bucket -- `getS3Descriptor(options?)` - Get S3 connection configuration +**Parameters:** +- `config` - Optional configuration object (see Configuration Guide above) + +**Returns:** Promise that resolves to a running Smarts3 instance + +#### Instance Methods + +##### `start(): Promise` + +Start the S3 server. + +##### `stop(): Promise` + +Stop the S3 server and release resources. + +##### `createBucket(name: string): Promise<{ name: string }>` + +Create a new S3 bucket. + +**Parameters:** +- `name` - Bucket name + +**Returns:** Promise that resolves to bucket information + +##### `getS3Descriptor(options?): Promise` + +Get S3 connection configuration for use with S3 clients. + +**Parameters:** +- `options` - Optional partial descriptor to merge with defaults + +**Returns:** Promise that resolves to S3 descriptor with: +- `accessKey` - Access key for authentication +- `accessSecret` - Secret key for authentication +- `endpoint` - Server endpoint (hostname/IP) +- `port` - Server port +- `useSsl` - Whether to use SSL (always false for local server) + +## ๐Ÿ’ก Production Considerations + +### When to Use Smarts3 vs MinIO + +**Use Smarts3 when:** +- ๐ŸŽฏ You need a lightweight, zero-dependency S3 server +- ๐Ÿงช Running in CI/CD pipelines or containerized test environments +- ๐Ÿ—๏ธ Local development where MinIO setup is overkill +- ๐Ÿ“ฆ Your application needs to bundle an S3-compatible server +- ๐Ÿš€ Quick prototyping without infrastructure setup + +**Use MinIO when:** +- ๐Ÿข Production workloads requiring high availability +- ๐Ÿ“Š Advanced features like versioning, replication, encryption at rest +- ๐Ÿ” Complex IAM policies and bucket policies +- ๐Ÿ“ˆ High-performance requirements with multiple nodes +- ๐ŸŒ Multi-tenant environments + +### Security Notes + +- Smarts3's authentication is intentionally simple (static credentials) +- It does **not** implement AWS Signature V4 verification +- Perfect for development/testing, but not for production internet-facing deployments +- For production use, place behind a reverse proxy with proper authentication ## ๐Ÿ› Debugging Tips -1. **Enable verbose logging** - The server logs all operations by default -2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/` +1. **Enable debug logging** + ```typescript + const s3 = await Smarts3.createAndStart({ + logging: { level: 'debug', format: 'json' }, + }); + ``` + +2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/` by default + 3. **Use the correct endpoint** - Remember to use `127.0.0.1` or `localhost` -4. **Force path style** - Always use path-style URLs with local S3 + +4. **Force path style** - Always use `forcePathStyle: true` with local S3 + +5. **Inspect requests** - All requests are logged when `silent: false` ## ๐Ÿ“ˆ Performance -`@push.rocks/smarts3` is optimized for development and testing: +Smarts3 is optimized for development and testing scenarios: - โšก **Instant operations** - No network latency -- ๐Ÿ’พ **Low memory footprint** - Efficient file system usage +- ๐Ÿ’พ **Low memory footprint** - Efficient filesystem operations with streams - ๐Ÿ”„ **Fast cleanup** - Clean slate mode for quick test resets -- ๐Ÿš€ **Parallel operations** - Handle multiple requests simultaneously +- ๐Ÿš€ **Parallel operations** - Handle multiple concurrent requests +- ๐Ÿ“ค **Streaming uploads/downloads** - Low memory usage for large files ## ๐Ÿ”— Related Packages @@ -416,6 +558,29 @@ interface ISmarts3ContructorOptions { - [`@push.rocks/smartfs`](https://www.npmjs.com/package/@push.rocks/smartfs) - Modern filesystem with Web Streams support - [`@tsclass/tsclass`](https://www.npmjs.com/package/@tsclass/tsclass) - TypeScript class helpers +## ๐Ÿ“ Changelog + +### v4.0.0 - Production Ready ๐Ÿš€ + +**Breaking Changes:** +- Configuration format changed from flat to nested structure +- Old format: `{ port: 3000, cleanSlate: true }` +- New format: `{ server: { port: 3000 }, storage: { cleanSlate: true } }` + +**New Features:** +- โœจ Production configuration system with comprehensive options +- ๐Ÿ“Š Structured logging with multiple levels and formats +- ๐ŸŒ Full CORS middleware support +- ๐Ÿ” Simple static credentials authentication +- ๐Ÿ“ค Complete multipart upload support for large files +- ๐Ÿ”ง Flexible configuration with sensible defaults + +**Improvements:** +- Removed smartbucket from production dependencies (dev-only) +- Migrated to @push.rocks/smartfs for modern filesystem operations +- Enhanced error handling and logging throughout +- Better TypeScript types and documentation + ## License and Legal Information This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository. @@ -428,7 +593,7 @@ This project is owned and maintained by Task Venture Capital GmbH. The names and ### Company Information -Task Venture Capital GmbH +Task Venture Capital GmbH Registered at District court Bremen HRB 35230 HB, Germany For any legal inquiries or if you require further information, please contact us via email at hello@task.vc. diff --git a/ts/00_commitinfo_data.ts b/ts/00_commitinfo_data.ts index fc4057a..68360ac 100644 --- a/ts/00_commitinfo_data.ts +++ b/ts/00_commitinfo_data.ts @@ -3,6 +3,6 @@ */ export const commitinfo = { name: '@push.rocks/smarts3', - version: '4.0.0', + version: '5.0.0', description: 'A Node.js TypeScript package to create a local S3 endpoint for simulating AWS S3 operations using mapped local directories for development and testing purposes.' }