push.rocks/smarts3
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

BREAKING CHANGE(core): Production-ready S3-compatible server: nested config, multipart uploads, CORS, structured logging, SmartFS migration and improved error handling

Browse Source
This commit is contained in:
Jürgen Kunz
2025-11-23 22:46:42 +00:00
parent c074a5d2ed
commit 3c0a54e08b
3 changed files with 348 additions and 171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
changelog.md
View File

@@ -1,5 +1,17 @@
# Changelog
## 2025-11-23 - 5.0.0 - BREAKING CHANGE(core)
Production-ready S3-compatible server: nested config, multipart uploads, CORS, structured logging, SmartFS migration and improved error handling
- Breaking change: configuration format migrated from flat to nested structure (server, storage, auth, cors, logging, limits). Update existing configs accordingly.
- Implemented full multipart upload support (initiate, upload part, complete, abort) with on-disk part management and final assembly.
- Added CORS middleware with configurable origins, methods, headers, exposed headers, maxAge and credentials support.
- Structured, configurable logging (levels: error|warn|info|debug; formats: text|json) and request/response logging middleware.
- Simple static credential authentication middleware (configurable list of credentials).
- Migrated filesystem operations to @push.rocks/smartfs (Web Streams interoperability) and removed smartbucket from production dependencies.
- Improved S3-compatible error handling and XML responses (S3Error class and XML utilities).
- Exposed Smarts3Server and made store/multipart managers accessible for tests and advanced usage; added helper methods like getS3Descriptor and createBucket.
## 2025-11-23 - 4.0.0 - BREAKING CHANGE(Smarts3)
Migrate Smarts3 configuration to nested server/storage objects and remove legacy flat config support

503
readme.md
View File

@@ -1,16 +1,20 @@
# @push.rocks/smarts3 🚀
**Mock S3 made simple** - A powerful Node.js TypeScript package for creating a local S3 endpoint that simulates AWS S3 operations using mapped local directories. Perfect for development and testing!
**Production-ready S3-compatible server** - A powerful, lightweight Node.js TypeScript package that brings full S3 API compatibility to your local filesystem. Perfect for development, testing, and scenarios where running MinIO is out of scope!
## 🌟 Features
- 🏃 **Lightning-fast local S3 simulation** - No more waiting for cloud operations during development
- ⚡ **Native custom S3 server** - Built on Node.js http module with zero framework dependencies
- ⚡ **Production-ready architecture** - Built on Node.js http module with zero framework dependencies
- 🔄 **Full AWS S3 API compatibility** - Drop-in replacement for AWS SDK v3 and other S3 clients
- 📂 **Local directory mapping** - Your buckets live right on your filesystem with Windows-compatible encoding
- 📂 **Local directory mapping** - Your buckets live right on your filesystem
- 🔐 **Simple authentication** - Static credential-based auth for secure access
- 🌐 **CORS support** - Configurable cross-origin resource sharing
- 📊 **Structured logging** - Multiple levels (error/warn/info/debug) and formats (text/JSON)
- 📤 **Multipart uploads** - Full support for large file uploads (>5MB)
- 🧪 **Perfect for testing** - Reliable, repeatable tests without cloud dependencies
- 🎯 **TypeScript-first** - Built with TypeScript for excellent type safety and IDE support
- 🔧 **Zero configuration** - Works out of the box with sensible defaults
- 🔧 **Flexible configuration** - Comprehensive config system with sensible defaults
- 🧹 **Clean slate mode** - Start fresh on every test run
## Issue Reporting and Security
@@ -39,10 +43,15 @@ Get up and running in seconds:
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
// Start your local S3 server
// Start your local S3 server with minimal config
const s3Server = await Smarts3.createAndStart({
port: 3000,
cleanSlate: true, // Start with empty buckets
server: {
port: 3000,
silent: false,
},
storage: {
cleanSlate: true, // Start with empty buckets
},
});
// Create a bucket
@@ -55,44 +64,165 @@ const s3Config = await s3Server.getS3Descriptor();
await s3Server.stop();
```
## 📖 Detailed Usage Guide
## 📖 Configuration Guide
### 🏗️ Setting Up Your S3 Server
### Complete Configuration Options
The `Smarts3` class provides a simple interface for managing your local S3 server:
Smarts3 uses a comprehensive nested configuration structure:
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
import { Smarts3, ISmarts3Config } from '@push.rocks/smarts3';
// Configuration options
const config = {
port: 3000, // Port to run the server on (default: 3000)
cleanSlate: true, // Clear all data on start (default: false)
const config: ISmarts3Config = {
// Server configuration
server: {
port: 3000, // Port to listen on (default: 3000)
address: '0.0.0.0', // Bind address (default: '0.0.0.0')
silent: false, // Disable all console output (default: false)
},
// Storage configuration
storage: {
directory: './buckets', // Directory to store buckets (default: .nogit/bucketsDir)
cleanSlate: false, // Clear all data on start (default: false)
},
// Authentication configuration
auth: {
enabled: false, // Enable authentication (default: false)
credentials: [ // List of valid credentials
{
accessKeyId: 'YOUR_ACCESS_KEY',
secretAccessKey: 'YOUR_SECRET_KEY',
},
],
},
// CORS configuration
cors: {
enabled: false, // Enable CORS (default: false)
allowedOrigins: ['*'], // Allowed origins (default: ['*'])
allowedMethods: [ // Allowed HTTP methods
'GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'
],
allowedHeaders: ['*'], // Allowed headers (default: ['*'])
exposedHeaders: [ // Headers exposed to client
'ETag', 'x-amz-request-id', 'x-amz-version-id'
],
maxAge: 86400, // Preflight cache duration in seconds
allowCredentials: false, // Allow credentials (default: false)
},
// Logging configuration
logging: {
level: 'info', // Log level: 'error' | 'warn' | 'info' | 'debug'
format: 'text', // Log format: 'text' | 'json'
enabled: true, // Enable logging (default: true)
},
// Request limits
limits: {
maxObjectSize: 5 * 1024 * 1024 * 1024, // 5GB max object size
maxMetadataSize: 2048, // 2KB max metadata size
requestTimeout: 300000, // 5 minutes request timeout
},
};
// Create and start in one go
const s3Server = await Smarts3.createAndStart(config);
// Or create and start separately
const s3Server = new Smarts3(config);
await s3Server.start();
```
### 🪣 Working with Buckets
### Simple Configuration Examples
Creating and managing buckets is straightforward:
**Development Mode (Default)**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
storage: { cleanSlate: true },
});
```
**Production Mode with Auth**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
auth: {
enabled: true,
credentials: [
{
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
},
],
},
logging: {
level: 'warn',
format: 'json',
},
});
```
**CORS-Enabled for Web Apps**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
cors: {
enabled: true,
allowedOrigins: ['http://localhost:8080', 'https://app.example.com'],
allowCredentials: true,
},
});
```
## 🪣 Working with Buckets
### Creating Buckets
```typescript
// Create a new bucket
const bucket = await s3Server.createBucket('my-bucket');
// The bucket is now ready to use!
console.log(`Created bucket: ${bucket.name}`);
```
### 📤 Uploading Files
## 📤 File Operations
Use the powerful `SmartBucket` integration for file operations:
### Using AWS SDK v3
```typescript
import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
// Get connection config
const config = await s3Server.getS3Descriptor();
// Configure AWS SDK client
const s3Client = new S3Client({
endpoint: `http://${config.endpoint}:${config.port}`,
region: 'us-east-1',
credentials: {
accessKeyId: config.accessKey,
secretAccessKey: config.accessSecret,
},
forcePathStyle: true,
});
// Upload a file
await s3Client.send(new PutObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
Body: 'Hello from AWS SDK!',
ContentType: 'text/plain',
}));
// Download a file
const response = await s3Client.send(new GetObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
}));
const content = await response.Body.transformToString();
console.log(content); // "Hello from AWS SDK!"
```
### Using SmartBucket
```typescript
import { SmartBucket } from '@push.rocks/smartbucket';
@@ -102,63 +232,74 @@ const s3Config = await s3Server.getS3Descriptor();
// Create a SmartBucket instance
const smartbucket = new SmartBucket(s3Config);
// Get your bucket
const bucket = await smartbucket.getBucket('my-bucket');
// Upload a file
const baseDir = await bucket.getBaseDirectory();
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');
// Upload with more control
// Upload files
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');
await baseDir.fastPut({
path: 'documents/important.pdf',
contents: Buffer.from(yourPdfData),
});
```
### 📥 Downloading Files
Retrieve your files easily:
```typescript
// Get file contents as string
// Download files
const content = await baseDir.fastGet('path/to/file.txt');
console.log(content); // "Hello, S3! 🎉"
// Get file as Buffer
const buffer = await baseDir.fastGetBuffer('documents/important.pdf');
```
### 📋 Listing Files
Browse your bucket contents:
```typescript
// List all files in the bucket
// List files
const files = await baseDir.listFiles();
files.forEach((file) => {
console.log(`📄 ${file.name} (${file.size} bytes)`);
});
// List files with a specific prefix
const docs = await baseDir.listFiles('documents/');
// Delete files
await baseDir.fastDelete('old-file.txt');
```
### 🗑️ Deleting Files
## 📤 Multipart Uploads
Clean up when needed:
Smarts3 supports multipart uploads for large files (>5MB):
```typescript
// Delete a single file
await baseDir.fastDelete('old-file.txt');
import {
S3Client,
CreateMultipartUploadCommand,
UploadPartCommand,
CompleteMultipartUploadCommand
} from '@aws-sdk/client-s3';
// Delete multiple files
const filesToDelete = ['temp1.txt', 'temp2.txt', 'temp3.txt'];
for (const file of filesToDelete) {
await baseDir.fastDelete(file);
const s3Client = new S3Client(/* ... */);
// 1. Initiate multipart upload
const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
}));
// 2. Upload parts (in parallel if desired)
const parts = [];
for (let i = 0; i < numParts; i++) {
const part = await s3Client.send(new UploadPartCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
PartNumber: i + 1,
Body: partData[i],
}));
parts.push({
PartNumber: i + 1,
ETag: part.ETag,
});
}
// 3. Complete the upload
await s3Client.send(new CompleteMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
MultipartUpload: { Parts: parts },
}));
```
## 🧪 Testing Integration
@@ -173,8 +314,8 @@ describe('S3 Operations', () => {
beforeAll(async () => {
s3Server = await Smarts3.createAndStart({
port: 9999,
cleanSlate: true,
server: { port: 9999, silent: true },
storage: { cleanSlate: true },
});
});
@@ -200,8 +341,8 @@ describe('S3 Operations', () => {
before(async () => {
s3Server = await Smarts3.createAndStart({
port: 9999,
cleanSlate: true,
server: { port: 9999, silent: true },
storage: { cleanSlate: true },
});
});
@@ -216,40 +357,7 @@ describe('S3 Operations', () => {
});
```
## 🔌 AWS SDK Integration
Use `smarts3` with the official AWS SDK:
```typescript
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { Smarts3 } from '@push.rocks/smarts3';
// Start local S3
const s3Server = await Smarts3.createAndStart({ port: 3000 });
const config = await s3Server.getS3Descriptor();
// Configure AWS SDK
const s3Client = new S3Client({
endpoint: `http://${config.endpoint}:${config.port}`,
region: 'us-east-1',
credentials: {
accessKeyId: config.accessKey,
secretAccessKey: config.accessSecret,
},
forcePathStyle: true,
});
// Use AWS SDK as normal
const command = new PutObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
Body: 'Hello from AWS SDK!',
});
await s3Client.send(command);
```
## 🎯 Real-World Examples
## 🎯 Real-World Use Cases
### CI/CD Pipeline Testing
@@ -258,10 +366,13 @@ await s3Client.send(command);
import { Smarts3 } from '@push.rocks/smarts3';
export async function setupTestEnvironment() {
// Start S3 server for CI tests
const s3 = await Smarts3.createAndStart({
port: process.env.S3_PORT || 3000,
cleanSlate: true,
server: {
port: process.env.S3_PORT || 3000,
silent: true,
},
storage: { cleanSlate: true },
logging: { level: 'error' }, // Only log errors in CI
});
// Create test buckets
@@ -281,8 +392,15 @@ import { Smarts3 } from '@push.rocks/smarts3';
import express from 'express';
async function startDevelopmentServer() {
// Start local S3
const s3 = await Smarts3.createAndStart({ port: 3000 });
// Start local S3 with CORS for local development
const s3 = await Smarts3.createAndStart({
server: { port: 3000 },
cors: {
enabled: true,
allowedOrigins: ['http://localhost:8080'],
},
});
await s3.createBucket('user-uploads');
// Start your API server
@@ -302,13 +420,16 @@ async function startDevelopmentServer() {
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
import { SmartBucket } from '@push.rocks/smartbucket';
async function testDataMigration() {
const s3 = await Smarts3.createAndStart({ cleanSlate: true });
const s3 = await Smarts3.createAndStart({
storage: { cleanSlate: true },
});
// Create source and destination buckets
const sourceBucket = await s3.createBucket('legacy-data');
const destBucket = await s3.createBucket('new-data');
await s3.createBucket('legacy-data');
await s3.createBucket('new-data');
// Populate source with test data
const config = await s3.getS3Descriptor();
@@ -316,15 +437,8 @@ async function testDataMigration() {
const source = await smartbucket.getBucket('legacy-data');
const sourceDir = await source.getBaseDirectory();
// Add test files
await sourceDir.fastStore(
'user-1.json',
JSON.stringify({ id: 1, name: 'Alice' }),
);
await sourceDir.fastStore(
'user-2.json',
JSON.stringify({ id: 2, name: 'Bob' }),
);
await sourceDir.fastStore('user-1.json', JSON.stringify({ id: 1, name: 'Alice' }));
await sourceDir.fastStore('user-2.json', JSON.stringify({ id: 2, name: 'Bob' }));
// Run your migration logic
await runMigration(config);
@@ -338,77 +452,105 @@ async function testDataMigration() {
}
```
## 🛠️ Advanced Configuration
### Custom S3 Descriptor Options
When integrating with different S3 clients, you can customize the connection details:
```typescript
const customDescriptor = await s3Server.getS3Descriptor({
endpoint: 'localhost', // Custom endpoint
port: 3001, // Different port
useSsl: false, // SSL configuration
// Add any additional options your S3 client needs
});
```
### Environment-Based Configuration
```typescript
const config = {
port: parseInt(process.env.S3_PORT || '3000'),
cleanSlate: process.env.NODE_ENV === 'test',
};
const s3Server = await Smarts3.createAndStart(config);
```
## 🤝 Use Cases
- **🧪 Unit & Integration Testing** - Test S3 operations without AWS credentials or internet
- **🏗️ Local Development** - Develop cloud features offline with full S3 compatibility
- **📚 Teaching & Demos** - Perfect for workshops and tutorials without AWS setup
- **🔄 CI/CD Pipelines** - Reliable S3 operations in containerized test environments
- **🎭 Mocking & Stubbing** - Replace real S3 calls in test suites
- **📊 Data Migration Testing** - Safely test data migrations locally before production
## 🔧 API Reference
### Smarts3 Class
#### Constructor Options
#### Static Methods
```typescript
interface ISmarts3ContructorOptions {
port?: number; // Server port (default: 3000)
cleanSlate?: boolean; // Clear storage on start (default: false)
}
```
##### `createAndStart(config?: ISmarts3Config): Promise<Smarts3>`
#### Methods
Create and start a Smarts3 instance in one call.
- `static createAndStart(options)` - Create and start server in one call
- `start()` - Start the S3 server
- `stop()` - Stop the S3 server
- `createBucket(name)` - Create a new bucket
- `getS3Descriptor(options?)` - Get S3 connection configuration
**Parameters:**
- `config` - Optional configuration object (see Configuration Guide above)
**Returns:** Promise that resolves to a running Smarts3 instance
#### Instance Methods
##### `start(): Promise<void>`
Start the S3 server.
##### `stop(): Promise<void>`
Stop the S3 server and release resources.
##### `createBucket(name: string): Promise<{ name: string }>`
Create a new S3 bucket.
**Parameters:**
- `name` - Bucket name
**Returns:** Promise that resolves to bucket information
##### `getS3Descriptor(options?): Promise<IS3Descriptor>`
Get S3 connection configuration for use with S3 clients.
**Parameters:**
- `options` - Optional partial descriptor to merge with defaults
**Returns:** Promise that resolves to S3 descriptor with:
- `accessKey` - Access key for authentication
- `accessSecret` - Secret key for authentication
- `endpoint` - Server endpoint (hostname/IP)
- `port` - Server port
- `useSsl` - Whether to use SSL (always false for local server)
## 💡 Production Considerations
### When to Use Smarts3 vs MinIO
**Use Smarts3 when:**
- 🎯 You need a lightweight, zero-dependency S3 server
- 🧪 Running in CI/CD pipelines or containerized test environments
- 🏗️ Local development where MinIO setup is overkill
- 📦 Your application needs to bundle an S3-compatible server
- 🚀 Quick prototyping without infrastructure setup
**Use MinIO when:**
- 🏢 Production workloads requiring high availability
- 📊 Advanced features like versioning, replication, encryption at rest
- 🔐 Complex IAM policies and bucket policies
- 📈 High-performance requirements with multiple nodes
- 🌐 Multi-tenant environments
### Security Notes
- Smarts3's authentication is intentionally simple (static credentials)
- It does **not** implement AWS Signature V4 verification
- Perfect for development/testing, but not for production internet-facing deployments
- For production use, place behind a reverse proxy with proper authentication
## 🐛 Debugging Tips
1. **Enable verbose logging** - The server logs all operations by default
2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/`
1. **Enable debug logging**
```typescript
const s3 = await Smarts3.createAndStart({
logging: { level: 'debug', format: 'json' },
});
```
2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/` by default
3. **Use the correct endpoint** - Remember to use `127.0.0.1` or `localhost`
4. **Force path style** - Always use path-style URLs with local S3
4. **Force path style** - Always use `forcePathStyle: true` with local S3
5. **Inspect requests** - All requests are logged when `silent: false`
## 📈 Performance
`@push.rocks/smarts3` is optimized for development and testing:
Smarts3 is optimized for development and testing scenarios:
- ⚡ **Instant operations** - No network latency
- 💾 **Low memory footprint** - Efficient file system usage
- 💾 **Low memory footprint** - Efficient filesystem operations with streams
- 🔄 **Fast cleanup** - Clean slate mode for quick test resets
- 🚀 **Parallel operations** - Handle multiple requests simultaneously
- 🚀 **Parallel operations** - Handle multiple concurrent requests
- 📤 **Streaming uploads/downloads** - Low memory usage for large files
## 🔗 Related Packages
@@ -416,6 +558,29 @@ interface ISmarts3ContructorOptions {
- [`@push.rocks/smartfs`](https://www.npmjs.com/package/@push.rocks/smartfs) - Modern filesystem with Web Streams support
- [`@tsclass/tsclass`](https://www.npmjs.com/package/@tsclass/tsclass) - TypeScript class helpers
## 📝 Changelog
### v4.0.0 - Production Ready 🚀
**Breaking Changes:**
- Configuration format changed from flat to nested structure
- Old format: `{ port: 3000, cleanSlate: true }`
- New format: `{ server: { port: 3000 }, storage: { cleanSlate: true } }`
**New Features:**
- ✨ Production configuration system with comprehensive options
- 📊 Structured logging with multiple levels and formats
- 🌐 Full CORS middleware support
- 🔐 Simple static credentials authentication
- 📤 Complete multipart upload support for large files
- 🔧 Flexible configuration with sensible defaults
**Improvements:**
- Removed smartbucket from production dependencies (dev-only)
- Migrated to @push.rocks/smartfs for modern filesystem operations
- Enhanced error handling and logging throughout
- Better TypeScript types and documentation
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.

2
ts/00_commitinfo_data.ts
View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smarts3',
version: '4.0.0',
version: '5.0.0',
description: 'A Node.js TypeScript package to create a local S3 endpoint for simulating AWS S3 operations using mapped local directories for development and testing purposes.'
}