16 Commits
v3.0.3 ... main

Author SHA1 Message Date
54a0c2fb65 v5.1.0
Some checks failed
Default (tags) / security (push) Successful in 38s
Default (tags) / test (push) Failing after 37s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 23:31:26 +00:00
648ff98c2d feat(multipart): Implement full multipart upload support with persistent manager, periodic cleanup, and API integration 2025-11-23 23:31:26 +00:00
d6f178bde6 v5.0.2
Some checks failed
Default (tags) / security (push) Successful in 24s
Default (tags) / test (push) Failing after 35s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:53:39 +00:00
ffaef5cb15 fix(readme): Clarify contribution agreement requirement in README 2025-11-23 22:53:39 +00:00
d4cc1d43ea v5.0.1
Some checks failed
Default (tags) / security (push) Successful in 35s
Default (tags) / test (push) Failing after 35s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:52:19 +00:00
759becdd04 fix(docs): Clarify README wording about S3 compatibility and AWS SDK usage 2025-11-23 22:52:19 +00:00
51e8836227 v5.0.0
Some checks failed
Default (tags) / security (push) Successful in 25s
Default (tags) / test (push) Failing after 35s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:46:42 +00:00
3c0a54e08b BREAKING CHANGE(core): Production-ready S3-compatible server: nested config, multipart uploads, CORS, structured logging, SmartFS migration and improved error handling 2025-11-23 22:46:42 +00:00
c074a5d2ed v4.0.0
Some checks failed
Default (tags) / security (push) Successful in 36s
Default (tags) / test (push) Failing after 37s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:42:47 +00:00
a9ba9de6be BREAKING CHANGE(Smarts3): Migrate Smarts3 configuration to nested server/storage objects and remove legacy flat config support 2025-11-23 22:42:47 +00:00
263e7a58b9 v3.2.0
Some checks failed
Default (tags) / security (push) Successful in 25s
Default (tags) / test (push) Failing after 35s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:41:46 +00:00
74b81d7ba8 feat(multipart): Add multipart upload support with MultipartUploadManager and controller integration 2025-11-23 22:41:46 +00:00
0d4837184f v3.1.0
Some checks failed
Default (tags) / security (push) Successful in 38s
Default (tags) / test (push) Failing after 36s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:37:32 +00:00
7f3de92961 feat(logging): Add structured Logger and integrate into Smarts3Server; pass full config to server 2025-11-23 22:37:32 +00:00
a7bc902dd0 v3.0.4
Some checks failed
Default (tags) / security (push) Successful in 34s
Default (tags) / test (push) Failing after 36s
Default (tags) / release (push) Has been skipped
Default (tags) / metadata (push) Has been skipped
2025-11-23 22:31:44 +00:00
95d78d0d08 fix(smarts3): Use filesystem store for bucket creation and remove smartbucket runtime dependency 2025-11-23 22:31:44 +00:00
15 changed files with 1587 additions and 238 deletions

View File

@@ -1,5 +1,70 @@
# Changelog
## 2025-11-23 - 5.1.0 - feat(multipart)
Implement full multipart upload support with persistent manager, periodic cleanup, and API integration
- Add IMultipartConfig to server config with defaults (expirationDays: 7, cleanupIntervalMinutes: 60) and merge into existing config flow
- Introduce MultipartUploadManager: persistent upload metadata on disk, part upload/assembly, restore uploads on startup, listParts/listUploads, abort/cleanup functionality
- Start and stop multipart cleanup task from Smarts3Server lifecycle (startCleanupTask on start, stopCleanupTask on stop) with configurable interval and expiration
- ObjectController: support multipart endpoints (initiate, upload part, complete, abort) and move assembled final object into the object store on completion; set ETag headers and return proper XML responses
- BucketController: support listing in-progress multipart uploads via ?uploads query parameter and return S3-compatible XML
- Persist multipart state to disk and restore on initialization to survive restarts; perform automatic cleanup of expired uploads
## 2025-11-23 - 5.0.2 - fix(readme)
Clarify contribution agreement requirement in README
- Updated the Issue Reporting and Security section in readme.md to make it explicit that developers must sign and comply with the contribution agreement (and complete identification) before obtaining a code.foss.global account to submit pull requests.
## 2025-11-23 - 5.0.1 - fix(docs)
Clarify README wording about S3 compatibility and AWS SDK usage
- Update README wording to "Full S3 API compatibility" and clarify it works seamlessly with AWS SDK v3 and other S3 clients
## 2025-11-23 - 5.0.0 - BREAKING CHANGE(core)
Production-ready S3-compatible server: nested config, multipart uploads, CORS, structured logging, SmartFS migration and improved error handling
- Breaking change: configuration format migrated from flat to nested structure (server, storage, auth, cors, logging, limits). Update existing configs accordingly.
- Implemented full multipart upload support (initiate, upload part, complete, abort) with on-disk part management and final assembly.
- Added CORS middleware with configurable origins, methods, headers, exposed headers, maxAge and credentials support.
- Structured, configurable logging (levels: error|warn|info|debug; formats: text|json) and request/response logging middleware.
- Simple static credential authentication middleware (configurable list of credentials).
- Migrated filesystem operations to @push.rocks/smartfs (Web Streams interoperability) and removed smartbucket from production dependencies.
- Improved S3-compatible error handling and XML responses (S3Error class and XML utilities).
- Exposed Smarts3Server and made store/multipart managers accessible for tests and advanced usage; added helper methods like getS3Descriptor and createBucket.
## 2025-11-23 - 4.0.0 - BREAKING CHANGE(Smarts3)
Migrate Smarts3 configuration to nested server/storage objects and remove legacy flat config support
- Smarts3.createAndStart() and Smarts3 constructor now accept ISmarts3Config with nested `server` and `storage` objects.
- Removed support for the legacy flat config shape (top-level `port` and `cleanSlate`) / ILegacySmarts3Config.
- Updated tests to use new config shape (server:{ port, silent } and storage:{ cleanSlate }).
- mergeConfig and Smarts3Server now rely on the nested config shape; consumers must update their initialization code.
## 2025-11-23 - 3.2.0 - feat(multipart)
Add multipart upload support with MultipartUploadManager and controller integration
- Introduce MultipartUploadManager (ts/classes/multipart-manager.ts) to manage multipart upload lifecycle and store parts on disk
- Wire multipart manager into server and request context (S3Context, Smarts3Server) and initialize multipart storage on server start
- Add multipart-related routes and handlers in ObjectController: initiate (POST ?uploads), upload part (PUT ?partNumber&uploadId), complete (POST ?uploadId), and abort (DELETE ?uploadId)
- On complete, combine parts into final object and store via existing FilesystemStore workflow
- Expose multipart manager on Smarts3Server for controller access
## 2025-11-23 - 3.1.0 - feat(logging)
Add structured Logger and integrate into Smarts3Server; pass full config to server
- Introduce a new Logger class (ts/classes/logger.ts) providing leveled logging (error, warn, info, debug), text/json formats and an enable flag.
- Integrate Logger into Smarts3Server: use structured logging for server lifecycle events, HTTP request/response logging and S3 errors instead of direct console usage.
- Smarts3 now passes the full merged configuration into Smarts3Server (config.logging can control logging behavior).
- Server start/stop messages and internal request/error logs are emitted via the Logger and respect the configured logging level/format and silent option.
## 2025-11-23 - 3.0.4 - fix(smarts3)
Use filesystem store for bucket creation and remove smartbucket runtime dependency
- Switched createBucket to call the internal FilesystemStore.createBucket instead of using @push.rocks/smartbucket
- Made Smarts3Server.store public so Smarts3 can access the filesystem store directly
- Removed runtime import/export of @push.rocks/smartbucket from plugins and moved @push.rocks/smartbucket to devDependencies in package.json
- Updated createBucket to return a simple { name } object after creating the bucket via the filesystem store
## 2025-11-23 - 3.0.3 - fix(filesystem)
Migrate filesystem implementation to @push.rocks/smartfs and add Web Streams handling

View File

@@ -1,6 +1,6 @@
{
"name": "@push.rocks/smarts3",
"version": "3.0.3",
"version": "5.1.0",
"private": false,
"description": "A Node.js TypeScript package to create a local S3 endpoint for simulating AWS S3 operations using mapped local directories for development and testing purposes.",
"main": "dist_ts/index.js",
@@ -19,6 +19,7 @@
"@git.zone/tsbundle": "^2.5.2",
"@git.zone/tsrun": "^2.0.0",
"@git.zone/tstest": "^3.1.0",
"@push.rocks/smartbucket": "^4.3.0",
"@types/node": "^22.9.0"
},
"browserslist": [
@@ -37,7 +38,6 @@
"readme.md"
],
"dependencies": {
"@push.rocks/smartbucket": "^4.3.0",
"@push.rocks/smartfs": "^1.1.0",
"@push.rocks/smartpath": "^6.0.0",
"@push.rocks/smartxml": "^2.0.0",

6
pnpm-lock.yaml generated
View File

@@ -8,9 +8,6 @@ importers:
.:
dependencies:
'@push.rocks/smartbucket':
specifier: ^4.3.0
version: 4.3.0
'@push.rocks/smartfs':
specifier: ^1.1.0
version: 1.1.0
@@ -39,6 +36,9 @@ importers:
'@git.zone/tstest':
specifier: ^3.1.0
version: 3.1.0(socks@2.8.7)(typescript@5.9.3)
'@push.rocks/smartbucket':
specifier: ^4.3.0
version: 4.3.0
'@types/node':
specifier: ^22.9.0
version: 22.19.1

509
readme.md
View File

@@ -1,21 +1,25 @@
# @push.rocks/smarts3 🚀
**Mock S3 made simple** - A powerful Node.js TypeScript package for creating a local S3 endpoint that simulates AWS S3 operations using mapped local directories. Perfect for development and testing!
**Production-ready S3-compatible server** - A powerful, lightweight Node.js TypeScript package that brings full S3 API compatibility to your local filesystem. Perfect for development, testing, and scenarios where running MinIO is out of scope!
## 🌟 Features
- 🏃 **Lightning-fast local S3 simulation** - No more waiting for cloud operations during development
-**Native custom S3 server** - Built on Node.js http module with zero framework dependencies
- 🔄 **Full AWS S3 API compatibility** - Drop-in replacement for AWS SDK v3 and other S3 clients
- 📂 **Local directory mapping** - Your buckets live right on your filesystem with Windows-compatible encoding
-**Production-ready architecture** - Built on Node.js http module with zero framework dependencies
- 🔄 **Full S3 API compatibility** - Works seamlessly with AWS SDK v3 and any other S3 client
- 📂 **Local directory mapping** - Your buckets live right on your filesystem
- 🔐 **Simple authentication** - Static credential-based auth for secure access
- 🌐 **CORS support** - Configurable cross-origin resource sharing
- 📊 **Structured logging** - Multiple levels (error/warn/info/debug) and formats (text/JSON)
- 📤 **Multipart uploads** - Full support for large file uploads (>5MB)
- 🧪 **Perfect for testing** - Reliable, repeatable tests without cloud dependencies
- 🎯 **TypeScript-first** - Built with TypeScript for excellent type safety and IDE support
- 🔧 **Zero configuration** - Works out of the box with sensible defaults
- 🔧 **Flexible configuration** - Comprehensive config system with sensible defaults
- 🧹 **Clean slate mode** - Start fresh on every test run
## Issue Reporting and Security
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who want to sign a contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
## 📦 Installation
@@ -39,10 +43,15 @@ Get up and running in seconds:
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
// Start your local S3 server
// Start your local S3 server with minimal config
const s3Server = await Smarts3.createAndStart({
port: 3000,
cleanSlate: true, // Start with empty buckets
server: {
port: 3000,
silent: false,
},
storage: {
cleanSlate: true, // Start with empty buckets
},
});
// Create a bucket
@@ -55,44 +64,165 @@ const s3Config = await s3Server.getS3Descriptor();
await s3Server.stop();
```
## 📖 Detailed Usage Guide
## 📖 Configuration Guide
### 🏗️ Setting Up Your S3 Server
### Complete Configuration Options
The `Smarts3` class provides a simple interface for managing your local S3 server:
Smarts3 uses a comprehensive nested configuration structure:
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
import { Smarts3, ISmarts3Config } from '@push.rocks/smarts3';
// Configuration options
const config = {
port: 3000, // Port to run the server on (default: 3000)
cleanSlate: true, // Clear all data on start (default: false)
const config: ISmarts3Config = {
// Server configuration
server: {
port: 3000, // Port to listen on (default: 3000)
address: '0.0.0.0', // Bind address (default: '0.0.0.0')
silent: false, // Disable all console output (default: false)
},
// Storage configuration
storage: {
directory: './buckets', // Directory to store buckets (default: .nogit/bucketsDir)
cleanSlate: false, // Clear all data on start (default: false)
},
// Authentication configuration
auth: {
enabled: false, // Enable authentication (default: false)
credentials: [ // List of valid credentials
{
accessKeyId: 'YOUR_ACCESS_KEY',
secretAccessKey: 'YOUR_SECRET_KEY',
},
],
},
// CORS configuration
cors: {
enabled: false, // Enable CORS (default: false)
allowedOrigins: ['*'], // Allowed origins (default: ['*'])
allowedMethods: [ // Allowed HTTP methods
'GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'
],
allowedHeaders: ['*'], // Allowed headers (default: ['*'])
exposedHeaders: [ // Headers exposed to client
'ETag', 'x-amz-request-id', 'x-amz-version-id'
],
maxAge: 86400, // Preflight cache duration in seconds
allowCredentials: false, // Allow credentials (default: false)
},
// Logging configuration
logging: {
level: 'info', // Log level: 'error' | 'warn' | 'info' | 'debug'
format: 'text', // Log format: 'text' | 'json'
enabled: true, // Enable logging (default: true)
},
// Request limits
limits: {
maxObjectSize: 5 * 1024 * 1024 * 1024, // 5GB max object size
maxMetadataSize: 2048, // 2KB max metadata size
requestTimeout: 300000, // 5 minutes request timeout
},
};
// Create and start in one go
const s3Server = await Smarts3.createAndStart(config);
// Or create and start separately
const s3Server = new Smarts3(config);
await s3Server.start();
```
### 🪣 Working with Buckets
### Simple Configuration Examples
Creating and managing buckets is straightforward:
**Development Mode (Default)**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
storage: { cleanSlate: true },
});
```
**Production Mode with Auth**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
auth: {
enabled: true,
credentials: [
{
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
},
],
},
logging: {
level: 'warn',
format: 'json',
},
});
```
**CORS-Enabled for Web Apps**
```typescript
const s3Server = await Smarts3.createAndStart({
server: { port: 3000 },
cors: {
enabled: true,
allowedOrigins: ['http://localhost:8080', 'https://app.example.com'],
allowCredentials: true,
},
});
```
## 🪣 Working with Buckets
### Creating Buckets
```typescript
// Create a new bucket
const bucket = await s3Server.createBucket('my-bucket');
// The bucket is now ready to use!
console.log(`Created bucket: ${bucket.name}`);
```
### 📤 Uploading Files
## 📤 File Operations
Use the powerful `SmartBucket` integration for file operations:
### Using AWS SDK v3
```typescript
import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';
// Get connection config
const config = await s3Server.getS3Descriptor();
// Configure AWS SDK client
const s3Client = new S3Client({
endpoint: `http://${config.endpoint}:${config.port}`,
region: 'us-east-1',
credentials: {
accessKeyId: config.accessKey,
secretAccessKey: config.accessSecret,
},
forcePathStyle: true,
});
// Upload a file
await s3Client.send(new PutObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
Body: 'Hello from AWS SDK!',
ContentType: 'text/plain',
}));
// Download a file
const response = await s3Client.send(new GetObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
}));
const content = await response.Body.transformToString();
console.log(content); // "Hello from AWS SDK!"
```
### Using SmartBucket
```typescript
import { SmartBucket } from '@push.rocks/smartbucket';
@@ -102,63 +232,74 @@ const s3Config = await s3Server.getS3Descriptor();
// Create a SmartBucket instance
const smartbucket = new SmartBucket(s3Config);
// Get your bucket
const bucket = await smartbucket.getBucket('my-bucket');
// Upload a file
const baseDir = await bucket.getBaseDirectory();
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');
// Upload with more control
// Upload files
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');
await baseDir.fastPut({
path: 'documents/important.pdf',
contents: Buffer.from(yourPdfData),
});
```
### 📥 Downloading Files
Retrieve your files easily:
```typescript
// Get file contents as string
// Download files
const content = await baseDir.fastGet('path/to/file.txt');
console.log(content); // "Hello, S3! 🎉"
// Get file as Buffer
const buffer = await baseDir.fastGetBuffer('documents/important.pdf');
```
### 📋 Listing Files
Browse your bucket contents:
```typescript
// List all files in the bucket
// List files
const files = await baseDir.listFiles();
files.forEach((file) => {
console.log(`📄 ${file.name} (${file.size} bytes)`);
});
// List files with a specific prefix
const docs = await baseDir.listFiles('documents/');
// Delete files
await baseDir.fastDelete('old-file.txt');
```
### 🗑️ Deleting Files
## 📤 Multipart Uploads
Clean up when needed:
Smarts3 supports multipart uploads for large files (>5MB):
```typescript
// Delete a single file
await baseDir.fastDelete('old-file.txt');
import {
S3Client,
CreateMultipartUploadCommand,
UploadPartCommand,
CompleteMultipartUploadCommand
} from '@aws-sdk/client-s3';
// Delete multiple files
const filesToDelete = ['temp1.txt', 'temp2.txt', 'temp3.txt'];
for (const file of filesToDelete) {
await baseDir.fastDelete(file);
const s3Client = new S3Client(/* ... */);
// 1. Initiate multipart upload
const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
}));
// 2. Upload parts (in parallel if desired)
const parts = [];
for (let i = 0; i < numParts; i++) {
const part = await s3Client.send(new UploadPartCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
PartNumber: i + 1,
Body: partData[i],
}));
parts.push({
PartNumber: i + 1,
ETag: part.ETag,
});
}
// 3. Complete the upload
await s3Client.send(new CompleteMultipartUploadCommand({
Bucket: 'my-bucket',
Key: 'large-file.bin',
UploadId,
MultipartUpload: { Parts: parts },
}));
```
## 🧪 Testing Integration
@@ -173,8 +314,8 @@ describe('S3 Operations', () => {
beforeAll(async () => {
s3Server = await Smarts3.createAndStart({
port: 9999,
cleanSlate: true,
server: { port: 9999, silent: true },
storage: { cleanSlate: true },
});
});
@@ -200,8 +341,8 @@ describe('S3 Operations', () => {
before(async () => {
s3Server = await Smarts3.createAndStart({
port: 9999,
cleanSlate: true,
server: { port: 9999, silent: true },
storage: { cleanSlate: true },
});
});
@@ -216,40 +357,7 @@ describe('S3 Operations', () => {
});
```
## 🔌 AWS SDK Integration
Use `smarts3` with the official AWS SDK:
```typescript
import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { Smarts3 } from '@push.rocks/smarts3';
// Start local S3
const s3Server = await Smarts3.createAndStart({ port: 3000 });
const config = await s3Server.getS3Descriptor();
// Configure AWS SDK
const s3Client = new S3Client({
endpoint: `http://${config.endpoint}:${config.port}`,
region: 'us-east-1',
credentials: {
accessKeyId: config.accessKey,
secretAccessKey: config.accessSecret,
},
forcePathStyle: true,
});
// Use AWS SDK as normal
const command = new PutObjectCommand({
Bucket: 'my-bucket',
Key: 'test-file.txt',
Body: 'Hello from AWS SDK!',
});
await s3Client.send(command);
```
## 🎯 Real-World Examples
## 🎯 Real-World Use Cases
### CI/CD Pipeline Testing
@@ -258,10 +366,13 @@ await s3Client.send(command);
import { Smarts3 } from '@push.rocks/smarts3';
export async function setupTestEnvironment() {
// Start S3 server for CI tests
const s3 = await Smarts3.createAndStart({
port: process.env.S3_PORT || 3000,
cleanSlate: true,
server: {
port: process.env.S3_PORT || 3000,
silent: true,
},
storage: { cleanSlate: true },
logging: { level: 'error' }, // Only log errors in CI
});
// Create test buckets
@@ -281,8 +392,15 @@ import { Smarts3 } from '@push.rocks/smarts3';
import express from 'express';
async function startDevelopmentServer() {
// Start local S3
const s3 = await Smarts3.createAndStart({ port: 3000 });
// Start local S3 with CORS for local development
const s3 = await Smarts3.createAndStart({
server: { port: 3000 },
cors: {
enabled: true,
allowedOrigins: ['http://localhost:8080'],
},
});
await s3.createBucket('user-uploads');
// Start your API server
@@ -302,13 +420,16 @@ async function startDevelopmentServer() {
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
import { SmartBucket } from '@push.rocks/smartbucket';
async function testDataMigration() {
const s3 = await Smarts3.createAndStart({ cleanSlate: true });
const s3 = await Smarts3.createAndStart({
storage: { cleanSlate: true },
});
// Create source and destination buckets
const sourceBucket = await s3.createBucket('legacy-data');
const destBucket = await s3.createBucket('new-data');
await s3.createBucket('legacy-data');
await s3.createBucket('new-data');
// Populate source with test data
const config = await s3.getS3Descriptor();
@@ -316,15 +437,8 @@ async function testDataMigration() {
const source = await smartbucket.getBucket('legacy-data');
const sourceDir = await source.getBaseDirectory();
// Add test files
await sourceDir.fastStore(
'user-1.json',
JSON.stringify({ id: 1, name: 'Alice' }),
);
await sourceDir.fastStore(
'user-2.json',
JSON.stringify({ id: 2, name: 'Bob' }),
);
await sourceDir.fastStore('user-1.json', JSON.stringify({ id: 1, name: 'Alice' }));
await sourceDir.fastStore('user-2.json', JSON.stringify({ id: 2, name: 'Bob' }));
// Run your migration logic
await runMigration(config);
@@ -338,77 +452,105 @@ async function testDataMigration() {
}
```
## 🛠️ Advanced Configuration
### Custom S3 Descriptor Options
When integrating with different S3 clients, you can customize the connection details:
```typescript
const customDescriptor = await s3Server.getS3Descriptor({
endpoint: 'localhost', // Custom endpoint
port: 3001, // Different port
useSsl: false, // SSL configuration
// Add any additional options your S3 client needs
});
```
### Environment-Based Configuration
```typescript
const config = {
port: parseInt(process.env.S3_PORT || '3000'),
cleanSlate: process.env.NODE_ENV === 'test',
};
const s3Server = await Smarts3.createAndStart(config);
```
## 🤝 Use Cases
- **🧪 Unit & Integration Testing** - Test S3 operations without AWS credentials or internet
- **🏗️ Local Development** - Develop cloud features offline with full S3 compatibility
- **📚 Teaching & Demos** - Perfect for workshops and tutorials without AWS setup
- **🔄 CI/CD Pipelines** - Reliable S3 operations in containerized test environments
- **🎭 Mocking & Stubbing** - Replace real S3 calls in test suites
- **📊 Data Migration Testing** - Safely test data migrations locally before production
## 🔧 API Reference
### Smarts3 Class
#### Constructor Options
#### Static Methods
```typescript
interface ISmarts3ContructorOptions {
port?: number; // Server port (default: 3000)
cleanSlate?: boolean; // Clear storage on start (default: false)
}
```
##### `createAndStart(config?: ISmarts3Config): Promise<Smarts3>`
#### Methods
Create and start a Smarts3 instance in one call.
- `static createAndStart(options)` - Create and start server in one call
- `start()` - Start the S3 server
- `stop()` - Stop the S3 server
- `createBucket(name)` - Create a new bucket
- `getS3Descriptor(options?)` - Get S3 connection configuration
**Parameters:**
- `config` - Optional configuration object (see Configuration Guide above)
**Returns:** Promise that resolves to a running Smarts3 instance
#### Instance Methods
##### `start(): Promise<void>`
Start the S3 server.
##### `stop(): Promise<void>`
Stop the S3 server and release resources.
##### `createBucket(name: string): Promise<{ name: string }>`
Create a new S3 bucket.
**Parameters:**
- `name` - Bucket name
**Returns:** Promise that resolves to bucket information
##### `getS3Descriptor(options?): Promise<IS3Descriptor>`
Get S3 connection configuration for use with S3 clients.
**Parameters:**
- `options` - Optional partial descriptor to merge with defaults
**Returns:** Promise that resolves to S3 descriptor with:
- `accessKey` - Access key for authentication
- `accessSecret` - Secret key for authentication
- `endpoint` - Server endpoint (hostname/IP)
- `port` - Server port
- `useSsl` - Whether to use SSL (always false for local server)
## 💡 Production Considerations
### When to Use Smarts3 vs MinIO
**Use Smarts3 when:**
- 🎯 You need a lightweight, zero-dependency S3 server
- 🧪 Running in CI/CD pipelines or containerized test environments
- 🏗️ Local development where MinIO setup is overkill
- 📦 Your application needs to bundle an S3-compatible server
- 🚀 Quick prototyping without infrastructure setup
**Use MinIO when:**
- 🏢 Production workloads requiring high availability
- 📊 Advanced features like versioning, replication, encryption at rest
- 🔐 Complex IAM policies and bucket policies
- 📈 High-performance requirements with multiple nodes
- 🌐 Multi-tenant environments
### Security Notes
- Smarts3's authentication is intentionally simple (static credentials)
- It does **not** implement AWS Signature V4 verification
- Perfect for development/testing, but not for production internet-facing deployments
- For production use, place behind a reverse proxy with proper authentication
## 🐛 Debugging Tips
1. **Enable verbose logging** - The server logs all operations by default
2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/`
1. **Enable debug logging**
```typescript
const s3 = await Smarts3.createAndStart({
logging: { level: 'debug', format: 'json' },
});
```
2. **Check the buckets directory** - Find your data in `.nogit/bucketsDir/` by default
3. **Use the correct endpoint** - Remember to use `127.0.0.1` or `localhost`
4. **Force path style** - Always use path-style URLs with local S3
4. **Force path style** - Always use `forcePathStyle: true` with local S3
5. **Inspect requests** - All requests are logged when `silent: false`
## 📈 Performance
`@push.rocks/smarts3` is optimized for development and testing:
Smarts3 is optimized for development and testing scenarios:
- ⚡ **Instant operations** - No network latency
- 💾 **Low memory footprint** - Efficient file system usage
- 💾 **Low memory footprint** - Efficient filesystem operations with streams
- 🔄 **Fast cleanup** - Clean slate mode for quick test resets
- 🚀 **Parallel operations** - Handle multiple requests simultaneously
- 🚀 **Parallel operations** - Handle multiple concurrent requests
- 📤 **Streaming uploads/downloads** - Low memory usage for large files
## 🔗 Related Packages
@@ -416,6 +558,29 @@ interface ISmarts3ContructorOptions {
- [`@push.rocks/smartfs`](https://www.npmjs.com/package/@push.rocks/smartfs) - Modern filesystem with Web Streams support
- [`@tsclass/tsclass`](https://www.npmjs.com/package/@tsclass/tsclass) - TypeScript class helpers
## 📝 Changelog
### v4.0.0 - Production Ready 🚀
**Breaking Changes:**
- Configuration format changed from flat to nested structure
- Old format: `{ port: 3000, cleanSlate: true }`
- New format: `{ server: { port: 3000 }, storage: { cleanSlate: true } }`
**New Features:**
- ✨ Production configuration system with comprehensive options
- 📊 Structured logging with multiple levels and formats
- 🌐 Full CORS middleware support
- 🔐 Simple static credentials authentication
- 📤 Complete multipart upload support for large files
- 🔧 Flexible configuration with sensible defaults
**Improvements:**
- Removed smartbucket from production dependencies (dev-only)
- Migrated to @push.rocks/smartfs for modern filesystem operations
- Enhanced error handling and logging throughout
- Better TypeScript types and documentation
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
@@ -428,7 +593,7 @@ This project is owned and maintained by Task Venture Capital GmbH. The names and
### Company Information
Task Venture Capital GmbH
Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

View File

@@ -18,9 +18,13 @@ async function streamToString(stream: Readable): Promise<string> {
tap.test('should start the S3 server and configure client', async () => {
testSmarts3Instance = await smarts3.Smarts3.createAndStart({
port: 3337,
cleanSlate: true,
silent: true,
server: {
port: 3337,
silent: true,
},
storage: {
cleanSlate: true,
},
});
const descriptor = await testSmarts3Instance.getS3Descriptor();

View File

@@ -7,8 +7,12 @@ let testSmarts3Instance: smarts3.Smarts3;
tap.test('should create a smarts3 instance and run it', async (toolsArg) => {
testSmarts3Instance = await smarts3.Smarts3.createAndStart({
port: 3333,
cleanSlate: true,
server: {
port: 3333,
},
storage: {
cleanSlate: true,
},
});
console.log(`Let the instance run for 2 seconds`);
await toolsArg.delayFor(2000);

View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smarts3',
version: '3.0.3',
version: '5.1.0',
description: 'A Node.js TypeScript package to create a local S3 endpoint for simulating AWS S3 operations using mapped local directories for development and testing purposes.'
}

View File

@@ -2,6 +2,7 @@ import * as plugins from '../plugins.js';
import { S3Error } from './s3-error.js';
import { createXml } from '../utils/xml.utils.js';
import type { FilesystemStore } from './filesystem-store.js';
import type { MultipartUploadManager } from './multipart-manager.js';
import type { Readable } from 'stream';
/**
@@ -14,6 +15,7 @@ export class S3Context {
public params: Record<string, string> = {};
public query: Record<string, string> = {};
public store: FilesystemStore;
public multipart: MultipartUploadManager;
private req: plugins.http.IncomingMessage;
private res: plugins.http.ServerResponse;
@@ -23,11 +25,13 @@ export class S3Context {
constructor(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
store: FilesystemStore
store: FilesystemStore,
multipart: MultipartUploadManager
) {
this.req = req;
this.res = res;
this.store = store;
this.multipart = multipart;
this.method = req.method || 'GET';
this.headers = req.headers;

130
ts/classes/logger.ts Normal file
View File

@@ -0,0 +1,130 @@
import type { ILoggingConfig } from '../index.js';
/**
* Log levels in order of severity
*/
const LOG_LEVELS = {
error: 0,
warn: 1,
info: 2,
debug: 3,
} as const;
type LogLevel = keyof typeof LOG_LEVELS;
/**
* Structured logger with configurable levels and formats
*/
export class Logger {
private config: Required<ILoggingConfig>;
private minLevel: number;
constructor(config: ILoggingConfig) {
// Apply defaults for any missing config
this.config = {
level: config.level ?? 'info',
format: config.format ?? 'text',
enabled: config.enabled ?? true,
};
this.minLevel = LOG_LEVELS[this.config.level];
}
/**
* Check if a log level should be output
*/
private shouldLog(level: LogLevel): boolean {
if (!this.config.enabled) {
return false;
}
return LOG_LEVELS[level] <= this.minLevel;
}
/**
* Format a log message
*/
private format(level: LogLevel, message: string, meta?: Record<string, any>): string {
const timestamp = new Date().toISOString();
if (this.config.format === 'json') {
return JSON.stringify({
timestamp,
level,
message,
...(meta || {}),
});
}
// Text format
const metaStr = meta ? ` ${JSON.stringify(meta)}` : '';
return `[${timestamp}] ${level.toUpperCase()}: ${message}${metaStr}`;
}
/**
* Log at error level
*/
public error(message: string, meta?: Record<string, any>): void {
if (this.shouldLog('error')) {
console.error(this.format('error', message, meta));
}
}
/**
* Log at warn level
*/
public warn(message: string, meta?: Record<string, any>): void {
if (this.shouldLog('warn')) {
console.warn(this.format('warn', message, meta));
}
}
/**
* Log at info level
*/
public info(message: string, meta?: Record<string, any>): void {
if (this.shouldLog('info')) {
console.log(this.format('info', message, meta));
}
}
/**
* Log at debug level
*/
public debug(message: string, meta?: Record<string, any>): void {
if (this.shouldLog('debug')) {
console.log(this.format('debug', message, meta));
}
}
/**
* Log HTTP request
*/
public request(method: string, url: string, meta?: Record<string, any>): void {
this.info(`${method} ${url}`, meta);
}
/**
* Log HTTP response
*/
public response(method: string, url: string, statusCode: number, duration: number): void {
const level: LogLevel = statusCode >= 500 ? 'error' : statusCode >= 400 ? 'warn' : 'info';
if (this.shouldLog(level)) {
const message = `${method} ${url} - ${statusCode} (${duration}ms)`;
if (level === 'error') {
this.error(message, { statusCode, duration });
} else if (level === 'warn') {
this.warn(message, { statusCode, duration });
} else {
this.info(message, { statusCode, duration });
}
}
}
/**
* Log S3 error
*/
public s3Error(code: string, message: string, status: number): void {
this.error(`[S3Error] ${code}: ${message}`, { code, status });
}
}

View File

@@ -0,0 +1,430 @@
import * as plugins from '../plugins.js';
import { Readable } from 'stream';
/**
* Multipart upload metadata
*/
export interface IMultipartUpload {
uploadId: string;
bucket: string;
key: string;
initiated: Date;
parts: Map<number, IPartInfo>;
metadata: Record<string, string>;
}
/**
* Part information
*/
export interface IPartInfo {
partNumber: number;
etag: string;
size: number;
lastModified: Date;
}
/**
* Serializable version of upload metadata for disk persistence
*/
interface ISerializableUpload {
uploadId: string;
bucket: string;
key: string;
initiated: string; // ISO date string
metadata: Record<string, string>;
parts: Array<{
partNumber: number;
etag: string;
size: number;
lastModified: string; // ISO date string
}>;
}
/**
* Manages multipart upload state and storage
*/
export class MultipartUploadManager {
private uploads: Map<string, IMultipartUpload> = new Map();
private uploadDir: string;
private cleanupInterval: NodeJS.Timeout | null = null;
private expirationDays: number;
private cleanupIntervalMinutes: number;
constructor(
private rootDir: string,
expirationDays: number = 7,
cleanupIntervalMinutes: number = 60
) {
this.uploadDir = plugins.path.join(rootDir, '.multipart');
this.expirationDays = expirationDays;
this.cleanupIntervalMinutes = cleanupIntervalMinutes;
}
/**
* Initialize multipart uploads directory
*/
public async initialize(): Promise<void> {
await plugins.smartfs.directory(this.uploadDir).recursive().create();
await this.restoreUploadsFromDisk();
}
/**
* Save upload metadata to disk for persistence
*/
private async saveUploadMetadata(uploadId: string): Promise<void> {
const upload = this.uploads.get(uploadId);
if (!upload) {
return;
}
const metadataPath = plugins.path.join(this.uploadDir, uploadId, 'metadata.json');
const serializable: ISerializableUpload = {
uploadId: upload.uploadId,
bucket: upload.bucket,
key: upload.key,
initiated: upload.initiated.toISOString(),
metadata: upload.metadata,
parts: Array.from(upload.parts.values()).map(part => ({
partNumber: part.partNumber,
etag: part.etag,
size: part.size,
lastModified: part.lastModified.toISOString(),
})),
};
await plugins.smartfs.file(metadataPath).write(JSON.stringify(serializable, null, 2));
}
/**
* Restore uploads from disk on initialization
*/
private async restoreUploadsFromDisk(): Promise<void> {
const uploadDirExists = await plugins.smartfs.directory(this.uploadDir).exists();
if (!uploadDirExists) {
return;
}
const entries = await plugins.smartfs.directory(this.uploadDir).includeStats().list();
for (const entry of entries) {
if (!entry.isDirectory) {
continue;
}
const uploadId = entry.name;
const metadataPath = plugins.path.join(this.uploadDir, uploadId, 'metadata.json');
// Check if metadata.json exists
const metadataExists = await plugins.smartfs.file(metadataPath).exists();
if (!metadataExists) {
// Orphaned upload directory - clean it up
console.warn(`Orphaned multipart upload directory found: ${uploadId}, cleaning up`);
await plugins.smartfs.directory(plugins.path.join(this.uploadDir, uploadId)).recursive().delete();
continue;
}
try {
// Read and parse metadata
const metadataContent = await plugins.smartfs.file(metadataPath).read();
const serialized: ISerializableUpload = JSON.parse(metadataContent as string);
// Restore to memory
const parts = new Map<number, IPartInfo>();
for (const part of serialized.parts) {
parts.set(part.partNumber, {
partNumber: part.partNumber,
etag: part.etag,
size: part.size,
lastModified: new Date(part.lastModified),
});
}
this.uploads.set(uploadId, {
uploadId: serialized.uploadId,
bucket: serialized.bucket,
key: serialized.key,
initiated: new Date(serialized.initiated),
parts,
metadata: serialized.metadata,
});
console.log(`Restored multipart upload: ${uploadId} (${serialized.bucket}/${serialized.key})`);
} catch (error) {
// Corrupted metadata - clean up
console.error(`Failed to restore multipart upload ${uploadId}:`, error);
await plugins.smartfs.directory(plugins.path.join(this.uploadDir, uploadId)).recursive().delete();
}
}
}
/**
* Generate a unique upload ID
*/
private generateUploadId(): string {
return plugins.crypto.randomBytes(16).toString('hex');
}
/**
* Initiate a new multipart upload
*/
public async initiateUpload(
bucket: string,
key: string,
metadata: Record<string, string>
): Promise<string> {
const uploadId = this.generateUploadId();
this.uploads.set(uploadId, {
uploadId,
bucket,
key,
initiated: new Date(),
parts: new Map(),
metadata,
});
// Create directory for this upload's parts
const uploadPath = plugins.path.join(this.uploadDir, uploadId);
await plugins.smartfs.directory(uploadPath).recursive().create();
// Persist metadata to disk
await this.saveUploadMetadata(uploadId);
return uploadId;
}
/**
* Upload a part
*/
public async uploadPart(
uploadId: string,
partNumber: number,
stream: Readable
): Promise<IPartInfo> {
const upload = this.uploads.get(uploadId);
if (!upload) {
throw new Error('No such upload');
}
const partPath = plugins.path.join(this.uploadDir, uploadId, `part-${partNumber}`);
// Write part to disk
const webWriteStream = await plugins.smartfs.file(partPath).writeStream();
const writer = webWriteStream.getWriter();
let size = 0;
const hash = plugins.crypto.createHash('md5');
for await (const chunk of stream) {
const buffer = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk);
await writer.write(new Uint8Array(buffer));
hash.update(buffer);
size += buffer.length;
}
await writer.close();
const etag = hash.digest('hex');
const partInfo: IPartInfo = {
partNumber,
etag,
size,
lastModified: new Date(),
};
upload.parts.set(partNumber, partInfo);
// Persist updated metadata
await this.saveUploadMetadata(uploadId);
return partInfo;
}
/**
* Complete multipart upload - combine all parts
*/
public async completeUpload(
uploadId: string,
parts: Array<{ PartNumber: number; ETag: string }>
): Promise<{ etag: string; size: number }> {
const upload = this.uploads.get(uploadId);
if (!upload) {
throw new Error('No such upload');
}
// Verify all parts are uploaded
for (const part of parts) {
const uploadedPart = upload.parts.get(part.PartNumber);
if (!uploadedPart) {
throw new Error(`Part ${part.PartNumber} not uploaded`);
}
// Normalize ETag format (remove quotes if present)
const normalizedETag = part.ETag.replace(/"/g, '');
if (uploadedPart.etag !== normalizedETag) {
throw new Error(`Part ${part.PartNumber} ETag mismatch`);
}
}
// Sort parts by part number
const sortedParts = parts.sort((a, b) => a.PartNumber - b.PartNumber);
// Combine parts into final object
const finalPath = plugins.path.join(this.uploadDir, uploadId, 'final');
const webWriteStream = await plugins.smartfs.file(finalPath).writeStream();
const writer = webWriteStream.getWriter();
const hash = plugins.crypto.createHash('md5');
let totalSize = 0;
for (const part of sortedParts) {
const partPath = plugins.path.join(this.uploadDir, uploadId, `part-${part.PartNumber}`);
// Read part and write to final file
const partContent = await plugins.smartfs.file(partPath).read();
const buffer = Buffer.isBuffer(partContent) ? partContent : Buffer.from(partContent as string);
await writer.write(new Uint8Array(buffer));
hash.update(buffer);
totalSize += buffer.length;
}
await writer.close();
const etag = hash.digest('hex');
return { etag, size: totalSize };
}
/**
* Get the final combined file path
*/
public getFinalPath(uploadId: string): string {
return plugins.path.join(this.uploadDir, uploadId, 'final');
}
/**
* Get upload metadata
*/
public getUpload(uploadId: string): IMultipartUpload | undefined {
return this.uploads.get(uploadId);
}
/**
* Abort multipart upload - clean up parts
*/
public async abortUpload(uploadId: string): Promise<void> {
const upload = this.uploads.get(uploadId);
if (!upload) {
throw new Error('No such upload');
}
// Delete upload directory
const uploadPath = plugins.path.join(this.uploadDir, uploadId);
await plugins.smartfs.directory(uploadPath).recursive().delete();
// Remove from memory
this.uploads.delete(uploadId);
}
/**
* Clean up upload after completion
*/
public async cleanupUpload(uploadId: string): Promise<void> {
const uploadPath = plugins.path.join(this.uploadDir, uploadId);
await plugins.smartfs.directory(uploadPath).recursive().delete();
this.uploads.delete(uploadId);
}
/**
* List all in-progress uploads for a bucket
*/
public listUploads(bucket?: string): IMultipartUpload[] {
const uploads = Array.from(this.uploads.values());
if (bucket) {
return uploads.filter((u) => u.bucket === bucket);
}
return uploads;
}
/**
* List parts for an upload
*/
public listParts(uploadId: string): IPartInfo[] {
const upload = this.uploads.get(uploadId);
if (!upload) {
throw new Error('No such upload');
}
return Array.from(upload.parts.values()).sort((a, b) => a.partNumber - b.partNumber);
}
/**
* Start automatic cleanup task for expired uploads
*/
public startCleanupTask(): void {
if (this.cleanupInterval) {
console.warn('Cleanup task is already running');
return;
}
// Run cleanup immediately on start
this.performCleanup().catch(err => {
console.error('Failed to perform initial multipart cleanup:', err);
});
// Then schedule periodic cleanup
const intervalMs = this.cleanupIntervalMinutes * 60 * 1000;
this.cleanupInterval = setInterval(() => {
this.performCleanup().catch(err => {
console.error('Failed to perform scheduled multipart cleanup:', err);
});
}, intervalMs);
console.log(`Multipart cleanup task started (interval: ${this.cleanupIntervalMinutes} minutes, expiration: ${this.expirationDays} days)`);
}
/**
* Stop automatic cleanup task
*/
public stopCleanupTask(): void {
if (this.cleanupInterval) {
clearInterval(this.cleanupInterval);
this.cleanupInterval = null;
console.log('Multipart cleanup task stopped');
}
}
/**
* Perform cleanup of expired uploads
*/
private async performCleanup(): Promise<void> {
const now = Date.now();
const expirationMs = this.expirationDays * 24 * 60 * 60 * 1000;
const expiredUploads: string[] = [];
// Find expired uploads
for (const [uploadId, upload] of this.uploads.entries()) {
const age = now - upload.initiated.getTime();
if (age > expirationMs) {
expiredUploads.push(uploadId);
}
}
if (expiredUploads.length === 0) {
return;
}
console.log(`Cleaning up ${expiredUploads.length} expired multipart upload(s)`);
// Delete expired uploads
for (const uploadId of expiredUploads) {
try {
await this.abortUpload(uploadId);
console.log(`Deleted expired multipart upload: ${uploadId}`);
} catch (err) {
console.error(`Failed to delete expired upload ${uploadId}:`, err);
}
}
}
}

View File

@@ -4,9 +4,12 @@ import { MiddlewareStack } from './middleware-stack.js';
import { S3Context } from './context.js';
import { FilesystemStore } from './filesystem-store.js';
import { S3Error } from './s3-error.js';
import { Logger } from './logger.js';
import { MultipartUploadManager } from './multipart-manager.js';
import { ServiceController } from '../controllers/service.controller.js';
import { BucketController } from '../controllers/bucket.controller.js';
import { ObjectController } from '../controllers/object.controller.js';
import type { ISmarts3Config } from '../index.js';
export interface ISmarts3ServerOptions {
port?: number;
@@ -14,6 +17,7 @@ export interface ISmarts3ServerOptions {
directory?: string;
cleanSlate?: boolean;
silent?: boolean;
config?: Required<ISmarts3Config>;
}
/**
@@ -24,20 +28,69 @@ export class Smarts3Server {
private httpServer?: plugins.http.Server;
private router: S3Router;
private middlewares: MiddlewareStack;
private store: FilesystemStore;
private options: Required<ISmarts3ServerOptions>;
public store: FilesystemStore; // Made public for direct access from Smarts3 class
public multipart: MultipartUploadManager; // Made public for controller access
private options: Required<Omit<ISmarts3ServerOptions, 'config'>>;
private config: Required<ISmarts3Config>;
private logger: Logger;
constructor(options: ISmarts3ServerOptions = {}) {
this.options = {
port: 3000,
address: '0.0.0.0',
directory: plugins.path.join(process.cwd(), '.nogit/bucketsDir'),
cleanSlate: false,
silent: false,
...options,
port: options.port ?? 3000,
address: options.address ?? '0.0.0.0',
directory: options.directory ?? plugins.path.join(process.cwd(), '.nogit/bucketsDir'),
cleanSlate: options.cleanSlate ?? false,
silent: options.silent ?? false,
};
// Store config for middleware and feature configuration
// If no config provided, create minimal default (for backward compatibility)
this.config = options.config ?? {
server: {
port: this.options.port,
address: this.options.address,
silent: this.options.silent,
},
storage: {
directory: this.options.directory,
cleanSlate: this.options.cleanSlate,
},
auth: {
enabled: false,
credentials: [{ accessKeyId: 'S3RVER', secretAccessKey: 'S3RVER' }],
},
cors: {
enabled: false,
allowedOrigins: ['*'],
allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'],
allowedHeaders: ['*'],
exposedHeaders: ['ETag', 'x-amz-request-id', 'x-amz-version-id'],
maxAge: 86400,
allowCredentials: false,
},
logging: {
level: 'info',
format: 'text',
enabled: true,
},
limits: {
maxObjectSize: 5 * 1024 * 1024 * 1024,
maxMetadataSize: 2048,
requestTimeout: 300000,
},
multipart: {
expirationDays: 7,
cleanupIntervalMinutes: 60,
},
};
this.logger = new Logger(this.config.logging);
this.store = new FilesystemStore(this.options.directory);
this.multipart = new MultipartUploadManager(
this.options.directory,
this.config.multipart.expirationDays,
this.config.multipart.cleanupIntervalMinutes
);
this.router = new S3Router();
this.middlewares = new MiddlewareStack();
@@ -49,20 +102,118 @@ export class Smarts3Server {
* Setup middleware stack
*/
private setupMiddlewares(): void {
// Logger middleware
if (!this.options.silent) {
// CORS middleware (must be first to handle preflight requests)
if (this.config.cors.enabled) {
this.middlewares.use(async (req, res, ctx, next) => {
const start = Date.now();
console.log(`${req.method} ${req.url}`);
console.log(` Headers:`, JSON.stringify(req.headers, null, 2).slice(0, 200));
const origin = req.headers.origin || req.headers.referer;
// Check if origin is allowed
const allowedOrigins = this.config.cors.allowedOrigins || ['*'];
const isOriginAllowed =
allowedOrigins.includes('*') ||
(origin && allowedOrigins.includes(origin));
if (isOriginAllowed) {
// Set CORS headers
res.setHeader(
'Access-Control-Allow-Origin',
allowedOrigins.includes('*') ? '*' : origin || '*'
);
if (this.config.cors.allowCredentials) {
res.setHeader('Access-Control-Allow-Credentials', 'true');
}
// Handle preflight OPTIONS request
if (req.method === 'OPTIONS') {
res.setHeader(
'Access-Control-Allow-Methods',
(this.config.cors.allowedMethods || []).join(', ')
);
res.setHeader(
'Access-Control-Allow-Headers',
(this.config.cors.allowedHeaders || []).join(', ')
);
if (this.config.cors.maxAge) {
res.setHeader(
'Access-Control-Max-Age',
String(this.config.cors.maxAge)
);
}
res.writeHead(204);
res.end();
return; // Don't call next() for OPTIONS
}
// Set exposed headers for actual requests
if (this.config.cors.exposedHeaders && this.config.cors.exposedHeaders.length > 0) {
res.setHeader(
'Access-Control-Expose-Headers',
this.config.cors.exposedHeaders.join(', ')
);
}
}
await next();
const duration = Date.now() - start;
console.log(`${req.method} ${req.url} - ${res.statusCode} (${duration}ms)`);
});
}
// TODO: Add authentication middleware
// TODO: Add CORS middleware
// Authentication middleware (simple static credentials)
if (this.config.auth.enabled) {
this.middlewares.use(async (req, res, ctx, next) => {
const authHeader = req.headers.authorization;
// Extract access key from Authorization header
let accessKeyId: string | undefined;
if (authHeader) {
// Support multiple auth formats:
// 1. AWS accessKeyId:signature
// 2. AWS4-HMAC-SHA256 Credential=accessKeyId/date/region/service/aws4_request, ...
if (authHeader.startsWith('AWS ')) {
accessKeyId = authHeader.substring(4).split(':')[0];
} else if (authHeader.startsWith('AWS4-HMAC-SHA256')) {
const credentialMatch = authHeader.match(/Credential=([^/]+)\//);
accessKeyId = credentialMatch ? credentialMatch[1] : undefined;
}
}
// Check if access key is valid
const isValid = this.config.auth.credentials.some(
(cred) => cred.accessKeyId === accessKeyId
);
if (!isValid) {
ctx.throw('AccessDenied', 'Access Denied');
return;
}
await next();
});
}
// Logger middleware
if (!this.options.silent && this.config.logging.enabled) {
this.middlewares.use(async (req, res, ctx, next) => {
const start = Date.now();
// Log request
this.logger.request(req.method || 'UNKNOWN', req.url || '/', {
headers: req.headers,
});
await next();
// Log response
const duration = Date.now() - start;
this.logger.response(
req.method || 'UNKNOWN',
req.url || '/',
res.statusCode || 500,
duration
);
});
}
}
/**
@@ -80,6 +231,7 @@ export class Smarts3Server {
// Object level (/:bucket/:key*)
this.router.put('/:bucket/:key*', ObjectController.putObject);
this.router.post('/:bucket/:key*', ObjectController.postObject); // For multipart operations
this.router.get('/:bucket/:key*', ObjectController.getObject);
this.router.head('/:bucket/:key*', ObjectController.headObject);
this.router.delete('/:bucket/:key*', ObjectController.deleteObject);
@@ -92,7 +244,7 @@ export class Smarts3Server {
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse
): Promise<void> {
const context = new S3Context(req, res, this.store);
const context = new S3Context(req, res, this.store, this.multipart);
try {
// Execute middleware stack
@@ -122,11 +274,14 @@ export class Smarts3Server {
): Promise<void> {
const s3Error = err instanceof S3Error ? err : S3Error.fromError(err);
if (!this.options.silent) {
console.error(`[S3Error] ${s3Error.code}: ${s3Error.message}`);
if (s3Error.status >= 500) {
console.error(err.stack || err);
}
// Log the error
this.logger.s3Error(s3Error.code, s3Error.message, s3Error.status);
// Log stack trace for server errors
if (s3Error.status >= 500) {
this.logger.debug('Error stack trace', {
stack: err.stack || err.toString(),
});
}
// Send error response
@@ -147,6 +302,12 @@ export class Smarts3Server {
// Initialize store
await this.store.initialize();
// Initialize multipart upload manager
await this.multipart.initialize();
// Start multipart cleanup task
this.multipart.startCleanupTask();
// Clean slate if requested
if (this.options.cleanSlate) {
await this.store.reset();
@@ -155,7 +316,10 @@ export class Smarts3Server {
// Create HTTP server
this.httpServer = plugins.http.createServer((req, res) => {
this.handleRequest(req, res).catch((err) => {
console.error('Fatal error in request handler:', err);
this.logger.error('Fatal error in request handler', {
error: err.message,
stack: err.stack,
});
if (!res.headersSent) {
res.writeHead(500, { 'Content-Type': 'text/plain' });
res.end('Internal Server Error');
@@ -169,9 +333,7 @@ export class Smarts3Server {
if (err) {
reject(err);
} else {
if (!this.options.silent) {
console.log(`S3 server listening on ${this.options.address}:${this.options.port}`);
}
this.logger.info(`S3 server listening on ${this.options.address}:${this.options.port}`);
resolve();
}
});
@@ -186,14 +348,15 @@ export class Smarts3Server {
return;
}
// Stop multipart cleanup task
this.multipart.stopCleanupTask();
await new Promise<void>((resolve, reject) => {
this.httpServer!.close((err?: Error) => {
if (err) {
reject(err);
} else {
if (!this.options.silent) {
console.log('S3 server stopped');
}
this.logger.info('S3 server stopped');
resolve();
}
});

View File

@@ -54,8 +54,9 @@ export class BucketController {
}
/**
* GET /:bucket - List objects
* GET /:bucket - List objects or multipart uploads
* Supports both V1 and V2 listing (V2 uses list-type=2 query param)
* Multipart uploads listing is triggered by ?uploads query parameter
*/
public static async listObjects(
req: plugins.http.IncomingMessage,
@@ -64,6 +65,12 @@ export class BucketController {
params: Record<string, string>
): Promise<void> {
const { bucket } = params;
// Check if this is a ListMultipartUploads request
if (ctx.query.uploads !== undefined) {
return BucketController.listMultipartUploads(req, res, ctx, params);
}
const isV2 = ctx.query['list-type'] === '2';
const result = await ctx.store.listObjects(bucket, {
@@ -127,4 +134,47 @@ export class BucketController {
});
}
}
/**
* GET /:bucket?uploads - List multipart uploads
*/
private static async listMultipartUploads(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
const { bucket } = params;
// Get all multipart uploads for this bucket
const uploads = ctx.multipart.listUploads(bucket);
// Build XML response
await ctx.sendXML({
ListMultipartUploadsResult: {
'@_xmlns': 'http://s3.amazonaws.com/doc/2006-03-01/',
Bucket: bucket,
KeyMarker: '',
UploadIdMarker: '',
MaxUploads: 1000,
IsTruncated: false,
...(uploads.length > 0 && {
Upload: uploads.map((upload) => ({
Key: upload.key,
UploadId: upload.uploadId,
Initiator: {
ID: 'S3RVER',
DisplayName: 'S3RVER',
},
Owner: {
ID: 'S3RVER',
DisplayName: 'S3RVER',
},
StorageClass: 'STANDARD',
Initiated: upload.initiated.toISOString(),
})),
}),
},
});
}
}

View File

@@ -6,7 +6,7 @@ import type { S3Context } from '../classes/context.js';
*/
export class ObjectController {
/**
* PUT /:bucket/:key* - Upload object or copy object
* PUT /:bucket/:key* - Upload object, copy object, or upload part
*/
public static async putObject(
req: plugins.http.IncomingMessage,
@@ -16,6 +16,11 @@ export class ObjectController {
): Promise<void> {
const { bucket, key } = params;
// Check if this is a multipart upload part
if (ctx.query.partNumber && ctx.query.uploadId) {
return ObjectController.uploadPart(req, res, ctx, params);
}
// Check if this is a COPY operation
const copySource = ctx.headers['x-amz-copy-source'] as string | undefined;
if (copySource) {
@@ -133,7 +138,7 @@ export class ObjectController {
}
/**
* DELETE /:bucket/:key* - Delete object
* DELETE /:bucket/:key* - Delete object or abort multipart upload
*/
public static async deleteObject(
req: plugins.http.IncomingMessage,
@@ -143,6 +148,11 @@ export class ObjectController {
): Promise<void> {
const { bucket, key } = params;
// Check if this is an abort multipart upload
if (ctx.query.uploadId) {
return ObjectController.abortMultipartUpload(req, res, ctx, params);
}
await ctx.store.deleteObject(bucket, key);
ctx.status(204).send('');
}
@@ -201,4 +211,168 @@ export class ObjectController {
},
});
}
/**
* POST /:bucket/:key* - Initiate or complete multipart upload
*/
public static async postObject(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
// Check if this is initiate multipart upload
if (ctx.query.uploads !== undefined) {
return ObjectController.initiateMultipartUpload(req, res, ctx, params);
}
// Check if this is complete multipart upload
if (ctx.query.uploadId) {
return ObjectController.completeMultipartUpload(req, res, ctx, params);
}
ctx.throw('InvalidRequest', 'Invalid POST request');
}
/**
* Initiate Multipart Upload (POST with ?uploads)
*/
private static async initiateMultipartUpload(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
const { bucket, key } = params;
// Extract metadata from headers
const metadata: Record<string, string> = {};
for (const [header, value] of Object.entries(ctx.headers)) {
if (header.startsWith('x-amz-meta-')) {
metadata[header] = value as string;
}
if (header === 'content-type' && value) {
metadata['content-type'] = value as string;
}
}
// Initiate upload
const uploadId = await ctx.multipart.initiateUpload(bucket, key, metadata);
// Send XML response
await ctx.sendXML({
InitiateMultipartUploadResult: {
Bucket: bucket,
Key: key,
UploadId: uploadId,
},
});
}
/**
* Upload Part (PUT with ?partNumber&uploadId)
*/
private static async uploadPart(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
const uploadId = ctx.query.uploadId!;
const partNumber = parseInt(ctx.query.partNumber!);
if (isNaN(partNumber) || partNumber < 1 || partNumber > 10000) {
ctx.throw('InvalidPartNumber', 'Part number must be between 1 and 10000');
}
// Upload the part
const partInfo = await ctx.multipart.uploadPart(
uploadId,
partNumber,
ctx.getRequestStream() as any as import('stream').Readable
);
// Set ETag header (part ETag)
ctx.setHeader('ETag', `"${partInfo.etag}"`);
ctx.status(200).send('');
}
/**
* Complete Multipart Upload (POST with ?uploadId)
*/
private static async completeMultipartUpload(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
const { bucket, key } = params;
const uploadId = ctx.query.uploadId!;
// Read and parse request body (XML with part list)
const body = await ctx.readBody();
// Parse XML to extract parts
// Expected format: <CompleteMultipartUpload><Part><PartNumber>1</PartNumber><ETag>"etag"</ETag></Part>...</CompleteMultipartUpload>
const partMatches = body.matchAll(/<Part>.*?<PartNumber>(\d+)<\/PartNumber>.*?<ETag>(.*?)<\/ETag>.*?<\/Part>/gs);
const parts: Array<{ PartNumber: number; ETag: string }> = [];
for (const match of partMatches) {
parts.push({
PartNumber: parseInt(match[1]),
ETag: match[2],
});
}
// Complete the upload
const result = await ctx.multipart.completeUpload(uploadId, parts);
// Get upload metadata
const upload = ctx.multipart.getUpload(uploadId);
if (!upload) {
ctx.throw('NoSuchUpload', 'The specified upload does not exist');
}
// Move final file to object store
const finalPath = ctx.multipart.getFinalPath(uploadId);
const finalContent = await plugins.smartfs.file(finalPath).read();
const finalStream = plugins.http.IncomingMessage.prototype;
// Create a readable stream from the buffer
const { Readable } = await import('stream');
const finalReadableStream = Readable.from([finalContent]);
// Store the final object
await ctx.store.putObject(bucket, key, finalReadableStream, upload.metadata);
// Clean up multipart upload data
await ctx.multipart.cleanupUpload(uploadId);
// Send XML response
await ctx.sendXML({
CompleteMultipartUploadResult: {
Location: `/${bucket}/${key}`,
Bucket: bucket,
Key: key,
ETag: `"${result.etag}"`,
},
});
}
/**
* Abort Multipart Upload (DELETE with ?uploadId)
*/
private static async abortMultipartUpload(
req: plugins.http.IncomingMessage,
res: plugins.http.ServerResponse,
ctx: S3Context,
params: Record<string, string>
): Promise<void> {
const uploadId = ctx.query.uploadId!;
// Abort and cleanup
await ctx.multipart.abortUpload(uploadId);
ctx.status(204).send('');
}
}

View File

@@ -2,39 +2,202 @@ import * as plugins from './plugins.js';
import * as paths from './paths.js';
import { Smarts3Server } from './classes/smarts3-server.js';
export interface ISmarts3ContructorOptions {
/**
* Authentication configuration
*/
export interface IAuthConfig {
enabled: boolean;
credentials: Array<{
accessKeyId: string;
secretAccessKey: string;
}>;
}
/**
* CORS configuration
*/
export interface ICorsConfig {
enabled: boolean;
allowedOrigins?: string[];
allowedMethods?: string[];
allowedHeaders?: string[];
exposedHeaders?: string[];
maxAge?: number;
allowCredentials?: boolean;
}
/**
* Logging configuration
*/
export interface ILoggingConfig {
level?: 'error' | 'warn' | 'info' | 'debug';
format?: 'text' | 'json';
enabled?: boolean;
}
/**
* Request limits configuration
*/
export interface ILimitsConfig {
maxObjectSize?: number;
maxMetadataSize?: number;
requestTimeout?: number;
}
/**
* Multipart upload configuration
*/
export interface IMultipartConfig {
expirationDays?: number;
cleanupIntervalMinutes?: number;
}
/**
* Server configuration
*/
export interface IServerConfig {
port?: number;
address?: string;
silent?: boolean;
}
/**
* Storage configuration
*/
export interface IStorageConfig {
directory?: string;
cleanSlate?: boolean;
}
/**
* Complete smarts3 configuration
*/
export interface ISmarts3Config {
server?: IServerConfig;
storage?: IStorageConfig;
auth?: IAuthConfig;
cors?: ICorsConfig;
logging?: ILoggingConfig;
limits?: ILimitsConfig;
multipart?: IMultipartConfig;
}
/**
* Default configuration values
*/
const DEFAULT_CONFIG: ISmarts3Config = {
server: {
port: 3000,
address: '0.0.0.0',
silent: false,
},
storage: {
directory: paths.bucketsDir,
cleanSlate: false,
},
auth: {
enabled: false,
credentials: [
{
accessKeyId: 'S3RVER',
secretAccessKey: 'S3RVER',
},
],
},
cors: {
enabled: false,
allowedOrigins: ['*'],
allowedMethods: ['GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'],
allowedHeaders: ['*'],
exposedHeaders: ['ETag', 'x-amz-request-id', 'x-amz-version-id'],
maxAge: 86400,
allowCredentials: false,
},
logging: {
level: 'info',
format: 'text',
enabled: true,
},
limits: {
maxObjectSize: 5 * 1024 * 1024 * 1024, // 5GB
maxMetadataSize: 2048,
requestTimeout: 300000, // 5 minutes
},
multipart: {
expirationDays: 7,
cleanupIntervalMinutes: 60,
},
};
/**
* Merge user config with defaults (deep merge)
*/
function mergeConfig(userConfig: ISmarts3Config): Required<ISmarts3Config> {
return {
server: {
...DEFAULT_CONFIG.server!,
...(userConfig.server || {}),
},
storage: {
...DEFAULT_CONFIG.storage!,
...(userConfig.storage || {}),
},
auth: {
...DEFAULT_CONFIG.auth!,
...(userConfig.auth || {}),
},
cors: {
...DEFAULT_CONFIG.cors!,
...(userConfig.cors || {}),
},
logging: {
...DEFAULT_CONFIG.logging!,
...(userConfig.logging || {}),
},
limits: {
...DEFAULT_CONFIG.limits!,
...(userConfig.limits || {}),
},
multipart: {
...DEFAULT_CONFIG.multipart!,
...(userConfig.multipart || {}),
},
};
}
/**
* Main Smarts3 class - production-ready S3-compatible server
*/
export class Smarts3 {
// STATIC
public static async createAndStart(
optionsArg: ConstructorParameters<typeof Smarts3>[0],
) {
const smartS3Instance = new Smarts3(optionsArg);
public static async createAndStart(configArg: ISmarts3Config = {}) {
const smartS3Instance = new Smarts3(configArg);
await smartS3Instance.start();
return smartS3Instance;
}
// INSTANCE
public options: ISmarts3ContructorOptions;
public config: Required<ISmarts3Config>;
public s3Instance: Smarts3Server;
constructor(optionsArg: ISmarts3ContructorOptions) {
this.options = optionsArg;
constructor(configArg: ISmarts3Config = {}) {
this.config = mergeConfig(configArg);
}
public async start() {
this.s3Instance = new Smarts3Server({
port: this.options.port || 3000,
address: '0.0.0.0',
directory: paths.bucketsDir,
cleanSlate: this.options.cleanSlate || false,
silent: false,
port: this.config.server.port,
address: this.config.server.address,
directory: this.config.storage.directory,
cleanSlate: this.config.storage.cleanSlate,
silent: this.config.server.silent,
config: this.config, // Pass full config to server
});
await this.s3Instance.start();
console.log('s3 server is running');
if (!this.config.server.silent) {
console.log('s3 server is running');
}
}
public async getS3Descriptor(
@@ -48,11 +211,9 @@ export class Smarts3 {
}
public async createBucket(bucketNameArg: string) {
const smartbucketInstance = new plugins.smartbucket.SmartBucket(
await this.getS3Descriptor(),
);
const bucket = await smartbucketInstance.createBucket(bucketNameArg);
return bucket;
// Call the filesystem store directly instead of using the client library
await this.s3Instance.store.createBucket(bucketNameArg);
return { name: bucketNameArg };
}
public async stop() {

View File

@@ -7,7 +7,6 @@ import * as url from 'url';
export { path, http, crypto, url };
// @push.rocks scope
import * as smartbucket from '@push.rocks/smartbucket';
import { SmartFs, SmartFsProviderNode } from '@push.rocks/smartfs';
import * as smartpath from '@push.rocks/smartpath';
import { SmartXml } from '@push.rocks/smartxml';
@@ -15,7 +14,7 @@ import { SmartXml } from '@push.rocks/smartxml';
// Create SmartFs instance with Node.js provider
export const smartfs = new SmartFs(new SmartFsProviderNode());
export { smartbucket, smartpath, SmartXml };
export { smartpath, SmartXml };
// @tsclass scope
import * as tsclass from '@tsclass/tsclass';