@push.rocks/smarts3 🚀

Production-ready S3-compatible server - A powerful, lightweight Node.js TypeScript package that brings full S3 API compatibility to your local filesystem. Perfect for development, testing, and scenarios where running MinIO is out of scope!

🌟 Features

• 🏃 Lightning-fast local S3 simulation - No more waiting for cloud operations during development
• ⚡ Production-ready architecture - Built on Node.js http module with zero framework dependencies
• 🔄 Full S3 API compatibility - Works seamlessly with AWS SDK v3 and any other S3 client
• 📂 Local directory mapping - Your buckets live right on your filesystem
• 🔐 Simple authentication - Static credential-based auth for secure access
• 🌐 CORS support - Configurable cross-origin resource sharing
• 📊 Structured logging - Multiple levels (error/warn/info/debug) and formats (text/JSON)
• 📤 Multipart uploads - Full support for large file uploads (>5MB)
• 🧪 Perfect for testing - Reliable, repeatable tests without cloud dependencies
• 🎯 TypeScript-first - Built with TypeScript for excellent type safety and IDE support
• 🔧 Flexible configuration - Comprehensive config system with sensible defaults
• 🧹 Clean slate mode - Start fresh on every test run

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

📦 Installation

Install using your favorite package manager:

# Using npm
npm install @push.rocks/smarts3 --save-dev

# Using pnpm (recommended)
pnpm add @push.rocks/smarts3 -D

# Using yarn
yarn add @push.rocks/smarts3 --dev

🚀 Quick Start

Get up and running in seconds:

import { Smarts3 } from '@push.rocks/smarts3';

// Start your local S3 server with minimal config
const s3Server = await Smarts3.createAndStart({
  server: {
    port: 3000,
    silent: false,
  },
  storage: {
    cleanSlate: true, // Start with empty buckets
  },
});

// Create a bucket
const bucket = await s3Server.createBucket('my-awesome-bucket');

// Get S3 connection details for use with AWS SDK or other S3 clients
const s3Config = await s3Server.getS3Descriptor();

// When you're done
await s3Server.stop();

📖 Configuration Guide

Complete Configuration Options

Smarts3 uses a comprehensive nested configuration structure:

import { Smarts3, ISmarts3Config } from '@push.rocks/smarts3';

const config: ISmarts3Config = {
  // Server configuration
  server: {
    port: 3000,              // Port to listen on (default: 3000)
    address: '0.0.0.0',      // Bind address (default: '0.0.0.0')
    silent: false,           // Disable all console output (default: false)
  },

  // Storage configuration
  storage: {
    directory: './buckets',  // Directory to store buckets (default: .nogit/bucketsDir)
    cleanSlate: false,       // Clear all data on start (default: false)
  },

  // Authentication configuration
  auth: {
    enabled: false,          // Enable authentication (default: false)
    credentials: [           // List of valid credentials
      {
        accessKeyId: 'YOUR_ACCESS_KEY',
        secretAccessKey: 'YOUR_SECRET_KEY',
      },
    ],
  },

  // CORS configuration
  cors: {
    enabled: false,          // Enable CORS (default: false)
    allowedOrigins: ['*'],   // Allowed origins (default: ['*'])
    allowedMethods: [        // Allowed HTTP methods
      'GET', 'POST', 'PUT', 'DELETE', 'HEAD', 'OPTIONS'
    ],
    allowedHeaders: ['*'],   // Allowed headers (default: ['*'])
    exposedHeaders: [        // Headers exposed to client
      'ETag', 'x-amz-request-id', 'x-amz-version-id'
    ],
    maxAge: 86400,           // Preflight cache duration in seconds
    allowCredentials: false, // Allow credentials (default: false)
  },

  // Logging configuration
  logging: {
    level: 'info',           // Log level: 'error' | 'warn' | 'info' | 'debug'
    format: 'text',          // Log format: 'text' | 'json'
    enabled: true,           // Enable logging (default: true)
  },

  // Request limits
  limits: {
    maxObjectSize: 5 * 1024 * 1024 * 1024,  // 5GB max object size
    maxMetadataSize: 2048,                   // 2KB max metadata size
    requestTimeout: 300000,                  // 5 minutes request timeout
  },
};

const s3Server = await Smarts3.createAndStart(config);

Simple Configuration Examples

Development Mode (Default)

const s3Server = await Smarts3.createAndStart({
  server: { port: 3000 },
  storage: { cleanSlate: true },
});

Production Mode with Auth

const s3Server = await Smarts3.createAndStart({
  server: { port: 3000 },
  auth: {
    enabled: true,
    credentials: [
      {
        accessKeyId: process.env.S3_ACCESS_KEY,
        secretAccessKey: process.env.S3_SECRET_KEY,
      },
    ],
  },
  logging: {
    level: 'warn',
    format: 'json',
  },
});

CORS-Enabled for Web Apps

const s3Server = await Smarts3.createAndStart({
  server: { port: 3000 },
  cors: {
    enabled: true,
    allowedOrigins: ['http://localhost:8080', 'https://app.example.com'],
    allowCredentials: true,
  },
});

🪣 Working with Buckets

Creating Buckets

// Create a new bucket
const bucket = await s3Server.createBucket('my-bucket');
console.log(`Created bucket: ${bucket.name}`);

📤 File Operations

Using AWS SDK v3

import { S3Client, PutObjectCommand, GetObjectCommand } from '@aws-sdk/client-s3';

// Get connection config
const config = await s3Server.getS3Descriptor();

// Configure AWS SDK client
const s3Client = new S3Client({
  endpoint: `http://${config.endpoint}:${config.port}`,
  region: 'us-east-1',
  credentials: {
    accessKeyId: config.accessKey,
    secretAccessKey: config.accessSecret,
  },
  forcePathStyle: true,
});

// Upload a file
await s3Client.send(new PutObjectCommand({
  Bucket: 'my-bucket',
  Key: 'test-file.txt',
  Body: 'Hello from AWS SDK!',
  ContentType: 'text/plain',
}));

// Download a file
const response = await s3Client.send(new GetObjectCommand({
  Bucket: 'my-bucket',
  Key: 'test-file.txt',
}));

const content = await response.Body.transformToString();
console.log(content); // "Hello from AWS SDK!"

Using SmartBucket

import { SmartBucket } from '@push.rocks/smartbucket';

// Get connection configuration
const s3Config = await s3Server.getS3Descriptor();

// Create a SmartBucket instance
const smartbucket = new SmartBucket(s3Config);
const bucket = await smartbucket.getBucket('my-bucket');
const baseDir = await bucket.getBaseDirectory();

// Upload files
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');
await baseDir.fastPut({
  path: 'documents/important.pdf',
  contents: Buffer.from(yourPdfData),
});

// Download files
const content = await baseDir.fastGet('path/to/file.txt');
const buffer = await baseDir.fastGetBuffer('documents/important.pdf');

// List files
const files = await baseDir.listFiles();
files.forEach((file) => {
  console.log(`📄 ${file.name} (${file.size} bytes)`);
});

// Delete files
await baseDir.fastDelete('old-file.txt');

📤 Multipart Uploads

Smarts3 supports multipart uploads for large files (>5MB):

import {
  S3Client,
  CreateMultipartUploadCommand,
  UploadPartCommand,
  CompleteMultipartUploadCommand
} from '@aws-sdk/client-s3';

const s3Client = new S3Client(/* ... */);

// 1. Initiate multipart upload
const { UploadId } = await s3Client.send(new CreateMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
}));

// 2. Upload parts (in parallel if desired)
const parts = [];
for (let i = 0; i < numParts; i++) {
  const part = await s3Client.send(new UploadPartCommand({
    Bucket: 'my-bucket',
    Key: 'large-file.bin',
    UploadId,
    PartNumber: i + 1,
    Body: partData[i],
  }));

  parts.push({
    PartNumber: i + 1,
    ETag: part.ETag,
  });
}

// 3. Complete the upload
await s3Client.send(new CompleteMultipartUploadCommand({
  Bucket: 'my-bucket',
  Key: 'large-file.bin',
  UploadId,
  MultipartUpload: { Parts: parts },
}));

🧪 Testing Integration

Using with Jest

import { Smarts3 } from '@push.rocks/smarts3';

describe('S3 Operations', () => {
  let s3Server: Smarts3;

  beforeAll(async () => {
    s3Server = await Smarts3.createAndStart({
      server: { port: 9999, silent: true },
      storage: { cleanSlate: true },
    });
  });

  afterAll(async () => {
    await s3Server.stop();
  });

  test('should upload and retrieve a file', async () => {
    const bucket = await s3Server.createBucket('test-bucket');
    // Your test logic here
  });
});

Using with Mocha

import { Smarts3 } from '@push.rocks/smarts3';
import { expect } from 'chai';

describe('S3 Operations', () => {
  let s3Server: Smarts3;

  before(async () => {
    s3Server = await Smarts3.createAndStart({
      server: { port: 9999, silent: true },
      storage: { cleanSlate: true },
    });
  });

  after(async () => {
    await s3Server.stop();
  });

  it('should upload and retrieve a file', async () => {
    const bucket = await s3Server.createBucket('test-bucket');
    // Your test logic here
  });
});

🎯 Real-World Use Cases

CI/CD Pipeline Testing

// ci-test.ts
import { Smarts3 } from '@push.rocks/smarts3';

export async function setupTestEnvironment() {
  const s3 = await Smarts3.createAndStart({
    server: {
      port: process.env.S3_PORT || 3000,
      silent: true,
    },
    storage: { cleanSlate: true },
    logging: { level: 'error' }, // Only log errors in CI
  });

  // Create test buckets
  await s3.createBucket('uploads');
  await s3.createBucket('processed');
  await s3.createBucket('archive');

  return s3;
}

Microservice Development

// dev-server.ts
import { Smarts3 } from '@push.rocks/smarts3';
import express from 'express';

async function startDevelopmentServer() {
  // Start local S3 with CORS for local development
  const s3 = await Smarts3.createAndStart({
    server: { port: 3000 },
    cors: {
      enabled: true,
      allowedOrigins: ['http://localhost:8080'],
    },
  });

  await s3.createBucket('user-uploads');

  // Start your API server
  const app = express();

  app.post('/upload', async (req, res) => {
    // Your upload logic using local S3
  });

  app.listen(8080, () => {
    console.log('🚀 Dev server running with local S3!');
  });
}

Data Migration Testing

import { Smarts3 } from '@push.rocks/smarts3';
import { SmartBucket } from '@push.rocks/smartbucket';

async function testDataMigration() {
  const s3 = await Smarts3.createAndStart({
    storage: { cleanSlate: true },
  });

  // Create source and destination buckets
  await s3.createBucket('legacy-data');
  await s3.createBucket('new-data');

  // Populate source with test data
  const config = await s3.getS3Descriptor();
  const smartbucket = new SmartBucket(config);
  const source = await smartbucket.getBucket('legacy-data');
  const sourceDir = await source.getBaseDirectory();

  await sourceDir.fastStore('user-1.json', JSON.stringify({ id: 1, name: 'Alice' }));
  await sourceDir.fastStore('user-2.json', JSON.stringify({ id: 2, name: 'Bob' }));

  // Run your migration logic
  await runMigration(config);

  // Verify migration results
  const dest = await smartbucket.getBucket('new-data');
  const destDir = await dest.getBaseDirectory();
  const migratedFiles = await destDir.listFiles();

  console.log(`✅ Migrated ${migratedFiles.length} files successfully!`);
}

🔧 API Reference

Smarts3 Class

Static Methods

createAndStart(config?: ISmarts3Config): Promise<Smarts3>

Create and start a Smarts3 instance in one call.

Parameters:

• config - Optional configuration object (see Configuration Guide above)

Returns: Promise that resolves to a running Smarts3 instance

Instance Methods

start(): Promise<void>

Start the S3 server.

stop(): Promise<void>

Stop the S3 server and release resources.

createBucket(name: string): Promise<{ name: string }>

Create a new S3 bucket.

Parameters:

• name - Bucket name

Returns: Promise that resolves to bucket information

getS3Descriptor(options?): Promise<IS3Descriptor>

Get S3 connection configuration for use with S3 clients.

Parameters:

• options - Optional partial descriptor to merge with defaults

Returns: Promise that resolves to S3 descriptor with:

• accessKey - Access key for authentication
• accessSecret - Secret key for authentication
• endpoint - Server endpoint (hostname/IP)
• port - Server port
• useSsl - Whether to use SSL (always false for local server)

💡 Production Considerations

When to Use Smarts3 vs MinIO

Use Smarts3 when:

• 🎯 You need a lightweight, zero-dependency S3 server
• 🧪 Running in CI/CD pipelines or containerized test environments
• 🏗️ Local development where MinIO setup is overkill
• 📦 Your application needs to bundle an S3-compatible server
• 🚀 Quick prototyping without infrastructure setup

Use MinIO when:

• 🏢 Production workloads requiring high availability
• 📊 Advanced features like versioning, replication, encryption at rest
• 🔐 Complex IAM policies and bucket policies
• 📈 High-performance requirements with multiple nodes
• 🌐 Multi-tenant environments

Security Notes

• Smarts3's authentication is intentionally simple (static credentials)
• It does not implement AWS Signature V4 verification
• Perfect for development/testing, but not for production internet-facing deployments
• For production use, place behind a reverse proxy with proper authentication

🐛 Debugging Tips

1. Enable debug logging

   const s3 = await Smarts3.createAndStart({
     logging: { level: 'debug', format: 'json' },
   });
   

2. Check the buckets directory - Find your data in .nogit/bucketsDir/ by default

3. Use the correct endpoint - Remember to use 127.0.0.1 or localhost

4. Force path style - Always use forcePathStyle: true with local S3

5. Inspect requests - All requests are logged when silent: false

📈 Performance

Smarts3 is optimized for development and testing scenarios:

• ⚡ Instant operations - No network latency
• 💾 Low memory footprint - Efficient filesystem operations with streams
• 🔄 Fast cleanup - Clean slate mode for quick test resets
• 🚀 Parallel operations - Handle multiple concurrent requests
• 📤 Streaming uploads/downloads - Low memory usage for large files

🔗 Related Packages

• @push.rocks/smartbucket - Powerful S3 abstraction layer
• @push.rocks/smartfs - Modern filesystem with Web Streams support
• @tsclass/tsclass - TypeScript class helpers

📝 Changelog

v4.0.0 - Production Ready 🚀

Breaking Changes:

• Configuration format changed from flat to nested structure
• Old format: { port: 3000, cleanSlate: true }
• New format: { server: { port: 3000 }, storage: { cleanSlate: true } }

New Features:

• ✨ Production configuration system with comprehensive options
• 📊 Structured logging with multiple levels and formats
• 🌐 Full CORS middleware support
• 🔐 Simple static credentials authentication
• 📤 Complete multipart upload support for large files
• 🔧 Flexible configuration with sensible defaults

Improvements:

• Removed smartbucket from production dependencies (dev-only)
• Migrated to @push.rocks/smartfs for modern filesystem operations
• Enhanced error handling and logging throughout
• Better TypeScript types and documentation

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

Company Information

Task Venture Capital GmbH Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.