Mock S3 made simple - A powerful Node.js TypeScript package for creating a local S3 endpoint that simulates AWS S3 operations using mapped local directories. Perfect for development and testing!

🌟 Features

🏃 Lightning-fast local S3 simulation - No more waiting for cloud operations during development
🔄 Full AWS S3 API compatibility - Drop-in replacement for S3 in your tests
📂 Local directory mapping - Your buckets live right on your filesystem
🧪 Perfect for testing - Reliable, repeatable tests without cloud dependencies
🎯 TypeScript-first - Built with TypeScript for excellent type safety and IDE support
🔧 Zero configuration - Works out of the box with sensible defaults
🧹 Clean slate mode - Start fresh on every test run

📦 Installation

Install using your favorite package manager:

# Using npm
npm install @push.rocks/smarts3 --save-dev

# Using pnpm (recommended)
pnpm add @push.rocks/smarts3 -D

# Using yarn
yarn add @push.rocks/smarts3 --dev

🚀 Quick Start

Get up and running in seconds:

import { Smarts3 } from '@push.rocks/smarts3';

// Start your local S3 server
const s3Server = await Smarts3.createAndStart({
  port: 3000,
  cleanSlate: true // Start with empty buckets
});

// Create a bucket
const bucket = await s3Server.createBucket('my-awesome-bucket');

// Get S3 connection details for use with AWS SDK or other S3 clients
const s3Config = await s3Server.getS3Descriptor();

// When you're done
await s3Server.stop();

📖 Detailed Usage Guide

🏗️ Setting Up Your S3 Server

The Smarts3 class provides a simple interface for managing your local S3 server:

import { Smarts3 } from '@push.rocks/smarts3';

// Configuration options
const config = {
  port: 3000,        // Port to run the server on (default: 3000)
  cleanSlate: true   // Clear all data on start (default: false)
};

// Create and start in one go
const s3Server = await Smarts3.createAndStart(config);

// Or create and start separately
const s3Server = new Smarts3(config);
await s3Server.start();

🪣 Working with Buckets

Creating and managing buckets is straightforward:

// Create a new bucket
const bucket = await s3Server.createBucket('my-bucket');

// The bucket is now ready to use!
console.log(`Created bucket: ${bucket.name}`);

📤 Uploading Files

Use the powerful SmartBucket integration for file operations:

import { SmartBucket } from '@push.rocks/smartbucket';

// Get connection configuration
const s3Config = await s3Server.getS3Descriptor();

// Create a SmartBucket instance
const smartbucket = new SmartBucket(s3Config);

// Get your bucket
const bucket = await smartbucket.getBucket('my-bucket');

// Upload a file
const baseDir = await bucket.getBaseDirectory();
await baseDir.fastStore('path/to/file.txt', 'Hello, S3! 🎉');

// Upload with more control
await baseDir.fastPut({
  path: 'documents/important.pdf',
  contents: Buffer.from(yourPdfData)
});

📥 Downloading Files

Retrieve your files easily:

// Get file contents as string
const content = await baseDir.fastGet('path/to/file.txt');
console.log(content); // "Hello, S3! 🎉"

// Get file as Buffer
const buffer = await baseDir.fastGetBuffer('documents/important.pdf');

📋 Listing Files

Browse your bucket contents:

// List all files in the bucket
const files = await baseDir.listFiles();

files.forEach(file => {
  console.log(`📄 ${file.name} (${file.size} bytes)`);
});

// List files with a specific prefix
const docs = await baseDir.listFiles('documents/');

🗑️ Deleting Files

Clean up when needed:

// Delete a single file
await baseDir.fastDelete('old-file.txt');

// Delete multiple files
const filesToDelete = ['temp1.txt', 'temp2.txt', 'temp3.txt'];
for (const file of filesToDelete) {
  await baseDir.fastDelete(file);
}

🧪 Testing Integration

Using with Jest

import { Smarts3 } from '@push.rocks/smarts3';

describe('S3 Operations', () => {
  let s3Server: Smarts3;

  beforeAll(async () => {
    s3Server = await Smarts3.createAndStart({
      port: 9999,
      cleanSlate: true
    });
  });

  afterAll(async () => {
    await s3Server.stop();
  });

  test('should upload and retrieve a file', async () => {
    const bucket = await s3Server.createBucket('test-bucket');
    // Your test logic here
  });
});

Using with Mocha

import { Smarts3 } from '@push.rocks/smarts3';
import { expect } from 'chai';

describe('S3 Operations', () => {
  let s3Server: Smarts3;

  before(async () => {
    s3Server = await Smarts3.createAndStart({
      port: 9999,
      cleanSlate: true
    });
  });

  after(async () => {
    await s3Server.stop();
  });

  it('should upload and retrieve a file', async () => {
    const bucket = await s3Server.createBucket('test-bucket');
    // Your test logic here
  });
});

🔌 AWS SDK Integration

Use smarts3 with the official AWS SDK:

import { S3Client, PutObjectCommand } from '@aws-sdk/client-s3';
import { Smarts3 } from '@push.rocks/smarts3';

// Start local S3
const s3Server = await Smarts3.createAndStart({ port: 3000 });
const config = await s3Server.getS3Descriptor();

// Configure AWS SDK
const s3Client = new S3Client({
  endpoint: `http://${config.endpoint}:${config.port}`,
  region: 'us-east-1',
  credentials: {
    accessKeyId: config.accessKey,
    secretAccessKey: config.accessSecret
  },
  forcePathStyle: true
});

// Use AWS SDK as normal
const command = new PutObjectCommand({
  Bucket: 'my-bucket',
  Key: 'test-file.txt',
  Body: 'Hello from AWS SDK!'
});

await s3Client.send(command);

🎯 Real-World Examples

CI/CD Pipeline Testing

// ci-test.ts
import { Smarts3 } from '@push.rocks/smarts3';

export async function setupTestEnvironment() {
  // Start S3 server for CI tests
  const s3 = await Smarts3.createAndStart({
    port: process.env.S3_PORT || 3000,
    cleanSlate: true
  });

  // Create test buckets
  await s3.createBucket('uploads');
  await s3.createBucket('processed');
  await s3.createBucket('archive');

  return s3;
}

Microservice Development

// dev-server.ts
import { Smarts3 } from '@push.rocks/smarts3';
import express from 'express';

async function startDevelopmentServer() {
  // Start local S3
  const s3 = await Smarts3.createAndStart({ port: 3000 });
  await s3.createBucket('user-uploads');

  // Start your API server
  const app = express();
  
  app.post('/upload', async (req, res) => {
    // Your upload logic using local S3
  });

  app.listen(8080, () => {
    console.log('🚀 Dev server running with local S3!');
  });
}

Data Migration Testing

import { Smarts3 } from '@push.rocks/smarts3';

async function testDataMigration() {
  const s3 = await Smarts3.createAndStart({ cleanSlate: true });
  
  // Create source and destination buckets
  const sourceBucket = await s3.createBucket('legacy-data');
  const destBucket = await s3.createBucket('new-data');

  // Populate source with test data
  const config = await s3.getS3Descriptor();
  const smartbucket = new SmartBucket(config);
  const source = await smartbucket.getBucket('legacy-data');
  const sourceDir = await source.getBaseDirectory();
  
  // Add test files
  await sourceDir.fastStore('user-1.json', JSON.stringify({ id: 1, name: 'Alice' }));
  await sourceDir.fastStore('user-2.json', JSON.stringify({ id: 2, name: 'Bob' }));

  // Run your migration logic
  await runMigration(config);

  // Verify migration results
  const dest = await smartbucket.getBucket('new-data');
  const destDir = await dest.getBaseDirectory();
  const migratedFiles = await destDir.listFiles();
  
  console.log(`✅ Migrated ${migratedFiles.length} files successfully!`);
}

🛠️ Advanced Configuration

Custom S3 Descriptor Options

When integrating with different S3 clients, you can customize the connection details:

const customDescriptor = await s3Server.getS3Descriptor({
  endpoint: 'localhost',  // Custom endpoint
  port: 3001,            // Different port
  useSsl: false,         // SSL configuration
  // Add any additional options your S3 client needs
});

Environment-Based Configuration

const config = {
  port: parseInt(process.env.S3_PORT || '3000'),
  cleanSlate: process.env.NODE_ENV === 'test'
};

const s3Server = await Smarts3.createAndStart(config);

🤝 Use Cases

🧪 Unit & Integration Testing - Test S3 operations without AWS credentials or internet
🏗️ Local Development - Develop cloud features offline with full S3 compatibility
📚 Teaching & Demos - Perfect for workshops and tutorials without AWS setup
🔄 CI/CD Pipelines - Reliable S3 operations in containerized test environments
🎭 Mocking & Stubbing - Replace real S3 calls in test suites
📊 Data Migration Testing - Safely test data migrations locally before production

🔧 API Reference

Smarts3 Class

Constructor Options

interface ISmarts3ContructorOptions {
  port?: number;        // Server port (default: 3000)
  cleanSlate?: boolean; // Clear storage on start (default: false)
}

Methods

static createAndStart(options) - Create and start server in one call
start() - Start the S3 server
stop() - Stop the S3 server
createBucket(name) - Create a new bucket
getS3Descriptor(options?) - Get S3 connection configuration

🐛 Debugging Tips

Enable verbose logging - The server logs all operations by default
Check the buckets directory - Find your data in .nogit/bucketsDir/
Use the correct endpoint - Remember to use 127.0.0.1 or localhost
Force path style - Always use path-style URLs with local S3

📈 Performance

@push.rocks/smarts3 is optimized for development and testing:

⚡ Instant operations - No network latency
💾 Low memory footprint - Efficient file system usage
🔄 Fast cleanup - Clean slate mode for quick test resets
🚀 Parallel operations - Handle multiple requests simultaneously

@push.rocks/smartbucket - Powerful S3 abstraction layer
@push.rocks/smartfile - Advanced file system operations
@tsclass/tsclass - TypeScript class helpers
s3rver - The underlying S3 server implementation

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

readme.md

@push.rocks/smarts3 🚀