A powerful, cloud-agnostic TypeScript library for object storage that makes S3 feel like a modern filesystem. Built for developers who demand simplicity, type-safety, and advanced features like metadata management, file locking, intelligent trash handling, and memory-efficient streaming.

Why SmartBucket? 🎯

🌍 Cloud Agnostic - Write once, run on AWS S3, MinIO, DigitalOcean Spaces, Backblaze B2, Wasabi, or any S3-compatible storage
🚀 Modern TypeScript - First-class TypeScript support with complete type definitions and async/await patterns
💾 Memory Efficient - Handle millions of files with async generators, RxJS observables, and cursor pagination
🗑️ Smart Trash System - Recover accidentally deleted files with built-in trash and restore functionality
🔒 File Locking - Prevent concurrent modifications with built-in locking mechanisms
🏷️ Rich Metadata - Attach custom metadata to any file for powerful organization and search
🌊 Streaming Support - Efficient handling of large files with Node.js and Web streams
📁 Directory-like API - Intuitive filesystem-like operations on object storage
⚡ Fail-Fast - Strict-by-default API catches errors immediately with precise stack traces

Quick Start 🚀

import { SmartBucket } from '@push.rocks/smartbucket';

// Connect to your storage
const storage = new SmartBucket({
  accessKey: 'your-access-key',
  accessSecret: 'your-secret-key',
  endpoint: 's3.amazonaws.com',
  port: 443,
  useSsl: true
});

// Get or create a bucket
const bucket = await storage.getBucketByName('my-app-data');

// Upload a file
await bucket.fastPut({
  path: 'users/profile.json',
  contents: JSON.stringify({ name: 'Alice', role: 'admin' })
});

// Download it back
const data = await bucket.fastGet({ path: 'users/profile.json' });
console.log('📄', JSON.parse(data.toString()));

// List files efficiently (even with millions of objects!)
for await (const key of bucket.listAllObjects('users/')) {
  console.log('🔍 Found:', key);
}

Install 📦

# Using pnpm (recommended)
pnpm add @push.rocks/smartbucket

# Using npm
npm install @push.rocks/smartbucket --save

🏁 Getting Started

First, set up your storage connection:

import { SmartBucket } from '@push.rocks/smartbucket';

// Initialize with your cloud storage credentials
const smartBucket = new SmartBucket({
  accessKey: 'your-access-key',
  accessSecret: 'your-secret-key',
  endpoint: 's3.amazonaws.com', // Or your provider's endpoint
  port: 443,
  useSsl: true,
  region: 'us-east-1' // Optional, defaults to 'us-east-1'
});

For MinIO or self-hosted S3:

const smartBucket = new SmartBucket({
  accessKey: 'minioadmin',
  accessSecret: 'minioadmin',
  endpoint: 'localhost',
  port: 9000,
  useSsl: false // MinIO often runs without SSL locally
});

🗂️ Working with Buckets

Creating Buckets

// Create a new bucket
const myBucket = await smartBucket.createBucket('my-awesome-bucket');
console.log(`✅ Bucket created: ${myBucket.name}`);

Getting Existing Buckets

// Get a bucket reference (throws if not found - strict by default!)
const existingBucket = await smartBucket.getBucketByName('existing-bucket');

// Check first, then get (non-throwing approach)
if (await smartBucket.bucketExists('maybe-exists')) {
  const bucket = await smartBucket.getBucketByName('maybe-exists');
  console.log('✅ Found bucket:', bucket.name);
}

Removing Buckets

// Delete a bucket (must be empty)
await smartBucket.removeBucket('old-bucket');
console.log('🗑️ Bucket removed');

📁 File Operations

Upload Files

const bucket = await smartBucket.getBucketByName('my-bucket');

// Simple file upload (returns File object)
const file = await bucket.fastPut({
  path: 'documents/report.pdf',
  contents: Buffer.from('Your file content here')
});
console.log('✅ Uploaded:', file.path);

// Upload with string content
await bucket.fastPut({
  path: 'notes/todo.txt',
  contents: 'Buy milk\nCall mom\nRule the world'
});

// Upload with overwrite control
const uploadedFile = await bucket.fastPut({
  path: 'images/logo.png',
  contents: imageBuffer,
  overwrite: true // Set to true to replace existing files
});

// Error handling: fastPut throws if file exists and overwrite is false
try {
  await bucket.fastPut({
    path: 'existing-file.txt',
    contents: 'new content'
  });
} catch (error) {
  console.error('❌ Upload failed:', error.message);
  // Error: Object already exists at path 'existing-file.txt' in bucket 'my-bucket'.
  // Set overwrite:true to replace it.
}

Download Files

// Get file as Buffer
const fileContent = await bucket.fastGet({
  path: 'documents/report.pdf'
});
console.log(`📄 File size: ${fileContent.length} bytes`);

// Get file as string
const textContent = fileContent.toString('utf-8');

// Parse JSON files directly
const jsonData = JSON.parse(fileContent.toString());

Check File Existence

const exists = await bucket.fastExists({
  path: 'documents/report.pdf'
});
console.log(`File exists: ${exists ? '✅' : '❌'}`);

Delete Files

// Permanent deletion
await bucket.fastRemove({
  path: 'old-file.txt'
});
console.log('🗑️ File deleted permanently');

Copy & Move Files

// Copy file within bucket
await bucket.fastCopy({
  sourcePath: 'original/file.txt',
  destinationPath: 'backup/file-copy.txt'
});
console.log('📋 File copied');

// Move file (copy + delete original)
await bucket.fastMove({
  sourcePath: 'temp/draft.txt',
  destinationPath: 'final/document.txt'
});
console.log('📦 File moved');

📋 Memory-Efficient Listing

SmartBucket provides three powerful patterns for listing objects, optimized for handling millions of files efficiently:

Async Generators (Recommended) ⭐

Memory-efficient streaming using native JavaScript async iteration:

// List all objects with prefix - streams one at a time!
for await (const key of bucket.listAllObjects('documents/')) {
  console.log(`📄 Found: ${key}`);

  // Process each file individually (memory efficient!)
  const content = await bucket.fastGet({ path: key });
  processFile(content);

  // Early termination support
  if (shouldStop()) break;
}

// List all objects (no prefix)
const allKeys: string[] = [];
for await (const key of bucket.listAllObjects()) {
  allKeys.push(key);
}

// Find objects matching glob patterns
for await (const key of bucket.findByGlob('**/*.json')) {
  console.log(`📦 JSON file: ${key}`);
}

// Complex glob patterns
for await (const key of bucket.findByGlob('npm/packages/*/index.json')) {
  // Matches: npm/packages/foo/index.json, npm/packages/bar/index.json
  console.log(`📦 Package index: ${key}`);
}

// More glob examples
for await (const key of bucket.findByGlob('logs/**/*.log')) {
  console.log('📋 Log file:', key);
}

for await (const key of bucket.findByGlob('images/*.{jpg,png,gif}')) {
  console.log('🖼️  Image:', key);
}

Why use async generators?

✅ Processes one item at a time (constant memory usage)
✅ Supports early termination with break
✅ Native JavaScript - no dependencies
✅ Perfect for large buckets with millions of objects
✅ Works seamlessly with for await...of loops

RxJS Observables

Perfect for reactive pipelines and complex data transformations:

import { filter, take, map } from 'rxjs/operators';

// Stream keys as Observable with powerful operators
bucket.listAllObjectsObservable('logs/')
  .pipe(
    filter(key => key.endsWith('.log')),
    take(100),
    map(key => ({ key, timestamp: Date.now() }))
  )
  .subscribe({
    next: (item) => console.log(`📋 Log file: ${item.key}`),
    error: (err) => console.error('❌ Error:', err),
    complete: () => console.log('✅ Listing complete')
  });

// Simple subscription without operators
bucket.listAllObjectsObservable('data/')
  .subscribe({
    next: (key) => processKey(key),
    complete: () => console.log('✅ Done')
  });

// Combine with other observables
import { merge } from 'rxjs';

const logs$ = bucket.listAllObjectsObservable('logs/');
const backups$ = bucket.listAllObjectsObservable('backups/');

merge(logs$, backups$)
  .pipe(filter(key => key.includes('2024')))
  .subscribe(key => console.log('📅 2024 file:', key));

Why use observables?

✅ Rich operator ecosystem (filter, map, debounce, etc.)
✅ Composable with other RxJS streams
✅ Perfect for reactive architectures
✅ Great for complex transformations

Cursor Pattern

Explicit pagination control for UI and resumable operations:

// Create cursor with custom page size
const cursor = bucket.createCursor('uploads/', { pageSize: 100 });

// Fetch pages manually
while (cursor.hasMore()) {
  const page = await cursor.next();
  console.log(`📄 Page has ${page.keys.length} items`);

  for (const key of page.keys) {
    console.log(`  - ${key}`);
  }

  if (page.done) {
    console.log('✅ Reached end');
    break;
  }
}

// Save and restore cursor state (perfect for resumable operations!)
const token = cursor.getToken();
// Store token in database or session...

// ... later, in a different request ...
const newCursor = bucket.createCursor('uploads/', { pageSize: 100 });
newCursor.setToken(token); // Resume from saved position!
const nextPage = await cursor.next();

// Reset cursor to start over
cursor.reset();
const firstPage = await cursor.next(); // Back to the beginning

Why use cursors?

✅ Perfect for UI pagination (prev/next buttons)
✅ Save/restore state for resumable operations
✅ Explicit control over page fetching
✅ Great for implementing "Load More" buttons

Convenience Methods

// Collect all keys into array (⚠️ WARNING: loads everything into memory!)
const allKeys = await bucket.listAllObjectsArray('images/');
console.log(`📦 Found ${allKeys.length} images`);

// Only use for small result sets
const smallList = await bucket.listAllObjectsArray('config/');
if (smallList.length < 100) {
  // Safe to process in memory
  smallList.forEach(key => console.log(key));
}

Performance Comparison:

Method	Memory Usage	Best For	Supports Early Exit
Async Generator	O(1) - constant	Most use cases, large datasets	✅ Yes
Observable	O(1) - constant	Reactive pipelines, RxJS apps	✅ Yes
Cursor	O(pageSize)	UI pagination, resumable ops	✅ Yes
Array	O(n) - grows with results	Small datasets (<10k items)	❌ No

📂 Directory Management

SmartBucket provides powerful directory-like operations for organizing your files:

// Get base directory
const baseDir = await bucket.getBaseDirectory();

// List directories and files
const directories = await baseDir.listDirectories();
const files = await baseDir.listFiles();

console.log(`📁 Found ${directories.length} directories`);
console.log(`📄 Found ${files.length} files`);

// Navigate subdirectories
const subDir = await baseDir.getSubDirectoryByName('projects/2024');

// Create nested file
await subDir.fastPut({
  path: 'report.pdf',
  contents: reportBuffer
});

// Get directory tree structure
const tree = await subDir.getTreeArray();
console.log('🌳 Directory tree:', tree);

// Get directory path
console.log('📂 Base path:', subDir.getBasePath()); // "projects/2024/"

// Create empty file as placeholder
await subDir.createEmptyFile('placeholder.txt');

🌊 Streaming Operations

Handle large files efficiently with streaming support:

Download Streams

// Node.js stream (for file I/O, HTTP responses, etc.)
const nodeStream = await bucket.fastGetStream(
  { path: 'large-video.mp4' },
  'nodestream'
);

// Pipe to file
import * as fs from 'node:fs';
nodeStream.pipe(fs.createWriteStream('local-video.mp4'));

// Pipe to HTTP response
app.get('/download', async (req, res) => {
  const stream = await bucket.fastGetStream(
    { path: 'file.pdf' },
    'nodestream'
  );
  res.setHeader('Content-Type', 'application/pdf');
  stream.pipe(res);
});

// Web stream (for modern browser/Deno environments)
const webStream = await bucket.fastGetStream(
  { path: 'large-file.zip' },
  'webstream'
);

Upload Streams

import * as fs from 'node:fs';

// Stream upload from file
const readStream = fs.createReadStream('big-data.csv');
await bucket.fastPutStream({
  path: 'uploads/big-data.csv',
  stream: readStream,
  metadata: {
    contentType: 'text/csv',
    userMetadata: {
      uploadedBy: 'data-team',
      version: '2.0'
    }
  }
});
console.log('✅ Large file uploaded via stream');

Reactive Streams with RxJS

// Get file as ReplaySubject for reactive programming
const replaySubject = await bucket.fastGetReplaySubject({
  path: 'data/sensor-readings.json',
  chunkSize: 1024
});

// Multiple subscribers can consume the same data
replaySubject.subscribe({
  next: (chunk) => processChunk(chunk),
  complete: () => console.log('✅ Stream complete')
});

replaySubject.subscribe({
  next: (chunk) => logChunk(chunk)
});

🔒 File Locking

Prevent concurrent modifications with built-in file locking:

const file = await bucket.getBaseDirectory()
  .getFile({ path: 'important-config.json' });

// Lock file for 10 minutes
await file.lock({ timeoutMillis: 600000 });
console.log('🔒 File locked');

// Try to modify locked file (will throw error)
try {
  await file.delete();
} catch (error) {
  console.log('❌ Cannot delete locked file');
}

// Check lock status
const isLocked = await file.isLocked();
console.log(`Lock status: ${isLocked ? '🔒 Locked' : '🔓 Unlocked'}`);

// Unlock when done
await file.unlock();
console.log('🔓 File unlocked');

Lock use cases:

🔄 Prevent concurrent writes during critical updates
🔐 Protect configuration files during deployment
🚦 Coordinate distributed workers
🛡️ Ensure data consistency

🏷️ Metadata Management

Attach and manage rich metadata for your files:

const file = await bucket.getBaseDirectory()
  .getFile({ path: 'document.pdf' });

// Get metadata handler
const metadata = await file.getMetaData();

// Set custom metadata
await metadata.setCustomMetaData({
  key: 'author',
  value: 'John Doe'
});

await metadata.setCustomMetaData({
  key: 'department',
  value: 'Engineering'
});

await metadata.setCustomMetaData({
  key: 'version',
  value: '1.0.0'
});

// Retrieve metadata
const author = await metadata.getCustomMetaData({ key: 'author' });
console.log(`📝 Author: ${author}`);

// Get all metadata
const allMeta = await metadata.getAllCustomMetaData();
console.log('📋 All metadata:', allMeta);
// { author: 'John Doe', department: 'Engineering', version: '1.0.0' }

// Check if metadata exists
const hasMetadata = await metadata.hasMetaData();
console.log(`Has metadata: ${hasMetadata ? '✅' : '❌'}`);

Metadata use cases:

👤 Track file ownership and authorship
🏷️ Add tags and categories for search
📊 Store processing status or workflow state
🔍 Enable rich querying and filtering
📝 Maintain audit trails

🗑️ Trash & Recovery

SmartBucket includes an intelligent trash system for safe file deletion and recovery:

const file = await bucket.getBaseDirectory()
  .getFile({ path: 'important-data.xlsx' });

// Move to trash instead of permanent deletion
await file.delete({ mode: 'trash' });
console.log('🗑️ File moved to trash (can be restored!)');

// Permanent deletion (use with caution!)
await file.delete({ mode: 'permanent' });
console.log('💀 File permanently deleted (cannot be recovered)');

// Access trash
const trash = await bucket.getTrash();
const trashDir = await trash.getTrashDir();
const trashedFiles = await trashDir.listFiles();
console.log(`📦 ${trashedFiles.length} files in trash`);

// Restore from trash
const trashedFile = await bucket.getBaseDirectory()
  .getFile({
    path: 'important-data.xlsx',
    getFromTrash: true
  });

await trashedFile.restore({ useOriginalPath: true });
console.log('♻️ File restored to original location');

// Or restore to a different location
await trashedFile.restore({
  useOriginalPath: false,
  restorePath: 'recovered/important-data.xlsx'
});
console.log('♻️ File restored to new location');

// Empty trash permanently
await trash.emptyTrash();
console.log('🧹 Trash emptied');

Trash features:

♻️ Recover accidentally deleted files
🏷️ Preserves original path in metadata
⏰ Tracks deletion timestamp
🔍 List and inspect trashed files
🧹 Bulk empty trash operation

⚡ Advanced Features

File Statistics

// Get detailed file statistics
const stats = await bucket.fastStat({ path: 'document.pdf' });
console.log(`📊 Size: ${stats.size} bytes`);
console.log(`📅 Last modified: ${stats.lastModified}`);
console.log(`🏷️ ETag: ${stats.etag}`);
console.log(`🗂️ Storage class: ${stats.storageClass}`);

Magic Bytes Detection

Detect file types by examining the first bytes (useful for validation):

// Read first bytes for file type detection
const magicBytes = await bucket.getMagicBytes({
  path: 'mystery-file',
  length: 16
});
console.log(`🔮 Magic bytes: ${magicBytes.toString('hex')}`);

// Or from a File object
const file = await bucket.getBaseDirectory()
  .getFile({ path: 'image.jpg' });
const magic = await file.getMagicBytes({ length: 4 });

// Check file signatures
if (magic[0] === 0xFF && magic[1] === 0xD8) {
  console.log('📸 This is a JPEG image');
} else if (magic[0] === 0x89 && magic[1] === 0x50) {
  console.log('🖼️  This is a PNG image');
}

JSON Data Operations

const file = await bucket.getBaseDirectory()
  .getFile({ path: 'config.json' });

// Read JSON data
const config = await file.getJsonData();
console.log('⚙️ Config loaded:', config);

// Update JSON data
config.version = '2.0';
config.updated = new Date().toISOString();
config.features.push('newFeature');

await file.writeJsonData(config);
console.log('💾 Config updated');

Directory & File Type Detection

// Check if path is a directory
const isDir = await bucket.isDirectory({ path: 'uploads/' });

// Check if path is a file
const isFile = await bucket.isFile({ path: 'uploads/document.pdf' });

console.log(`Is directory: ${isDir ? '📁' : '❌'}`);
console.log(`Is file: ${isFile ? '📄' : '❌'}`);

Clean Bucket Contents

// Remove all files and directories (⚠️ use with caution!)
await bucket.cleanAllContents();
console.log('🧹 Bucket cleaned');

☁️ Cloud Provider Support

SmartBucket works seamlessly with all major S3-compatible providers:

Provider	Status	Notes
AWS S3	✅ Full support	Native S3 API
MinIO	✅ Full support	Self-hosted, perfect for development
DigitalOcean Spaces	✅ Full support	Cost-effective S3-compatible
Backblaze B2	✅ Full support	Very affordable storage
Wasabi	✅ Full support	High-performance hot storage
Google Cloud Storage	✅ Full support	Via S3-compatible API
Cloudflare R2	✅ Full support	Zero egress fees
Any S3-compatible	✅ Full support	Works with any S3-compatible provider

The library automatically handles provider quirks and optimizes operations for each platform while maintaining a consistent API.

Configuration examples:

// AWS S3
const awsStorage = new SmartBucket({
  accessKey: process.env.AWS_ACCESS_KEY_ID,
  accessSecret: process.env.AWS_SECRET_ACCESS_KEY,
  endpoint: 's3.amazonaws.com',
  region: 'us-east-1',
  useSsl: true
});

// MinIO (local development)
const minioStorage = new SmartBucket({
  accessKey: 'minioadmin',
  accessSecret: 'minioadmin',
  endpoint: 'localhost',
  port: 9000,
  useSsl: false
});

// DigitalOcean Spaces
const doStorage = new SmartBucket({
  accessKey: process.env.DO_SPACES_KEY,
  accessSecret: process.env.DO_SPACES_SECRET,
  endpoint: 'nyc3.digitaloceanspaces.com',
  region: 'nyc3',
  useSsl: true
});

// Backblaze B2
const b2Storage = new SmartBucket({
  accessKey: process.env.B2_KEY_ID,
  accessSecret: process.env.B2_APPLICATION_KEY,
  endpoint: 's3.us-west-002.backblazeb2.com',
  region: 'us-west-002',
  useSsl: true
});

🔧 Advanced Configuration

// Environment-based configuration with @push.rocks/qenv
import { Qenv } from '@push.rocks/qenv';

const qenv = new Qenv('./', './.nogit/');

const smartBucket = new SmartBucket({
  accessKey: await qenv.getEnvVarOnDemandStrict('S3_ACCESS_KEY'),
  accessSecret: await qenv.getEnvVarOnDemandStrict('S3_SECRET'),
  endpoint: await qenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
  port: parseInt(await qenv.getEnvVarOnDemandStrict('S3_PORT')),
  useSsl: await qenv.getEnvVarOnDemandStrict('S3_USE_SSL') === 'true',
  region: await qenv.getEnvVarOnDemandStrict('S3_REGION')
});

🧪 Testing

SmartBucket is thoroughly tested with 82 comprehensive tests covering all features:

# Run all tests
pnpm test

# Run specific test file
pnpm tstest test/test.listing.node+deno.ts --verbose

# Run tests with log file
pnpm test --logfile

🛡️ Error Handling Best Practices

SmartBucket uses a strict-by-default approach - methods throw errors instead of returning null:

// ✅ Good: Check existence first
if (await bucket.fastExists({ path: 'file.txt' })) {
  const content = await bucket.fastGet({ path: 'file.txt' });
  process(content);
}

// ✅ Good: Try/catch for expected failures
try {
  const file = await bucket.fastGet({ path: 'might-not-exist.txt' });
  process(file);
} catch (error) {
  console.log('File not found, using default');
  useDefault();
}

// ✅ Good: Explicit overwrite control
try {
  await bucket.fastPut({
    path: 'existing-file.txt',
    contents: 'new data',
    overwrite: false // Explicitly fail if exists
  });
} catch (error) {
  console.log('File already exists');
}

// ❌ Bad: Assuming file exists without checking
const content = await bucket.fastGet({ path: 'file.txt' }); // May throw!

💡 Best Practices

Always use strict mode for critical operations to catch errors early
Check existence first with fastExists(), bucketExists(), etc. before operations
Implement proper error handling for network and permission issues
Use streaming for large files (>100MB) to optimize memory usage
Leverage metadata for organizing and searching files
Enable trash mode for important data to prevent accidental loss
Lock files during critical operations to prevent race conditions
Use async generators for listing large buckets to avoid memory issues
Set explicit overwrite flags to prevent accidental file overwrites
Clean up resources properly when done

📊 Performance Tips

Listing: Use async generators or cursors for buckets with >10,000 objects
Uploads: Use streams for files >100MB
Downloads: Use streams for files you'll process incrementally
Metadata: Cache metadata when reading frequently
Locking: Keep lock durations as short as possible
Glob patterns: Be specific to reduce objects scanned

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

readme.md

@push.rocks/smartbucket 🪣