smartbucket/readme.md

# @push.rocks/smartbucket 🪣

> A powerful, cloud-agnostic TypeScript library for object storage with advanced features like file locking, metadata management, and intelligent trash handling.

## Install 📦

To install `@push.rocks/smartbucket`, run:

```bash
npm install @push.rocks/smartbucket --save
```

Or if you're using pnpm (recommended):

```bash
pnpm add @push.rocks/smartbucket
```

## Usage 🚀

### Introduction

`@push.rocks/smartbucket` provides a unified, cloud-agnostic API for object storage operations across major providers like AWS S3, Google Cloud Storage, MinIO, and more. It abstracts away provider-specific complexities while offering advanced features like metadata management, file locking, streaming operations, and intelligent trash management.

### Table of Contents

1. [🏁 Getting Started](#-getting-started)
2. [🗂️ Working with Buckets](#️-working-with-buckets)
3. [📁 File Operations](#-file-operations)
4. [📂 Directory Management](#-directory-management)
5. [🌊 Streaming Operations](#-streaming-operations)
6. [🔒 File Locking](#-file-locking)
7. [🏷️ Metadata Management](#️-metadata-management)
8. [🗑️ Trash & Recovery](#️-trash--recovery)
9. [⚡ Advanced Features](#-advanced-features)
10. [☁️ Cloud Provider Support](#️-cloud-provider-support)

### 🏁 Getting Started

First, set up your storage connection:

```typescript
import { SmartBucket } from '@push.rocks/smartbucket';

// Initialize with your cloud storage credentials
const smartBucket = new SmartBucket({
  accessKey: 'your-access-key',
  accessSecret: 'your-secret-key',
  endpoint: 's3.amazonaws.com', // Or your provider's endpoint
  port: 443,
  useSsl: true,
  region: 'us-east-1' // Optional, defaults to 'us-east-1'
});
```

### 🗂️ Working with Buckets

#### Creating Buckets

```typescript
// Create a new bucket
const myBucket = await smartBucket.createBucket('my-awesome-bucket');
console.log(`✅ Bucket created: ${myBucket.name}`);
```

#### Getting Existing Buckets

```typescript
// Get a bucket reference
const existingBucket = await smartBucket.getBucketByName('existing-bucket');

// Or use strict mode (throws if bucket doesn't exist)
const bucketStrict = await smartBucket.getBucketByNameStrict('must-exist-bucket');
```

#### Removing Buckets

```typescript
// Delete a bucket (must be empty)
await smartBucket.removeBucket('old-bucket');
console.log('🗑️ Bucket removed');
```

### 📁 File Operations

#### Upload Files

```typescript
const bucket = await smartBucket.getBucketByName('my-bucket');

// Simple file upload
await bucket.fastPut({
  path: 'documents/report.pdf',
  contents: Buffer.from('Your file content here')
});

// Upload with string content
await bucket.fastPut({
  path: 'notes/todo.txt',
  contents: 'Buy milk\nCall mom\nRule the world'
});

// Strict upload (returns File object)
const uploadedFile = await bucket.fastPutStrict({
  path: 'images/logo.png',
  contents: imageBuffer,
  overwrite: true // Optional: control overwrite behavior
});
```

#### Download Files

```typescript
// Get file as Buffer
const fileContent = await bucket.fastGet({
  path: 'documents/report.pdf'
});
console.log(`📄 File size: ${fileContent.length} bytes`);

// Get file as string
const textContent = fileContent.toString('utf-8');
```

#### Check File Existence

```typescript
const exists = await bucket.fastExists({
  path: 'documents/report.pdf'
});
console.log(`File exists: ${exists ? '✅' : '❌'}`);
```

#### Delete Files

```typescript
// Permanent deletion
await bucket.fastRemove({
  path: 'old-file.txt'
});
```

#### Copy & Move Files

```typescript
// Copy file within bucket
await bucket.fastCopy({
  sourcePath: 'original/file.txt',
  destinationPath: 'backup/file-copy.txt'
});

// Move file (copy + delete original)
await bucket.fastMove({
  sourcePath: 'temp/draft.txt',
  destinationPath: 'final/document.txt'
});
```

### 📂 Directory Management

SmartBucket provides powerful directory-like operations for organizing your files:

```typescript
// Get base directory
const baseDir = await bucket.getBaseDirectory();

// List directories and files
const directories = await baseDir.listDirectories();
const files = await baseDir.listFiles();

console.log(`📁 Found ${directories.length} directories`);
console.log(`📄 Found ${files.length} files`);

// Navigate subdirectories
const subDir = await baseDir.getSubDirectoryByName('projects/2024');

// Create nested file
await subDir.fastPut({
  path: 'report.pdf',
  contents: reportBuffer
});

// Get directory tree structure
const tree = await subDir.getTreeArray();
console.log('🌳 Directory tree:', tree);

// Create empty file as placeholder
await subDir.createEmptyFile('placeholder.txt');
```

### 🌊 Streaming Operations

Handle large files efficiently with streaming:

#### Download Streams

```typescript
// Node.js stream
const nodeStream = await bucket.fastGetStream(
  { path: 'large-video.mp4' },
  'nodestream'
);
nodeStream.pipe(fs.createWriteStream('local-video.mp4'));

// Web stream (for modern environments)
const webStream = await bucket.fastGetStream(
  { path: 'large-file.zip' },
  'webstream'
);
```

#### Upload Streams

```typescript
// Stream upload from file
const readStream = fs.createReadStream('big-data.csv');
await bucket.fastPutStream({
  path: 'uploads/big-data.csv',
  stream: readStream,
  metadata: {
    contentType: 'text/csv',
    userMetadata: {
      uploadedBy: 'data-team',
      version: '2.0'
    }
  }
});
```

#### Reactive Streams with RxJS

```typescript
// Get file as ReplaySubject for reactive programming
const replaySubject = await bucket.fastGetReplaySubject({
  path: 'data/sensor-readings.json',
  chunkSize: 1024
});

replaySubject.subscribe({
  next: (chunk) => processChunk(chunk),
  complete: () => console.log('✅ Stream complete')
});
```

### 🔒 File Locking

Prevent accidental modifications with file locking:

```typescript
const file = await bucket.getBaseDirectory()
  .getFileStrict({ path: 'important-config.json' });

// Lock file for 10 minutes
await file.lock({ timeoutMillis: 600000 });
console.log('🔒 File locked');

// Try to modify locked file (will throw error)
try {
  await file.delete();
} catch (error) {
  console.log('❌ Cannot delete locked file');
}

// Unlock when done
await file.unlock();
console.log('🔓 File unlocked');
```

### 🏷️ Metadata Management

Attach and manage metadata for your files:

```typescript
const file = await bucket.getBaseDirectory()
  .getFileStrict({ path: 'document.pdf' });

// Get metadata handler
const metadata = await file.getMetaData();

// Set custom metadata
await metadata.setCustomMetaData({
  key: 'author',
  value: 'John Doe'
});

await metadata.setCustomMetaData({
  key: 'department',
  value: 'Engineering'
});

// Retrieve metadata
const author = await metadata.getCustomMetaData({ key: 'author' });
console.log(`📝 Author: ${author}`);

// Get all metadata
const allMeta = await metadata.getAllCustomMetaData();
console.log('📋 All metadata:', allMeta);
```

### 🗑️ Trash & Recovery

SmartBucket includes an intelligent trash system for safe file deletion:

```typescript
const file = await bucket.getBaseDirectory()
  .getFileStrict({ path: 'important-data.xlsx' });

// Move to trash instead of permanent deletion
await file.delete({ mode: 'trash' });
console.log('🗑️ File moved to trash');

// Access trash
const trash = await bucket.getTrash();
const trashDir = await trash.getTrashDir();
const trashedFiles = await trashDir.listFiles();
console.log(`📦 ${trashedFiles.length} files in trash`);

// Restore from trash
const trashedFile = await bucket.getBaseDirectory()
  .getFileStrict({ 
    path: 'important-data.xlsx',
    getFromTrash: true
  });

await trashedFile.restore({ useOriginalPath: true });
console.log('♻️ File restored successfully');

// Permanent deletion from trash
await trash.emptyTrash();
console.log('🧹 Trash emptied');
```

### ⚡ Advanced Features

#### File Statistics

```typescript
// Get detailed file statistics
const stats = await bucket.fastStat({ path: 'document.pdf' });
console.log(`📊 Size: ${stats.size} bytes`);
console.log(`📅 Last modified: ${stats.lastModified}`);
console.log(`🏷️ ETag: ${stats.etag}`);
```

#### Magic Bytes Detection

```typescript
// Read first bytes for file type detection
const magicBytes = await bucket.getMagicBytes({
  path: 'mystery-file',
  length: 16
});

// Or from a File object
const file = await bucket.getBaseDirectory()
  .getFileStrict({ path: 'image.jpg' });
const magic = await file.getMagicBytes({ length: 4 });
console.log(`🔮 Magic bytes: ${magic.toString('hex')}`);
```

#### JSON Data Operations

```typescript
const file = await bucket.getBaseDirectory()
  .getFileStrict({ path: 'config.json' });

// Read JSON data
const config = await file.getJsonData();
console.log('⚙️ Config loaded:', config);

// Update JSON data
config.version = '2.0';
config.updated = new Date().toISOString();
await file.writeJsonData(config);
console.log('💾 Config updated');
```

#### Directory & File Type Detection

```typescript
// Check if path is a directory
const isDir = await bucket.isDirectory({ path: 'uploads/' });

// Check if path is a file
const isFile = await bucket.isFile({ path: 'uploads/document.pdf' });

console.log(`Is directory: ${isDir ? '📁' : '❌'}`);
console.log(`Is file: ${isFile ? '📄' : '❌'}`);
```

#### Clean Bucket Contents

```typescript
// Remove all files and directories (use with caution!)
await bucket.cleanAllContents();
console.log('🧹 Bucket cleaned');
```

### ☁️ Cloud Provider Support

SmartBucket works seamlessly with:

- ✅ **AWS S3** - Full compatibility with S3 API
- ✅ **Google Cloud Storage** - Via S3-compatible API
- ✅ **MinIO** - Self-hosted S3-compatible storage
- ✅ **DigitalOcean Spaces** - S3-compatible object storage
- ✅ **Backblaze B2** - Cost-effective cloud storage
- ✅ **Wasabi** - High-performance S3-compatible storage
- ✅ **Any S3-compatible provider**

The library automatically handles provider quirks and optimizes operations for each platform while maintaining a consistent API.

### 🔧 Advanced Configuration

```typescript
// Configure with custom options
const smartBucket = new SmartBucket({
  accessKey: process.env.S3_ACCESS_KEY,
  accessSecret: process.env.S3_SECRET_KEY,
  endpoint: process.env.S3_ENDPOINT,
  port: 443,
  useSsl: true,
  region: 'eu-central-1',
  // Additional S3 client options can be passed through
});

// Environment-based configuration
import { Qenv } from '@push.rocks/qenv';
const qenv = new Qenv('./', './.nogit/');

const smartBucket = new SmartBucket({
  accessKey: await qenv.getEnvVarOnDemandStrict('S3_ACCESS_KEY'),
  accessSecret: await qenv.getEnvVarOnDemandStrict('S3_SECRET'),
  endpoint: await qenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
});
```

### 🧪 Testing

SmartBucket is thoroughly tested. Run tests with:

```bash
pnpm test
```

### 🤝 Best Practices

1. **Always use strict mode** for critical operations to catch errors early
2. **Implement proper error handling** for network and permission issues
3. **Use streaming** for large files to optimize memory usage
4. **Leverage metadata** for organizing and searching files
5. **Enable trash mode** for important data to prevent accidental loss
6. **Lock files** during critical operations to prevent race conditions
7. **Clean up resources** properly when done

## License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository. 

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

### Company Information

Task Venture Capital GmbH  
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.