@push.rocks/smartbucket

A TypeScript library for cloud-independent object storage, providing features like bucket creation, file and directory management, and data streaming.

Install

To install @push.rocks/smartbucket, you need to have Node.js and npm (Node Package Manager) installed. If they are installed, you can add @push.rocks/smartbucket to your project by running the following command in your project's root directory:

npm install @push.rocks/smartbucket --save

This command will download and install @push.rocks/smartbucket along with its required dependencies into your project's node_modules directory and save it as a dependency in your project's package.json file.

Usage

@push.rocks/smartbucket is a TypeScript module designed to provide simple cloud-independent object storage functionality. It wraps various cloud storage providers such as AWS S3, Google Cloud Storage, and others, offering a unified API to manage storage buckets and objects within those buckets.

In this guide, we will delve into the usage of SmartBucket, covering its full range of features from setting up the library to advanced usage scenarios.

Table of Contents

1. Setting Up
2. Creating a New Bucket
3. Listing Buckets
4. Working with Files
   • Uploading Files
   • Downloading Files
   • Deleting Files
   • Streaming Files
5. Working with Directories
6. Advanced Features
   • Bucket Policies
   • Object Metadata
   • Cloud Agnostic

Setting Up

First, ensure you are using ECMAScript modules (ESM) and TypeScript in your project for best compatibility. Here's how to import and initialize SmartBucket in a TypeScript file:

import {
  SmartBucket,
  Bucket,
  Directory,
  File
} from '@push.rocks/smartbucket';

const mySmartBucket = new SmartBucket({
  accessKey: "yourAccessKey",
  accessSecret: "yourSecretKey",
  endpoint: "yourEndpointURL",
  port: 443,            // Default is 443, can be customized for specific endpoint
  useSsl: true          // Defaults to true
});

Make sure to replace "yourAccessKey", "yourSecretKey", and "yourEndpointURL" with your actual credentials and endpoint URL. The port and useSsl options are optional and can be omitted if the defaults are acceptable.

Creating a New Bucket

To create a new bucket:

async function createBucket(bucketName: string) {
  try {
    const myBucket: Bucket = await mySmartBucket.createBucket(bucketName);
    console.log(`Bucket ${bucketName} created successfully.`);
  } catch (error) {
    console.error("Error creating bucket:", error);
  }
}

// Use the function
createBucket("exampleBucket");

Bucket names must be unique across the storage service.

Listing Buckets

Currently, SmartBucket does not include a direct method to list all buckets, but you can access the underlying client provided by the cloud storage SDK to perform such operations, depending on the SDK's capabilities.

Working with Files

Uploading Files

To upload an object to a bucket:

async function uploadFile(bucketName: string, filePath: string, fileContent: Buffer | string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    await myBucket.fastPut({ path: filePath, contents: fileContent });
    console.log(`File uploaded to ${bucketName} at ${filePath}`);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
uploadFile("exampleBucket", "path/to/object.txt", "Hello, world!");

Downloading Files

To download an object:

async function downloadFile(bucketName: string, filePath: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    const fileContent: Buffer = await myBucket.fastGet({ path: filePath });
    console.log("Downloaded file content:", fileContent.toString());
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
downloadFile("exampleBucket", "path/to/object.txt");

Deleting Files

To delete an object from a bucket:

async function deleteFile(bucketName: string, filePath: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    await myBucket.fastRemove({ path: filePath });
    console.log(`File at ${filePath} deleted from ${bucketName}.`);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
deleteFile("exampleBucket", "path/to/object.txt");

Streaming Files

SmartBucket allows you to work with file streams, which can be useful for handling large files.

To read a file as a stream:

import { ReplaySubject } from '@push.rocks/smartrx';

async function readFileStream(bucketName: string, filePath: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    const fileStream: ReplaySubject<Buffer> = await myBucket.fastGetStream({ path: filePath });
    fileStream.subscribe({
      next(chunk: Buffer) {
        console.log("Chunk received:", chunk.toString());
      },
      complete() {
        console.log("File read completed.");
      },
      error(err) {
        console.error("Error reading file stream:", err);
      }
    });
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
readFileStream("exampleBucket", "path/to/object.txt");

To write a file as a stream:

import { Readable } from 'stream';

async function writeFileStream(bucketName: string, filePath: string, readableStream: Readable) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    await myBucket.fastPutStream({ path: filePath, dataStream: readableStream });
    console.log(`File streamed to ${bucketName} at ${filePath}`);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Create a readable stream from a string
const readable = new Readable();
readable.push('Hello world streamed as a file!');
readable.push(null); // End of stream

// Use the function
writeFileStream("exampleBucket", "path/to/streamedObject.txt", readable);

Working with Directories

@push.rocks/smartbucket offers abstractions for directories within buckets for easier object management. You can create, list, and delete directories using the Directory class.

To list the contents of a directory:

async function listDirectoryContents(bucketName: string, directoryPath: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    const baseDirectory: Directory = await myBucket.getBaseDirectory();
    const targetDirectory: Directory = await baseDirectory.getSubDirectoryByName(directoryPath);
    
    console.log('Listing directories:');
    const directories = await targetDirectory.listDirectories();
    directories.forEach(dir => {
      console.log(`- ${dir.name}`);
    });

    console.log('Listing files:');
    const files = await targetDirectory.listFiles();
    files.forEach(file => {
      console.log(`- ${file.name}`);
    });
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
listDirectoryContents("exampleBucket", "some/directory/path");

To create a file within a directory:

async function createFileInDirectory(bucketName: string, directoryPath: string, fileName: string, fileContent: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    const baseDirectory: Directory = await myBucket.getBaseDirectory();
    const targetDirectory: Directory = await baseDirectory.getSubDirectoryByName(directoryPath);
    await targetDirectory.createEmptyFile(fileName); // Create an empty file
    const file = new File({ directoryRefArg: targetDirectory, fileName });
    await file.updateWithContents({ contents: fileContent });
    console.log(`File created: ${fileName}`);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
createFileInDirectory("exampleBucket", "some/directory", "newfile.txt", "Hello, world!");

Advanced Features

Bucket Policies

Manage bucket policies to control access permissions. This feature depends on the policies provided by the storage service (e.g., AWS S3, MinIO).

Object Metadata

Retrieve and modify object metadata. Metadata can be useful for storing additional information about an object.

To retrieve metadata:

async function getObjectMetadata(bucketName: string, filePath: string) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    const metadata = await mySmartBucket.minioClient.statObject(bucketName, filePath);
    console.log("Object metadata:", metadata);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
getObjectMetadata("exampleBucket", "path/to/object.txt");

To update metadata:

async function updateObjectMetadata(bucketName: string, filePath: string, newMetadata: { [key: string]: string }) {
  const myBucket: Bucket = await mySmartBucket.getBucketByName(bucketName);
  if (myBucket) {
    await myBucket.copyObject({
      objectKey: filePath,
      nativeMetadata: newMetadata,
      deleteExistingNativeMetadata: false,
    });
    console.log(`Metadata updated for ${filePath}`);
  } else {
    console.error(`Bucket ${bucketName} does not exist.`);
  }
}

// Use the function
updateObjectMetadata("exampleBucket", "path/to/object.txt", {
  customKey: "customValue"
});

Cloud Agnostic

@push.rocks/smartbucket is designed to work with multiple cloud providers, allowing for easier migration or multi-cloud strategies. This means you can switch from one provider to another with minimal changes to your codebase.

Remember, each cloud provider has specific features and limitations. @push.rocks/smartbucket aims to abstract common functionalities, but always refer to the specific cloud provider's documentation for advanced features or limitations.

This guide covers the basic to advanced scenarios of using @push.rocks/smartbucket. For further details, refer to the API documentation and examples.

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

Company Information

Task Venture Capital GmbH
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.