2024-10-26 14:14:41 +02:00
parent c931a0459a
commit 1a1ceba76d
15 changed files with 3497 additions and 19083 deletions
							
							
							
						
@@ -1,85 +1,178 @@
```markdown
# @push.rocks/smarts3
create an s3 endpoint that maps to a local directory
A Node.js TypeScript package to create a local S3 endpoint for development and testing using mapped local directories, simulating AWS S3.
## Install
To use @push.rocks/smarts3 in your project, you'll need to install it via npm. You can do so by running the following command in your project's root directory:
To integrate `@push.rocks/smarts3` with your project, you need to install it via npm. Execute the following command within your project's root directory:
```sh
npm install @push.rocks/smarts3 --save
```
This will add `@push.rocks/smarts3` as a dependency to your project's `package.json` file and download the package into the `node_modules` directory.
This command will add `@push.rocks/smarts3` as a dependency in your project's `package.json` file and download the package into the `node_modules` directory.
## Usage
@push.rocks/smarts3 is designed to make it easy for developers to create an S3 compatible endpoint that maps to a local directory. This can be particularly useful for local development and testing, allowing you to mimic S3's functionality without the need for an actual S3 bucket.
### Overview
Below, we'll go through how to get started, set up an S3 server, manage buckets, and perform common operations.
The `@push.rocks/smarts3` module allows users to create a mock S3 endpoint that maps to a local directory using `s3rver`. This simulation of AWS S3 operations facilitates development and testing by enabling file uploads, bucket creation, and other interactions locally. This local setup is ideal for developers looking to test cloud file storage operations without requiring access to a real AWS S3 instance.
### Setting up the S3 Server
In this comprehensive guide, we will explore setting up a local S3 server, performing operations like creating buckets and uploading files, and how to effectively integrate this into your development workflow.
First, let's see how to set up and start an S3 server instance.
### Setting Up the Environment
To begin any operations, your environment must be configured correctly. Here’s a simple setup procedure:
```typescript
import * as path from 'path';
import { promises as fs } from 'fs';
async function setupEnvironment() {
  const packageDir = path.resolve();
  const nogitDir = path.join(packageDir, './.nogit');
  const bucketsDir = path.join(nogitDir, 'bucketsDir');
  try {
    await fs.mkdir(bucketsDir, { recursive: true });
  } catch (error) {
    console.error('Failed to create buckets directory!', error);
    throw error;
  }
  console.log('Environment setup complete.');
}
setupEnvironment().catch(console.error);
```
This script sets up a directory structure required for the `smarts3` server, ensuring that the directories needed for bucket storage exist before starting the server.
### Starting the S3 Server
Once your environment is set up, start an instance of the `smarts3` server. This acts as your local mock S3 endpoint:
```typescript
import { Smarts3 } from '@push.rocks/smarts3';
async function startServer() {
  // Creating and starting the Smarts3 instance
  const smarts3Instance = await Smarts3.createAndStart({
    port: 3000, // optional, defaults to 3000
    cleanSlate: true // optional, if set to true, it will empty the directory on start
    port: 3000,
    cleanSlate: true,
  });
  console.log('S3 server is up and running...');
  console.log('S3 server is up and running at http://localhost:3000');
  return smarts3Instance;
}
startServer().catch(console.error);
```
### Creating a Bucket
**Parameters:**
- **Port**: Specify the port for the local S3 server. Defaults to `3000`.
- **CleanSlate**: If `true`, clears the storage directory each time the server starts, providing a fresh test state.
With the server up and running, you can now create buckets.
### Creating and Managing Buckets
With your server running, create buckets for storing files. A bucket in S3 acts similarly to a root directory.
```typescript
async function createBucket(smarts3Instance: Smarts3) {
  // Creating a new bucket
  const bucket = await smarts3Instance.createBucket('my-awesome-bucket');
async function createBucket(smarts3Instance: Smarts3, bucketName: string) {
  const bucket = await smarts3Instance.createBucket(bucketName);
  console.log(`Bucket created: ${bucket.name}`);
}
// Assuming startServer() has been called and smarts3Instance has been received
createBucket(smarts3Instance).catch(console.error);
startServer()
  .then((smarts3Instance) => createBucket(smarts3Instance, 'my-awesome-bucket'))
  .catch(console.error);
```
### Accessing Buckets and Uploading Files
### Uploading and Managing Files
To perform actions like uploading files, you will need to work with the underlying `SmartBucket` module. The `SmartBucket` module is part of the @push.rocks ecosystem and integrated into smarts3 for easy bucket and file management.
Uploading files to a bucket uses the `SmartBucket` module, part of the `@push.rocks/smartbucket` ecosystem:
```typescript
import { SmartBucket, Bucket } from '@pushrocks/smartbucket';
import { SmartBucket } from '@push.rocks/smartbucket';
async function uploadFile(smarts3Instance: Smarts3) {
  // Getting the S3 descriptor to configure SmartBucket
async function uploadFile(smarts3Instance: Smarts3, bucketName: string, filePath: string, fileContent: string) {
  const s3Descriptor = await smarts3Instance.getS3Descriptor();
  
  const smartbucketInstance = new SmartBucket(s3Descriptor);
  const bucket: Bucket = await smartbucketInstance.getBucket('my-awesome-bucket');
  const bucket = await smartbucketInstance.getBucket(bucketName);
  // Now let's upload a file to the bucket
  const baseDirectory = await bucket.getBaseDirectory();
  await baseDirectory.fastStore('hello.txt', 'Hello, world!');
  console.log('File uploaded successfully.');
  await bucket.getBaseDirectory().fastStore(filePath, fileContent);
  console.log(`File "${filePath}" uploaded successfully to bucket "${bucketName}".`);
}
uploadFile(smarts3Instance).catch(console.error);
startServer()
  .then(async (smarts3Instance) => {
    await createBucket(smarts3Instance, 'my-awesome-bucket');
    await uploadFile(smarts3Instance, 'my-awesome-bucket', 'hello.txt', 'Hello, world!');
  })
  .catch(console.error);
```
### Listing Files in a Bucket
Listing files within a bucket allows you to manage its contents conveniently:
```typescript
async function listFiles(smarts3Instance: Smarts3, bucketName: string) {
  const s3Descriptor = await smarts3Instance.getS3Descriptor();
  const smartbucketInstance = new SmartBucket(s3Descriptor);
  const bucket = await smartbucketInstance.getBucket(bucketName);
  const baseDirectory = await bucket.getBaseDirectory();
  const files = await baseDirectory.listFiles();
  console.log(`Files in bucket "${bucketName}":`, files);
}
startServer()
  .then(async (smarts3Instance) => {
    await createBucket(smarts3Instance, 'my-awesome-bucket');
    await listFiles(smarts3Instance, 'my-awesome-bucket');
  })
  .catch(console.error);
```
### Deleting a File
Managing storage efficiently involves deleting files when necessary:
```typescript
async function deleteFile(smarts3Instance: Smarts3, bucketName: string, filePath: string) {
  const s3Descriptor = await smarts3Instance.getS3Descriptor();
  const smartbucketInstance = new SmartBucket(s3Descriptor);
  const bucket = await smartbucketInstance.getBucket(bucketName);
  await bucket.getBaseDirectory().fastDelete(filePath);
  console.log(`File "${filePath}" deleted from bucket "${bucketName}".`);
}
startServer()
  .then(async (smarts3Instance) => {
    await createBucket(smarts3Instance, 'my-awesome-bucket');
    await deleteFile(smarts3Instance, 'my-awesome-bucket', 'hello.txt');
  })
  .catch(console.error);
```
### Scenario Integrations
#### Development and Testing
1. **Feature Development:** Use `@push.rocks/smarts3` to simulate file upload endpoints, ensuring your application handles file operations correctly before going live.
   
2. **Continuous Integration/Continuous Deployment (CI/CD):** Integrate with CI/CD pipelines to automatically test file interactions.
3. **Data Migration Testing:** Simulate data migrations between buckets to perfect processes before implementation on actual S3.
4. **Onboarding New Developers:** Offer new team members hands-on practice with mock setups to improve their understanding without real-world consequences.
### Stopping the Server
Finally, when you are done, you can stop the Smarts3 server as follows:
Safely shutting down the server when tasks are complete ensures system resources are managed well:
```typescript
async function stopServer(smarts3Instance: Smarts3) {
							
							
							
								
							
						
@@ -87,12 +180,16 @@ async function stopServer(smarts3Instance: Smarts3) {
  console.log('S3 server has been stopped.');
}
stopServer(smarts3Instance).catch(console.error);
startServer()
  .then(async (smarts3Instance) => {
    await createBucket(smarts3Instance, 'my-awesome-bucket');
    await stopServer(smarts3Instance);
  })
  .catch(console.error);
```
### Complete Integration
Integrating @push.rocks/smarts3 into your development workflow offers a seamless way to mimic Amazon S3 locally. By creating an S3-compatible endpoint that maps to a local directory, you can develop and test applications that interact with S3 without incurring any cost or requiring internet connectivity. Thanks to its API, @push.rocks/smarts3 makes it easy to programmatically manage your local S3 server, create and delete buckets, and upload or download files, ensuring that your applications can be designed with cloud scalability in mind, right from the start.
In this guide, we walked through setting up and fully utilizing the `@push.rocks/smarts3` package for local development and testing. The package simulates AWS S3 operations, reducing dependency on remote services and allowing efficient development iteration cycles. By implementing the practices and scripts shared here, you can ensure a seamless and productive development experience using the local S3 simulation capabilities of `@push.rocks/smarts3`.
```
## License and Legal Information