Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| dd6efa4908 | |||
| 1f4b7319d3 | |||
| b8e5d9a222 | |||
| 429375a643 | |||
| e147a077f3 | |||
| 5889396134 |
120
changelog.md
120
changelog.md
@@ -1,5 +1,125 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 2025-11-20 - 4.1.0 - feat(core)
|
||||||
|
Add S3 endpoint normalization, directory pagination, improved metadata checks, trash support, and related tests
|
||||||
|
|
||||||
|
- Add normalizeS3Descriptor helper to sanitize and normalize various S3 endpoint formats and emit warnings for mismatches (helpers.ts).
|
||||||
|
- Use normalized endpoint and credentials when constructing S3 client in SmartBucket (classes.smartbucket.ts).
|
||||||
|
- Implement paginated listing helper listObjectsV2AllPages in Directory and use it for listFiles and listDirectories to aggregate Contents and CommonPrefixes across pages (classes.directory.ts).
|
||||||
|
- Improve MetaData.hasMetaData to catch NotFound errors and return false instead of throwing (classes.metadata.ts).
|
||||||
|
- Export metadata and trash modules from index (ts/index.ts) and add a Trash class with utilities for trashed files and key encoding (classes.trash.ts).
|
||||||
|
- Enhance Bucket operations: fastCopy now preserves or replaces native metadata correctly, cleanAllContents supports paginated deletion, and improved fastExists error handling (classes.bucket.ts).
|
||||||
|
- Fix Directory.getSubDirectoryByName to construct new Directory instances with the correct parent directory reference.
|
||||||
|
- Add tests covering metadata absence and pagination behavior (test/test.local.node+deno.ts).
|
||||||
|
|
||||||
|
## 2025-11-20 - 4.0.1 - fix(plugins)
|
||||||
|
Use explicit node: imports for native path and stream modules in ts/plugins.ts
|
||||||
|
|
||||||
|
- Replaced imports of 'path' and 'stream' with 'node:path' and 'node:stream' in ts/plugins.ts.
|
||||||
|
- Ensures correct ESM resolution of Node built-ins when package.json type is 'module' and avoids accidental conflicts with userland packages.
|
||||||
|
|
||||||
|
## 2025-11-20 - 4.0.0 - BREAKING CHANGE(core)
|
||||||
|
Make API strict-by-default: remove *Strict variants, throw on not-found/exists conflicts, add explicit exists() methods, update docs/tests and bump deps
|
||||||
|
|
||||||
|
- Breaking: Core API methods are strict by default and now throw errors instead of returning null when targets are missing or already exist (e.g. getBucketByName, getFile, getSubDirectoryByName, fastPut, fastPutStream).
|
||||||
|
- Removed *Strict variants: fastPutStrict, getBucketByNameStrict, getFileStrict, getSubDirectoryByNameStrict — use the base methods which are now strict.
|
||||||
|
- Added explicit existence checks: bucketExists (SmartBucket), fileExists (Directory/fileExists), directoryExists (Directory.directoryExists), and fastExists (Bucket.fastExists) to allow non-throwing checks before operations.
|
||||||
|
- Return type updates: fastPut now returns Promise<File> (no null), getBucketByName/getFile/getSubDirectoryByName now return the respective objects or throw.
|
||||||
|
- Improved error messages to guide callers (e.g. suggest setting overwrite:true on fastPut when object exists).
|
||||||
|
- Updated README, changelog and tests to reflect the new strict semantics and usage patterns.
|
||||||
|
- Developer/runtime dependency bumps: @git.zone/tsbuild, @git.zone/tsrun, @git.zone/tstest, @aws-sdk/client-s3, @push.rocks/smartstring, @tsclass/tsclass (version bumps recorded in package.json).
|
||||||
|
- Major version bump to 4.0.0 to reflect breaking API changes.
|
||||||
|
|
||||||
|
## 2025-11-20 - 4.0.0 - BREAKING: Strict by default + exists methods
|
||||||
|
Complete API overhaul: all methods throw by default, removed all *Strict variants, added dedicated exists methods
|
||||||
|
|
||||||
|
**Breaking Changes:**
|
||||||
|
|
||||||
|
**Putters (Write Operations):**
|
||||||
|
- `fastPut`: Return type `Promise<File | null>` → `Promise<File>`, throws when file exists and overwrite is false
|
||||||
|
- `fastPutStream`: Now throws when file exists and overwrite is false (previously returned silently)
|
||||||
|
- `fastPutStrict`: **Removed** - use `fastPut` directly
|
||||||
|
|
||||||
|
**Getters (Read Operations):**
|
||||||
|
- `getBucketByName`: Return type `Promise<Bucket | null>` → `Promise<Bucket>`, throws when bucket not found
|
||||||
|
- `getBucketByNameStrict`: **Removed** - use `getBucketByName` directly
|
||||||
|
- `getFile`: Return type `Promise<File | null>` → `Promise<File>`, throws when file not found
|
||||||
|
- `getFileStrict`: **Removed** - use `getFile` directly
|
||||||
|
- `getSubDirectoryByName`: Return type `Promise<Directory | null>` → `Promise<Directory>`, throws when directory not found
|
||||||
|
- `getSubDirectoryByNameStrict`: **Removed** - use `getSubDirectoryByName` directly
|
||||||
|
|
||||||
|
**New Methods (Existence Checks):**
|
||||||
|
- `bucket.fastExists({ path })` - ✅ Already existed
|
||||||
|
- `directory.fileExists({ path })` - **NEW** - Check if file exists
|
||||||
|
- `directory.directoryExists(name)` - **NEW** - Check if subdirectory exists
|
||||||
|
- `smartBucket.bucketExists(name)` - **NEW** - Check if bucket exists
|
||||||
|
|
||||||
|
**Benefits:**
|
||||||
|
- ✅ **Simpler API**: Removed 4 redundant *Strict methods
|
||||||
|
- ✅ **Type-safe**: No nullable returns - `Promise<T>` not `Promise<T | null>`
|
||||||
|
- ✅ **Fail-fast**: Errors throw immediately with precise stack traces
|
||||||
|
- ✅ **Consistent**: All methods behave the same way
|
||||||
|
- ✅ **Explicit**: Use exists() to check, then get() to retrieve
|
||||||
|
- ✅ **Better debugging**: Error location is always precise
|
||||||
|
|
||||||
|
**Migration Guide:**
|
||||||
|
|
||||||
|
```typescript
|
||||||
|
// ============================================
|
||||||
|
// Pattern 1: Check then Get (Recommended)
|
||||||
|
// ============================================
|
||||||
|
|
||||||
|
// Before (v3.x):
|
||||||
|
const bucket = await smartBucket.getBucketByName('my-bucket');
|
||||||
|
if (bucket) {
|
||||||
|
// use bucket
|
||||||
|
}
|
||||||
|
|
||||||
|
// After (v4.0):
|
||||||
|
if (await smartBucket.bucketExists('my-bucket')) {
|
||||||
|
const bucket = await smartBucket.getBucketByName('my-bucket'); // guaranteed to exist
|
||||||
|
// use bucket
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================
|
||||||
|
// Pattern 2: Try/Catch
|
||||||
|
// ============================================
|
||||||
|
|
||||||
|
// Before (v3.x):
|
||||||
|
const file = await directory.getFile({ path: 'file.txt' });
|
||||||
|
if (!file) {
|
||||||
|
// Handle not found
|
||||||
|
}
|
||||||
|
|
||||||
|
// After (v4.0):
|
||||||
|
try {
|
||||||
|
const file = await directory.getFile({ path: 'file.txt' });
|
||||||
|
// use file
|
||||||
|
} catch (error) {
|
||||||
|
// Handle not found
|
||||||
|
}
|
||||||
|
|
||||||
|
// ============================================
|
||||||
|
// Pattern 3: Remove *Strict calls
|
||||||
|
// ============================================
|
||||||
|
|
||||||
|
// Before (v3.x):
|
||||||
|
const file = await directory.getFileStrict({ path: 'file.txt' });
|
||||||
|
|
||||||
|
// After (v4.0):
|
||||||
|
const file = await directory.getFile({ path: 'file.txt' }); // already strict
|
||||||
|
|
||||||
|
// ============================================
|
||||||
|
// Pattern 4: Write Operations
|
||||||
|
// ============================================
|
||||||
|
|
||||||
|
// Before (v3.x):
|
||||||
|
const file = await bucket.fastPutStrict({ path: 'file.txt', contents: 'data' });
|
||||||
|
|
||||||
|
// After (v4.0):
|
||||||
|
const file = await bucket.fastPut({ path: 'file.txt', contents: 'data' }); // already strict
|
||||||
|
```
|
||||||
|
|
||||||
## 2025-08-18 - 3.3.10 - fix(helpers)
|
## 2025-08-18 - 3.3.10 - fix(helpers)
|
||||||
Normalize and robustly parse S3 endpoint configuration; use normalized descriptor in SmartBucket and update dev tooling
|
Normalize and robustly parse S3 endpoint configuration; use normalized descriptor in SmartBucket and update dev tooling
|
||||||
|
|
||||||
|
|||||||
14
package.json
14
package.json
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@push.rocks/smartbucket",
|
"name": "@push.rocks/smartbucket",
|
||||||
"version": "3.3.10",
|
"version": "4.1.0",
|
||||||
"description": "A TypeScript library providing a cloud-agnostic interface for managing object storage with functionalities like bucket management, file and directory operations, and advanced features such as metadata handling and file locking.",
|
"description": "A TypeScript library providing a cloud-agnostic interface for managing object storage with functionalities like bucket management, file and directory operations, and advanced features such as metadata handling and file locking.",
|
||||||
"main": "dist_ts/index.js",
|
"main": "dist_ts/index.js",
|
||||||
"typings": "dist_ts/index.d.ts",
|
"typings": "dist_ts/index.d.ts",
|
||||||
@@ -12,22 +12,22 @@
|
|||||||
"build": "(tsbuild --web --allowimplicitany)"
|
"build": "(tsbuild --web --allowimplicitany)"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"@git.zone/tsbuild": "^2.6.7",
|
"@git.zone/tsbuild": "^3.1.0",
|
||||||
"@git.zone/tsrun": "^1.2.49",
|
"@git.zone/tsrun": "^2.0.0",
|
||||||
"@git.zone/tstest": "^2.3.4",
|
"@git.zone/tstest": "^3.0.1",
|
||||||
"@push.rocks/qenv": "^6.1.3",
|
"@push.rocks/qenv": "^6.1.3",
|
||||||
"@push.rocks/tapbundle": "^6.0.3"
|
"@push.rocks/tapbundle": "^6.0.3"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@aws-sdk/client-s3": "^3.864.0",
|
"@aws-sdk/client-s3": "^3.936.0",
|
||||||
"@push.rocks/smartmime": "^2.0.4",
|
"@push.rocks/smartmime": "^2.0.4",
|
||||||
"@push.rocks/smartpath": "^6.0.0",
|
"@push.rocks/smartpath": "^6.0.0",
|
||||||
"@push.rocks/smartpromise": "^4.2.3",
|
"@push.rocks/smartpromise": "^4.2.3",
|
||||||
"@push.rocks/smartrx": "^3.0.10",
|
"@push.rocks/smartrx": "^3.0.10",
|
||||||
"@push.rocks/smartstream": "^3.2.5",
|
"@push.rocks/smartstream": "^3.2.5",
|
||||||
"@push.rocks/smartstring": "^4.0.15",
|
"@push.rocks/smartstring": "^4.1.0",
|
||||||
"@push.rocks/smartunique": "^3.0.9",
|
"@push.rocks/smartunique": "^3.0.9",
|
||||||
"@tsclass/tsclass": "^9.2.0"
|
"@tsclass/tsclass": "^9.3.0"
|
||||||
},
|
},
|
||||||
"private": false,
|
"private": false,
|
||||||
"files": [
|
"files": [
|
||||||
|
|||||||
3958
pnpm-lock.yaml
generated
3958
pnpm-lock.yaml
generated
File diff suppressed because it is too large
Load Diff
@@ -1,3 +1,5 @@
|
|||||||
* The project uses the official s3 client, not the minio client.
|
* The project uses the official s3 client, not the minio client.
|
||||||
* notice the difference between *Strict methods and the normal methods.
|
* **All methods throw by default** (strict mode): - Put operations: `fastPut`, `fastPutStream` throw when file exists and overwrite is false - Get operations: `getBucketByName`, `getFile`, `getSubDirectoryByName` throw when not found
|
||||||
|
* **Use exists() methods to check before getting**: `bucketExists`, `fileExists`, `directoryExists`, `fastExists`
|
||||||
|
* **No *Strict methods**: All removed (fastPutStrict, getBucketByNameStrict, getFileStrict, getSubDirectoryByNameStrict)
|
||||||
* metadata is handled though the MetaData class. Important!
|
* metadata is handled though the MetaData class. Important!
|
||||||
|
|||||||
21
readme.md
21
readme.md
@@ -88,8 +88,8 @@ console.log('🗑️ Bucket removed');
|
|||||||
```typescript
|
```typescript
|
||||||
const bucket = await smartBucket.getBucketByName('my-bucket');
|
const bucket = await smartBucket.getBucketByName('my-bucket');
|
||||||
|
|
||||||
// Simple file upload
|
// Simple file upload (returns File object)
|
||||||
await bucket.fastPut({
|
const file = await bucket.fastPut({
|
||||||
path: 'documents/report.pdf',
|
path: 'documents/report.pdf',
|
||||||
contents: Buffer.from('Your file content here')
|
contents: Buffer.from('Your file content here')
|
||||||
});
|
});
|
||||||
@@ -100,12 +100,23 @@ await bucket.fastPut({
|
|||||||
contents: 'Buy milk\nCall mom\nRule the world'
|
contents: 'Buy milk\nCall mom\nRule the world'
|
||||||
});
|
});
|
||||||
|
|
||||||
// Strict upload (returns File object)
|
// Upload with overwrite control
|
||||||
const uploadedFile = await bucket.fastPutStrict({
|
const uploadedFile = await bucket.fastPut({
|
||||||
path: 'images/logo.png',
|
path: 'images/logo.png',
|
||||||
contents: imageBuffer,
|
contents: imageBuffer,
|
||||||
overwrite: true // Optional: control overwrite behavior
|
overwrite: true // Set to true to replace existing files
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Error handling: fastPut throws if file exists and overwrite is false
|
||||||
|
try {
|
||||||
|
await bucket.fastPut({
|
||||||
|
path: 'existing-file.txt',
|
||||||
|
contents: 'new content'
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.error('Upload failed:', error.message);
|
||||||
|
// Error: Object already exists at path 'existing-file.txt' in bucket 'my-bucket'. Set overwrite:true to replace it.
|
||||||
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Download Files
|
#### Download Files
|
||||||
|
|||||||
76
test/test.local.node+deno.ts
Normal file
76
test/test.local.node+deno.ts
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
import { expect, tap } from '@git.zone/tstest/tapbundle';
|
||||||
|
|
||||||
|
import * as plugins from '../ts/plugins.js';
|
||||||
|
import * as smartbucket from '../ts/index.js';
|
||||||
|
|
||||||
|
class FakeS3Client {
|
||||||
|
private callIndex = 0;
|
||||||
|
|
||||||
|
constructor(private readonly pages: Array<Partial<plugins.s3.ListObjectsV2Output>>) {}
|
||||||
|
|
||||||
|
public async send(_command: any) {
|
||||||
|
const page = this.pages[this.callIndex] || { Contents: [], CommonPrefixes: [], IsTruncated: false };
|
||||||
|
this.callIndex += 1;
|
||||||
|
return page;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
tap.test('MetaData.hasMetaData should return false when metadata file does not exist', async () => {
|
||||||
|
const fakeFile = {
|
||||||
|
name: 'file.txt',
|
||||||
|
parentDirectoryRef: {
|
||||||
|
async getFile() {
|
||||||
|
throw new Error(`File not found at path 'file.txt.metadata'`);
|
||||||
|
},
|
||||||
|
},
|
||||||
|
} as unknown as smartbucket.File;
|
||||||
|
|
||||||
|
const hasMetaData = await smartbucket.MetaData.hasMetaData({ file: fakeFile });
|
||||||
|
expect(hasMetaData).toBeFalse();
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.test('getSubDirectoryByName should create correct parent chain for new nested directories', async () => {
|
||||||
|
const fakeSmartbucket = { s3Client: new FakeS3Client([{ Contents: [], CommonPrefixes: [] }]) } as unknown as smartbucket.SmartBucket;
|
||||||
|
const bucket = new smartbucket.Bucket(fakeSmartbucket, 'test-bucket');
|
||||||
|
const baseDirectory = new smartbucket.Directory(bucket, null as any, '');
|
||||||
|
|
||||||
|
const nestedDirectory = await baseDirectory.getSubDirectoryByName('level1/level2', { getEmptyDirectory: true });
|
||||||
|
|
||||||
|
expect(nestedDirectory.name).toEqual('level2');
|
||||||
|
expect(nestedDirectory.parentDirectoryRef.name).toEqual('level1');
|
||||||
|
expect(nestedDirectory.getBasePath()).toEqual('level1/level2/');
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.test('listFiles should aggregate results across paginated ListObjectsV2 responses', async () => {
|
||||||
|
const firstPage = {
|
||||||
|
Contents: Array.from({ length: 1000 }, (_, index) => ({ Key: `file-${index}` })),
|
||||||
|
IsTruncated: true,
|
||||||
|
NextContinuationToken: 'token-1',
|
||||||
|
};
|
||||||
|
const secondPage = {
|
||||||
|
Contents: Array.from({ length: 200 }, (_, index) => ({ Key: `file-${1000 + index}` })),
|
||||||
|
IsTruncated: false,
|
||||||
|
};
|
||||||
|
const fakeSmartbucket = { s3Client: new FakeS3Client([firstPage, secondPage]) } as unknown as smartbucket.SmartBucket;
|
||||||
|
const bucket = new smartbucket.Bucket(fakeSmartbucket, 'test-bucket');
|
||||||
|
const baseDirectory = new smartbucket.Directory(bucket, null as any, '');
|
||||||
|
|
||||||
|
const files = await baseDirectory.listFiles();
|
||||||
|
expect(files.length).toEqual(1200);
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.test('listDirectories should aggregate CommonPrefixes across pagination', async () => {
|
||||||
|
const fakeSmartbucket = {
|
||||||
|
s3Client: new FakeS3Client([
|
||||||
|
{ CommonPrefixes: [{ Prefix: 'dirA/' }], IsTruncated: true, NextContinuationToken: 'token-1' },
|
||||||
|
{ CommonPrefixes: [{ Prefix: 'dirB/' }], IsTruncated: false },
|
||||||
|
]),
|
||||||
|
} as unknown as smartbucket.SmartBucket;
|
||||||
|
const bucket = new smartbucket.Bucket(fakeSmartbucket, 'test-bucket');
|
||||||
|
const baseDirectory = new smartbucket.Directory(bucket, null as any, '');
|
||||||
|
|
||||||
|
const directories = await baseDirectory.listDirectories();
|
||||||
|
expect(directories.map((d) => d.name)).toEqual(['dirA', 'dirB']);
|
||||||
|
});
|
||||||
|
|
||||||
|
export default tap.start();
|
||||||
@@ -16,7 +16,7 @@ tap.test('should create a valid smartbucket', async () => {
|
|||||||
endpoint: await testQenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
|
endpoint: await testQenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
|
||||||
});
|
});
|
||||||
expect(testSmartbucket).toBeInstanceOf(smartbucket.SmartBucket);
|
expect(testSmartbucket).toBeInstanceOf(smartbucket.SmartBucket);
|
||||||
myBucket = await testSmartbucket.getBucketByNameStrict(await testQenv.getEnvVarOnDemandStrict('S3_BUCKET'),);
|
myBucket = await testSmartbucket.getBucketByName(await testQenv.getEnvVarOnDemandStrict('S3_BUCKET'),);
|
||||||
expect(myBucket).toBeInstanceOf(smartbucket.Bucket);
|
expect(myBucket).toBeInstanceOf(smartbucket.Bucket);
|
||||||
expect(myBucket.name).toEqual('test-pushrocks-smartbucket');
|
expect(myBucket.name).toEqual('test-pushrocks-smartbucket');
|
||||||
});
|
});
|
||||||
@@ -17,7 +17,7 @@ tap.test('should create a valid smartbucket', async () => {
|
|||||||
endpoint: await testQenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
|
endpoint: await testQenv.getEnvVarOnDemandStrict('S3_ENDPOINT'),
|
||||||
});
|
});
|
||||||
expect(testSmartbucket).toBeInstanceOf(smartbucket.SmartBucket);
|
expect(testSmartbucket).toBeInstanceOf(smartbucket.SmartBucket);
|
||||||
myBucket = await testSmartbucket.getBucketByNameStrict(await testQenv.getEnvVarOnDemandStrict('S3_BUCKET'),);
|
myBucket = await testSmartbucket.getBucketByName(await testQenv.getEnvVarOnDemandStrict('S3_BUCKET'),);
|
||||||
expect(myBucket).toBeInstanceOf(smartbucket.Bucket);
|
expect(myBucket).toBeInstanceOf(smartbucket.Bucket);
|
||||||
expect(myBucket.name).toEqual('test-pushrocks-smartbucket');
|
expect(myBucket.name).toEqual('test-pushrocks-smartbucket');
|
||||||
});
|
});
|
||||||
@@ -30,7 +30,7 @@ tap.test('should clean all contents', async () => {
|
|||||||
|
|
||||||
tap.test('should delete a file into the normally', async () => {
|
tap.test('should delete a file into the normally', async () => {
|
||||||
const path = 'trashtest/trashme.txt';
|
const path = 'trashtest/trashme.txt';
|
||||||
const file = await myBucket.fastPutStrict({
|
const file = await myBucket.fastPut({
|
||||||
path,
|
path,
|
||||||
contents: 'I\'m in the trash test content!',
|
contents: 'I\'m in the trash test content!',
|
||||||
});
|
});
|
||||||
@@ -44,7 +44,7 @@ tap.test('should delete a file into the normally', async () => {
|
|||||||
|
|
||||||
tap.test('should put a file into the trash', async () => {
|
tap.test('should put a file into the trash', async () => {
|
||||||
const path = 'trashtest/trashme.txt';
|
const path = 'trashtest/trashme.txt';
|
||||||
const file = await myBucket.fastPutStrict({
|
const file = await myBucket.fastPut({
|
||||||
path,
|
path,
|
||||||
contents: 'I\'m in the trash test content!',
|
contents: 'I\'m in the trash test content!',
|
||||||
});
|
});
|
||||||
@@ -76,7 +76,7 @@ tap.test('should put a file into the trash', async () => {
|
|||||||
|
|
||||||
tap.test('should restore a file from trash', async () => {
|
tap.test('should restore a file from trash', async () => {
|
||||||
const baseDirectory = await myBucket.getBaseDirectory();
|
const baseDirectory = await myBucket.getBaseDirectory();
|
||||||
const file = await baseDirectory.getFileStrict({
|
const file = await baseDirectory.getFile({
|
||||||
path: 'trashtest/trashme.txt',
|
path: 'trashtest/trashme.txt',
|
||||||
getFromTrash: true
|
getFromTrash: true
|
||||||
});
|
});
|
||||||
@@ -3,6 +3,6 @@
|
|||||||
*/
|
*/
|
||||||
export const commitinfo = {
|
export const commitinfo = {
|
||||||
name: '@push.rocks/smartbucket',
|
name: '@push.rocks/smartbucket',
|
||||||
version: '3.3.10',
|
version: '4.1.0',
|
||||||
description: 'A TypeScript library providing a cloud-agnostic interface for managing object storage with functionalities like bucket management, file and directory operations, and advanced features such as metadata handling and file locking.'
|
description: 'A TypeScript library providing a cloud-agnostic interface for managing object storage with functionalities like bucket management, file and directory operations, and advanced features such as metadata handling and file locking.'
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ import { Trash } from './classes.trash.js';
|
|||||||
* operate in S3 basic fashion on blobs of data.
|
* operate in S3 basic fashion on blobs of data.
|
||||||
*/
|
*/
|
||||||
export class Bucket {
|
export class Bucket {
|
||||||
public static async getBucketByName(smartbucketRef: SmartBucket, bucketNameArg: string) {
|
public static async getBucketByName(smartbucketRef: SmartBucket, bucketNameArg: string): Promise<Bucket> {
|
||||||
const command = new plugins.s3.ListBucketsCommand({});
|
const command = new plugins.s3.ListBucketsCommand({});
|
||||||
const buckets = await smartbucketRef.s3Client.send(command);
|
const buckets = await smartbucketRef.s3Client.send(command);
|
||||||
const foundBucket = buckets.Buckets!.find((bucket) => bucket.Name === bucketNameArg);
|
const foundBucket = buckets.Buckets!.find((bucket) => bucket.Name === bucketNameArg);
|
||||||
@@ -24,8 +24,7 @@ export class Bucket {
|
|||||||
console.log(`Taking this as base for new Bucket instance`);
|
console.log(`Taking this as base for new Bucket instance`);
|
||||||
return new this(smartbucketRef, bucketNameArg);
|
return new this(smartbucketRef, bucketNameArg);
|
||||||
} else {
|
} else {
|
||||||
console.log(`did not find bucket by name: ${bucketNameArg}`);
|
throw new Error(`Bucket '${bucketNameArg}' not found.`);
|
||||||
return null;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -71,7 +70,7 @@ export class Bucket {
|
|||||||
}
|
}
|
||||||
const checkPath = await helpers.reducePathDescriptorToPath(pathDescriptorArg);
|
const checkPath = await helpers.reducePathDescriptorToPath(pathDescriptorArg);
|
||||||
const baseDirectory = await this.getBaseDirectory();
|
const baseDirectory = await this.getBaseDirectory();
|
||||||
return await baseDirectory.getSubDirectoryByNameStrict(checkPath, {
|
return await baseDirectory.getSubDirectoryByName(checkPath, {
|
||||||
getEmptyDirectory: true,
|
getEmptyDirectory: true,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
@@ -88,15 +87,16 @@ export class Bucket {
|
|||||||
contents: string | Buffer;
|
contents: string | Buffer;
|
||||||
overwrite?: boolean;
|
overwrite?: boolean;
|
||||||
}
|
}
|
||||||
): Promise<File | null> {
|
): Promise<File> {
|
||||||
try {
|
try {
|
||||||
const reducedPath = await helpers.reducePathDescriptorToPath(optionsArg);
|
const reducedPath = await helpers.reducePathDescriptorToPath(optionsArg);
|
||||||
const exists = await this.fastExists({ path: reducedPath });
|
const exists = await this.fastExists({ path: reducedPath });
|
||||||
|
|
||||||
if (exists && !optionsArg.overwrite) {
|
if (exists && !optionsArg.overwrite) {
|
||||||
const errorText = `Object already exists at path '${reducedPath}' in bucket '${this.name}'.`;
|
throw new Error(
|
||||||
console.error(errorText);
|
`Object already exists at path '${reducedPath}' in bucket '${this.name}'. ` +
|
||||||
return null;
|
`Set overwrite:true to replace it.`
|
||||||
|
);
|
||||||
} else if (exists && optionsArg.overwrite) {
|
} else if (exists && optionsArg.overwrite) {
|
||||||
console.log(
|
console.log(
|
||||||
`Overwriting existing object at path '${reducedPath}' in bucket '${this.name}'.`
|
`Overwriting existing object at path '${reducedPath}' in bucket '${this.name}'.`
|
||||||
@@ -129,13 +129,6 @@ export class Bucket {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
public async fastPutStrict(...args: Parameters<Bucket['fastPut']>) {
|
|
||||||
const file = await this.fastPut(...args);
|
|
||||||
if (!file) {
|
|
||||||
throw new Error(`File not stored at path '${args[0].path}'`);
|
|
||||||
}
|
|
||||||
return file;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* get file
|
* get file
|
||||||
@@ -259,10 +252,10 @@ export class Bucket {
|
|||||||
const exists = await this.fastExists({ path: optionsArg.path });
|
const exists = await this.fastExists({ path: optionsArg.path });
|
||||||
|
|
||||||
if (exists && !optionsArg.overwrite) {
|
if (exists && !optionsArg.overwrite) {
|
||||||
console.error(
|
throw new Error(
|
||||||
`Object already exists at path '${optionsArg.path}' in bucket '${this.name}'.`
|
`Object already exists at path '${optionsArg.path}' in bucket '${this.name}'. ` +
|
||||||
|
`Set overwrite:true to replace it.`
|
||||||
);
|
);
|
||||||
return;
|
|
||||||
} else if (exists && optionsArg.overwrite) {
|
} else if (exists && optionsArg.overwrite) {
|
||||||
console.log(
|
console.log(
|
||||||
`Overwriting existing object at path '${optionsArg.path}' in bucket '${this.name}'.`
|
`Overwriting existing object at path '${optionsArg.path}' in bucket '${this.name}'.`
|
||||||
@@ -460,7 +453,7 @@ export class Bucket {
|
|||||||
Range: `bytes=0-${optionsArg.length - 1}`,
|
Range: `bytes=0-${optionsArg.length - 1}`,
|
||||||
});
|
});
|
||||||
const response = await this.smartbucketRef.s3Client.send(command);
|
const response = await this.smartbucketRef.s3Client.send(command);
|
||||||
const chunks = [];
|
const chunks: Buffer[] = [];
|
||||||
const stream = response.Body as any; // SdkStreamMixin includes readable stream
|
const stream = response.Body as any; // SdkStreamMixin includes readable stream
|
||||||
|
|
||||||
for await (const chunk of stream) {
|
for await (const chunk of stream) {
|
||||||
|
|||||||
@@ -69,7 +69,7 @@ export class Directory {
|
|||||||
path: string;
|
path: string;
|
||||||
createWithContents?: string | Buffer;
|
createWithContents?: string | Buffer;
|
||||||
getFromTrash?: boolean;
|
getFromTrash?: boolean;
|
||||||
}): Promise<File | null> {
|
}): Promise<File> {
|
||||||
const pathDescriptor = {
|
const pathDescriptor = {
|
||||||
directory: this,
|
directory: this,
|
||||||
path: optionsArg.path,
|
path: optionsArg.path,
|
||||||
@@ -83,7 +83,7 @@ export class Directory {
|
|||||||
return trashedFile;
|
return trashedFile;
|
||||||
}
|
}
|
||||||
if (!exists && !optionsArg.createWithContents) {
|
if (!exists && !optionsArg.createWithContents) {
|
||||||
return null;
|
throw new Error(`File not found at path '${optionsArg.path}'`);
|
||||||
}
|
}
|
||||||
if (!exists && optionsArg.createWithContents) {
|
if (!exists && optionsArg.createWithContents) {
|
||||||
await File.create({
|
await File.create({
|
||||||
@@ -98,32 +98,66 @@ export class Directory {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* gets a file strictly
|
* Check if a file exists in this directory
|
||||||
* @param args
|
|
||||||
* @returns
|
|
||||||
*/
|
*/
|
||||||
public async getFileStrict(...args: Parameters<Directory['getFile']>) {
|
public async fileExists(optionsArg: { path: string }): Promise<boolean> {
|
||||||
const file = await this.getFile(...args);
|
const pathDescriptor = {
|
||||||
if (!file) {
|
directory: this,
|
||||||
throw new Error(`File not found at path '${args[0].path}'`);
|
path: optionsArg.path,
|
||||||
}
|
};
|
||||||
return file;
|
return this.bucketRef.fastExists({
|
||||||
|
path: await helpers.reducePathDescriptorToPath(pathDescriptor),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Check if a subdirectory exists
|
||||||
|
*/
|
||||||
|
public async directoryExists(dirNameArg: string): Promise<boolean> {
|
||||||
|
const directories = await this.listDirectories();
|
||||||
|
return directories.some(dir => dir.name === dirNameArg);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Collects all ListObjectsV2 pages for a prefix.
|
||||||
|
*/
|
||||||
|
private async listObjectsV2AllPages(prefix: string, delimiter?: string) {
|
||||||
|
const allContents: plugins.s3._Object[] = [];
|
||||||
|
const allCommonPrefixes: plugins.s3.CommonPrefix[] = [];
|
||||||
|
let continuationToken: string | undefined;
|
||||||
|
|
||||||
|
do {
|
||||||
|
const command = new plugins.s3.ListObjectsV2Command({
|
||||||
|
Bucket: this.bucketRef.name,
|
||||||
|
Prefix: prefix,
|
||||||
|
Delimiter: delimiter,
|
||||||
|
ContinuationToken: continuationToken,
|
||||||
|
});
|
||||||
|
const response = await this.bucketRef.smartbucketRef.s3Client.send(command);
|
||||||
|
|
||||||
|
if (response.Contents) {
|
||||||
|
allContents.push(...response.Contents);
|
||||||
|
}
|
||||||
|
if (response.CommonPrefixes) {
|
||||||
|
allCommonPrefixes.push(...response.CommonPrefixes);
|
||||||
|
}
|
||||||
|
|
||||||
|
continuationToken = response.IsTruncated ? response.NextContinuationToken : undefined;
|
||||||
|
} while (continuationToken);
|
||||||
|
|
||||||
|
return { contents: allContents, commonPrefixes: allCommonPrefixes };
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* lists all files
|
* lists all files
|
||||||
*/
|
*/
|
||||||
public async listFiles(): Promise<File[]> {
|
public async listFiles(): Promise<File[]> {
|
||||||
const command = new plugins.s3.ListObjectsV2Command({
|
const { contents } = await this.listObjectsV2AllPages(this.getBasePath(), '/');
|
||||||
Bucket: this.bucketRef.name,
|
|
||||||
Prefix: this.getBasePath(),
|
|
||||||
Delimiter: '/',
|
|
||||||
});
|
|
||||||
const response = await this.bucketRef.smartbucketRef.s3Client.send(command);
|
|
||||||
const fileArray: File[] = [];
|
const fileArray: File[] = [];
|
||||||
|
|
||||||
response.Contents?.forEach((item) => {
|
contents.forEach((item) => {
|
||||||
if (item.Key && !item.Key.endsWith('/')) {
|
if (item.Key && !item.Key.endsWith('/')) {
|
||||||
const subtractedPath = item.Key.replace(this.getBasePath(), '');
|
const subtractedPath = item.Key.replace(this.getBasePath(), '');
|
||||||
if (!subtractedPath.includes('/')) {
|
if (!subtractedPath.includes('/')) {
|
||||||
@@ -145,16 +179,11 @@ export class Directory {
|
|||||||
*/
|
*/
|
||||||
public async listDirectories(): Promise<Directory[]> {
|
public async listDirectories(): Promise<Directory[]> {
|
||||||
try {
|
try {
|
||||||
const command = new plugins.s3.ListObjectsV2Command({
|
const { commonPrefixes } = await this.listObjectsV2AllPages(this.getBasePath(), '/');
|
||||||
Bucket: this.bucketRef.name,
|
|
||||||
Prefix: this.getBasePath(),
|
|
||||||
Delimiter: '/',
|
|
||||||
});
|
|
||||||
const response = await this.bucketRef.smartbucketRef.s3Client.send(command);
|
|
||||||
const directoryArray: Directory[] = [];
|
const directoryArray: Directory[] = [];
|
||||||
|
|
||||||
if (response.CommonPrefixes) {
|
if (commonPrefixes) {
|
||||||
response.CommonPrefixes.forEach((item) => {
|
commonPrefixes.forEach((item) => {
|
||||||
if (item.Prefix) {
|
if (item.Prefix) {
|
||||||
const subtractedPath = item.Prefix.replace(this.getBasePath(), '');
|
const subtractedPath = item.Prefix.replace(this.getBasePath(), '');
|
||||||
if (subtractedPath.endsWith('/')) {
|
if (subtractedPath.endsWith('/')) {
|
||||||
@@ -206,7 +235,7 @@ export class Directory {
|
|||||||
* if the path is a file path, it will be treated as a file and the parent directory will be returned
|
* if the path is a file path, it will be treated as a file and the parent directory will be returned
|
||||||
*/
|
*/
|
||||||
couldBeFilePath?: boolean;
|
couldBeFilePath?: boolean;
|
||||||
} = {}): Promise<Directory | null> {
|
} = {}): Promise<Directory> {
|
||||||
|
|
||||||
const dirNameArray = dirNameArg.split('/').filter(str => str.trim() !== "");
|
const dirNameArray = dirNameArg.split('/').filter(str => str.trim() !== "");
|
||||||
|
|
||||||
@@ -226,7 +255,7 @@ export class Directory {
|
|||||||
return returnDirectory;
|
return returnDirectory;
|
||||||
}
|
}
|
||||||
if (optionsArg.getEmptyDirectory || optionsArg.createWithInitializerFile) {
|
if (optionsArg.getEmptyDirectory || optionsArg.createWithInitializerFile) {
|
||||||
returnDirectory = new Directory(this.bucketRef, this, dirNameToSearch);
|
returnDirectory = new Directory(this.bucketRef, directoryArg, dirNameToSearch);
|
||||||
}
|
}
|
||||||
if (isFinalDirectory && optionsArg.createWithInitializerFile) {
|
if (isFinalDirectory && optionsArg.createWithInitializerFile) {
|
||||||
returnDirectory?.createEmptyFile('00init.txt');
|
returnDirectory?.createEmptyFile('00init.txt');
|
||||||
@@ -253,16 +282,12 @@ export class Directory {
|
|||||||
wantedDirectory = await getDirectory(directoryToSearchIn, dirNameToSearch, counter === dirNameArray.length);
|
wantedDirectory = await getDirectory(directoryToSearchIn, dirNameToSearch, counter === dirNameArray.length);
|
||||||
}
|
}
|
||||||
|
|
||||||
return wantedDirectory || null;
|
if (!wantedDirectory) {
|
||||||
|
throw new Error(`Directory not found at path '${dirNameArg}'`);
|
||||||
|
}
|
||||||
|
return wantedDirectory;
|
||||||
}
|
}
|
||||||
|
|
||||||
public async getSubDirectoryByNameStrict(...args: Parameters<Directory['getSubDirectoryByName']>) {
|
|
||||||
const directory = await this.getSubDirectoryByName(...args);
|
|
||||||
if (!directory) {
|
|
||||||
throw new Error(`Directory not found at path '${args[0]}'`);
|
|
||||||
}
|
|
||||||
return directory;
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* moves the directory
|
* moves the directory
|
||||||
@@ -360,7 +385,7 @@ export class Directory {
|
|||||||
*/
|
*/
|
||||||
mode?: 'permanent' | 'trash';
|
mode?: 'permanent' | 'trash';
|
||||||
}) {
|
}) {
|
||||||
const file = await this.getFileStrict({
|
const file = await this.getFile({
|
||||||
path: optionsArg.path,
|
path: optionsArg.path,
|
||||||
});
|
});
|
||||||
await file.delete({
|
await file.delete({
|
||||||
|
|||||||
@@ -245,7 +245,7 @@ export class File {
|
|||||||
|
|
||||||
// lets update references of this
|
// lets update references of this
|
||||||
const baseDirectory = await this.parentDirectoryRef.bucketRef.getBaseDirectory();
|
const baseDirectory = await this.parentDirectoryRef.bucketRef.getBaseDirectory();
|
||||||
this.parentDirectoryRef = await baseDirectory.getSubDirectoryByNameStrict(
|
this.parentDirectoryRef = await baseDirectory.getSubDirectoryByName(
|
||||||
await helpers.reducePathDescriptorToPath(pathDescriptorArg),
|
await helpers.reducePathDescriptorToPath(pathDescriptorArg),
|
||||||
{
|
{
|
||||||
couldBeFilePath: true,
|
couldBeFilePath: true,
|
||||||
|
|||||||
@@ -4,11 +4,23 @@ import { File } from './classes.file.js';
|
|||||||
|
|
||||||
export class MetaData {
|
export class MetaData {
|
||||||
public static async hasMetaData(optionsArg: { file: File }) {
|
public static async hasMetaData(optionsArg: { file: File }) {
|
||||||
// lets find the existing metadata file
|
// try finding the existing metadata file; return false if it doesn't exist
|
||||||
const existingFile = await optionsArg.file.parentDirectoryRef.getFile({
|
try {
|
||||||
path: optionsArg.file.name + '.metadata',
|
const existingFile = await optionsArg.file.parentDirectoryRef.getFile({
|
||||||
});
|
path: optionsArg.file.name + '.metadata',
|
||||||
return !!existingFile;
|
});
|
||||||
|
return !!existingFile;
|
||||||
|
} catch (error: any) {
|
||||||
|
const message = error?.message || '';
|
||||||
|
const isNotFound =
|
||||||
|
message.includes('File not found') ||
|
||||||
|
error?.name === 'NotFound' ||
|
||||||
|
error?.$metadata?.httpStatusCode === 404;
|
||||||
|
if (isNotFound) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// static
|
// static
|
||||||
@@ -17,7 +29,7 @@ export class MetaData {
|
|||||||
metaData.fileRef = optionsArg.file;
|
metaData.fileRef = optionsArg.file;
|
||||||
|
|
||||||
// lets find the existing metadata file
|
// lets find the existing metadata file
|
||||||
metaData.metadataFile = await metaData.fileRef.parentDirectoryRef.getFileStrict({
|
metaData.metadataFile = await metaData.fileRef.parentDirectoryRef.getFile({
|
||||||
path: metaData.fileRef.name + '.metadata',
|
path: metaData.fileRef.name + '.metadata',
|
||||||
createWithContents: '{}',
|
createWithContents: '{}',
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -42,11 +42,12 @@ export class SmartBucket {
|
|||||||
return Bucket.getBucketByName(this, bucketNameArg);
|
return Bucket.getBucketByName(this, bucketNameArg);
|
||||||
}
|
}
|
||||||
|
|
||||||
public async getBucketByNameStrict(...args: Parameters<SmartBucket['getBucketByName']>) {
|
/**
|
||||||
const bucket = await this.getBucketByName(...args);
|
* Check if a bucket exists
|
||||||
if (!bucket) {
|
*/
|
||||||
throw new Error(`Bucket ${args[0]} does not exist.`);
|
public async bucketExists(bucketNameArg: string): Promise<boolean> {
|
||||||
}
|
const command = new plugins.s3.ListBucketsCommand({});
|
||||||
return bucket;
|
const buckets = await this.s3Client.send(command);
|
||||||
|
return buckets.Buckets?.some(bucket => bucket.Name === bucketNameArg) ?? false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ export class Trash {
|
|||||||
const trashDir = await this.getTrashDir();
|
const trashDir = await this.getTrashDir();
|
||||||
const originalPath = await helpers.reducePathDescriptorToPath(pathDescriptor);
|
const originalPath = await helpers.reducePathDescriptorToPath(pathDescriptor);
|
||||||
const trashKey = await this.getTrashKeyByOriginalBasePath(originalPath);
|
const trashKey = await this.getTrashKeyByOriginalBasePath(originalPath);
|
||||||
return trashDir.getFileStrict({ path: trashKey });
|
return trashDir.getFile({ path: trashKey });
|
||||||
}
|
}
|
||||||
|
|
||||||
public async getTrashKeyByOriginalBasePath (originalPath: string): Promise<string> {
|
public async getTrashKeyByOriginalBasePath (originalPath: string): Promise<string> {
|
||||||
|
|||||||
@@ -2,3 +2,5 @@ export * from './classes.smartbucket.js';
|
|||||||
export * from './classes.bucket.js';
|
export * from './classes.bucket.js';
|
||||||
export * from './classes.directory.js';
|
export * from './classes.directory.js';
|
||||||
export * from './classes.file.js';
|
export * from './classes.file.js';
|
||||||
|
export * from './classes.metadata.js';
|
||||||
|
export * from './classes.trash.js';
|
||||||
|
|||||||
@@ -1,8 +1,8 @@
|
|||||||
// plugins.ts
|
// plugins.ts
|
||||||
|
|
||||||
// node native
|
// node native
|
||||||
import * as path from 'path';
|
import * as path from 'node:path';
|
||||||
import * as stream from 'stream';
|
import * as stream from 'node:stream';
|
||||||
|
|
||||||
export { path, stream };
|
export { path, stream };
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user