feat(listing): Add memory-efficient listing APIs: async generator, RxJS observable, and cursor pagination; export ListCursor and Minimatch; add minimatch dependency; bump to 4.2.0

This commit is contained in:
2025-11-20 15:14:11 +00:00
parent dd6efa4908
commit 65c7bcf12c
13 changed files with 1080 additions and 95 deletions

View File

@@ -1,5 +1,60 @@
# Changelog
## 2025-11-20 - 4.3.0 - feat(listing)
Add memory-efficient listing APIs: async generator, RxJS observable, and cursor pagination; export ListCursor and Minimatch; add minimatch dependency; bump to 4.2.0
- Added memory-efficient listing methods on Bucket: listAllObjects (async generator), listAllObjectsObservable (RxJS Observable), createCursor (returns ListCursor) and listAllObjectsArray (convenience array collector).
- New ListCursor class (ts/classes.listcursor.ts) providing page-based iteration: next(), hasMore(), reset(), getToken()/setToken().
- Added glob matching helper findByGlob(pattern) using minimatch (exported via plugins.Minimatch).
- Exported ListCursor from ts/index.ts and exported Minimatch via ts/plugins.ts.
- Added minimatch dependency in package.json and bumped package version to 4.2.0; increased test timeout to 120s.
- Updated tests to read S3_SECRETKEY, S3_PORT and to assert bucket name from env (test/test.node+deno.ts, test/test.trash.node+deno.ts).
- No breaking changes: new APIs are additive and existing behavior preserved.
## 2025-11-20 - 4.2.0 - feat(listing)
Add memory-efficient listing with async generators, RxJS observables, and cursor pagination for huge buckets
**New Memory-Efficient Listing Methods:**
**Async Generator (Recommended for most use cases):**
- `Bucket.listAllObjects(prefix?)` - Stream object keys one at a time using `for await...of`
- `Bucket.findByGlob(pattern)` - Find objects matching glob patterns (e.g., `**/*.json`, `npm/packages/*/index.json`)
- Memory efficient, supports early termination, composable
**RxJS Observable (For complex reactive pipelines):**
- `Bucket.listAllObjectsObservable(prefix?)` - Emit keys as Observable for use with RxJS operators (filter, map, take, etc.)
- Perfect for complex data transformations and reactive architectures
**Cursor Pattern (For manual pagination control):**
- `Bucket.createCursor(prefix?, options?)` - Create cursor for explicit page-by-page iteration
- `ListCursor.next()` - Fetch next page of results
- `ListCursor.hasMore()` - Check if more results available
- `ListCursor.reset()` - Reset to beginning
- `ListCursor.getToken()` / `ListCursor.setToken()` - Save/restore pagination state
- Ideal for UI pagination and resumable operations
**Convenience Methods:**
- `Bucket.listAllObjectsArray(prefix?)` - Collect all keys into array (WARNING: loads all into memory)
**Benefits:**
- ✅ Memory-efficient streaming for buckets with millions of objects
- ✅ Three patterns for different use cases (generators, observables, cursors)
- ✅ Support for early termination and incremental processing
- ✅ Glob pattern matching with minimatch
- ✅ Full TypeScript support with proper types
- ✅ Zero breaking changes - all new methods
**Dependencies:**
- Added `minimatch` for glob pattern support
**Files Changed:**
- `ts/classes.bucket.ts` - Added all listing methods
- `ts/classes.listcursor.ts` - NEW: Cursor implementation
- `ts/plugins.ts` - Export Minimatch
- `ts/index.ts` - Export ListCursor
- `test/test.listing.node+deno.ts` - NEW: Comprehensive listing tests
- `package.json` - Added minimatch dependency
## 2025-11-20 - 4.1.0 - feat(core)
Add S3 endpoint normalization, directory pagination, improved metadata checks, trash support, and related tests