v6.0.0

- Introduced a detailed guide on query filters in the README, covering basic filtering, comparison operators, array operators, logical operators, element operators, and advanced filtering patterns.
- Implemented a type-safe filtering system in `classes.doc.ts` with `MongoFilterCondition` and `MongoFilter` types to support MongoDB operators while maintaining nested type safety.
- Enhanced error handling for invalid operators and conditions in the filtering logic.
- Added extensive tests for various filtering scenarios, including basic, comparison, array, logical, and complex filters, ensuring robust functionality and performance.
- Implemented security measures to prevent the use of dangerous operators like `$where` and validate operator usage.

.vscode/settings.json

@@ -22,5 +22,6 @@
         }
       }
     }
-  ]
+  ],
+  "deno.enable": false
 }

changelog.md

@@ -1,5 +1,253 @@
 # Changelog
 
+## 2025-11-17 - 6.0.0 - BREAKING CHANGE(decorators)
+Migrate to TC39 Stage 3 decorators and refactor decorator metadata handling; update class initialization, lucene adapter fixes and docs
+
+- Switch all decorators to TC39 Stage 3 signatures and metadata usage (use context.metadata and context.addInitializer) — affects svDb, globalSvDb, searchable, unI, index, Collection and managed.
+- Refactor Collection/managed decorators to read and initialize prototype/constructor properties from context.metadata to ensure prototype properties are available before instance creation (ts/classes.collection.ts).
+- Improve search implementation: add a Lucene parser and transformer with safer MongoDB query generation, wildcard/fuzzy handling and properly structured boolean operators (ts/classes.lucene.adapter.ts).
+- Search integration updated to use the new adapter and handle advanced Lucene syntax and edge cases more robustly.
+- Bump dev tooling versions: @git.zone/tsbuild -> ^3.1.0 and @git.zone/tsrun -> ^2.0.0.
+- Documentation: update README and add readme.hints.md describing the TC39 decorator migration, minimum TypeScript (>=5.2) and Deno notes; tests adjusted accordingly.
+- Clean up project memory/config files related to the previous decorator approach and Deno configuration adjustments.
+
+## 2025-11-17 - 5.16.7 - fix(classes.collection)
+Improve Deno and TypeScript compatibility: Collection decorator _svDbOptions forwarding and config cleanup
+
+- Collection decorator: capture original constructor and forward _svDbOptions to ensure property decorator options (serialize/deserialize) remain accessible in Deno environments.
+- Collection decorator: keep instance getter defined on prototype for Deno compatibility (no behavior change, clarifies forwarding logic).
+- Build/config: removed experimentalDecorators and useDefineForClassFields from deno.json and tsconfig.json to avoid Deno/TS build issues and rely on default compilation settings.
+
+## 2025-11-17 - 5.16.6 - fix(classes)
+Add Deno compatibility, prototype-safe decorators and safe collection accessor; bump a few deps
+
+- Add deno.json to enable experimentalDecorators and target ES2022/DOM for Deno builds.
+- Introduce getCollectionSafe() on SmartDataDbDoc and use it for save/update/delete/findOne to avoid runtime errors when instance 'collection' is not present.
+- Change several instance properties (globalSaveableProperties, uniqueIndexes, regularIndexes, saveableProperties) to 'declare' so decorator-set prototype properties are not shadowed (Deno compatibility).
+- Enhance @Collection decorator: capture original constructor/prototype for Deno, define prototype getter for collection on decorated class, attach docCtor for searchableFields, and forward _svDbOptions to the original constructor to preserve serializer metadata.
+- Improve text/search index handling by relying on docCtor.searchableFields and guarding text index creation.
+- Bump dependencies/devDependencies: @push.rocks/smartmongo -> ^2.0.14, @git.zone/tsbuild -> ^2.7.1, @git.zone/tstest -> ^2.8.1.
+- These are non-breaking runtime compatibility and developer-experience fixes; intended as a patch release.
+
+## 2025-11-16 - 5.16.5 - fix(watcher)
+Update dependencies, tooling and watcher import; add .serena cache ignore
+
+- Bump runtime dependencies: @push.rocks/smartlog 3.1.8 → 3.1.10, @push.rocks/smartstring 4.0.15 → 4.1.0, @push.rocks/taskbuffer 3.1.7 → 3.4.0, @tsclass/tsclass 9.2.0 → 9.3.0, mongodb 6.18.0 → 6.20.0
+- Bump devDependencies: @git.zone/tsbuild 2.6.7 → 2.6.8, @git.zone/tsrun 1.2.44 → 1.6.2, @git.zone/tstest 2.3.5 → 2.6.2
+- Switch EventEmitter import to node:events in ts/classes.watcher.ts to use the namespaced Node import
+- Add .serena/.gitignore to ignore /cache
+
+## 2025-08-18 - 5.16.4 - fix(classes.doc (convertFilterForMongoDb))
+Improve filter conversion: handle logical operators, merge operator objects, add nested filter tests and docs, and fix test script
+
+- Fix package.json test script: remove stray dot in tstest --verbose argument to ensure tests run correctly
+- Enhance convertFilterForMongoDb in ts/classes.doc.ts to properly handle logical operators ($and, $or, $nor, $not) and return them recursively
+- Merge operator objects for the same field path (e.g. combining $gte and $lte) to avoid overwriting operator clauses when object and dot-notation are mixed
+- Add validation/guards for operator argument types (e.g. $in, $nin, $all must be arrays; $size must be numeric) and preserve existing behavior blocking $where for security
+- Add comprehensive nested filter tests in test/test.filters.ts to cover deep nested object queries, $elemMatch, array size, $all, $in on nested fields and more
+- Expand README filtering section with detailed examples for basic filtering, deep nested filters, comparison operators, array operations, logical and element operators, and advanced patterns
+
+## 2025-08-18 - 5.16.3 - fix(docs)
+Add local Claude settings and remove outdated codex.md
+
+- Added .claude/settings.local.json to store local Claude/assistant permissions and configuration.
+- Removed codex.md (project overview) — documentation file deleted.
+- No runtime/library code changes; documentation/configuration-only update, bump patch version.
+
+## 2025-08-18 - 5.16.2 - fix(readme)
+Update README: clarify examples, expand search/cursor/docs and add local Claude settings
+
+- Refined README wording and structure: clearer Quick Start, improved examples and developer-focused phrasing
+- Expanded documentation for search, cursors, change streams, distributed coordination, transactions and EasyStore with more concrete code examples
+- Adjusted code examples to show safer defaults (ID generation, status/tags, connection pooling) and improved best-practices guidance
+- Added .claude/settings.local.json to provide local assistant/CI permission configuration
+
+## 2025-08-12 - 5.16.1 - fix(core)
+Improve error handling and logging; enhance search query sanitization; update dependency versions and documentation
+
+- Replaced console.log and console.warn with structured logger.log calls throughout the core modules
+- Enhanced database initialization with try/catch and proper URI credential encoding
+- Improved search query conversion by disallowing dangerous operators (e.g. $where) and securely escaping regex patterns
+- Bumped dependency versions (smartlog, @tsclass/tsclass, mongodb, etc.) in package.json
+- Added detailed project memories including code style, project overview, and suggested commands for developers
+- Updated README with improved instructions, feature highlights, and quick start sections
+
+## 2025-04-25 - 5.16.0 - feat(watcher)
+Enhance change stream watchers with buffering and EventEmitter support; update dependency versions
+
+- Bumped smartmongo from ^2.0.11 to ^2.0.12 and smartrx from ^3.0.7 to ^3.0.10
+- Upgraded @tsclass/tsclass to ^9.0.0 and mongodb to ^6.16.0
+- Refactored the watch API to accept additional options (bufferTimeMs, fullDocument) for improved change stream handling
+- Modified SmartdataDbWatcher to extend EventEmitter and support event notifications
+
+## 2025-04-24 - 5.15.1 - fix(cursor)
+Improve cursor usage documentation and refactor getCursor API to support native cursor modifiers
+
+- Updated examples in readme.md to demonstrate manual iteration using cursor.next() and proper cursor closing.
+- Refactored the getCursor method in classes.doc.ts to accept session and modifier options, consolidating cursor handling.
+- Added new tests in test/test.cursor.ts to verify cursor operations, including limits, sorting, and skipping.
+
+## 2025-04-24 - 5.15.0 - feat(svDb)
+Enhance svDb decorator to support custom serialization and deserialization options
+
+- Added an optional options parameter to the svDb decorator to accept serialize/deserialize functions
+- Updated instance creation logic (updateFromDb) to apply custom deserialization if provided
+- Updated createSavableObject to use custom serialization when available
+
+## 2025-04-23 - 5.14.1 - fix(db operations)
+Update transaction API to consistently pass optional session parameters across database operations
+
+- Revised transaction support in readme to use startSession without await and showcased session usage in getInstance and save calls
+- Updated methods in classes.collection.ts to accept an optional session parameter for findOne, getCursor, findAll, insert, update, delete, and getCount
+- Enhanced SmartDataDbDoc save and delete methods to propagate session parameters
+- Improved overall consistency of transactional APIs across the library
+
+## 2025-04-23 - 5.14.0 - feat(doc)
+Implement support for beforeSave, afterSave, beforeDelete, and afterDelete lifecycle hooks in document save and delete operations to allow custom logic execution during these critical moments.
+
+- Calls beforeSave hook if defined before performing insert or update.
+- Calls afterSave hook after a document is saved.
+- Calls beforeDelete hook before deletion and afterDelete hook afterward.
+- Ensures _updatedAt timestamp is refreshed during save operations.
+
+## 2025-04-22 - 5.13.1 - fix(search)
+Improve search query parsing for implicit AND queries by preserving quoted substrings and better handling free terms, quoted phrases, and field:value tokens.
+
+- Replace previous implicit AND logic with tokenization that preserves quoted substrings
+- Support both free term and field:value tokens with wildcards inside quotes
+- Ensure errors are thrown for non-searchable fields in field-specific queries
+
+## 2025-04-22 - 5.13.0 - feat(search)
+Improve search query handling and update documentation
+
+- Added 'codex.md' providing a high-level project overview and detailed search API documentation.
+- Enhanced search parsing in SmartDataDbDoc to support combined free-term and quoted field phrase queries.
+- Introduced a new fallback branch in the search method to handle free term with quoted field input.
+- Updated tests in test/test.search.ts to cover new combined query scenarios and ensure robust behavior.
+
+## 2025-04-22 - 5.12.2 - fix(search)
+Fix handling of quoted wildcard patterns in field-specific search queries and add tests for location-based wildcard phrase searches
+
+- Strip surrounding quotes from wildcard patterns in field queries to correctly transform them to regex
+- Introduce new tests in test/test.search.ts to validate exact quoted and unquoted wildcard searches on a location field
+
+## 2025-04-22 - 5.12.1 - fix(search)
+Improve implicit AND logic for mixed free term and field queries in search and enhance wildcard field handling.
+
+- Updated regex for field:value parsing to capture full value with wildcards.
+- Added explicit handling for free terms by converting to regex across searchable fields.
+- Improved error messaging for attempts to search non-searchable fields.
+- Extended tests to cover combined free term and wildcard field searches, including error cases.
+
+## 2025-04-22 - 5.12.0 - feat(doc/search)
+Enhance search functionality with filter and validate options for advanced query control
+
+- Added 'filter' option to merge additional MongoDB query constraints in search
+- Introduced 'validate' hook to post-process and filter fetched documents
+- Refactored underlying execQuery function to support additional search options
+- Updated tests to cover new search scenarios and fallback mechanisms
+
+## 2025-04-22 - 5.11.4 - fix(search)
+Implement implicit AND logic for mixed simple term and field:value queries in search
+
+- Added a new branch to detect and handle search queries that mix field:value pairs with plain terms without explicit operators
+- Builds an implicit $and filter when query parts contain colon(s) but lack explicit boolean operators or quotes
+- Ensures proper parsing and improved robustness of search filters
+
+## 2025-04-22 - 5.11.3 - fix(lucene adapter and search tests)
+Improve range query parsing in Lucene adapter and expand search test coverage
+
+- Added a new 'testSearch' script in package.json to run search tests.
+- Introduced advanced search tests for range queries and combined field filters in test/search.advanced.ts.
+- Enhanced robustness tests in test/search.ts for wildcard and empty query scenarios.
+- Fixed token validation in the parseRange method of the Lucene adapter to ensure proper error handling.
+
+## 2025-04-21 - 5.11.2 - fix(readme)
+Update readme to clarify usage of searchable fields retrieval
+
+- Replaced getSearchableFields('Product') with Product.getSearchableFields()
+- Updated documentation to reference the static method Class.getSearchableFields()
+
+## 2025-04-21 - 5.11.1 - fix(doc)
+Refactor searchable fields API and improve collection registration.
+
+- Removed the standalone getSearchableFields utility in favor of a static method on document classes.
+- Updated tests to use the new static method (e.g., Product.getSearchableFields()).
+- Ensured the Collection decorator attaches a docCtor property to correctly register searchable fields.
+- Added try/catch in test cleanup to gracefully handle dropDatabase errors.
+
+## 2025-04-21 - 5.11.0 - feat(ts/classes.lucene.adapter)
+Expose luceneWildcardToRegex method to allow external usage and enhance regex transformation capabilities.
+
+- Changed luceneWildcardToRegex from private to public in ts/classes.lucene.adapter.ts.
+
+## 2025-04-21 - 5.10.0 - feat(search)
+Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests
+
+- Updated readme.md with detailed Lucene‑style search examples and use cases
+- Enhanced LuceneToMongoTransformer to properly handle wildcard conversion and regex escaping
+- Improved search query parsing in SmartDataDbDoc for field-specific, multi-term, and advanced Lucene syntax
+- Added new advanced search tests covering boolean operators, grouping, quoted phrases, and wildcard queries
+
+## 2025-04-18 - 5.9.2 - fix(documentation)
+Update search API documentation to replace deprecated searchWithLucene examples with the unified search(query) API and clarify its behavior.
+
+- Replaced 'searchWithLucene' examples with 'search(query)' in the README.
+- Updated explanation to detail field-specific exact match, partial word regex search, multi-word literal matching, and handling of empty queries.
+- Clarified guidelines for creating MongoDB text indexes on searchable fields for optimized search performance.
+
+## 2025-04-18 - 5.9.1 - fix(search)
+Refactor search tests to use unified search API and update text index type casting
+
+- Replaced all calls from searchWithLucene with search in test/search tests
+- Updated text index specification in the collection class to use proper type casting
+
+## 2025-04-18 - 5.9.0 - feat(collections/search)
+Improve text index creation and search fallback mechanisms in collections and document search methods
+
+- Auto-create a compound text index on all searchable fields in SmartdataCollection with a one-time flag to prevent duplicate index creation.
+- Refine the search method in SmartDataDbDoc to support exact field matches and safe regex fallback for non-Lucene queries.
+
+## 2025-04-17 - 5.8.4 - fix(core)
+Update commit metadata with no functional code changes
+
+- Commit info and documentation refreshed
+- No code or test changes detected in the diff
+
+## 2025-04-17 - 5.8.3 - fix(readme)
+Improve readme documentation on data models and connection management
+
+- Clarify that data models use @Collection, @unI, @svDb, @index, and @searchable decorators
+- Document that ObjectId and Buffer fields are stored as BSON types natively without extra decorators
+- Update connection management section to use 'db.close()' instead of 'db.disconnect()'
+- Revise license section to reference the MIT License without including additional legal details
+
+## 2025-04-14 - 5.8.2 - fix(classes.doc.ts)
+Ensure collection initialization before creating a cursor in getCursorExtended
+
+- Added 'await collection.init()' to guarantee that the MongoDB collection is initialized before using the cursor
+- Prevents potential runtime errors when accessing collection.mongoDbCollection
+
+## 2025-04-14 - 5.8.1 - fix(cursor, doc)
+Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.
+
+- Specify Promise<T> as return type for next() in SmartdataDbCursor and cast return value to T.
+- Specify Promise<T[]> as return type for toArray() in SmartdataDbCursor and cast return value to T[].
+- Update getCursorExtended to return Promise<SmartdataDbCursor<T>> for clearer type safety.
+
+## 2025-04-14 - 5.8.0 - feat(cursor)
+Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances
+
+- Introduced asynchronous toArray method in SmartdataDbCursor which retrieves all documents from the MongoDB cursor
+- Maps each native document to a SmartDataDbDoc instance using createInstanceFromMongoDbNativeDoc for consistent API usage
+
+## 2025-04-14 - 5.7.0 - feat(SmartDataDbDoc)
+Add extended cursor method getCursorExtended for flexible cursor modifications
+
+- Introduces getCursorExtended in classes.doc.ts to allow modifier functions for MongoDB cursors
+- Wraps the modified cursor with SmartdataDbCursor for improved API consistency
+- Enhances querying capabilities by enabling customized cursor transformations
+
 ## 2025-04-07 - 5.6.0 - feat(indexing)
 Add support for regular index creation in documents and collections
 

package.json

@@ -1,13 +1,14 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.6.0",
+  "version": "6.0.0",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",
   "typings": "dist_ts/index.d.ts",
   "type": "module",
   "scripts": {
-    "test": "tstest test/",
+    "test": "tstest test/ --verbose --logfile --timeout 120",
+    "testSearch": "tsx test/test.search.ts",
     "build": "tsbuild --web --allowimplicitany",
     "buildDocs": "tsdoc"
   },
@@ -22,26 +23,26 @@
   },
   "homepage": "https://code.foss.global/push.rocks/smartdata#readme",
   "dependencies": {
-    "@push.rocks/lik": "^6.0.14",
+    "@push.rocks/lik": "^6.2.2",
     "@push.rocks/smartdelay": "^3.0.1",
-    "@push.rocks/smartlog": "^3.0.2",
-    "@push.rocks/smartmongo": "^2.0.11",
+    "@push.rocks/smartlog": "^3.1.10",
+    "@push.rocks/smartmongo": "^2.0.14",
     "@push.rocks/smartpromise": "^4.0.2",
-    "@push.rocks/smartrx": "^3.0.7",
-    "@push.rocks/smartstring": "^4.0.15",
+    "@push.rocks/smartrx": "^3.0.10",
+    "@push.rocks/smartstring": "^4.1.0",
     "@push.rocks/smarttime": "^4.0.6",
     "@push.rocks/smartunique": "^3.0.8",
-    "@push.rocks/taskbuffer": "^3.1.7",
-    "@tsclass/tsclass": "^8.2.0",
-    "mongodb": "^6.15.0"
+    "@push.rocks/taskbuffer": "^3.4.0",
+    "@tsclass/tsclass": "^9.3.0",
+    "mongodb": "^6.20.0"
   },
   "devDependencies": {
-    "@git.zone/tsbuild": "^2.3.2",
-    "@git.zone/tsrun": "^1.2.44",
-    "@git.zone/tstest": "^1.0.77",
-    "@push.rocks/qenv": "^6.0.5",
-    "@push.rocks/tapbundle": "^5.6.2",
-    "@types/node": "^22.14.0"
+    "@git.zone/tsbuild": "^3.1.0",
+    "@git.zone/tsrun": "^2.0.0",
+    "@git.zone/tstest": "^2.8.1",
+    "@push.rocks/qenv": "^6.1.3",
+    "@push.rocks/tapbundle": "^6.0.3",
+    "@types/node": "^22.15.2"
   },
   "files": [
     "ts/**/*",

readme.hints.md

@@ -0,0 +1,80 @@
+# Project Memory - Smartdata
+
+## TC39 Decorator Migration (v6.0.0) - ✅ COMPLETED
+
+### Final Status: All Tests Passing (157/157)
+Migration successfully completed on 2025-11-17.
+
+### What Changed:
+- ✅ Removed `experimentalDecorators` from tsconfig.json
+- ✅ Refactored all 7 decorators to TC39 Stage 3 syntax
+  - 5 property decorators: @globalSvDb, @svDb, @unI, @index, @searchable
+  - 2 class decorators: @Collection, @managed
+- ✅ Implemented context.metadata pattern for shared decorator state
+- ✅ All tests passing across Node.js and Deno runtimes
+
+### Critical Discovery: TC39 Metadata Access Pattern
+**THE KEY INSIGHT**: In TC39 decorators, metadata is NOT accessed via `constructor[Symbol.metadata]`. Instead:
+- **Field decorators**: Write to `context.metadata`
+- **Class decorators**: Read from `context.metadata` (same shared object!)
+- The `context.metadata` object is shared between all decorators on the same class
+- Attempting to write to `constructor[Symbol.metadata]` throws: "Cannot assign to read only property"
+
+### Implementation Pattern:
+```typescript
+// Field decorator - stores metadata
+export function svDb() {
+  return (value: undefined, context: ClassFieldDecoratorContext) => {
+    const metadata = context.metadata as ISmartdataDecoratorMetadata;
+    if (!metadata.saveableProperties) {
+      metadata.saveableProperties = [];
+    }
+    metadata.saveableProperties.push(String(context.name));
+  };
+}
+
+// Class decorator - reads metadata and initializes prototype
+export function Collection(dbArg: SmartdataDb) {
+  return function(value: Function, context: ClassDecoratorContext) => {
+    const metadata = context.metadata as ISmartdataDecoratorMetadata;
+    if (metadata?.saveableProperties) {
+      decoratedClass.prototype.saveableProperties = [...metadata.saveableProperties];
+    }
+    return decoratedClass;
+  };
+}
+```
+
+### Runtime Compatibility:
+- ✅ **Node.js v23.8.0**: Full TC39 support
+- ✅ **Deno v2.5.4**: Full TC39 support
+- ❌ **Bun v1.3.0**: No TC39 support (uses legacy decorators only)
+  - Removed "+bun" from test filenames to skip Bun tests
+
+### Key Technical Notes:
+1. **Metadata Initialization Timing**: Class decorators run AFTER field decorators, allowing them to read accumulated metadata and initialize prototypes before any instances are created
+2. **Prototype vs Instance Properties**: Properties set on prototype are accessible via `this.propertyName` in instances
+3. **TypeScript Lib Support**: TypeScript 5.9.3 includes built-in decorator types (no custom lib configuration needed)
+4. **Interface Naming**: Used `ISmartdataDecoratorMetadata` extending `DecoratorMetadataObject` for type safety
+
+### Files Modified:
+- ts/classes.doc.ts (property decorators + metadata interface)
+- ts/classes.collection.ts (class decorators + prototype initialization)
+- tsconfig.json (removed experimentalDecorators flag)
+- test/*.ts (renamed files to remove "+bun" suffix)
+
+### Test Results:
+All 157 tests passing across 10 test files:
+- test.cursor.ts: 7/7
+- test.deno.ts: 11/11 (queries working correctly!)
+- test.search.advanced.ts: 41/41
+- test.typescript.ts: 4/4
+- test.watch.ts: 5/5
+- And 5 more test files
+
+### Migration Learnings for Future Reference:
+1. `context.metadata` is the ONLY way to share state between decorators
+2. Class decorators must initialize prototypes from metadata immediately
+3. `Symbol.metadata` on constructors is read-only (managed by runtime)
+4. Field decorators run before class decorators (guaranteed order)
+5. TypeScript 5.2+ has built-in TC39 decorator support

test/test.cursor.ts

@@ -0,0 +1,97 @@
+import { tap, expect } from '@push.rocks/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import { smartunique } from '../ts/plugins.js';
+import * as smartdata from '../ts/index.js';
+
+// Set up database connection
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define a simple document model for cursor tests
+@smartdata.Collection(() => testDb)
+class CursorTest extends smartdata.SmartDataDbDoc<CursorTest, CursorTest> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  public name: string;
+
+  @smartdata.svDb()
+  public order: number;
+
+  constructor(name: string, order: number) {
+    super();
+    this.name = name;
+    this.order = order;
+  }
+}
+
+// Initialize the in-memory MongoDB and SmartdataDB
+tap.test('cursor init: start Mongo and SmartdataDb', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(
+    await smartmongoInstance.getMongoDescriptor(),
+  );
+  await testDb.init();
+});
+
+// Insert sample documents
+tap.test('cursor insert: save 5 test documents', async () => {
+  for (let i = 1; i <= 5; i++) {
+    const doc = new CursorTest(`item${i}`, i);
+    await doc.save();
+  }
+  const count = await CursorTest.getCount({});
+  expect(count).toEqual(5);
+});
+
+// Test that toArray returns all documents
+tap.test('cursor toArray: retrieves all documents', async () => {
+  const cursor = await CursorTest.getCursor({});
+  const all = await cursor.toArray();
+  expect(all.length).toEqual(5);
+});
+
+// Test iteration via forEach
+tap.test('cursor forEach: iterates through all documents', async () => {
+  const names: string[] = [];
+  const cursor = await CursorTest.getCursor({});
+  await cursor.forEach(async (item) => {
+    names.push(item.name);
+  });
+  expect(names.length).toEqual(5);
+  expect(names).toContain('item3');
+});
+
+// Test native cursor modifiers: limit
+tap.test('cursor modifier limit: only two documents', async () => {
+  const cursor = await CursorTest.getCursor({}, { modifier: (c) => c.limit(2) });
+  const limited = await cursor.toArray();
+  expect(limited.length).toEqual(2);
+});
+
+// Test native cursor modifiers: sort and skip
+tap.test('cursor modifier sort & skip: returns correct order', async () => {
+  const cursor = await CursorTest.getCursor({}, {
+    modifier: (c) => c.sort({ order: -1 }).skip(1),
+  });
+  const results = await cursor.toArray();
+  // Skipped the first (order 5), next should be 4,3,2,1
+  expect(results.length).toEqual(4);
+  expect(results[0].order).toEqual(4);
+});
+
+// Cleanup: drop database, close connections, stop Mongo
+tap.test('cursor cleanup: drop DB and stop', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir(
+      `.nogit/dbdump/test.cursor.ts`,
+    );
+  }
+  // Ensure process exits after cleanup
+  setTimeout(() => process.exit(), 2000);
+});
+
+export default tap.start();

test/test.deno.ts

@@ -0,0 +1,255 @@
+// TODO: Decorator support during testing for bun and deno in @git.zone/tstest
+
+import { tap, expect } from '@push.rocks/tapbundle';
+import { Qenv } from '@push.rocks/qenv';
+import * as smartmongo from '@push.rocks/smartmongo';
+import { smartunique } from '../ts/plugins.js';
+
+import * as mongodb from 'mongodb';
+
+const testQenv = new Qenv(process.cwd(), process.cwd() + '/.nogit/');
+
+console.log(process.memoryUsage());
+
+// the tested module
+import * as smartdata from '../ts/index.js';
+
+// =======================================
+// Connecting to the database server
+// =======================================
+
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+const totalCars = 2000;
+
+tap.test('should create a testinstance as database', async () => {
+  const databaseName = `test-smartdata-deno-${smartunique.shortId()}`;
+  testDb = new smartdata.SmartdataDb({
+    mongoDbUrl: await testQenv.getEnvVarOnDemand('MONGODB_URL'),
+    mongoDbName: databaseName,
+  });
+  await testDb.init();
+});
+
+// =======================================
+// The actual tests
+// =======================================
+
+// ------
+// Collections
+// ------
+
+@smartdata.Collection(() => {
+  return testDb;
+})
+class Car extends smartdata.SmartDataDbDoc<Car, Car> {
+  @smartdata.unI()
+  public index: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  public color: string;
+
+  @smartdata.svDb()
+  public brand: string;
+
+  @smartdata.svDb()
+  public testBuffer = Buffer.from('hello');
+
+  @smartdata.svDb()
+  deepData = {
+    sodeep: 'yes',
+  };
+
+  constructor(colorArg: string, brandArg: string) {
+    super();
+    this.color = colorArg;
+    this.brand = brandArg;
+  }
+}
+
+tap.test('should create a new id', async () => {
+  const newid = await Car.getNewId();
+  console.log(newid);
+});
+
+tap.test('should save the car to the db', async (toolsArg) => {
+  const myCar = new Car('red', 'Volvo');
+  console.log('Car.collection.smartdataDb:', (Car.collection as any).smartdataDb?.mongoDb?.databaseName);
+  console.log('Car.collection.collectionName:', (Car.collection as any).collectionName);
+  console.log('testDb.mongoDb.databaseName:', testDb.mongoDb.databaseName);
+  await myCar.save();
+
+  const myCar2 = new Car('red', 'Volvo');
+  await myCar2.save();
+
+  let counter = 0;
+
+  const gottenCarInstance = await Car.getInstance({});
+  console.log(gottenCarInstance.testBuffer instanceof mongodb.Binary);
+  process.memoryUsage();
+  do {
+    const myCar3 = new Car('red', 'Renault');
+    await myCar3.save();
+    counter++;
+    if (counter % 100 === 0) {
+      console.log(
+        `Filled database with ${counter} of ${totalCars} Cars and memory usage ${
+          process.memoryUsage().rss / 1e6
+        } MB`,
+      );
+    }
+  } while (counter < totalCars);
+  console.log(process.memoryUsage());
+
+  // DEBUG: Check what's actually in the database
+  const savedCount = await Car.getCount({});
+  console.log('Total cars saved in DB:', savedCount);
+  const renaultCount = await Car.getCount({ brand: 'Renault' });
+  console.log('Renault cars in DB:', renaultCount);
+
+  // Check what's actually in the first saved car
+  const firstCar = await Car.getInstance({});
+  console.log('First car data:', JSON.stringify({
+    color: firstCar?.color,
+    brand: firstCar?.brand,
+    index: firstCar?.index
+  }));
+});
+
+tap.test('expect to get instance of Car with shallow match', async () => {
+  console.log('Before query - testDb.mongoDb.databaseName:', testDb.mongoDb.databaseName);
+  console.log('Before query - Car.collection.smartdataDb:', (Car.collection as any).smartdataDb?.mongoDb?.databaseName);
+  console.log('Before query - Car.collection.collectionName:', (Car.collection as any).collectionName);
+
+  const totalQueryCycles = totalCars / 2;
+  let counter = 0;
+  do {
+    const timeStart = Date.now();
+    const myCars = await Car.getInstances({
+      brand: 'Renault',
+    });
+    if (counter % 10 === 0) {
+      console.log(
+        `performed ${counter} of ${totalQueryCycles} total query cycles: took ${
+          Date.now() - timeStart
+        }ms to query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`,
+      );
+      console.log('myCars.length:', myCars.length);
+      console.log('myCars[0]:', myCars[0]);
+    }
+    expect(myCars[0].deepData.sodeep).toEqual('yes');
+    expect(myCars[0].brand).toEqual('Renault');
+    counter++;
+  } while (counter < totalQueryCycles);
+});
+
+tap.test('expect to get instance of Car with deep match', async () => {
+  const totalQueryCycles = totalCars / 6;
+  let counter = 0;
+  do {
+    const timeStart = Date.now();
+    const myCars2 = await Car.getInstances({
+      deepData: {
+        sodeep: 'yes',
+      },
+    });
+    if (counter % 10 === 0) {
+      console.log(
+        `performed ${counter} of ${totalQueryCycles} total query cycles: took ${
+          Date.now() - timeStart
+        }ms to deep query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`,
+      );
+    }
+    expect(myCars2[0].deepData.sodeep).toEqual('yes');
+    expect(myCars2[0].brand).toEqual('Volvo');
+    counter++;
+  } while (counter < totalQueryCycles);
+});
+
+tap.test('expect to get instance of Car and update it', async () => {
+  const myCar = await Car.getInstance<Car>({
+    brand: 'Volvo',
+  });
+  expect(myCar.color).toEqual('red');
+  myCar.color = 'blue';
+  await myCar.save();
+});
+
+tap.test('should be able to delete an instance of car', async () => {
+  const myCars = await Car.getInstances({
+    brand: 'Volvo',
+    color: 'blue',
+  });
+  console.log(myCars);
+  expect(myCars[0].color).toEqual('blue');
+  for (const myCar of myCars) {
+    await myCar.delete();
+  }
+
+  const myCar2 = await Car.getInstance<Car>({
+    brand: 'Volvo',
+  });
+  expect(myCar2.color).toEqual('red');
+});
+
+// tslint:disable-next-line: max-classes-per-file
+@smartdata.Collection(() => {
+  return testDb;
+})
+class Truck extends smartdata.SmartDataDbDoc<Car, Car> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  public color: string;
+
+  @smartdata.svDb()
+  public brand: string;
+
+  constructor(colorArg: string, brandArg: string) {
+    super();
+    this.color = colorArg;
+    this.brand = brandArg;
+  }
+}
+
+tap.test('should store a new Truck', async () => {
+  const truck = new Truck('blue', 'MAN');
+  await truck.save();
+  const myTruck2 = await Truck.getInstance({ color: 'blue' });
+  expect(myTruck2.color).toEqual('blue');
+  myTruck2.color = 'red';
+  await myTruck2.save();
+  const myTruck3 = await Truck.getInstance({ color: 'blue' });
+  expect(myTruck3).toBeNull();
+});
+
+tap.test('should return a count', async () => {
+  const truckCount = await Truck.getCount();
+  expect(truckCount).toEqual(1);
+});
+
+tap.test('should use a cursor', async () => {
+  const cursor = await Car.getCursor({});
+  let counter = 0;
+  await cursor.forEach(async (carArg) => {
+    counter++;
+    counter % 50 === 0 ? console.log(`50 more of ${carArg.color}`) : null;
+  });
+});
+
+// =======================================
+// close the database connection
+// =======================================
+tap.test('close', async () => {
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir('./.nogit/dbdump/test.ts');
+  } else {
+    await testDb.mongoDb.dropDatabase();
+    await testDb.close();
+  }
+  setTimeout(() => process.exit(), 2000);
+});
+
+tap.start({ throwOnError: true });

test/test.filters.ts

@@ -0,0 +1,819 @@
+import { tap, expect } from '@git.zone/tstest/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import * as smartunique from '@push.rocks/smartunique';
+import * as smartdata from '../ts/index.js';
+
+const { SmartdataDb, Collection, svDb, unI, index } = smartdata;
+
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define test document classes
+@Collection(() => testDb)
+class TestUser extends smartdata.SmartDataDbDoc<TestUser, TestUser> {
+  @unI()
+  public id: string = smartunique.shortId();
+
+  @svDb()
+  public name: string;
+
+  @svDb()
+  public age: number;
+
+  @svDb()
+  public email: string;
+
+  @svDb()
+  public roles: string[];
+
+  @svDb()
+  public tags: string[];
+
+  @svDb()
+  public status: 'active' | 'inactive' | 'pending';
+
+  @svDb()
+  public metadata: {
+    lastLogin?: Date;
+    loginCount?: number;
+    preferences?: Record<string, any>;
+  };
+
+  @svDb()
+  public scores: number[];
+
+  constructor(data: Partial<TestUser> = {}) {
+    super();
+    Object.assign(this, data);
+  }
+}
+
+@Collection(() => testDb)
+class TestOrder extends smartdata.SmartDataDbDoc<TestOrder, TestOrder> {
+  @unI()
+  public id: string = smartunique.shortId();
+
+  @svDb()
+  public userId: string;
+
+  @svDb()
+  public items: Array<{
+    product: string;
+    quantity: number;
+    price: number;
+  }>;
+
+  @svDb()
+  public totalAmount: number;
+
+  @svDb()
+  public status: string;
+
+  @svDb()
+  public tags: string[];
+
+  constructor(data: Partial<TestOrder> = {}) {
+    super();
+    Object.assign(this, data);
+  }
+}
+
+// Setup and teardown
+tap.test('should create a test database instance', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(await smartmongoInstance.getMongoDescriptor());
+  await testDb.init();
+  expect(testDb).toBeInstanceOf(SmartdataDb);
+});
+
+tap.test('should create test data', async () => {
+  // Create test users
+  const users = [
+    new TestUser({
+      name: 'John Doe',
+      age: 30,
+      email: 'john@example.com',
+      roles: ['admin', 'user'],
+      tags: ['javascript', 'nodejs', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 5, lastLogin: new Date() },
+      scores: [85, 90, 78]
+    }),
+    new TestUser({
+      name: 'Jane Smith',
+      age: 25,
+      email: 'jane@example.com',
+      roles: ['user'],
+      tags: ['python', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 3 },
+      scores: [92, 88, 95]
+    }),
+    new TestUser({
+      name: 'Bob Johnson',
+      age: 35,
+      email: 'bob@example.com',
+      roles: ['moderator', 'user'],
+      tags: ['javascript', 'react', 'nodejs'],
+      status: 'inactive',
+      metadata: { loginCount: 0 },
+      scores: [70, 75, 80]
+    }),
+    new TestUser({
+      name: 'Alice Brown',
+      age: 28,
+      email: 'alice@example.com',
+      roles: ['admin'],
+      tags: ['typescript', 'angular', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 10 },
+      scores: [95, 98, 100]
+    }),
+    new TestUser({
+      name: 'Charlie Wilson',
+      age: 22,
+      email: 'charlie@example.com',
+      roles: ['user'],
+      tags: ['golang', 'kubernetes'],
+      status: 'pending',
+      metadata: { loginCount: 1 },
+      scores: [60, 65]
+    })
+  ];
+
+  for (const user of users) {
+    await user.save();
+  }
+
+  // Create test orders
+  const orders = [
+    new TestOrder({
+      userId: users[0].id,
+      items: [
+        { product: 'laptop', quantity: 1, price: 1200 },
+        { product: 'mouse', quantity: 2, price: 25 }
+      ],
+      totalAmount: 1250,
+      status: 'completed',
+      tags: ['electronics', 'priority']
+    }),
+    new TestOrder({
+      userId: users[1].id,
+      items: [
+        { product: 'book', quantity: 3, price: 15 },
+        { product: 'pen', quantity: 5, price: 2 }
+      ],
+      totalAmount: 55,
+      status: 'pending',
+      tags: ['stationery']
+    }),
+    new TestOrder({
+      userId: users[0].id,
+      items: [
+        { product: 'laptop', quantity: 2, price: 1200 },
+        { product: 'keyboard', quantity: 2, price: 80 }
+      ],
+      totalAmount: 2560,
+      status: 'processing',
+      tags: ['electronics', 'bulk']
+    })
+  ];
+
+  for (const order of orders) {
+    await order.save();
+  }
+
+  const savedUsers = await TestUser.getInstances({});
+  const savedOrders = await TestOrder.getInstances({});
+  expect(savedUsers.length).toEqual(5);
+  expect(savedOrders.length).toEqual(3);
+});
+
+// ============= BASIC FILTER TESTS =============
+tap.test('should filter by simple equality', async () => {
+  const users = await TestUser.getInstances({ name: 'John Doe' });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter by multiple fields (implicit AND)', async () => {
+  const users = await TestUser.getInstances({ 
+    status: 'active',
+    age: 30
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter by nested object fields', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.loginCount': 5
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+// ============= COMPREHENSIVE NESTED FILTER TESTS =============
+tap.test('should filter by nested object with direct object syntax', async () => {
+  // Direct nested object matching (exact match)
+  const users = await TestUser.getInstances({
+    metadata: { 
+      loginCount: 5,
+      lastLogin: (await TestUser.getInstances({}))[0].metadata.lastLogin // Get the exact date
+    }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter by partial nested object match', async () => {
+  // When using object syntax, only specified fields must match
+  const users = await TestUser.getInstances({
+    metadata: { loginCount: 5 }  // Only checks loginCount, ignores other fields
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should combine nested object and dot notation', async () => {
+  const users = await TestUser.getInstances({
+    metadata: { loginCount: { $gte: 3 } },  // Object syntax with operator
+    'metadata.loginCount': { $lte: 10 }     // Dot notation with operator
+  });
+  expect(users.length).toEqual(3);  // Jane (3), John (5), and Alice (10) have loginCount between 3-10
+});
+
+tap.test('should filter nested fields with operators using dot notation', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.loginCount': { $gte: 5 }
+  });
+  expect(users.length).toEqual(2);  // John (5) and Alice (10)
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should filter nested fields with multiple operators', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.loginCount': { $gte: 3, $lt: 10 }
+  });
+  expect(users.length).toEqual(2);  // Jane (3) and John (5)
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Jane Smith', 'John Doe']);
+});
+
+tap.test('should handle deeply nested object structures', async () => {
+  // First, create a user with deep nesting in preferences
+  const deepUser = new TestUser({
+    name: 'Deep Nester',
+    age: 40,
+    email: 'deep@example.com',
+    roles: ['admin'],
+    tags: [],
+    status: 'active',
+    metadata: {
+      loginCount: 1,
+      preferences: {
+        theme: {
+          colors: {
+            primary: '#000000',
+            secondary: '#ffffff'
+          },
+          fonts: {
+            heading: 'Arial',
+            body: 'Helvetica'
+          }
+        },
+        notifications: {
+          email: true,
+          push: false
+        }
+      }
+    },
+    scores: []
+  });
+  await deepUser.save();
+
+  // Test deep nesting with dot notation
+  const deepResults = await TestUser.getInstances({
+    'metadata.preferences.theme.colors.primary': '#000000'
+  });
+  expect(deepResults.length).toEqual(1);
+  expect(deepResults[0].name).toEqual('Deep Nester');
+
+  // Test deep nesting with operators
+  const boolResults = await TestUser.getInstances({
+    'metadata.preferences.notifications.email': { $eq: true }
+  });
+  expect(boolResults.length).toEqual(1);
+  expect(boolResults[0].name).toEqual('Deep Nester');
+
+  // Clean up
+  await deepUser.delete();
+});
+
+tap.test('should filter arrays of nested objects using $elemMatch', async () => {
+  const orders = await TestOrder.getInstances({
+    items: {
+      $elemMatch: {
+        product: 'laptop',
+        price: { $gte: 1000 }
+      }
+    }
+  });
+  expect(orders.length).toEqual(2);  // Both laptop orders have price >= 1000
+});
+
+tap.test('should filter nested arrays with dot notation', async () => {
+  // Query for any order that has an item with specific product
+  const orders = await TestOrder.getInstances({
+    'items.product': 'laptop'
+  });
+  expect(orders.length).toEqual(2);  // Two orders contain laptops
+});
+
+tap.test('should combine nested object filters with logical operators', async () => {
+  const users = await TestUser.getInstances({
+    $or: [
+      { 'metadata.loginCount': { $gte: 10 } },  // Alice has 10
+      { 
+        $and: [
+          { 'metadata.loginCount': { $lt: 5 } },  // Jane has 3, Bob has 0, Charlie has 1
+          { status: 'active' }  // Jane is active, Bob is inactive, Charlie is pending
+        ]
+      }
+    ]
+  });
+  expect(users.length).toEqual(2);  // Alice (loginCount >= 10), Jane (loginCount < 5 AND active)
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'Jane Smith']);
+});
+
+tap.test('should handle null and undefined in nested fields', async () => {
+  // Users without lastLogin
+  const noLastLogin = await TestUser.getInstances({
+    'metadata.lastLogin': { $exists: false }
+  });
+  expect(noLastLogin.length).toEqual(4);  // Everyone except John
+
+  // Users with preferences (none have it set)
+  const withPreferences = await TestUser.getInstances({
+    'metadata.preferences': { $exists: true }
+  });
+  expect(withPreferences.length).toEqual(0);
+});
+
+tap.test('should filter nested arrays by size', async () => {
+  // Create an order with specific number of items
+  const multiItemOrder = new TestOrder({
+    userId: 'test-user',
+    items: [
+      { product: 'item1', quantity: 1, price: 10 },
+      { product: 'item2', quantity: 2, price: 20 },
+      { product: 'item3', quantity: 3, price: 30 },
+      { product: 'item4', quantity: 4, price: 40 }
+    ],
+    totalAmount: 100,
+    status: 'pending',
+    tags: ['test']
+  });
+  await multiItemOrder.save();
+
+  const fourItemOrders = await TestOrder.getInstances({
+    items: { $size: 4 }
+  });
+  expect(fourItemOrders.length).toEqual(1);
+
+  // Clean up
+  await multiItemOrder.delete();
+});
+
+tap.test('should handle nested field comparison between documents', async () => {
+  // Find users where loginCount equals their age divided by 6 (John: 30/6=5)
+  const users = await TestUser.getInstances({
+    $and: [
+      { 'metadata.loginCount': 5 },
+      { age: 30 }
+    ]
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter using $in on nested fields', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.loginCount': { $in: [0, 1, 5] }
+  });
+  expect(users.length).toEqual(3);  // Bob (0), Charlie (1), John (5)
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'Charlie Wilson', 'John Doe']);
+});
+
+tap.test('should filter nested arrays with $all', async () => {
+  // Create an order with multiple tags
+  const taggedOrder = new TestOrder({
+    userId: 'test-user',
+    items: [{ product: 'test', quantity: 1, price: 10 }],
+    totalAmount: 10,
+    status: 'completed',
+    tags: ['urgent', 'priority', 'electronics']
+  });
+  await taggedOrder.save();
+
+  const priorityElectronics = await TestOrder.getInstances({
+    tags: { $all: ['priority', 'electronics'] }
+  });
+  expect(priorityElectronics.length).toEqual(2);  // Original order and new one
+
+  // Clean up
+  await taggedOrder.delete();
+});
+
+// ============= COMPARISON OPERATOR TESTS =============
+tap.test('should filter using $gt operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gt: 30 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Bob Johnson');
+});
+
+tap.test('should filter using $gte operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gte: 30 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'John Doe']);
+});
+
+tap.test('should filter using $lt operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $lt: 25 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Charlie Wilson');
+});
+
+tap.test('should filter using $lte operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $lte: 25 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Charlie Wilson', 'Jane Smith']);
+});
+
+tap.test('should filter using $ne operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $ne: 'active' }
+  });
+  expect(users.length).toEqual(2);
+  const statuses = users.map(u => u.status).sort();
+  expect(statuses).toEqual(['inactive', 'pending']);
+});
+
+tap.test('should filter using multiple comparison operators', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gte: 25, $lt: 30 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'Jane Smith']);
+});
+
+// ============= ARRAY OPERATOR TESTS =============
+tap.test('should filter using $in operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $in: ['active', 'pending'] }
+  });
+  expect(users.length).toEqual(4);
+  expect(users.every(u => ['active', 'pending'].includes(u.status))).toEqual(true);
+});
+
+tap.test('should filter arrays using $in operator', async () => {
+  const users = await TestUser.getInstances({
+    roles: { $in: ['admin'] }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should filter using $nin operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $nin: ['inactive', 'pending'] }
+  });
+  expect(users.length).toEqual(3);
+  expect(users.every(u => u.status === 'active')).toEqual(true);
+});
+
+tap.test('should filter arrays using $all operator', async () => {
+  const users = await TestUser.getInstances({
+    tags: { $all: ['javascript', 'nodejs'] }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'John Doe']);
+});
+
+tap.test('should filter arrays using $size operator', async () => {
+  const users = await TestUser.getInstances({
+    scores: { $size: 2 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Charlie Wilson');
+});
+
+tap.test('should filter arrays using $elemMatch operator', async () => {
+  const orders = await TestOrder.getInstances({
+    items: {
+      $elemMatch: {
+        product: 'laptop',
+        quantity: { $gte: 2 }
+      }
+    }
+  });
+  expect(orders.length).toEqual(1);
+  expect(orders[0].totalAmount).toEqual(2560);
+});
+
+tap.test('should filter using $elemMatch with single condition', async () => {
+  const orders = await TestOrder.getInstances({
+    items: {
+      $elemMatch: {
+        price: { $gt: 100 }
+      }
+    }
+  });
+  expect(orders.length).toEqual(2);
+  expect(orders.every(o => o.items.some(i => i.price > 100))).toEqual(true);
+});
+
+// ============= LOGICAL OPERATOR TESTS =============
+tap.test('should filter using $or operator', async () => {
+  const users = await TestUser.getInstances({
+    $or: [
+      { age: { $lt: 25 } },
+      { status: 'inactive' }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'Charlie Wilson']);
+});
+
+tap.test('should filter using $and operator', async () => {
+  const users = await TestUser.getInstances({
+    $and: [
+      { status: 'active' },
+      { age: { $gte: 28 } }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should filter using $nor operator', async () => {
+  const users = await TestUser.getInstances({
+    $nor: [
+      { status: 'inactive' },
+      { age: { $lt: 25 } }
+    ]
+  });
+  expect(users.length).toEqual(3);
+  expect(users.every(u => u.status !== 'inactive' && u.age >= 25)).toEqual(true);
+});
+
+tap.test('should filter using nested logical operators', async () => {
+  const users = await TestUser.getInstances({
+    $or: [
+      {
+        $and: [
+          { status: 'active' },
+          { roles: { $in: ['admin'] } }
+        ]
+      },
+      { age: { $lt: 23 } }
+    ]
+  });
+  expect(users.length).toEqual(3);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'Charlie Wilson', 'John Doe']);
+});
+
+// ============= ELEMENT OPERATOR TESTS =============
+tap.test('should filter using $exists operator', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.lastLogin': { $exists: true }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter using $exists false', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.preferences': { $exists: false }
+  });
+  expect(users.length).toEqual(5);
+});
+
+// ============= COMPLEX FILTER TESTS =============
+tap.test('should handle complex nested filters', async () => {
+  const users = await TestUser.getInstances({
+    $and: [
+      { status: 'active' },
+      {
+        $or: [
+          { age: { $gte: 30 } },
+          { roles: { $all: ['admin'] } }
+        ]
+      },
+      { tags: { $in: ['mongodb'] } }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should combine multiple operator types', async () => {
+  const orders = await TestOrder.getInstances({
+    $and: [
+      { totalAmount: { $gte: 100 } },
+      { status: { $in: ['completed', 'processing'] } },
+      { tags: { $in: ['electronics'] } }
+    ]
+  });
+  expect(orders.length).toEqual(2);
+  expect(orders.every(o => o.totalAmount >= 100)).toEqual(true);
+});
+
+// ============= ERROR HANDLING TESTS =============
+tap.test('should throw error for $where operator', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      $where: 'this.age > 25'
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$where.*not allowed/);
+});
+
+tap.test('should throw error for invalid $in value', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      status: { $in: 'active' as any } // Should be an array
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$in.*requires.*array/);
+});
+
+tap.test('should throw error for invalid $size value', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      scores: { $size: '3' as any } // Should be a number
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$size.*requires.*numeric/);
+});
+
+tap.test('should throw error for dots in field names', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      'some.nested.field': { 'invalid.key': 'value' }
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/keys cannot contain dots/);
+});
+
+// ============= EDGE CASE TESTS =============
+tap.test('should handle empty filter (return all)', async () => {
+  const users = await TestUser.getInstances({});
+  expect(users.length).toEqual(5);
+});
+
+tap.test('should handle null values in filter', async () => {
+  // First, create a user with null email
+  const nullUser = new TestUser({
+    name: 'Null User',
+    age: 40,
+    email: null as any,
+    roles: ['user'],
+    tags: [],
+    status: 'active',
+    metadata: {},
+    scores: []
+  });
+  await nullUser.save();
+
+  const users = await TestUser.getInstances({ email: null });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Null User');
+
+  // Clean up
+  await nullUser.delete();
+});
+
+tap.test('should handle arrays as direct equality match', async () => {
+  // This tests that arrays without operators are treated as equality matches
+  const users = await TestUser.getInstances({
+    roles: ['user'] // Exact match for array
+  });
+  expect(users.length).toEqual(2);  // Both Jane and Charlie have exactly ['user']
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Charlie Wilson', 'Jane Smith']);
+});
+
+tap.test('should handle regex operator', async () => {
+  const users = await TestUser.getInstances({
+    name: { $regex: '^J', $options: 'i' }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Jane Smith', 'John Doe']);
+});
+
+tap.test('should handle unknown operators by letting MongoDB reject them', async () => {
+  // Unknown operators should be passed through to MongoDB, which will reject them
+  let error: Error | null = null;
+  
+  try {
+    await TestUser.getInstances({
+      age: { $unknownOp: 30 } as any
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/unknown operator.*\$unknownOp/);
+});
+
+// ============= PERFORMANCE TESTS =============
+tap.test('should efficiently filter large result sets', async () => {
+  // Create many test documents
+  const manyUsers = [];
+  for (let i = 0; i < 100; i++) {
+    manyUsers.push(new TestUser({
+      name: `User ${i}`,
+      age: 20 + (i % 40),
+      email: `user${i}@example.com`,
+      roles: i % 3 === 0 ? ['admin'] : ['user'],
+      tags: i % 2 === 0 ? ['even', 'test'] : ['odd', 'test'],
+      status: i % 4 === 0 ? 'inactive' : 'active',
+      metadata: { loginCount: i },
+      scores: [i, i + 10, i + 20]
+    }));
+  }
+
+  // Save in batches for efficiency
+  for (const user of manyUsers) {
+    await user.save();
+  }
+
+  // Complex filter that should still be fast
+  const startTime = Date.now();
+  const filtered = await TestUser.getInstances({
+    $and: [
+      { age: { $gte: 30, $lt: 40 } },
+      { status: 'active' },
+      { tags: { $in: ['even'] } },
+      { 'metadata.loginCount': { $gte: 20 } }
+    ]
+  });
+  const duration = Date.now() - startTime;
+
+  console.log(`Complex filter on 100+ documents took ${duration}ms`);
+  expect(duration).toBeLessThan(1000); // Should complete in under 1 second
+  expect(filtered.length).toBeGreaterThan(0);
+
+  // Clean up
+  for (const user of manyUsers) {
+    await user.delete();
+  }
+});
+
+// ============= CLEANUP =============
+tap.test('should clean up test database', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  await smartmongoInstance.stop();
+});
+
+export default tap.start();

test/test.node.ts

@@ -1,3 +1,5 @@
+// TODO: Decorator support during testing for bun and deno in @git.zone/tstest
+
 import { tap, expect } from '@push.rocks/tapbundle';
 import { Qenv } from '@push.rocks/qenv';
 import * as smartmongo from '@push.rocks/smartmongo';

test/test.search.advanced.node.ts

@@ -0,0 +1,202 @@
+import { tap, expect } from '@push.rocks/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import * as smartdata from '../ts/index.js';
+import { searchable } from '../ts/classes.doc.js';
+import { smartunique } from '../ts/plugins.js';
+
+// Set up database connection
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define a test class for advanced search scenarios
+@smartdata.Collection(() => testDb)
+class Product extends smartdata.SmartDataDbDoc<Product, Product> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  @searchable()
+  public name: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public description: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public category: string;
+
+  @smartdata.svDb()
+  public price: number;
+
+  constructor(
+    nameArg: string,
+    descriptionArg: string,
+    categoryArg: string,
+    priceArg: number,
+  ) {
+    super();
+    this.name = nameArg;
+    this.description = descriptionArg;
+    this.category = categoryArg;
+    this.price = priceArg;
+  }
+}
+
+// Initialize DB and insert sample products
+tap.test('setup advanced search database', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(
+    await smartmongoInstance.getMongoDescriptor(),
+  );
+  await testDb.init();
+});
+
+tap.test('insert products for advanced search', async () => {
+  const products = [
+    new Product(
+      'Night Owl Lamp',
+      'Bright lamp for night reading',
+      'Lighting',
+      29,
+    ),
+    new Product(
+      'Day Light Lamp',
+      'Daytime lamp with adjustable brightness',
+      'Lighting',
+      39,
+    ),
+    new Product(
+      'Office Chair',
+      'Ergonomic chair for office',
+      'Furniture',
+      199,
+    ),
+    new Product(
+      'Gaming Chair',
+      'Comfortable for long gaming sessions',
+      'Furniture',
+      299,
+    ),
+    new Product(
+      'iPhone 12',
+      'Latest iPhone with A14 Bionic chip',
+      'Electronics',
+      999,
+    ),
+    new Product(
+      'AirPods',
+      'Wireless earbuds with noise cancellation',
+      'Electronics',
+      249,
+    ),
+  ];
+  for (const p of products) {
+    await p.save();
+  }
+  const all = await Product.getInstances({});
+  expect(all.length).toEqual(products.length);
+});
+
+// Simple exact field:value matching
+tap.test('simpleExact: category:Furniture returns chairs', async () => {
+  const res = await Product.search('category:Furniture');
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// simpleExact invalid field should throw
+tap.test('simpleExact invalid field errors', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:29');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+// Quoted phrase search
+tap.test('quoted phrase "Bright lamp" matches Night Owl Lamp', async () => {
+  const res = await Product.search('"Bright lamp"');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+tap.test("quoted phrase 'night reading' matches Night Owl Lamp", async () => {
+  const res = await Product.search("'night reading'");
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+
+tap.test('wildcard description:*gaming* matches Gaming Chair', async () => {
+  const res = await Product.search('description:*gaming*');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Gaming Chair');
+});
+
+// Boolean AND and OR
+tap.test('boolean AND: category:Lighting AND lamp', async () => {
+  const res = await Product.search('category:Lighting AND lamp');
+  expect(res.length).toEqual(2);
+});
+
+tap.test('boolean OR: Furniture OR Electronics', async () => {
+  const res = await Product.search('Furniture OR Electronics');
+  expect(res.length).toEqual(4);
+});
+
+// Multi-term unquoted -> AND across terms
+tap.test('multi-term unquoted adjustable brightness', async () => {
+  const res = await Product.search('adjustable brightness');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Day Light Lamp');
+});
+
+tap.test('multi-term unquoted Night Lamp', async () => {
+  const res = await Product.search('Night Lamp');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+// Grouping with parentheses
+tap.test('grouping: (Furniture OR Electronics) AND Chair', async () => {
+  const res = await Product.search(
+    '(Furniture OR Electronics) AND Chair',
+  );
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// Additional range and combined query tests
+tap.test('range query price:[30 TO 300] returns expected products', async () => {
+  const res = await Product.search('price:[30 TO 300]');
+  // Expect products with price between 30 and 300 inclusive: Day Light Lamp, Gaming Chair, Office Chair, AirPods
+  expect(res.length).toEqual(4);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['AirPods', 'Day Light Lamp', 'Gaming Chair', 'Office Chair']);
+});
+
+tap.test('should filter category and price range', async () => {
+  const res = await Product.search('category:Lighting AND price:[30 TO 40]');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Day Light Lamp');
+});
+
+// Teardown
+tap.test('cleanup advanced search database', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir(
+      `.nogit/dbdump/test.search.advanced.ts`,
+    );
+  }
+  setTimeout(() => process.exit(), 2000);
+});
+
+tap.start({ throwOnError: true });

test/test.search.ts

@@ -4,11 +4,13 @@ import { smartunique } from '../ts/plugins.js';
 
 // Import the smartdata library
 import * as smartdata from '../ts/index.js';
-import { searchable, getSearchableFields } from '../ts/classes.doc.js';
+import { searchable } from '../ts/classes.doc.js';
 
 // Set up database connection
 let smartmongoInstance: smartmongo.SmartMongo;
 let testDb: smartdata.SmartdataDb;
+// Class for location-based wildcard/phrase tests
+let LocationDoc: any;
 
 // Define a test class with searchable fields using the standard SmartDataDbDoc
 @smartdata.Collection(() => testDb)
@@ -72,7 +74,7 @@ tap.test('should create test products with searchable fields', async () => {
 
 tap.test('should retrieve searchable fields for a class', async () => {
   // Use the getSearchableFields function to verify our searchable fields
-  const searchableFields = getSearchableFields('Product');
+  const searchableFields = Product.getSearchableFields();
   console.log('Searchable fields:', searchableFields);
 
   expect(searchableFields.length).toEqual(3);
@@ -104,21 +106,21 @@ tap.test('should search products by basic search method', async () => {
   }
 });
 
-tap.test('should search products with searchWithLucene method', async () => {
+tap.test('should search products with search method', async () => {
   // Using the robust searchWithLucene method
-  const wirelessResults = await Product.searchWithLucene('wireless');
+    const wirelessResults = await Product.search('wireless');
     console.log(
-    `Found ${wirelessResults.length} products matching 'wireless' using searchWithLucene`,
+      `Found ${wirelessResults.length} products matching 'wireless' using search`,
     );
 
   expect(wirelessResults.length).toEqual(1);
   expect(wirelessResults[0].name).toEqual('AirPods');
 });
 
-tap.test('should search products by category with searchWithLucene', async () => {
+tap.test('should search products by category with search', async () => {
   // Using field-specific search with searchWithLucene
-  const kitchenResults = await Product.searchWithLucene('category:Kitchen');
-  console.log(`Found ${kitchenResults.length} products in Kitchen category using searchWithLucene`);
+  const kitchenResults = await Product.search('category:Kitchen');
+  console.log(`Found ${kitchenResults.length} products in Kitchen category using search`);
 
   expect(kitchenResults.length).toEqual(2);
   expect(kitchenResults[0].category).toEqual('Kitchen');
@@ -127,7 +129,7 @@ tap.test('should search products by category with searchWithLucene', async () =>
 
 tap.test('should search products with partial word matches', async () => {
   // Testing partial word matches
-  const proResults = await Product.searchWithLucene('Pro');
+  const proResults = await Product.search('Pro');
   console.log(`Found ${proResults.length} products matching 'Pro'`);
 
   // Should match both "MacBook Pro" and "professionals" in description
@@ -136,7 +138,7 @@ tap.test('should search products with partial word matches', async () => {
 
 tap.test('should search across multiple searchable fields', async () => {
   // Test searching across all searchable fields
-  const bookResults = await Product.searchWithLucene('book');
+  const bookResults = await Product.search('book');
   console.log(`Found ${bookResults.length} products matching 'book' across all fields`);
 
   // Should match "MacBook" in name and "Books" in category
@@ -145,8 +147,8 @@ tap.test('should search across multiple searchable fields', async () => {
 
 tap.test('should handle case insensitive searches', async () => {
   // Test case insensitivity
-  const electronicsResults = await Product.searchWithLucene('electronics');
-  const ElectronicsResults = await Product.searchWithLucene('Electronics');
+  const electronicsResults = await Product.search('electronics');
+  const ElectronicsResults = await Product.search('Electronics');
 
   console.log(`Found ${electronicsResults.length} products matching lowercase 'electronics'`);
   console.log(`Found ${ElectronicsResults.length} products matching capitalized 'Electronics'`);
@@ -166,14 +168,14 @@ tap.test('should demonstrate search fallback mechanisms', async () => {
 
   // Use a simpler term that should be found in descriptions
   // Avoid using "OR" operator which requires a text index
-  const results = await Product.searchWithLucene('high');
+  const results = await Product.search('high');
   console.log(`Found ${results.length} products matching 'high'`);
 
   // "High-speed blender" contains "high"
   expect(results.length).toBeGreaterThan(0);
 
   // Try another fallback example that won't need $text
-  const powerfulResults = await Product.searchWithLucene('powerful');
+  const powerfulResults = await Product.search('powerful');
   console.log(`Found ${powerfulResults.length} products matching 'powerful'`);
 
   // "Powerful laptop for professionals" contains "powerful"
@@ -192,6 +194,208 @@ tap.test('should explain the advantages of the integrated approach', async () =>
   expect(true).toEqual(true);
 });
 
+// Additional robustness tests
+tap.test('should search exact name using field:value', async () => {
+  const nameResults = await Product.search('name:AirPods');
+  expect(nameResults.length).toEqual(1);
+  expect(nameResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should throw when searching non-searchable field', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:129');
+  } catch (err) {
+    error = err as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+tap.test('empty query should return all products', async () => {
+  const allResults = await Product.search('');
+  expect(allResults.length).toEqual(8);
+});
+
+tap.test('should search multi-word term across fields', async () => {
+  const termResults = await Product.search('iPhone 12');
+  expect(termResults.length).toEqual(1);
+  expect(termResults[0].name).toEqual('iPhone 12');
+});
+
+// Additional search scenarios
+tap.test('should return zero results for non-existent terms', async () => {
+  const noResults = await Product.search('NonexistentTerm');
+  expect(noResults.length).toEqual(0);
+});
+
+tap.test('should search products by description term "noise"', async () => {
+  const noiseResults = await Product.search('noise');
+  expect(noiseResults.length).toEqual(1);
+  expect(noiseResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should search products by description term "flagship"', async () => {
+  const flagshipResults = await Product.search('flagship');
+  expect(flagshipResults.length).toEqual(1);
+  expect(flagshipResults[0].name).toEqual('Galaxy S21');
+});
+
+tap.test('should search numeric strings "12"', async () => {
+  const twelveResults = await Product.search('12');
+  expect(twelveResults.length).toEqual(1);
+  expect(twelveResults[0].name).toEqual('iPhone 12');
+});
+
+tap.test('should search hyphenated terms "high-speed"', async () => {
+  const hyphenResults = await Product.search('high-speed');
+  expect(hyphenResults.length).toEqual(1);
+  expect(hyphenResults[0].name).toEqual('Blender');
+});
+
+tap.test('should search hyphenated terms "E-reader"', async () => {
+  const ereaderResults = await Product.search('E-reader');
+  expect(ereaderResults.length).toEqual(1);
+  expect(ereaderResults[0].name).toEqual('Kindle Paperwhite');
+});
+
+// Additional robustness tests
+tap.test('should return all products for empty search', async () => {
+  const searchResults = await Product.search('');
+  const allProducts = await Product.getInstances({});
+  expect(searchResults.length).toEqual(allProducts.length);
+});
+
+tap.test('should support wildcard plain term across all fields', async () => {
+  const results = await Product.search('*book*');
+  const names = results.map((r) => r.name).sort();
+  expect(names).toEqual(['Harry Potter', 'Kindle Paperwhite', 'MacBook Pro']);
+});
+
+tap.test('should support wildcard plain term with question mark pattern', async () => {
+  const results = await Product.search('?one?');
+  const names = results.map((r) => r.name).sort();
+  expect(names).toEqual(['Galaxy S21', 'iPhone 12']);
+});
+
+// Filter and Validation tests
+tap.test('should apply filter option to restrict results', async () => {
+  // search term 'book' across all fields but restrict to Books category
+  const bookFiltered = await Product.search('book', { filter: { category: 'Books' } });
+  expect(bookFiltered.length).toEqual(2);
+  bookFiltered.forEach((p) => expect(p.category).toEqual('Books'));
+});
+tap.test('should apply validate hook to post-filter results', async () => {
+  // return only products with price > 500
+  const expensive = await Product.search('', { validate: (p) => p.price > 500 });
+  expect(expensive.length).toBeGreaterThan(0);
+  expensive.forEach((p) => expect(p.price).toBeGreaterThan(500));
+});
+
+// Tests for quoted and wildcard field-specific phrases
+tap.test('setup location test products', async () => {
+  @smartdata.Collection(() => testDb)
+  class LD extends smartdata.SmartDataDbDoc<LD, LD> {
+    @smartdata.unI() public id: string = smartunique.shortId();
+    @smartdata.svDb() @searchable() public location: string;
+    constructor(loc: string) { super(); this.location = loc; }
+  }
+  // Assign to outer variable for subsequent tests
+  LocationDoc = LD;
+  const locations = ['Berlin', 'Frankfurt am Main', 'Frankfurt am Oder', 'London'];
+  for (const loc of locations) {
+    await new LocationDoc(loc).save();
+  }
+});
+tap.test('should search exact quoted field phrase', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt am Main"');
+  expect(results.length).toEqual(1);
+  expect(results[0].location).toEqual('Frankfurt am Main');
+});
+tap.test('should search wildcard quoted field phrase', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt am *"');
+  const names = results.map((d: any) => d.location).sort();
+  expect(names).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+tap.test('should search unquoted wildcard field', async () => {
+  const results = await (LocationDoc as any).search('location:Frankfurt*');
+  const names = results.map((d: any) => d.location).sort();
+  expect(names).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+
+// Combined free-term + field phrase/wildcard tests
+let CombinedDoc: any;
+tap.test('setup combined docs for free-term and location tests', async () => {
+  @smartdata.Collection(() => testDb)
+  class CD extends smartdata.SmartDataDbDoc<CD, CD> {
+    @smartdata.unI() public id: string = smartunique.shortId();
+    @smartdata.svDb() @searchable() public name: string;
+    @smartdata.svDb() @searchable() public location: string;
+    constructor(name: string, location: string) { super(); this.name = name; this.location = location; }
+  }
+  CombinedDoc = CD;
+  const docs = [
+    new CombinedDoc('TypeScript', 'Berlin'),
+    new CombinedDoc('TypeScript', 'Frankfurt am Main'),
+    new CombinedDoc('TypeScript', 'Frankfurt am Oder'),
+    new CombinedDoc('JavaScript', 'Berlin'),
+  ];
+  for (const d of docs) await d.save();
+});
+tap.test('should search free term and exact quoted field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Berlin"');
+  expect(res.length).toEqual(1);
+  expect(res[0].location).toEqual('Berlin');
+});
+tap.test('should not match free term with non-matching quoted field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Frankfurt d"');
+  expect(res.length).toEqual(0);
+});
+tap.test('should search free term with quoted wildcard field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Frankfurt am *"');
+  const locs = res.map((r: any) => r.location).sort();
+  expect(locs).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+// Quoted exact field phrase without wildcard should return no matches if no exact match
+tap.test('should not match location:"Frankfurt d"', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt d"');
+  expect(results.length).toEqual(0);
+});
+
+// Combined free-term and field wildcard tests
+tap.test('should combine free term and wildcard field search', async () => {
+  const results = await Product.search('book category:Book*');
+  expect(results.length).toEqual(2);
+  results.forEach((p) => expect(p.category).toEqual('Books'));
+});
+tap.test('should not match when free term matches but wildcard field does not', async () => {
+  const results = await Product.search('book category:Kitchen*');
+  expect(results.length).toEqual(0);
+});
+
+// Non-searchable field should cause an error for combined queries
+tap.test('should throw when combining term with non-searchable field', async () => {
+  let error: Error;
+  try {
+    await Product.search('book location:Berlin');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+tap.test('should throw when combining term with non-searchable wildcard field', async () => {
+  let error: Error;
+  try {
+    await Product.search('book location:Berlin*');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+// Close database connection
 tap.test('close database connection', async () => {
   await testDb.mongoDb.dropDatabase();
   await testDb.close();

test/test.watch.ts

@@ -60,11 +60,52 @@ tap.test('should watch a collection', async (toolsArg) => {
   await done.promise;
 });
 
+// ======= New tests for EventEmitter and buffering support =======
+tap.test('should emit change via EventEmitter', async (tools) => {
+  const done = tools.defer();
+  const watcher = await House.watch({});
+  watcher.on('change', async (houseArg) => {
+    // Expect a House instance
+    expect(houseArg).toBeDefined();
+    // Clean up
+    await watcher.stop();
+    done.resolve();
+  });
+  // Trigger an insert to generate a change event
+  const h = new House();
+  await h.save();
+  await done.promise;
+});
+
+tap.test('should buffer change events when bufferTimeMs is set', async (tools) => {
+  const done = tools.defer();
+  // bufferTimeMs collects events into arrays every 50ms
+  const watcher = await House.watch({}, { bufferTimeMs: 50 });
+  let received: House[];
+  watcher.changeSubject.subscribe(async (batch: House[]) => {
+    if (batch && batch.length > 0) {
+      received = batch;
+      await watcher.stop();
+      done.resolve();
+    }
+  });
+  // Rapidly insert multiple docs
+  const docs = [new House(), new House(), new House()];
+  for (const doc of docs) await doc.save();
+  await done.promise;
+  // All inserts should be in one buffered batch
+  expect(received.length).toEqual(docs.length);
+});
+
 // =======================================
 // close the database connection
 // =======================================
 tap.test('close', async () => {
+  try {
     await testDb.mongoDb.dropDatabase();
+  } catch (err) {
+    console.warn('dropDatabase error ignored in cleanup:', err.message || err);
+  }
   await testDb.close();
   if (smartmongoInstance) {
     await smartmongoInstance.stop();

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.6.0',
+  version: '6.0.0',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/classes.collection.ts

@@ -4,6 +4,7 @@ import { SmartdataDbCursor } from './classes.cursor.js';
 import { SmartDataDbDoc, type IIndexOptions } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { CollectionFactory } from './classes.collectionfactory.js';
+import { logger } from './logging.js';
 
 export interface IFindOptions {
   limit?: number;
@@ -25,23 +26,94 @@ const collectionFactory = new CollectionFactory();
  * @param dbArg
  */
 export function Collection(dbArg: SmartdataDb | TDelayed<SmartdataDb>) {
-  return function classDecorator<T extends { new (...args: any[]): {} }>(constructor: T) {
+  return function classDecorator(value: Function, context: ClassDecoratorContext) {
+    if (context.kind !== 'class') {
+      throw new Error('Collection can only decorate classes');
+    }
+
+    // Capture original constructor for _svDbOptions forwarding
+    const originalConstructor = value as any;
+    const constructor = value as { new (...args: any[]): any };
+
+    const getCollection = () => {
+      if (!(dbArg instanceof SmartdataDb)) {
+        dbArg = dbArg();
+      }
+      const coll = collectionFactory.getCollection(constructor.name, dbArg);
+      // Attach document constructor for searchableFields lookup
+      if (!(coll as any).docCtor) {
+        (coll as any).docCtor = decoratedClass;
+      }
+      return coll;
+    };
+
     const decoratedClass = class extends constructor {
       public static className = constructor.name;
       public static get collection() {
-        if (!(dbArg instanceof SmartdataDb)) {
-          dbArg = dbArg();
-        }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        return getCollection();
       }
       public get collection() {
-        if (!(dbArg instanceof SmartdataDb)) {
-          dbArg = dbArg();
-        }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        return getCollection();
       }
     };
-    return decoratedClass;
+
+    // Ensure instance getter works in Deno by defining it on the prototype
+    Object.defineProperty(decoratedClass.prototype, 'collection', {
+      get: getCollection,
+      enumerable: false,
+      configurable: true
+    });
+
+    // Deno compatibility note: Property decorators set properties on the prototype.
+    // Since we removed instance property declarations from SmartDataDbDoc,
+    // the decorator-set prototype properties are now accessible without shadowing.
+    // No manual forwarding needed - natural prototype inheritance works!
+
+    // Point to original constructor's _svDbOptions
+    Object.defineProperty(decoratedClass, '_svDbOptions', {
+      get() { return originalConstructor._svDbOptions; },
+      set(value) { originalConstructor._svDbOptions = value; },
+      configurable: true
+    });
+
+    // Initialize prototype properties from context.metadata (TC39 decorator metadata)
+    // This ensures prototype properties are available before any instance is created
+    const metadata = context.metadata as any;
+    if (metadata) {
+      const proto = decoratedClass.prototype;
+
+      // Initialize globalSaveableProperties
+      if (metadata.globalSaveableProperties && !proto.globalSaveableProperties) {
+        proto.globalSaveableProperties = [...metadata.globalSaveableProperties];
+      }
+
+      // Initialize saveableProperties
+      if (metadata.saveableProperties && !proto.saveableProperties) {
+        proto.saveableProperties = [...metadata.saveableProperties];
+      }
+
+      // Initialize uniqueIndexes
+      if (metadata.uniqueIndexes && !proto.uniqueIndexes) {
+        proto.uniqueIndexes = [...metadata.uniqueIndexes];
+      }
+
+      // Initialize regularIndexes
+      if (metadata.regularIndexes && !proto.regularIndexes) {
+        proto.regularIndexes = [...metadata.regularIndexes];
+      }
+
+      // Initialize searchableFields on constructor (not prototype)
+      if (metadata.searchableFields && !Array.isArray((decoratedClass as any).searchableFields)) {
+        (decoratedClass as any).searchableFields = [...metadata.searchableFields];
+      }
+
+      // Initialize _svDbOptions from metadata
+      if (metadata._svDbOptions && !originalConstructor._svDbOptions) {
+        originalConstructor._svDbOptions = { ...metadata._svDbOptions };
+      }
+    }
+
+    return decoratedClass as any;
   };
 }
 
@@ -59,7 +131,13 @@ export const setDefaultManagerForDoc = <T,>(managerArg: IManager, dbDocArg: T):
  * @param dbArg
  */
 export function managed<TManager extends IManager>(managerArg?: TManager | TDelayed<TManager>) {
-  return function classDecorator<T extends { new (...args: any[]): any }>(constructor: T) {
+  return function classDecorator(value: Function, context: ClassDecoratorContext) {
+    if (context.kind !== 'class') {
+      throw new Error('managed can only decorate classes');
+    }
+
+    const constructor = value as { new (...args: any[]): any };
+
     const decoratedClass = class extends constructor {
       public static className = constructor.name;
       public static get collection() {
@@ -109,7 +187,46 @@ export function managed<TManager extends IManager>(managerArg?: TManager | TDela
         return manager;
       }
     };
-    return decoratedClass;
+
+    // Initialize prototype properties from context.metadata (TC39 decorator metadata)
+    // This ensures prototype properties are available before any instance is created
+    const originalConstructor = value as any;
+    const metadata = context.metadata as any;
+    if (metadata) {
+      const proto = decoratedClass.prototype;
+
+      // Initialize globalSaveableProperties
+      if (metadata.globalSaveableProperties && !proto.globalSaveableProperties) {
+        proto.globalSaveableProperties = [...metadata.globalSaveableProperties];
+      }
+
+      // Initialize saveableProperties
+      if (metadata.saveableProperties && !proto.saveableProperties) {
+        proto.saveableProperties = [...metadata.saveableProperties];
+      }
+
+      // Initialize uniqueIndexes
+      if (metadata.uniqueIndexes && !proto.uniqueIndexes) {
+        proto.uniqueIndexes = [...metadata.uniqueIndexes];
+      }
+
+      // Initialize regularIndexes
+      if (metadata.regularIndexes && !proto.regularIndexes) {
+        proto.regularIndexes = [...metadata.regularIndexes];
+      }
+
+      // Initialize searchableFields on constructor (not prototype)
+      if (metadata.searchableFields && !Array.isArray((decoratedClass as any).searchableFields)) {
+        (decoratedClass as any).searchableFields = [...metadata.searchableFields];
+      }
+
+      // Initialize _svDbOptions from metadata
+      if (metadata._svDbOptions && !originalConstructor._svDbOptions) {
+        originalConstructor._svDbOptions = { ...metadata._svDbOptions };
+      }
+    }
+
+    return decoratedClass as any;
   };
 }
 
@@ -128,6 +245,8 @@ export class SmartdataCollection<T> {
   public smartdataDb: SmartdataDb;
   public uniqueIndexes: string[] = [];
   public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+  // flag to ensure text index is created only once
+  private textIndexCreated: boolean = false;
 
   constructor(classNameArg: string, smartDataDbArg: SmartdataDb) {
     // tell the collection where it belongs
@@ -150,19 +269,31 @@ export class SmartdataCollection<T> {
       });
       if (!wantedCollection) {
         await this.smartdataDb.mongoDb.createCollection(this.collectionName);
-        console.log(`Successfully initiated Collection ${this.collectionName}`);
+        logger.log('info', `Successfully initiated Collection ${this.collectionName}`);
       }
       this.mongoDbCollection = this.smartdataDb.mongoDb.collection(this.collectionName);
+      // Auto-create a compound text index on all searchable fields
+      // Use document constructor's searchableFields registered via decorator
+      const docCtor = (this as any).docCtor;
+      const searchableFields: string[] = docCtor?.searchableFields || [];
+      if (searchableFields.length > 0 && !this.textIndexCreated) {
+        // Build a compound text index spec
+        const indexSpec: Record<string, 'text'> = {};
+        searchableFields.forEach(f => { indexSpec[f] = 'text'; });
+        // Cast to any to satisfy TypeScript IndexSpecification typing
+        await this.mongoDbCollection.createIndex(indexSpec as any, { name: 'smartdata_text_index' });
+        this.textIndexCreated = true;
+      }
     }
   }
 
   /**
    * mark unique index
    */
-  public markUniqueIndexes(keyArrayArg: string[] = []) {
+  public async markUniqueIndexes(keyArrayArg: string[] = []) {
     for (const key of keyArrayArg) {
       if (!this.uniqueIndexes.includes(key)) {
-        this.mongoDbCollection.createIndex(key, {
+        await this.mongoDbCollection.createIndex({ [key]: 1 }, {
           unique: true,
         });
         // make sure we only call this once and not for every doc we create
@@ -174,12 +305,12 @@ export class SmartdataCollection<T> {
   /**
    * creates regular indexes for the collection
    */
-  public createRegularIndexes(indexesArg: Array<{field: string, options: IIndexOptions}> = []) {
+  public async createRegularIndexes(indexesArg: Array<{field: string, options: IIndexOptions}> = []) {
     for (const indexDef of indexesArg) {
       // Check if we've already created this index
       const indexKey = indexDef.field;
       if (!this.regularIndexes.some(i => i.field === indexKey)) {
-        this.mongoDbCollection.createIndex(
+        await this.mongoDbCollection.createIndex(
           { [indexDef.field]: 1 }, // Simple single-field index
           indexDef.options
         );
@@ -199,53 +330,74 @@ export class SmartdataCollection<T> {
   /**
    * finds an object in the DbCollection
    */
-  public async findOne(filterObject: any): Promise<any> {
+  public async findOne(
+    filterObject: any,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ): Promise<any> {
     await this.init();
-    const cursor = this.mongoDbCollection.find(filterObject);
-    const result = await cursor.next();
-    cursor.close();
-    return result;
+    // Use MongoDB driver's findOne with optional session
+    return this.mongoDbCollection.findOne(filterObject, { session: opts?.session });
   }
 
   public async getCursor(
     filterObjectArg: any,
     dbDocArg: typeof SmartDataDbDoc,
+    opts?: { session?: plugins.mongodb.ClientSession }
   ): Promise<SmartdataDbCursor<any>> {
     await this.init();
-    const cursor = this.mongoDbCollection.find(filterObjectArg);
+    const cursor = this.mongoDbCollection.find(filterObjectArg, { session: opts?.session });
     return new SmartdataDbCursor(cursor, dbDocArg);
   }
 
   /**
    * finds an object in the DbCollection
    */
-  public async findAll(filterObject: any): Promise<any[]> {
+  public async findAll(
+    filterObject: any,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ): Promise<any[]> {
     await this.init();
-    const cursor = this.mongoDbCollection.find(filterObject);
+    const cursor = this.mongoDbCollection.find(filterObject, { session: opts?.session });
     const result = await cursor.toArray();
     cursor.close();
     return result;
   }
 
   /**
-   * watches the collection while applying a filter
+   * Watches the collection, returning a SmartdataDbWatcher with RxJS and EventEmitter support.
+   * @param filterObject match filter for change stream
+   * @param opts optional MongoDB ChangeStreamOptions & { bufferTimeMs } to buffer events
+   * @param smartdataDbDocArg document class for instance creation
    */
   public async watch(
     filterObject: any,
-    smartdataDbDocArg: typeof SmartDataDbDoc,
+    opts: (plugins.mongodb.ChangeStreamOptions & { bufferTimeMs?: number }) = {},
+    smartdataDbDocArg?: typeof SmartDataDbDoc,
   ): Promise<SmartdataDbWatcher> {
     await this.init();
+    // Extract bufferTimeMs from options
+    const { bufferTimeMs, fullDocument, ...otherOptions } = opts || {};
+    // Determine fullDocument behavior: default to 'updateLookup'
+    const changeStreamOptions: plugins.mongodb.ChangeStreamOptions = {
+      ...otherOptions,
+      fullDocument:
+        fullDocument === undefined
+          ? 'updateLookup'
+          : (fullDocument as any) === true
+          ? 'updateLookup'
+          : fullDocument,
+    } as any;
+    // Build pipeline with match if provided
+    const pipeline = filterObject ? [{ $match: filterObject }] : [];
     const changeStream = this.mongoDbCollection.watch(
-      [
-        {
-          $match: filterObject,
-        },
-      ],
-      {
-        fullDocument: 'updateLookup',
-      },
+      pipeline,
+      changeStreamOptions,
+    );
+    const smartdataWatcher = new SmartdataDbWatcher(
+      changeStream,
+      smartdataDbDocArg,
+      { bufferTimeMs },
     );
-    const smartdataWatcher = new SmartdataDbWatcher(changeStream, smartdataDbDocArg);
     await smartdataWatcher.readyDeferred.promise;
     return smartdataWatcher;
   }
@@ -253,7 +405,10 @@ export class SmartdataCollection<T> {
   /**
    * create an object in the database
    */
-  public async insert(dbDocArg: T & SmartDataDbDoc<T, unknown>): Promise<any> {
+  public async insert(
+    dbDocArg: T & SmartDataDbDoc<T, unknown>,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ): Promise<any> {
     await this.init();
     await this.checkDoc(dbDocArg);
     this.markUniqueIndexes(dbDocArg.uniqueIndexes);
@@ -264,14 +419,17 @@ export class SmartdataCollection<T> {
     }
     
     const saveableObject = await dbDocArg.createSavableObject();
-    const result = await this.mongoDbCollection.insertOne(saveableObject);
+    const result = await this.mongoDbCollection.insertOne(saveableObject, { session: opts?.session });
     return result;
   }
 
   /**
    * inserts object into the DbCollection
    */
-  public async update(dbDocArg: T & SmartDataDbDoc<T, unknown>): Promise<any> {
+  public async update(
+    dbDocArg: T & SmartDataDbDoc<T, unknown>,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ): Promise<any> {
     await this.init();
     await this.checkDoc(dbDocArg);
     const identifiableObject = await dbDocArg.createIdentifiableObject();
@@ -286,21 +444,27 @@ export class SmartdataCollection<T> {
     const result = await this.mongoDbCollection.updateOne(
       identifiableObject,
       { $set: updateableObject },
-      { upsert: true },
+      { upsert: true, session: opts?.session },
     );
     return result;
   }
 
-  public async delete(dbDocArg: T & SmartDataDbDoc<T, unknown>): Promise<any> {
+  public async delete(
+    dbDocArg: T & SmartDataDbDoc<T, unknown>,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ): Promise<any> {
     await this.init();
     await this.checkDoc(dbDocArg);
     const identifiableObject = await dbDocArg.createIdentifiableObject();
-    await this.mongoDbCollection.deleteOne(identifiableObject);
+    await this.mongoDbCollection.deleteOne(identifiableObject, { session: opts?.session });
   }
 
-  public async getCount(filterObject: any) {
+  public async getCount(
+    filterObject: any,
+    opts?: { session?: plugins.mongodb.ClientSession }
+  ) {
     await this.init();
-    return this.mongoDbCollection.countDocuments(filterObject);
+    return this.mongoDbCollection.countDocuments(filterObject, { session: opts?.session });
   }
 
   /**

ts/classes.cursor.ts

@@ -15,14 +15,14 @@ export class SmartdataDbCursor<T = any> {
     this.smartdataDbDoc = dbDocArg;
   }
 
-  public async next(closeAtEnd = true) {
+  public async next(closeAtEnd = true): Promise<T> {
     const result = this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(
       await this.mongodbCursor.next(),
     );
     if (!result && closeAtEnd) {
       await this.close();
     }
-    return result;
+    return result as T;
   }
 
   public async forEach(forEachFuncArg: (itemArg: T) => Promise<any>, closeCursorAtEnd = true) {
@@ -40,6 +40,11 @@ export class SmartdataDbCursor<T = any> {
     }
   }
 
+  public async toArray(): Promise<T[]> {
+    const result = await this.mongodbCursor.toArray();
+    return result.map((itemArg) => this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(itemArg)) as T[];
+  }
+
   public async close() {
     await this.mongodbCursor.close();
   }

ts/classes.db.ts

@@ -35,24 +35,43 @@ export class SmartdataDb {
    * connects to the database that was specified during instance creation
    */
   public async init(): Promise<any> {
+    try {
+      // Safely encode credentials to handle special characters
+      const encodedUser = this.smartdataOptions.mongoDbUser 
+        ? encodeURIComponent(this.smartdataOptions.mongoDbUser) 
+        : '';
+      const encodedPass = this.smartdataOptions.mongoDbPass 
+        ? encodeURIComponent(this.smartdataOptions.mongoDbPass) 
+        : '';
+      
       const finalConnectionUrl = this.smartdataOptions.mongoDbUrl
-      .replace('<USERNAME>', this.smartdataOptions.mongoDbUser)
-      .replace('<username>', this.smartdataOptions.mongoDbUser)
-      .replace('<USER>', this.smartdataOptions.mongoDbUser)
-      .replace('<user>', this.smartdataOptions.mongoDbUser)
-      .replace('<PASSWORD>', this.smartdataOptions.mongoDbPass)
-      .replace('<password>', this.smartdataOptions.mongoDbPass)
+        .replace('<USERNAME>', encodedUser)
+        .replace('<username>', encodedUser)
+        .replace('<USER>', encodedUser)
+        .replace('<user>', encodedUser)
+        .replace('<PASSWORD>', encodedPass)
+        .replace('<password>', encodedPass)
         .replace('<DBNAME>', this.smartdataOptions.mongoDbName)
         .replace('<dbname>', this.smartdataOptions.mongoDbName);
 
-    this.mongoDbClient = await plugins.mongodb.MongoClient.connect(finalConnectionUrl, {
-      maxPoolSize: 100,
-      maxIdleTimeMS: 10,
-    });
+      const clientOptions: plugins.mongodb.MongoClientOptions = {
+        maxPoolSize: (this.smartdataOptions as any).maxPoolSize ?? 100,
+        maxIdleTimeMS: (this.smartdataOptions as any).maxIdleTimeMS ?? 300000, // 5 minutes default
+        serverSelectionTimeoutMS: (this.smartdataOptions as any).serverSelectionTimeoutMS ?? 30000,
+        retryWrites: true,
+      };
+
+      this.mongoDbClient = await plugins.mongodb.MongoClient.connect(finalConnectionUrl, clientOptions);
       this.mongoDb = this.mongoDbClient.db(this.smartdataOptions.mongoDbName);
       this.status = 'connected';
       this.statusConnectedDeferred.resolve();
-    console.log(`Connected to database ${this.smartdataOptions.mongoDbName}`);
+      logger.log('info', `Connected to database ${this.smartdataOptions.mongoDbName}`);
+    } catch (error) {
+      this.status = 'disconnected';
+      this.statusConnectedDeferred.reject(error);
+      logger.log('error', `Failed to connect to database ${this.smartdataOptions.mongoDbName}: ${error.message}`);
+      throw error;
+    }
   }
 
   /**
@@ -63,6 +82,12 @@ export class SmartdataDb {
     this.status = 'disconnected';
     logger.log('info', `disconnected from database ${this.smartdataOptions.mongoDbName}`);
   }
+  /**
+   * Start a MongoDB client session for transactions
+   */
+  public startSession(): plugins.mongodb.ClientSession {
+    return this.mongoDbClient.startSession();
+  }
 
   // handle table to class distribution
 

ts/classes.distributedcoordinator.ts

@@ -3,6 +3,7 @@ import { SmartdataDb } from './classes.db.js';
 import { managed, setDefaultManagerForDoc } from './classes.collection.js';
 import { SmartDataDbDoc, svDb, unI } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
+import { logger } from './logging.js';
 
 @managed()
 export class DistributedClass extends SmartDataDbDoc<DistributedClass, DistributedClass> {
@@ -63,11 +64,11 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
         this.ownInstance.data.elected = false;
       }
       if (this.ownInstance?.data.status === 'stopped') {
-        console.log(`stopping a distributed instance that has not been started yet.`);
+        logger.log('warn', `stopping a distributed instance that has not been started yet.`);
       }
       this.ownInstance.data.status = 'stopped';
       await this.ownInstance.save();
-      console.log(`stopped ${this.ownInstance.id}`);
+      logger.log('info', `stopped ${this.ownInstance.id}`);
     });
   }
 
@@ -83,17 +84,17 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
   public async sendHeartbeat() {
     await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
       if (this.ownInstance.data.status === 'stopped') {
-        console.log(`aborted sending heartbeat because status is stopped`);
+        logger.log('debug', `aborted sending heartbeat because status is stopped`);
         return;
       }
       await this.ownInstance.updateFromDb();
       this.ownInstance.data.lastUpdated = Date.now();
       await this.ownInstance.save();
-      console.log(`sent heartbeat for ${this.ownInstance.id}`);
+      logger.log('debug', `sent heartbeat for ${this.ownInstance.id}`);
       const allInstances = DistributedClass.getInstances({});
     });
     if (this.ownInstance.data.status === 'stopped') {
-      console.log(`aborted sending heartbeat because status is stopped`);
+      logger.log('info', `aborted sending heartbeat because status is stopped`);
       return;
     }
     const eligibleLeader = await this.getEligibleLeader();
@@ -120,7 +121,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
         await this.ownInstance.save();
       });
     } else {
-      console.warn(`distributed instance already initialized`);
+      logger.log('warn', `distributed instance already initialized`);
     }
 
     // lets enable the heartbeat
@@ -149,24 +150,24 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
   public async checkAndMaybeLead() {
     await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
       this.ownInstance.data.status = 'initializing';
-      this.ownInstance.save();
+      await this.ownInstance.save();
     });
     if (await this.getEligibleLeader()) {
       await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
         await this.ownInstance.updateFromDb();
         this.ownInstance.data.status = 'settled';
         await this.ownInstance.save();
-        console.log(`${this.ownInstance.id} settled as follower`);
+        logger.log('info', `${this.ownInstance.id} settled as follower`);
       });
       return;
     } else if (
       (await DistributedClass.getInstances({})).find((instanceArg) => {
-        instanceArg.data.status === 'bidding' &&
+        return instanceArg.data.status === 'bidding' &&
           instanceArg.data.biddingStartTime <= Date.now() - 4000 &&
           instanceArg.data.biddingStartTime >= Date.now() - 30000;
       })
     ) {
-      console.log('too late to the bidding party... waiting for next round.');
+      logger.log('info', 'too late to the bidding party... waiting for next round.');
       return;
     } else {
       await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
@@ -175,9 +176,9 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
         this.ownInstance.data.biddingStartTime = Date.now();
         this.ownInstance.data.biddingShortcode = plugins.smartunique.shortId();
         await this.ownInstance.save();
-        console.log('bidding code stored.');
+        logger.log('info', 'bidding code stored.');
       });
-      console.log(`bidding for leadership...`);
+      logger.log('info', `bidding for leadership...`);
       await plugins.smartdelay.delayFor(plugins.smarttime.getMilliSecondsFromUnits({ seconds: 5 }));
       await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
         let biddingInstances = await DistributedClass.getInstances({});
@@ -187,7 +188,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
             instanceArg.data.lastUpdated >=
               Date.now() - plugins.smarttime.getMilliSecondsFromUnits({ seconds: 10 }),
         );
-        console.log(`found ${biddingInstances.length} bidding instances...`);
+        logger.log('info', `found ${biddingInstances.length} bidding instances...`);
         this.ownInstance.data.elected = true;
         for (const biddingInstance of biddingInstances) {
           if (biddingInstance.data.biddingShortcode < this.ownInstance.data.biddingShortcode) {
@@ -195,7 +196,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
           }
         }
         await plugins.smartdelay.delayFor(5000);
-        console.log(`settling with status elected = ${this.ownInstance.data.elected}`);
+        logger.log('info', `settling with status elected = ${this.ownInstance.data.elected}`);
         this.ownInstance.data.status = 'settled';
         await this.ownInstance.save();
       });
@@ -226,11 +227,11 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
     this.distributedWatcher.changeSubject.subscribe({
       next: async (distributedDoc) => {
         if (!distributedDoc) {
-          console.log(`registered deletion of instance...`);
+          logger.log('info', `registered deletion of instance...`);
           return;
         }
-        console.log(distributedDoc);
-        console.log(`registered change for ${distributedDoc.id}`);
+        logger.log('info', distributedDoc);
+        logger.log('info', `registered change for ${distributedDoc.id}`);
         distributedDoc;
       },
     });
@@ -252,7 +253,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
   ): Promise<plugins.taskbuffer.distributedCoordination.IDistributedTaskRequestResult> {
     await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
       if (!this.ownInstance) {
-        console.error('instance need to be started first...');
+        logger.log('error', 'instance need to be started first...');
         return;
       }
       await this.ownInstance.updateFromDb();
@@ -268,7 +269,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
       return taskRequestResult;
     });
     if (!result) {
-      console.warn('no result found for task request...');
+      logger.log('warn', 'no result found for task request...');
       return null;
     }
     return result;
@@ -285,7 +286,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
         );
       });
       if (!existingInfoBasis) {
-        console.warn('trying to update a non existing task request... aborting!');
+        logger.log('warn', 'trying to update a non existing task request... aborting!');
         return;
       }
       Object.assign(existingInfoBasis, infoBasisArg);
@@ -293,8 +294,10 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
       plugins.smartdelay.delayFor(60000).then(() => {
         this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
           const indexToRemove = this.ownInstance.data.taskRequests.indexOf(existingInfoBasis);
-          this.ownInstance.data.taskRequests.splice(indexToRemove, indexToRemove);
+          if (indexToRemove >= 0) {
+            this.ownInstance.data.taskRequests.splice(indexToRemove, 1);
             await this.ownInstance.save();
+          }
         });
       });
     });

ts/classes.easystore.ts

@@ -18,7 +18,7 @@ export class EasyStore<T> {
       public nameId: string;
 
       @svDb()
-      public ephermal: {
+      public ephemeral: {
         activated: boolean;
         timeout: number;
       };
@@ -32,8 +32,8 @@ export class EasyStore<T> {
     return SmartdataEasyStore;
   })();
 
-  constructor(nameIdArg: string, smnartdataDbRefArg: SmartdataDb) {
-    this.smartdataDbRef = smnartdataDbRefArg;
+  constructor(nameIdArg: string, smartdataDbRefArg: SmartdataDb) {
+    this.smartdataDbRef = smartdataDbRefArg;
     this.nameId = nameIdArg;
   }
 
@@ -110,10 +110,12 @@ export class EasyStore<T> {
     await easyStore.save();
   }
 
-  public async cleanUpEphermal() {
-    while (
-      (await this.smartdataDbRef.statusConnectedDeferred.promise) &&
-      this.smartdataDbRef.status === 'connected'
-    ) {}
+  public async cleanUpEphemeral() {
+    // Clean up ephemeral data periodically while connected
+    while (this.smartdataDbRef.status === 'connected') {
+      await plugins.smartdelay.delayFor(60000); // Check every minute
+      // TODO: Implement actual cleanup logic for ephemeral data
+      // For now, this prevents the infinite CPU loop
+    }
   }
 }

ts/classes.lucene.adapter.ts

@@ -2,6 +2,7 @@
  * Lucene to MongoDB query adapter for SmartData
  */
 import * as plugins from './plugins.js';
+import { logger } from './logging.js';
 
 // Types
 type NodeType =
@@ -290,11 +291,11 @@ export class LuceneParser {
     const includeLower = this.tokens[this.pos] === '[';
     const includeUpper = this.tokens[this.pos + 4] === ']';
 
-    this.pos++; // Skip open bracket
-
+    // Ensure tokens for lower, TO, upper, and closing bracket exist
     if (this.pos + 4 >= this.tokens.length) {
       throw new Error('Invalid range query syntax');
     }
+    this.pos++; // Skip open bracket
 
     const lower = this.tokens[this.pos];
     this.pos++;
@@ -329,7 +330,16 @@ export class LuceneParser {
  * FIXED VERSION - proper MongoDB query structure
  */
 export class LuceneToMongoTransformer {
-  constructor() {}
+  private defaultFields: string[];
+  constructor(defaultFields: string[] = []) {
+    this.defaultFields = defaultFields;
+  }
+  /**
+   * Escape special characters for use in RegExp patterns
+   */
+  private escapeRegex(input: string): string {
+    return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+  }
 
   /**
    * Transform a Lucene AST node to a MongoDB query
@@ -366,18 +376,21 @@ export class LuceneToMongoTransformer {
    * FIXED: properly structured $or query for multiple fields
    */
   private transformTerm(node: TermNode, searchFields?: string[]): any {
-    // If specific fields are provided, search across those fields
-    if (searchFields && searchFields.length > 0) {
-      // Create an $or query to search across multiple fields
-      const orConditions = searchFields.map((field) => ({
-        [field]: { $regex: node.value, $options: 'i' },
-      }));
-
-      return { $or: orConditions };
+    // Build regex pattern, support wildcard (*) and fuzzy (?) if present
+    const term = node.value;
+    // Determine regex pattern: wildcard conversion or exact escape
+    let pattern: string;
+    if (term.includes('*') || term.includes('?')) {
+      pattern = this.luceneWildcardToRegex(term);
+    } else {
+      pattern = this.escapeRegex(term);
     }
-
-    // Otherwise, use text search (requires a text index on desired fields)
-    return { $text: { $search: node.value } };
+    // Search across provided fields or default fields
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
+    }));
+    return { $or: orConditions };
   }
 
   /**
@@ -385,19 +398,16 @@ export class LuceneToMongoTransformer {
    * FIXED: properly structured $or query for multiple fields
    */
   private transformPhrase(node: PhraseNode, searchFields?: string[]): any {
-    // If specific fields are provided, search phrase across those fields
-    if (searchFields && searchFields.length > 0) {
-      const orConditions = searchFields.map((field) => ({
-        [field]: { $regex: `${node.value.replace(/\s+/g, '\\s+')}`, $options: 'i' },
+    // Use regex across provided fields or default fields, respecting word boundaries
+    const parts = node.value.split(/\s+/).map((t) => this.escapeRegex(t));
+    const pattern = parts.join('\\s+');
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
     }));
-
     return { $or: orConditions };
   }
 
-    // For phrases, we use a regex to ensure exact matches
-    return { $text: { $search: `"${node.value}"` } };
-  }
-
   /**
    * Transform a field query to MongoDB query
    */
@@ -429,9 +439,14 @@ export class LuceneToMongoTransformer {
       };
     }
 
-    // Special case for exact term matches on fields
+    // Special case for exact term matches on fields (supporting wildcard characters)
     if (node.value.type === 'TERM') {
-      return { [node.field]: { $regex: (node.value as TermNode).value, $options: 'i' } };
+      const val = (node.value as TermNode).value;
+      if (val.includes('*') || val.includes('?')) {
+        const regex = this.luceneWildcardToRegex(val);
+        return { [node.field]: { $regex: regex, $options: 'i' } };
+      }
+      return { [node.field]: { $regex: val, $options: 'i' } };
     }
 
     // Special case for phrase matches on fields
@@ -626,7 +641,7 @@ export class LuceneToMongoTransformer {
   /**
    * Convert Lucene wildcards to MongoDB regex patterns
    */
-  private luceneWildcardToRegex(wildcardPattern: string): string {
+  public luceneWildcardToRegex(wildcardPattern: string): string {
     // Replace Lucene wildcards with regex equivalents
     // * => .*
     // ? => .
@@ -691,7 +706,8 @@ export class SmartdataLuceneAdapter {
    */
   constructor(defaultSearchFields?: string[]) {
     this.parser = new LuceneParser();
-    this.transformer = new LuceneToMongoTransformer();
+    // Pass default searchable fields into transformer
+    this.transformer = new LuceneToMongoTransformer(defaultSearchFields || []);
     if (defaultSearchFields) {
       this.defaultSearchFields = defaultSearchFields;
     }
@@ -704,7 +720,7 @@ export class SmartdataLuceneAdapter {
    */
   convert(luceneQuery: string, searchFields?: string[]): any {
     try {
-      // For simple single term queries, create a simpler query structure
+      // For simple single-term queries (no field:, boolean, grouping), use simpler regex
       if (
         !luceneQuery.includes(':') &&
         !luceneQuery.includes(' AND ') &&
@@ -713,13 +729,17 @@ export class SmartdataLuceneAdapter {
         !luceneQuery.includes('(') &&
         !luceneQuery.includes('[')
       ) {
-        // This is a simple term, use a more direct approach
         const fieldsToSearch = searchFields || this.defaultSearchFields;
-
         if (fieldsToSearch && fieldsToSearch.length > 0) {
+          // Handle wildcard characters in query
+          let pattern = luceneQuery;
+          if (luceneQuery.includes('*') || luceneQuery.includes('?')) {
+            // Use transformer to convert wildcard pattern
+            pattern = this.transformer.luceneWildcardToRegex(luceneQuery);
+          }
           return {
             $or: fieldsToSearch.map((field) => ({
-              [field]: { $regex: luceneQuery, $options: 'i' },
+              [field]: { $regex: pattern, $options: 'i' },
             })),
           };
         }
@@ -735,7 +755,7 @@ export class SmartdataLuceneAdapter {
       // Transform the AST to a MongoDB query
       return this.transformWithFields(ast, fieldsToSearch);
     } catch (error) {
-      console.error(`Failed to convert Lucene query "${luceneQuery}":`, error);
+      logger.log('error', `Failed to convert Lucene query "${luceneQuery}":`, error);
       throw new Error(`Failed to convert Lucene query: ${error}`);
     }
   }

ts/classes.watcher.ts

@@ -1,37 +1,73 @@
 import { SmartDataDbDoc } from './classes.doc.js';
 import * as plugins from './plugins.js';
+import { EventEmitter } from 'node:events';
 
 /**
  * a wrapper for the native mongodb cursor. Exposes better
  */
-export class SmartdataDbWatcher<T = any> {
+/**
+ * Wraps a MongoDB ChangeStream with RxJS and EventEmitter support.
+ */
+export class SmartdataDbWatcher<T = any> extends EventEmitter {
   // STATIC
   public readyDeferred = plugins.smartpromise.defer();
 
   // INSTANCE
   private changeStream: plugins.mongodb.ChangeStream<T>;
-
-  public changeSubject = new plugins.smartrx.rxjs.Subject<T>();
+  private rawSubject: plugins.smartrx.rxjs.Subject<T>;
+  /** Emits change documents (or arrays of documents if buffered) */
+  public changeSubject: any;
+  /**
+   * @param changeStreamArg native MongoDB ChangeStream
+   * @param smartdataDbDocArg document class for instance creation
+   * @param opts.bufferTimeMs optional milliseconds to buffer events via RxJS
+   */
   constructor(
     changeStreamArg: plugins.mongodb.ChangeStream<T>,
     smartdataDbDocArg: typeof SmartDataDbDoc,
+    opts?: { bufferTimeMs?: number },
   ) {
+    super();
+    this.rawSubject = new plugins.smartrx.rxjs.Subject<T>();
+    // Apply buffering if requested
+    if (opts && opts.bufferTimeMs) {
+      this.changeSubject = this.rawSubject.pipe(plugins.smartrx.rxjs.ops.bufferTime(opts.bufferTimeMs));
+    } else {
+      this.changeSubject = this.rawSubject;
+    }
     this.changeStream = changeStreamArg;
     this.changeStream.on('change', async (item: any) => {
-      if (!item.fullDocument) {
-        this.changeSubject.next(null);
-        return;
+      let docInstance: T = null;
+      if (item.fullDocument) {
+        docInstance = smartdataDbDocArg.createInstanceFromMongoDbNativeDoc(
+          item.fullDocument
+        ) as any as T;
       }
-      this.changeSubject.next(
-        smartdataDbDocArg.createInstanceFromMongoDbNativeDoc(item.fullDocument) as any as T,
-      );
+      // Notify subscribers
+      this.rawSubject.next(docInstance);
+      this.emit('change', docInstance);
     });
+    // Signal readiness after one tick
     plugins.smartdelay.delayFor(0).then(() => {
       this.readyDeferred.resolve();
     });
   }
 
-  public async close() {
+  /**
+   * Close the change stream, complete the RxJS subject, and remove listeners.
+   */
+  public async close(): Promise<void> {
+    // Close MongoDB ChangeStream
     await this.changeStream.close();
+    // Complete the subject to teardown any buffering operators
+    this.rawSubject.complete();
+    // Remove all EventEmitter listeners
+    this.removeAllListeners();
+  }
+  /**
+   * Alias for close(), matching README usage
+   */
+  public async stop(): Promise<void> {
+    return this.close();
   }
 }

tsconfig.json

@@ -1,7 +1,5 @@
 {
   "compilerOptions": {
-    "experimentalDecorators": true,
-    "useDefineForClassFields": false,
     "target": "ES2022",
     "module": "NodeNext",
     "moduleResolution": "NodeNext",