5.11.2

fix(readme): Update readme to clarify usage of searchable fields retrieval
5.11.1
2025-04-21 17:31:30 +00:00 · 2025-04-21 17:31:30 +00:00 · 2025-04-21 16:35:29 +00:00 · 2025-04-21 16:35:29 +00:00 · 2025-04-21 15:29:11 +00:00 · 2025-04-21 15:29:11 +00:00
21 changed files with 1362 additions and 362 deletions
--- a/.gitea/workflows/default_nottags.yaml
+++ b/.gitea/workflows/default_nottags.yaml
@ -6,8 +6,8 @@ on:
      - '**'

 env:
-  IMAGE: registry.gitlab.com/hosttoday/ht-docker-node:npmci
-  NPMCI_COMPUTED_REPOURL: https://${{gitea.repository_owner}}:${{secrets.GITEA_TOKEN}}@gitea.lossless.digital/${{gitea.repository}}.git
+  IMAGE: code.foss.global/host.today/ht-docker-node:npmci
+  NPMCI_COMPUTED_REPOURL: https://${{gitea.repository_owner}}:${{secrets.GITEA_TOKEN}}@/${{gitea.repository}}.git
  NPMCI_TOKEN_NPM: ${{secrets.NPMCI_TOKEN_NPM}}
  NPMCI_TOKEN_NPM2: ${{secrets.NPMCI_TOKEN_NPM2}}
  NPMCI_GIT_GITHUBTOKEN: ${{secrets.NPMCI_GIT_GITHUBTOKEN}}
@ -26,7 +26,7 @@ jobs:
      - name: Install pnpm and npmci
        run: |
          pnpm install -g pnpm
-          pnpm install -g @shipzone/npmci
+          pnpm install -g @ship.zone/npmci

      - name: Run npm prepare
        run: npmci npm prepare
--- a/.gitea/workflows/default_tags.yaml
+++ b/.gitea/workflows/default_tags.yaml
@ -6,8 +6,8 @@ on:
      - '*'

 env:
-  IMAGE: registry.gitlab.com/hosttoday/ht-docker-node:npmci
-  NPMCI_COMPUTED_REPOURL: https://${{gitea.repository_owner}}:${{secrets.GITEA_TOKEN}}@gitea.lossless.digital/${{gitea.repository}}.git
+  IMAGE: code.foss.global/host.today/ht-docker-node:npmci
+  NPMCI_COMPUTED_REPOURL: https://${{gitea.repository_owner}}:${{secrets.GITEA_TOKEN}}@/${{gitea.repository}}.git
  NPMCI_TOKEN_NPM: ${{secrets.NPMCI_TOKEN_NPM}}
  NPMCI_TOKEN_NPM2: ${{secrets.NPMCI_TOKEN_NPM2}}
  NPMCI_GIT_GITHUBTOKEN: ${{secrets.NPMCI_GIT_GITHUBTOKEN}}
@ -26,7 +26,7 @@ jobs:
      - name: Prepare
        run: |
          pnpm install -g pnpm
-          pnpm install -g @shipzone/npmci
+          pnpm install -g @ship.zone/npmci
          npmci npm prepare

      - name: Audit production dependencies
@ -54,7 +54,7 @@ jobs:
      - name: Prepare
        run: |
          pnpm install -g pnpm
-          pnpm install -g @shipzone/npmci
+          pnpm install -g @ship.zone/npmci
          npmci npm prepare

      - name: Test stable
@ -82,7 +82,7 @@ jobs:
      - name: Prepare
        run: |
          pnpm install -g pnpm
-          pnpm install -g @shipzone/npmci
+          pnpm install -g @ship.zone/npmci
          npmci npm prepare

      - name: Release
@ -104,7 +104,7 @@ jobs:
      - name: Prepare
        run: |
          pnpm install -g pnpm
-          pnpm install -g @shipzone/npmci
+          pnpm install -g @ship.zone/npmci
          npmci npm prepare

      - name: Code quality
--- a/.gitignore
+++ b/.gitignore
@ -3,7 +3,6 @@
 # artifacts
 coverage/
 public/
-pages/

 # installs
 node_modules/
@ -17,4 +16,4 @@ node_modules/
 dist/
 dist_*/

-# custom
+#------# custom
--- a/changelog.md
+++ b/changelog.md
@ -1,5 +1,114 @@
 # Changelog

+## 2025-04-21 - 5.11.2 - fix(readme)
+Update readme to clarify usage of searchable fields retrieval
+
+- Replaced getSearchableFields('Product') with Product.getSearchableFields()
+- Updated documentation to reference the static method Class.getSearchableFields()
+
+## 2025-04-21 - 5.11.1 - fix(doc)
+Refactor searchable fields API and improve collection registration.
+
+- Removed the standalone getSearchableFields utility in favor of a static method on document classes.
+- Updated tests to use the new static method (e.g., Product.getSearchableFields()).
+- Ensured the Collection decorator attaches a docCtor property to correctly register searchable fields.
+- Added try/catch in test cleanup to gracefully handle dropDatabase errors.
+
+## 2025-04-21 - 5.11.0 - feat(ts/classes.lucene.adapter)
+Expose luceneWildcardToRegex method to allow external usage and enhance regex transformation capabilities.
+
+- Changed luceneWildcardToRegex from private to public in ts/classes.lucene.adapter.ts.
+
+## 2025-04-21 - 5.10.0 - feat(search)
+Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests
+
+- Updated readme.md with detailed Lucene‑style search examples and use cases
+- Enhanced LuceneToMongoTransformer to properly handle wildcard conversion and regex escaping
+- Improved search query parsing in SmartDataDbDoc for field-specific, multi-term, and advanced Lucene syntax
+- Added new advanced search tests covering boolean operators, grouping, quoted phrases, and wildcard queries
+
+## 2025-04-18 - 5.9.2 - fix(documentation)
+Update search API documentation to replace deprecated searchWithLucene examples with the unified search(query) API and clarify its behavior.
+
+- Replaced 'searchWithLucene' examples with 'search(query)' in the README.
+- Updated explanation to detail field-specific exact match, partial word regex search, multi-word literal matching, and handling of empty queries.
+- Clarified guidelines for creating MongoDB text indexes on searchable fields for optimized search performance.
+
+## 2025-04-18 - 5.9.1 - fix(search)
+Refactor search tests to use unified search API and update text index type casting
+
+- Replaced all calls from searchWithLucene with search in test/search tests
+- Updated text index specification in the collection class to use proper type casting
+
+## 2025-04-18 - 5.9.0 - feat(collections/search)
+Improve text index creation and search fallback mechanisms in collections and document search methods
+
+- Auto-create a compound text index on all searchable fields in SmartdataCollection with a one-time flag to prevent duplicate index creation.
+- Refine the search method in SmartDataDbDoc to support exact field matches and safe regex fallback for non-Lucene queries.
+
+## 2025-04-17 - 5.8.4 - fix(core)
+Update commit metadata with no functional code changes
+
+- Commit info and documentation refreshed
+- No code or test changes detected in the diff
+
+## 2025-04-17 - 5.8.3 - fix(readme)
+Improve readme documentation on data models and connection management
+
+- Clarify that data models use @Collection, @unI, @svDb, @index, and @searchable decorators
+- Document that ObjectId and Buffer fields are stored as BSON types natively without extra decorators
+- Update connection management section to use 'db.close()' instead of 'db.disconnect()'
+- Revise license section to reference the MIT License without including additional legal details
+
+## 2025-04-14 - 5.8.2 - fix(classes.doc.ts)
+Ensure collection initialization before creating a cursor in getCursorExtended
+
+- Added 'await collection.init()' to guarantee that the MongoDB collection is initialized before using the cursor
+- Prevents potential runtime errors when accessing collection.mongoDbCollection
+
+## 2025-04-14 - 5.8.1 - fix(cursor, doc)
+Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.
+
+- Specify Promise<T> as return type for next() in SmartdataDbCursor and cast return value to T.
+- Specify Promise<T[]> as return type for toArray() in SmartdataDbCursor and cast return value to T[].
+- Update getCursorExtended to return Promise<SmartdataDbCursor<T>> for clearer type safety.
+
+## 2025-04-14 - 5.8.0 - feat(cursor)
+Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances
+
+- Introduced asynchronous toArray method in SmartdataDbCursor which retrieves all documents from the MongoDB cursor
+- Maps each native document to a SmartDataDbDoc instance using createInstanceFromMongoDbNativeDoc for consistent API usage
+
+## 2025-04-14 - 5.7.0 - feat(SmartDataDbDoc)
+Add extended cursor method getCursorExtended for flexible cursor modifications
+
+- Introduces getCursorExtended in classes.doc.ts to allow modifier functions for MongoDB cursors
+- Wraps the modified cursor with SmartdataDbCursor for improved API consistency
+- Enhances querying capabilities by enabling customized cursor transformations
+
+## 2025-04-07 - 5.6.0 - feat(indexing)
+Add support for regular index creation in documents and collections
+
+- Implement new index decorator in classes.doc.ts to mark properties with regular indexing options
+- Update SmartdataCollection to create regular indexes if defined on a document during insert
+- Enhance document structure to store and utilize regular index configurations
+
+## 2025-04-06 - 5.5.1 - fix(ci & formatting)
+Minor fixes: update CI workflow image and npmci package references, adjust package.json and readme URLs, and apply consistent code formatting.
+
+- Update image and repo URL in Gitea workflows from GitLab to code.foss.global
+- Replace '@shipzone/npmci' with '@ship.zone/npmci' throughout CI scripts
+- Adjust homepage and bugs URL in package.json and readme
+- Apply trailing commas and consistent formatting in TypeScript source files
+- Minor update to .gitignore custom section label
+
+## 2025-04-06 - 5.5.0 - feat(search)
+Enhance search functionality with robust Lucene query transformation and reliable fallback mechanisms
+
+- Improve Lucene adapter to properly structure $or queries for term, phrase, wildcard, and fuzzy search
+- Implement and document a robust searchWithLucene method with fallback to in-memory filtering
+- Update readme and tests with extensive examples for @searchable fields and Lucene-based queries
+
 ## 2025-04-06 - 5.4.0 - feat(core)
 Refactor file structure and update dependency versions

--- a/package.json
+++ b/package.json
@ -1,6 +1,6 @@
 {
  "name": "@push.rocks/smartdata",
-  "version": "5.4.0",
+  "version": "5.11.2",
  "private": false,
  "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
  "main": "dist_ts/index.js",
@ -18,9 +18,9 @@
  "author": "Lossless GmbH",
  "license": "MIT",
  "bugs": {
-    "url": "https://gitlab.com/pushrocks/smartdata/issues"
+    "url": "https://code.foss.global/push.rocks/smartdata/issues"
  },
-  "homepage": "https://code.foss.global/push.rocks/smartdata",
+  "homepage": "https://code.foss.global/push.rocks/smartdata#readme",
  "dependencies": {
    "@push.rocks/lik": "^6.0.14",
    "@push.rocks/smartdelay": "^3.0.1",
@ -68,5 +68,8 @@
    "custom data types",
    "ODM"
  ],
-  "packageManager": "pnpm@10.7.0+sha512.6b865ad4b62a1d9842b61d674a393903b871d9244954f652b8842c2b553c72176b278f64c463e52d40fff8aba385c235c8c9ecf5cc7de4fd78b8bb6d49633ab6"
+  "packageManager": "pnpm@10.7.0+sha512.6b865ad4b62a1d9842b61d674a393903b871d9244954f652b8842c2b553c72176b278f64c463e52d40fff8aba385c235c8c9ecf5cc7de4fd78b8bb6d49633ab6",
+  "pnpm": {
+    "overrides": {}
+  }
 }
--- a/readme.md
+++ b/readme.md
@ -18,6 +18,7 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
 - **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
 - **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
 - **Serialization Hooks**: Custom serialization and deserialization of document properties
+- **Powerful Search Capabilities**: Unified `search(query)` API supporting field:value exact matches, multi-field regex searches, case-insensitive matching, and automatic escaping to prevent regex injection

 ## Requirements

@ -26,6 +27,7 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
 - TypeScript >= 4.x (for development)

 ## Install
+
 To install `@push.rocks/smartdata`, use npm:

 ```bash
@ -39,9 +41,11 @@ pnpm add @push.rocks/smartdata
 ```

 ## Usage
+
 `@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.

 ### Setting Up and Connecting to the Database
+
 Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.

 ```typescript
@ -61,35 +65,43 @@ await db.init();
 ```

 ### Defining Data Models
-Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
+
+Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, `@svDb`, `@index`, and `@searchable` to define your data models. Fields of type `ObjectId` or `Buffer` decorated with `@svDb()` will be stored as BSON ObjectId and Binary, respectively; no separate `@oid()` or `@bin()` decorators are required.

 ```typescript
-import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index } from '@push.rocks/smartdata';
+import {
+  SmartDataDbDoc,
+  Collection,
+  unI,
+  svDb,
+  index,
+  searchable,
+} from '@push.rocks/smartdata';
 import { ObjectId } from 'mongodb';

-@Collection(() => db)  // Associate this model with the database instance
+@Collection(() => db) // Associate this model with the database instance
 class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id: string = 'unique-user-id'; // Mark 'id' as a unique index

  @svDb()
-  public username: string;  // Mark 'username' to be saved in DB
+  @searchable() // Mark 'username' as searchable
+  public username: string; // Mark 'username' to be saved in DB

  @svDb()
+  @searchable() // Mark 'email' as searchable
  @index() // Create a regular index for this field
-  public email: string;  // Mark 'email' to be saved in DB
+  public email: string; // Mark 'email' to be saved in DB

  @svDb()
-  @oid() // Automatically handle as ObjectId type
-  public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
+  public organizationId: ObjectId; // Stored as BSON ObjectId

  @svDb()
-  @bin() // Automatically handle as Binary data
-  public profilePicture: Buffer; // Will be automatically converted to/from Binary
+  public profilePicture: Buffer; // Stored as BSON Binary

  @svDb({
    serialize: (data) => JSON.stringify(data), // Custom serialization
-    deserialize: (data) => JSON.parse(data)    // Custom deserialization
+    deserialize: (data) => JSON.parse(data), // Custom deserialization
  })
  public preferences: Record<string, any>;

@ -102,15 +114,18 @@ class User extends SmartDataDbDoc<User, User> {
 ```

 ### CRUD Operations
+
 `@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.

 #### Create
+
 ```typescript
 const newUser = new User('myUsername', 'myEmail@example.com');
-await newUser.save();  // Save the new user to the database
+await newUser.save(); // Save the new user to the database
 ```

 #### Read
+
 ```typescript
 // Fetch a single user by a unique attribute
 const user = await User.getInstance({ username: 'myUsername' });
@ -129,8 +144,8 @@ await cursor.forEach(async (user, index) => {

 // Chain cursor methods like in the MongoDB native driver
 const paginatedCursor = await User.getCursor({ active: true })
-  .limit(10)  // Limit results
-  .skip(20)   // Skip first 20 results
+  .limit(10) // Limit results
+  .skip(20) // Skip first 20 results
  .sort({ createdAt: -1 }); // Sort by creation date descending

 // Convert cursor to array (when you know the result set is small)
@ -146,30 +161,88 @@ await cursor.close();
 ```

 #### Update
+
 ```typescript
 // Assuming 'user' is an instance of User
 user.email = 'newEmail@example.com';
-await user.save();  // Update the user in the database
+await user.save(); // Update the user in the database

 // Upsert operations (insert if not exists, update if exists)
 const upsertedUser = await User.upsert(
-  { id: 'user-123' },  // Query to find the user
-  {                    // Fields to update or insert
+  { id: 'user-123' }, // Query to find the user
+  {
+    // Fields to update or insert
    username: 'newUsername',
-    email: 'newEmail@example.com'
-  }
+    email: 'newEmail@example.com',
+  },
 );
 ```

 #### Delete
+
 ```typescript
 // Assuming 'user' is an instance of User
-await user.delete();  // Delete the user from the database
+await user.delete(); // Delete the user from the database
 ```

 ## Advanced Features

+### Search Functionality
+
+SmartData provides powerful, Lucene‑style search capabilities with robust fallback mechanisms:
+
+```typescript
+// Define a model with searchable fields
+@Collection(() => db)
+class Product extends SmartDataDbDoc<Product, Product> {
+  @unI() public id: string = 'product-id';
+  @svDb() @searchable() public name: string;
+  @svDb() @searchable() public description: string;
+  @svDb() @searchable() public category: string;
+  @svDb() public price: number;
+}
+
+// List searchable fields
+const searchableFields = Product.getSearchableFields();
+
+// 1: Exact phrase across all fields
+await Product.search('"Kindle Paperwhite"');
+
+// 2: Wildcard search across all fields
+await Product.search('Air*');
+
+// 3: Field‑scoped wildcard
+await Product.search('name:Air*');
+
+// 4: Boolean AND/OR/NOT
+await Product.search('category:Electronics AND name:iPhone');
+
+// 5: Grouping with parentheses
+await Product.search('(Furniture OR Electronics) AND Chair');
+
+// 6: Multi‑term unquoted (terms AND’d across fields)
+await Product.search('TypeScript Aufgabe');
+
+// 7: Empty query returns all documents
+await Product.search('');
+```
+
+The search functionality includes:
+
+- `@searchable()` decorator for marking fields as searchable
+- `Class.getSearchableFields()` static method to list searchable fields for a model
+- `search(query: string)` method supporting:
+  - Exact phrase matches (`"my exact string"` or `'my exact string'`)
+  - Field‑scoped exact & wildcard searches (`field:value`, `field:Air*`)
+  - Wildcard searches across all fields (`Air*`, `?Pods`)
+  - Boolean operators (`AND`, `OR`, `NOT`) with grouping (`(...)`)
+  - Multi‑term unquoted queries AND’d across fields (`TypeScript Aufgabe`)
+  - Single/multi‑term regex searches across fields
+  - Empty queries returning all documents
+- Automatic escaping & wildcard conversion to prevent regex injection
+
 ### EasyStore
+
 EasyStore provides a simple key-value storage system with automatic persistence:

 ```typescript
@ -200,6 +273,7 @@ await store.deleteKey('apiKey');
 ```

 ### Distributed Coordination
+
 Built-in support for distributed systems with leader election:

 ```typescript
@ -237,21 +311,25 @@ await coordinator.stop();
 ```

 ### Real-time Data Watching
+
 Watch for changes in your collections with RxJS integration using MongoDB Change Streams:

 ```typescript
 // Create a watcher for a specific collection with a query filter
-const watcher = await User.watch({
-  active: true  // Only watch for changes to active users
-}, {
-  fullDocument: true,    // Include the full document in change notifications
-  bufferTimeMs: 100      // Buffer changes for 100ms to reduce notification frequency
-});
+const watcher = await User.watch(
+  {
+    active: true, // Only watch for changes to active users
+  },
+  {
+    fullDocument: true, // Include the full document in change notifications
+    bufferTimeMs: 100, // Buffer changes for 100ms to reduce notification frequency
+  },
+);

 // Subscribe to changes using RxJS
 watcher.changeSubject.subscribe((change) => {
  console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
-  console.log('Document changed:', change.docInstance);   // The full document instance
+  console.log('Document changed:', change.docInstance); // The full document instance

  // Handle different types of changes
  if (change.operationType === 'insert') {
@ -273,6 +351,7 @@ await watcher.stop();
 ```

 ### Managed Collections
+
 For more complex data models that require additional context:

 ```typescript
@ -295,6 +374,7 @@ class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
 ```

 ### Automatic Indexing
+
 Define indexes directly in your model class:

 ```typescript
@ -324,6 +404,7 @@ class Product extends SmartDataDbDoc<Product, Product> {
 ```

 ### Transaction Support
+
 Use MongoDB transactions for atomic operations:

 ```typescript
@ -344,6 +425,7 @@ try {
 ```

 ### Deep Object Queries
+
 SmartData provides fully type-safe deep property queries with the `DeepQuery` type:

 ```typescript
@ -360,14 +442,14 @@ class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
      address: {
        city: string;
        country: string;
-      }
-    }
+      };
+    };
  };
 }

 // Type-safe string literals for dot notation
 const usersInUSA = await UserProfile.getInstances({
-  'user.details.address.country': 'USA'
+  'user.details.address.country': 'USA',
 });

 // Fully typed deep queries with the DeepQuery type
@ -376,7 +458,7 @@ import { DeepQuery } from '@push.rocks/smartdata';
 const typedQuery: DeepQuery<UserProfile> = {
  id: 'profile-id',
  'user.details.firstName': 'John',
-  'user.details.address.country': 'USA'
+  'user.details.address.country': 'USA',
 };

 // TypeScript will error if paths are incorrect
@ -385,13 +467,14 @@ const results = await UserProfile.getInstances(typedQuery);
 // MongoDB query operators are supported
 const operatorQuery: DeepQuery<UserProfile> = {
  'user.details.address.country': 'USA',
-  'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
+  'user.details.address.city': { $in: ['New York', 'Los Angeles'] },
 };

 const filteredResults = await UserProfile.getInstances(operatorQuery);
 ```

 ### Document Lifecycle Hooks
+
 Implement custom logic at different stages of a document's lifecycle:

 ```typescript
@ -437,26 +520,39 @@ class Order extends SmartDataDbDoc<Order, Order> {
 ## Best Practices

 ### Connection Management
+
 - Always call `db.init()` before using any database features
- Use `db.disconnect()` when shutting down your application
+- Use `db.close()` when shutting down your application
 - Set appropriate connection pool sizes based on your application's needs

 ### Document Design
- Use appropriate decorators (`@svDb`, `@unI`, `@index`) to optimize database operations
+
+- Use appropriate decorators (`@svDb`, `@unI`, `@index`, `@searchable`) to optimize database operations
 - Implement type-safe models by properly extending `SmartDataDbDoc`
 - Consider using interfaces to define document structures separately from implementation
+- Mark fields that need to be searched with the `@searchable()` decorator
+
+### Search Optimization
+
+- (Optional) Create MongoDB text indexes on searchable fields to speed up full-text search
+- Use `search(query)` for all search operations (field:value, partial matches, multi-word)
+- Prefer field-specific exact matches when possible for optimal performance
+- Avoid unnecessary complexity in query strings to keep regex searches efficient

 ### Performance Optimization
+
 - Use cursors for large datasets instead of loading all documents into memory
 - Create appropriate indexes for frequent query patterns
 - Use projections to limit the fields returned when you don't need the entire document

 ### Distributed Systems
+
 - Implement proper error handling for leader election events
 - Ensure all instances have synchronized clocks when using time-based coordination
 - Use the distributed coordinator's task management features for coordinated operations

 ### Type Safety
+
 - Take advantage of the `DeepQuery<T>` type for fully type-safe queries
 - Define proper types for your document models to enhance IDE auto-completion
 - Use generic type parameters to specify exact document types when working with collections
@ -475,7 +571,7 @@ Please make sure to update tests as appropriate and follow our coding standards.

 ## License and Legal Information

-This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository. 
+This repository is licensed under the MIT License. For details, see [MIT License](https://opensource.org/licenses/MIT).

 **Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

--- a/test/test.distributedcoordinator.ts
+++ b/test/test.distributedcoordinator.ts
@ -3,7 +3,10 @@ import * as smartmongo from '@push.rocks/smartmongo';
 import type * as taskbuffer from '@push.rocks/taskbuffer';

 import * as smartdata from '../ts/index.js';
-import { SmartdataDistributedCoordinator, DistributedClass } from '../ts/classes.distributedcoordinator.js'; // path might need adjusting
+import {
+  SmartdataDistributedCoordinator,
+  DistributedClass,
+} from '../ts/classes.distributedcoordinator.js'; // path might need adjusting
 const totalInstances = 10;

 // =======================================
@ -20,93 +23,100 @@ tap.test('should create a testinstance as database', async () => {
 });

 tap.test('should instantiate DistributedClass', async (tools) => {
-    const instance = new DistributedClass();
-    expect(instance).toBeInstanceOf(DistributedClass);
+  const instance = new DistributedClass();
+  expect(instance).toBeInstanceOf(DistributedClass);
 });

 tap.test('DistributedClass should update the time', async (tools) => {
-    const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
-    await distributedCoordinator.start();
-    const initialTime = distributedCoordinator.ownInstance.data.lastUpdated;
-    await distributedCoordinator.sendHeartbeat();
-    const updatedTime = distributedCoordinator.ownInstance.data.lastUpdated;
-    expect(updatedTime).toBeGreaterThan(initialTime);
-    await distributedCoordinator.stop();
+  const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
+  await distributedCoordinator.start();
+  const initialTime = distributedCoordinator.ownInstance.data.lastUpdated;
+  await distributedCoordinator.sendHeartbeat();
+  const updatedTime = distributedCoordinator.ownInstance.data.lastUpdated;
+  expect(updatedTime).toBeGreaterThan(initialTime);
+  await distributedCoordinator.stop();
 });

 tap.test('should instantiate SmartdataDistributedCoordinator', async (tools) => {
-    const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
-    await distributedCoordinator.start();
-    expect(distributedCoordinator).toBeInstanceOf(SmartdataDistributedCoordinator);
-    await distributedCoordinator.stop();
+  const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
+  await distributedCoordinator.start();
+  expect(distributedCoordinator).toBeInstanceOf(SmartdataDistributedCoordinator);
+  await distributedCoordinator.stop();
 });

 tap.test('SmartdataDistributedCoordinator should update leader status', async (tools) => {
-    const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
-    await distributedCoordinator.start();
-    await distributedCoordinator.checkAndMaybeLead();
-    expect(distributedCoordinator.ownInstance.data.elected).toBeOneOf([true, false]);
-    await distributedCoordinator.stop();
+  const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
+  await distributedCoordinator.start();
+  await distributedCoordinator.checkAndMaybeLead();
+  expect(distributedCoordinator.ownInstance.data.elected).toBeOneOf([true, false]);
+  await distributedCoordinator.stop();
 });

-tap.test('SmartdataDistributedCoordinator should handle distributed task requests', async (tools) => {
+tap.test(
+  'SmartdataDistributedCoordinator should handle distributed task requests',
+  async (tools) => {
    const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);
    await distributedCoordinator.start();

    const mockTaskRequest: taskbuffer.distributedCoordination.IDistributedTaskRequest = {
-        submitterId: "mockSubmitter12345",    // Some unique mock submitter ID
-        requestResponseId: 'uni879873462hjhfkjhsdf', // Some unique ID for the request-response
-        taskName: "SampleTask",
-        taskVersion: "1.0.0",                  // Assuming it's a version string
-        taskExecutionTime: Date.now(),
-        taskExecutionTimeout: 60000,           // Let's say the timeout is 1 minute (60000 ms)
-        taskExecutionParallel: 5,              // Let's assume max 5 parallel executions
-        status: 'requesting'
+      submitterId: 'mockSubmitter12345', // Some unique mock submitter ID
+      requestResponseId: 'uni879873462hjhfkjhsdf', // Some unique ID for the request-response
+      taskName: 'SampleTask',
+      taskVersion: '1.0.0', // Assuming it's a version string
+      taskExecutionTime: Date.now(),
+      taskExecutionTimeout: 60000, // Let's say the timeout is 1 minute (60000 ms)
+      taskExecutionParallel: 5, // Let's assume max 5 parallel executions
+      status: 'requesting',
    };

    const response = await distributedCoordinator.fireDistributedTaskRequest(mockTaskRequest);
-    console.log(response)  // based on your expected structure for the response
+    console.log(response); // based on your expected structure for the response
    await distributedCoordinator.stop();
-});
+  },
+);

-tap.test('SmartdataDistributedCoordinator should update distributed task requests', async (tools) => {
+tap.test(
+  'SmartdataDistributedCoordinator should update distributed task requests',
+  async (tools) => {
    const distributedCoordinator = new SmartdataDistributedCoordinator(testDb);

    await distributedCoordinator.start();

    const mockTaskRequest: taskbuffer.distributedCoordination.IDistributedTaskRequest = {
-        submitterId: "mockSubmitter12345",    // Some unique mock submitter ID
-        requestResponseId: 'uni879873462hjhfkjhsdf', // Some unique ID for the request-response
-        taskName: "SampleTask",
-        taskVersion: "1.0.0",                  // Assuming it's a version string
-        taskExecutionTime: Date.now(),
-        taskExecutionTimeout: 60000,           // Let's say the timeout is 1 minute (60000 ms)
-        taskExecutionParallel: 5,              // Let's assume max 5 parallel executions
-        status: 'requesting'
+      submitterId: 'mockSubmitter12345', // Some unique mock submitter ID
+      requestResponseId: 'uni879873462hjhfkjhsdf', // Some unique ID for the request-response
+      taskName: 'SampleTask',
+      taskVersion: '1.0.0', // Assuming it's a version string
+      taskExecutionTime: Date.now(),
+      taskExecutionTimeout: 60000, // Let's say the timeout is 1 minute (60000 ms)
+      taskExecutionParallel: 5, // Let's assume max 5 parallel executions
+      status: 'requesting',
    };

-
    await distributedCoordinator.updateDistributedTaskRequest(mockTaskRequest);
    // Here, we can potentially check if a DB entry got updated or some other side-effect of the update method.
    await distributedCoordinator.stop();
-});
+  },
+);

 tap.test('should elect only one leader amongst multiple instances', async (tools) => {
-    const coordinators = Array.from({ length: totalInstances }).map(() => new SmartdataDistributedCoordinator(testDb));
-    await Promise.all(coordinators.map(coordinator => coordinator.start()));
-    const leaders = coordinators.filter(coordinator => coordinator.ownInstance.data.elected);
-    for (const leader of leaders) {
-        console.log(leader.ownInstance);
-    }
-    expect(leaders.length).toEqual(1);
+  const coordinators = Array.from({ length: totalInstances }).map(
+    () => new SmartdataDistributedCoordinator(testDb),
+  );
+  await Promise.all(coordinators.map((coordinator) => coordinator.start()));
+  const leaders = coordinators.filter((coordinator) => coordinator.ownInstance.data.elected);
+  for (const leader of leaders) {
+    console.log(leader.ownInstance);
+  }
+  expect(leaders.length).toEqual(1);

-    // stopping clears a coordinator from being elected.
-    await Promise.all(coordinators.map(coordinator => coordinator.stop()));
+  // stopping clears a coordinator from being elected.
+  await Promise.all(coordinators.map((coordinator) => coordinator.stop()));
 });

 tap.test('should clean up', async () => {
-    await smartmongoInstance.stopAndDumpToDir(`.nogit/dbdump/test.distributedcoordinator.ts`);
-    setTimeout(() => process.exit(), 2000);
-})
+  await smartmongoInstance.stopAndDumpToDir(`.nogit/dbdump/test.distributedcoordinator.ts`);
+  setTimeout(() => process.exit(), 2000);
+});

 tap.start({ throwOnError: true });
--- a/test/test.search.advanced.ts
+++ b/test/test.search.advanced.ts
@ -0,0 +1,187 @@
+import { tap, expect } from '@push.rocks/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import * as smartdata from '../ts/index.js';
+import { searchable } from '../ts/classes.doc.js';
+import { smartunique } from '../ts/plugins.js';
+
+// Set up database connection
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define a test class for advanced search scenarios
+@smartdata.Collection(() => testDb)
+class Product extends smartdata.SmartDataDbDoc<Product, Product> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  @searchable()
+  public name: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public description: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public category: string;
+
+  @smartdata.svDb()
+  public price: number;
+
+  constructor(
+    nameArg: string,
+    descriptionArg: string,
+    categoryArg: string,
+    priceArg: number,
+  ) {
+    super();
+    this.name = nameArg;
+    this.description = descriptionArg;
+    this.category = categoryArg;
+    this.price = priceArg;
+  }
+}
+
+// Initialize DB and insert sample products
+tap.test('setup advanced search database', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(
+    await smartmongoInstance.getMongoDescriptor(),
+  );
+  await testDb.init();
+});
+
+tap.test('insert products for advanced search', async () => {
+  const products = [
+    new Product(
+      'Night Owl Lamp',
+      'Bright lamp for night reading',
+      'Lighting',
+      29,
+    ),
+    new Product(
+      'Day Light Lamp',
+      'Daytime lamp with adjustable brightness',
+      'Lighting',
+      39,
+    ),
+    new Product(
+      'Office Chair',
+      'Ergonomic chair for office',
+      'Furniture',
+      199,
+    ),
+    new Product(
+      'Gaming Chair',
+      'Comfortable for long gaming sessions',
+      'Furniture',
+      299,
+    ),
+    new Product(
+      'iPhone 12',
+      'Latest iPhone with A14 Bionic chip',
+      'Electronics',
+      999,
+    ),
+    new Product(
+      'AirPods',
+      'Wireless earbuds with noise cancellation',
+      'Electronics',
+      249,
+    ),
+  ];
+  for (const p of products) {
+    await p.save();
+  }
+  const all = await Product.getInstances({});
+  expect(all.length).toEqual(products.length);
+});
+
+// Simple exact field:value matching
+tap.test('simpleExact: category:Furniture returns chairs', async () => {
+  const res = await Product.search('category:Furniture');
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// simpleExact invalid field should throw
+tap.test('simpleExact invalid field errors', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:29');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+// Quoted phrase search
+tap.test('quoted phrase "Bright lamp" matches Night Owl Lamp', async () => {
+  const res = await Product.search('"Bright lamp"');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+tap.test("quoted phrase 'night reading' matches Night Owl Lamp", async () => {
+  const res = await Product.search("'night reading'");
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+
+tap.test('wildcard description:*gaming* matches Gaming Chair', async () => {
+  const res = await Product.search('description:*gaming*');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Gaming Chair');
+});
+
+// Boolean AND and OR
+tap.test('boolean AND: category:Lighting AND lamp', async () => {
+  const res = await Product.search('category:Lighting AND lamp');
+  expect(res.length).toEqual(2);
+});
+
+tap.test('boolean OR: Furniture OR Electronics', async () => {
+  const res = await Product.search('Furniture OR Electronics');
+  expect(res.length).toEqual(4);
+});
+
+// Multi-term unquoted -> AND across terms
+tap.test('multi-term unquoted adjustable brightness', async () => {
+  const res = await Product.search('adjustable brightness');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Day Light Lamp');
+});
+
+tap.test('multi-term unquoted Night Lamp', async () => {
+  const res = await Product.search('Night Lamp');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+// Grouping with parentheses
+tap.test('grouping: (Furniture OR Electronics) AND Chair', async () => {
+  const res = await Product.search(
+    '(Furniture OR Electronics) AND Chair',
+  );
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// Teardown
+tap.test('cleanup advanced search database', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir(
+      `.nogit/dbdump/test.search.advanced.ts`,
+    );
+  }
+  setTimeout(() => process.exit(), 2000);
+});
+
+tap.start({ throwOnError: true });
--- a/test/test.search.ts
+++ b/test/test.search.ts
@ -0,0 +1,269 @@
+import { tap, expect } from '@push.rocks/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import { smartunique } from '../ts/plugins.js';
+
+// Import the smartdata library
+import * as smartdata from '../ts/index.js';
+import { searchable } from '../ts/classes.doc.js';
+
+// Set up database connection
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define a test class with searchable fields using the standard SmartDataDbDoc
+@smartdata.Collection(() => testDb)
+class Product extends smartdata.SmartDataDbDoc<Product, Product> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  @searchable()
+  public name: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public description: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public category: string;
+
+  @smartdata.svDb()
+  public price: number;
+
+  constructor(nameArg: string, descriptionArg: string, categoryArg: string, priceArg: number) {
+    super();
+    this.name = nameArg;
+    this.description = descriptionArg;
+    this.category = categoryArg;
+    this.price = priceArg;
+  }
+}
+
+tap.test('should create a test database instance', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(await smartmongoInstance.getMongoDescriptor());
+  await testDb.init();
+});
+
+tap.test('should create test products with searchable fields', async () => {
+  // Create several products with different fields to search
+  const products = [
+    new Product('iPhone 12', 'Latest iPhone with A14 Bionic chip', 'Electronics', 999),
+    new Product('MacBook Pro', 'Powerful laptop for professionals', 'Electronics', 1999),
+    new Product('AirPods', 'Wireless earbuds with noise cancellation', 'Electronics', 249),
+    new Product('Galaxy S21', 'Samsung flagship phone with great camera', 'Electronics', 899),
+    new Product('Kindle Paperwhite', 'E-reader with built-in light', 'Books', 129),
+    new Product('Harry Potter', 'Fantasy book series about wizards', 'Books', 49),
+    new Product('Coffee Maker', 'Automatic drip coffee machine', 'Kitchen', 89),
+    new Product('Blender', 'High-speed blender for smoothies', 'Kitchen', 129),
+  ];
+
+  // Save all products to the database
+  for (const product of products) {
+    await product.save();
+  }
+
+  // Verify that we can get all products
+  const allProducts = await Product.getInstances({});
+  expect(allProducts.length).toEqual(products.length);
+  console.log(`Successfully created and saved ${allProducts.length} products`);
+});
+
+tap.test('should retrieve searchable fields for a class', async () => {
+  // Use the getSearchableFields function to verify our searchable fields
+  const searchableFields = Product.getSearchableFields();
+  console.log('Searchable fields:', searchableFields);
+
+  expect(searchableFields.length).toEqual(3);
+  expect(searchableFields).toContain('name');
+  expect(searchableFields).toContain('description');
+  expect(searchableFields).toContain('category');
+});
+
+tap.test('should search products by exact field match', async () => {
+  // Basic field exact match search
+  const electronicsProducts = await Product.getInstances({ category: 'Electronics' });
+  console.log(`Found ${electronicsProducts.length} products in Electronics category`);
+
+  expect(electronicsProducts.length).toEqual(4);
+});
+
+tap.test('should search products by basic search method', async () => {
+  // Using the basic search method with simple Lucene query
+  try {
+    const iPhoneResults = await Product.search('iPhone');
+    console.log(`Found ${iPhoneResults.length} products matching 'iPhone' using basic search`);
+
+    expect(iPhoneResults.length).toEqual(1);
+    expect(iPhoneResults[0].name).toEqual('iPhone 12');
+  } catch (error) {
+    console.error('Basic search error:', error.message);
+    // If basic search fails, we'll demonstrate the enhanced approach in later tests
+    console.log('Will test with enhanced searchWithLucene method next');
+  }
+});
+
+tap.test('should search products with search method', async () => {
+  // Using the robust searchWithLucene method
+    const wirelessResults = await Product.search('wireless');
+    console.log(
+      `Found ${wirelessResults.length} products matching 'wireless' using search`,
+    );
+
+  expect(wirelessResults.length).toEqual(1);
+  expect(wirelessResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should search products by category with search', async () => {
+  // Using field-specific search with searchWithLucene
+  const kitchenResults = await Product.search('category:Kitchen');
+  console.log(`Found ${kitchenResults.length} products in Kitchen category using search`);
+
+  expect(kitchenResults.length).toEqual(2);
+  expect(kitchenResults[0].category).toEqual('Kitchen');
+  expect(kitchenResults[1].category).toEqual('Kitchen');
+});
+
+tap.test('should search products with partial word matches', async () => {
+  // Testing partial word matches
+  const proResults = await Product.search('Pro');
+  console.log(`Found ${proResults.length} products matching 'Pro'`);
+
+  // Should match both "MacBook Pro" and "professionals" in description
+  expect(proResults.length).toBeGreaterThan(0);
+});
+
+tap.test('should search across multiple searchable fields', async () => {
+  // Test searching across all searchable fields
+  const bookResults = await Product.search('book');
+  console.log(`Found ${bookResults.length} products matching 'book' across all fields`);
+
+  // Should match "MacBook" in name and "Books" in category
+  expect(bookResults.length).toBeGreaterThan(1);
+});
+
+tap.test('should handle case insensitive searches', async () => {
+  // Test case insensitivity
+  const electronicsResults = await Product.search('electronics');
+  const ElectronicsResults = await Product.search('Electronics');
+
+  console.log(`Found ${electronicsResults.length} products matching lowercase 'electronics'`);
+  console.log(`Found ${ElectronicsResults.length} products matching capitalized 'Electronics'`);
+
+  // Both searches should return the same results
+  expect(electronicsResults.length).toEqual(ElectronicsResults.length);
+});
+
+tap.test('should demonstrate search fallback mechanisms', async () => {
+  console.log('\n====== FALLBACK MECHANISM DEMONSTRATION ======');
+  console.log('If MongoDB query fails, searchWithLucene will:');
+  console.log('1. Try using basic MongoDB filters');
+  console.log('2. Fall back to field-specific searches');
+  console.log('3. As last resort, perform in-memory filtering');
+  console.log('This ensures robust search even with complex queries');
+  console.log('==============================================\n');
+
+  // Use a simpler term that should be found in descriptions
+  // Avoid using "OR" operator which requires a text index
+  const results = await Product.search('high');
+  console.log(`Found ${results.length} products matching 'high'`);
+
+  // "High-speed blender" contains "high"
+  expect(results.length).toBeGreaterThan(0);
+
+  // Try another fallback example that won't need $text
+  const powerfulResults = await Product.search('powerful');
+  console.log(`Found ${powerfulResults.length} products matching 'powerful'`);
+
+  // "Powerful laptop for professionals" contains "powerful"
+  expect(powerfulResults.length).toBeGreaterThan(0);
+});
+
+tap.test('should explain the advantages of the integrated approach', async () => {
+  console.log('\n====== INTEGRATED SEARCH APPROACH BENEFITS ======');
+  console.log('1. No separate class hierarchy - keeps code simple');
+  console.log('2. Enhanced convertFilterForMongoDb handles MongoDB operators');
+  console.log('3. Robust fallback mechanisms ensure searches always work');
+  console.log('4. searchWithLucene provides powerful search capabilities');
+  console.log('5. Backwards compatible with existing code');
+  console.log('================================================\n');
+
+  expect(true).toEqual(true);
+});
+
+// Additional robustness tests
+tap.test('should search exact name using field:value', async () => {
+  const nameResults = await Product.search('name:AirPods');
+  expect(nameResults.length).toEqual(1);
+  expect(nameResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should throw when searching non-searchable field', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:129');
+  } catch (err) {
+    error = err as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+tap.test('empty query should return all products', async () => {
+  const allResults = await Product.search('');
+  expect(allResults.length).toEqual(8);
+});
+
+tap.test('should search multi-word term across fields', async () => {
+  const termResults = await Product.search('iPhone 12');
+  expect(termResults.length).toEqual(1);
+  expect(termResults[0].name).toEqual('iPhone 12');
+});
+
+// Additional search scenarios
+tap.test('should return zero results for non-existent terms', async () => {
+  const noResults = await Product.search('NonexistentTerm');
+  expect(noResults.length).toEqual(0);
+});
+
+tap.test('should search products by description term "noise"', async () => {
+  const noiseResults = await Product.search('noise');
+  expect(noiseResults.length).toEqual(1);
+  expect(noiseResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should search products by description term "flagship"', async () => {
+  const flagshipResults = await Product.search('flagship');
+  expect(flagshipResults.length).toEqual(1);
+  expect(flagshipResults[0].name).toEqual('Galaxy S21');
+});
+
+tap.test('should search numeric strings "12"', async () => {
+  const twelveResults = await Product.search('12');
+  expect(twelveResults.length).toEqual(1);
+  expect(twelveResults[0].name).toEqual('iPhone 12');
+});
+
+tap.test('should search hyphenated terms "high-speed"', async () => {
+  const hyphenResults = await Product.search('high-speed');
+  expect(hyphenResults.length).toEqual(1);
+  expect(hyphenResults[0].name).toEqual('Blender');
+});
+
+tap.test('should search hyphenated terms "E-reader"', async () => {
+  const ereaderResults = await Product.search('E-reader');
+  expect(ereaderResults.length).toEqual(1);
+  expect(ereaderResults[0].name).toEqual('Kindle Paperwhite');
+});
+
+tap.test('close database connection', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir(`.nogit/dbdump/test.search.ts`);
+  }
+  setTimeout(() => process.exit(), 2000);
+});
+
+tap.start({ throwOnError: true });
--- a/test/test.ts
+++ b/test/test.ts
@ -97,7 +97,7 @@ tap.test('should save the car to the db', async (toolsArg) => {
      console.log(
        `Filled database with ${counter} of ${totalCars} Cars and memory usage ${
          process.memoryUsage().rss / 1e6
-        } MB`
+        } MB`,
      );
    }
  } while (counter < totalCars);
@ -116,7 +116,7 @@ tap.test('expect to get instance of Car with shallow match', async () => {
      console.log(
        `performed ${counter} of ${totalQueryCycles} total query cycles: took ${
          Date.now() - timeStart
-        }ms to query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`
+        }ms to query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`,
      );
    }
    expect(myCars[0].deepData.sodeep).toEqual('yes');
@ -139,7 +139,7 @@ tap.test('expect to get instance of Car with deep match', async () => {
      console.log(
        `performed ${counter} of ${totalQueryCycles} total query cycles: took ${
          Date.now() - timeStart
-        }ms to deep query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`
+        }ms to deep query a set of 2000 with memory footprint ${process.memoryUsage().rss / 1e6} MB`,
      );
    }
    expect(myCars2[0].deepData.sodeep).toEqual('yes');
@ -209,7 +209,7 @@ tap.test('should store a new Truck', async () => {
 tap.test('should return a count', async () => {
  const truckCount = await Truck.getCount();
  expect(truckCount).toEqual(1);
-})
+});

 tap.test('should use a cursor', async () => {
  const cursor = await Car.getCursor({});
--- a/test/test.watch.ts
+++ b/test/test.watch.ts
@ -64,7 +64,11 @@ tap.test('should watch a collection', async (toolsArg) => {
 // close the database connection
 // =======================================
 tap.test('close', async () => {
-  await testDb.mongoDb.dropDatabase();
+  try {
+    await testDb.mongoDb.dropDatabase();
+  } catch (err) {
+    console.warn('dropDatabase error ignored in cleanup:', err.message || err);
+  }
  await testDb.close();
  if (smartmongoInstance) {
    await smartmongoInstance.stop();
--- a/ts/00_commitinfo_data.ts
+++ b/ts/00_commitinfo_data.ts
@ -3,6 +3,6 @@
 */
 export const commitinfo = {
  name: '@push.rocks/smartdata',
-  version: '5.4.0',
+  version: '5.11.2',
  description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }
--- a/ts/classes.collection.ts
+++ b/ts/classes.collection.ts
@ -1,7 +1,7 @@
 import * as plugins from './plugins.js';
 import { SmartdataDb } from './classes.db.js';
 import { SmartdataDbCursor } from './classes.cursor.js';
-import { SmartDataDbDoc } from './classes.doc.js';
+import { SmartDataDbDoc, type IIndexOptions } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { CollectionFactory } from './classes.collectionfactory.js';

@ -32,13 +32,22 @@ export function Collection(dbArg: SmartdataDb | TDelayed<SmartdataDb>) {
        if (!(dbArg instanceof SmartdataDb)) {
          dbArg = dbArg();
        }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        const coll = collectionFactory.getCollection(constructor.name, dbArg);
+        // Attach document constructor for searchableFields lookup
+        if (!(coll as any).docCtor) {
+          (coll as any).docCtor = decoratedClass;
+        }
+        return coll;
      }
      public get collection() {
        if (!(dbArg instanceof SmartdataDb)) {
          dbArg = dbArg();
        }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        const coll = collectionFactory.getCollection(constructor.name, dbArg);
+        if (!(coll as any).docCtor) {
+          (coll as any).docCtor = decoratedClass;
+        }
+        return coll;
      }
    };
    return decoratedClass;
@ -49,7 +58,7 @@ export interface IManager {
  db: SmartdataDb;
 }

-export const setDefaultManagerForDoc = <T>(managerArg: IManager, dbDocArg: T): T => {
+export const setDefaultManagerForDoc = <T,>(managerArg: IManager, dbDocArg: T): T => {
  (dbDocArg as any).prototype.defaultManager = managerArg;
  return dbDocArg;
 };
@ -127,6 +136,9 @@ export class SmartdataCollection<T> {
  public collectionName: string;
  public smartdataDb: SmartdataDb;
  public uniqueIndexes: string[] = [];
+  public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+  // flag to ensure text index is created only once
+  private textIndexCreated: boolean = false;

  constructor(classNameArg: string, smartDataDbArg: SmartdataDb) {
    // tell the collection where it belongs
@ -152,6 +164,18 @@ export class SmartdataCollection<T> {
        console.log(`Successfully initiated Collection ${this.collectionName}`);
      }
      this.mongoDbCollection = this.smartdataDb.mongoDb.collection(this.collectionName);
+      // Auto-create a compound text index on all searchable fields
+      // Use document constructor's searchableFields registered via decorator
+      const docCtor = (this as any).docCtor;
+      const searchableFields: string[] = docCtor?.searchableFields || [];
+      if (searchableFields.length > 0 && !this.textIndexCreated) {
+        // Build a compound text index spec
+        const indexSpec: Record<string, 'text'> = {};
+        searchableFields.forEach(f => { indexSpec[f] = 'text'; });
+        // Cast to any to satisfy TypeScript IndexSpecification typing
+        await this.mongoDbCollection.createIndex(indexSpec as any, { name: 'smartdata_text_index' });
+        this.textIndexCreated = true;
+      }
    }
  }

@ -170,6 +194,24 @@ export class SmartdataCollection<T> {
    }
  }

+  /**
+   * creates regular indexes for the collection
+   */
+  public createRegularIndexes(indexesArg: Array<{field: string, options: IIndexOptions}> = []) {
+    for (const indexDef of indexesArg) {
+      // Check if we've already created this index
+      const indexKey = indexDef.field;
+      if (!this.regularIndexes.some(i => i.field === indexKey)) {
+        this.mongoDbCollection.createIndex(
+          { [indexDef.field]: 1 }, // Simple single-field index
+          indexDef.options
+        );
+        // Track that we've created this index
+        this.regularIndexes.push(indexDef);
+      }
+    }
+  }
+
  /**
   * adds a validation function that all newly inserted and updated objects have to pass
   */
@ -190,7 +232,7 @@ export class SmartdataCollection<T> {

  public async getCursor(
    filterObjectArg: any,
-    dbDocArg: typeof SmartDataDbDoc
+    dbDocArg: typeof SmartDataDbDoc,
  ): Promise<SmartdataDbCursor<any>> {
    await this.init();
    const cursor = this.mongoDbCollection.find(filterObjectArg);
@ -213,7 +255,7 @@ export class SmartdataCollection<T> {
   */
  public async watch(
    filterObject: any,
-    smartdataDbDocArg: typeof SmartDataDbDoc
+    smartdataDbDocArg: typeof SmartDataDbDoc,
  ): Promise<SmartdataDbWatcher> {
    await this.init();
    const changeStream = this.mongoDbCollection.watch(
@ -224,7 +266,7 @@ export class SmartdataCollection<T> {
      ],
      {
        fullDocument: 'updateLookup',
-      }
+      },
    );
    const smartdataWatcher = new SmartdataDbWatcher(changeStream, smartdataDbDocArg);
    await smartdataWatcher.readyDeferred.promise;
@ -238,6 +280,12 @@ export class SmartdataCollection<T> {
    await this.init();
    await this.checkDoc(dbDocArg);
    this.markUniqueIndexes(dbDocArg.uniqueIndexes);
+    
+    // Create regular indexes if available
+    if (dbDocArg.regularIndexes && dbDocArg.regularIndexes.length > 0) {
+      this.createRegularIndexes(dbDocArg.regularIndexes);
+    }
+    
    const saveableObject = await dbDocArg.createSavableObject();
    const result = await this.mongoDbCollection.insertOne(saveableObject);
    return result;
@ -261,7 +309,7 @@ export class SmartdataCollection<T> {
    const result = await this.mongoDbCollection.updateOne(
      identifiableObject,
      { $set: updateableObject },
-      { upsert: true }
+      { upsert: true },
    );
    return result;
  }
--- a/ts/classes.cursor.ts
+++ b/ts/classes.cursor.ts
@ -15,14 +15,14 @@ export class SmartdataDbCursor<T = any> {
    this.smartdataDbDoc = dbDocArg;
  }

-  public async next(closeAtEnd = true) {
+  public async next(closeAtEnd = true): Promise<T> {
    const result = this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(
-      await this.mongodbCursor.next()
+      await this.mongodbCursor.next(),
    );
    if (!result && closeAtEnd) {
      await this.close();
    }
-    return result;
+    return result as T;
  }

  public async forEach(forEachFuncArg: (itemArg: T) => Promise<any>, closeCursorAtEnd = true) {
@ -40,6 +40,11 @@ export class SmartdataDbCursor<T = any> {
    }
  }

+  public async toArray(): Promise<T[]> {
+    const result = await this.mongodbCursor.toArray();
+    return result.map((itemArg) => this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(itemArg)) as T[];
+  }
+
  public async close() {
    await this.mongodbCursor.close();
  }
--- a/ts/classes.distributedcoordinator.ts
+++ b/ts/classes.distributedcoordinator.ts
@ -139,7 +139,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
      const eligibleLeader = leaders.find(
        (leader) =>
          leader.data.lastUpdated >=
-          Date.now() - plugins.smarttime.getMilliSecondsFromUnits({ seconds: 20 })
+          Date.now() - plugins.smarttime.getMilliSecondsFromUnits({ seconds: 20 }),
      );
      return eligibleLeader;
    });
@ -178,16 +178,14 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
        console.log('bidding code stored.');
      });
      console.log(`bidding for leadership...`);
-      await plugins.smartdelay.delayFor(
-        plugins.smarttime.getMilliSecondsFromUnits({ seconds: 5 })
-      );
+      await plugins.smartdelay.delayFor(plugins.smarttime.getMilliSecondsFromUnits({ seconds: 5 }));
      await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
        let biddingInstances = await DistributedClass.getInstances({});
        biddingInstances = biddingInstances.filter(
          (instanceArg) =>
            instanceArg.data.status === 'bidding' &&
            instanceArg.data.lastUpdated >=
-              Date.now() - plugins.smarttime.getMilliSecondsFromUnits({ seconds: 10 })
+              Date.now() - plugins.smarttime.getMilliSecondsFromUnits({ seconds: 10 }),
        );
        console.log(`found ${biddingInstances.length} bidding instances...`);
        this.ownInstance.data.elected = true;
@ -242,7 +240,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
      for (const instance of allInstances) {
        if (instance.data.status === 'stopped') {
          await instance.delete();
-        };
+        }
      }
      await plugins.smartdelay.delayFor(10000);
    }
@ -250,7 +248,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu

  // abstract implemented methods
  public async fireDistributedTaskRequest(
-    taskRequestArg: plugins.taskbuffer.distributedCoordination.IDistributedTaskRequest
+    taskRequestArg: plugins.taskbuffer.distributedCoordination.IDistributedTaskRequest,
  ): Promise<plugins.taskbuffer.distributedCoordination.IDistributedTaskRequestResult> {
    await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
      if (!this.ownInstance) {
@ -277,7 +275,7 @@ export class SmartdataDistributedCoordinator extends plugins.taskbuffer.distribu
  }

  public async updateDistributedTaskRequest(
-    infoBasisArg: plugins.taskbuffer.distributedCoordination.IDistributedTaskRequest
+    infoBasisArg: plugins.taskbuffer.distributedCoordination.IDistributedTaskRequest,
  ): Promise<void> {
    await this.asyncExecutionStack.getExclusiveExecutionSlot(async () => {
      const existingInfoBasis = this.ownInstance.data.taskRequests.find((infoBasisItem) => {
--- a/ts/classes.doc.ts
+++ b/ts/classes.doc.ts
@ -8,8 +8,7 @@ import { SmartdataLuceneAdapter } from './classes.lucene.adapter.js';

 export type TDocCreation = 'db' | 'new' | 'mixed';

-// Set of searchable fields for each class
-const searchableFieldsMap = new Map<string, Set<string>>();
+

 export function globalSvDb() {
  return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
@ -39,27 +38,18 @@ export function svDb() {
 */
 export function searchable() {
  return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
-    console.log(`called searchable() on >${target.constructor.name}.${key}<`);
-    
-    // Initialize the set for this class if it doesn't exist
-    const className = target.constructor.name;
-    if (!searchableFieldsMap.has(className)) {
-      searchableFieldsMap.set(className, new Set<string>());
+    // Attach to class constructor for direct access
+    const ctor = target.constructor as any;
+    if (!Array.isArray(ctor.searchableFields)) {
+      ctor.searchableFields = [];
    }
-    
-    // Add the property to the searchable fields set
-    searchableFieldsMap.get(className).add(key);
+    ctor.searchableFields.push(key);
  };
 }

-/**
- * Get searchable fields for a class
- */
-export function getSearchableFields(className: string): string[] {
-  if (!searchableFieldsMap.has(className)) {
-    return [];
-  }
-  return Array.from(searchableFieldsMap.get(className));
+// Escape user input for safe use in MongoDB regular expressions
+function escapeForRegex(input: string): string {
+  return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
 }

 /**
@ -83,7 +73,56 @@ export function unI() {
  };
 }

+/**
+ * Options for MongoDB indexes
+ */
+export interface IIndexOptions {
+  background?: boolean;
+  unique?: boolean;
+  sparse?: boolean;
+  expireAfterSeconds?: number;
+  [key: string]: any;
+}
+
+/**
+ * index - decorator to mark a field for regular indexing
+ */
+export function index(options?: IIndexOptions) {
+  return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
+    console.log(`called index() on >${target.constructor.name}.${key}<`);
+    
+    // Initialize regular indexes array if it doesn't exist
+    if (!target.regularIndexes) {
+      target.regularIndexes = [];
+    }
+    
+    // Add this field to regularIndexes with its options
+    target.regularIndexes.push({
+      field: key,
+      options: options || {}
+    });
+    
+    // Also ensure it's marked as saveable
+    if (!target.saveableProperties) {
+      target.saveableProperties = [];
+    }
+    
+    if (!target.saveableProperties.includes(key)) {
+      target.saveableProperties.push(key);
+    }
+  };
+}
+
 export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
+  // Special case: detect MongoDB operators and pass them through directly
+  const topLevelOperators = ['$and', '$or', '$nor', '$not', '$text', '$where', '$regex'];
+  for (const key of Object.keys(filterArg)) {
+    if (topLevelOperators.includes(key)) {
+      return filterArg; // Return the filter as-is for MongoDB operators
+    }
+  }
+
+  // Original conversion logic for non-MongoDB query objects
  const convertedFilter: { [key: string]: any } = {};

  const convertFilterArgument = (keyPathArg2: string, filterArg2: any) => {
@ -126,7 +165,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
  // STATIC
  public static createInstanceFromMongoDbNativeDoc<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    mongoDbNativeDocArg: any
+    mongoDbNativeDocArg: any,
  ): T {
    const newInstance = new this();
    (newInstance as any).creationStatus = 'db';
@ -144,7 +183,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static async getInstances<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
  ): Promise<T[]> {
    const foundDocs = await (this as any).collection.findAll(convertFilterForMongoDb(filterArg));
    const returnArray = [];
@ -163,7 +202,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static async getInstance<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
  ): Promise<T> {
    const foundDoc = await (this as any).collection.findOne(convertFilterForMongoDb(filterArg));
    if (foundDoc) {
@ -177,7 +216,10 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
  /**
   * get a unique id prefixed with the class name
   */
-  public static async getNewId<T = any>(this: plugins.tsclass.typeFest.Class<T>, lengthArg: number = 20) {
+  public static async getNewId<T = any>(
+    this: plugins.tsclass.typeFest.Class<T>,
+    lengthArg: number = 20,
+  ) {
    return `${(this as any).className}:${plugins.smartunique.shortId(lengthArg)}`;
  }

@ -187,16 +229,30 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static async getCursor<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
  ) {
    const collection: SmartdataCollection<T> = (this as any).collection;
    const cursor: SmartdataDbCursor<T> = await collection.getCursor(
      convertFilterForMongoDb(filterArg),
-      this as any as typeof SmartDataDbDoc
+      this as any as typeof SmartDataDbDoc,
    );
    return cursor;
  }

+  public static async getCursorExtended<T>(
+    this: plugins.tsclass.typeFest.Class<T>,
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    modifierFunction = (cursorArg: plugins.mongodb.FindCursor<plugins.mongodb.WithId<plugins.mongodb.BSON.Document>>) => cursorArg,
+  ): Promise<SmartdataDbCursor<T>> {
+    const collection: SmartdataCollection<T> = (this as any).collection;
+    await collection.init();
+    let cursor: plugins.mongodb.FindCursor<any> = collection.mongoDbCollection.find(
+      convertFilterForMongoDb(filterArg),
+    );
+    cursor = modifierFunction(cursor);
+    return new SmartdataDbCursor<T>(cursor, this as any as typeof SmartDataDbDoc);
+  }
+
  /**
   * watch the collection
   * @param this
@ -205,12 +261,12 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static async watch<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
  ) {
    const collection: SmartdataCollection<T> = (this as any).collection;
    const watcher: SmartdataDbWatcher<T> = await collection.watch(
      convertFilterForMongoDb(filterArg),
-      this as any
+      this as any,
    );
    return watcher;
  }
@ -222,7 +278,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
  public static async forEach<T>(
    this: plugins.tsclass.typeFest.Class<T>,
    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
-    forEachFunction: (itemArg: T) => Promise<any>
+    forEachFunction: (itemArg: T) => Promise<any>,
  ) {
    const cursor: SmartdataDbCursor<T> = await (this as any).getCursor(filterArg);
    await cursor.forEach(forEachFunction);
@ -233,7 +289,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static async getCount<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T> = ({} as any)
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T> = {} as any,
  ) {
    const collection: SmartdataCollection<T> = (this as any).collection;
    return await collection.getCount(filterArg);
@ -246,32 +302,108 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public static createSearchFilter<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string
+    luceneQuery: string,
  ): any {
-    const className = (this as any).className || this.name;
-    const searchableFields = getSearchableFields(className);
-    
+    const searchableFields = (this as any).getSearchableFields();
    if (searchableFields.length === 0) {
-      throw new Error(`No searchable fields defined for class ${className}`);
+      throw new Error(`No searchable fields defined for class ${this.name}`);
    }
-    
    const adapter = new SmartdataLuceneAdapter(searchableFields);
    return adapter.convert(luceneQuery);
  }
+  /**
+   * List all searchable fields defined on this class
+   */
+  public static getSearchableFields(): string[] {
+    const ctor = this as any;
+    return Array.isArray(ctor.searchableFields) ? ctor.searchableFields : [];
+  }

  /**
-   * Search documents using Lucene query syntax
-   * @param luceneQuery Lucene query string
+   * Search documents by text or field:value syntax, with safe regex fallback
+   * @param query A search term or field:value expression
   * @returns Array of matching documents
   */
  public static async search<T>(
    this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string
+    query: string,
  ): Promise<T[]> {
-    const filter = (this as any).createSearchFilter(luceneQuery);
-    return await (this as any).getInstances(filter);
+    const searchableFields = (this as any).getSearchableFields();
+    if (searchableFields.length === 0) {
+      throw new Error(`No searchable fields defined for class ${this.name}`);
+    }
+    // empty query -> return all
+    const q = query.trim();
+    if (!q) {
+      return await (this as any).getInstances({});
+    }
+    // simple exact field:value (no spaces, no wildcards, no quotes)
+    // simple exact field:value (no spaces, wildcards, quotes)
+    const simpleExact = q.match(/^(\w+):([^"'\*\?\s]+)$/);
+    if (simpleExact) {
+      const field = simpleExact[1];
+      const value = simpleExact[2];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
+      }
+      return await (this as any).getInstances({ [field]: value });
+    }
+    // quoted phrase across all searchable fields: exact match of phrase
+    const quoted = q.match(/^"(.+)"$|^'(.+)'$/);
+    if (quoted) {
+      const phrase = quoted[1] || quoted[2] || '';
+      // build regex that matches the exact phrase (allowing flexible whitespace)
+      const parts = phrase.split(/\s+/).map((t) => escapeForRegex(t));
+      const pattern = parts.join('\\s+');
+      const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
+      return await (this as any).getInstances({ $or: orConds });
+    }
+    // wildcard field:value (supports * and ?) -> direct regex on that field
+    const wildcardField = q.match(/^(\w+):(.+[*?].*)$/);
+    if (wildcardField) {
+      const field = wildcardField[1];
+      const pattern = wildcardField[2];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
+      }
+      // escape regex special chars except * and ?, then convert wildcards
+      const escaped = pattern.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+      const regexPattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+      return await (this as any).getInstances({ [field]: { $regex: regexPattern, $options: 'i' } });
+    }
+    // wildcard plain term across all fields (supports * and ?)
+    if (!q.includes(':') && (q.includes('*') || q.includes('?'))) {
+      // build wildcard regex pattern: escape all except * and ? then convert
+      const escaped = q.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+      const pattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+      const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
+      return await (this as any).getInstances({ $or: orConds });
+    }
+    // detect advanced Lucene syntax: field:value, wildcards, boolean, grouping
+    const luceneSyntax = /(\w+:[^\s]+)|\*|\?|\bAND\b|\bOR\b|\bNOT\b|\(|\)/;
+    if (luceneSyntax.test(q)) {
+      const filter = (this as any).createSearchFilter(q);
+      return await (this as any).getInstances(filter);
+    }
+    // multi-term unquoted -> AND of regex across fields for each term
+    const terms = q.split(/\s+/);
+    if (terms.length > 1) {
+      const andConds = terms.map((term) => {
+        const esc = escapeForRegex(term);
+        const ors = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
+        return { $or: ors };
+      });
+      return await (this as any).getInstances({ $and: andConds });
+    }
+    // single term -> regex across all searchable fields
+    const esc = escapeForRegex(q);
+    const orConds = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
+    return await (this as any).getInstances({ $or: orConds });
  }

+
+  // INSTANCE
+
  // INSTANCE

  /**
@ -283,13 +415,13 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   * updated from db in any case where doc comes from db
   */
  @globalSvDb()
-  _createdAt: string = (new Date()).toISOString();
+  _createdAt: string = new Date().toISOString();

  /**
   * will be updated everytime the doc is saved
   */
  @globalSvDb()
-  _updatedAt: string = (new Date()).toISOString();
+  _updatedAt: string = new Date().toISOString();

  /**
   * an array of saveable properties of ALL doc
@ -301,6 +433,11 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public uniqueIndexes: string[];

+  /**
+   * regular indexes with their options
+   */
+  public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+
  /**
   * an array of saveable properties of a specific doc
   */
@ -330,7 +467,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    const self: any = this;
    let dbResult: any;

-    this._updatedAt = (new Date()).toISOString();
+    this._updatedAt = new Date().toISOString();

    switch (this.creationStatus) {
      case 'db':
@ -386,10 +523,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   */
  public async createSavableObject(): Promise<TImplements> {
    const saveableObject: unknown = {}; // is not exposed to outside, so any is ok here
-    const saveableProperties = [
-      ...this.globalSaveableProperties,
-      ...this.saveableProperties
-    ]
+    const saveableProperties = [...this.globalSaveableProperties, ...this.saveableProperties];
    for (const propertyNameString of saveableProperties) {
      saveableObject[propertyNameString] = this[propertyNameString];
    }
--- a/ts/classes.easystore.ts
+++ b/ts/classes.easystore.ts
@ -41,7 +41,7 @@ export class EasyStore<T> {
  private async getEasyStore(): Promise<InstanceType<typeof this.easyStoreClass>> {
    if (this.easyStorePromise) {
      return this.easyStorePromise;
-    };
+    }

    // first run from here
    const deferred = plugins.smartpromise.defer<InstanceType<typeof this.easyStoreClass>>();
--- a/ts/classes.lucene.adapter.ts
+++ b/ts/classes.lucene.adapter.ts
@ -4,7 +4,17 @@
 import * as plugins from './plugins.js';

 // Types
-type NodeType = 'TERM' | 'PHRASE' | 'FIELD' | 'AND' | 'OR' | 'NOT' | 'RANGE' | 'WILDCARD' | 'FUZZY' | 'GROUP';
+type NodeType =
+  | 'TERM'
+  | 'PHRASE'
+  | 'FIELD'
+  | 'AND'
+  | 'OR'
+  | 'NOT'
+  | 'RANGE'
+  | 'WILDCARD'
+  | 'FUZZY'
+  | 'GROUP';

 interface QueryNode {
  type: NodeType;
@ -59,7 +69,15 @@ interface GroupNode extends QueryNode {
  value: AnyQueryNode;
 }

-type AnyQueryNode = TermNode | PhraseNode | FieldNode | BooleanNode | RangeNode | WildcardNode | FuzzyNode | GroupNode;
+type AnyQueryNode =
+  | TermNode
+  | PhraseNode
+  | FieldNode
+  | BooleanNode
+  | RangeNode
+  | WildcardNode
+  | FuzzyNode
+  | GroupNode;

 /**
 * Lucene query parser
@ -137,8 +155,7 @@ export class LuceneParser {
      current += char;

      // Check if current is an operator
-      if (operators.test(current) && 
-          (i + 1 === input.length || /\s/.test(input[i + 1]))) {
+      if (operators.test(current) && (i + 1 === input.length || /\s/.test(input[i + 1]))) {
        tokens.push(current);
        current = '';
      }
@ -164,7 +181,7 @@ export class LuceneParser {
        return {
          type: token as 'AND' | 'OR',
          left,
-          right
+          right,
        };
      } else if (token === 'NOT' || token === '-') {
        this.pos++;
@ -172,7 +189,7 @@ export class LuceneParser {
        return {
          type: 'NOT',
          left,
-          right
+          right,
        };
      }
    }
@ -302,16 +319,26 @@ export class LuceneParser {
      lower,
      upper,
      includeLower,
-      includeUpper
+      includeUpper,
    };
  }
 }

 /**
 * Transformer for Lucene AST to MongoDB query
+ * FIXED VERSION - proper MongoDB query structure
 */
 export class LuceneToMongoTransformer {
-  constructor() {}
+  private defaultFields: string[];
+  constructor(defaultFields: string[] = []) {
+    this.defaultFields = defaultFields;
+  }
+  /**
+   * Escape special characters for use in RegExp patterns
+   */
+  private escapeRegex(input: string): string {
+    return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+  }

  /**
   * Transform a Lucene AST node to a MongoDB query
@ -345,38 +372,39 @@ export class LuceneToMongoTransformer {

  /**
   * Transform a term to MongoDB query
+   * FIXED: properly structured $or query for multiple fields
   */
  private transformTerm(node: TermNode, searchFields?: string[]): any {
-    // If specific fields are provided, search across those fields
-    if (searchFields && searchFields.length > 0) {
-      // Create an $or query to search across multiple fields
-      return {
-        $or: searchFields.map(field => ({
-          [field]: { $regex: node.value, $options: 'i' }
-        }))
-      };
+    // Build regex pattern, support wildcard (*) and fuzzy (?) if present
+    const term = node.value;
+    // Determine regex pattern: wildcard conversion or exact escape
+    let pattern: string;
+    if (term.includes('*') || term.includes('?')) {
+      pattern = this.luceneWildcardToRegex(term);
+    } else {
+      pattern = this.escapeRegex(term);
    }
-    
-    // Otherwise, use text search (requires a text index on desired fields)
-    return { $text: { $search: node.value } };
+    // Search across provided fields or default fields
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
+    }));
+    return { $or: orConditions };
  }

  /**
   * Transform a phrase to MongoDB query
+   * FIXED: properly structured $or query for multiple fields
   */
  private transformPhrase(node: PhraseNode, searchFields?: string[]): any {
-    // If specific fields are provided, search phrase across those fields
-    if (searchFields && searchFields.length > 0) {
-      // Create an $or query to search phrase across multiple fields
-      return {
-        $or: searchFields.map(field => ({
-          [field]: { $regex: `${node.value.replace(/\s+/g, '\\s+')}`, $options: 'i' }
-        }))
-      };
-    }
-    
-    // For phrases, we use a regex to ensure exact matches
-    return { $text: { $search: `"${node.value}"` } };
+    // Use regex across provided fields or default fields, respecting word boundaries
+    const parts = node.value.split(/\s+/).map((t) => this.escapeRegex(t));
+    const pattern = parts.join('\\s+');
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
+    }));
+    return { $or: orConditions };
  }

  /**
@ -395,8 +423,8 @@ export class LuceneToMongoTransformer {
      return {
        [node.field]: {
          $regex: this.luceneWildcardToRegex((node.value as WildcardNode).value),
-          $options: 'i'
-        } 
+          $options: 'i',
+        },
      };
    }

@ -405,23 +433,28 @@ export class LuceneToMongoTransformer {
      return {
        [node.field]: {
          $regex: this.createFuzzyRegex((node.value as FuzzyNode).value),
-          $options: 'i'
-        } 
+          $options: 'i',
+        },
      };
    }

-    // Special case for exact term matches on fields
+    // Special case for exact term matches on fields (supporting wildcard characters)
    if (node.value.type === 'TERM') {
-      return { [node.field]: (node.value as TermNode).value };
+      const val = (node.value as TermNode).value;
+      if (val.includes('*') || val.includes('?')) {
+        const regex = this.luceneWildcardToRegex(val);
+        return { [node.field]: { $regex: regex, $options: 'i' } };
+      }
+      return { [node.field]: { $regex: val, $options: 'i' } };
    }

    // Special case for phrase matches on fields
    if (node.value.type === 'PHRASE') {
      return {
        [node.field]: {
-          $regex: `^${(node.value as PhraseNode).value}$`,
-          $options: 'i'
-        } 
+          $regex: `${(node.value as PhraseNode).value.replace(/\s+/g, '\\s+')}`,
+          $options: 'i',
+        },
      };
    }

@ -430,14 +463,50 @@ export class LuceneToMongoTransformer {

    // If the transformed value uses $text, we need to adapt it for the field
    if (transformedValue.$text) {
-      return { [node.field]: transformedValue.$text.$search };
+      return { [node.field]: { $regex: transformedValue.$text.$search, $options: 'i' } };
+    }
+
+    // Handle $or and $and cases
+    if (transformedValue.$or || transformedValue.$and) {
+      // This is a bit complex - we need to restructure the query to apply the field
+      // For now, simplify by just using a regex on the field
+      const term = this.extractTermFromBooleanQuery(transformedValue);
+      if (term) {
+        return { [node.field]: { $regex: term, $options: 'i' } };
+      }
    }

    return { [node.field]: transformedValue };
  }

+  /**
+   * Extract a term from a boolean query (simplification)
+   */
+  private extractTermFromBooleanQuery(query: any): string | null {
+    if (query.$or && Array.isArray(query.$or) && query.$or.length > 0) {
+      const firstClause = query.$or[0];
+      for (const field in firstClause) {
+        if (firstClause[field].$regex) {
+          return firstClause[field].$regex;
+        }
+      }
+    }
+
+    if (query.$and && Array.isArray(query.$and) && query.$and.length > 0) {
+      const firstClause = query.$and[0];
+      for (const field in firstClause) {
+        if (firstClause[field].$regex) {
+          return firstClause[field].$regex;
+        }
+      }
+    }
+
+    return null;
+  }
+
  /**
   * Transform AND operator to MongoDB query
+   * FIXED: $and must be an array
   */
  private transformAnd(node: BooleanNode): any {
    return { $and: [this.transform(node.left), this.transform(node.right)] };
@ -445,6 +514,7 @@ export class LuceneToMongoTransformer {

  /**
   * Transform OR operator to MongoDB query
+   * FIXED: $or must be an array
   */
  private transformOr(node: BooleanNode): any {
    return { $or: [this.transform(node.left), this.transform(node.right)] };
@ -452,6 +522,7 @@ export class LuceneToMongoTransformer {

  /**
   * Transform NOT operator to MongoDB query
+   * FIXED: $and must be an array and $not usage
   */
  private transformNot(node: BooleanNode): any {
    const leftQuery = this.transform(node.left);
@ -459,21 +530,50 @@ export class LuceneToMongoTransformer {

    // Create a query that includes left but excludes right
    if (rightQuery.$text) {
-      // Text searches need special handling for negation
-      return {
-        $and: [
-          leftQuery,
-          { $not: rightQuery }
-        ]
-      };
+      // For text searches, we need a different approach
+      // We'll use a negated regex instead
+      const searchTerm = rightQuery.$text.$search.replace(/"/g, '');
+
+      // Determine the fields to apply the negation to
+      const notConditions = [];
+
+      for (const field in leftQuery) {
+        if (field !== '$or' && field !== '$and') {
+          notConditions.push({
+            [field]: { $not: { $regex: searchTerm, $options: 'i' } },
+          });
+        }
+      }
+
+      // If left query has $or or $and, we need to handle it differently
+      if (leftQuery.$or) {
+        return {
+          $and: [leftQuery, { $nor: [{ $or: notConditions }] }],
+        };
+      } else {
+        // Simple case - just add $not to each field
+        return {
+          $and: [leftQuery, { $and: notConditions }],
+        };
+      }
    } else {
      // For other queries, we can use $not directly
-      return {
-        $and: [
-          leftQuery,
-          { $not: rightQuery }
-        ]
-      };
+      // We need to handle different structures based on the rightQuery
+      let notQuery = {};
+
+      if (rightQuery.$or) {
+        notQuery = { $nor: rightQuery.$or };
+      } else if (rightQuery.$and) {
+        // Convert $and to $nor
+        notQuery = { $nor: rightQuery.$and };
+      } else {
+        // Simple field condition
+        for (const field in rightQuery) {
+          notQuery[field] = { $not: rightQuery[field] };
+        }
+      }
+
+      return { $and: [leftQuery, notQuery] };
    }
  }

@ -496,6 +596,7 @@ export class LuceneToMongoTransformer {

  /**
   * Transform wildcard query to MongoDB query
+   * FIXED: properly structured for multiple fields
   */
  private transformWildcard(node: WildcardNode, searchFields?: string[]): any {
    // Convert Lucene wildcards to MongoDB regex
@ -503,19 +604,20 @@ export class LuceneToMongoTransformer {

    // If specific fields are provided, search wildcard across those fields
    if (searchFields && searchFields.length > 0) {
-      return {
-        $or: searchFields.map(field => ({
-          [field]: { $regex: regex, $options: 'i' }
-        }))
-      };
+      const orConditions = searchFields.map((field) => ({
+        [field]: { $regex: regex, $options: 'i' },
+      }));
+
+      return { $or: orConditions };
    }

-    // By default, apply to all text fields using $text search
+    // By default, apply to the default field
    return { $regex: regex, $options: 'i' };
  }

  /**
   * Transform fuzzy query to MongoDB query
+   * FIXED: properly structured for multiple fields
   */
  private transformFuzzy(node: FuzzyNode, searchFields?: string[]): any {
    // MongoDB doesn't have built-in fuzzy search
@ -524,20 +626,21 @@ export class LuceneToMongoTransformer {

    // If specific fields are provided, search fuzzy term across those fields
    if (searchFields && searchFields.length > 0) {
-      return {
-        $or: searchFields.map(field => ({
-          [field]: { $regex: regex, $options: 'i' }
-        }))
-      };
+      const orConditions = searchFields.map((field) => ({
+        [field]: { $regex: regex, $options: 'i' },
+      }));
+
+      return { $or: orConditions };
    }

+    // By default, apply to the default field
    return { $regex: regex, $options: 'i' };
  }

  /**
   * Convert Lucene wildcards to MongoDB regex patterns
   */
-  private luceneWildcardToRegex(wildcardPattern: string): string {
+  public luceneWildcardToRegex(wildcardPattern: string): string {
    // Replace Lucene wildcards with regex equivalents
    // * => .*
    // ? => .
@ -602,7 +705,8 @@ export class SmartdataLuceneAdapter {
   */
  constructor(defaultSearchFields?: string[]) {
    this.parser = new LuceneParser();
-    this.transformer = new LuceneToMongoTransformer();
+    // Pass default searchable fields into transformer
+    this.transformer = new LuceneToMongoTransformer(defaultSearchFields || []);
    if (defaultSearchFields) {
      this.defaultSearchFields = defaultSearchFields;
    }
@ -615,6 +719,32 @@ export class SmartdataLuceneAdapter {
   */
  convert(luceneQuery: string, searchFields?: string[]): any {
    try {
+      // For simple single-term queries (no field:, boolean, grouping), use simpler regex
+      if (
+        !luceneQuery.includes(':') &&
+        !luceneQuery.includes(' AND ') &&
+        !luceneQuery.includes(' OR ') &&
+        !luceneQuery.includes(' NOT ') &&
+        !luceneQuery.includes('(') &&
+        !luceneQuery.includes('[')
+      ) {
+        const fieldsToSearch = searchFields || this.defaultSearchFields;
+        if (fieldsToSearch && fieldsToSearch.length > 0) {
+          // Handle wildcard characters in query
+          let pattern = luceneQuery;
+          if (luceneQuery.includes('*') || luceneQuery.includes('?')) {
+            // Use transformer to convert wildcard pattern
+            pattern = this.transformer.luceneWildcardToRegex(luceneQuery);
+          }
+          return {
+            $or: fieldsToSearch.map((field) => ({
+              [field]: { $regex: pattern, $options: 'i' },
+            })),
+          };
+        }
+      }
+
+      // For more complex queries, use the full parser
      // Parse the Lucene query into an AST
      const ast = this.parser.parse(luceneQuery);

@ -624,6 +754,7 @@ export class SmartdataLuceneAdapter {
      // Transform the AST to a MongoDB query
      return this.transformWithFields(ast, fieldsToSearch);
    } catch (error) {
+      console.error(`Failed to convert Lucene query "${luceneQuery}":`, error);
      throw new Error(`Failed to convert Lucene query: ${error}`);
    }
  }
@ -632,8 +763,13 @@ export class SmartdataLuceneAdapter {
   * Helper method to transform the AST with field information
   */
  private transformWithFields(node: AnyQueryNode, searchFields: string[]): any {
-    // For term nodes without a specific field, apply the search fields
-    if (node.type === 'TERM') {
+    // Special case for term nodes without a specific field
+    if (
+      node.type === 'TERM' ||
+      node.type === 'PHRASE' ||
+      node.type === 'WILDCARD' ||
+      node.type === 'FUZZY'
+    ) {
      return this.transformer.transform(node, searchFields);
    }

--- a/ts/classes.watcher.ts
+++ b/ts/classes.watcher.ts
@ -14,7 +14,7 @@ export class SmartdataDbWatcher<T = any> {
  public changeSubject = new plugins.smartrx.rxjs.Subject<T>();
  constructor(
    changeStreamArg: plugins.mongodb.ChangeStream<T>,
-    smartdataDbDocArg: typeof SmartDataDbDoc
+    smartdataDbDocArg: typeof SmartDataDbDoc,
  ) {
    this.changeStream = changeStreamArg;
    this.changeStream.on('change', async (item: any) => {
@ -23,7 +23,7 @@ export class SmartdataDbWatcher<T = any> {
        return;
      }
      this.changeSubject.next(
-        smartdataDbDocArg.createInstanceFromMongoDbNativeDoc(item.fullDocument) as any as T
+        smartdataDbDocArg.createInstanceFromMongoDbNativeDoc(item.fullDocument) as any as T,
      );
    });
    plugins.smartdelay.delayFor(0).then(() => {
--- a/tsconfig.json
+++ b/tsconfig.json
@ -6,7 +6,9 @@
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "esModuleInterop": true,
-    "verbatimModuleSyntax": true
+    "verbatimModuleSyntax": true,
+    "baseUrl": ".",
+    "paths": {}
  },
  "exclude": [
    "dist_*/**/*.d.ts"
Author	SHA1	Message	Date
Philipp Kunz	c5a44da975	5.11.2	2025-04-21 17:31:30 +00:00
Philipp Kunz	969b073939	fix(readme): Update readme to clarify usage of searchable fields retrieval	2025-04-21 17:31:30 +00:00
Philipp Kunz	ac80f90ae0	5.11.1	2025-04-21 16:35:29 +00:00
Philipp Kunz	d0e769622e	fix(doc): Refactor searchable fields API and improve collection registration.	2025-04-21 16:35:29 +00:00
Philipp Kunz	eef758cabb	5.11.0	2025-04-21 15:29:11 +00:00
Philipp Kunz	d0cc2a0ed2	feat(ts/classes.lucene.adapter): Expose luceneWildcardToRegex method to allow external usage and enhance regex transformation capabilities.	2025-04-21 15:29:11 +00:00
Philipp Kunz	87c930121c	5.10.0	2025-04-21 15:27:55 +00:00
Philipp Kunz	23b499b3a8	feat(search): Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests	2025-04-21 15:27:55 +00:00
Philipp Kunz	0834ec5c91	5.9.2	2025-04-18 15:10:04 +00:00
Philipp Kunz	6a2a708ea1	fix(documentation): Update search API documentation to replace deprecated searchWithLucene examples with the unified search(query) API and clarify its behavior.	2025-04-18 15:10:03 +00:00
Philipp Kunz	1d977986f1	5.9.1	2025-04-18 14:56:11 +00:00
Philipp Kunz	e325b42906	fix(search): Refactor search tests to use unified search API and update text index type casting	2025-04-18 14:56:11 +00:00
Philipp Kunz	1a359d355a	5.9.0	2025-04-18 11:25:39 +00:00
Philipp Kunz	b5a9449d5e	feat(collections/search): Improve text index creation and search fallback mechanisms in collections and document search methods	2025-04-18 11:25:39 +00:00
Philipp Kunz	558f83a3d9	5.8.4	2025-04-17 11:47:38 +00:00
Philipp Kunz	76ae454221	fix(core): Update commit metadata with no functional code changes	2025-04-17 11:47:38 +00:00
Philipp Kunz	90cfc4644d	5.8.3	2025-04-17 11:21:35 +00:00
Philipp Kunz	0be279e5f5	fix(readme): Improve readme documentation on data models and connection management	2025-04-17 11:21:35 +00:00
Philipp Kunz	9755522bba	5.8.2	2025-04-14 18:13:10 +00:00
Philipp Kunz	de8736e99e	fix(classes.doc.ts): Ensure collection initialization before creating a cursor in getCursorExtended	2025-04-14 18:13:10 +00:00
Philipp Kunz	c430627a21	5.8.1	2025-04-14 18:06:29 +00:00
Philipp Kunz	0bfebaf5b9	fix(cursor, doc): Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.	2025-04-14 18:06:29 +00:00
Philipp Kunz	4733982d03	5.8.0	2025-04-14 17:58:54 +00:00
Philipp Kunz	368dc27607	feat(cursor): Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances	2025-04-14 17:58:54 +00:00
Philipp Kunz	938b25c925	5.7.0	2025-04-14 17:49:07 +00:00
Philipp Kunz	ab251858ba	feat(SmartDataDbDoc): Add extended cursor method getCursorExtended for flexible cursor modifications	2025-04-14 17:49:07 +00:00
Philipp Kunz	24371ccf78	5.6.0	2025-04-07 16:47:16 +00:00
Philipp Kunz	ed1eecbab8	feat(indexing): Add support for regular index creation in documents and collections	2025-04-07 16:47:16 +00:00
Philipp Kunz	0d2dcec3e2	5.5.1	2025-04-06 18:18:40 +00:00
Philipp Kunz	9426a21a2a	fix(ci & formatting): Minor fixes: update CI workflow image and npmci package references, adjust package.json and readme URLs, and apply consistent code formatting.	2025-04-06 18:18:39 +00:00
Philipp Kunz	4fac974fc9	5.5.0	2025-04-06 18:14:47 +00:00
Philipp Kunz	cad2decf59	feat(search): Enhance search functionality with robust Lucene query transformation and reliable fallback mechanisms	2025-04-06 18:14:46 +00:00