5.10.0

changelog.md

@@ -1,5 +1,72 @@
 # Changelog
 
+## 2025-04-21 - 5.10.0 - feat(search)
+Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests
+
+- Updated readme.md with detailed Lucene‑style search examples and use cases
+- Enhanced LuceneToMongoTransformer to properly handle wildcard conversion and regex escaping
+- Improved search query parsing in SmartDataDbDoc for field-specific, multi-term, and advanced Lucene syntax
+- Added new advanced search tests covering boolean operators, grouping, quoted phrases, and wildcard queries
+
+## 2025-04-18 - 5.9.2 - fix(documentation)
+Update search API documentation to replace deprecated searchWithLucene examples with the unified search(query) API and clarify its behavior.
+
+- Replaced 'searchWithLucene' examples with 'search(query)' in the README.
+- Updated explanation to detail field-specific exact match, partial word regex search, multi-word literal matching, and handling of empty queries.
+- Clarified guidelines for creating MongoDB text indexes on searchable fields for optimized search performance.
+
+## 2025-04-18 - 5.9.1 - fix(search)
+Refactor search tests to use unified search API and update text index type casting
+
+- Replaced all calls from searchWithLucene with search in test/search tests
+- Updated text index specification in the collection class to use proper type casting
+
+## 2025-04-18 - 5.9.0 - feat(collections/search)
+Improve text index creation and search fallback mechanisms in collections and document search methods
+
+- Auto-create a compound text index on all searchable fields in SmartdataCollection with a one-time flag to prevent duplicate index creation.
+- Refine the search method in SmartDataDbDoc to support exact field matches and safe regex fallback for non-Lucene queries.
+
+## 2025-04-17 - 5.8.4 - fix(core)
+Update commit metadata with no functional code changes
+
+- Commit info and documentation refreshed
+- No code or test changes detected in the diff
+
+## 2025-04-17 - 5.8.3 - fix(readme)
+Improve readme documentation on data models and connection management
+
+- Clarify that data models use @Collection, @unI, @svDb, @index, and @searchable decorators
+- Document that ObjectId and Buffer fields are stored as BSON types natively without extra decorators
+- Update connection management section to use 'db.close()' instead of 'db.disconnect()'
+- Revise license section to reference the MIT License without including additional legal details
+
+## 2025-04-14 - 5.8.2 - fix(classes.doc.ts)
+Ensure collection initialization before creating a cursor in getCursorExtended
+
+- Added 'await collection.init()' to guarantee that the MongoDB collection is initialized before using the cursor
+- Prevents potential runtime errors when accessing collection.mongoDbCollection
+
+## 2025-04-14 - 5.8.1 - fix(cursor, doc)
+Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.
+
+- Specify Promise<T> as return type for next() in SmartdataDbCursor and cast return value to T.
+- Specify Promise<T[]> as return type for toArray() in SmartdataDbCursor and cast return value to T[].
+- Update getCursorExtended to return Promise<SmartdataDbCursor<T>> for clearer type safety.
+
+## 2025-04-14 - 5.8.0 - feat(cursor)
+Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances
+
+- Introduced asynchronous toArray method in SmartdataDbCursor which retrieves all documents from the MongoDB cursor
+- Maps each native document to a SmartDataDbDoc instance using createInstanceFromMongoDbNativeDoc for consistent API usage
+
+## 2025-04-14 - 5.7.0 - feat(SmartDataDbDoc)
+Add extended cursor method getCursorExtended for flexible cursor modifications
+
+- Introduces getCursorExtended in classes.doc.ts to allow modifier functions for MongoDB cursors
+- Wraps the modified cursor with SmartdataDbCursor for improved API consistency
+- Enhances querying capabilities by enabling customized cursor transformations
+
 ## 2025-04-07 - 5.6.0 - feat(indexing)
 Add support for regular index creation in documents and collections
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.6.0",
+  "version": "5.10.0",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",

readme.md

@@ -18,7 +18,7 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
 - **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
 - **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
 - **Serialization Hooks**: Custom serialization and deserialization of document properties
-- **Powerful Search Capabilities**: Lucene-like query syntax with field-specific search, advanced operators, and fallback mechanisms
+- **Powerful Search Capabilities**: Unified `search(query)` API supporting field:value exact matches, multi-field regex searches, case-insensitive matching, and automatic escaping to prevent regex injection
 
 ## Requirements
 
@@ -66,7 +66,7 @@ await db.init();
 
 ### Defining Data Models
 
-Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
+Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, `@svDb`, `@index`, and `@searchable` to define your data models. Fields of type `ObjectId` or `Buffer` decorated with `@svDb()` will be stored as BSON ObjectId and Binary, respectively; no separate `@oid()` or `@bin()` decorators are required.
 
 ```typescript
 import {
@@ -74,8 +74,6 @@ import {
   Collection,
   unI,
   svDb,
-  oid,
-  bin,
   index,
   searchable,
 } from '@push.rocks/smartdata';
@@ -96,12 +94,10 @@ class User extends SmartDataDbDoc<User, User> {
   public email: string; // Mark 'email' to be saved in DB
 
   @svDb()
-  @oid() // Automatically handle as ObjectId type
-  public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
+  public organizationId: ObjectId; // Stored as BSON ObjectId
 
   @svDb()
-  @bin() // Automatically handle as Binary data
-  public profilePicture: Buffer; // Will be automatically converted to/from Binary
+  public profilePicture: Buffer; // Stored as BSON Binary
 
   @svDb({
     serialize: (data) => JSON.stringify(data), // Custom serialization
@@ -193,72 +189,57 @@ await user.delete(); // Delete the user from the database
 
 ### Search Functionality
 
-SmartData provides powerful search capabilities with a Lucene-like query syntax and robust fallback mechanisms:
+SmartData provides powerful, Lucene‑style search capabilities with robust fallback mechanisms:
 
 ```typescript
 // Define a model with searchable fields
 @Collection(() => db)
 class Product extends SmartDataDbDoc<Product, Product> {
-  @unI()
-  public id: string = 'product-id';
-
-  @svDb()
-  @searchable() // Mark this field as searchable
-  public name: string;
-
-  @svDb()
-  @searchable() // Mark this field as searchable
-  public description: string;
-
-  @svDb()
-  @searchable() // Mark this field as searchable
-  public category: string;
-
-  @svDb()
-  public price: number;
+  @unI() public id: string = 'product-id';
+  @svDb() @searchable() public name: string;
+  @svDb() @searchable() public description: string;
+  @svDb() @searchable() public category: string;
+  @svDb() public price: number;
 }
 
-// Get all fields marked as searchable for a class
-const searchableFields = getSearchableFields('Product'); // ['name', 'description', 'category']
+// List searchable fields
+const searchableFields = getSearchableFields('Product');
 
-// Basic search across all searchable fields
-const iPhoneProducts = await Product.searchWithLucene('iPhone');
+// 1: Exact phrase across all fields
+await Product.search('"Kindle Paperwhite"');
 
-// Field-specific search
-const electronicsProducts = await Product.searchWithLucene('category:Electronics');
+// 2: Wildcard search across all fields
+await Product.search('Air*');
 
-// Search with wildcards
-const macProducts = await Product.searchWithLucene('Mac*');
+// 3: Field‑scoped wildcard
+await Product.search('name:Air*');
 
-// Search in specific fields with partial words
-const laptopResults = await Product.searchWithLucene('description:laptop');
+// 4: Boolean AND/OR/NOT
+await Product.search('category:Electronics AND name:iPhone');
 
-// Search is case-insensitive
-const results1 = await Product.searchWithLucene('electronics');
-const results2 = await Product.searchWithLucene('Electronics');
-// results1 and results2 will contain the same documents
+// 5: Grouping with parentheses
+await Product.search('(Furniture OR Electronics) AND Chair');
 
-// Using boolean operators (requires text index in MongoDB)
-const wirelessOrLaptop = await Product.searchWithLucene('wireless OR laptop');
+// 6: Multi‑term unquoted (terms AND’d across fields)
+await Product.search('TypeScript Aufgabe');
 
-// Negative searches
-const electronicsNotSamsung = await Product.searchWithLucene('Electronics NOT Samsung');
-
-// Phrase searches
-const exactPhrase = await Product.searchWithLucene('"high-speed blender"');
-
-// Grouping with parentheses
-const complexQuery = await Product.searchWithLucene('(wireless OR bluetooth) AND Electronics');
+// 7: Empty query returns all documents
+await Product.search('');
 ```
 
 The search functionality includes:
 
 - `@searchable()` decorator for marking fields as searchable
-- `getSearchableFields()` to retrieve all searchable fields for a class
-- `search()` method for basic search (requires MongoDB text index)
-- `searchWithLucene()` method with robust fallback mechanisms
-- Support for field-specific searches, wildcards, and boolean operators
-- Automatic fallback to regex-based search if MongoDB text search fails
+- `getSearchableFields()` to list searchable fields for a model
+- `search(query: string)` method supporting:
+  - Exact phrase matches (`"my exact string"` or `'my exact string'`)
+  - Field‑scoped exact & wildcard searches (`field:value`, `field:Air*`)
+  - Wildcard searches across all fields (`Air*`, `?Pods`)
+  - Boolean operators (`AND`, `OR`, `NOT`) with grouping (`(...)`)
+  - Multi‑term unquoted queries AND’d across fields (`TypeScript Aufgabe`)
+  - Single/multi‑term regex searches across fields
+  - Empty queries returning all documents
+- Automatic escaping & wildcard conversion to prevent regex injection
 
 ### EasyStore
 
@@ -541,7 +522,7 @@ class Order extends SmartDataDbDoc<Order, Order> {
 ### Connection Management
 
 - Always call `db.init()` before using any database features
-- Use `db.disconnect()` when shutting down your application
+- Use `db.close()` when shutting down your application
 - Set appropriate connection pool sizes based on your application's needs
 
 ### Document Design
@@ -553,10 +534,10 @@ class Order extends SmartDataDbDoc<Order, Order> {
 
 ### Search Optimization
 
-- Create MongoDB text indexes for collections that need advanced search operations
-- Use `searchWithLucene()` for robust searches with fallback mechanisms
-- Prefer field-specific searches when possible for better performance
-- Use simple term queries instead of boolean operators if you don't have text indexes
+- (Optional) Create MongoDB text indexes on searchable fields to speed up full-text search
+- Use `search(query)` for all search operations (field:value, partial matches, multi-word)
+- Prefer field-specific exact matches when possible for optimal performance
+- Avoid unnecessary complexity in query strings to keep regex searches efficient
 
 ### Performance Optimization
 
@@ -590,7 +571,7 @@ Please make sure to update tests as appropriate and follow our coding standards.
 
 ## License and Legal Information
 
-This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
+This repository is licensed under the MIT License. For details, see [MIT License](https://opensource.org/licenses/MIT).
 
 **Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
 

test/test.search.advanced.ts

@@ -0,0 +1,187 @@
+import { tap, expect } from '@push.rocks/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import * as smartdata from '../ts/index.js';
+import { searchable, getSearchableFields } from '../ts/classes.doc.js';
+import { smartunique } from '../ts/plugins.js';
+
+// Set up database connection
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define a test class for advanced search scenarios
+@smartdata.Collection(() => testDb)
+class Product extends smartdata.SmartDataDbDoc<Product, Product> {
+  @smartdata.unI()
+  public id: string = smartunique.shortId();
+
+  @smartdata.svDb()
+  @searchable()
+  public name: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public description: string;
+
+  @smartdata.svDb()
+  @searchable()
+  public category: string;
+
+  @smartdata.svDb()
+  public price: number;
+
+  constructor(
+    nameArg: string,
+    descriptionArg: string,
+    categoryArg: string,
+    priceArg: number,
+  ) {
+    super();
+    this.name = nameArg;
+    this.description = descriptionArg;
+    this.category = categoryArg;
+    this.price = priceArg;
+  }
+}
+
+// Initialize DB and insert sample products
+tap.test('setup advanced search database', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(
+    await smartmongoInstance.getMongoDescriptor(),
+  );
+  await testDb.init();
+});
+
+tap.test('insert products for advanced search', async () => {
+  const products = [
+    new Product(
+      'Night Owl Lamp',
+      'Bright lamp for night reading',
+      'Lighting',
+      29,
+    ),
+    new Product(
+      'Day Light Lamp',
+      'Daytime lamp with adjustable brightness',
+      'Lighting',
+      39,
+    ),
+    new Product(
+      'Office Chair',
+      'Ergonomic chair for office',
+      'Furniture',
+      199,
+    ),
+    new Product(
+      'Gaming Chair',
+      'Comfortable for long gaming sessions',
+      'Furniture',
+      299,
+    ),
+    new Product(
+      'iPhone 12',
+      'Latest iPhone with A14 Bionic chip',
+      'Electronics',
+      999,
+    ),
+    new Product(
+      'AirPods',
+      'Wireless earbuds with noise cancellation',
+      'Electronics',
+      249,
+    ),
+  ];
+  for (const p of products) {
+    await p.save();
+  }
+  const all = await Product.getInstances({});
+  expect(all.length).toEqual(products.length);
+});
+
+// Simple exact field:value matching
+tap.test('simpleExact: category:Furniture returns chairs', async () => {
+  const res = await Product.search('category:Furniture');
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// simpleExact invalid field should throw
+tap.test('simpleExact invalid field errors', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:29');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+// Quoted phrase search
+tap.test('quoted phrase "Bright lamp" matches Night Owl Lamp', async () => {
+  const res = await Product.search('"Bright lamp"');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+tap.test("quoted phrase 'night reading' matches Night Owl Lamp", async () => {
+  const res = await Product.search("'night reading'");
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+
+tap.test('wildcard description:*gaming* matches Gaming Chair', async () => {
+  const res = await Product.search('description:*gaming*');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Gaming Chair');
+});
+
+// Boolean AND and OR
+tap.test('boolean AND: category:Lighting AND lamp', async () => {
+  const res = await Product.search('category:Lighting AND lamp');
+  expect(res.length).toEqual(2);
+});
+
+tap.test('boolean OR: Furniture OR Electronics', async () => {
+  const res = await Product.search('Furniture OR Electronics');
+  expect(res.length).toEqual(4);
+});
+
+// Multi-term unquoted -> AND across terms
+tap.test('multi-term unquoted adjustable brightness', async () => {
+  const res = await Product.search('adjustable brightness');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Day Light Lamp');
+});
+
+tap.test('multi-term unquoted Night Lamp', async () => {
+  const res = await Product.search('Night Lamp');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Night Owl Lamp');
+});
+
+// Grouping with parentheses
+tap.test('grouping: (Furniture OR Electronics) AND Chair', async () => {
+  const res = await Product.search(
+    '(Furniture OR Electronics) AND Chair',
+  );
+  expect(res.length).toEqual(2);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['Gaming Chair', 'Office Chair']);
+});
+
+// Teardown
+tap.test('cleanup advanced search database', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  if (smartmongoInstance) {
+    await smartmongoInstance.stopAndDumpToDir(
+      `.nogit/dbdump/test.search.advanced.ts`,
+    );
+  }
+  setTimeout(() => process.exit(), 2000);
+});
+
+tap.start({ throwOnError: true });

test/test.search.ts

@@ -104,21 +104,21 @@ tap.test('should search products by basic search method', async () => {
   }
 });
 
-tap.test('should search products with searchWithLucene method', async () => {
+tap.test('should search products with search method', async () => {
   // Using the robust searchWithLucene method
-  const wirelessResults = await Product.searchWithLucene('wireless');
-  console.log(
-    `Found ${wirelessResults.length} products matching 'wireless' using searchWithLucene`,
-  );
+    const wirelessResults = await Product.search('wireless');
+    console.log(
+      `Found ${wirelessResults.length} products matching 'wireless' using search`,
+    );
 
   expect(wirelessResults.length).toEqual(1);
   expect(wirelessResults[0].name).toEqual('AirPods');
 });
 
-tap.test('should search products by category with searchWithLucene', async () => {
+tap.test('should search products by category with search', async () => {
   // Using field-specific search with searchWithLucene
-  const kitchenResults = await Product.searchWithLucene('category:Kitchen');
-  console.log(`Found ${kitchenResults.length} products in Kitchen category using searchWithLucene`);
+  const kitchenResults = await Product.search('category:Kitchen');
+  console.log(`Found ${kitchenResults.length} products in Kitchen category using search`);
 
   expect(kitchenResults.length).toEqual(2);
   expect(kitchenResults[0].category).toEqual('Kitchen');
@@ -127,7 +127,7 @@ tap.test('should search products by category with searchWithLucene', async () =>
 
 tap.test('should search products with partial word matches', async () => {
   // Testing partial word matches
-  const proResults = await Product.searchWithLucene('Pro');
+  const proResults = await Product.search('Pro');
   console.log(`Found ${proResults.length} products matching 'Pro'`);
 
   // Should match both "MacBook Pro" and "professionals" in description
@@ -136,7 +136,7 @@ tap.test('should search products with partial word matches', async () => {
 
 tap.test('should search across multiple searchable fields', async () => {
   // Test searching across all searchable fields
-  const bookResults = await Product.searchWithLucene('book');
+  const bookResults = await Product.search('book');
   console.log(`Found ${bookResults.length} products matching 'book' across all fields`);
 
   // Should match "MacBook" in name and "Books" in category
@@ -145,8 +145,8 @@ tap.test('should search across multiple searchable fields', async () => {
 
 tap.test('should handle case insensitive searches', async () => {
   // Test case insensitivity
-  const electronicsResults = await Product.searchWithLucene('electronics');
-  const ElectronicsResults = await Product.searchWithLucene('Electronics');
+  const electronicsResults = await Product.search('electronics');
+  const ElectronicsResults = await Product.search('Electronics');
 
   console.log(`Found ${electronicsResults.length} products matching lowercase 'electronics'`);
   console.log(`Found ${ElectronicsResults.length} products matching capitalized 'Electronics'`);
@@ -166,14 +166,14 @@ tap.test('should demonstrate search fallback mechanisms', async () => {
 
   // Use a simpler term that should be found in descriptions
   // Avoid using "OR" operator which requires a text index
-  const results = await Product.searchWithLucene('high');
+  const results = await Product.search('high');
   console.log(`Found ${results.length} products matching 'high'`);
 
   // "High-speed blender" contains "high"
   expect(results.length).toBeGreaterThan(0);
 
   // Try another fallback example that won't need $text
-  const powerfulResults = await Product.searchWithLucene('powerful');
+  const powerfulResults = await Product.search('powerful');
   console.log(`Found ${powerfulResults.length} products matching 'powerful'`);
 
   // "Powerful laptop for professionals" contains "powerful"
@@ -192,6 +192,71 @@ tap.test('should explain the advantages of the integrated approach', async () =>
   expect(true).toEqual(true);
 });
 
+// Additional robustness tests
+tap.test('should search exact name using field:value', async () => {
+  const nameResults = await Product.search('name:AirPods');
+  expect(nameResults.length).toEqual(1);
+  expect(nameResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should throw when searching non-searchable field', async () => {
+  let error: Error;
+  try {
+    await Product.search('price:129');
+  } catch (err) {
+    error = err as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+tap.test('empty query should return all products', async () => {
+  const allResults = await Product.search('');
+  expect(allResults.length).toEqual(8);
+});
+
+tap.test('should search multi-word term across fields', async () => {
+  const termResults = await Product.search('iPhone 12');
+  expect(termResults.length).toEqual(1);
+  expect(termResults[0].name).toEqual('iPhone 12');
+});
+
+// Additional search scenarios
+tap.test('should return zero results for non-existent terms', async () => {
+  const noResults = await Product.search('NonexistentTerm');
+  expect(noResults.length).toEqual(0);
+});
+
+tap.test('should search products by description term "noise"', async () => {
+  const noiseResults = await Product.search('noise');
+  expect(noiseResults.length).toEqual(1);
+  expect(noiseResults[0].name).toEqual('AirPods');
+});
+
+tap.test('should search products by description term "flagship"', async () => {
+  const flagshipResults = await Product.search('flagship');
+  expect(flagshipResults.length).toEqual(1);
+  expect(flagshipResults[0].name).toEqual('Galaxy S21');
+});
+
+tap.test('should search numeric strings "12"', async () => {
+  const twelveResults = await Product.search('12');
+  expect(twelveResults.length).toEqual(1);
+  expect(twelveResults[0].name).toEqual('iPhone 12');
+});
+
+tap.test('should search hyphenated terms "high-speed"', async () => {
+  const hyphenResults = await Product.search('high-speed');
+  expect(hyphenResults.length).toEqual(1);
+  expect(hyphenResults[0].name).toEqual('Blender');
+});
+
+tap.test('should search hyphenated terms "E-reader"', async () => {
+  const ereaderResults = await Product.search('E-reader');
+  expect(ereaderResults.length).toEqual(1);
+  expect(ereaderResults[0].name).toEqual('Kindle Paperwhite');
+});
+
 tap.test('close database connection', async () => {
   await testDb.mongoDb.dropDatabase();
   await testDb.close();

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.6.0',
+  version: '5.10.0',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/classes.collection.ts

@@ -1,7 +1,7 @@
 import * as plugins from './plugins.js';
 import { SmartdataDb } from './classes.db.js';
 import { SmartdataDbCursor } from './classes.cursor.js';
-import { SmartDataDbDoc, type IIndexOptions } from './classes.doc.js';
+import { SmartDataDbDoc, type IIndexOptions, getSearchableFields } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { CollectionFactory } from './classes.collectionfactory.js';
 
@@ -128,6 +128,8 @@ export class SmartdataCollection<T> {
   public smartdataDb: SmartdataDb;
   public uniqueIndexes: string[] = [];
   public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+  // flag to ensure text index is created only once
+  private textIndexCreated: boolean = false;
 
   constructor(classNameArg: string, smartDataDbArg: SmartdataDb) {
     // tell the collection where it belongs
@@ -153,6 +155,16 @@ export class SmartdataCollection<T> {
         console.log(`Successfully initiated Collection ${this.collectionName}`);
       }
       this.mongoDbCollection = this.smartdataDb.mongoDb.collection(this.collectionName);
+      // Auto-create a compound text index on all searchable fields
+      const searchableFields = getSearchableFields(this.collectionName);
+      if (searchableFields.length > 0 && !this.textIndexCreated) {
+        // Build a compound text index spec
+        const indexSpec: Record<string, 'text'> = {};
+        searchableFields.forEach(f => { indexSpec[f] = 'text'; });
+        // Cast to any to satisfy TypeScript IndexSpecification typing
+        await this.mongoDbCollection.createIndex(indexSpec as any, { name: 'smartdata_text_index' });
+        this.textIndexCreated = true;
+      }
     }
   }
 

ts/classes.cursor.ts

@@ -15,14 +15,14 @@ export class SmartdataDbCursor<T = any> {
     this.smartdataDbDoc = dbDocArg;
   }
 
-  public async next(closeAtEnd = true) {
+  public async next(closeAtEnd = true): Promise<T> {
     const result = this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(
       await this.mongodbCursor.next(),
     );
     if (!result && closeAtEnd) {
       await this.close();
     }
-    return result;
+    return result as T;
   }
 
   public async forEach(forEachFuncArg: (itemArg: T) => Promise<any>, closeCursorAtEnd = true) {
@@ -40,6 +40,11 @@ export class SmartdataDbCursor<T = any> {
     }
   }
 
+  public async toArray(): Promise<T[]> {
+    const result = await this.mongodbCursor.toArray();
+    return result.map((itemArg) => this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(itemArg)) as T[];
+  }
+
   public async close() {
     await this.mongodbCursor.close();
   }

ts/classes.doc.ts

@@ -61,6 +61,10 @@ export function getSearchableFields(className: string): string[] {
   }
   return Array.from(searchableFieldsMap.get(className));
 }
+// Escape user input for safe use in MongoDB regular expressions
+function escapeForRegex(input: string): string {
+  return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
 
 /**
  * unique index - decorator to mark a unique index
@@ -249,6 +253,20 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
     return cursor;
   }
 
+  public static async getCursorExtended<T>(
+    this: plugins.tsclass.typeFest.Class<T>,
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    modifierFunction = (cursorArg: plugins.mongodb.FindCursor<plugins.mongodb.WithId<plugins.mongodb.BSON.Document>>) => cursorArg,
+  ): Promise<SmartdataDbCursor<T>> {
+    const collection: SmartdataCollection<T> = (this as any).collection;
+    await collection.init();
+    let cursor: plugins.mongodb.FindCursor<any> = collection.mongoDbCollection.find(
+      convertFilterForMongoDb(filterArg),
+    );
+    cursor = modifierFunction(cursor);
+    return new SmartdataDbCursor<T>(cursor, this as any as typeof SmartDataDbDoc);
+  }
+
   /**
    * watch the collection
    * @param this
@@ -312,57 +330,88 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   }
 
   /**
-   * Search documents using Lucene query syntax
-   * @param luceneQuery Lucene query string
+   * Search documents by text or field:value syntax, with safe regex fallback
+   * @param query A search term or field:value expression
    * @returns Array of matching documents
    */
   public static async search<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string,
+    query: string,
   ): Promise<T[]> {
-    const filter = (this as any).createSearchFilter(luceneQuery);
-    return await (this as any).getInstances(filter);
-  }
-
-  /**
-   * Search documents using Lucene query syntax with robust error handling
-   * @param luceneQuery The Lucene query string to search with
-   * @returns Array of matching documents
-   */
-  public static async searchWithLucene<T>(
-    this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string,
-  ): Promise<T[]> {
-    try {
-      const className = (this as any).className || this.name;
-      const searchableFields = getSearchableFields(className);
-
-      if (searchableFields.length === 0) {
-        console.warn(
-          `No searchable fields defined for class ${className}, falling back to simple search`,
-        );
-        return (this as any).searchByTextAcrossFields(luceneQuery);
-      }
-
-      // Simple term search optimization
-      if (
-        !luceneQuery.includes(':') &&
-        !luceneQuery.includes(' AND ') &&
-        !luceneQuery.includes(' OR ') &&
-        !luceneQuery.includes(' NOT ')
-      ) {
-        return (this as any).searchByTextAcrossFields(luceneQuery);
-      }
-
-      // Try to use the Lucene-to-MongoDB conversion
-      const filter = (this as any).createSearchFilter(luceneQuery);
-      return await (this as any).getInstances(filter);
-    } catch (error) {
-      console.error(`Error in searchWithLucene: ${error.message}`);
-      return (this as any).searchByTextAcrossFields(luceneQuery);
+    const className = (this as any).className || this.name;
+    const searchableFields = getSearchableFields(className);
+    if (searchableFields.length === 0) {
+      throw new Error(`No searchable fields defined for class ${className}`);
     }
+    // empty query -> return all
+    const q = query.trim();
+    if (!q) {
+      return await (this as any).getInstances({});
+    }
+    // simple exact field:value (no spaces, no wildcards, no quotes)
+    const simpleExact = q.match(/^(\w+):([^"'\*\?\s]+)$/);
+    if (simpleExact) {
+      const field = simpleExact[1];
+      const value = simpleExact[2];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${className}`);
+      }
+      return await (this as any).getInstances({ [field]: value });
+    }
+    // quoted phrase across all searchable fields: exact match of phrase
+    const quoted = q.match(/^"(.+)"$|^'(.+)'$/);
+    if (quoted) {
+      const phrase = quoted[1] || quoted[2] || '';
+      // build regex that matches the exact phrase (allowing flexible whitespace)
+      const parts = phrase.split(/\s+/).map((t) => escapeForRegex(t));
+      const pattern = parts.join('\\s+');
+      const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
+      return await (this as any).getInstances({ $or: orConds });
+    }
+    // wildcard field:value (supports * and ?) -> direct regex on that field
+    const wildcardField = q.match(/^(\w+):(.+[*?].*)$/);
+    if (wildcardField) {
+      const field = wildcardField[1];
+      const pattern = wildcardField[2];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${className}`);
+      }
+      // escape regex special chars except * and ?, then convert wildcards
+      const escaped = pattern.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+      const regexPattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+      return await (this as any).getInstances({ [field]: { $regex: regexPattern, $options: 'i' } });
+    }
+    // wildcard plain term across all fields (supports * and ?)
+    if (!q.includes(':') && (q.includes('*') || q.includes('?'))) {
+      // build wildcard regex pattern: escape all except * and ? then convert
+      const escaped = q.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+      const pattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+      const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
+      return await (this as any).getInstances({ $or: orConds });
+    }
+    // detect advanced Lucene syntax: field:value, wildcards, boolean, grouping
+    const luceneSyntax = /(\w+:[^\s]+)|\*|\?|\bAND\b|\bOR\b|\bNOT\b|\(|\)/;
+    if (luceneSyntax.test(q)) {
+      const filter = (this as any).createSearchFilter(q);
+      return await (this as any).getInstances(filter);
+    }
+    // multi-term unquoted -> AND of regex across fields for each term
+    const terms = q.split(/\s+/);
+    if (terms.length > 1) {
+      const andConds = terms.map((term) => {
+        const esc = escapeForRegex(term);
+        const ors = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
+        return { $or: ors };
+      });
+      return await (this as any).getInstances({ $and: andConds });
+    }
+    // single term -> regex across all searchable fields
+    const esc = escapeForRegex(q);
+    const orConds = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
+    return await (this as any).getInstances({ $or: orConds });
   }
 
+
   /**
    * Search by text across all searchable fields (fallback method)
    * @param searchText The text to search for in all searchable fields

ts/classes.lucene.adapter.ts

@@ -329,7 +329,16 @@ export class LuceneParser {
  * FIXED VERSION - proper MongoDB query structure
  */
 export class LuceneToMongoTransformer {
-  constructor() {}
+  private defaultFields: string[];
+  constructor(defaultFields: string[] = []) {
+    this.defaultFields = defaultFields;
+  }
+  /**
+   * Escape special characters for use in RegExp patterns
+   */
+  private escapeRegex(input: string): string {
+    return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+  }
 
   /**
    * Transform a Lucene AST node to a MongoDB query
@@ -366,18 +375,21 @@ export class LuceneToMongoTransformer {
    * FIXED: properly structured $or query for multiple fields
    */
   private transformTerm(node: TermNode, searchFields?: string[]): any {
-    // If specific fields are provided, search across those fields
-    if (searchFields && searchFields.length > 0) {
-      // Create an $or query to search across multiple fields
-      const orConditions = searchFields.map((field) => ({
-        [field]: { $regex: node.value, $options: 'i' },
-      }));
-
-      return { $or: orConditions };
+    // Build regex pattern, support wildcard (*) and fuzzy (?) if present
+    const term = node.value;
+    // Determine regex pattern: wildcard conversion or exact escape
+    let pattern: string;
+    if (term.includes('*') || term.includes('?')) {
+      pattern = this.luceneWildcardToRegex(term);
+    } else {
+      pattern = this.escapeRegex(term);
     }
-
-    // Otherwise, use text search (requires a text index on desired fields)
-    return { $text: { $search: node.value } };
+    // Search across provided fields or default fields
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
+    }));
+    return { $or: orConditions };
   }
 
   /**
@@ -385,17 +397,14 @@ export class LuceneToMongoTransformer {
    * FIXED: properly structured $or query for multiple fields
    */
   private transformPhrase(node: PhraseNode, searchFields?: string[]): any {
-    // If specific fields are provided, search phrase across those fields
-    if (searchFields && searchFields.length > 0) {
-      const orConditions = searchFields.map((field) => ({
-        [field]: { $regex: `${node.value.replace(/\s+/g, '\\s+')}`, $options: 'i' },
-      }));
-
-      return { $or: orConditions };
-    }
-
-    // For phrases, we use a regex to ensure exact matches
-    return { $text: { $search: `"${node.value}"` } };
+    // Use regex across provided fields or default fields, respecting word boundaries
+    const parts = node.value.split(/\s+/).map((t) => this.escapeRegex(t));
+    const pattern = parts.join('\\s+');
+    const fields = searchFields && searchFields.length > 0 ? searchFields : this.defaultFields;
+    const orConditions = fields.map((field) => ({
+      [field]: { $regex: pattern, $options: 'i' },
+    }));
+    return { $or: orConditions };
   }
 
   /**
@@ -429,9 +438,14 @@ export class LuceneToMongoTransformer {
       };
     }
 
-    // Special case for exact term matches on fields
+    // Special case for exact term matches on fields (supporting wildcard characters)
     if (node.value.type === 'TERM') {
-      return { [node.field]: { $regex: (node.value as TermNode).value, $options: 'i' } };
+      const val = (node.value as TermNode).value;
+      if (val.includes('*') || val.includes('?')) {
+        const regex = this.luceneWildcardToRegex(val);
+        return { [node.field]: { $regex: regex, $options: 'i' } };
+      }
+      return { [node.field]: { $regex: val, $options: 'i' } };
     }
 
     // Special case for phrase matches on fields
@@ -691,7 +705,8 @@ export class SmartdataLuceneAdapter {
    */
   constructor(defaultSearchFields?: string[]) {
     this.parser = new LuceneParser();
-    this.transformer = new LuceneToMongoTransformer();
+    // Pass default searchable fields into transformer
+    this.transformer = new LuceneToMongoTransformer(defaultSearchFields || []);
     if (defaultSearchFields) {
       this.defaultSearchFields = defaultSearchFields;
     }
@@ -704,7 +719,7 @@ export class SmartdataLuceneAdapter {
    */
   convert(luceneQuery: string, searchFields?: string[]): any {
     try {
-      // For simple single term queries, create a simpler query structure
+      // For simple single-term queries (no field:, boolean, grouping), use simpler regex
       if (
         !luceneQuery.includes(':') &&
         !luceneQuery.includes(' AND ') &&
@@ -713,13 +728,17 @@ export class SmartdataLuceneAdapter {
         !luceneQuery.includes('(') &&
         !luceneQuery.includes('[')
       ) {
-        // This is a simple term, use a more direct approach
         const fieldsToSearch = searchFields || this.defaultSearchFields;
-
         if (fieldsToSearch && fieldsToSearch.length > 0) {
+          // Handle wildcard characters in query
+          let pattern = luceneQuery;
+          if (luceneQuery.includes('*') || luceneQuery.includes('?')) {
+            // Use transformer to convert wildcard pattern
+            pattern = this.transformer.luceneWildcardToRegex(luceneQuery);
+          }
           return {
             $or: fieldsToSearch.map((field) => ({
-              [field]: { $regex: luceneQuery, $options: 'i' },
+              [field]: { $regex: pattern, $options: 'i' },
             })),
           };
         }