5.13.0

changelog.md

@@ -1,5 +1,69 @@
 # Changelog
 
+## 2025-04-22 - 5.13.0 - feat(search)
+Improve search query handling and update documentation
+
+- Added 'codex.md' providing a high-level project overview and detailed search API documentation.
+- Enhanced search parsing in SmartDataDbDoc to support combined free-term and quoted field phrase queries.
+- Introduced a new fallback branch in the search method to handle free term with quoted field input.
+- Updated tests in test/test.search.ts to cover new combined query scenarios and ensure robust behavior.
+
+## 2025-04-22 - 5.12.2 - fix(search)
+Fix handling of quoted wildcard patterns in field-specific search queries and add tests for location-based wildcard phrase searches
+
+- Strip surrounding quotes from wildcard patterns in field queries to correctly transform them to regex
+- Introduce new tests in test/test.search.ts to validate exact quoted and unquoted wildcard searches on a location field
+
+## 2025-04-22 - 5.12.1 - fix(search)
+Improve implicit AND logic for mixed free term and field queries in search and enhance wildcard field handling.
+
+- Updated regex for field:value parsing to capture full value with wildcards.
+- Added explicit handling for free terms by converting to regex across searchable fields.
+- Improved error messaging for attempts to search non-searchable fields.
+- Extended tests to cover combined free term and wildcard field searches, including error cases.
+
+## 2025-04-22 - 5.12.0 - feat(doc/search)
+Enhance search functionality with filter and validate options for advanced query control
+
+- Added 'filter' option to merge additional MongoDB query constraints in search
+- Introduced 'validate' hook to post-process and filter fetched documents
+- Refactored underlying execQuery function to support additional search options
+- Updated tests to cover new search scenarios and fallback mechanisms
+
+## 2025-04-22 - 5.11.4 - fix(search)
+Implement implicit AND logic for mixed simple term and field:value queries in search
+
+- Added a new branch to detect and handle search queries that mix field:value pairs with plain terms without explicit operators
+- Builds an implicit $and filter when query parts contain colon(s) but lack explicit boolean operators or quotes
+- Ensures proper parsing and improved robustness of search filters
+
+## 2025-04-22 - 5.11.3 - fix(lucene adapter and search tests)
+Improve range query parsing in Lucene adapter and expand search test coverage
+
+- Added a new 'testSearch' script in package.json to run search tests.
+- Introduced advanced search tests for range queries and combined field filters in test/search.advanced.ts.
+- Enhanced robustness tests in test/search.ts for wildcard and empty query scenarios.
+- Fixed token validation in the parseRange method of the Lucene adapter to ensure proper error handling.
+
+## 2025-04-21 - 5.11.2 - fix(readme)
+Update readme to clarify usage of searchable fields retrieval
+
+- Replaced getSearchableFields('Product') with Product.getSearchableFields()
+- Updated documentation to reference the static method Class.getSearchableFields()
+
+## 2025-04-21 - 5.11.1 - fix(doc)
+Refactor searchable fields API and improve collection registration.
+
+- Removed the standalone getSearchableFields utility in favor of a static method on document classes.
+- Updated tests to use the new static method (e.g., Product.getSearchableFields()).
+- Ensured the Collection decorator attaches a docCtor property to correctly register searchable fields.
+- Added try/catch in test cleanup to gracefully handle dropDatabase errors.
+
+## 2025-04-21 - 5.11.0 - feat(ts/classes.lucene.adapter)
+Expose luceneWildcardToRegex method to allow external usage and enhance regex transformation capabilities.
+
+- Changed luceneWildcardToRegex from private to public in ts/classes.lucene.adapter.ts.
+
 ## 2025-04-21 - 5.10.0 - feat(search)
 Improve search functionality: update documentation, refine Lucene query transformation, and add advanced search tests
 

codex.md

@@ -0,0 +1,77 @@
+# SmartData Project Overview
+
+This document provides a high-level overview of the SmartData library (`@push.rocks/smartdata`), its architecture, core components, and key features—including recent enhancements to the search API.
+
+## 1. Project Purpose
+- A TypeScript‑first wrapper around MongoDB that supplies:
+  - Strongly‑typed document & collection classes
+  - Decorator‑based schema definition (no external schema files)
+  - Advanced search capabilities with Lucene‑style queries
+  - Built‑in support for real‑time data sync, distributed coordination, and key‑value EasyStore
+
+## 2. Core Concepts & Components
+- **SmartDataDb**: Manages the MongoDB connection, pooling, and initialization of collections.
+- **SmartDataDbDoc**: Base class for all document models; provides CRUD, upsert, and cursor APIs.
+- **Decorators**:
+  - `@Collection`: Associates a class with a MongoDB collection
+  - `@svDb()`: Marks a field as persisted to the DB
+  - `@unI()`: Marks a field as a unique index
+  - `@index()`: Adds a regular index
+  - `@searchable()`: Marks a field for inclusion in text searches or regex queries
+- **SmartdataCollection**: Wraps a MongoDB collection; auto‑creates indexes based on decorators.
+- **Lucene Adapter**: Parses a Lucene query string into an AST and transforms it to a MongoDB filter object.
+- **EasyStore**: A simple, schema‑less key‑value store built on top of MongoDB for sharing ephemeral data.
+- **Distributed Coordinator**: Leader election and task‑distribution API for building resilient, multi‑instance systems.
+- **Watcher**: Listens to change streams for real‑time updates and integrates with RxJS.
+
+## 3. Search API
+SmartData provides a unified `.search(query[, opts])` method on all models with `@searchable()` fields:
+
+- **Supported Syntax**:
+  1. Exact field:value (e.g. `field:Value`)
+  2. Quoted phrases (e.g. `"exact phrase"` or `'exact phrase'`)
+  3. Wildcards: `*` (zero or more chars) and `?` (single char)
+  4. Boolean operators: `AND`, `OR`, `NOT`
+  5. Grouping: parenthesis `(A OR B) AND C`
+  6. Range queries: `[num TO num]`, `{num TO num}`
+  7. Multi‑term unquoted: terms AND’d across all searchable fields
+  8. Empty query returns all documents
+
+- **Fallback Mechanisms**:
+  1. Text index based `$text` search (if supported)
+  2. Field‑scoped and multi‑field regex queries
+  3. In‑memory filtering for complex or unsupported cases
+
+### New Security & Extensibility Hooks
+The `.search(query, opts?)` signature now accepts a `SearchOptions<T>` object:
+```ts
+interface SearchOptions<T> {
+  filter?: Record<string, any>;        // Additional MongoDB filter AND‑merged
+  validate?: (doc: T) => boolean;      // Post‑fetch hook to drop results
+}
+```
+- **filter**: Enforces mandatory constraints (e.g. multi‑tenant isolation) directly in the Mongo query.
+- **validate**: An async function that runs after fetching; return `false` to exclude a document.
+
+## 4. Testing Strategy
+- Unit tests in `test/test.search.ts` cover basic search functionality and new options:
+  - Exact, wildcard, phrase, boolean and grouping cases
+  - Implicit AND and mixed free‑term + field searches
+  - Edge cases (non‑searchable fields, quoted wildcards, no matches)
+  - `filter` and `validate` tests ensure security hooks work as intended
+- Advanced search scenarios are covered in `test/test.search.advanced.ts`.
+
+## 5. Usage Example
+```ts
+// Basic search
+const prods = await Product.search('wireless earbuds');
+
+// Scoped search (only your organization’s items)
+const myItems = await Product.search('book', { filter: { ownerId } });
+
+// Post‑search validation (only cheap items)
+const cheapItems = await Product.search('', { validate: p => p.price < 50 });
+```
+
+---
+Last updated: 2025-04-22

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.10.0",
+  "version": "5.13.0",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",
@@ -8,6 +8,7 @@
   "type": "module",
   "scripts": {
     "test": "tstest test/",
+    "testSearch": "tsx test/test.search.ts",
     "build": "tsbuild --web --allowimplicitany",
     "buildDocs": "tsdoc"
   },

readme.md

@@ -203,7 +203,7 @@ class Product extends SmartDataDbDoc<Product, Product> {
 }
 
 // List searchable fields
-const searchableFields = getSearchableFields('Product');
+const searchableFields = Product.getSearchableFields();
 
 // 1: Exact phrase across all fields
 await Product.search('"Kindle Paperwhite"');
@@ -225,12 +225,16 @@ await Product.search('TypeScript Aufgabe');
 
 // 7: Empty query returns all documents
 await Product.search('');
+// 8: Scoped search with additional filter (e.g. multi-tenant isolation)
+await Product.search('book', { filter: { ownerId: currentUserId } });
+// 9: Post-search validation hook to drop unwanted results (e.g. price check)
+await Product.search('', { validate: (p) => p.price < 100 });
 ```
 
 The search functionality includes:
 
 - `@searchable()` decorator for marking fields as searchable
-- `getSearchableFields()` to list searchable fields for a model
+- `Class.getSearchableFields()` static method to list searchable fields for a model
 - `search(query: string)` method supporting:
   - Exact phrase matches (`"my exact string"` or `'my exact string'`)
   - Field‑scoped exact & wildcard searches (`field:value`, `field:Air*`)

test/test.search.advanced.ts

@@ -1,7 +1,7 @@
 import { tap, expect } from '@push.rocks/tapbundle';
 import * as smartmongo from '@push.rocks/smartmongo';
 import * as smartdata from '../ts/index.js';
-import { searchable, getSearchableFields } from '../ts/classes.doc.js';
+import { searchable } from '../ts/classes.doc.js';
 import { smartunique } from '../ts/plugins.js';
 
 // Set up database connection
@@ -172,6 +172,21 @@ tap.test('grouping: (Furniture OR Electronics) AND Chair', async () => {
   expect(names).toEqual(['Gaming Chair', 'Office Chair']);
 });
 
+// Additional range and combined query tests
+tap.test('range query price:[30 TO 300] returns expected products', async () => {
+  const res = await Product.search('price:[30 TO 300]');
+  // Expect products with price between 30 and 300 inclusive: Day Light Lamp, Gaming Chair, Office Chair, AirPods
+  expect(res.length).toEqual(4);
+  const names = res.map((r) => r.name).sort();
+  expect(names).toEqual(['AirPods', 'Day Light Lamp', 'Gaming Chair', 'Office Chair']);
+});
+
+tap.test('should filter category and price range', async () => {
+  const res = await Product.search('category:Lighting AND price:[30 TO 40]');
+  expect(res.length).toEqual(1);
+  expect(res[0].name).toEqual('Day Light Lamp');
+});
+
 // Teardown
 tap.test('cleanup advanced search database', async () => {
   await testDb.mongoDb.dropDatabase();

test/test.search.ts

@@ -4,11 +4,13 @@ import { smartunique } from '../ts/plugins.js';
 
 // Import the smartdata library
 import * as smartdata from '../ts/index.js';
-import { searchable, getSearchableFields } from '../ts/classes.doc.js';
+import { searchable } from '../ts/classes.doc.js';
 
 // Set up database connection
 let smartmongoInstance: smartmongo.SmartMongo;
 let testDb: smartdata.SmartdataDb;
+// Class for location-based wildcard/phrase tests
+let LocationDoc: any;
 
 // Define a test class with searchable fields using the standard SmartDataDbDoc
 @smartdata.Collection(() => testDb)
@@ -72,7 +74,7 @@ tap.test('should create test products with searchable fields', async () => {
 
 tap.test('should retrieve searchable fields for a class', async () => {
   // Use the getSearchableFields function to verify our searchable fields
-  const searchableFields = getSearchableFields('Product');
+  const searchableFields = Product.getSearchableFields();
   console.log('Searchable fields:', searchableFields);
 
   expect(searchableFields.length).toEqual(3);
@@ -257,6 +259,143 @@ tap.test('should search hyphenated terms "E-reader"', async () => {
   expect(ereaderResults[0].name).toEqual('Kindle Paperwhite');
 });
 
+// Additional robustness tests
+tap.test('should return all products for empty search', async () => {
+  const searchResults = await Product.search('');
+  const allProducts = await Product.getInstances({});
+  expect(searchResults.length).toEqual(allProducts.length);
+});
+
+tap.test('should support wildcard plain term across all fields', async () => {
+  const results = await Product.search('*book*');
+  const names = results.map((r) => r.name).sort();
+  expect(names).toEqual(['Harry Potter', 'Kindle Paperwhite', 'MacBook Pro']);
+});
+
+tap.test('should support wildcard plain term with question mark pattern', async () => {
+  const results = await Product.search('?one?');
+  const names = results.map((r) => r.name).sort();
+  expect(names).toEqual(['Galaxy S21', 'iPhone 12']);
+});
+
+// Filter and Validation tests
+tap.test('should apply filter option to restrict results', async () => {
+  // search term 'book' across all fields but restrict to Books category
+  const bookFiltered = await Product.search('book', { filter: { category: 'Books' } });
+  expect(bookFiltered.length).toEqual(2);
+  bookFiltered.forEach((p) => expect(p.category).toEqual('Books'));
+});
+tap.test('should apply validate hook to post-filter results', async () => {
+  // return only products with price > 500
+  const expensive = await Product.search('', { validate: (p) => p.price > 500 });
+  expect(expensive.length).toBeGreaterThan(0);
+  expensive.forEach((p) => expect(p.price).toBeGreaterThan(500));
+});
+
+// Tests for quoted and wildcard field-specific phrases
+tap.test('setup location test products', async () => {
+  @smartdata.Collection(() => testDb)
+  class LD extends smartdata.SmartDataDbDoc<LD, LD> {
+    @smartdata.unI() public id: string = smartunique.shortId();
+    @smartdata.svDb() @searchable() public location: string;
+    constructor(loc: string) { super(); this.location = loc; }
+  }
+  // Assign to outer variable for subsequent tests
+  LocationDoc = LD;
+  const locations = ['Berlin', 'Frankfurt am Main', 'Frankfurt am Oder', 'London'];
+  for (const loc of locations) {
+    await new LocationDoc(loc).save();
+  }
+});
+tap.test('should search exact quoted field phrase', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt am Main"');
+  expect(results.length).toEqual(1);
+  expect(results[0].location).toEqual('Frankfurt am Main');
+});
+tap.test('should search wildcard quoted field phrase', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt am *"');
+  const names = results.map((d: any) => d.location).sort();
+  expect(names).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+tap.test('should search unquoted wildcard field', async () => {
+  const results = await (LocationDoc as any).search('location:Frankfurt*');
+  const names = results.map((d: any) => d.location).sort();
+  expect(names).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+
+// Combined free-term + field phrase/wildcard tests
+let CombinedDoc: any;
+tap.test('setup combined docs for free-term and location tests', async () => {
+  @smartdata.Collection(() => testDb)
+  class CD extends smartdata.SmartDataDbDoc<CD, CD> {
+    @smartdata.unI() public id: string = smartunique.shortId();
+    @smartdata.svDb() @searchable() public name: string;
+    @smartdata.svDb() @searchable() public location: string;
+    constructor(name: string, location: string) { super(); this.name = name; this.location = location; }
+  }
+  CombinedDoc = CD;
+  const docs = [
+    new CombinedDoc('TypeScript', 'Berlin'),
+    new CombinedDoc('TypeScript', 'Frankfurt am Main'),
+    new CombinedDoc('TypeScript', 'Frankfurt am Oder'),
+    new CombinedDoc('JavaScript', 'Berlin'),
+  ];
+  for (const d of docs) await d.save();
+});
+tap.test('should search free term and exact quoted field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Berlin"');
+  expect(res.length).toEqual(1);
+  expect(res[0].location).toEqual('Berlin');
+});
+tap.test('should not match free term with non-matching quoted field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Frankfurt d"');
+  expect(res.length).toEqual(0);
+});
+tap.test('should search free term with quoted wildcard field phrase', async () => {
+  const res = await CombinedDoc.search('TypeScript location:"Frankfurt am *"');
+  const locs = res.map((r: any) => r.location).sort();
+  expect(locs).toEqual(['Frankfurt am Main', 'Frankfurt am Oder']);
+});
+// Quoted exact field phrase without wildcard should return no matches if no exact match
+tap.test('should not match location:"Frankfurt d"', async () => {
+  const results = await (LocationDoc as any).search('location:"Frankfurt d"');
+  expect(results.length).toEqual(0);
+});
+
+// Combined free-term and field wildcard tests
+tap.test('should combine free term and wildcard field search', async () => {
+  const results = await Product.search('book category:Book*');
+  expect(results.length).toEqual(2);
+  results.forEach((p) => expect(p.category).toEqual('Books'));
+});
+tap.test('should not match when free term matches but wildcard field does not', async () => {
+  const results = await Product.search('book category:Kitchen*');
+  expect(results.length).toEqual(0);
+});
+
+// Non-searchable field should cause an error for combined queries
+tap.test('should throw when combining term with non-searchable field', async () => {
+  let error: Error;
+  try {
+    await Product.search('book location:Berlin');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+tap.test('should throw when combining term with non-searchable wildcard field', async () => {
+  let error: Error;
+  try {
+    await Product.search('book location:Berlin*');
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error.message).toMatch(/not searchable/);
+});
+
+// Close database connection
 tap.test('close database connection', async () => {
   await testDb.mongoDb.dropDatabase();
   await testDb.close();

test/test.watch.ts

@@ -64,7 +64,11 @@ tap.test('should watch a collection', async (toolsArg) => {
 // close the database connection
 // =======================================
 tap.test('close', async () => {
+  try {
     await testDb.mongoDb.dropDatabase();
+  } catch (err) {
+    console.warn('dropDatabase error ignored in cleanup:', err.message || err);
+  }
   await testDb.close();
   if (smartmongoInstance) {
     await smartmongoInstance.stop();

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.10.0',
+  version: '5.13.0',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/classes.collection.ts

@@ -1,7 +1,7 @@
 import * as plugins from './plugins.js';
 import { SmartdataDb } from './classes.db.js';
 import { SmartdataDbCursor } from './classes.cursor.js';
-import { SmartDataDbDoc, type IIndexOptions, getSearchableFields } from './classes.doc.js';
+import { SmartDataDbDoc, type IIndexOptions } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { CollectionFactory } from './classes.collectionfactory.js';
 
@@ -32,13 +32,22 @@ export function Collection(dbArg: SmartdataDb | TDelayed<SmartdataDb>) {
         if (!(dbArg instanceof SmartdataDb)) {
           dbArg = dbArg();
         }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        const coll = collectionFactory.getCollection(constructor.name, dbArg);
+        // Attach document constructor for searchableFields lookup
+        if (!(coll as any).docCtor) {
+          (coll as any).docCtor = decoratedClass;
+        }
+        return coll;
       }
       public get collection() {
         if (!(dbArg instanceof SmartdataDb)) {
           dbArg = dbArg();
         }
-        return collectionFactory.getCollection(constructor.name, dbArg);
+        const coll = collectionFactory.getCollection(constructor.name, dbArg);
+        if (!(coll as any).docCtor) {
+          (coll as any).docCtor = decoratedClass;
+        }
+        return coll;
       }
     };
     return decoratedClass;
@@ -156,7 +165,9 @@ export class SmartdataCollection<T> {
       }
       this.mongoDbCollection = this.smartdataDb.mongoDb.collection(this.collectionName);
       // Auto-create a compound text index on all searchable fields
-      const searchableFields = getSearchableFields(this.collectionName);
+      // Use document constructor's searchableFields registered via decorator
+      const docCtor = (this as any).docCtor;
+      const searchableFields: string[] = docCtor?.searchableFields || [];
       if (searchableFields.length > 0 && !this.textIndexCreated) {
         // Build a compound text index spec
         const indexSpec: Record<string, 'text'> = {};

ts/classes.doc.ts

@@ -5,11 +5,19 @@ import { SmartdataDbCursor } from './classes.cursor.js';
 import { type IManager, SmartdataCollection } from './classes.collection.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { SmartdataLuceneAdapter } from './classes.lucene.adapter.js';
+/**
+ * Search options for `.search()`:
+ * - filter: additional MongoDB query to AND-merge
+ * - validate: post-fetch validator, return true to keep a doc
+ */
+export interface SearchOptions<T> {
+  filter?: Record<string, any>;
+  validate?: (doc: T) => Promise<boolean> | boolean;
+}
 
 export type TDocCreation = 'db' | 'new' | 'mixed';
 
-// Set of searchable fields for each class
+
-const searchableFieldsMap = new Map<string, Set<string>>();
 
 export function globalSvDb() {
   return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
@@ -39,28 +47,15 @@ export function svDb() {
  */
 export function searchable() {
   return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
-    console.log(`called searchable() on >${target.constructor.name}.${key}<`);
+    // Attach to class constructor for direct access
-
+    const ctor = target.constructor as any;
-    // Initialize the set for this class if it doesn't exist
+    if (!Array.isArray(ctor.searchableFields)) {
-    const className = target.constructor.name;
+      ctor.searchableFields = [];
-    if (!searchableFieldsMap.has(className)) {
-      searchableFieldsMap.set(className, new Set<string>());
     }
-
+    ctor.searchableFields.push(key);
-    // Add the property to the searchable fields set
-    searchableFieldsMap.get(className).add(key);
   };
 }
 
-/**
- * Get searchable fields for a class
- */
-export function getSearchableFields(className: string): string[] {
-  if (!searchableFieldsMap.has(className)) {
-    return [];
-  }
-  return Array.from(searchableFieldsMap.get(className));
-}
 // Escape user input for safe use in MongoDB regular expressions
 function escapeForRegex(input: string): string {
   return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
@@ -318,68 +313,103 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
     this: plugins.tsclass.typeFest.Class<T>,
     luceneQuery: string,
   ): any {
-    const className = (this as any).className || this.name;
+    const searchableFields = (this as any).getSearchableFields();
-    const searchableFields = getSearchableFields(className);
-
     if (searchableFields.length === 0) {
-      throw new Error(`No searchable fields defined for class ${className}`);
+      throw new Error(`No searchable fields defined for class ${this.name}`);
     }
-
     const adapter = new SmartdataLuceneAdapter(searchableFields);
     return adapter.convert(luceneQuery);
   }
+  /**
+   * List all searchable fields defined on this class
+   */
+  public static getSearchableFields(): string[] {
+    const ctor = this as any;
+    return Array.isArray(ctor.searchableFields) ? ctor.searchableFields : [];
+  }
+  /**
+   * Execute a query with optional hard filter and post-fetch validation
+   */
+  private static async execQuery<T>(
+    this: plugins.tsclass.typeFest.Class<T>,
+    baseFilter: Record<string, any>,
+    opts?: SearchOptions<T>
+  ): Promise<T[]> {
+    let mongoFilter = baseFilter || {};
+    if (opts?.filter) {
+      mongoFilter = { $and: [mongoFilter, opts.filter] };
+    }
+    let docs: T[] = await (this as any).getInstances(mongoFilter);
+    if (opts?.validate) {
+      const out: T[] = [];
+      for (const d of docs) {
+        if (await opts.validate(d)) out.push(d);
+      }
+      docs = out;
+    }
+    return docs;
+  }
 
   /**
    * Search documents by text or field:value syntax, with safe regex fallback
+   * Supports additional filtering and post-fetch validation via opts
    * @param query A search term or field:value expression
+   * @param opts Optional filter and validate hooks
    * @returns Array of matching documents
    */
   public static async search<T>(
     this: plugins.tsclass.typeFest.Class<T>,
     query: string,
+    opts?: SearchOptions<T>,
   ): Promise<T[]> {
-    const className = (this as any).className || this.name;
+    const searchableFields = (this as any).getSearchableFields();
-    const searchableFields = getSearchableFields(className);
     if (searchableFields.length === 0) {
-      throw new Error(`No searchable fields defined for class ${className}`);
+      throw new Error(`No searchable fields defined for class ${this.name}`);
     }
     // empty query -> return all
     const q = query.trim();
     if (!q) {
-      return await (this as any).getInstances({});
+      // empty query: fetch all, apply opts
+      return await (this as any).execQuery({}, opts);
     }
     // simple exact field:value (no spaces, no wildcards, no quotes)
+    // simple exact field:value (no spaces, wildcards, quotes)
     const simpleExact = q.match(/^(\w+):([^"'\*\?\s]+)$/);
     if (simpleExact) {
       const field = simpleExact[1];
       const value = simpleExact[2];
       if (!searchableFields.includes(field)) {
-        throw new Error(`Field '${field}' is not searchable for class ${className}`);
+        throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
       }
-      return await (this as any).getInstances({ [field]: value });
+      // simple field:value search
+      return await (this as any).execQuery({ [field]: value }, opts);
     }
     // quoted phrase across all searchable fields: exact match of phrase
     const quoted = q.match(/^"(.+)"$|^'(.+)'$/);
     if (quoted) {
       const phrase = quoted[1] || quoted[2] || '';
-      // build regex that matches the exact phrase (allowing flexible whitespace)
       const parts = phrase.split(/\s+/).map((t) => escapeForRegex(t));
       const pattern = parts.join('\\s+');
       const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
-      return await (this as any).getInstances({ $or: orConds });
+      return await (this as any).execQuery({ $or: orConds }, opts);
     }
     // wildcard field:value (supports * and ?) -> direct regex on that field
     const wildcardField = q.match(/^(\w+):(.+[*?].*)$/);
     if (wildcardField) {
       const field = wildcardField[1];
-      const pattern = wildcardField[2];
+      // Support quoted wildcard patterns: strip surrounding quotes
+      let pattern = wildcardField[2];
+      if ((pattern.startsWith('"') && pattern.endsWith('"')) ||
+          (pattern.startsWith("'") && pattern.endsWith("'"))) {
+        pattern = pattern.slice(1, -1);
+      }
       if (!searchableFields.includes(field)) {
-        throw new Error(`Field '${field}' is not searchable for class ${className}`);
+        throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
       }
       // escape regex special chars except * and ?, then convert wildcards
       const escaped = pattern.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
       const regexPattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
-      return await (this as any).getInstances({ [field]: { $regex: regexPattern, $options: 'i' } });
+      return await (this as any).execQuery({ [field]: { $regex: regexPattern, $options: 'i' } }, opts);
     }
     // wildcard plain term across all fields (supports * and ?)
     if (!q.includes(':') && (q.includes('*') || q.includes('?'))) {
@@ -387,13 +417,70 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
       const escaped = q.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
       const pattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
       const orConds = searchableFields.map((f) => ({ [f]: { $regex: pattern, $options: 'i' } }));
-      return await (this as any).getInstances({ $or: orConds });
+      return await (this as any).execQuery({ $or: orConds }, opts);
+    }
+    // implicit AND: combine free terms and field:value terms (with or without wildcards)
+    const parts = q.split(/\s+/);
+    const hasColon = parts.some((t) => t.includes(':'));
+    if (
+      parts.length > 1 && hasColon &&
+      !q.includes(' AND ') && !q.includes(' OR ') && !q.includes(' NOT ') &&
+      !q.includes('(') && !q.includes(')') &&
+      !q.includes('[') && !q.includes(']') &&
+      !q.includes('"') && !q.includes("'")
+    ) {
+      const andConds = parts.map((term) => {
+        const m = term.match(/^(\w+):(.+)$/);
+        if (m) {
+          const field = m[1];
+          const value = m[2];
+          if (!searchableFields.includes(field)) {
+            throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
+          }
+          if (value.includes('*') || value.includes('?')) {
+            // wildcard field search
+            const escaped = value.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+            const pattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+            return { [field]: { $regex: pattern, $options: 'i' } };
+          }
+          // exact field:value
+          return { [field]: value };
+        }
+        // free term -> regex across all searchable fields
+        const esc = escapeForRegex(term);
+        return { $or: searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } })) };
+      });
+      return await (this as any).execQuery({ $and: andConds }, opts);
+    }
+
+    // free term and quoted field phrase (exact or wildcard), e.g. 'term field:"phrase"' or 'term field:"ph*se"'
+    const freeWithQuotedField = q.match(/^(\S+)\s+(\w+):"(.+)"$/);
+    if (freeWithQuotedField) {
+      const freeTerm = freeWithQuotedField[1];
+      const field = freeWithQuotedField[2];
+      let phrase = freeWithQuotedField[3];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${this.name}`);
+      }
+      // free term condition across all searchable fields
+      const freeEsc = escapeForRegex(freeTerm);
+      const freeCond = { $or: searchableFields.map((f) => ({ [f]: { $regex: freeEsc, $options: 'i' } })) };
+      // field condition: exact match or wildcard pattern
+      let fieldCond;
+      if (phrase.includes('*') || phrase.includes('?')) {
+        const escaped = phrase.replace(/([.+^${}()|[\\]\\])/g, '\\$1');
+        const pattern = escaped.replace(/\*/g, '.*').replace(/\?/g, '.');
+        fieldCond = { [field]: { $regex: pattern, $options: 'i' } };
+      } else {
+        fieldCond = { [field]: phrase };
+      }
+      return await (this as any).execQuery({ $and: [freeCond, fieldCond] }, opts);
     }
     // detect advanced Lucene syntax: field:value, wildcards, boolean, grouping
     const luceneSyntax = /(\w+:[^\s]+)|\*|\?|\bAND\b|\bOR\b|\bNOT\b|\(|\)/;
     if (luceneSyntax.test(q)) {
       const filter = (this as any).createSearchFilter(q);
-      return await (this as any).getInstances(filter);
+      return await (this as any).execQuery(filter, opts);
     }
     // multi-term unquoted -> AND of regex across fields for each term
     const terms = q.split(/\s+/);
@@ -403,63 +490,16 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
         const ors = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
         return { $or: ors };
       });
-      return await (this as any).getInstances({ $and: andConds });
+      return await (this as any).execQuery({ $and: andConds }, opts);
     }
     // single term -> regex across all searchable fields
     const esc = escapeForRegex(q);
     const orConds = searchableFields.map((f) => ({ [f]: { $regex: esc, $options: 'i' } }));
-    return await (this as any).getInstances({ $or: orConds });
+    return await (this as any).execQuery({ $or: orConds }, opts);
   }
 
 
-  /**
+  // INSTANCE
-   * Search by text across all searchable fields (fallback method)
-   * @param searchText The text to search for in all searchable fields
-   * @returns Array of matching documents
-   */
-  private static async searchByTextAcrossFields<T>(
-    this: plugins.tsclass.typeFest.Class<T>,
-    searchText: string,
-  ): Promise<T[]> {
-    try {
-      const className = (this as any).className || this.name;
-      const searchableFields = getSearchableFields(className);
-
-      // Fallback to direct filter if we have searchable fields
-      if (searchableFields.length > 0) {
-        // Create a simple $or query with regex for each field
-        const orConditions = searchableFields.map((field) => ({
-          [field]: { $regex: searchText, $options: 'i' },
-        }));
-
-        const filter = { $or: orConditions };
-
-        try {
-          // Try with MongoDB filter first
-          return await (this as any).getInstances(filter);
-        } catch (error) {
-          console.warn('MongoDB filter failed, falling back to in-memory search');
-        }
-      }
-
-      // Last resort: get all and filter in memory
-      const allDocs = await (this as any).getInstances({});
-      const lowerSearchText = searchText.toLowerCase();
-
-      return allDocs.filter((doc: any) => {
-        for (const field of searchableFields) {
-          const value = doc[field];
-          if (value && typeof value === 'string' && value.toLowerCase().includes(lowerSearchText)) {
-            return true;
-          }
-        }
-        return false;
-      });
-    } catch (error) {
-      console.error(`Error in searchByTextAcrossFields: ${error.message}`);
-      return [];
-    }
-  }
 
   // INSTANCE
 

ts/classes.lucene.adapter.ts

@@ -290,11 +290,11 @@ export class LuceneParser {
     const includeLower = this.tokens[this.pos] === '[';
     const includeUpper = this.tokens[this.pos + 4] === ']';
 
-    this.pos++; // Skip open bracket
+    // Ensure tokens for lower, TO, upper, and closing bracket exist
-
     if (this.pos + 4 >= this.tokens.length) {
       throw new Error('Invalid range query syntax');
     }
+    this.pos++; // Skip open bracket
 
     const lower = this.tokens[this.pos];
     this.pos++;
@@ -640,7 +640,7 @@ export class LuceneToMongoTransformer {
   /**
    * Convert Lucene wildcards to MongoDB regex patterns
    */
-  private luceneWildcardToRegex(wildcardPattern: string): string {
+  public luceneWildcardToRegex(wildcardPattern: string): string {
     // Replace Lucene wildcards with regex equivalents
     // * => .*
     // ? => .