5.9.0

changelog.md

@@ -1,5 +1,58 @@
 # Changelog
 
+## 2025-04-18 - 5.9.0 - feat(collections/search)
+Improve text index creation and search fallback mechanisms in collections and document search methods
+
+- Auto-create a compound text index on all searchable fields in SmartdataCollection with a one-time flag to prevent duplicate index creation.
+- Refine the search method in SmartDataDbDoc to support exact field matches and safe regex fallback for non-Lucene queries.
+
+## 2025-04-17 - 5.8.4 - fix(core)
+Update commit metadata with no functional code changes
+
+- Commit info and documentation refreshed
+- No code or test changes detected in the diff
+
+## 2025-04-17 - 5.8.3 - fix(readme)
+Improve readme documentation on data models and connection management
+
+- Clarify that data models use @Collection, @unI, @svDb, @index, and @searchable decorators
+- Document that ObjectId and Buffer fields are stored as BSON types natively without extra decorators
+- Update connection management section to use 'db.close()' instead of 'db.disconnect()'
+- Revise license section to reference the MIT License without including additional legal details
+
+## 2025-04-14 - 5.8.2 - fix(classes.doc.ts)
+Ensure collection initialization before creating a cursor in getCursorExtended
+
+- Added 'await collection.init()' to guarantee that the MongoDB collection is initialized before using the cursor
+- Prevents potential runtime errors when accessing collection.mongoDbCollection
+
+## 2025-04-14 - 5.8.1 - fix(cursor, doc)
+Add explicit return types and casts to SmartdataDbCursor methods and update getCursorExtended signature in SmartDataDbDoc.
+
+- Specify Promise<T> as return type for next() in SmartdataDbCursor and cast return value to T.
+- Specify Promise<T[]> as return type for toArray() in SmartdataDbCursor and cast return value to T[].
+- Update getCursorExtended to return Promise<SmartdataDbCursor<T>> for clearer type safety.
+
+## 2025-04-14 - 5.8.0 - feat(cursor)
+Add toArray method to SmartdataDbCursor to convert raw MongoDB documents into initialized class instances
+
+- Introduced asynchronous toArray method in SmartdataDbCursor which retrieves all documents from the MongoDB cursor
+- Maps each native document to a SmartDataDbDoc instance using createInstanceFromMongoDbNativeDoc for consistent API usage
+
+## 2025-04-14 - 5.7.0 - feat(SmartDataDbDoc)
+Add extended cursor method getCursorExtended for flexible cursor modifications
+
+- Introduces getCursorExtended in classes.doc.ts to allow modifier functions for MongoDB cursors
+- Wraps the modified cursor with SmartdataDbCursor for improved API consistency
+- Enhances querying capabilities by enabling customized cursor transformations
+
+## 2025-04-07 - 5.6.0 - feat(indexing)
+Add support for regular index creation in documents and collections
+
+- Implement new index decorator in classes.doc.ts to mark properties with regular indexing options
+- Update SmartdataCollection to create regular indexes if defined on a document during insert
+- Enhance document structure to store and utilize regular index configurations
+
 ## 2025-04-06 - 5.5.1 - fix(ci & formatting)
 Minor fixes: update CI workflow image and npmci package references, adjust package.json and readme URLs, and apply consistent code formatting.
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.5.1",
+  "version": "5.9.0",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",

readme.md

@@ -66,7 +66,7 @@ await db.init();
 
 ### Defining Data Models
 
-Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
+Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, `@svDb`, `@index`, and `@searchable` to define your data models. Fields of type `ObjectId` or `Buffer` decorated with `@svDb()` will be stored as BSON ObjectId and Binary, respectively; no separate `@oid()` or `@bin()` decorators are required.
 
 ```typescript
 import {
@@ -74,8 +74,6 @@ import {
   Collection,
   unI,
   svDb,
-  oid,
-  bin,
   index,
   searchable,
 } from '@push.rocks/smartdata';
@@ -96,12 +94,10 @@ class User extends SmartDataDbDoc<User, User> {
   public email: string; // Mark 'email' to be saved in DB
 
   @svDb()
-  @oid() // Automatically handle as ObjectId type
-  public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
+  public organizationId: ObjectId; // Stored as BSON ObjectId
 
   @svDb()
-  @bin() // Automatically handle as Binary data
-  public profilePicture: Buffer; // Will be automatically converted to/from Binary
+  public profilePicture: Buffer; // Stored as BSON Binary
 
   @svDb({
     serialize: (data) => JSON.stringify(data), // Custom serialization
@@ -541,7 +537,7 @@ class Order extends SmartDataDbDoc<Order, Order> {
 ### Connection Management
 
 - Always call `db.init()` before using any database features
-- Use `db.disconnect()` when shutting down your application
+- Use `db.close()` when shutting down your application
 - Set appropriate connection pool sizes based on your application's needs
 
 ### Document Design
@@ -590,7 +586,7 @@ Please make sure to update tests as appropriate and follow our coding standards.
 
 ## License and Legal Information
 
-This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
+This repository is licensed under the MIT License. For details, see [MIT License](https://opensource.org/licenses/MIT).
 
 **Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
 

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.5.1',
+  version: '5.9.0',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/classes.collection.ts

@@ -1,7 +1,7 @@
 import * as plugins from './plugins.js';
 import { SmartdataDb } from './classes.db.js';
 import { SmartdataDbCursor } from './classes.cursor.js';
-import { SmartDataDbDoc } from './classes.doc.js';
+import { SmartDataDbDoc, type IIndexOptions, getSearchableFields } from './classes.doc.js';
 import { SmartdataDbWatcher } from './classes.watcher.js';
 import { CollectionFactory } from './classes.collectionfactory.js';
 
@@ -127,6 +127,9 @@ export class SmartdataCollection<T> {
   public collectionName: string;
   public smartdataDb: SmartdataDb;
   public uniqueIndexes: string[] = [];
+  public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+  // flag to ensure text index is created only once
+  private textIndexCreated: boolean = false;
 
   constructor(classNameArg: string, smartDataDbArg: SmartdataDb) {
     // tell the collection where it belongs
@@ -152,6 +155,14 @@ export class SmartdataCollection<T> {
         console.log(`Successfully initiated Collection ${this.collectionName}`);
       }
       this.mongoDbCollection = this.smartdataDb.mongoDb.collection(this.collectionName);
+      // Auto-create a compound text index on all searchable fields
+      const searchableFields = getSearchableFields(this.collectionName);
+      if (searchableFields.length > 0 && !this.textIndexCreated) {
+        const indexSpec: { [key: string]: string } = {};
+        searchableFields.forEach(f => { indexSpec[f] = 'text'; });
+        await this.mongoDbCollection.createIndex(indexSpec, { name: 'smartdata_text_index' });
+        this.textIndexCreated = true;
+      }
     }
   }
 
@@ -170,6 +181,24 @@ export class SmartdataCollection<T> {
     }
   }
 
+  /**
+   * creates regular indexes for the collection
+   */
+  public createRegularIndexes(indexesArg: Array<{field: string, options: IIndexOptions}> = []) {
+    for (const indexDef of indexesArg) {
+      // Check if we've already created this index
+      const indexKey = indexDef.field;
+      if (!this.regularIndexes.some(i => i.field === indexKey)) {
+        this.mongoDbCollection.createIndex(
+          { [indexDef.field]: 1 }, // Simple single-field index
+          indexDef.options
+        );
+        // Track that we've created this index
+        this.regularIndexes.push(indexDef);
+      }
+    }
+  }
+
   /**
    * adds a validation function that all newly inserted and updated objects have to pass
    */
@@ -238,6 +267,12 @@ export class SmartdataCollection<T> {
     await this.init();
     await this.checkDoc(dbDocArg);
     this.markUniqueIndexes(dbDocArg.uniqueIndexes);
+    
+    // Create regular indexes if available
+    if (dbDocArg.regularIndexes && dbDocArg.regularIndexes.length > 0) {
+      this.createRegularIndexes(dbDocArg.regularIndexes);
+    }
+    
     const saveableObject = await dbDocArg.createSavableObject();
     const result = await this.mongoDbCollection.insertOne(saveableObject);
     return result;
@@ -295,4 +330,4 @@ export class SmartdataCollection<T> {
     }
     return done.promise;
   }
-}
+}

ts/classes.cursor.ts

@@ -15,14 +15,14 @@ export class SmartdataDbCursor<T = any> {
     this.smartdataDbDoc = dbDocArg;
   }
 
-  public async next(closeAtEnd = true) {
+  public async next(closeAtEnd = true): Promise<T> {
     const result = this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(
       await this.mongodbCursor.next(),
     );
     if (!result && closeAtEnd) {
       await this.close();
     }
-    return result;
+    return result as T;
   }
 
   public async forEach(forEachFuncArg: (itemArg: T) => Promise<any>, closeCursorAtEnd = true) {
@@ -40,6 +40,11 @@ export class SmartdataDbCursor<T = any> {
     }
   }
 
+  public async toArray(): Promise<T[]> {
+    const result = await this.mongodbCursor.toArray();
+    return result.map((itemArg) => this.smartdataDbDoc.createInstanceFromMongoDbNativeDoc(itemArg)) as T[];
+  }
+
   public async close() {
     await this.mongodbCursor.close();
   }

ts/classes.doc.ts

@@ -61,6 +61,10 @@ export function getSearchableFields(className: string): string[] {
   }
   return Array.from(searchableFieldsMap.get(className));
 }
+// Escape user input for safe use in MongoDB regular expressions
+function escapeForRegex(input: string): string {
+  return input.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
+}
 
 /**
  * unique index - decorator to mark a unique index
@@ -83,6 +87,46 @@ export function unI() {
   };
 }
 
+/**
+ * Options for MongoDB indexes
+ */
+export interface IIndexOptions {
+  background?: boolean;
+  unique?: boolean;
+  sparse?: boolean;
+  expireAfterSeconds?: number;
+  [key: string]: any;
+}
+
+/**
+ * index - decorator to mark a field for regular indexing
+ */
+export function index(options?: IIndexOptions) {
+  return (target: SmartDataDbDoc<unknown, unknown>, key: string) => {
+    console.log(`called index() on >${target.constructor.name}.${key}<`);
+    
+    // Initialize regular indexes array if it doesn't exist
+    if (!target.regularIndexes) {
+      target.regularIndexes = [];
+    }
+    
+    // Add this field to regularIndexes with its options
+    target.regularIndexes.push({
+      field: key,
+      options: options || {}
+    });
+    
+    // Also ensure it's marked as saveable
+    if (!target.saveableProperties) {
+      target.saveableProperties = [];
+    }
+    
+    if (!target.saveableProperties.includes(key)) {
+      target.saveableProperties.push(key);
+    }
+  };
+}
+
 export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
   // Special case: detect MongoDB operators and pass them through directly
   const topLevelOperators = ['$and', '$or', '$nor', '$not', '$text', '$where', '$regex'];
@@ -209,6 +253,20 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
     return cursor;
   }
 
+  public static async getCursorExtended<T>(
+    this: plugins.tsclass.typeFest.Class<T>,
+    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    modifierFunction = (cursorArg: plugins.mongodb.FindCursor<plugins.mongodb.WithId<plugins.mongodb.BSON.Document>>) => cursorArg,
+  ): Promise<SmartdataDbCursor<T>> {
+    const collection: SmartdataCollection<T> = (this as any).collection;
+    await collection.init();
+    let cursor: plugins.mongodb.FindCursor<any> = collection.mongoDbCollection.find(
+      convertFilterForMongoDb(filterArg),
+    );
+    cursor = modifierFunction(cursor);
+    return new SmartdataDbCursor<T>(cursor, this as any as typeof SmartDataDbDoc);
+  }
+
   /**
    * watch the collection
    * @param this
@@ -272,57 +330,38 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   }
 
   /**
-   * Search documents using Lucene query syntax
-   * @param luceneQuery Lucene query string
+   * Search documents by text or field:value syntax, with safe regex fallback
+   * @param query A search term or field:value expression
    * @returns Array of matching documents
    */
   public static async search<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string,
+    query: string,
   ): Promise<T[]> {
-    const filter = (this as any).createSearchFilter(luceneQuery);
-    return await (this as any).getInstances(filter);
-  }
-
-  /**
-   * Search documents using Lucene query syntax with robust error handling
-   * @param luceneQuery The Lucene query string to search with
-   * @returns Array of matching documents
-   */
-  public static async searchWithLucene<T>(
-    this: plugins.tsclass.typeFest.Class<T>,
-    luceneQuery: string,
-  ): Promise<T[]> {
-    try {
-      const className = (this as any).className || this.name;
-      const searchableFields = getSearchableFields(className);
-
-      if (searchableFields.length === 0) {
-        console.warn(
-          `No searchable fields defined for class ${className}, falling back to simple search`,
-        );
-        return (this as any).searchByTextAcrossFields(luceneQuery);
-      }
-
-      // Simple term search optimization
-      if (
-        !luceneQuery.includes(':') &&
-        !luceneQuery.includes(' AND ') &&
-        !luceneQuery.includes(' OR ') &&
-        !luceneQuery.includes(' NOT ')
-      ) {
-        return (this as any).searchByTextAcrossFields(luceneQuery);
-      }
-
-      // Try to use the Lucene-to-MongoDB conversion
-      const filter = (this as any).createSearchFilter(luceneQuery);
-      return await (this as any).getInstances(filter);
-    } catch (error) {
-      console.error(`Error in searchWithLucene: ${error.message}`);
-      return (this as any).searchByTextAcrossFields(luceneQuery);
+    const className = (this as any).className || this.name;
+    const searchableFields = getSearchableFields(className);
+    if (searchableFields.length === 0) {
+      throw new Error(`No searchable fields defined for class ${className}`);
     }
+    // field:value exact match (case-sensitive for non-regex fields)
+    const fv = query.match(/^(\w+):(.+)$/);
+    if (fv) {
+      const field = fv[1];
+      const value = fv[2];
+      if (!searchableFields.includes(field)) {
+        throw new Error(`Field '${field}' is not searchable for class ${className}`);
+      }
+      return await (this as any).getInstances({ [field]: value });
+    }
+    // safe regex across all searchable fields (case-insensitive)
+    const escaped = escapeForRegex(query);
+    const orConditions = searchableFields.map((field) => ({
+      [field]: { $regex: escaped, $options: 'i' },
+    }));
+    return await (this as any).getInstances({ $or: orConditions });
   }
 
+
   /**
    * Search by text across all searchable fields (fallback method)
    * @param searchText The text to search for in all searchable fields
@@ -401,6 +440,11 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public uniqueIndexes: string[];
 
+  /**
+   * regular indexes with their options
+   */
+  public regularIndexes: Array<{field: string, options: IIndexOptions}> = [];
+
   /**
    * an array of saveable properties of a specific doc
    */
@@ -503,4 +547,4 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
     }
     return identifiableObject;
   }
-}
+}