5.16.3

- Introduced a detailed guide on query filters in the README, covering basic filtering, comparison operators, array operators, logical operators, element operators, and advanced filtering patterns.
- Implemented a type-safe filtering system in `classes.doc.ts` with `MongoFilterCondition` and `MongoFilter` types to support MongoDB operators while maintaining nested type safety.
- Enhanced error handling for invalid operators and conditions in the filtering logic.
- Added extensive tests for various filtering scenarios, including basic, comparison, array, logical, and complex filters, ensuring robust functionality and performance.
- Implemented security measures to prevent the use of dangerous operators like `$where` and validate operator usage.

changelog.md

@@ -1,5 +1,20 @@
 # Changelog
 
+## 2025-08-18 - 5.16.3 - fix(docs)
+Add local Claude settings and remove outdated codex.md
+
+- Added .claude/settings.local.json to store local Claude/assistant permissions and configuration.
+- Removed codex.md (project overview) — documentation file deleted.
+- No runtime/library code changes; documentation/configuration-only update, bump patch version.
+
+## 2025-08-18 - 5.16.2 - fix(readme)
+Update README: clarify examples, expand search/cursor/docs and add local Claude settings
+
+- Refined README wording and structure: clearer Quick Start, improved examples and developer-focused phrasing
+- Expanded documentation for search, cursors, change streams, distributed coordination, transactions and EasyStore with more concrete code examples
+- Adjusted code examples to show safer defaults (ID generation, status/tags, connection pooling) and improved best-practices guidance
+- Added .claude/settings.local.json to provide local assistant/CI permission configuration
+
 ## 2025-08-12 - 5.16.1 - fix(core)
 Improve error handling and logging; enhance search query sanitization; update dependency versions and documentation
 

codex.md

@@ -1,77 +0,0 @@
-# SmartData Project Overview
-
-This document provides a high-level overview of the SmartData library (`@push.rocks/smartdata`), its architecture, core components, and key features—including recent enhancements to the search API.
-
-## 1. Project Purpose
-- A TypeScript‑first wrapper around MongoDB that supplies:
-  - Strongly‑typed document & collection classes
-  - Decorator‑based schema definition (no external schema files)
-  - Advanced search capabilities with Lucene‑style queries
-  - Built‑in support for real‑time data sync, distributed coordination, and key‑value EasyStore
-
-## 2. Core Concepts & Components
-- **SmartDataDb**: Manages the MongoDB connection, pooling, and initialization of collections.
-- **SmartDataDbDoc**: Base class for all document models; provides CRUD, upsert, and cursor APIs.
-- **Decorators**:
-  - `@Collection`: Associates a class with a MongoDB collection
-  - `@svDb()`: Marks a field as persisted to the DB
-  - `@unI()`: Marks a field as a unique index
-  - `@index()`: Adds a regular index
-  - `@searchable()`: Marks a field for inclusion in text searches or regex queries
-- **SmartdataCollection**: Wraps a MongoDB collection; auto‑creates indexes based on decorators.
-- **Lucene Adapter**: Parses a Lucene query string into an AST and transforms it to a MongoDB filter object.
-- **EasyStore**: A simple, schema‑less key‑value store built on top of MongoDB for sharing ephemeral data.
-- **Distributed Coordinator**: Leader election and task‑distribution API for building resilient, multi‑instance systems.
-- **Watcher**: Listens to change streams for real‑time updates and integrates with RxJS.
-
-## 3. Search API
-SmartData provides a unified `.search(query[, opts])` method on all models with `@searchable()` fields:
-
-- **Supported Syntax**:
-  1. Exact field:value (e.g. `field:Value`)
-  2. Quoted phrases (e.g. `"exact phrase"` or `'exact phrase'`)
-  3. Wildcards: `*` (zero or more chars) and `?` (single char)
-  4. Boolean operators: `AND`, `OR`, `NOT`
-  5. Grouping: parenthesis `(A OR B) AND C`
-  6. Range queries: `[num TO num]`, `{num TO num}`
-  7. Multi‑term unquoted: terms AND’d across all searchable fields
-  8. Empty query returns all documents
-
-- **Fallback Mechanisms**:
-  1. Text index based `$text` search (if supported)
-  2. Field‑scoped and multi‑field regex queries
-  3. In‑memory filtering for complex or unsupported cases
-
-### New Security & Extensibility Hooks
-The `.search(query, opts?)` signature now accepts a `SearchOptions<T>` object:
-```ts
-interface SearchOptions<T> {
-  filter?: Record<string, any>;        // Additional MongoDB filter AND‑merged
-  validate?: (doc: T) => boolean;      // Post‑fetch hook to drop results
-}
-```
-- **filter**: Enforces mandatory constraints (e.g. multi‑tenant isolation) directly in the Mongo query.
-- **validate**: An async function that runs after fetching; return `false` to exclude a document.
-
-## 4. Testing Strategy
-- Unit tests in `test/test.search.ts` cover basic search functionality and new options:
-  - Exact, wildcard, phrase, boolean and grouping cases
-  - Implicit AND and mixed free‑term + field searches
-  - Edge cases (non‑searchable fields, quoted wildcards, no matches)
-  - `filter` and `validate` tests ensure security hooks work as intended
-- Advanced search scenarios are covered in `test/test.search.advanced.ts`.
-
-## 5. Usage Example
-```ts
-// Basic search
-const prods = await Product.search('wireless earbuds');
-
-// Scoped search (only your organization’s items)
-const myItems = await Product.search('book', { filter: { ownerId } });
-
-// Post‑search validation (only cheap items)
-const cheapItems = await Product.search('', { validate: p => p.price < 50 });
-```
-
----
-Last updated: 2025-04-22

package.json

@@ -1,13 +1,13 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.16.1",
+  "version": "5.16.3",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",
   "typings": "dist_ts/index.d.ts",
   "type": "module",
   "scripts": {
-    "test": "tstest test/ --verbose",
+    "test": "tstest test/ --verbose.  --logfile --timeout 120",
     "testSearch": "tsx test/test.search.ts",
     "build": "tsbuild --web --allowimplicitany",
     "buildDocs": "tsdoc"
@@ -37,10 +37,10 @@
     "mongodb": "^6.18.0"
   },
   "devDependencies": {
-    "@git.zone/tsbuild": "^2.6.4",
+    "@git.zone/tsbuild": "^2.6.7",
     "@git.zone/tsrun": "^1.2.44",
-    "@git.zone/tstest": "^2.3.2",
+    "@git.zone/tstest": "^2.3.5",
-    "@push.rocks/qenv": "^6.0.5",
+    "@push.rocks/qenv": "^6.1.3",
     "@push.rocks/tapbundle": "^6.0.3",
     "@types/node": "^22.15.2"
   },

readme.md

@@ -2,28 +2,28 @@
 
 [![npm version](https://badge.fury.io/js/@push.rocks%2Fsmartdata.svg)](https://www.npmjs.com/package/@push.rocks/smartdata)
 
-**The ultimate TypeScript-first MongoDB wrapper** that makes database operations beautiful, type-safe, and incredibly powerful. Built for modern applications that demand real-time performance, distributed coordination, and rock-solid reliability.
+**The ultimate TypeScript-first MongoDB wrapper** that turns your database into a type-safe powerhouse. Built for modern applications that demand real-time performance, distributed coordination, and rock-solid reliability.
 
 ## 🌟 Why SmartData?
 
-SmartData isn't just another MongoDB wrapper - it's a complete data management powerhouse that transforms how you work with databases:
+SmartData isn't just another MongoDB wrapper - it's a complete paradigm shift in how you work with databases:
 
-- 🔒 **100% Type-Safe**: Full TypeScript with decorators, generics, and deep query typing
+- 🔒 **100% Type-Safe**: Full TypeScript with decorators, generics, and compile-time query validation
-- ⚡ **Lightning Fast**: Connection pooling, cursor streaming, and optimized indexing
+- ⚡ **Lightning Fast**: Connection pooling, cursor streaming, and intelligent indexing
-- 🔄 **Real-time Sync**: MongoDB Change Streams with RxJS for reactive applications
+- 🔄 **Real-time Ready**: MongoDB Change Streams with RxJS for reactive applications
-- 🌍 **Distributed Ready**: Built-in leader election and task coordination
+- 🌍 **Distributed Systems**: Built-in leader election and task coordination
-- 🛡️ **Security First**: NoSQL injection prevention, credential encoding, and secure defaults
+- 🛡️ **Security First**: NoSQL injection prevention, automatic sanitization, and secure defaults
-- 🎯 **Developer Friendly**: Intuitive API, powerful search, and amazing DX
+- 🎯 **Developer Experience**: Intuitive API, powerful search, and IntelliSense that just works
 
 ## 📦 Installation
 
 ```bash
-# Using npm
-npm install @push.rocks/smartdata --save
-
 # Using pnpm (recommended)
 pnpm add @push.rocks/smartdata
 
+# Using npm
+npm install @push.rocks/smartdata
+
 # Using yarn
 yarn add @push.rocks/smartdata
 ```
@@ -48,13 +48,13 @@ const db = new SmartdataDb({
   mongoDbUser: 'username',
   mongoDbPass: 'password',
   
-  // Optional: Configure connection pooling (new!)
+  // Optional: Advanced connection pooling
-  maxPoolSize: 100,          // Max connections in pool (default: 100)
+  maxPoolSize: 100,          // Max connections in pool
-  maxIdleTimeMS: 300000,     // Max idle time (default: 5 minutes)
+  maxIdleTimeMS: 300000,     // Max idle time before connection close
-  serverSelectionTimeoutMS: 30000  // Connection timeout (default: 30s)
+  serverSelectionTimeoutMS: 30000  // Connection timeout
 });
 
-// Initialize with automatic retry and connection pooling
+// Initialize with automatic retry and health monitoring
 await db.init();
 ```
 
@@ -74,22 +74,25 @@ import { ObjectId } from 'mongodb';
 @Collection(() => db)
 class User extends SmartDataDbDoc<User, User> {
   @unI()
-  public id: string = 'unique-user-id';  // Unique index
+  public id: string;  // Unique index with automatic ID generation
   
   @svDb()
-  @searchable()  // Enable full-text search
+  @searchable()  // Enable Lucene-style searching
   public username: string;
   
   @svDb()
   @searchable()
-  @index({ unique: false })  // Regular index for performance
+  @index({ unique: false })  // Performance index
   public email: string;
   
   @svDb()
-  public organizationId: ObjectId;  // Automatically handled as BSON ObjectId
+  public status: 'active' | 'inactive' | 'pending';  // Full union type support
   
   @svDb()
-  public profilePicture: Buffer;    // Automatically handled as BSON Binary
+  public organizationId: ObjectId;  // Native MongoDB types
+  
+  @svDb()
+  public profilePicture: Buffer;    // Binary data support
   
   @svDb({
     // Custom serialization for complex objects
@@ -98,6 +101,9 @@ class User extends SmartDataDbDoc<User, User> {
   })
   public preferences: Record<string, any>;
   
+  @svDb()
+  public tags: string[];  // Array support with operators
+  
   @svDb()
   public createdAt: Date = new Date();
   
@@ -105,6 +111,7 @@ class User extends SmartDataDbDoc<User, User> {
     super();
     this.username = username;
     this.email = email;
+    this.id = User.getNewId();
   }
 }
 ```
@@ -114,18 +121,17 @@ class User extends SmartDataDbDoc<User, User> {
 ```typescript
 // ✨ Create
 const user = new User('johndoe', 'john@example.com');
+user.status = 'active';
+user.tags = ['developer', 'typescript'];
 await user.save();
 
-// 🔍 Read
+// 🔍 Read - with full type safety
 const foundUser = await User.getInstance({ username: 'johndoe' });
-const allUsers = await User.getInstances({ email: 'john@example.com' });
+const activeUsers = await User.getInstances({ status: 'active' });
 
 // ✏️ Update
 foundUser.email = 'newemail@example.com';
-await foundUser.save();
+foundUser.tags.push('mongodb');
-
-// 🔄 Upsert (update or insert)
-// Note: Upsert is handled automatically by save() - if document exists it updates, otherwise inserts
 await foundUser.save();
 
 // 🗑️ Delete
@@ -134,9 +140,58 @@ await foundUser.delete();
 
 ## 🔥 Advanced Features
 
+### 🎯 Type-Safe Query Filters
+
+SmartData provides the most advanced type-safe filtering system for MongoDB, supporting all operators while maintaining full IntelliSense:
+
+```typescript
+// Union types work perfectly
+const users = await User.getInstances({
+  status: { $in: ['active', 'pending'] }  // TypeScript validates these values!
+});
+
+// Comparison operators with type checking
+const adults = await User.getInstances({
+  age: { $gte: 18, $lt: 65 }  // Type-safe numeric comparisons
+});
+
+// Array operations with full type safety
+const experts = await User.getInstances({
+  tags: { $all: ['typescript', 'mongodb'] },  // Must have all tags
+  skills: { $size: 5 }  // Exactly 5 skills
+});
+
+// Complex nested queries
+const results = await Order.getInstances({
+  $and: [
+    { status: { $in: ['pending', 'processing'] } },
+    { 'items.price': { $gte: 100 } },  // Dot notation for nested fields
+    { 
+      items: { 
+        $elemMatch: {  // Array element matching
+          product: 'laptop',
+          quantity: { $gte: 2 }
+        }
+      }
+    }
+  ]
+});
+
+// Logical operators
+const urgentOrHighValue = await Order.getInstances({
+  $or: [
+    { priority: 'urgent' },
+    { totalAmount: { $gte: 1000 } }
+  ]
+});
+
+// Security: $where is automatically blocked
+// await User.getInstances({ $where: '...' });  // ❌ Throws security error
+```
+
 ### 🔎 Powerful Search Engine
 
-SmartData includes a Lucene-style search engine with automatic field indexing:
+Built-in Lucene-style search with automatic indexing:
 
 ```typescript
 @Collection(() => db)
@@ -148,31 +203,31 @@ class Product extends SmartDataDbDoc<Product, Product> {
   @svDb() public price: number;
 }
 
-// 🎯 Exact phrase search
+// Simple text search across all searchable fields
-await Product.search('"MacBook Pro 16"');
+const results = await Product.search('laptop');
 
-// 🔤 Wildcard search
+// Field-specific search
-await Product.search('Mac*');
+const electronics = await Product.search('category:Electronics');
 
-// 📁 Field-specific search
+// Wildcard searches
-await Product.search('category:Electronics');
+const matches = await Product.search('Mac*');
 
-// 🧮 Boolean operators
+// Boolean operators
-await Product.search('(laptop OR desktop) AND NOT gaming');
+const query = await Product.search('laptop AND NOT gaming');
 
-// 🔐 Secure multi-field search
+// Phrase search
-await Product.search('TypeScript MongoDB'); // Automatically escaped
+const exact = await Product.search('"MacBook Pro 16"');
 
-// 🏷️ Scoped search with filters
+// Combined with filters for powerful queries
-await Product.search('laptop', {
+const affordable = await Product.search('laptop', {
-  filter: { price: { $lt: 2000 } },
+  filter: { price: { $lte: 1500 } },
-  validate: (p) => p.inStock === true
+  validate: async (p) => p.inStock === true
 });
 ```
 
 ### 💾 EasyStore - Type-Safe Key-Value Storage
 
-Perfect for configuration, caching, and shared state:
+Perfect for configuration, caching, and session management:
 
 ```typescript
 interface AppConfig {
@@ -190,7 +245,7 @@ interface AppConfig {
 // Create a type-safe store
 const config = await db.createEasyStore<AppConfig>('app-config');
 
-// Write with full IntelliSense
+// Write with full type checking
 await config.writeKey('features', {
   darkMode: true,
   notifications: false
@@ -198,123 +253,132 @@ await config.writeKey('features', {
 
 // Read with guaranteed types
 const features = await config.readKey('features');
-// TypeScript knows: features.darkMode is boolean
+// TypeScript knows: features.darkMode is boolean ✅
 
-// Atomic operations
+// Atomic updates
-await config.writeAll({
+await config.updateKey('limits', (current) => ({
-  apiKey: 'new-key',
+  ...current,
-  limits: { maxUsers: 1000, maxStorage: 5000 }
+  maxUsers: current.maxUsers + 100
-});
+}));
 
-// Delete a key
+// Delete keys
 await config.deleteKey('features');
 
-// Wipe entire store
+// Clear entire store
 await config.wipe();
 ```
 
-### 🌐 Distributed Coordination
-
-Build resilient distributed systems with automatic leader election:
-
-```typescript
-const coordinator = new SmartdataDistributedCoordinator(db);
-
-// Start coordination with automatic heartbeat
-await coordinator.start();
-
-// Check if this instance is the leader
-const eligibleLeader = await coordinator.getEligibleLeader();
-const isLeader = eligibleLeader?.id === coordinator.id;
-
-if (isLeader) {
-  console.log('🎖️ This instance is now the leader!');
-  // Leader-specific tasks are handled internally by leadFunction()
-  // The coordinator automatically manages leader election and failover
-}
-
-// Fire distributed task requests (for taskbuffer integration)
-const result = await coordinator.fireDistributedTaskRequest({
-  taskName: 'maintenance',
-  taskExecutionTime: Date.now(),
-  requestResponseId: 'unique-id'
-});
-
-// Graceful shutdown
-await coordinator.stop();
-```
-
 ### 📡 Real-Time Change Streams
 
 React to database changes instantly with RxJS integration:
 
 ```typescript
-// Watch for specific changes
+// Watch for changes with automatic reconnection
 const watcher = await User.watch(
-  { active: true },  // Only watch active users
+  { status: 'active' },  // Filter which documents to watch
-  {
+  { 
-    fullDocument: 'updateLookup',  // Include full document
+    fullDocument: 'updateLookup',  // Get full document on updates
-    bufferTimeMs: 100,  // Buffer for performance
+    bufferTimeMs: 100  // Buffer changes for efficiency
   }
 );
 
-// Subscribe with RxJS (emits documents or arrays if buffered)
+// Subscribe to changes with RxJS
-watcher.changeSubject
+watcher.changeSubject.subscribe({
-  .pipe(
+  next: (change) => {
-    filter(user => user !== null),  // Filter out deletions
+    console.log('User changed:', change.fullDocument);
-  )
+    
-  .subscribe(user => {
+    switch(change.operationType) {
-    console.log(`📢 User change detected: ${user.username}`);
+      case 'insert':
-    sendNotification(user.email);
+        console.log('New user created');
-  });
+        break;
-
+      case 'update':
-// Or use EventEmitter pattern
+        console.log('User updated');
-watcher.on('change', (user) => {
+        break;
-  if (user) {
+      case 'delete':
-    console.log(`✏️ User changed: ${user.username}`);
+        console.log('User deleted');
-  } else {
+        break;
-    console.log(`👋 User deleted`);
+    }
-  }
+  },
+  error: (err) => console.error('Watch error:', err)
 });
 
+// Advanced: Watch with aggregation pipeline
+const complexWatcher = await Order.watch(
+  { status: 'pending' },
+  {
+    pipeline: [
+      { $match: { 'fullDocument.totalAmount': { $gte: 1000 } } },
+      { $addFields: { isHighValue: true } }
+    ]
+  }
+);
+
 // Clean up when done
-await watcher.stop();
+await watcher.close();
 ```
 
-### 🎯 Cursor Operations for Large Datasets
+### 🌐 Distributed Coordination
 
-Handle millions of documents efficiently:
+Build resilient distributed systems with automatic leader election:
 
 ```typescript
-// Create a cursor with modifiers
+import { SmartdataDistributedCoordinator } from '@push.rocks/smartdata';
+
+const coordinator = new SmartdataDistributedCoordinator(db);
+
+// Start coordination with automatic heartbeat
+await coordinator.start();
+
+// Leader election happens automatically
+coordinator.asyncExecutionScheduler.addJobHandler('maintenance', async () => {
+  // This code only runs on the leader instance
+  console.log('🎖️ Running maintenance as leader');
+  await performDatabaseCleanup();
+});
+
+// Check leadership status
+const isLeader = coordinator.isLeader;
+
+// Fire distributed tasks (integrates with @push.rocks/taskbuffer)
+const result = await coordinator.fireDistributedTaskRequest({
+  taskName: 'process-payments',
+  taskExecutionTime: Date.now(),
+  requestResponseId: 'unique-id'
+});
+
+// Graceful shutdown with leadership handoff
+await coordinator.stop();
+```
+
+### 🔄 Cursor Streaming for Large Datasets
+
+Process millions of documents without memory issues:
+
+```typescript
+// Get a cursor for memory-efficient iteration
 const cursor = await User.getCursor(
-  { active: true },
+  { status: 'active' },
-  { 
+  {
+    // Optional: Use MongoDB native cursor modifiers
     modifier: (cursor) => cursor
       .sort({ createdAt: -1 })
-      .skip(100)
+      .limit(10000)
-      .limit(50)
+      .project({ email: 1, username: 1 })
   }
 );
 
-// Stream processing - memory efficient
+// Process one at a time
 await cursor.forEach(async (user) => {
-  await processUser(user);
+  await sendEmail(user.email);
-  // Processes one at a time, minimal memory usage
+  // Each document is loaded individually
 });
 
-// Manual iteration
+// Or use async iteration
-let user;
+for await (const user of cursor) {
-while (user = await cursor.next()) {
+  if (shouldStop(user)) break;
-  if (shouldStop(user)) {
+  await processUser(user);
-    break;
-  }
-  await handleUser(user);
 }
 
-// Convert to array (only for small datasets!)
-const users = await cursor.toArray();
-
 // Always clean up
 await cursor.close();
 ```
@@ -331,15 +395,17 @@ try {
     // All operations in this block are atomic
     const sender = await User.getInstance(
       { id: 'user-1' }, 
-      session  // Pass session to all operations
+      { session }  // Pass session to all operations
     );
+    
     sender.balance -= 100;
     await sender.save({ session });
     
     const receiver = await User.getInstance(
       { id: 'user-2' }, 
-      session
+      { session }
     );
+    
     receiver.balance += 100;
     await receiver.save({ session });
     
@@ -349,9 +415,7 @@ try {
     }
   });
   
-  console.log('✅ Transaction completed successfully');
+  console.log('✅ Transaction completed');
-} catch (error) {
-  console.error('❌ Transaction failed, rolled back');
 } finally {
   await session.endSession();
 }
@@ -365,13 +429,8 @@ Handle complex data types with custom serializers:
 class Document extends SmartDataDbDoc<Document, Document> {
   @svDb({
     // Encrypt sensitive data before storing
-    serialize: async (value) => {
+    serialize: async (value) => await encrypt(value),
-      return await encrypt(value);
+    deserialize: async (value) => await decrypt(value)
-    },
-    // Decrypt when reading
-    deserialize: async (value) => {
-      return await decrypt(value);
-    }
   })
   public sensitiveData: string;
   
@@ -383,11 +442,18 @@ class Document extends SmartDataDbDoc<Document, Document> {
   public largePayload: any;
   
   @svDb({
-    // Store sets as arrays
+    // Store Sets as arrays
     serialize: (set) => Array.from(set),
     deserialize: (arr) => new Set(arr)
   })
   public tags: Set<string>;
+  
+  @svDb({
+    // Handle custom date formats
+    serialize: (date) => date?.toISOString(),
+    deserialize: (str) => str ? new Date(str) : null
+  })
+  public scheduledAt: Date | null;
 }
 ```
 
@@ -399,171 +465,280 @@ Add custom logic at any point in the document lifecycle:
 @Collection(() => db)
 class Order extends SmartDataDbDoc<Order, Order> {
   @unI() public id: string;
-  @svDb() public items: OrderItem[];
+  @svDb() public items: Array<{ product: string; quantity: number }>;
-  @svDb() public total: number;
+  @svDb() public totalAmount: number;
-  @svDb() public status: 'pending' | 'paid' | 'shipped';
+  @svDb() public status: string;
   
-  // Validate and calculate before saving
+  // Called before saving (create or update)
   async beforeSave() {
-    this.total = this.items.reduce((sum, item) => 
+    // Recalculate total
+    this.totalAmount = this.items.reduce((sum, item) => 
       sum + (item.price * item.quantity), 0
     );
     
-    if (this.items.length === 0) {
+    // Validate
-      throw new Error('Order must have items!');
+    if (this.totalAmount < 0) {
+      throw new Error('Invalid order total');
     }
   }
   
-  // Send notifications after saving
+  // Called after successful save
   async afterSave() {
-    if (this.status === 'paid') {
+    // Send notifications
-      await sendOrderConfirmation(this);
+    await notificationService.orderUpdated(this.id);
-      await notifyWarehouse(this);
+    
-    }
+    // Update cache
+    await cache.set(`order:${this.id}`, this);
   }
   
-  // Prevent deletion of shipped orders
+  // Called before deletion
   async beforeDelete() {
-    if (this.status === 'shipped') {
+    // Archive order
-      throw new Error('Cannot delete shipped orders!');
+    await archive.store(this);
+    
+    // Check if deletion is allowed
+    if (this.status === 'completed') {
+      throw new Error('Cannot delete completed orders');
     }
   }
   
-  // Audit logging
+  // Called after successful deletion
   async afterDelete() {
-    await auditLog.record({
+    // Clean up related data
-      action: 'order_deleted',
+    await cache.delete(`order:${this.id}`);
-      orderId: this.id,
+    
-      timestamp: new Date()
+    // Log deletion
-    });
+    console.log(`Order ${this.id} deleted at ${new Date()}`);
   }
 }
 ```
 
-### 🔍 Deep Query Type Safety
+### 🚀 Performance Optimization
 
-TypeScript knows your nested object structure:
+#### Intelligent Indexing
 
 ```typescript
-interface UserProfile {
-  personal: {
-    name: {
-      first: string;
-      last: string;
-    };
-    age: number;
-  };
-  address: {
-    street: string;
-    city: string;
-    country: string;
-  };
-}
-
 @Collection(() => db)
-class Profile extends SmartDataDbDoc<Profile, Profile> {
+class HighPerformanceDoc extends SmartDataDbDoc<HighPerformanceDoc, HighPerformanceDoc> {
-  @unI() public id: string;
+  @unI()  // Unique index
-  @svDb() public data: UserProfile;
+  public id: string;
+  
+  @index()  // Single field index
+  public userId: string;
+  
+  @index({ sparse: true })  // Sparse index for optional fields
+  public deletedAt?: Date;
+  
+  @index({ 
+    unique: false,
+    background: true,  // Non-blocking index creation
+    expireAfterSeconds: 86400  // TTL index
+  })
+  public sessionToken: string;
+  
+  // Compound indexes for complex queries
+  static async createIndexes() {
+    await this.collection.createIndex(
+      { userId: 1, createdAt: -1 },  // Compound index
+      { name: 'user_activity_idx' }
+    );
+    
+    // Text index for search
+    await this.collection.createIndex(
+      { title: 'text', content: 'text' },
+      { weights: { title: 10, content: 5 } }
+    );
+  }
 }
-
-// TypeScript enforces correct paths and types!
-const profiles = await Profile.getInstances({
-  'data.personal.name.first': 'John',  // ✅ Type-checked
-  'data.address.country': 'USA',       // ✅ Type-checked
-  'data.personal.age': { $gte: 18 },   // ✅ Type-checked
-  // 'data.invalid.path': 'value'      // ❌ TypeScript error!
-});
 ```
 
-## 🛡️ Security Features
+#### Connection Pool Management
 
-SmartData includes enterprise-grade security out of the box:
+```typescript
-
+const db = new SmartdataDb({
-- **🔐 Credential Security**: Automatic encoding of special characters in passwords
+  mongoDbUrl: 'mongodb://localhost:27017',
-- **💉 Injection Prevention**: NoSQL injection protection with query sanitization
+  mongoDbName: 'myapp',
-- **🚫 Dangerous Operator Blocking**: Prevents use of `$where` and other risky operators
+  
-- **🔒 Secure Defaults**: Production-ready connection settings out of the box
+  // Connection pool optimization
-- **🛑 Rate Limiting Ready**: Built-in connection pooling prevents connection exhaustion
+  maxPoolSize: 100,           // Maximum connections
+  minPoolSize: 10,            // Minimum connections to maintain
+  maxIdleTimeMS: 300000,      // Close idle connections after 5 minutes
+  waitQueueTimeoutMS: 5000,   // Max time to wait for available connection
+  
+  // Server selection
+  serverSelectionTimeoutMS: 30000,  // Timeout for selecting a server
+  heartbeatFrequencyMS: 10000,      // How often to check server status
+  
+  // Socket settings
+  socketTimeoutMS: 360000,    // Socket timeout (6 minutes)
+  family: 4,                  // Force IPv4
+});
+```
 
 ## 🎯 Best Practices
 
-### Connection Management
+### 1. Always Use TypeScript
+
+SmartData is built for TypeScript. Using JavaScript means missing out on:
+- Compile-time query validation
+- IntelliSense for MongoDB operators
+- Type-safe document updates
+- Automatic refactoring support
+
+### 2. Design Your Indexes Thoughtfully
+
 ```typescript
-// ✅ DO: Use connection pooling options
+// ✅ Good: Index fields you query frequently
-const db = new SmartdataDb({
+@index() public userId: string;
-  mongoDbUrl: 'mongodb://localhost:27017/myapp',
+@index() public status: string;
-  maxPoolSize: 50,  // Adjust based on your load
-  maxIdleTimeMS: 300000  // 5 minutes
-});
 
-// ✅ DO: Always close connections on shutdown
+// ❌ Bad: Over-indexing
-process.on('SIGTERM', async () => {
+@index() public everyField: string;  // Don't index everything!
-  await db.close();
-  process.exit(0);
-});
-
-// ❌ DON'T: Create multiple DB instances for the same database
 ```
 
-### Performance Optimization
+### 3. Use Transactions for Critical Operations
+
 ```typescript
-// ✅ DO: Use cursors for large datasets
+// ✅ Good: Atomic operations for financial transactions
+await session.withTransaction(async () => {
+  await debitAccount(fromAccount, amount, { session });
+  await creditAccount(toAccount, amount, { session });
+});
+
+// ❌ Bad: Multiple operations without transactions
+await debitAccount(fromAccount, amount);  // What if this fails?
+await creditAccount(toAccount, amount);
+```
+
+### 4. Leverage Cursors for Large Datasets
+
+```typescript
+// ✅ Good: Stream processing
 const cursor = await LargeCollection.getCursor({});
 await cursor.forEach(async (doc) => {
-  await processDocument(doc);
+  await processOne(doc);
 });
 
-// ❌ DON'T: Load everything into memory
+// ❌ Bad: Loading everything into memory
-const allDocs = await LargeCollection.getInstances({}); // Could OOM!
+const allDocs = await LargeCollection.getInstances({});  // Could OOM!
-
-// ✅ DO: Create indexes for frequent queries
-@index() public frequentlyQueried: string;
-
-// ✅ DO: Use projections when you don't need all fields
-const cursor = await User.getCursor(
-  { active: true },
-  { projection: { username: 1, email: 1 } }
-);
 ```
 
-### Type Safety
+### 5. Implement Proper Error Handling
+
 ```typescript
-// ✅ DO: Leverage TypeScript's type system
+// ✅ Good: Graceful error handling
-interface StrictUserData {
+try {
-  verified: boolean;
+  const user = await User.getInstance({ id: userId });
-  roles: ('admin' | 'user' | 'guest')[];
+  if (!user) {
+    throw new Error('User not found');
+  }
+  await processUser(user);
+} catch (error) {
+  logger.error('Failed to process user:', error);
+  await notificationService.alertAdmin(error);
 }
 
-@Collection(() => db)
+// ❌ Bad: Ignoring errors
-class StrictUser extends SmartDataDbDoc<StrictUser, StrictUser> {
+const user = await User.getInstance({ id: userId });
-  @svDb() public data: StrictUserData;  // Fully typed!
+await processUser(user);  // What if user is null?
-}
-
-// ✅ DO: Use DeepQuery for nested queries
-import { DeepQuery } from '@push.rocks/smartdata';
-
-const query: DeepQuery<StrictUser> = {
-  'data.verified': true,
-  'data.roles': { $in: ['admin'] }
-};
 ```
 
-## 📊 Performance Benchmarks
+## 🔧 Troubleshooting
 
-SmartData has been battle-tested in production environments:
+### Connection Issues
 
-- **🚀 Connection Pooling**: 100+ concurrent connections with <10ms latency
+```typescript
-- **⚡ Query Performance**: Indexed searches return in <5ms for millions of documents
+// Enable debug logging
-- **📦 Memory Efficient**: Stream processing keeps memory under 100MB for any dataset size
+const db = new SmartdataDb({
-- **🔄 Real-time Updates**: Change streams deliver updates in <50ms
+  mongoDbUrl: 'mongodb://localhost:27017',
+  mongoDbName: 'myapp',
+  // Add connection event handlers
+});
 
-## 🤝 Support
+db.mongoClient.on('error', (err) => {
+  console.error('MongoDB connection error:', err);
+});
 
-Need help? We've got you covered:
+db.mongoClient.on('timeout', () => {
+  console.error('MongoDB connection timeout');
+});
 
-- 📖 **Documentation**: Full API docs at [https://code.foss.global/push.rocks/smartdata](https://code.foss.global/push.rocks/smartdata)
+// Check connection status
-- 💬 **Issues**: Report bugs at [GitLab Issues](https://code.foss.global/push.rocks/smartdata/issues)
+const isConnected = db.status === 'connected';
-- 📧 **Email**: Reach out to hello@task.vc for enterprise support
+```
+
+### Memory Leaks
+
+Always clean up resources:
+
+```typescript
+// Watchers
+const watcher = await User.watch({});
+// ... use watcher
+await watcher.close();  // Always close!
+
+// Cursors
+const cursor = await User.getCursor({});
+// ... use cursor
+await cursor.close();  // Always close!
+
+// Sessions
+const session = db.startSession();
+// ... use session
+await session.endSession();  // Always end!
+```
+
+### Performance Issues
+
+```typescript
+// Profile slow queries
+await db.mongoDb.setProfilingLevel('all');
+
+// Check query execution
+const stats = await User.collection.mongoDbCollection
+  .find({ complex: 'query' })
+  .explain('executionStats');
+  
+console.log('Query execution time:', stats.executionTimeMillis);
+console.log('Documents examined:', stats.totalDocsExamined);
+console.log('Index used:', stats.executionStats.executionStages.indexName);
+```
+
+## 📚 API Reference
+
+### Core Classes
+
+- `SmartdataDb` - Database connection and management
+- `SmartDataDbDoc` - Base class for all documents
+- `SmartdataCollection` - Collection management
+- `SmartdataDbCursor` - Cursor for streaming operations
+- `SmartdataDbWatcher` - Change stream watcher
+- `SmartdataDistributedCoordinator` - Distributed coordination
+
+### Decorators
+
+- `@Collection(dbGetter)` - Define collection for a document class
+- `@unI()` - Create unique index
+- `@index(options)` - Create regular index
+- `@svDb(options)` - Mark field as saveable with optional serialization
+- `@searchable()` - Enable text search on field
+
+### Key Methods
+
+- `getInstances(filter)` - Find multiple documents
+- `getInstance(filter)` - Find single document
+- `getCursor(filter)` - Get cursor for streaming
+- `watch(filter, options)` - Watch for changes
+- `search(query, options)` - Lucene-style search
+- `save(options)` - Save document
+- `delete(options)` - Delete document
+- `updateFromDb()` - Refresh from database
+
+## 🌍 Community & Support
+
+- 📖 **Documentation**: [https://code.foss.global/push.rocks/smartdata](https://code.foss.global/push.rocks/smartdata)
+- 💬 **Issues**: [GitLab Issues](https://code.foss.global/push.rocks/smartdata/issues)
+- 📧 **Email**: hello@task.vc
 
 ## License and Legal Information
 

test/test.filters.ts

@@ -0,0 +1,604 @@
+import { tap, expect } from '@git.zone/tstest/tapbundle';
+import * as smartmongo from '@push.rocks/smartmongo';
+import * as smartunique from '@push.rocks/smartunique';
+import * as smartdata from '../ts/index.js';
+
+const { SmartdataDb, Collection, svDb, unI, index } = smartdata;
+
+let smartmongoInstance: smartmongo.SmartMongo;
+let testDb: smartdata.SmartdataDb;
+
+// Define test document classes
+@Collection(() => testDb)
+class TestUser extends smartdata.SmartDataDbDoc<TestUser, TestUser> {
+  @unI()
+  public id: string = smartunique.shortId();
+
+  @svDb()
+  public name: string;
+
+  @svDb()
+  public age: number;
+
+  @svDb()
+  public email: string;
+
+  @svDb()
+  public roles: string[];
+
+  @svDb()
+  public tags: string[];
+
+  @svDb()
+  public status: 'active' | 'inactive' | 'pending';
+
+  @svDb()
+  public metadata: {
+    lastLogin?: Date;
+    loginCount?: number;
+    preferences?: Record<string, any>;
+  };
+
+  @svDb()
+  public scores: number[];
+
+  constructor(data: Partial<TestUser> = {}) {
+    super();
+    Object.assign(this, data);
+  }
+}
+
+@Collection(() => testDb)
+class TestOrder extends smartdata.SmartDataDbDoc<TestOrder, TestOrder> {
+  @unI()
+  public id: string = smartunique.shortId();
+
+  @svDb()
+  public userId: string;
+
+  @svDb()
+  public items: Array<{
+    product: string;
+    quantity: number;
+    price: number;
+  }>;
+
+  @svDb()
+  public totalAmount: number;
+
+  @svDb()
+  public status: string;
+
+  @svDb()
+  public tags: string[];
+
+  constructor(data: Partial<TestOrder> = {}) {
+    super();
+    Object.assign(this, data);
+  }
+}
+
+// Setup and teardown
+tap.test('should create a test database instance', async () => {
+  smartmongoInstance = await smartmongo.SmartMongo.createAndStart();
+  testDb = new smartdata.SmartdataDb(await smartmongoInstance.getMongoDescriptor());
+  await testDb.init();
+  expect(testDb).toBeInstanceOf(SmartdataDb);
+});
+
+tap.test('should create test data', async () => {
+  // Create test users
+  const users = [
+    new TestUser({
+      name: 'John Doe',
+      age: 30,
+      email: 'john@example.com',
+      roles: ['admin', 'user'],
+      tags: ['javascript', 'nodejs', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 5, lastLogin: new Date() },
+      scores: [85, 90, 78]
+    }),
+    new TestUser({
+      name: 'Jane Smith',
+      age: 25,
+      email: 'jane@example.com',
+      roles: ['user'],
+      tags: ['python', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 3 },
+      scores: [92, 88, 95]
+    }),
+    new TestUser({
+      name: 'Bob Johnson',
+      age: 35,
+      email: 'bob@example.com',
+      roles: ['moderator', 'user'],
+      tags: ['javascript', 'react', 'nodejs'],
+      status: 'inactive',
+      metadata: { loginCount: 0 },
+      scores: [70, 75, 80]
+    }),
+    new TestUser({
+      name: 'Alice Brown',
+      age: 28,
+      email: 'alice@example.com',
+      roles: ['admin'],
+      tags: ['typescript', 'angular', 'mongodb'],
+      status: 'active',
+      metadata: { loginCount: 10 },
+      scores: [95, 98, 100]
+    }),
+    new TestUser({
+      name: 'Charlie Wilson',
+      age: 22,
+      email: 'charlie@example.com',
+      roles: ['user'],
+      tags: ['golang', 'kubernetes'],
+      status: 'pending',
+      metadata: { loginCount: 1 },
+      scores: [60, 65]
+    })
+  ];
+
+  for (const user of users) {
+    await user.save();
+  }
+
+  // Create test orders
+  const orders = [
+    new TestOrder({
+      userId: users[0].id,
+      items: [
+        { product: 'laptop', quantity: 1, price: 1200 },
+        { product: 'mouse', quantity: 2, price: 25 }
+      ],
+      totalAmount: 1250,
+      status: 'completed',
+      tags: ['electronics', 'priority']
+    }),
+    new TestOrder({
+      userId: users[1].id,
+      items: [
+        { product: 'book', quantity: 3, price: 15 },
+        { product: 'pen', quantity: 5, price: 2 }
+      ],
+      totalAmount: 55,
+      status: 'pending',
+      tags: ['stationery']
+    }),
+    new TestOrder({
+      userId: users[0].id,
+      items: [
+        { product: 'laptop', quantity: 2, price: 1200 },
+        { product: 'keyboard', quantity: 2, price: 80 }
+      ],
+      totalAmount: 2560,
+      status: 'processing',
+      tags: ['electronics', 'bulk']
+    })
+  ];
+
+  for (const order of orders) {
+    await order.save();
+  }
+
+  const savedUsers = await TestUser.getInstances({});
+  const savedOrders = await TestOrder.getInstances({});
+  expect(savedUsers.length).toEqual(5);
+  expect(savedOrders.length).toEqual(3);
+});
+
+// ============= BASIC FILTER TESTS =============
+tap.test('should filter by simple equality', async () => {
+  const users = await TestUser.getInstances({ name: 'John Doe' });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter by multiple fields (implicit AND)', async () => {
+  const users = await TestUser.getInstances({ 
+    status: 'active',
+    age: 30
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter by nested object fields', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.loginCount': 5
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+// ============= COMPARISON OPERATOR TESTS =============
+tap.test('should filter using $gt operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gt: 30 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Bob Johnson');
+});
+
+tap.test('should filter using $gte operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gte: 30 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'John Doe']);
+});
+
+tap.test('should filter using $lt operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $lt: 25 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Charlie Wilson');
+});
+
+tap.test('should filter using $lte operator', async () => {
+  const users = await TestUser.getInstances({
+    age: { $lte: 25 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Charlie Wilson', 'Jane Smith']);
+});
+
+tap.test('should filter using $ne operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $ne: 'active' }
+  });
+  expect(users.length).toEqual(2);
+  const statuses = users.map(u => u.status).sort();
+  expect(statuses).toEqual(['inactive', 'pending']);
+});
+
+tap.test('should filter using multiple comparison operators', async () => {
+  const users = await TestUser.getInstances({
+    age: { $gte: 25, $lt: 30 }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'Jane Smith']);
+});
+
+// ============= ARRAY OPERATOR TESTS =============
+tap.test('should filter using $in operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $in: ['active', 'pending'] }
+  });
+  expect(users.length).toEqual(4);
+  expect(users.every(u => ['active', 'pending'].includes(u.status))).toEqual(true);
+});
+
+tap.test('should filter arrays using $in operator', async () => {
+  const users = await TestUser.getInstances({
+    roles: { $in: ['admin'] }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should filter using $nin operator', async () => {
+  const users = await TestUser.getInstances({
+    status: { $nin: ['inactive', 'pending'] }
+  });
+  expect(users.length).toEqual(3);
+  expect(users.every(u => u.status === 'active')).toEqual(true);
+});
+
+tap.test('should filter arrays using $all operator', async () => {
+  const users = await TestUser.getInstances({
+    tags: { $all: ['javascript', 'nodejs'] }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'John Doe']);
+});
+
+tap.test('should filter arrays using $size operator', async () => {
+  const users = await TestUser.getInstances({
+    scores: { $size: 2 }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Charlie Wilson');
+});
+
+tap.test('should filter arrays using $elemMatch operator', async () => {
+  const orders = await TestOrder.getInstances({
+    items: {
+      $elemMatch: {
+        product: 'laptop',
+        quantity: { $gte: 2 }
+      }
+    }
+  });
+  expect(orders.length).toEqual(1);
+  expect(orders[0].totalAmount).toEqual(2560);
+});
+
+tap.test('should filter using $elemMatch with single condition', async () => {
+  const orders = await TestOrder.getInstances({
+    items: {
+      $elemMatch: {
+        price: { $gt: 100 }
+      }
+    }
+  });
+  expect(orders.length).toEqual(2);
+  expect(orders.every(o => o.items.some(i => i.price > 100))).toEqual(true);
+});
+
+// ============= LOGICAL OPERATOR TESTS =============
+tap.test('should filter using $or operator', async () => {
+  const users = await TestUser.getInstances({
+    $or: [
+      { age: { $lt: 25 } },
+      { status: 'inactive' }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Bob Johnson', 'Charlie Wilson']);
+});
+
+tap.test('should filter using $and operator', async () => {
+  const users = await TestUser.getInstances({
+    $and: [
+      { status: 'active' },
+      { age: { $gte: 28 } }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should filter using $nor operator', async () => {
+  const users = await TestUser.getInstances({
+    $nor: [
+      { status: 'inactive' },
+      { age: { $lt: 25 } }
+    ]
+  });
+  expect(users.length).toEqual(3);
+  expect(users.every(u => u.status !== 'inactive' && u.age >= 25)).toEqual(true);
+});
+
+tap.test('should filter using nested logical operators', async () => {
+  const users = await TestUser.getInstances({
+    $or: [
+      {
+        $and: [
+          { status: 'active' },
+          { roles: { $in: ['admin'] } }
+        ]
+      },
+      { age: { $lt: 23 } }
+    ]
+  });
+  expect(users.length).toEqual(3);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'Charlie Wilson', 'John Doe']);
+});
+
+// ============= ELEMENT OPERATOR TESTS =============
+tap.test('should filter using $exists operator', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.lastLogin': { $exists: true }
+  });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('John Doe');
+});
+
+tap.test('should filter using $exists false', async () => {
+  const users = await TestUser.getInstances({
+    'metadata.preferences': { $exists: false }
+  });
+  expect(users.length).toEqual(5);
+});
+
+// ============= COMPLEX FILTER TESTS =============
+tap.test('should handle complex nested filters', async () => {
+  const users = await TestUser.getInstances({
+    $and: [
+      { status: 'active' },
+      {
+        $or: [
+          { age: { $gte: 30 } },
+          { roles: { $all: ['admin'] } }
+        ]
+      },
+      { tags: { $in: ['mongodb'] } }
+    ]
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Alice Brown', 'John Doe']);
+});
+
+tap.test('should combine multiple operator types', async () => {
+  const orders = await TestOrder.getInstances({
+    $and: [
+      { totalAmount: { $gte: 100 } },
+      { status: { $in: ['completed', 'processing'] } },
+      { tags: { $in: ['electronics'] } }
+    ]
+  });
+  expect(orders.length).toEqual(2);
+  expect(orders.every(o => o.totalAmount >= 100)).toEqual(true);
+});
+
+// ============= ERROR HANDLING TESTS =============
+tap.test('should throw error for $where operator', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      $where: 'this.age > 25'
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$where.*not allowed/);
+});
+
+tap.test('should throw error for invalid $in value', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      status: { $in: 'active' as any } // Should be an array
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$in.*requires.*array/);
+});
+
+tap.test('should throw error for invalid $size value', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      scores: { $size: '3' as any } // Should be a number
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/\$size.*requires.*numeric/);
+});
+
+tap.test('should throw error for dots in field names', async () => {
+  let error: Error | null = null;
+  try {
+    await TestUser.getInstances({
+      'some.nested.field': { 'invalid.key': 'value' }
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/keys cannot contain dots/);
+});
+
+// ============= EDGE CASE TESTS =============
+tap.test('should handle empty filter (return all)', async () => {
+  const users = await TestUser.getInstances({});
+  expect(users.length).toEqual(5);
+});
+
+tap.test('should handle null values in filter', async () => {
+  // First, create a user with null email
+  const nullUser = new TestUser({
+    name: 'Null User',
+    age: 40,
+    email: null as any,
+    roles: ['user'],
+    tags: [],
+    status: 'active',
+    metadata: {},
+    scores: []
+  });
+  await nullUser.save();
+
+  const users = await TestUser.getInstances({ email: null });
+  expect(users.length).toEqual(1);
+  expect(users[0].name).toEqual('Null User');
+
+  // Clean up
+  await nullUser.delete();
+});
+
+tap.test('should handle arrays as direct equality match', async () => {
+  // This tests that arrays without operators are treated as equality matches
+  const users = await TestUser.getInstances({
+    roles: ['user'] // Exact match for array
+  });
+  expect(users.length).toEqual(2);  // Both Jane and Charlie have exactly ['user']
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Charlie Wilson', 'Jane Smith']);
+});
+
+tap.test('should handle regex operator', async () => {
+  const users = await TestUser.getInstances({
+    name: { $regex: '^J', $options: 'i' }
+  });
+  expect(users.length).toEqual(2);
+  const names = users.map(u => u.name).sort();
+  expect(names).toEqual(['Jane Smith', 'John Doe']);
+});
+
+tap.test('should handle unknown operators by letting MongoDB reject them', async () => {
+  // Unknown operators should be passed through to MongoDB, which will reject them
+  let error: Error | null = null;
+  
+  try {
+    await TestUser.getInstances({
+      age: { $unknownOp: 30 } as any
+    });
+  } catch (e) {
+    error = e as Error;
+  }
+
+  expect(error).toBeTruthy();
+  expect(error?.message).toMatch(/unknown operator.*\$unknownOp/);
+});
+
+// ============= PERFORMANCE TESTS =============
+tap.test('should efficiently filter large result sets', async () => {
+  // Create many test documents
+  const manyUsers = [];
+  for (let i = 0; i < 100; i++) {
+    manyUsers.push(new TestUser({
+      name: `User ${i}`,
+      age: 20 + (i % 40),
+      email: `user${i}@example.com`,
+      roles: i % 3 === 0 ? ['admin'] : ['user'],
+      tags: i % 2 === 0 ? ['even', 'test'] : ['odd', 'test'],
+      status: i % 4 === 0 ? 'inactive' : 'active',
+      metadata: { loginCount: i },
+      scores: [i, i + 10, i + 20]
+    }));
+  }
+
+  // Save in batches for efficiency
+  for (const user of manyUsers) {
+    await user.save();
+  }
+
+  // Complex filter that should still be fast
+  const startTime = Date.now();
+  const filtered = await TestUser.getInstances({
+    $and: [
+      { age: { $gte: 30, $lt: 40 } },
+      { status: 'active' },
+      { tags: { $in: ['even'] } },
+      { 'metadata.loginCount': { $gte: 20 } }
+    ]
+  });
+  const duration = Date.now() - startTime;
+
+  console.log(`Complex filter on 100+ documents took ${duration}ms`);
+  expect(duration).toBeLessThan(1000); // Should complete in under 1 second
+  expect(filtered.length).toBeGreaterThan(0);
+
+  // Clean up
+  for (const user of manyUsers) {
+    await user.delete();
+  }
+});
+
+// ============= CLEANUP =============
+tap.test('should clean up test database', async () => {
+  await testDb.mongoDb.dropDatabase();
+  await testDb.close();
+  await smartmongoInstance.stop();
+});
+
+export default tap.start();

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.16.1',
+  version: '5.16.3',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/classes.doc.ts

@@ -151,9 +151,55 @@ export function index(options?: IIndexOptions) {
   };
 }
 
+// Helper type to extract element type from arrays or return T itself
+type ElementOf<T> = T extends ReadonlyArray<infer U> ? U : T;
+
+// Type for $in/$nin values - arrays of the element type
+type InValues<T> = ReadonlyArray<ElementOf<T>>;
+
+// Type that allows MongoDB operators on leaf values while maintaining nested type safety
+export type MongoFilterCondition<T> = T | {
+  $eq?: T;
+  $ne?: T;
+  $gt?: T;
+  $gte?: T;
+  $lt?: T;
+  $lte?: T;
+  $in?: InValues<T>;
+  $nin?: InValues<T>;
+  $exists?: boolean;
+  $type?: string | number;
+  $regex?: string | RegExp;
+  $options?: string;
+  $all?: T extends ReadonlyArray<infer U> ? ReadonlyArray<U> : never;
+  $elemMatch?: T extends ReadonlyArray<infer U> ? MongoFilter<U> : never;
+  $size?: T extends ReadonlyArray<any> ? number : never;
+  $not?: MongoFilterCondition<T>;
+};
+
+export type MongoFilter<T> = {
+  [K in keyof T]?: T[K] extends object 
+    ? T[K] extends any[]
+      ? MongoFilterCondition<T[K]>  // Arrays can have operators
+      : MongoFilter<T[K]> | MongoFilterCondition<T[K]>  // Objects can be nested or have operators
+    : MongoFilterCondition<T[K]>;  // Primitives get operators
+} & {
+  // Logical operators
+  $and?: MongoFilter<T>[];
+  $or?: MongoFilter<T>[];
+  $nor?: MongoFilter<T>[];
+  $not?: MongoFilter<T>;
+  // Allow any string key for dot notation (we lose type safety here but maintain flexibility)
+  [key: string]: any;
+};
+
 export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
+  // SECURITY: Block $where to prevent server-side JS execution
+  if (filterArg.$where !== undefined) {
+    throw new Error('$where operator is not allowed for security reasons');
+  }
+  
   // Special case: detect MongoDB operators and pass them through directly
-  // SECURITY: Removed $where to prevent server-side JS execution
   const topLevelOperators = ['$and', '$or', '$nor', '$not', '$text', '$regex'];
   for (const key of Object.keys(filterArg)) {
     if (topLevelOperators.includes(key)) {
@@ -166,26 +212,76 @@ export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
 
   const convertFilterArgument = (keyPathArg2: string, filterArg2: any) => {
     if (Array.isArray(filterArg2)) {
-      // FIX: Properly handle arrays for operators like $in, $all, or plain equality
+      // Arrays are typically used as values for operators like $in or as direct equality matches
       convertedFilter[keyPathArg2] = filterArg2;
       return;
     } else if (typeof filterArg2 === 'object' && filterArg2 !== null) {
-      for (const key of Object.keys(filterArg2)) {
+      // Check if this is an object with MongoDB operators
-        if (key.startsWith('$')) {
+      const keys = Object.keys(filterArg2);
-          // Prevent dangerous operators
+      const hasOperators = keys.some(key => key.startsWith('$'));
-          if (key === '$where') {
+      
-            throw new Error('$where operator is not allowed for security reasons');
+      if (hasOperators) {
-          }
+        // This object contains MongoDB operators
-          convertedFilter[keyPathArg2] = filterArg2;
+        // Validate and pass through allowed operators
-          return;
+        const allowedOperators = [
-        } else if (key.includes('.')) {
+          // Comparison operators
+          '$eq', '$ne', '$gt', '$gte', '$lt', '$lte',
+          // Array operators
+          '$in', '$nin', '$all', '$elemMatch', '$size',
+          // Element operators
+          '$exists', '$type',
+          // Evaluation operators (safe ones only)
+          '$regex', '$options', '$text', '$mod',
+          // Logical operators (nested)
+          '$and', '$or', '$nor', '$not'
+        ];
+        
+        // Check for dangerous operators
+        if (keys.includes('$where')) {
+          throw new Error('$where operator is not allowed for security reasons');
+        }
+        
+        // Validate all operators are in the allowed list
+        const invalidOperators = keys.filter(key => 
+          key.startsWith('$') && !allowedOperators.includes(key)
+        );
+        
+        if (invalidOperators.length > 0) {
+          console.warn(`Warning: Unknown MongoDB operators detected: ${invalidOperators.join(', ')}`);
+        }
+        
+        // For array operators, ensure the values are appropriate
+        if (filterArg2.$in && !Array.isArray(filterArg2.$in)) {
+          throw new Error('$in operator requires an array value');
+        }
+        if (filterArg2.$nin && !Array.isArray(filterArg2.$nin)) {
+          throw new Error('$nin operator requires an array value');
+        }
+        if (filterArg2.$all && !Array.isArray(filterArg2.$all)) {
+          throw new Error('$all operator requires an array value');
+        }
+        if (filterArg2.$size && typeof filterArg2.$size !== 'number') {
+          throw new Error('$size operator requires a numeric value');
+        }
+        
+        // Pass the operator object through
+        convertedFilter[keyPathArg2] = filterArg2;
+        return;
+      }
+      
+      // No operators, check for dots in keys
+      for (const key of keys) {
+        if (key.includes('.')) {
           throw new Error('keys cannot contain dots');
         }
       }
-      for (const key of Object.keys(filterArg2)) {
+      
+      // Recursively process nested objects
+      for (const key of keys) {
         convertFilterArgument(`${keyPathArg2}.${key}`, filterArg2[key]);
       }
     } else {
+      // Primitive values
       convertedFilter[keyPathArg2] = filterArg2;
     }
   };
@@ -227,12 +323,12 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
   /**
    * gets all instances as array
    * @param this
-   * @param filterArg
+   * @param filterArg - Type-safe MongoDB filter with nested object support and operators
    * @returns
    */
   public static async getInstances<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    filterArg: MongoFilter<T>,
     opts?: { session?: plugins.mongodb.ClientSession }
   ): Promise<T[]> {
     // Pass session through to findAll for transactional queries
@@ -256,7 +352,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public static async getInstance<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    filterArg: MongoFilter<T>,
     opts?: { session?: plugins.mongodb.ClientSession }
   ): Promise<T> {
     // Retrieve one document, with optional session for transactions
@@ -289,7 +385,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public static async getCursor<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    filterArg: MongoFilter<T>,
     opts?: {
       session?: plugins.mongodb.ClientSession;
       modifier?: (cursorArg: plugins.mongodb.FindCursor<plugins.mongodb.WithId<plugins.mongodb.BSON.Document>>) => plugins.mongodb.FindCursor<plugins.mongodb.WithId<plugins.mongodb.BSON.Document>>;
@@ -319,7 +415,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public static async watch<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    filterArg: MongoFilter<T>,
     opts?: plugins.mongodb.ChangeStreamOptions & { bufferTimeMs?: number },
   ): Promise<SmartdataDbWatcher<T>> {
     const collection: SmartdataCollection<T> = (this as any).collection;
@@ -337,7 +433,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public static async forEach<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T>,
+    filterArg: MongoFilter<T>,
     forEachFunction: (itemArg: T) => Promise<any>,
   ) {
     const cursor: SmartdataDbCursor<T> = await (this as any).getCursor(filterArg);
@@ -349,7 +445,7 @@ export class SmartDataDbDoc<T extends TImplements, TImplements, TManager extends
    */
   public static async getCount<T>(
     this: plugins.tsclass.typeFest.Class<T>,
-    filterArg: plugins.tsclass.typeFest.PartialDeep<T> = {} as any,
+    filterArg: MongoFilter<T> = {} as any,
   ) {
     const collection: SmartdataCollection<T> = (this as any).collection;
     return await collection.getCount(filterArg);