smartdata/codex.md

# SmartData Project Overview

This document provides a high-level overview of the SmartData library (`@push.rocks/smartdata`), its architecture, core components, and key features—including recent enhancements to the search API.

## 1. Project Purpose
- A TypeScript‑first wrapper around MongoDB that supplies:
  - Strongly‑typed document & collection classes
  - Decorator‑based schema definition (no external schema files)
  - Advanced search capabilities with Lucene‑style queries
  - Built‑in support for real‑time data sync, distributed coordination, and key‑value EasyStore

## 2. Core Concepts & Components
- **SmartDataDb**: Manages the MongoDB connection, pooling, and initialization of collections.
- **SmartDataDbDoc**: Base class for all document models; provides CRUD, upsert, and cursor APIs.
- **Decorators**:
  - `@Collection`: Associates a class with a MongoDB collection
  - `@svDb()`: Marks a field as persisted to the DB
  - `@unI()`: Marks a field as a unique index
  - `@index()`: Adds a regular index
  - `@searchable()`: Marks a field for inclusion in text searches or regex queries
- **SmartdataCollection**: Wraps a MongoDB collection; auto‑creates indexes based on decorators.
- **Lucene Adapter**: Parses a Lucene query string into an AST and transforms it to a MongoDB filter object.
- **EasyStore**: A simple, schema‑less key‑value store built on top of MongoDB for sharing ephemeral data.
- **Distributed Coordinator**: Leader election and task‑distribution API for building resilient, multi‑instance systems.
- **Watcher**: Listens to change streams for real‑time updates and integrates with RxJS.

## 3. Search API
SmartData provides a unified `.search(query[, opts])` method on all models with `@searchable()` fields:

- **Supported Syntax**:
  1. Exact field:value (e.g. `field:Value`)
  2. Quoted phrases (e.g. `"exact phrase"` or `'exact phrase'`)
  3. Wildcards: `*` (zero or more chars) and `?` (single char)
  4. Boolean operators: `AND`, `OR`, `NOT`
  5. Grouping: parenthesis `(A OR B) AND C`
  6. Range queries: `[num TO num]`, `{num TO num}`
  7. Multi‑term unquoted: terms AND’d across all searchable fields
  8. Empty query returns all documents

- **Fallback Mechanisms**:
  1. Text index based `$text` search (if supported)
  2. Field‑scoped and multi‑field regex queries
  3. In‑memory filtering for complex or unsupported cases

### New Security & Extensibility Hooks
The `.search(query, opts?)` signature now accepts a `SearchOptions<T>` object:
```ts
interface SearchOptions<T> {
  filter?: Record<string, any>;        // Additional MongoDB filter AND‑merged
  validate?: (doc: T) => boolean;      // Post‑fetch hook to drop results
}
```
- **filter**: Enforces mandatory constraints (e.g. multi‑tenant isolation) directly in the Mongo query.
- **validate**: An async function that runs after fetching; return `false` to exclude a document.

## 4. Testing Strategy
- Unit tests in `test/test.search.ts` cover basic search functionality and new options:
  - Exact, wildcard, phrase, boolean and grouping cases
  - Implicit AND and mixed free‑term + field searches
  - Edge cases (non‑searchable fields, quoted wildcards, no matches)
  - `filter` and `validate` tests ensure security hooks work as intended
- Advanced search scenarios are covered in `test/test.search.advanced.ts`.

## 5. Usage Example
```ts
// Basic search
const prods = await Product.search('wireless earbuds');

// Scoped search (only your organization’s items)
const myItems = await Product.search('book', { filter: { ownerId } });

// Post‑search validation (only cheap items)
const cheapItems = await Product.search('', { validate: p => p.price < 50 });
```

---
Last updated: 2025-04-22