smartdata/readme.md

# @push.rocks/smartdata

[![npm version](https://badge.fury.io/js/@push.rocks%2Fsmartdata.svg)](https://www.npmjs.com/package/@push.rocks/smartdata)

A powerful TypeScript-first MongoDB wrapper that provides advanced features for distributed systems, real-time data synchronization, and easy data management.

## Features

- **Type-Safe MongoDB Integration**: Full TypeScript support with decorators for schema definition
- **Document Management**: Type-safe CRUD operations with automatic timestamp tracking
- **EasyStore**: Simple key-value storage with automatic persistence and sharing between instances
- **Distributed Coordination**: Built-in support for leader election and distributed task management
- **Real-time Data Sync**: Watchers for real-time data changes with RxJS integration
- **Connection Management**: Automatic connection handling with connection pooling
- **Collection Management**: Type-safe collection operations with automatic indexing
- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>`
- **MongoDB Compatibility**: Support for all MongoDB query operators and advanced features
- **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
- **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
- **Serialization Hooks**: Custom serialization and deserialization of document properties
- **Powerful Search Capabilities**: Unified `search(query)` API supporting field:value exact matches, multi-field regex searches, case-insensitive matching, and automatic escaping to prevent regex injection

## Requirements

- Node.js >= 16.x
- MongoDB >= 4.4
- TypeScript >= 4.x (for development)

## Install

To install `@push.rocks/smartdata`, use npm:

```bash
npm install @push.rocks/smartdata --save
```

Or with pnpm:

```bash
pnpm add @push.rocks/smartdata
```

## Usage

`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.

### Setting Up and Connecting to the Database

Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.

```typescript
import { SmartdataDb } from '@push.rocks/smartdata';

// Create a new instance of SmartdataDb with MongoDB connection details
const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://<USERNAME>:<PASSWORD>@localhost:27017/<DBNAME>',
  mongoDbName: 'your-database-name',
  mongoDbUser: 'your-username',
  mongoDbPass: 'your-password',
});

// Initialize and connect to the database
// This sets up a connection pool with max 100 connections
await db.init();
```

### Defining Data Models

Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, `@svDb`, `@index`, and `@searchable` to define your data models. Fields of type `ObjectId` or `Buffer` decorated with `@svDb()` will be stored as BSON ObjectId and Binary, respectively; no separate `@oid()` or `@bin()` decorators are required.

```typescript
import {
  SmartDataDbDoc,
  Collection,
  unI,
  svDb,
  index,
  searchable,
} from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';

@Collection(() => db) // Associate this model with the database instance
class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id: string = 'unique-user-id'; // Mark 'id' as a unique index

  @svDb()
  @searchable() // Mark 'username' as searchable
  public username: string; // Mark 'username' to be saved in DB

  @svDb()
  @searchable() // Mark 'email' as searchable
  @index() // Create a regular index for this field
  public email: string; // Mark 'email' to be saved in DB

  @svDb()
  public organizationId: ObjectId; // Stored as BSON ObjectId

  @svDb()
  public profilePicture: Buffer; // Stored as BSON Binary

  @svDb({
    serialize: (data) => JSON.stringify(data), // Custom serialization
    deserialize: (data) => JSON.parse(data), // Custom deserialization
  })
  public preferences: Record<string, any>;

  constructor(username: string, email: string) {
    super();
    this.username = username;
    this.email = email;
  }
}
```

### CRUD Operations

`@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.

#### Create

```typescript
const newUser = new User('myUsername', 'myEmail@example.com');
await newUser.save(); // Save the new user to the database
```

#### Read

```typescript
// Fetch a single user by a unique attribute
const user = await User.getInstance({ username: 'myUsername' });

// Fetch multiple users that match criteria
const users = await User.getInstances({ email: 'myEmail@example.com' });

// Using a cursor for large collections
const cursor = await User.getCursor({ active: true });

// Process documents one at a time (memory efficient)
await cursor.forEach(async (user, index) => {
  // Process each user with its position
  console.log(`Processing user ${index}: ${user.username}`);
});

// Chain cursor methods like in the MongoDB native driver
const paginatedCursor = await User.getCursor({ active: true })
  .limit(10) // Limit results
  .skip(20) // Skip first 20 results
  .sort({ createdAt: -1 }); // Sort by creation date descending

// Convert cursor to array (when you know the result set is small)
const userArray = await paginatedCursor.toArray();

// Other cursor operations
const nextUser = await cursor.next(); // Get the next document
const hasMoreUsers = await cursor.hasNext(); // Check if more documents exist
const count = await cursor.count(); // Get the count of documents in the cursor

// Always close cursors when done with them
await cursor.close();
```

#### Update

```typescript
// Assuming 'user' is an instance of User
user.email = 'newEmail@example.com';
await user.save(); // Update the user in the database

// Upsert operations (insert if not exists, update if exists)
const upsertedUser = await User.upsert(
  { id: 'user-123' }, // Query to find the user
  {
    // Fields to update or insert
    username: 'newUsername',
    email: 'newEmail@example.com',
  },
);
```

#### Delete

```typescript
// Assuming 'user' is an instance of User
await user.delete(); // Delete the user from the database
```

## Advanced Features

### Search Functionality

SmartData provides powerful, Lucene‑style search capabilities with robust fallback mechanisms:

```typescript
// Define a model with searchable fields
@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() public id: string = 'product-id';
  @svDb() @searchable() public name: string;
  @svDb() @searchable() public description: string;
  @svDb() @searchable() public category: string;
  @svDb() public price: number;
}

// List searchable fields
const searchableFields = Product.getSearchableFields();

// 1: Exact phrase across all fields
await Product.search('"Kindle Paperwhite"');

// 2: Wildcard search across all fields
await Product.search('Air*');

// 3: Field‑scoped wildcard
await Product.search('name:Air*');

// 4: Boolean AND/OR/NOT
await Product.search('category:Electronics AND name:iPhone');

// 5: Grouping with parentheses
await Product.search('(Furniture OR Electronics) AND Chair');

// 6: Multi‑term unquoted (terms AND’d across fields)
await Product.search('TypeScript Aufgabe');

// 7: Empty query returns all documents
await Product.search('');
```

The search functionality includes:

- `@searchable()` decorator for marking fields as searchable
- `Class.getSearchableFields()` static method to list searchable fields for a model
- `search(query: string)` method supporting:
  - Exact phrase matches (`"my exact string"` or `'my exact string'`)
  - Field‑scoped exact & wildcard searches (`field:value`, `field:Air*`)
  - Wildcard searches across all fields (`Air*`, `?Pods`)
  - Boolean operators (`AND`, `OR`, `NOT`) with grouping (`(...)`)
  - Multi‑term unquoted queries AND’d across fields (`TypeScript Aufgabe`)
  - Single/multi‑term regex searches across fields
  - Empty queries returning all documents
- Automatic escaping & wildcard conversion to prevent regex injection

### EasyStore

EasyStore provides a simple key-value storage system with automatic persistence:

```typescript
// Create an EasyStore instance with a specific type
interface ConfigStore {
  apiKey: string;
  settings: {
    theme: string;
    notifications: boolean;
  };
}

// Create a type-safe EasyStore
const store = await db.createEasyStore<ConfigStore>('app-config');

// Write and read data with full type safety
await store.writeKey('apiKey', 'secret-api-key-123');
await store.writeKey('settings', { theme: 'dark', notifications: true });

const apiKey = await store.readKey('apiKey'); // Type: string
const settings = await store.readKey('settings'); // Type: { theme: string, notifications: boolean }

// Check if a key exists
const hasKey = await store.hasKey('apiKey'); // true

// Delete a key
await store.deleteKey('apiKey');
```

### Distributed Coordination

Built-in support for distributed systems with leader election:

```typescript
// Create a distributed coordinator
const coordinator = new SmartdataDistributedCoordinator(db);

// Start coordination
await coordinator.start();

// Handle leadership changes
coordinator.on('leadershipChange', (isLeader) => {
  if (isLeader) {
    // This instance is now the leader
    // Run leader-specific tasks
    startPeriodicJobs();
  } else {
    // This instance is no longer the leader
    stopPeriodicJobs();
  }
});

// Access leadership status anytime
if (coordinator.isLeader) {
  // Run leader-only operations
}

// Execute a task only on the leader
await coordinator.executeIfLeader(async () => {
  // This code only runs on the leader instance
  await runImportantTask();
});

// Stop coordination when shutting down
await coordinator.stop();
```

### Real-time Data Watching

Watch for changes in your collections with RxJS integration using MongoDB Change Streams:

```typescript
// Create a watcher for a specific collection with a query filter
const watcher = await User.watch(
  {
    active: true, // Only watch for changes to active users
  },
  {
    fullDocument: true, // Include the full document in change notifications
    bufferTimeMs: 100, // Buffer changes for 100ms to reduce notification frequency
  },
);

// Subscribe to changes using RxJS
watcher.changeSubject.subscribe((change) => {
  console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
  console.log('Document changed:', change.docInstance); // The full document instance

  // Handle different types of changes
  if (change.operationType === 'insert') {
    console.log('New user created:', change.docInstance.username);
  } else if (change.operationType === 'update') {
    console.log('User updated:', change.docInstance.username);
  } else if (change.operationType === 'delete') {
    console.log('User deleted');
  }
});

// Manual observation with event emitter pattern
watcher.on('change', (change) => {
  console.log('Document changed:', change);
});

// Stop watching when no longer needed
await watcher.stop();
```

### Managed Collections

For more complex data models that require additional context:

```typescript
@Collection(() => db)
class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
  @unI()
  public id: string = 'unique-id';

  @svDb()
  public data: string;

  @managed()
  public manager: YourCustomManager;

  // The manager can provide additional functionality
  async specialOperation() {
    return this.manager.doSomethingSpecial(this);
  }
}
```

### Automatic Indexing

Define indexes directly in your model class:

```typescript
@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() // Unique index
  public id: string = 'product-id';

  @svDb()
  @index() // Regular index for faster queries
  public category: string;

  @svDb()
  @index({ sparse: true }) // Sparse index with options
  public optionalField?: string;

  // Compound indexes can be defined in the collection decorator
  @Collection(() => db, {
    indexMap: {
      compoundIndex: {
        fields: { category: 1, name: 1 },
        options: { background: true }
      }
    }
  })
}
```

### Transaction Support

Use MongoDB transactions for atomic operations:

```typescript
const session = await db.startSession();
try {
  await session.withTransaction(async () => {
    const user = await User.getInstance({ id: 'user-id' }, { session });
    user.balance -= 100;
    await user.save({ session });

    const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
    recipient.balance += 100;
    await user.save({ session });
  });
} finally {
  await session.endSession();
}
```

### Deep Object Queries

SmartData provides fully type-safe deep property queries with the `DeepQuery` type:

```typescript
// If your document has nested objects
class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
  @unI()
  public id: string = 'profile-id';

  @svDb()
  public user: {
    details: {
      firstName: string;
      lastName: string;
      address: {
        city: string;
        country: string;
      };
    };
  };
}

// Type-safe string literals for dot notation
const usersInUSA = await UserProfile.getInstances({
  'user.details.address.country': 'USA',
});

// Fully typed deep queries with the DeepQuery type
import { DeepQuery } from '@push.rocks/smartdata';

const typedQuery: DeepQuery<UserProfile> = {
  id: 'profile-id',
  'user.details.firstName': 'John',
  'user.details.address.country': 'USA',
};

// TypeScript will error if paths are incorrect
const results = await UserProfile.getInstances(typedQuery);

// MongoDB query operators are supported
const operatorQuery: DeepQuery<UserProfile> = {
  'user.details.address.country': 'USA',
  'user.details.address.city': { $in: ['New York', 'Los Angeles'] },
};

const filteredResults = await UserProfile.getInstances(operatorQuery);
```

### Document Lifecycle Hooks

Implement custom logic at different stages of a document's lifecycle:

```typescript
@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
  @unI()
  public id: string = 'order-id';

  @svDb()
  public total: number;

  @svDb()
  public items: string[];

  // Called before saving the document
  async beforeSave() {
    // Calculate total based on items
    this.total = await calculateTotal(this.items);

    // Validate the document
    if (this.items.length === 0) {
      throw new Error('Order must have at least one item');
    }
  }

  // Called after the document is saved
  async afterSave() {
    // Notify other systems about the saved order
    await notifyExternalSystems(this);
  }

  // Called before deleting the document
  async beforeDelete() {
    // Check if order can be deleted
    const canDelete = await checkOrderDeletable(this.id);
    if (!canDelete) {
      throw new Error('Order cannot be deleted');
    }
  }
}
```

## Best Practices

### Connection Management

- Always call `db.init()` before using any database features
- Use `db.close()` when shutting down your application
- Set appropriate connection pool sizes based on your application's needs

### Document Design

- Use appropriate decorators (`@svDb`, `@unI`, `@index`, `@searchable`) to optimize database operations
- Implement type-safe models by properly extending `SmartDataDbDoc`
- Consider using interfaces to define document structures separately from implementation
- Mark fields that need to be searched with the `@searchable()` decorator

### Search Optimization

- (Optional) Create MongoDB text indexes on searchable fields to speed up full-text search
- Use `search(query)` for all search operations (field:value, partial matches, multi-word)
- Prefer field-specific exact matches when possible for optimal performance
- Avoid unnecessary complexity in query strings to keep regex searches efficient

### Performance Optimization

- Use cursors for large datasets instead of loading all documents into memory
- Create appropriate indexes for frequent query patterns
- Use projections to limit the fields returned when you don't need the entire document

### Distributed Systems

- Implement proper error handling for leader election events
- Ensure all instances have synchronized clocks when using time-based coordination
- Use the distributed coordinator's task management features for coordinated operations

### Type Safety

- Take advantage of the `DeepQuery<T>` type for fully type-safe queries
- Define proper types for your document models to enhance IDE auto-completion
- Use generic type parameters to specify exact document types when working with collections

## Contributing

We welcome contributions to @push.rocks/smartdata! Here's how you can help:

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

Please make sure to update tests as appropriate and follow our coding standards.

## License and Legal Information

This repository is licensed under the MIT License. For details, see [MIT License](https://opensource.org/licenses/MIT).

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

### Company Information

Task Venture Capital GmbH  
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.