smartdata/readme.md

# @push.rocks/smartdata 🚀

[![npm version](https://badge.fury.io/js/@push.rocks%2Fsmartdata.svg)](https://www.npmjs.com/package/@push.rocks/smartdata)

**The ultimate TypeScript-first MongoDB wrapper** that turns your database into a type-safe powerhouse. Built for modern applications that demand real-time performance, distributed coordination, and rock-solid reliability.

## 🌟 Why SmartData?

SmartData isn't just another MongoDB wrapper - it's a complete paradigm shift in how you work with databases:

- 🔒 **100% Type-Safe**: Full TypeScript with decorators, generics, and compile-time query validation
- ⚡ **Lightning Fast**: Connection pooling, cursor streaming, and intelligent indexing
- 🔄 **Real-time Ready**: MongoDB Change Streams with RxJS for reactive applications
- 🌍 **Distributed Systems**: Built-in leader election and task coordination
- 🛡️ **Security First**: NoSQL injection prevention, automatic sanitization, and secure defaults
- 🎯 **Developer Experience**: Intuitive API, powerful search, and IntelliSense that just works

## 📦 Installation

```bash
# Using pnpm (recommended)
pnpm add @push.rocks/smartdata

# Using npm
npm install @push.rocks/smartdata

# Using yarn
yarn add @push.rocks/smartdata
```

## 🚦 Requirements

- **Node.js** >= 16.x
- **MongoDB** >= 4.4
- **TypeScript** >= 4.x (for development)

## 🎯 Quick Start

### 1️⃣ Connect to Your Database

```typescript
import { SmartdataDb } from '@push.rocks/smartdata';

// Create a database instance with smart defaults
const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017/myapp',
  mongoDbName: 'myapp',
  mongoDbUser: 'username',
  mongoDbPass: 'password',
  
  // Optional: Advanced connection pooling
  maxPoolSize: 100,          // Max connections in pool
  maxIdleTimeMS: 300000,     // Max idle time before connection close
  serverSelectionTimeoutMS: 30000  // Connection timeout
});

// Initialize with automatic retry and health monitoring
await db.init();
```

### 2️⃣ Define Your Data Models

```typescript
import {
  SmartDataDbDoc,
  Collection,
  unI,
  svDb,
  index,
  searchable,
} from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';

@Collection(() => db)
class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id: string;  // Unique index with automatic ID generation
  
  @svDb()
  @searchable()  // Enable Lucene-style searching
  public username: string;
  
  @svDb()
  @searchable()
  @index({ unique: false })  // Performance index
  public email: string;
  
  @svDb()
  public status: 'active' | 'inactive' | 'pending';  // Full union type support
  
  @svDb()
  public organizationId: ObjectId;  // Native MongoDB types
  
  @svDb()
  public profilePicture: Buffer;    // Binary data support
  
  @svDb({
    // Custom serialization for complex objects
    serialize: (data) => JSON.stringify(data),
    deserialize: (data) => JSON.parse(data),
  })
  public preferences: Record<string, any>;
  
  @svDb()
  public tags: string[];  // Array support with operators
  
  @svDb()
  public createdAt: Date = new Date();
  
  constructor(username: string, email: string) {
    super();
    this.username = username;
    this.email = email;
    this.id = User.getNewId();
  }
}
```

### 3️⃣ Perform CRUD Operations

```typescript
// ✨ Create
const user = new User('johndoe', 'john@example.com');
user.status = 'active';
user.tags = ['developer', 'typescript'];
await user.save();

// 🔍 Read - with full type safety
const foundUser = await User.getInstance({ username: 'johndoe' });
const activeUsers = await User.getInstances({ status: 'active' });

// ✏️ Update
foundUser.email = 'newemail@example.com';
foundUser.tags.push('mongodb');
await foundUser.save();

// 🗑️ Delete
await foundUser.delete();
```

## 🔥 Advanced Features

### 🎯 Type-Safe Query Filters

SmartData provides the most advanced type-safe filtering system for MongoDB, supporting all operators while maintaining full IntelliSense:

```typescript
// Union types work perfectly
const users = await User.getInstances({
  status: { $in: ['active', 'pending'] }  // TypeScript validates these values!
});

// Comparison operators with type checking
const adults = await User.getInstances({
  age: { $gte: 18, $lt: 65 }  // Type-safe numeric comparisons
});

// Array operations with full type safety
const experts = await User.getInstances({
  tags: { $all: ['typescript', 'mongodb'] },  // Must have all tags
  skills: { $size: 5 }  // Exactly 5 skills
});

// Complex nested queries
const results = await Order.getInstances({
  $and: [
    { status: { $in: ['pending', 'processing'] } },
    { 'items.price': { $gte: 100 } },  // Dot notation for nested fields
    { 
      items: { 
        $elemMatch: {  // Array element matching
          product: 'laptop',
          quantity: { $gte: 2 }
        }
      }
    }
  ]
});

// Logical operators
const urgentOrHighValue = await Order.getInstances({
  $or: [
    { priority: 'urgent' },
    { totalAmount: { $gte: 1000 } }
  ]
});

// Security: $where is automatically blocked
// await User.getInstances({ $where: '...' });  // ❌ Throws security error
```

### 🔎 Powerful Search Engine

Built-in Lucene-style search with automatic indexing:

```typescript
@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() public id: string;
  @svDb() @searchable() public name: string;
  @svDb() @searchable() public description: string;
  @svDb() @searchable() public category: string;
  @svDb() public price: number;
}

// Simple text search across all searchable fields
const results = await Product.search('laptop');

// Field-specific search
const electronics = await Product.search('category:Electronics');

// Wildcard searches
const matches = await Product.search('Mac*');

// Boolean operators
const query = await Product.search('laptop AND NOT gaming');

// Phrase search
const exact = await Product.search('"MacBook Pro 16"');

// Combined with filters for powerful queries
const affordable = await Product.search('laptop', {
  filter: { price: { $lte: 1500 } },
  validate: async (p) => p.inStock === true
});
```

### 💾 EasyStore - Type-Safe Key-Value Storage

Perfect for configuration, caching, and session management:

```typescript
interface AppConfig {
  apiKey: string;
  features: {
    darkMode: boolean;
    notifications: boolean;
  };
  limits: {
    maxUsers: number;
    maxStorage: number;
  };
}

// Create a type-safe store
const config = await db.createEasyStore<AppConfig>('app-config');

// Write with full type checking
await config.writeKey('features', {
  darkMode: true,
  notifications: false
});

// Read with guaranteed types
const features = await config.readKey('features');
// TypeScript knows: features.darkMode is boolean ✅

// Atomic updates
await config.updateKey('limits', (current) => ({
  ...current,
  maxUsers: current.maxUsers + 100
}));

// Delete keys
await config.deleteKey('features');

// Clear entire store
await config.wipe();
```

### 📡 Real-Time Change Streams

React to database changes instantly with RxJS integration:

```typescript
// Watch for changes with automatic reconnection
const watcher = await User.watch(
  { status: 'active' },  // Filter which documents to watch
  { 
    fullDocument: 'updateLookup',  // Get full document on updates
    bufferTimeMs: 100  // Buffer changes for efficiency
  }
);

// Subscribe to changes with RxJS
watcher.changeSubject.subscribe({
  next: (change) => {
    console.log('User changed:', change.fullDocument);
    
    switch(change.operationType) {
      case 'insert':
        console.log('New user created');
        break;
      case 'update':
        console.log('User updated');
        break;
      case 'delete':
        console.log('User deleted');
        break;
    }
  },
  error: (err) => console.error('Watch error:', err)
});

// Advanced: Watch with aggregation pipeline
const complexWatcher = await Order.watch(
  { status: 'pending' },
  {
    pipeline: [
      { $match: { 'fullDocument.totalAmount': { $gte: 1000 } } },
      { $addFields: { isHighValue: true } }
    ]
  }
);

// Clean up when done
await watcher.close();
```

### 🌐 Distributed Coordination

Build resilient distributed systems with automatic leader election:

```typescript
import { SmartdataDistributedCoordinator } from '@push.rocks/smartdata';

const coordinator = new SmartdataDistributedCoordinator(db);

// Start coordination with automatic heartbeat
await coordinator.start();

// Leader election happens automatically
coordinator.asyncExecutionScheduler.addJobHandler('maintenance', async () => {
  // This code only runs on the leader instance
  console.log('🎖️ Running maintenance as leader');
  await performDatabaseCleanup();
});

// Check leadership status
const isLeader = coordinator.isLeader;

// Fire distributed tasks (integrates with @push.rocks/taskbuffer)
const result = await coordinator.fireDistributedTaskRequest({
  taskName: 'process-payments',
  taskExecutionTime: Date.now(),
  requestResponseId: 'unique-id'
});

// Graceful shutdown with leadership handoff
await coordinator.stop();
```

### 🔄 Cursor Streaming for Large Datasets

Process millions of documents without memory issues:

```typescript
// Get a cursor for memory-efficient iteration
const cursor = await User.getCursor(
  { status: 'active' },
  {
    // Optional: Use MongoDB native cursor modifiers
    modifier: (cursor) => cursor
      .sort({ createdAt: -1 })
      .limit(10000)
      .project({ email: 1, username: 1 })
  }
);

// Process one at a time
await cursor.forEach(async (user) => {
  await sendEmail(user.email);
  // Each document is loaded individually
});

// Or use async iteration
for await (const user of cursor) {
  if (shouldStop(user)) break;
  await processUser(user);
}

// Always clean up
await cursor.close();
```

### 🔐 Transaction Support

Ensure data consistency with MongoDB transactions:

```typescript
const session = db.startSession();

try {
  await session.withTransaction(async () => {
    // All operations in this block are atomic
    const sender = await User.getInstance(
      { id: 'user-1' }, 
      { session }  // Pass session to all operations
    );
    
    sender.balance -= 100;
    await sender.save({ session });
    
    const receiver = await User.getInstance(
      { id: 'user-2' }, 
      { session }
    );
    
    receiver.balance += 100;
    await receiver.save({ session });
    
    // If anything fails, everything rolls back
    if (sender.balance < 0) {
      throw new Error('Insufficient funds!');
    }
  });
  
  console.log('✅ Transaction completed');
} finally {
  await session.endSession();
}
```

### 🎨 Custom Serialization

Handle complex data types with custom serializers:

```typescript
class Document extends SmartDataDbDoc<Document, Document> {
  @svDb({
    // Encrypt sensitive data before storing
    serialize: async (value) => await encrypt(value),
    deserialize: async (value) => await decrypt(value)
  })
  public sensitiveData: string;
  
  @svDb({
    // Compress large JSON objects
    serialize: (value) => compress(JSON.stringify(value)),
    deserialize: (value) => JSON.parse(decompress(value))
  })
  public largePayload: any;
  
  @svDb({
    // Store Sets as arrays
    serialize: (set) => Array.from(set),
    deserialize: (arr) => new Set(arr)
  })
  public tags: Set<string>;
  
  @svDb({
    // Handle custom date formats
    serialize: (date) => date?.toISOString(),
    deserialize: (str) => str ? new Date(str) : null
  })
  public scheduledAt: Date | null;
}
```

### 🎣 Lifecycle Hooks

Add custom logic at any point in the document lifecycle:

```typescript
@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
  @unI() public id: string;
  @svDb() public items: Array<{ product: string; quantity: number }>;
  @svDb() public totalAmount: number;
  @svDb() public status: string;
  
  // Called before saving (create or update)
  async beforeSave() {
    // Recalculate total
    this.totalAmount = this.items.reduce((sum, item) => 
      sum + (item.price * item.quantity), 0
    );
    
    // Validate
    if (this.totalAmount < 0) {
      throw new Error('Invalid order total');
    }
  }
  
  // Called after successful save
  async afterSave() {
    // Send notifications
    await notificationService.orderUpdated(this.id);
    
    // Update cache
    await cache.set(`order:${this.id}`, this);
  }
  
  // Called before deletion
  async beforeDelete() {
    // Archive order
    await archive.store(this);
    
    // Check if deletion is allowed
    if (this.status === 'completed') {
      throw new Error('Cannot delete completed orders');
    }
  }
  
  // Called after successful deletion
  async afterDelete() {
    // Clean up related data
    await cache.delete(`order:${this.id}`);
    
    // Log deletion
    console.log(`Order ${this.id} deleted at ${new Date()}`);
  }
}
```

### 🚀 Performance Optimization

#### Intelligent Indexing

```typescript
@Collection(() => db)
class HighPerformanceDoc extends SmartDataDbDoc<HighPerformanceDoc, HighPerformanceDoc> {
  @unI()  // Unique index
  public id: string;
  
  @index()  // Single field index
  public userId: string;
  
  @index({ sparse: true })  // Sparse index for optional fields
  public deletedAt?: Date;
  
  @index({ 
    unique: false,
    background: true,  // Non-blocking index creation
    expireAfterSeconds: 86400  // TTL index
  })
  public sessionToken: string;
  
  // Compound indexes for complex queries
  static async createIndexes() {
    await this.collection.createIndex(
      { userId: 1, createdAt: -1 },  // Compound index
      { name: 'user_activity_idx' }
    );
    
    // Text index for search
    await this.collection.createIndex(
      { title: 'text', content: 'text' },
      { weights: { title: 10, content: 5 } }
    );
  }
}
```

#### Connection Pool Management

```typescript
const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017',
  mongoDbName: 'myapp',
  
  // Connection pool optimization
  maxPoolSize: 100,           // Maximum connections
  minPoolSize: 10,            // Minimum connections to maintain
  maxIdleTimeMS: 300000,      // Close idle connections after 5 minutes
  waitQueueTimeoutMS: 5000,   // Max time to wait for available connection
  
  // Server selection
  serverSelectionTimeoutMS: 30000,  // Timeout for selecting a server
  heartbeatFrequencyMS: 10000,      // How often to check server status
  
  // Socket settings
  socketTimeoutMS: 360000,    // Socket timeout (6 minutes)
  family: 4,                  // Force IPv4
});
```

## 🎯 Best Practices

### 1. Always Use TypeScript

SmartData is built for TypeScript. Using JavaScript means missing out on:
- Compile-time query validation
- IntelliSense for MongoDB operators
- Type-safe document updates
- Automatic refactoring support

### 2. Design Your Indexes Thoughtfully

```typescript
// ✅ Good: Index fields you query frequently
@index() public userId: string;
@index() public status: string;

// ❌ Bad: Over-indexing
@index() public everyField: string;  // Don't index everything!
```

### 3. Use Transactions for Critical Operations

```typescript
// ✅ Good: Atomic operations for financial transactions
await session.withTransaction(async () => {
  await debitAccount(fromAccount, amount, { session });
  await creditAccount(toAccount, amount, { session });
});

// ❌ Bad: Multiple operations without transactions
await debitAccount(fromAccount, amount);  // What if this fails?
await creditAccount(toAccount, amount);
```

### 4. Leverage Cursors for Large Datasets

```typescript
// ✅ Good: Stream processing
const cursor = await LargeCollection.getCursor({});
await cursor.forEach(async (doc) => {
  await processOne(doc);
});

// ❌ Bad: Loading everything into memory
const allDocs = await LargeCollection.getInstances({});  // Could OOM!
```

### 5. Implement Proper Error Handling

```typescript
// ✅ Good: Graceful error handling
try {
  const user = await User.getInstance({ id: userId });
  if (!user) {
    throw new Error('User not found');
  }
  await processUser(user);
} catch (error) {
  logger.error('Failed to process user:', error);
  await notificationService.alertAdmin(error);
}

// ❌ Bad: Ignoring errors
const user = await User.getInstance({ id: userId });
await processUser(user);  // What if user is null?
```

## 🔧 Troubleshooting

### Connection Issues

```typescript
// Enable debug logging
const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://localhost:27017',
  mongoDbName: 'myapp',
  // Add connection event handlers
});

db.mongoClient.on('error', (err) => {
  console.error('MongoDB connection error:', err);
});

db.mongoClient.on('timeout', () => {
  console.error('MongoDB connection timeout');
});

// Check connection status
const isConnected = db.status === 'connected';
```

### Memory Leaks

Always clean up resources:

```typescript
// Watchers
const watcher = await User.watch({});
// ... use watcher
await watcher.close();  // Always close!

// Cursors
const cursor = await User.getCursor({});
// ... use cursor
await cursor.close();  // Always close!

// Sessions
const session = db.startSession();
// ... use session
await session.endSession();  // Always end!
```

### Performance Issues

```typescript
// Profile slow queries
await db.mongoDb.setProfilingLevel('all');

// Check query execution
const stats = await User.collection.mongoDbCollection
  .find({ complex: 'query' })
  .explain('executionStats');
  
console.log('Query execution time:', stats.executionTimeMillis);
console.log('Documents examined:', stats.totalDocsExamined);
console.log('Index used:', stats.executionStats.executionStages.indexName);
```

## 📚 API Reference

### Core Classes

- `SmartdataDb` - Database connection and management
- `SmartDataDbDoc` - Base class for all documents
- `SmartdataCollection` - Collection management
- `SmartdataDbCursor` - Cursor for streaming operations
- `SmartdataDbWatcher` - Change stream watcher
- `SmartdataDistributedCoordinator` - Distributed coordination

### Decorators

- `@Collection(dbGetter)` - Define collection for a document class
- `@unI()` - Create unique index
- `@index(options)` - Create regular index
- `@svDb(options)` - Mark field as saveable with optional serialization
- `@searchable()` - Enable text search on field

### Key Methods

- `getInstances(filter)` - Find multiple documents
- `getInstance(filter)` - Find single document
- `getCursor(filter)` - Get cursor for streaming
- `watch(filter, options)` - Watch for changes
- `search(query, options)` - Lucene-style search
- `save(options)` - Save document
- `delete(options)` - Delete document
- `updateFromDb()` - Refresh from database

## 🌍 Community & Support

- 📖 **Documentation**: [https://code.foss.global/push.rocks/smartdata](https://code.foss.global/push.rocks/smartdata)
- 💬 **Issues**: [GitLab Issues](https://code.foss.global/push.rocks/smartdata/issues)
- 📧 **Email**: hello@task.vc

## License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository. 

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

### Company Information

Task Venture Capital GmbH  
Registered at District court Bremen HRB 35230 HB, Germany

For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.