A powerful TypeScript-first MongoDB wrapper that provides advanced features for distributed systems, real-time data synchronization, and easy data management.

Features

Type-Safe MongoDB Integration: Full TypeScript support with decorators for schema definition
Document Management: Type-safe CRUD operations with automatic timestamp tracking
EasyStore: Simple key-value storage with automatic persistence and sharing between instances
Distributed Coordination: Built-in support for leader election and distributed task management
Real-time Data Sync: Watchers for real-time data changes with RxJS integration
Connection Management: Automatic connection handling with connection pooling
Collection Management: Type-safe collection operations with automatic indexing
Deep Query Type Safety: Fully type-safe queries for nested object properties with DeepQuery<T>
MongoDB Compatibility: Support for all MongoDB query operators and advanced features
Enhanced Cursors: Chainable, type-safe cursor API with memory efficient data processing
Type Conversion: Automatic handling of MongoDB types like ObjectId and Binary data
Serialization Hooks: Custom serialization and deserialization of document properties

Requirements

Node.js >= 16.x
MongoDB >= 4.4
TypeScript >= 4.x (for development)

Install

To install @push.rocks/smartdata, use npm:

npm install @push.rocks/smartdata --save

Or with pnpm:

pnpm add @push.rocks/smartdata

Usage

@push.rocks/smartdata enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.

Setting Up and Connecting to the Database

Before interacting with the database, you need to set up and establish a connection. The SmartdataDb class handles connection pooling and automatic reconnection.

import { SmartdataDb } from '@push.rocks/smartdata';

// Create a new instance of SmartdataDb with MongoDB connection details
const db = new SmartdataDb({
  mongoDbUrl: 'mongodb://<USERNAME>:<PASSWORD>@localhost:27017/<DBNAME>',
  mongoDbName: 'your-database-name',
  mongoDbUser: 'your-username',
  mongoDbPass: 'your-password',
});

// Initialize and connect to the database
// This sets up a connection pool with max 100 connections
await db.init();

Defining Data Models

Data models in @push.rocks/smartdata are classes that represent collections and documents in your MongoDB database. Use decorators such as @Collection, @unI, and @svDb to define your data models.

import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index } from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';

@Collection(() => db)  // Associate this model with the database instance
class User extends SmartDataDbDoc<User, User> {
  @unI()
  public id: string = 'unique-user-id'; // Mark 'id' as a unique index
  
  @svDb()
  public username: string;  // Mark 'username' to be saved in DB
  
  @svDb()
  @index() // Create a regular index for this field
  public email: string;  // Mark 'email' to be saved in DB
  
  @svDb()
  @oid() // Automatically handle as ObjectId type
  public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
  
  @svDb()
  @bin() // Automatically handle as Binary data
  public profilePicture: Buffer; // Will be automatically converted to/from Binary
  
  @svDb({ 
    serialize: (data) => JSON.stringify(data), // Custom serialization
    deserialize: (data) => JSON.parse(data)    // Custom deserialization
  })
  public preferences: Record<string, any>;
  
  constructor(username: string, email: string) {
    super();
    this.username = username;
    this.email = email;
  }
}

CRUD Operations

@push.rocks/smartdata simplifies CRUD operations with intuitive methods on model instances.

Create

const newUser = new User('myUsername', 'myEmail@example.com');
await newUser.save();  // Save the new user to the database

Read

// Fetch a single user by a unique attribute
const user = await User.getInstance({ username: 'myUsername' });

// Fetch multiple users that match criteria
const users = await User.getInstances({ email: 'myEmail@example.com' });

// Using a cursor for large collections
const cursor = await User.getCursor({ active: true });

// Process documents one at a time (memory efficient)
await cursor.forEach(async (user, index) => {
  // Process each user with its position
  console.log(`Processing user ${index}: ${user.username}`);
});

// Chain cursor methods like in the MongoDB native driver
const paginatedCursor = await User.getCursor({ active: true })
  .limit(10)  // Limit results
  .skip(20)   // Skip first 20 results
  .sort({ createdAt: -1 }); // Sort by creation date descending

// Convert cursor to array (when you know the result set is small)
const userArray = await paginatedCursor.toArray();

// Other cursor operations
const nextUser = await cursor.next(); // Get the next document
const hasMoreUsers = await cursor.hasNext(); // Check if more documents exist
const count = await cursor.count(); // Get the count of documents in the cursor

// Always close cursors when done with them
await cursor.close();

Update

// Assuming 'user' is an instance of User
user.email = 'newEmail@example.com';
await user.save();  // Update the user in the database

// Upsert operations (insert if not exists, update if exists)
const upsertedUser = await User.upsert(
  { id: 'user-123' },  // Query to find the user
  {                    // Fields to update or insert
    username: 'newUsername', 
    email: 'newEmail@example.com'
  }
);

Delete

// Assuming 'user' is an instance of User
await user.delete();  // Delete the user from the database

Advanced Features

EasyStore

EasyStore provides a simple key-value storage system with automatic persistence:

// Create an EasyStore instance with a specific type
interface ConfigStore {
  apiKey: string;
  settings: {
    theme: string;
    notifications: boolean;
  };
}

// Create a type-safe EasyStore
const store = await db.createEasyStore<ConfigStore>('app-config');

// Write and read data with full type safety
await store.writeKey('apiKey', 'secret-api-key-123');
await store.writeKey('settings', { theme: 'dark', notifications: true });

const apiKey = await store.readKey('apiKey'); // Type: string
const settings = await store.readKey('settings'); // Type: { theme: string, notifications: boolean }

// Check if a key exists
const hasKey = await store.hasKey('apiKey'); // true

// Delete a key
await store.deleteKey('apiKey');

Distributed Coordination

Built-in support for distributed systems with leader election:

// Create a distributed coordinator
const coordinator = new SmartdataDistributedCoordinator(db);

// Start coordination
await coordinator.start();

// Handle leadership changes
coordinator.on('leadershipChange', (isLeader) => {
  if (isLeader) {
    // This instance is now the leader
    // Run leader-specific tasks
    startPeriodicJobs();
  } else {
    // This instance is no longer the leader
    stopPeriodicJobs();
  }
});

// Access leadership status anytime
if (coordinator.isLeader) {
  // Run leader-only operations
}

// Execute a task only on the leader
await coordinator.executeIfLeader(async () => {
  // This code only runs on the leader instance
  await runImportantTask();
});

// Stop coordination when shutting down
await coordinator.stop();

Real-time Data Watching

Watch for changes in your collections with RxJS integration using MongoDB Change Streams:

// Create a watcher for a specific collection with a query filter
const watcher = await User.watch({
  active: true  // Only watch for changes to active users
}, {
  fullDocument: true,    // Include the full document in change notifications
  bufferTimeMs: 100      // Buffer changes for 100ms to reduce notification frequency
});

// Subscribe to changes using RxJS
watcher.changeSubject.subscribe((change) => {
  console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
  console.log('Document changed:', change.docInstance);   // The full document instance
  
  // Handle different types of changes
  if (change.operationType === 'insert') {
    console.log('New user created:', change.docInstance.username);
  } else if (change.operationType === 'update') {
    console.log('User updated:', change.docInstance.username);
  } else if (change.operationType === 'delete') {
    console.log('User deleted');
  }
});

// Manual observation with event emitter pattern
watcher.on('change', (change) => {
  console.log('Document changed:', change);
});

// Stop watching when no longer needed
await watcher.stop();

Managed Collections

For more complex data models that require additional context:

@Collection(() => db)
class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
  @unI()
  public id: string = 'unique-id';
  
  @svDb()
  public data: string;
  
  @managed()
  public manager: YourCustomManager;
  
  // The manager can provide additional functionality
  async specialOperation() {
    return this.manager.doSomethingSpecial(this);
  }
}

Automatic Indexing

Define indexes directly in your model class:

@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
  @unI() // Unique index
  public id: string = 'product-id';
  
  @svDb()
  @index() // Regular index for faster queries
  public category: string;
  
  @svDb()
  @index({ sparse: true }) // Sparse index with options
  public optionalField?: string;
  
  // Compound indexes can be defined in the collection decorator
  @Collection(() => db, {
    indexMap: {
      compoundIndex: { 
        fields: { category: 1, name: 1 },
        options: { background: true }
      }
    }
  })
}

Transaction Support

Use MongoDB transactions for atomic operations:

const session = await db.startSession();
try {
  await session.withTransaction(async () => {
    const user = await User.getInstance({ id: 'user-id' }, { session });
    user.balance -= 100;
    await user.save({ session });
    
    const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
    recipient.balance += 100;
    await user.save({ session });
  });
} finally {
  await session.endSession();
}

Deep Object Queries

SmartData provides fully type-safe deep property queries with the DeepQuery type:

// If your document has nested objects
class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
  @unI()
  public id: string = 'profile-id';
  
  @svDb()
  public user: {
    details: {
      firstName: string;
      lastName: string;
      address: {
        city: string;
        country: string;
      }
    }
  };
}

// Type-safe string literals for dot notation
const usersInUSA = await UserProfile.getInstances({
  'user.details.address.country': 'USA'
});

// Fully typed deep queries with the DeepQuery type
import { DeepQuery } from '@push.rocks/smartdata';

const typedQuery: DeepQuery<UserProfile> = {
  id: 'profile-id',
  'user.details.firstName': 'John',
  'user.details.address.country': 'USA'
};

// TypeScript will error if paths are incorrect
const results = await UserProfile.getInstances(typedQuery);

// MongoDB query operators are supported
const operatorQuery: DeepQuery<UserProfile> = {
  'user.details.address.country': 'USA',
  'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
};

const filteredResults = await UserProfile.getInstances(operatorQuery);

Document Lifecycle Hooks

Implement custom logic at different stages of a document's lifecycle:

@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
  @unI()
  public id: string = 'order-id';
  
  @svDb()
  public total: number;
  
  @svDb()
  public items: string[];
  
  // Called before saving the document
  async beforeSave() {
    // Calculate total based on items
    this.total = await calculateTotal(this.items);
    
    // Validate the document
    if (this.items.length === 0) {
      throw new Error('Order must have at least one item');
    }
  }
  
  // Called after the document is saved
  async afterSave() {
    // Notify other systems about the saved order
    await notifyExternalSystems(this);
  }
  
  // Called before deleting the document
  async beforeDelete() {
    // Check if order can be deleted
    const canDelete = await checkOrderDeletable(this.id);
    if (!canDelete) {
      throw new Error('Order cannot be deleted');
    }
  }
}

Best Practices

Connection Management

Always call db.init() before using any database features
Use db.disconnect() when shutting down your application
Set appropriate connection pool sizes based on your application's needs

Document Design

Use appropriate decorators (@svDb, @unI, @index) to optimize database operations
Implement type-safe models by properly extending SmartDataDbDoc
Consider using interfaces to define document structures separately from implementation

Performance Optimization

Use cursors for large datasets instead of loading all documents into memory
Create appropriate indexes for frequent query patterns
Use projections to limit the fields returned when you don't need the entire document

Distributed Systems

Implement proper error handling for leader election events
Ensure all instances have synchronized clocks when using time-based coordination
Use the distributed coordinator's task management features for coordinated operations

Type Safety

Take advantage of the DeepQuery<T> type for fully type-safe queries
Define proper types for your document models to enhance IDE auto-completion
Use generic type parameters to specify exact document types when working with collections

Contributing

We welcome contributions to @push.rocks/smartdata! Here's how you can help:

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Please make sure to update tests as appropriate and follow our coding standards.

License and Legal Information

This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the license file within this repository.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.

readme.md

@push.rocks/smartdata