5.3.0

changelog.md

@@ -1,5 +1,41 @@
 # Changelog
 
+## 2025-03-10 - 5.3.0 - feat(docs)
+Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.
+
+- Added pnpm installation command
+- Updated User model example to include ObjectId, Binary, and custom serialization
+- Expanded CRUD operations examples with cursor methods and deep query support
+- Enhanced sections on EasyStore, real-time data watching with RxJS integration, and managed collections
+- Included detailed examples for transactions, deep object queries, and document lifecycle hooks
+
+## 2025-02-03 - 5.2.12 - fix(documentation)
+Remove license badge from README
+
+- Removed the license badge from the README file, ensuring compliance with branding guidelines.
+
+## 2025-02-03 - 5.2.11 - fix(documentation)
+Updated project documentation for accuracy and added advanced feature details
+
+- Added details for EasyStore, Distributed Coordination, and Real-time Data Watching features.
+- Updated database connection setup instructions to include user authentication.
+- Re-organized advanced usage section to showcase additional features separately.
+
+## 2024-09-05 - 5.2.10 - fix(smartdata.classes.doc)
+Fix issue with array handling in convertFilterForMongoDb function
+
+- Corrected the logic to properly handle array filters in the convertFilterForMongoDb function to avoid incorrect assignments.
+
+## 2024-09-05 - 5.2.9 - fix(smartdata.classes.doc)
+Fixed issue with convertFilterForMongoDb to handle array operators.
+
+- Updated the convertFilterForMongoDb function in smartdata.classes.doc.ts to properly handle array operators like $in and $all.
+
+## 2024-09-05 - 5.2.8 - fix(smartdata.classes.doc)
+Fix key handling in convertFilterForMongoDb function
+
+- Fixed an issue in convertFilterForMongoDb that allowed keys with dots which could cause errors.
+
 ## 2024-09-05 - 5.2.7 - fix(core)
 Fixed issue with handling filter keys containing dots in smartdata.classes.doc.ts
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@push.rocks/smartdata",
-  "version": "5.2.7",
+  "version": "5.3.0",
   "private": false,
   "description": "An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.",
   "main": "dist_ts/index.js",

readme.md

@@ -1,5 +1,29 @@
 # @push.rocks/smartdata
-do more with data
+
+[![npm version](https://badge.fury.io/js/@push.rocks%2Fsmartdata.svg)](https://www.npmjs.com/package/@push.rocks/smartdata)
+
+A powerful TypeScript-first MongoDB wrapper that provides advanced features for distributed systems, real-time data synchronization, and easy data management.
+
+## Features
+
+- **Type-Safe MongoDB Integration**: Full TypeScript support with decorators for schema definition
+- **Document Management**: Type-safe CRUD operations with automatic timestamp tracking
+- **EasyStore**: Simple key-value storage with automatic persistence and sharing between instances
+- **Distributed Coordination**: Built-in support for leader election and distributed task management
+- **Real-time Data Sync**: Watchers for real-time data changes with RxJS integration
+- **Connection Management**: Automatic connection handling with connection pooling
+- **Collection Management**: Type-safe collection operations with automatic indexing
+- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>` 
+- **MongoDB Compatibility**: Support for all MongoDB query operators and advanced features
+- **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
+- **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
+- **Serialization Hooks**: Custom serialization and deserialization of document properties
+
+## Requirements
+
+- Node.js >= 16.x
+- MongoDB >= 4.4
+- TypeScript >= 4.x (for development)
 
 ## Install
 To install `@push.rocks/smartdata`, use npm:
@@ -8,26 +32,31 @@ To install `@push.rocks/smartdata`, use npm:
 npm install @push.rocks/smartdata --save
 ```
 
-This will add `@push.rocks/smartdata` to your project's dependencies.
+Or with pnpm:
+
+```bash
+pnpm add @push.rocks/smartdata
+```
 
 ## Usage
-`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage. Below are various scenarios demonstrating how to utilize this package effectively in a project.
+`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.
 
 ### Setting Up and Connecting to the Database
-Before interacting with the database, you need to set up and establish a connection. This is done by creating an instance of `SmartdataDb` and calling its `init` method with your MongoDB connection details.
+Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.
 
 ```typescript
 import { SmartdataDb } from '@push.rocks/smartdata';
 
 // Create a new instance of SmartdataDb with MongoDB connection details
 const db = new SmartdataDb({
-  mongoDbUrl: 'mongodb://localhost:27017',
+  mongoDbUrl: 'mongodb://<USERNAME>:<PASSWORD>@localhost:27017/<DBNAME>',
   mongoDbName: 'your-database-name',
   mongoDbUser: 'your-username',
   mongoDbPass: 'your-password',
 });
 
 // Initialize and connect to the database
+// This sets up a connection pool with max 100 connections
 await db.init();
 ```
 
@@ -35,7 +64,8 @@ await db.init();
 Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
 
 ```typescript
-import { SmartDataDbDoc, Collection, unI, svDb } from '@push.rocks/smartdata';
+import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index } from '@push.rocks/smartdata';
+import { ObjectId } from 'mongodb';
 
 @Collection(() => db)  // Associate this model with the database instance
 class User extends SmartDataDbDoc<User, User> {
@@ -46,8 +76,23 @@ class User extends SmartDataDbDoc<User, User> {
   public username: string;  // Mark 'username' to be saved in DB
   
   @svDb()
+  @index() // Create a regular index for this field
   public email: string;  // Mark 'email' to be saved in DB
   
+  @svDb()
+  @oid() // Automatically handle as ObjectId type
+  public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
+  
+  @svDb()
+  @bin() // Automatically handle as Binary data
+  public profilePicture: Buffer; // Will be automatically converted to/from Binary
+  
+  @svDb({ 
+    serialize: (data) => JSON.stringify(data), // Custom serialization
+    deserialize: (data) => JSON.parse(data)    // Custom deserialization
+  })
+  public preferences: Record<string, any>;
+  
   constructor(username: string, email: string) {
     super();
     this.username = username;
@@ -56,7 +101,7 @@ class User extends SmartDataDbDoc<User, User> {
 }
 ```
 
-### Performing CRUD Operations
+### CRUD Operations
 `@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.
 
 #### Create
@@ -72,6 +117,32 @@ const user = await User.getInstance({ username: 'myUsername' });
 
 // Fetch multiple users that match criteria
 const users = await User.getInstances({ email: 'myEmail@example.com' });
+
+// Using a cursor for large collections
+const cursor = await User.getCursor({ active: true });
+
+// Process documents one at a time (memory efficient)
+await cursor.forEach(async (user, index) => {
+  // Process each user with its position
+  console.log(`Processing user ${index}: ${user.username}`);
+});
+
+// Chain cursor methods like in the MongoDB native driver
+const paginatedCursor = await User.getCursor({ active: true })
+  .limit(10)  // Limit results
+  .skip(20)   // Skip first 20 results
+  .sort({ createdAt: -1 }); // Sort by creation date descending
+
+// Convert cursor to array (when you know the result set is small)
+const userArray = await paginatedCursor.toArray();
+
+// Other cursor operations
+const nextUser = await cursor.next(); // Get the next document
+const hasMoreUsers = await cursor.hasNext(); // Check if more documents exist
+const count = await cursor.count(); // Get the count of documents in the cursor
+
+// Always close cursors when done with them
+await cursor.close();
 ```
 
 #### Update
@@ -79,6 +150,15 @@ const users = await User.getInstances({ email: 'myEmail@example.com' });
 // Assuming 'user' is an instance of User
 user.email = 'newEmail@example.com';
 await user.save();  // Update the user in the database
+
+// Upsert operations (insert if not exists, update if exists)
+const upsertedUser = await User.upsert(
+  { id: 'user-123' },  // Query to find the user
+  {                    // Fields to update or insert
+    username: 'newUsername', 
+    email: 'newEmail@example.com'
+  }
+);
 ```
 
 #### Delete
@@ -87,13 +167,311 @@ await user.save();  // Update the user in the database
 await user.delete();  // Delete the user from the database
 ```
 
-### Advanced Usage
+## Advanced Features
-`@push.rocks/smartdata` also supports advanced features like watching for real-time changes in the database, handling distributed data coordination, and more. These features utilize MongoDB's capabilities to provide real-time data syncing and distributed systems coordination.
 
-### Conclusion
+### EasyStore
-With its focus on TypeScript, modern JavaScript syntax, and leveraging MongoDB's features, `@push.rocks/smartdata` offers a powerful toolkit for data handling and operations management in Node.js applications. Its design for ease of use, coupled with advanced features, makes it a versatile choice for developers looking to build efficient and scalable data-driven applications.
+EasyStore provides a simple key-value storage system with automatic persistence:
 
-For more details on usage and additional features, refer to the [official documentation](https://gitlab.com/push.rocks/smartdata#README) and explore the various classes and methods provided by `@push.rocks/smartdata`.
+```typescript
+// Create an EasyStore instance with a specific type
+interface ConfigStore {
+  apiKey: string;
+  settings: {
+    theme: string;
+    notifications: boolean;
+  };
+}
+
+// Create a type-safe EasyStore
+const store = await db.createEasyStore<ConfigStore>('app-config');
+
+// Write and read data with full type safety
+await store.writeKey('apiKey', 'secret-api-key-123');
+await store.writeKey('settings', { theme: 'dark', notifications: true });
+
+const apiKey = await store.readKey('apiKey'); // Type: string
+const settings = await store.readKey('settings'); // Type: { theme: string, notifications: boolean }
+
+// Check if a key exists
+const hasKey = await store.hasKey('apiKey'); // true
+
+// Delete a key
+await store.deleteKey('apiKey');
+```
+
+### Distributed Coordination
+Built-in support for distributed systems with leader election:
+
+```typescript
+// Create a distributed coordinator
+const coordinator = new SmartdataDistributedCoordinator(db);
+
+// Start coordination
+await coordinator.start();
+
+// Handle leadership changes
+coordinator.on('leadershipChange', (isLeader) => {
+  if (isLeader) {
+    // This instance is now the leader
+    // Run leader-specific tasks
+    startPeriodicJobs();
+  } else {
+    // This instance is no longer the leader
+    stopPeriodicJobs();
+  }
+});
+
+// Access leadership status anytime
+if (coordinator.isLeader) {
+  // Run leader-only operations
+}
+
+// Execute a task only on the leader
+await coordinator.executeIfLeader(async () => {
+  // This code only runs on the leader instance
+  await runImportantTask();
+});
+
+// Stop coordination when shutting down
+await coordinator.stop();
+```
+
+### Real-time Data Watching
+Watch for changes in your collections with RxJS integration using MongoDB Change Streams:
+
+```typescript
+// Create a watcher for a specific collection with a query filter
+const watcher = await User.watch({
+  active: true  // Only watch for changes to active users
+}, {
+  fullDocument: true,    // Include the full document in change notifications
+  bufferTimeMs: 100      // Buffer changes for 100ms to reduce notification frequency
+});
+
+// Subscribe to changes using RxJS
+watcher.changeSubject.subscribe((change) => {
+  console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
+  console.log('Document changed:', change.docInstance);   // The full document instance
+  
+  // Handle different types of changes
+  if (change.operationType === 'insert') {
+    console.log('New user created:', change.docInstance.username);
+  } else if (change.operationType === 'update') {
+    console.log('User updated:', change.docInstance.username);
+  } else if (change.operationType === 'delete') {
+    console.log('User deleted');
+  }
+});
+
+// Manual observation with event emitter pattern
+watcher.on('change', (change) => {
+  console.log('Document changed:', change);
+});
+
+// Stop watching when no longer needed
+await watcher.stop();
+```
+
+### Managed Collections
+For more complex data models that require additional context:
+
+```typescript
+@Collection(() => db)
+class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
+  @unI()
+  public id: string = 'unique-id';
+  
+  @svDb()
+  public data: string;
+  
+  @managed()
+  public manager: YourCustomManager;
+  
+  // The manager can provide additional functionality
+  async specialOperation() {
+    return this.manager.doSomethingSpecial(this);
+  }
+}
+```
+
+### Automatic Indexing
+Define indexes directly in your model class:
+
+```typescript
+@Collection(() => db)
+class Product extends SmartDataDbDoc<Product, Product> {
+  @unI() // Unique index
+  public id: string = 'product-id';
+  
+  @svDb()
+  @index() // Regular index for faster queries
+  public category: string;
+  
+  @svDb()
+  @index({ sparse: true }) // Sparse index with options
+  public optionalField?: string;
+  
+  // Compound indexes can be defined in the collection decorator
+  @Collection(() => db, {
+    indexMap: {
+      compoundIndex: { 
+        fields: { category: 1, name: 1 },
+        options: { background: true }
+      }
+    }
+  })
+}
+```
+
+### Transaction Support
+Use MongoDB transactions for atomic operations:
+
+```typescript
+const session = await db.startSession();
+try {
+  await session.withTransaction(async () => {
+    const user = await User.getInstance({ id: 'user-id' }, { session });
+    user.balance -= 100;
+    await user.save({ session });
+    
+    const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
+    recipient.balance += 100;
+    await user.save({ session });
+  });
+} finally {
+  await session.endSession();
+}
+```
+
+### Deep Object Queries
+SmartData provides fully type-safe deep property queries with the `DeepQuery` type:
+
+```typescript
+// If your document has nested objects
+class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
+  @unI()
+  public id: string = 'profile-id';
+  
+  @svDb()
+  public user: {
+    details: {
+      firstName: string;
+      lastName: string;
+      address: {
+        city: string;
+        country: string;
+      }
+    }
+  };
+}
+
+// Type-safe string literals for dot notation
+const usersInUSA = await UserProfile.getInstances({
+  'user.details.address.country': 'USA'
+});
+
+// Fully typed deep queries with the DeepQuery type
+import { DeepQuery } from '@push.rocks/smartdata';
+
+const typedQuery: DeepQuery<UserProfile> = {
+  id: 'profile-id',
+  'user.details.firstName': 'John',
+  'user.details.address.country': 'USA'
+};
+
+// TypeScript will error if paths are incorrect
+const results = await UserProfile.getInstances(typedQuery);
+
+// MongoDB query operators are supported
+const operatorQuery: DeepQuery<UserProfile> = {
+  'user.details.address.country': 'USA',
+  'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
+};
+
+const filteredResults = await UserProfile.getInstances(operatorQuery);
+```
+
+### Document Lifecycle Hooks
+Implement custom logic at different stages of a document's lifecycle:
+
+```typescript
+@Collection(() => db)
+class Order extends SmartDataDbDoc<Order, Order> {
+  @unI()
+  public id: string = 'order-id';
+  
+  @svDb()
+  public total: number;
+  
+  @svDb()
+  public items: string[];
+  
+  // Called before saving the document
+  async beforeSave() {
+    // Calculate total based on items
+    this.total = await calculateTotal(this.items);
+    
+    // Validate the document
+    if (this.items.length === 0) {
+      throw new Error('Order must have at least one item');
+    }
+  }
+  
+  // Called after the document is saved
+  async afterSave() {
+    // Notify other systems about the saved order
+    await notifyExternalSystems(this);
+  }
+  
+  // Called before deleting the document
+  async beforeDelete() {
+    // Check if order can be deleted
+    const canDelete = await checkOrderDeletable(this.id);
+    if (!canDelete) {
+      throw new Error('Order cannot be deleted');
+    }
+  }
+}
+```
+
+## Best Practices
+
+### Connection Management
+- Always call `db.init()` before using any database features
+- Use `db.disconnect()` when shutting down your application
+- Set appropriate connection pool sizes based on your application's needs
+
+### Document Design
+- Use appropriate decorators (`@svDb`, `@unI`, `@index`) to optimize database operations
+- Implement type-safe models by properly extending `SmartDataDbDoc`
+- Consider using interfaces to define document structures separately from implementation
+
+### Performance Optimization
+- Use cursors for large datasets instead of loading all documents into memory
+- Create appropriate indexes for frequent query patterns
+- Use projections to limit the fields returned when you don't need the entire document
+
+### Distributed Systems
+- Implement proper error handling for leader election events
+- Ensure all instances have synchronized clocks when using time-based coordination
+- Use the distributed coordinator's task management features for coordinated operations
+
+### Type Safety
+- Take advantage of the `DeepQuery<T>` type for fully type-safe queries
+- Define proper types for your document models to enhance IDE auto-completion
+- Use generic type parameters to specify exact document types when working with collections
+
+## Contributing
+
+We welcome contributions to @push.rocks/smartdata! Here's how you can help:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+Please make sure to update tests as appropriate and follow our coding standards.
 
 ## License and Legal Information
 

ts/00_commitinfo_data.ts

@@ -3,6 +3,6 @@
  */
 export const commitinfo = {
   name: '@push.rocks/smartdata',
-  version: '5.2.7',
+  version: '5.3.0',
   description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
 }

ts/smartdata.classes.doc.ts

@@ -53,15 +53,18 @@ export function unI() {
 
 export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
   const convertedFilter: { [key: string]: any } = {};
+
   const convertFilterArgument = (keyPathArg2: string, filterArg2: any) => {
-    if (typeof filterArg2 === 'object') {
+    if (Array.isArray(filterArg2)) {
+      // Directly assign arrays (they might be using operators like $in or $all)
+      convertFilterArgument(keyPathArg2, filterArg2[0]);
+    } else if (typeof filterArg2 === 'object' && filterArg2 !== null) {
       for (const key of Object.keys(filterArg2)) {
         if (key.startsWith('$')) {
           convertedFilter[keyPathArg2] = filterArg2;
           return;
         } else if (key.includes('.')) {
-          convertedFilter[keyPathArg2] = filterArg2;
+          throw new Error('keys cannot contain dots');
-          return;
         }
       }
       for (const key of Object.keys(filterArg2)) {
@@ -71,6 +74,7 @@ export const convertFilterForMongoDb = (filterArg: { [key: string]: any }) => {
       convertedFilter[keyPathArg2] = filterArg2;
     }
   };
+
   for (const key of Object.keys(filterArg)) {
     convertFilterArgument(key, filterArg[key]);
   }