push.rocks/smartdata
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(docs): Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.

Browse Source
This commit is contained in:
Philipp Kunz 2025-03-10 23:55:44 +00:00
parent c8d51a30d8
commit ebaf3e685c
3 changed files with 336 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
changelog.md
View File

@ -1,5 +1,14 @@
# Changelog
## 2025-03-10 - 5.3.0 - feat(docs)
Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.
- Added pnpm installation command
- Updated User model example to include ObjectId, Binary, and custom serialization
- Expanded CRUD operations examples with cursor methods and deep query support
- Enhanced sections on EasyStore, real-time data watching with RxJS integration, and managed collections
- Included detailed examples for transactions, deep object queries, and document lifecycle hooks
## 2025-02-03 - 5.2.12 - fix(documentation)
Remove license badge from README

345
readme.md
View File

@ -7,11 +7,17 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
## Features
- **Type-Safe MongoDB Integration**: Full TypeScript support with decorators for schema definition
- **Document Management**: Type-safe CRUD operations with automatic timestamp tracking
- **EasyStore**: Simple key-value storage with automatic persistence and sharing between instances
- **Distributed Coordination**: Built-in support for leader election and distributed task management
- **Real-time Data Sync**: Watchers for real-time data changes and synchronization
- **Real-time Data Sync**: Watchers for real-time data changes with RxJS integration
- **Connection Management**: Automatic connection handling with connection pooling
- **Collection Management**: Type-safe collection operations with automatic indexing
- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>`
- **MongoDB Compatibility**: Support for all MongoDB query operators and advanced features
- **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
- **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
- **Serialization Hooks**: Custom serialization and deserialization of document properties
## Requirements
@ -26,10 +32,14 @@ To install `@push.rocks/smartdata`, use npm:
npm install @push.rocks/smartdata --save
```
This will add `@push.rocks/smartdata` to your project's dependencies.
Or with pnpm:
```bash
pnpm add @push.rocks/smartdata
```
## Usage
`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage. Below are various scenarios demonstrating how to utilize this package effectively in a project.
`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.
### Setting Up and Connecting to the Database
Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.
@ -54,7 +64,8 @@ await db.init();
Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
```typescript
import { SmartDataDbDoc, Collection, unI, svDb } from '@push.rocks/smartdata';
import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index } from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';
@Collection(() => db) // Associate this model with the database instance
class User extends SmartDataDbDoc<User, User> {
@ -65,8 +76,23 @@ class User extends SmartDataDbDoc<User, User> {
public username: string; // Mark 'username' to be saved in DB
@svDb()
@index() // Create a regular index for this field
public email: string; // Mark 'email' to be saved in DB
@svDb()
@oid() // Automatically handle as ObjectId type
public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
@svDb()
@bin() // Automatically handle as Binary data
public profilePicture: Buffer; // Will be automatically converted to/from Binary
@svDb({
serialize: (data) => JSON.stringify(data), // Custom serialization
deserialize: (data) => JSON.parse(data) // Custom deserialization
})
public preferences: Record<string, any>;
constructor(username: string, email: string) {
super();
this.username = username;
@ -75,7 +101,7 @@ class User extends SmartDataDbDoc<User, User> {
}
```
### Performing CRUD Operations
### CRUD Operations
`@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.
#### Create
@ -91,6 +117,32 @@ const user = await User.getInstance({ username: 'myUsername' });
// Fetch multiple users that match criteria
const users = await User.getInstances({ email: 'myEmail@example.com' });
// Using a cursor for large collections
const cursor = await User.getCursor({ active: true });
// Process documents one at a time (memory efficient)
await cursor.forEach(async (user, index) => {
// Process each user with its position
console.log(`Processing user ${index}: ${user.username}`);
});
// Chain cursor methods like in the MongoDB native driver
const paginatedCursor = await User.getCursor({ active: true })
.limit(10) // Limit results
.skip(20) // Skip first 20 results
.sort({ createdAt: -1 }); // Sort by creation date descending
// Convert cursor to array (when you know the result set is small)
const userArray = await paginatedCursor.toArray();
// Other cursor operations
const nextUser = await cursor.next(); // Get the next document
const hasMoreUsers = await cursor.hasNext(); // Check if more documents exist
const count = await cursor.count(); // Get the count of documents in the cursor
// Always close cursors when done with them
await cursor.close();
```
#### Update
@ -98,6 +150,15 @@ const users = await User.getInstances({ email: 'myEmail@example.com' });
// Assuming 'user' is an instance of User
user.email = 'newEmail@example.com';
await user.save(); // Update the user in the database
// Upsert operations (insert if not exists, update if exists)
const upsertedUser = await User.upsert(
{ id: 'user-123' }, // Query to find the user
{ // Fields to update or insert
username: 'newUsername',
email: 'newEmail@example.com'
}
);
```
#### Delete
@ -106,21 +167,39 @@ await user.save(); // Update the user in the database
await user.delete(); // Delete the user from the database
```
### Advanced Features
## Advanced Features
#### EasyStore
### EasyStore
EasyStore provides a simple key-value storage system with automatic persistence:
```typescript
// Create an EasyStore instance
const store = await db.createEasyStore<YourDataType>('store-name');
// Create an EasyStore instance with a specific type
interface ConfigStore {
apiKey: string;
settings: {
theme: string;
notifications: boolean;
};
}
// Write and read data
await store.writeKey('key', value);
const value = await store.readKey('key');
// Create a type-safe EasyStore
const store = await db.createEasyStore<ConfigStore>('app-config');
// Write and read data with full type safety
await store.writeKey('apiKey', 'secret-api-key-123');
await store.writeKey('settings', { theme: 'dark', notifications: true });
const apiKey = await store.readKey('apiKey'); // Type: string
const settings = await store.readKey('settings'); // Type: { theme: string, notifications: boolean }
// Check if a key exists
const hasKey = await store.hasKey('apiKey'); // true
// Delete a key
await store.deleteKey('apiKey');
```
#### Distributed Coordination
### Distributed Coordination
Built-in support for distributed systems with leader election:
```typescript
@ -134,27 +213,253 @@ await coordinator.start();
coordinator.on('leadershipChange', (isLeader) => {
if (isLeader) {
// This instance is now the leader
// Run leader-specific tasks
startPeriodicJobs();
} else {
// This instance is no longer the leader
stopPeriodicJobs();
}
});
// Access leadership status anytime
if (coordinator.isLeader) {
// Run leader-only operations
}
// Execute a task only on the leader
await coordinator.executeIfLeader(async () => {
// This code only runs on the leader instance
await runImportantTask();
});
// Stop coordination when shutting down
await coordinator.stop();
```
#### Real-time Data Watching
Watch for changes in your collections:
### Real-time Data Watching
Watch for changes in your collections with RxJS integration using MongoDB Change Streams:
```typescript
const watcher = new SmartdataDbWatcher(collection);
// Create a watcher for a specific collection with a query filter
const watcher = await User.watch({
active: true // Only watch for changes to active users
}, {
fullDocument: true, // Include the full document in change notifications
bufferTimeMs: 100 // Buffer changes for 100ms to reduce notification frequency
});
// Subscribe to changes using RxJS
watcher.changeSubject.subscribe((change) => {
console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
console.log('Document changed:', change.docInstance); // The full document instance
// Handle different types of changes
if (change.operationType === 'insert') {
console.log('New user created:', change.docInstance.username);
} else if (change.operationType === 'update') {
console.log('User updated:', change.docInstance.username);
} else if (change.operationType === 'delete') {
console.log('User deleted');
}
});
// Manual observation with event emitter pattern
watcher.on('change', (change) => {
console.log('Document changed:', change);
});
await watcher.start();
// Stop watching when no longer needed
await watcher.stop();
```
### Conclusion
With its focus on TypeScript, modern JavaScript syntax, and leveraging MongoDB's features, `@push.rocks/smartdata` offers a powerful toolkit for data handling and operations management in Node.js applications. Its design for ease of use, coupled with advanced features, makes it a versatile choice for developers looking to build efficient and scalable data-driven applications.
### Managed Collections
For more complex data models that require additional context:
For more details on usage and additional features, refer to the [official documentation](https://gitlab.com/push.rocks/smartdata/-/blob/master/README.md) and explore the various classes and methods provided by `@push.rocks/smartdata`.
```typescript
@Collection(() => db)
class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
@unI()
public id: string = 'unique-id';
@svDb()
public data: string;
@managed()
public manager: YourCustomManager;
// The manager can provide additional functionality
async specialOperation() {
return this.manager.doSomethingSpecial(this);
}
}
```
### Automatic Indexing
Define indexes directly in your model class:
```typescript
@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
@unI() // Unique index
public id: string = 'product-id';
@svDb()
@index() // Regular index for faster queries
public category: string;
@svDb()
@index({ sparse: true }) // Sparse index with options
public optionalField?: string;
// Compound indexes can be defined in the collection decorator
@Collection(() => db, {
indexMap: {
compoundIndex: {
fields: { category: 1, name: 1 },
options: { background: true }
}
}
})
}
```
### Transaction Support
Use MongoDB transactions for atomic operations:
```typescript
const session = await db.startSession();
try {
await session.withTransaction(async () => {
const user = await User.getInstance({ id: 'user-id' }, { session });
user.balance -= 100;
await user.save({ session });
const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
recipient.balance += 100;
await user.save({ session });
});
} finally {
await session.endSession();
}
```
### Deep Object Queries
SmartData provides fully type-safe deep property queries with the `DeepQuery` type:
```typescript
// If your document has nested objects
class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
@unI()
public id: string = 'profile-id';
@svDb()
public user: {
details: {
firstName: string;
lastName: string;
address: {
city: string;
country: string;
}
}
};
}
// Type-safe string literals for dot notation
const usersInUSA = await UserProfile.getInstances({
'user.details.address.country': 'USA'
});
// Fully typed deep queries with the DeepQuery type
import { DeepQuery } from '@push.rocks/smartdata';
const typedQuery: DeepQuery<UserProfile> = {
id: 'profile-id',
'user.details.firstName': 'John',
'user.details.address.country': 'USA'
};
// TypeScript will error if paths are incorrect
const results = await UserProfile.getInstances(typedQuery);
// MongoDB query operators are supported
const operatorQuery: DeepQuery<UserProfile> = {
'user.details.address.country': 'USA',
'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
};
const filteredResults = await UserProfile.getInstances(operatorQuery);
```
### Document Lifecycle Hooks
Implement custom logic at different stages of a document's lifecycle:
```typescript
@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
@unI()
public id: string = 'order-id';
@svDb()
public total: number;
@svDb()
public items: string[];
// Called before saving the document
async beforeSave() {
// Calculate total based on items
this.total = await calculateTotal(this.items);
// Validate the document
if (this.items.length === 0) {
throw new Error('Order must have at least one item');
}
}
// Called after the document is saved
async afterSave() {
// Notify other systems about the saved order
await notifyExternalSystems(this);
}
// Called before deleting the document
async beforeDelete() {
// Check if order can be deleted
const canDelete = await checkOrderDeletable(this.id);
if (!canDelete) {
throw new Error('Order cannot be deleted');
}
}
}
```
## Best Practices
### Connection Management
- Always call `db.init()` before using any database features
- Use `db.disconnect()` when shutting down your application
- Set appropriate connection pool sizes based on your application's needs
### Document Design
- Use appropriate decorators (`@svDb`, `@unI`, `@index`) to optimize database operations
- Implement type-safe models by properly extending `SmartDataDbDoc`
- Consider using interfaces to define document structures separately from implementation
### Performance Optimization
- Use cursors for large datasets instead of loading all documents into memory
- Create appropriate indexes for frequent query patterns
- Use projections to limit the fields returned when you don't need the entire document
### Distributed Systems
- Implement proper error handling for leader election events
- Ensure all instances have synchronized clocks when using time-based coordination
- Use the distributed coordinator's task management features for coordinated operations
### Type Safety
- Take advantage of the `DeepQuery<T>` type for fully type-safe queries
- Define proper types for your document models to enhance IDE auto-completion
- Use generic type parameters to specify exact document types when working with collections
## Contributing

2
ts/00_commitinfo_data.ts
View File

@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smartdata',
version: '5.2.12',
version: '5.3.0',
description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
}