push.rocks/smartdata
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(docs): Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.

Browse Source
This commit is contained in:
Philipp Kunz 2025-03-10 23:55:44 +00:00
parent c8d51a30d8
commit ebaf3e685c
3 changed files with 336 additions and 22 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
changelog.md
View File

@ -1,5 +1,14 @@
# Changelog
## 2025-03-10 - 5.3.0 - feat(docs)
Enhance documentation with updated installation instructions and comprehensive usage examples covering advanced features such as deep queries, automatic indexing, and distributed coordination.
- Added pnpm installation command
- Updated User model example to include ObjectId, Binary, and custom serialization
- Expanded CRUD operations examples with cursor methods and deep query support
- Enhanced sections on EasyStore, real-time data watching with RxJS integration, and managed collections
- Included detailed examples for transactions, deep object queries, and document lifecycle hooks
## 2025-02-03 - 5.2.12 - fix(documentation)
Remove license badge from README

347
readme.md
View File

@ -7,11 +7,17 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
## Features
- **Type-Safe MongoDB Integration**: Full TypeScript support with decorators for schema definition
- **Document Management**: Type-safe CRUD operations with automatic timestamp tracking
- **EasyStore**: Simple key-value storage with automatic persistence and sharing between instances
- **Distributed Coordination**: Built-in support for leader election and distributed task management
- **Real-time Data Sync**: Watchers for real-time data changes and synchronization
- **Real-time Data Sync**: Watchers for real-time data changes with RxJS integration
- **Connection Management**: Automatic connection handling with connection pooling
- **Collection Management**: Type-safe collection operations with automatic indexing
- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>`
- **MongoDB Compatibility**: Support for all MongoDB query operators and advanced features
- **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
- **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
- **Serialization Hooks**: Custom serialization and deserialization of document properties
## Requirements
@ -26,10 +32,14 @@ To install `@push.rocks/smartdata`, use npm:
npm install @push.rocks/smartdata --save
```
This will add `@push.rocks/smartdata` to your project's dependencies.
Or with pnpm:
```bash
pnpm add @push.rocks/smartdata
```
## Usage
`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage. Below are various scenarios demonstrating how to utilize this package effectively in a project.
`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.
### Setting Up and Connecting to the Database
Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.
@ -54,7 +64,8 @@ await db.init();
Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
```typescript
import { SmartDataDbDoc, Collection, unI, svDb } from '@push.rocks/smartdata';
import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index } from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';
@Collection(() => db) // Associate this model with the database instance
class User extends SmartDataDbDoc<User, User> {
@ -65,8 +76,23 @@ class User extends SmartDataDbDoc<User, User> {
public username: string; // Mark 'username' to be saved in DB
@svDb()
@index() // Create a regular index for this field
public email: string; // Mark 'email' to be saved in DB
@svDb()
@oid() // Automatically handle as ObjectId type
public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
@svDb()
@bin() // Automatically handle as Binary data
public profilePicture: Buffer; // Will be automatically converted to/from Binary
@svDb({
serialize: (data) => JSON.stringify(data), // Custom serialization
deserialize: (data) => JSON.parse(data) // Custom deserialization
})
public preferences: Record<string, any>;
constructor(username: string, email: string) {
super();
this.username = username;
@ -75,7 +101,7 @@ class User extends SmartDataDbDoc<User, User> {
}
```
### Performing CRUD Operations
### CRUD Operations
`@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.
#### Create
@ -91,6 +117,32 @@ const user = await User.getInstance({ username: 'myUsername' });
// Fetch multiple users that match criteria
const users = await User.getInstances({ email: 'myEmail@example.com' });
// Using a cursor for large collections
const cursor = await User.getCursor({ active: true });
// Process documents one at a time (memory efficient)
await cursor.forEach(async (user, index) => {
// Process each user with its position
console.log(`Processing user ${index}: ${user.username}`);
});
// Chain cursor methods like in the MongoDB native driver
const paginatedCursor = await User.getCursor({ active: true })
.limit(10) // Limit results
.skip(20) // Skip first 20 results
.sort({ createdAt: -1 }); // Sort by creation date descending
// Convert cursor to array (when you know the result set is small)
const userArray = await paginatedCursor.toArray();
// Other cursor operations
const nextUser = await cursor.next(); // Get the next document
const hasMoreUsers = await cursor.hasNext(); // Check if more documents exist
const count = await cursor.count(); // Get the count of documents in the cursor
// Always close cursors when done with them
await cursor.close();
```
#### Update
@ -98,6 +150,15 @@ const users = await User.getInstances({ email: 'myEmail@example.com' });
// Assuming 'user' is an instance of User
user.email = 'newEmail@example.com';
await user.save(); // Update the user in the database
// Upsert operations (insert if not exists, update if exists)
const upsertedUser = await User.upsert(
{ id: 'user-123' }, // Query to find the user
{ // Fields to update or insert
username: 'newUsername',
email: 'newEmail@example.com'
}
);
```
#### Delete
@ -106,21 +167,39 @@ await user.save(); // Update the user in the database
await user.delete(); // Delete the user from the database
```
### Advanced Features
## Advanced Features
#### EasyStore
### EasyStore
EasyStore provides a simple key-value storage system with automatic persistence:
```typescript
// Create an EasyStore instance
const store = await db.createEasyStore<YourDataType>('store-name');
// Create an EasyStore instance with a specific type
interface ConfigStore {
apiKey: string;
settings: {
theme: string;
notifications: boolean;
};
}
// Write and read data
await store.writeKey('key', value);
const value = await store.readKey('key');
// Create a type-safe EasyStore
const store = await db.createEasyStore<ConfigStore>('app-config');
// Write and read data with full type safety
await store.writeKey('apiKey', 'secret-api-key-123');
await store.writeKey('settings', { theme: 'dark', notifications: true });
const apiKey = await store.readKey('apiKey'); // Type: string
const settings = await store.readKey('settings'); // Type: { theme: string, notifications: boolean }
// Check if a key exists
const hasKey = await store.hasKey('apiKey'); // true
// Delete a key
await store.deleteKey('apiKey');
```
#### Distributed Coordination
### Distributed Coordination
Built-in support for distributed systems with leader election:
```typescript
@ -134,27 +213,253 @@ await coordinator.start();
coordinator.on('leadershipChange', (isLeader) => {
if (isLeader) {
// This instance is now the leader
// Run leader-specific tasks
startPeriodicJobs();
} else {
// This instance is no longer the leader
stopPeriodicJobs();
}
});
// Access leadership status anytime
if (coordinator.isLeader) {
// Run leader-only operations
}
// Execute a task only on the leader
await coordinator.executeIfLeader(async () => {
// This code only runs on the leader instance
await runImportantTask();
});
// Stop coordination when shutting down
await coordinator.stop();
```
#### Real-time Data Watching
Watch for changes in your collections:
### Real-time Data Watching
Watch for changes in your collections with RxJS integration using MongoDB Change Streams:
```typescript
const watcher = new SmartdataDbWatcher(collection);
// Create a watcher for a specific collection with a query filter
const watcher = await User.watch({
active: true // Only watch for changes to active users
}, {
fullDocument: true, // Include the full document in change notifications
bufferTimeMs: 100 // Buffer changes for 100ms to reduce notification frequency
});
// Subscribe to changes using RxJS
watcher.changeSubject.subscribe((change) => {
console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
console.log('Document changed:', change.docInstance); // The full document instance
// Handle different types of changes
if (change.operationType === 'insert') {
console.log('New user created:', change.docInstance.username);
} else if (change.operationType === 'update') {
console.log('User updated:', change.docInstance.username);
} else if (change.operationType === 'delete') {
console.log('User deleted');
}
});
// Manual observation with event emitter pattern
watcher.on('change', (change) => {
console.log('Document changed:', change);
});
await watcher.start();
// Stop watching when no longer needed
await watcher.stop();
```
### Conclusion
With its focus on TypeScript, modern JavaScript syntax, and leveraging MongoDB's features, `@push.rocks/smartdata` offers a powerful toolkit for data handling and operations management in Node.js applications. Its design for ease of use, coupled with advanced features, makes it a versatile choice for developers looking to build efficient and scalable data-driven applications.
### Managed Collections
For more complex data models that require additional context:
For more details on usage and additional features, refer to the [official documentation](https://gitlab.com/push.rocks/smartdata/-/blob/master/README.md) and explore the various classes and methods provided by `@push.rocks/smartdata`.
```typescript
@Collection(() => db)
class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
@unI()
public id: string = 'unique-id';
@svDb()
public data: string;
@managed()
public manager: YourCustomManager;
// The manager can provide additional functionality
async specialOperation() {
return this.manager.doSomethingSpecial(this);
}
}
```
### Automatic Indexing
Define indexes directly in your model class:
```typescript
@Collection(() => db)
class Product extends SmartDataDbDoc<Product, Product> {
@unI() // Unique index
public id: string = 'product-id';
@svDb()
@index() // Regular index for faster queries
public category: string;
@svDb()
@index({ sparse: true }) // Sparse index with options
public optionalField?: string;
// Compound indexes can be defined in the collection decorator
@Collection(() => db, {
indexMap: {
compoundIndex: {
fields: { category: 1, name: 1 },
options: { background: true }
}
}
})
}
```
### Transaction Support
Use MongoDB transactions for atomic operations:
```typescript
const session = await db.startSession();
try {
await session.withTransaction(async () => {
const user = await User.getInstance({ id: 'user-id' }, { session });
user.balance -= 100;
await user.save({ session });
const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
recipient.balance += 100;
await user.save({ session });
});
} finally {
await session.endSession();
}
```
### Deep Object Queries
SmartData provides fully type-safe deep property queries with the `DeepQuery` type:
```typescript
// If your document has nested objects
class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
@unI()
public id: string = 'profile-id';
@svDb()
public user: {
details: {
firstName: string;
lastName: string;
address: {
city: string;
country: string;
}
}
};
}
// Type-safe string literals for dot notation
const usersInUSA = await UserProfile.getInstances({
'user.details.address.country': 'USA'
});
// Fully typed deep queries with the DeepQuery type
import { DeepQuery } from '@push.rocks/smartdata';
const typedQuery: DeepQuery<UserProfile> = {
id: 'profile-id',
'user.details.firstName': 'John',
'user.details.address.country': 'USA'
};
// TypeScript will error if paths are incorrect
const results = await UserProfile.getInstances(typedQuery);
// MongoDB query operators are supported
const operatorQuery: DeepQuery<UserProfile> = {
'user.details.address.country': 'USA',
'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
};
const filteredResults = await UserProfile.getInstances(operatorQuery);
```
### Document Lifecycle Hooks
Implement custom logic at different stages of a document's lifecycle:
```typescript
@Collection(() => db)
class Order extends SmartDataDbDoc<Order, Order> {
@unI()
public id: string = 'order-id';
@svDb()
public total: number;
@svDb()
public items: string[];
// Called before saving the document
async beforeSave() {
// Calculate total based on items
this.total = await calculateTotal(this.items);
// Validate the document
if (this.items.length === 0) {
throw new Error('Order must have at least one item');
}
}
// Called after the document is saved
async afterSave() {
// Notify other systems about the saved order
await notifyExternalSystems(this);
}
// Called before deleting the document
async beforeDelete() {
// Check if order can be deleted
const canDelete = await checkOrderDeletable(this.id);
if (!canDelete) {
throw new Error('Order cannot be deleted');
}
}
}
```
## Best Practices
### Connection Management
- Always call `db.init()` before using any database features
- Use `db.disconnect()` when shutting down your application
- Set appropriate connection pool sizes based on your application's needs
### Document Design
- Use appropriate decorators (`@svDb`, `@unI`, `@index`) to optimize database operations
- Implement type-safe models by properly extending `SmartDataDbDoc`
- Consider using interfaces to define document structures separately from implementation
### Performance Optimization
- Use cursors for large datasets instead of loading all documents into memory
- Create appropriate indexes for frequent query patterns
- Use projections to limit the fields returned when you don't need the entire document
### Distributed Systems
- Implement proper error handling for leader election events
- Ensure all instances have synchronized clocks when using time-based coordination
- Use the distributed coordinator's task management features for coordinated operations
### Type Safety
- Take advantage of the `DeepQuery<T>` type for fully type-safe queries
- Define proper types for your document models to enhance IDE auto-completion
- Use generic type parameters to specify exact document types when working with collections
## Contributing
@ -185,4 +490,4 @@ Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.

2
ts/00_commitinfo_data.ts
View File

@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smartdata',
version: '5.2.12',
version: '5.3.0',
description: 'An advanced library for NoSQL data organization and manipulation using TypeScript with support for MongoDB, data validation, collections, and custom data types.'
}