push.rocks/smartdata
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(ci & formatting): Minor fixes: update CI workflow image and npmci package references, adjust package.json and readme URLs, and apply consistent code formatting.

Browse Source
This commit is contained in:
Philipp Kunz
2025-04-06 18:18:39 +00:00
parent 4fac974fc9
commit 9426a21a2a
19 changed files with 437 additions and 354 deletions
Download Patch File Download Diff File Expand all files Collapse all files

152
readme.md
View File

@@ -13,7 +13,7 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
- **Real-time Data Sync**: Watchers for real-time data changes with RxJS integration
- **Connection Management**: Automatic connection handling with connection pooling
- **Collection Management**: Type-safe collection operations with automatic indexing
- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>`
- **Deep Query Type Safety**: Fully type-safe queries for nested object properties with `DeepQuery<T>`
- **MongoDB Compatibility**: Support for all MongoDB query operators and advanced features
- **Enhanced Cursors**: Chainable, type-safe cursor API with memory efficient data processing
- **Type Conversion**: Automatic handling of MongoDB types like ObjectId and Binary data
@@ -27,6 +27,7 @@ A powerful TypeScript-first MongoDB wrapper that provides advanced features for
- TypeScript >= 4.x (for development)
## Install
To install `@push.rocks/smartdata`, use npm:
```bash
@@ -40,9 +41,11 @@ pnpm add @push.rocks/smartdata
```
## Usage
`@push.rocks/smartdata` enables efficient data handling and operation management with a focus on using MongoDB. It leverages TypeScript for strong typing and ESM syntax for modern JavaScript usage.
### Setting Up and Connecting to the Database
Before interacting with the database, you need to set up and establish a connection. The `SmartdataDb` class handles connection pooling and automatic reconnection.
```typescript
@@ -62,40 +65,50 @@ await db.init();
```
### Defining Data Models
Data models in `@push.rocks/smartdata` are classes that represent collections and documents in your MongoDB database. Use decorators such as `@Collection`, `@unI`, and `@svDb` to define your data models.
```typescript
import { SmartDataDbDoc, Collection, unI, svDb, oid, bin, index, searchable } from '@push.rocks/smartdata';
import {
SmartDataDbDoc,
Collection,
unI,
svDb,
oid,
bin,
index,
searchable,
} from '@push.rocks/smartdata';
import { ObjectId } from 'mongodb';
@Collection(() => db) // Associate this model with the database instance
@Collection(() => db) // Associate this model with the database instance
class User extends SmartDataDbDoc<User, User> {
@unI()
public id: string = 'unique-user-id'; // Mark 'id' as a unique index
@svDb()
@searchable() // Mark 'username' as searchable
public username: string; // Mark 'username' to be saved in DB
public username: string; // Mark 'username' to be saved in DB
@svDb()
@searchable() // Mark 'email' as searchable
@index() // Create a regular index for this field
public email: string; // Mark 'email' to be saved in DB
public email: string; // Mark 'email' to be saved in DB
@svDb()
@oid() // Automatically handle as ObjectId type
public organizationId: ObjectId; // Will be automatically converted to/from ObjectId
@svDb()
@bin() // Automatically handle as Binary data
public profilePicture: Buffer; // Will be automatically converted to/from Binary
@svDb({
@svDb({
serialize: (data) => JSON.stringify(data), // Custom serialization
deserialize: (data) => JSON.parse(data) // Custom deserialization
deserialize: (data) => JSON.parse(data), // Custom deserialization
})
public preferences: Record<string, any>;
constructor(username: string, email: string) {
super();
this.username = username;
@@ -105,15 +118,18 @@ class User extends SmartDataDbDoc<User, User> {
```
### CRUD Operations
`@push.rocks/smartdata` simplifies CRUD operations with intuitive methods on model instances.
#### Create
```typescript
const newUser = new User('myUsername', 'myEmail@example.com');
await newUser.save(); // Save the new user to the database
await newUser.save(); // Save the new user to the database
```
#### Read
```typescript
// Fetch a single user by a unique attribute
const user = await User.getInstance({ username: 'myUsername' });
@@ -132,8 +148,8 @@ await cursor.forEach(async (user, index) => {
// Chain cursor methods like in the MongoDB native driver
const paginatedCursor = await User.getCursor({ active: true })
.limit(10) // Limit results
.skip(20) // Skip first 20 results
.limit(10) // Limit results
.skip(20) // Skip first 20 results
.sort({ createdAt: -1 }); // Sort by creation date descending
// Convert cursor to array (when you know the result set is small)
@@ -149,30 +165,34 @@ await cursor.close();
```
#### Update
```typescript
// Assuming 'user' is an instance of User
user.email = 'newEmail@example.com';
await user.save(); // Update the user in the database
await user.save(); // Update the user in the database
// Upsert operations (insert if not exists, update if exists)
const upsertedUser = await User.upsert(
{ id: 'user-123' }, // Query to find the user
{ // Fields to update or insert
username: 'newUsername',
email: 'newEmail@example.com'
}
{ id: 'user-123' }, // Query to find the user
{
// Fields to update or insert
username: 'newUsername',
email: 'newEmail@example.com',
},
);
```
#### Delete
```typescript
// Assuming 'user' is an instance of User
await user.delete(); // Delete the user from the database
await user.delete(); // Delete the user from the database
```
## Advanced Features
### Search Functionality
SmartData provides powerful search capabilities with a Lucene-like query syntax and robust fallback mechanisms:
```typescript
@@ -181,19 +201,19 @@ SmartData provides powerful search capabilities with a Lucene-like query syntax
class Product extends SmartDataDbDoc<Product, Product> {
@unI()
public id: string = 'product-id';
@svDb()
@searchable() // Mark this field as searchable
public name: string;
@svDb()
@searchable() // Mark this field as searchable
public description: string;
@svDb()
@searchable() // Mark this field as searchable
public category: string;
@svDb()
public price: number;
}
@@ -232,6 +252,7 @@ const complexQuery = await Product.searchWithLucene('(wireless OR bluetooth) AND
```
The search functionality includes:
- `@searchable()` decorator for marking fields as searchable
- `getSearchableFields()` to retrieve all searchable fields for a class
- `search()` method for basic search (requires MongoDB text index)
@@ -240,6 +261,7 @@ The search functionality includes:
- Automatic fallback to regex-based search if MongoDB text search fails
### EasyStore
EasyStore provides a simple key-value storage system with automatic persistence:
```typescript
@@ -270,6 +292,7 @@ await store.deleteKey('apiKey');
```
### Distributed Coordination
Built-in support for distributed systems with leader election:
```typescript
@@ -307,22 +330,26 @@ await coordinator.stop();
```
### Real-time Data Watching
Watch for changes in your collections with RxJS integration using MongoDB Change Streams:
```typescript
// Create a watcher for a specific collection with a query filter
const watcher = await User.watch({
active: true // Only watch for changes to active users
}, {
fullDocument: true, // Include the full document in change notifications
bufferTimeMs: 100 // Buffer changes for 100ms to reduce notification frequency
});
const watcher = await User.watch(
{
active: true, // Only watch for changes to active users
},
{
fullDocument: true, // Include the full document in change notifications
bufferTimeMs: 100, // Buffer changes for 100ms to reduce notification frequency
},
);
// Subscribe to changes using RxJS
watcher.changeSubject.subscribe((change) => {
console.log('Change operation:', change.operationType); // 'insert', 'update', 'delete', etc.
console.log('Document changed:', change.docInstance); // The full document instance
console.log('Document changed:', change.docInstance); // The full document instance
// Handle different types of changes
if (change.operationType === 'insert') {
console.log('New user created:', change.docInstance.username);
@@ -343,6 +370,7 @@ await watcher.stop();
```
### Managed Collections
For more complex data models that require additional context:
```typescript
@@ -350,13 +378,13 @@ For more complex data models that require additional context:
class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
@unI()
public id: string = 'unique-id';
@svDb()
public data: string;
@managed()
public manager: YourCustomManager;
// The manager can provide additional functionality
async specialOperation() {
return this.manager.doSomethingSpecial(this);
@@ -365,6 +393,7 @@ class ManagedDoc extends SmartDataDbDoc<ManagedDoc, ManagedDoc> {
```
### Automatic Indexing
Define indexes directly in your model class:
```typescript
@@ -372,19 +401,19 @@ Define indexes directly in your model class:
class Product extends SmartDataDbDoc<Product, Product> {
@unI() // Unique index
public id: string = 'product-id';
@svDb()
@index() // Regular index for faster queries
public category: string;
@svDb()
@index({ sparse: true }) // Sparse index with options
public optionalField?: string;
// Compound indexes can be defined in the collection decorator
@Collection(() => db, {
indexMap: {
compoundIndex: {
compoundIndex: {
fields: { category: 1, name: 1 },
options: { background: true }
}
@@ -394,6 +423,7 @@ class Product extends SmartDataDbDoc<Product, Product> {
```
### Transaction Support
Use MongoDB transactions for atomic operations:
```typescript
@@ -403,7 +433,7 @@ try {
const user = await User.getInstance({ id: 'user-id' }, { session });
user.balance -= 100;
await user.save({ session });
const recipient = await User.getInstance({ id: 'recipient-id' }, { session });
recipient.balance += 100;
await user.save({ session });
@@ -414,6 +444,7 @@ try {
```
### Deep Object Queries
SmartData provides fully type-safe deep property queries with the `DeepQuery` type:
```typescript
@@ -421,7 +452,7 @@ SmartData provides fully type-safe deep property queries with the `DeepQuery` ty
class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
@unI()
public id: string = 'profile-id';
@svDb()
public user: {
details: {
@@ -430,14 +461,14 @@ class UserProfile extends SmartDataDbDoc<UserProfile, UserProfile> {
address: {
city: string;
country: string;
}
}
};
};
};
}
// Type-safe string literals for dot notation
const usersInUSA = await UserProfile.getInstances({
'user.details.address.country': 'USA'
'user.details.address.country': 'USA',
});
// Fully typed deep queries with the DeepQuery type
@@ -446,7 +477,7 @@ import { DeepQuery } from '@push.rocks/smartdata';
const typedQuery: DeepQuery<UserProfile> = {
id: 'profile-id',
'user.details.firstName': 'John',
'user.details.address.country': 'USA'
'user.details.address.country': 'USA',
};
// TypeScript will error if paths are incorrect
@@ -455,13 +486,14 @@ const results = await UserProfile.getInstances(typedQuery);
// MongoDB query operators are supported
const operatorQuery: DeepQuery<UserProfile> = {
'user.details.address.country': 'USA',
'user.details.address.city': { $in: ['New York', 'Los Angeles'] }
'user.details.address.city': { $in: ['New York', 'Los Angeles'] },
};
const filteredResults = await UserProfile.getInstances(operatorQuery);
```
### Document Lifecycle Hooks
Implement custom logic at different stages of a document's lifecycle:
```typescript
@@ -469,30 +501,30 @@ Implement custom logic at different stages of a document's lifecycle:
class Order extends SmartDataDbDoc<Order, Order> {
@unI()
public id: string = 'order-id';
@svDb()
public total: number;
@svDb()
public items: string[];
// Called before saving the document
async beforeSave() {
// Calculate total based on items
this.total = await calculateTotal(this.items);
// Validate the document
if (this.items.length === 0) {
throw new Error('Order must have at least one item');
}
}
// Called after the document is saved
async afterSave() {
// Notify other systems about the saved order
await notifyExternalSystems(this);
}
// Called before deleting the document
async beforeDelete() {
// Check if order can be deleted
@@ -507,33 +539,39 @@ class Order extends SmartDataDbDoc<Order, Order> {
## Best Practices
### Connection Management
- Always call `db.init()` before using any database features
- Use `db.disconnect()` when shutting down your application
- Set appropriate connection pool sizes based on your application's needs
### Document Design
- Use appropriate decorators (`@svDb`, `@unI`, `@index`, `@searchable`) to optimize database operations
- Implement type-safe models by properly extending `SmartDataDbDoc`
- Consider using interfaces to define document structures separately from implementation
- Mark fields that need to be searched with the `@searchable()` decorator
### Search Optimization
- Create MongoDB text indexes for collections that need advanced search operations
- Use `searchWithLucene()` for robust searches with fallback mechanisms
- Prefer field-specific searches when possible for better performance
- Use simple term queries instead of boolean operators if you don't have text indexes
### Performance Optimization
- Use cursors for large datasets instead of loading all documents into memory
- Create appropriate indexes for frequent query patterns
- Use projections to limit the fields returned when you don't need the entire document
### Distributed Systems
- Implement proper error handling for leader election events
- Ensure all instances have synchronized clocks when using time-based coordination
- Use the distributed coordinator's task management features for coordinated operations
### Type Safety
- Take advantage of the `DeepQuery<T>` type for fully type-safe queries
- Define proper types for your document models to enhance IDE auto-completion
- Use generic type parameters to specify exact document types when working with collections
@@ -552,7 +590,7 @@ Please make sure to update tests as appropriate and follow our coding standards.
## License and Legal Information
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
This repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.
@@ -567,4 +605,4 @@ Registered at District court Bremen HRB 35230 HB, Germany
For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.
By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.