Compare commits
8 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| c65c285296 | |||
| b72174ca7b | |||
| 1ff4d219af | |||
| 09f60de56f | |||
| e6a36ecb5f | |||
| 6a37a773ea | |||
| 1fff277698 | |||
| 0ad7f316c4 |
36
changelog.md
36
changelog.md
@@ -1,5 +1,41 @@
|
||||
# Changelog
|
||||
|
||||
## 2026-02-03 - 5.1.0 - feat(localtsmdb)
|
||||
export ILocalTsmDbConnectionInfo and expand LocalTsmDb/TsmDB documentation and examples
|
||||
|
||||
- Exported new type ILocalTsmDbConnectionInfo from ts_local (ts/index.ts)
|
||||
- Added LocalTsmDb configuration example, methods table, and ConnectionInfo interface to README
|
||||
- Documented Unix socket vs TCP connection modes and updated usage examples (TCP and socket examples)
|
||||
- Expanded TsmDB docs: additional server properties, aggregation stages, regex examples, index operations, database ops, checksums, and wire protocol commands
|
||||
- Updated architecture notes to include Unix socket support and new engine components (QueryEngine, UpdateEngine, AggregationEngine)
|
||||
|
||||
## 2026-02-03 - 5.0.0 - BREAKING CHANGE(localtsmdb)
|
||||
add Unix socket support and change LocalTsmDb API to return connection info instead of a MongoClient
|
||||
|
||||
- LocalTsmDb.start() now returns ILocalTsmDbConnectionInfo { socketPath, connectionUri } instead of a connected MongoClient
|
||||
- Removed internal MongoClient management: consumers must create/connect/close their own MongoClient using the returned connectionUri (close client before calling db.stop())
|
||||
- Added ILocalTsmDbConnectionInfo type and getConnectionInfo() (replaces getClient())
|
||||
- TsmdbServer: added socketPath option to listen on Unix sockets, cleans up stale socket files on start/stop, and encodes socket paths in getConnectionUri()
|
||||
- LocalTsmDb can auto-generate socket paths in the OS temp dir; LocalTsmDb no longer depends on the mongodb package internally (lightweight Unix socket wrapper)
|
||||
- Updated docs and tests to use MongoClient externally and to demonstrate socketPath/connectionUri workflow
|
||||
- ts_local plugins no longer export net (net usage moved to server implementation)
|
||||
|
||||
## 2026-02-03 - 4.3.0 - feat(docs)
|
||||
add LocalTsmDb documentation and examples; update README code samples and imports; correct examples and variable names; update package author
|
||||
|
||||
- Introduce LocalTsmDb: zero-config local database with automatic persistence, auto port discovery, and pre-connected client (added Quick Start, API, Features, and testing examples).
|
||||
- Expand comparison table to include LocalTsmDb alongside SmartMongo and TsmDB.
|
||||
- Update README examples: new LocalTsmDb usage, reorder options (LocalTsmDb, TsmDB, SmartMongo), rename test DB variable (db -> testDb), and adjust test snippets for Jest/Mocha and tap.
|
||||
- Adjust code snippets and API notes: switch some example imports to use tsmdb, replace FileStorageAdapter references, change planner.createPlan to await planner.plan, and use wal.getEntriesAfter(...) without awaiting.
|
||||
- Update package.json author from 'Lossless GmbH' to 'Task Venture Capital GmbH'.
|
||||
|
||||
## 2026-02-03 - 4.2.1 - fix(package.json)
|
||||
replace main and typings with exports field pointing to ./dist_ts/index.js
|
||||
|
||||
- Added package.json exports field mapping "." to ./dist_ts/index.js to declare the package entrypoint.
|
||||
- Removed main (dist_ts/index.js) and typings (dist_ts/index.d.ts) entries.
|
||||
- Note: switching to exports improves Node resolution but removing the typings entry may affect TypeScript consumers expecting index.d.ts.
|
||||
|
||||
## 2026-02-01 - 4.2.0 - feat(tsmdb)
|
||||
implement TsmDB Mongo-wire-compatible server, add storage/engine modules and reorganize exports
|
||||
|
||||
|
||||
@@ -1,12 +1,13 @@
|
||||
{
|
||||
"name": "@push.rocks/smartmongo",
|
||||
"version": "4.2.0",
|
||||
"version": "5.1.0",
|
||||
"private": false,
|
||||
"description": "A module for creating and managing a local MongoDB instance for testing purposes.",
|
||||
"main": "dist_ts/index.js",
|
||||
"typings": "dist_ts/index.d.ts",
|
||||
"exports": {
|
||||
".": "./dist_ts/index.js"
|
||||
},
|
||||
"type": "module",
|
||||
"author": "Lossless GmbH",
|
||||
"author": "Task Venture Capital GmbH",
|
||||
"license": "MIT",
|
||||
"scripts": {
|
||||
"test": "(tstest test/. --verbose --logfile --timeout 60)",
|
||||
|
||||
367
readme.md
367
readme.md
@@ -1,6 +1,6 @@
|
||||
# @push.rocks/smartmongo
|
||||
|
||||
A powerful MongoDB toolkit for testing and development — featuring both a real MongoDB memory server (**SmartMongo**) and an ultra-fast, lightweight wire-protocol-compatible in-memory database server (**TsmDB**). 🚀
|
||||
A powerful MongoDB toolkit for testing and development — featuring a real MongoDB memory server (**SmartMongo**), an ultra-fast wire-protocol-compatible in-memory database server (**TsmDB**), and a zero-config local database (**LocalTsmDb**). 🚀
|
||||
|
||||
## Install
|
||||
|
||||
@@ -16,21 +16,88 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
|
||||
|
||||
## Overview
|
||||
|
||||
`@push.rocks/smartmongo` provides two powerful approaches for MongoDB in testing and development:
|
||||
`@push.rocks/smartmongo` provides three powerful approaches for MongoDB in testing and development:
|
||||
|
||||
| Feature | SmartMongo | TsmDB |
|
||||
|---------|------------|---------|
|
||||
| **Type** | Real MongoDB (memory server) | Pure TypeScript wire protocol server |
|
||||
| **Speed** | ~2-5s startup | ⚡ Instant startup (~5ms) |
|
||||
| **Compatibility** | 100% MongoDB | MongoDB driver compatible |
|
||||
| **Dependencies** | Downloads MongoDB binary | Zero external dependencies |
|
||||
| **Replication** | ✅ Full replica set support | Single node emulation |
|
||||
| **Use Case** | Integration testing | Unit testing, CI/CD |
|
||||
| **Persistence** | Dump to directory | Optional file/memory persistence |
|
||||
| Feature | SmartMongo | TsmDB | LocalTsmDb |
|
||||
|---------|------------|-------|------------|
|
||||
| **Type** | Real MongoDB (memory server) | Wire protocol server | Zero-config local DB |
|
||||
| **Speed** | ~2-5s startup | ⚡ Instant (~5ms) | ⚡ Instant (Unix socket) |
|
||||
| **Compatibility** | 100% MongoDB | MongoDB driver compatible | MongoDB driver compatible |
|
||||
| **Dependencies** | Downloads MongoDB binary | Zero external deps | Zero external deps (no MongoDB driver!) |
|
||||
| **Connection** | TCP | TCP or Unix socket | Unix socket (default) |
|
||||
| **Replication** | ✅ Full replica set | Single node | Single node |
|
||||
| **Persistence** | Dump to directory | Memory or file | File-based (automatic) |
|
||||
| **Use Case** | Integration testing | Unit testing, CI/CD | Quick prototyping, local dev |
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
### Option 1: SmartMongo (Real MongoDB)
|
||||
### Option 1: LocalTsmDb (Zero-Config Local Database) ⭐
|
||||
|
||||
The easiest way to get started — just point it at a folder and you have a persistent MongoDB-compatible database using Unix sockets. No port conflicts, no MongoDB driver dependency in LocalTsmDb!
|
||||
|
||||
```typescript
|
||||
import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient } from 'mongodb';
|
||||
|
||||
// Create a local database backed by files
|
||||
const db = new LocalTsmDb({ folderPath: './my-data' });
|
||||
|
||||
// Start and get connection info (Unix socket path + connection URI)
|
||||
const { connectionUri } = await db.start();
|
||||
|
||||
// Connect with your own MongoDB client
|
||||
const client = new MongoClient(connectionUri, { directConnection: true });
|
||||
await client.connect();
|
||||
|
||||
// Use exactly like MongoDB
|
||||
const users = client.db('myapp').collection('users');
|
||||
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
|
||||
|
||||
const user = await users.findOne({ name: 'Alice' });
|
||||
console.log(user); // { _id: ObjectId(...), name: 'Alice', email: 'alice@example.com' }
|
||||
|
||||
// Data persists to disk automatically!
|
||||
await client.close();
|
||||
await db.stop();
|
||||
|
||||
// Later... data is still there
|
||||
const db2 = new LocalTsmDb({ folderPath: './my-data' });
|
||||
const { connectionUri: uri2 } = await db2.start();
|
||||
const client2 = new MongoClient(uri2, { directConnection: true });
|
||||
await client2.connect();
|
||||
const savedUser = await client2.db('myapp').collection('users').findOne({ name: 'Alice' });
|
||||
// savedUser exists!
|
||||
```
|
||||
|
||||
### Option 2: TsmDB (Wire Protocol Server)
|
||||
|
||||
A lightweight, pure TypeScript MongoDB-compatible server — use the official `mongodb` driver directly!
|
||||
|
||||
```typescript
|
||||
import { tsmdb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient } from 'mongodb';
|
||||
|
||||
// Start TsmDB server (TCP mode)
|
||||
const server = new tsmdb.TsmdbServer({ port: 27017 });
|
||||
await server.start();
|
||||
|
||||
// Connect with the official MongoDB driver
|
||||
const client = new MongoClient('mongodb://127.0.0.1:27017');
|
||||
await client.connect();
|
||||
|
||||
// Use exactly like real MongoDB
|
||||
const db = client.db('myapp');
|
||||
await db.collection('users').insertOne({ name: 'Alice', age: 30 });
|
||||
|
||||
const user = await db.collection('users').findOne({ name: 'Alice' });
|
||||
console.log(user); // { _id: ObjectId(...), name: 'Alice', age: 30 }
|
||||
|
||||
// Clean up
|
||||
await client.close();
|
||||
await server.stop();
|
||||
```
|
||||
|
||||
### Option 3: SmartMongo (Real MongoDB)
|
||||
|
||||
Spin up a real MongoDB replica set in memory — perfect for integration tests that need full MongoDB compatibility.
|
||||
|
||||
@@ -51,36 +118,91 @@ console.log(descriptor.mongoDbUrl); // mongodb://127.0.0.1:xxxxx/...
|
||||
await mongo.stop();
|
||||
```
|
||||
|
||||
### Option 2: TsmDB (Wire Protocol Server)
|
||||
---
|
||||
|
||||
A lightweight, pure TypeScript MongoDB-compatible server that speaks the wire protocol — use the official `mongodb` driver directly!
|
||||
## 📖 LocalTsmDb API
|
||||
|
||||
The simplest option for local development and prototyping — lightweight, Unix socket-based, and automatic persistence.
|
||||
|
||||
### Configuration
|
||||
|
||||
```typescript
|
||||
import { tsmdb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient } from 'mongodb';
|
||||
import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
import type { ILocalTsmDbOptions, ILocalTsmDbConnectionInfo } from '@push.rocks/smartmongo';
|
||||
|
||||
// Start TsmDB server
|
||||
const server = new tsmdb.TsmdbServer({ port: 27017 });
|
||||
await server.start();
|
||||
const options: ILocalTsmDbOptions = {
|
||||
folderPath: './data', // Required: where to store data
|
||||
socketPath: '/tmp/my.sock', // Optional: custom socket path (default: auto-generated)
|
||||
};
|
||||
|
||||
// Connect with the official MongoDB driver!
|
||||
const client = new MongoClient('mongodb://127.0.0.1:27017');
|
||||
await client.connect();
|
||||
|
||||
// Use exactly like real MongoDB
|
||||
const db = client.db('myapp');
|
||||
await db.collection('users').insertOne({ name: 'Alice', age: 30 });
|
||||
|
||||
const user = await db.collection('users').findOne({ name: 'Alice' });
|
||||
console.log(user); // { _id: ObjectId(...), name: 'Alice', age: 30 }
|
||||
|
||||
// Clean up
|
||||
await client.close();
|
||||
await server.stop();
|
||||
const db = new LocalTsmDb(options);
|
||||
```
|
||||
|
||||
### Methods
|
||||
|
||||
| Method | Returns | Description |
|
||||
|--------|---------|-------------|
|
||||
| `start()` | `Promise<ILocalTsmDbConnectionInfo>` | Starts the server and returns connection info |
|
||||
| `stop()` | `Promise<void>` | Stops the server and cleans up the socket |
|
||||
| `getConnectionInfo()` | `ILocalTsmDbConnectionInfo` | Returns current connection info |
|
||||
| `getConnectionUri()` | `string` | Returns the MongoDB connection URI |
|
||||
| `getServer()` | `TsmdbServer` | Returns the underlying TsmDB server instance |
|
||||
| `running` | `boolean` | Property indicating if the server is running |
|
||||
|
||||
### Connection Info
|
||||
|
||||
The `start()` method returns an `ILocalTsmDbConnectionInfo` object:
|
||||
|
||||
```typescript
|
||||
interface ILocalTsmDbConnectionInfo {
|
||||
socketPath: string; // The Unix socket file path, e.g., /tmp/smartmongo-abc123.sock
|
||||
connectionUri: string; // MongoDB URI, e.g., mongodb://%2Ftmp%2Fsmartmongo-abc123.sock
|
||||
}
|
||||
```
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```typescript
|
||||
import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient } from 'mongodb';
|
||||
|
||||
const db = new LocalTsmDb({ folderPath: './data' });
|
||||
|
||||
// Start and get connection info
|
||||
const { socketPath, connectionUri } = await db.start();
|
||||
console.log(socketPath); // /tmp/smartmongo-abc123.sock (auto-generated)
|
||||
console.log(connectionUri); // mongodb://%2Ftmp%2Fsmartmongo-abc123.sock
|
||||
|
||||
// Connect with your own MongoDB client
|
||||
const client = new MongoClient(connectionUri, { directConnection: true });
|
||||
await client.connect();
|
||||
|
||||
// Use the client
|
||||
const users = client.db('mydb').collection('users');
|
||||
await users.insertOne({ name: 'Alice' });
|
||||
|
||||
// Check status
|
||||
console.log(db.running); // true
|
||||
|
||||
// Stop when done (close your client first!)
|
||||
await client.close();
|
||||
await db.stop();
|
||||
```
|
||||
|
||||
### Features
|
||||
|
||||
- 🔌 **Unix Sockets** — No port conflicts, faster IPC than TCP
|
||||
- 💾 **Automatic Persistence** — Data saved to files, survives restarts
|
||||
- 🪶 **Lightweight** — No MongoDB driver dependency in LocalTsmDb itself
|
||||
- 🎯 **Zero Config** — Just specify a folder path and you're good to go
|
||||
- 🔗 **Connection URI** — Ready-to-use URI for your own MongoClient
|
||||
|
||||
---
|
||||
|
||||
## 📖 SmartMongo API
|
||||
|
||||
Full MongoDB replica set in memory using `mongodb-memory-server`.
|
||||
|
||||
### Creating an Instance
|
||||
|
||||
```typescript
|
||||
@@ -116,27 +238,44 @@ await mongo.stopAndDumpToDir('./test-data');
|
||||
await mongo.stopAndDumpToDir('./test-data', (doc) => `${doc.collection}-${doc._id}.bson`);
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔧 TsmDB API
|
||||
|
||||
Pure TypeScript MongoDB wire protocol server. No external dependencies.
|
||||
|
||||
### Server Configuration
|
||||
|
||||
```typescript
|
||||
import { tsmdb } from '@push.rocks/smartmongo';
|
||||
|
||||
// TCP mode (default)
|
||||
const server = new tsmdb.TsmdbServer({
|
||||
port: 27017, // Default MongoDB port
|
||||
host: '127.0.0.1', // Bind address
|
||||
storage: 'memory', // 'memory' or 'file'
|
||||
storagePath: './data', // For file-based storage
|
||||
port: 27017, // Default MongoDB port
|
||||
host: '127.0.0.1', // Bind address
|
||||
storage: 'memory', // 'memory' or 'file'
|
||||
storagePath: './data', // For file-based storage
|
||||
});
|
||||
|
||||
// Unix socket mode (no port conflicts!)
|
||||
const server = new tsmdb.TsmdbServer({
|
||||
socketPath: '/tmp/my-tsmdb.sock',
|
||||
storage: 'file',
|
||||
storagePath: './data',
|
||||
});
|
||||
|
||||
await server.start();
|
||||
console.log(server.getConnectionUri()); // mongodb://127.0.0.1:27017
|
||||
console.log(server.getConnectionUri());
|
||||
// TCP: mongodb://127.0.0.1:27017
|
||||
// Socket: mongodb://%2Ftmp%2Fmy-tsmdb.sock
|
||||
|
||||
// Server properties
|
||||
console.log(server.running); // true
|
||||
console.log(server.getUptime()); // seconds
|
||||
console.log(server.getConnectionCount()); // active connections
|
||||
console.log(server.port); // 27017 (TCP mode)
|
||||
console.log(server.host); // '127.0.0.1' (TCP mode)
|
||||
console.log(server.socketPath); // '/tmp/my-tsmdb.sock' (socket mode)
|
||||
console.log(server.getUptime()); // seconds since start
|
||||
console.log(server.getConnectionCount()); // active client connections
|
||||
|
||||
await server.stop();
|
||||
```
|
||||
@@ -146,6 +285,7 @@ await server.stop();
|
||||
TsmDB supports the core MongoDB operations via the wire protocol:
|
||||
|
||||
#### 🔹 CRUD Operations
|
||||
|
||||
```typescript
|
||||
// Insert
|
||||
await collection.insertOne({ name: 'Bob' });
|
||||
@@ -175,6 +315,7 @@ const result = await collection.findOneAndUpdate(
|
||||
```
|
||||
|
||||
#### 🔹 Query Operators
|
||||
|
||||
```typescript
|
||||
// Comparison
|
||||
{ age: { $eq: 25 } }
|
||||
@@ -197,9 +338,14 @@ const result = await collection.findOneAndUpdate(
|
||||
{ tags: { $all: ['mongodb', 'database'] } }
|
||||
{ scores: { $elemMatch: { $gte: 80, $lt: 90 } } }
|
||||
{ tags: { $size: 3 } }
|
||||
|
||||
// Regex
|
||||
{ name: { $regex: /^Al/i } }
|
||||
{ email: { $regex: '@example\\.com$' } }
|
||||
```
|
||||
|
||||
#### 🔹 Update Operators
|
||||
|
||||
```typescript
|
||||
{ $set: { name: 'New Name' } }
|
||||
{ $unset: { tempField: '' } }
|
||||
@@ -212,9 +358,12 @@ const result = await collection.findOneAndUpdate(
|
||||
{ $addToSet: { tags: 'unique-tag' } }
|
||||
{ $pop: { queue: 1 } } // Remove last
|
||||
{ $pop: { queue: -1 } } // Remove first
|
||||
{ $rename: { oldField: 'newField' } }
|
||||
{ $currentDate: { lastModified: true } }
|
||||
```
|
||||
|
||||
#### 🔹 Aggregation Pipeline
|
||||
|
||||
```typescript
|
||||
const results = await collection.aggregate([
|
||||
{ $match: { status: 'active' } },
|
||||
@@ -225,17 +374,28 @@ const results = await collection.aggregate([
|
||||
]).toArray();
|
||||
```
|
||||
|
||||
Supported stages: `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, and more.
|
||||
**Supported stages:** `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, `$replaceRoot`, `$set`, `$unset`, and more.
|
||||
|
||||
**Supported group accumulators:** `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`, `$push`, `$addToSet`, `$count`.
|
||||
|
||||
#### 🔹 Index Operations
|
||||
|
||||
```typescript
|
||||
// Create indexes
|
||||
await collection.createIndex({ email: 1 }, { unique: true });
|
||||
await collection.createIndex({ name: 1, age: -1 });
|
||||
await collection.createIndex({ location: '2dsphere' }); // Geospatial
|
||||
|
||||
// List indexes
|
||||
const indexes = await collection.listIndexes().toArray();
|
||||
|
||||
// Drop indexes
|
||||
await collection.dropIndex('email_1');
|
||||
await collection.dropIndexes(); // Drop all except _id
|
||||
```
|
||||
|
||||
#### 🔹 Database Operations
|
||||
|
||||
```typescript
|
||||
// List databases
|
||||
const dbs = await client.db().admin().listDatabases();
|
||||
@@ -249,9 +409,13 @@ await db.dropCollection('oldcollection');
|
||||
|
||||
// Drop database
|
||||
await db.dropDatabase();
|
||||
|
||||
// Database stats
|
||||
const stats = await db.stats();
|
||||
```
|
||||
|
||||
#### 🔹 Count & Distinct
|
||||
|
||||
```typescript
|
||||
// Count documents
|
||||
const total = await collection.countDocuments({});
|
||||
@@ -264,6 +428,7 @@ const activeDepts = await collection.distinct('department', { status: 'active' }
|
||||
```
|
||||
|
||||
#### 🔹 Bulk Operations
|
||||
|
||||
```typescript
|
||||
const result = await collection.bulkWrite([
|
||||
{ insertOne: { document: { name: 'Bulk1' } } },
|
||||
@@ -292,22 +457,22 @@ const server = new tsmdb.TsmdbServer({
|
||||
persistIntervalMs: 30000 // Save every 30 seconds
|
||||
});
|
||||
|
||||
// File-based - persistent storage with optional checksums
|
||||
import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';
|
||||
|
||||
const adapter = new FileStorageAdapter('./data/tsmdb', {
|
||||
enableChecksums: true, // CRC32 checksums for data integrity
|
||||
strictChecksums: false // Log warnings vs throw on mismatch
|
||||
// File-based - persistent storage with checksums
|
||||
const server = new tsmdb.TsmdbServer({
|
||||
storage: 'file',
|
||||
storagePath: './data/tsmdb'
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚡ Performance & Reliability Features
|
||||
|
||||
TsmDB includes enterprise-grade features for robustness:
|
||||
|
||||
### 🔍 Index-Accelerated Queries
|
||||
|
||||
Indexes are automatically used to accelerate queries. Instead of scanning all documents, TsmDB uses:
|
||||
Indexes are automatically used to accelerate queries:
|
||||
|
||||
- **Hash indexes** for equality queries (`$eq`, `$in`)
|
||||
- **B-tree indexes** for range queries (`$gt`, `$gte`, `$lt`, `$lte`)
|
||||
@@ -331,14 +496,14 @@ import { tsmdb } from '@push.rocks/smartmongo';
|
||||
|
||||
// For debugging, you can access the query planner
|
||||
const planner = new tsmdb.QueryPlanner(indexEngine);
|
||||
const plan = planner.createPlan(filter);
|
||||
const plan = await planner.plan(filter);
|
||||
|
||||
console.log(plan);
|
||||
// {
|
||||
// type: 'IXSCAN', // or 'IXSCAN_RANGE', 'COLLSCAN'
|
||||
// indexName: 'email_1',
|
||||
// estimatedCost: 1,
|
||||
// selectivity: 0.001
|
||||
// selectivity: 0.01,
|
||||
// indexCovering: true
|
||||
// }
|
||||
```
|
||||
|
||||
@@ -357,10 +522,10 @@ await wal.initialize();
|
||||
// - Timestamp
|
||||
// - Operation type (insert, update, delete, checkpoint)
|
||||
// - Document data (BSON serialized)
|
||||
// - CRC32 checksum
|
||||
// - CRC32 checksum for integrity
|
||||
|
||||
// Recovery support
|
||||
const entries = await wal.getEntriesAfter(lastCheckpointLsn);
|
||||
const entries = wal.getEntriesAfter(lastCheckpointLsn);
|
||||
```
|
||||
|
||||
### 🔐 Session Management
|
||||
@@ -393,33 +558,67 @@ try {
|
||||
File-based storage supports CRC32 checksums to detect corruption:
|
||||
|
||||
```typescript
|
||||
import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';
|
||||
|
||||
const adapter = new FileStorageAdapter('./data', {
|
||||
enableChecksums: true,
|
||||
strictChecksums: true // Throw error on corruption (vs warning)
|
||||
});
|
||||
import { tsmdb } from '@push.rocks/smartmongo';
|
||||
|
||||
// Checksums are used internally for WAL and data integrity
|
||||
// Documents are checksummed on write, verified on read
|
||||
// Checksums are automatically stripped before returning to client
|
||||
const checksum = tsmdb.calculateDocumentChecksum(doc);
|
||||
const isValid = tsmdb.verifyChecksum(docWithChecksum);
|
||||
```
|
||||
|
||||
### 📋 Supported Wire Protocol Commands
|
||||
|
||||
| Category | Commands |
|
||||
|----------|----------|
|
||||
| **Handshake** | `hello`, `isMaster` |
|
||||
| **Handshake** | `hello`, `isMaster`, `ismaster` |
|
||||
| **CRUD** | `find`, `insert`, `update`, `delete`, `findAndModify`, `getMore`, `killCursors` |
|
||||
| **Aggregation** | `aggregate`, `count`, `distinct` |
|
||||
| **Indexes** | `createIndexes`, `dropIndexes`, `listIndexes` |
|
||||
| **Transactions** | `startTransaction`, `commitTransaction`, `abortTransaction` |
|
||||
| **Sessions** | `startSession`, `endSessions` |
|
||||
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats` |
|
||||
| **Sessions** | `startSession`, `endSessions`, `refreshSessions` |
|
||||
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats`, `connectionStatus` |
|
||||
|
||||
TsmDB supports MongoDB wire protocol versions 0-21, compatible with MongoDB 3.6 through 7.0 drivers.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Testing Examples
|
||||
|
||||
### Jest/Mocha with LocalTsmDb
|
||||
|
||||
```typescript
|
||||
import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient, Db } from 'mongodb';
|
||||
|
||||
let db: LocalTsmDb;
|
||||
let client: MongoClient;
|
||||
|
||||
beforeAll(async () => {
|
||||
db = new LocalTsmDb({ folderPath: './test-data' });
|
||||
const { connectionUri } = await db.start();
|
||||
client = new MongoClient(connectionUri, { directConnection: true });
|
||||
await client.connect();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await client.close();
|
||||
await db.stop();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean slate for each test
|
||||
await client.db('test').dropDatabase();
|
||||
});
|
||||
|
||||
test('should insert and find user', async () => {
|
||||
const users = client.db('test').collection('users');
|
||||
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
|
||||
|
||||
const user = await users.findOne({ name: 'Alice' });
|
||||
expect(user?.email).toBe('alice@example.com');
|
||||
});
|
||||
```
|
||||
|
||||
### Jest/Mocha with TsmDB
|
||||
|
||||
```typescript
|
||||
@@ -428,7 +627,7 @@ import { MongoClient, Db } from 'mongodb';
|
||||
|
||||
let server: tsmdb.TsmdbServer;
|
||||
let client: MongoClient;
|
||||
let db: Db;
|
||||
let testDb: Db;
|
||||
|
||||
beforeAll(async () => {
|
||||
server = new tsmdb.TsmdbServer({ port: 27117 });
|
||||
@@ -436,7 +635,7 @@ beforeAll(async () => {
|
||||
|
||||
client = new MongoClient('mongodb://127.0.0.1:27117');
|
||||
await client.connect();
|
||||
db = client.db('test');
|
||||
testDb = client.db('test');
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
@@ -445,12 +644,11 @@ afterAll(async () => {
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
// Clean slate for each test
|
||||
await db.dropDatabase();
|
||||
await testDb.dropDatabase();
|
||||
});
|
||||
|
||||
test('should insert and find user', async () => {
|
||||
const users = db.collection('users');
|
||||
const users = testDb.collection('users');
|
||||
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
|
||||
|
||||
const user = await users.findOne({ name: 'Alice' });
|
||||
@@ -462,22 +660,21 @@ test('should insert and find user', async () => {
|
||||
|
||||
```typescript
|
||||
import { expect, tap } from '@git.zone/tstest/tapbundle';
|
||||
import { tsmdb } from '@push.rocks/smartmongo';
|
||||
import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
import { MongoClient } from 'mongodb';
|
||||
|
||||
let server: tsmdb.TsmdbServer;
|
||||
let db: LocalTsmDb;
|
||||
let client: MongoClient;
|
||||
|
||||
tap.test('setup', async () => {
|
||||
server = new tsmdb.TsmdbServer({ port: 27117 });
|
||||
await server.start();
|
||||
client = new MongoClient('mongodb://127.0.0.1:27117');
|
||||
db = new LocalTsmDb({ folderPath: './test-data' });
|
||||
const { connectionUri } = await db.start();
|
||||
client = new MongoClient(connectionUri, { directConnection: true });
|
||||
await client.connect();
|
||||
});
|
||||
|
||||
tap.test('should perform CRUD operations', async () => {
|
||||
const db = client.db('test');
|
||||
const col = db.collection('items');
|
||||
const col = client.db('test').collection('items');
|
||||
|
||||
// Create
|
||||
const result = await col.insertOne({ name: 'Widget', price: 9.99 });
|
||||
@@ -500,14 +697,25 @@ tap.test('should perform CRUD operations', async () => {
|
||||
|
||||
tap.test('teardown', async () => {
|
||||
await client.close();
|
||||
await server.stop();
|
||||
await db.stop();
|
||||
});
|
||||
|
||||
export default tap.start();
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🏗️ Architecture
|
||||
|
||||
### Module Structure
|
||||
|
||||
```
|
||||
@push.rocks/smartmongo
|
||||
├── SmartMongo → Real MongoDB memory server (mongodb-memory-server wrapper)
|
||||
├── tsmdb → Wire protocol server with full engine stack
|
||||
└── LocalTsmDb → Lightweight Unix socket wrapper (no MongoDB driver dependency)
|
||||
```
|
||||
|
||||
### TsmDB Wire Protocol Stack
|
||||
|
||||
```
|
||||
@@ -515,7 +723,7 @@ export default tap.start();
|
||||
│ Official MongoDB Driver │
|
||||
│ (mongodb npm) │
|
||||
└─────────────────────────┬───────────────────────────────────┘
|
||||
│ TCP + OP_MSG/BSON
|
||||
│ TCP/Unix Socket + OP_MSG/BSON
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ TsmdbServer │
|
||||
@@ -554,11 +762,16 @@ export default tap.start();
|
||||
| **WireProtocol** | Parses MongoDB OP_MSG binary protocol |
|
||||
| **CommandRouter** | Routes commands to appropriate handlers |
|
||||
| **QueryPlanner** | Analyzes queries and selects execution strategy |
|
||||
| **QueryEngine** | Executes queries with filter matching |
|
||||
| **UpdateEngine** | Processes update operators (`$set`, `$inc`, etc.) |
|
||||
| **AggregationEngine** | Executes aggregation pipelines |
|
||||
| **IndexEngine** | Manages B-tree and hash indexes |
|
||||
| **SessionEngine** | Tracks client sessions and timeouts |
|
||||
| **TransactionEngine** | Handles ACID transaction semantics |
|
||||
| **WAL** | Write-ahead logging for durability |
|
||||
|
||||
---
|
||||
|
||||
## License and Legal Information
|
||||
|
||||
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.
|
||||
|
||||
@@ -3,6 +3,6 @@
|
||||
*/
|
||||
export const commitinfo = {
|
||||
name: '@push.rocks/smartmongo',
|
||||
version: '4.2.0',
|
||||
version: '5.1.0',
|
||||
description: 'A module for creating and managing a local MongoDB instance for testing purposes.'
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@ export * as tsmdb from './ts_tsmdb/index.js';
|
||||
|
||||
// Export LocalTsmDb from ts_local
|
||||
export { LocalTsmDb } from './ts_local/index.js';
|
||||
export type { ILocalTsmDbOptions } from './ts_local/index.js';
|
||||
export type { ILocalTsmDbOptions, ILocalTsmDbConnectionInfo } from './ts_local/index.js';
|
||||
|
||||
// Export commitinfo
|
||||
export { commitinfo };
|
||||
|
||||
@@ -1,98 +1,106 @@
|
||||
import * as plugins from './plugins.js';
|
||||
import * as crypto from 'crypto';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { TsmdbServer } from '../ts_tsmdb/index.js';
|
||||
import type { MongoClient } from 'mongodb';
|
||||
|
||||
/**
|
||||
* Connection information returned by LocalTsmDb.start()
|
||||
*/
|
||||
export interface ILocalTsmDbConnectionInfo {
|
||||
/** The Unix socket file path */
|
||||
socketPath: string;
|
||||
/** MongoDB connection URI ready for MongoClient */
|
||||
connectionUri: string;
|
||||
}
|
||||
|
||||
export interface ILocalTsmDbOptions {
|
||||
/** Required: where to store data */
|
||||
folderPath: string;
|
||||
port?: number;
|
||||
host?: string;
|
||||
/** Optional: custom socket path (default: auto-generated in /tmp) */
|
||||
socketPath?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* LocalTsmDb - Convenience class for local MongoDB-compatible database
|
||||
* LocalTsmDb - Lightweight local MongoDB-compatible database using Unix sockets
|
||||
*
|
||||
* This class wraps TsmdbServer and provides a simple interface for
|
||||
* starting a local file-based MongoDB-compatible server and connecting to it.
|
||||
* starting a local file-based MongoDB-compatible server. Returns connection
|
||||
* info that you can use with your own MongoDB driver instance.
|
||||
*
|
||||
* @example
|
||||
* ```typescript
|
||||
* import { LocalTsmDb } from '@push.rocks/smartmongo';
|
||||
* import { MongoClient } from 'mongodb';
|
||||
*
|
||||
* const db = new LocalTsmDb({ folderPath: './data' });
|
||||
* const client = await db.start();
|
||||
* const { connectionUri } = await db.start();
|
||||
*
|
||||
* // Connect with your own MongoDB client
|
||||
* const client = new MongoClient(connectionUri, { directConnection: true });
|
||||
* await client.connect();
|
||||
*
|
||||
* // Use the MongoDB client
|
||||
* const collection = client.db('mydb').collection('users');
|
||||
* await collection.insertOne({ name: 'Alice' });
|
||||
*
|
||||
* // When done
|
||||
* await client.close();
|
||||
* await db.stop();
|
||||
* ```
|
||||
*/
|
||||
export class LocalTsmDb {
|
||||
private options: ILocalTsmDbOptions;
|
||||
private server: TsmdbServer | null = null;
|
||||
private client: MongoClient | null = null;
|
||||
private generatedSocketPath: string | null = null;
|
||||
|
||||
constructor(options: ILocalTsmDbOptions) {
|
||||
this.options = options;
|
||||
}
|
||||
|
||||
/**
|
||||
* Find an available port starting from the given port
|
||||
* Generate a unique socket path in /tmp
|
||||
*/
|
||||
private async findAvailablePort(startPort = 27017): Promise<number> {
|
||||
return new Promise((resolve, reject) => {
|
||||
const server = plugins.net.createServer();
|
||||
server.listen(startPort, '127.0.0.1', () => {
|
||||
const addr = server.address();
|
||||
const port = typeof addr === 'object' && addr ? addr.port : startPort;
|
||||
server.close(() => resolve(port));
|
||||
});
|
||||
server.on('error', () => {
|
||||
this.findAvailablePort(startPort + 1).then(resolve).catch(reject);
|
||||
});
|
||||
});
|
||||
private generateSocketPath(): string {
|
||||
const randomId = crypto.randomBytes(8).toString('hex');
|
||||
return path.join(os.tmpdir(), `smartmongo-${randomId}.sock`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Start the local TsmDB server and return a connected MongoDB client
|
||||
* Start the local TsmDB server and return connection info
|
||||
*/
|
||||
async start(): Promise<MongoClient> {
|
||||
if (this.server && this.client) {
|
||||
async start(): Promise<ILocalTsmDbConnectionInfo> {
|
||||
if (this.server) {
|
||||
throw new Error('LocalTsmDb is already running');
|
||||
}
|
||||
|
||||
const port = this.options.port ?? await this.findAvailablePort();
|
||||
const host = this.options.host ?? '127.0.0.1';
|
||||
// Use provided socket path or generate one
|
||||
this.generatedSocketPath = this.options.socketPath ?? this.generateSocketPath();
|
||||
|
||||
this.server = new TsmdbServer({
|
||||
port,
|
||||
host,
|
||||
socketPath: this.generatedSocketPath,
|
||||
storage: 'file',
|
||||
storagePath: this.options.folderPath,
|
||||
});
|
||||
await this.server.start();
|
||||
|
||||
// Dynamically import mongodb to avoid requiring it as a hard dependency
|
||||
const mongodb = await import('mongodb');
|
||||
this.client = new mongodb.MongoClient(this.server.getConnectionUri(), {
|
||||
directConnection: true,
|
||||
serverSelectionTimeoutMS: 5000,
|
||||
});
|
||||
await this.client.connect();
|
||||
|
||||
return this.client;
|
||||
return {
|
||||
socketPath: this.generatedSocketPath,
|
||||
connectionUri: this.server.getConnectionUri(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the MongoDB client (throws if not started)
|
||||
* Get connection info (throws if not started)
|
||||
*/
|
||||
getClient(): MongoClient {
|
||||
if (!this.client) {
|
||||
getConnectionInfo(): ILocalTsmDbConnectionInfo {
|
||||
if (!this.server || !this.generatedSocketPath) {
|
||||
throw new Error('LocalTsmDb is not running. Call start() first.');
|
||||
}
|
||||
return this.client;
|
||||
return {
|
||||
socketPath: this.generatedSocketPath,
|
||||
connectionUri: this.server.getConnectionUri(),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -123,16 +131,13 @@ export class LocalTsmDb {
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop the local TsmDB server and close the client connection
|
||||
* Stop the local TsmDB server
|
||||
*/
|
||||
async stop(): Promise<void> {
|
||||
if (this.client) {
|
||||
await this.client.close();
|
||||
this.client = null;
|
||||
}
|
||||
if (this.server) {
|
||||
await this.server.stop();
|
||||
this.server = null;
|
||||
this.generatedSocketPath = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
export { LocalTsmDb } from './classes.localtsmdb.js';
|
||||
export type { ILocalTsmDbOptions } from './classes.localtsmdb.js';
|
||||
export type { ILocalTsmDbOptions, ILocalTsmDbConnectionInfo } from './classes.localtsmdb.js';
|
||||
|
||||
@@ -1,4 +1,3 @@
|
||||
import * as smartpromise from '@push.rocks/smartpromise';
|
||||
import * as net from 'net';
|
||||
|
||||
export { smartpromise, net };
|
||||
export { smartpromise };
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
import * as net from 'net';
|
||||
import * as fs from 'fs/promises';
|
||||
import * as plugins from '../plugins.js';
|
||||
import { WireProtocol, OP_QUERY } from './WireProtocol.js';
|
||||
import { CommandRouter } from './CommandRouter.js';
|
||||
@@ -10,10 +11,12 @@ import type { IStorageAdapter } from '../storage/IStorageAdapter.js';
|
||||
* Server configuration options
|
||||
*/
|
||||
export interface ITsmdbServerOptions {
|
||||
/** Port to listen on (default: 27017) */
|
||||
/** Port to listen on (default: 27017) - ignored if socketPath is set */
|
||||
port?: number;
|
||||
/** Host to bind to (default: 127.0.0.1) */
|
||||
/** Host to bind to (default: 127.0.0.1) - ignored if socketPath is set */
|
||||
host?: string;
|
||||
/** Unix socket path - if set, server listens on socket instead of TCP */
|
||||
socketPath?: string;
|
||||
/** Storage type: 'memory' or 'file' (default: 'memory') */
|
||||
storage?: 'memory' | 'file';
|
||||
/** Path for file storage (required if storage is 'file') */
|
||||
@@ -54,7 +57,7 @@ interface IConnectionState {
|
||||
* ```
|
||||
*/
|
||||
export class TsmdbServer {
|
||||
private options: Required<ITsmdbServerOptions>;
|
||||
private options: Required<Omit<ITsmdbServerOptions, 'socketPath'>> & { socketPath: string };
|
||||
private server: net.Server | null = null;
|
||||
private storage: IStorageAdapter;
|
||||
private commandRouter: CommandRouter;
|
||||
@@ -62,11 +65,14 @@ export class TsmdbServer {
|
||||
private connectionIdCounter = 0;
|
||||
private isRunning = false;
|
||||
private startTime: Date = new Date();
|
||||
private useSocket: boolean;
|
||||
|
||||
constructor(options: ITsmdbServerOptions = {}) {
|
||||
this.useSocket = !!options.socketPath;
|
||||
this.options = {
|
||||
port: options.port ?? 27017,
|
||||
host: options.host ?? '127.0.0.1',
|
||||
socketPath: options.socketPath ?? '',
|
||||
storage: options.storage ?? 'memory',
|
||||
storagePath: options.storagePath ?? './data',
|
||||
persistPath: options.persistPath ?? '',
|
||||
@@ -119,6 +125,18 @@ export class TsmdbServer {
|
||||
// Initialize storage
|
||||
await this.storage.initialize();
|
||||
|
||||
// Clean up stale socket file if using Unix socket
|
||||
if (this.useSocket && this.options.socketPath) {
|
||||
try {
|
||||
await fs.unlink(this.options.socketPath);
|
||||
} catch (err: any) {
|
||||
// Ignore ENOENT (file doesn't exist)
|
||||
if (err.code !== 'ENOENT') {
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
this.server = net.createServer((socket) => {
|
||||
this.handleConnection(socket);
|
||||
@@ -132,11 +150,21 @@ export class TsmdbServer {
|
||||
}
|
||||
});
|
||||
|
||||
this.server.listen(this.options.port, this.options.host, () => {
|
||||
this.isRunning = true;
|
||||
this.startTime = new Date();
|
||||
resolve();
|
||||
});
|
||||
if (this.useSocket && this.options.socketPath) {
|
||||
// Listen on Unix socket
|
||||
this.server.listen(this.options.socketPath, () => {
|
||||
this.isRunning = true;
|
||||
this.startTime = new Date();
|
||||
resolve();
|
||||
});
|
||||
} else {
|
||||
// Listen on TCP
|
||||
this.server.listen(this.options.port, this.options.host, () => {
|
||||
this.isRunning = true;
|
||||
this.startTime = new Date();
|
||||
resolve();
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
@@ -161,9 +189,22 @@ export class TsmdbServer {
|
||||
await this.storage.close();
|
||||
|
||||
return new Promise((resolve) => {
|
||||
this.server!.close(() => {
|
||||
this.server!.close(async () => {
|
||||
this.isRunning = false;
|
||||
this.server = null;
|
||||
|
||||
// Clean up socket file if using Unix socket
|
||||
if (this.useSocket && this.options.socketPath) {
|
||||
try {
|
||||
await fs.unlink(this.options.socketPath);
|
||||
} catch (err: any) {
|
||||
// Ignore ENOENT (file doesn't exist)
|
||||
if (err.code !== 'ENOENT') {
|
||||
console.error('Failed to remove socket file:', err);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
resolve();
|
||||
});
|
||||
});
|
||||
@@ -275,9 +316,21 @@ export class TsmdbServer {
|
||||
* Get the connection URI for this server
|
||||
*/
|
||||
getConnectionUri(): string {
|
||||
if (this.useSocket && this.options.socketPath) {
|
||||
// URL-encode the socket path (replace / with %2F)
|
||||
const encodedPath = encodeURIComponent(this.options.socketPath);
|
||||
return `mongodb://${encodedPath}`;
|
||||
}
|
||||
return `mongodb://${this.options.host}:${this.options.port}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the socket path (if using Unix socket mode)
|
||||
*/
|
||||
get socketPath(): string | undefined {
|
||||
return this.useSocket ? this.options.socketPath : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Check if the server is running
|
||||
*/
|
||||
|
||||
Reference in New Issue
Block a user