smartmongo/readme.md

# @push.rocks/smartmongo

A powerful MongoDB toolkit for testing and development — featuring both a real MongoDB memory server (**SmartMongo**) and an ultra-fast, lightweight wire-protocol-compatible in-memory database server (**TsmDB**). 🚀

## Install

```bash
npm install @push.rocks/smartmongo --save-dev
# or
pnpm add -D @push.rocks/smartmongo
```

## Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.

## Overview

`@push.rocks/smartmongo` provides two powerful approaches for MongoDB in testing and development:

| Feature | SmartMongo | TsmDB |
|---------|------------|---------|
| **Type** | Real MongoDB (memory server) | Pure TypeScript wire protocol server |
| **Speed** | ~2-5s startup | ⚡ Instant startup (~5ms) |
| **Compatibility** | 100% MongoDB | MongoDB driver compatible |
| **Dependencies** | Downloads MongoDB binary | Zero external dependencies |
| **Replication** | ✅ Full replica set support | Single node emulation |
| **Use Case** | Integration testing | Unit testing, CI/CD |
| **Persistence** | Dump to directory | Optional file/memory persistence |

## 🚀 Quick Start

### Option 1: SmartMongo (Real MongoDB)

Spin up a real MongoDB replica set in memory — perfect for integration tests that need full MongoDB compatibility.

```typescript
import { SmartMongo } from '@push.rocks/smartmongo';

// Start a MongoDB replica set
const mongo = await SmartMongo.createAndStart();

// Get connection details
const descriptor = await mongo.getMongoDescriptor();
console.log(descriptor.mongoDbUrl); // mongodb://127.0.0.1:xxxxx/...

// Use with your MongoDB client or ORM
// ... run your tests ...

// Clean up
await mongo.stop();
```

### Option 2: TsmDB (Wire Protocol Server)

A lightweight, pure TypeScript MongoDB-compatible server that speaks the wire protocol — use the official `mongodb` driver directly!

```typescript
import { tsmdb } from '@push.rocks/smartmongo';
import { MongoClient } from 'mongodb';

// Start TsmDB server
const server = new tsmdb.TsmdbServer({ port: 27017 });
await server.start();

// Connect with the official MongoDB driver!
const client = new MongoClient('mongodb://127.0.0.1:27017');
await client.connect();

// Use exactly like real MongoDB
const db = client.db('myapp');
await db.collection('users').insertOne({ name: 'Alice', age: 30 });

const user = await db.collection('users').findOne({ name: 'Alice' });
console.log(user); // { _id: ObjectId(...), name: 'Alice', age: 30 }

// Clean up
await client.close();
await server.stop();
```

## 📖 SmartMongo API

### Creating an Instance

```typescript
import { SmartMongo } from '@push.rocks/smartmongo';

// Default: single replica
const mongo = await SmartMongo.createAndStart();

// Multiple replicas for testing replication
const mongo = await SmartMongo.createAndStart(3);
```

### Getting Connection Details

```typescript
const descriptor = await mongo.getMongoDescriptor();
// {
//   mongoDbName: 'smartmongo_testdatabase',
//   mongoDbUrl: 'mongodb://127.0.0.1:xxxxx/?replicaSet=testset'
// }
```

### Stopping & Cleanup

```typescript
// Simple stop (data discarded)
await mongo.stop();

// Stop and dump data to disk for inspection
await mongo.stopAndDumpToDir('./test-data');

// With custom file naming
await mongo.stopAndDumpToDir('./test-data', (doc) => `${doc.collection}-${doc._id}.bson`);
```

## 🔧 TsmDB API

### Server Configuration

```typescript
import { tsmdb } from '@push.rocks/smartmongo';

const server = new tsmdb.TsmdbServer({
  port: 27017,           // Default MongoDB port
  host: '127.0.0.1',     // Bind address
  storage: 'memory',     // 'memory' or 'file'
  storagePath: './data', // For file-based storage
});

await server.start();
console.log(server.getConnectionUri()); // mongodb://127.0.0.1:27017

// Server properties
console.log(server.running);              // true
console.log(server.getUptime());          // seconds
console.log(server.getConnectionCount()); // active connections

await server.stop();
```

### Supported MongoDB Operations

TsmDB supports the core MongoDB operations via the wire protocol:

#### 🔹 CRUD Operations
```typescript
// Insert
await collection.insertOne({ name: 'Bob' });
await collection.insertMany([{ a: 1 }, { a: 2 }]);

// Find
const doc = await collection.findOne({ name: 'Bob' });
const docs = await collection.find({ age: { $gte: 18 } }).toArray();

// Update
await collection.updateOne({ name: 'Bob' }, { $set: { age: 25 } });
await collection.updateMany({ active: false }, { $set: { archived: true } });

// Delete
await collection.deleteOne({ name: 'Bob' });
await collection.deleteMany({ archived: true });

// Replace
await collection.replaceOne({ _id: id }, { name: 'New Bob', age: 30 });

// Find and Modify
const result = await collection.findOneAndUpdate(
  { name: 'Bob' },
  { $inc: { visits: 1 } },
  { returnDocument: 'after' }
);
```

#### 🔹 Query Operators
```typescript
// Comparison
{ age: { $eq: 25 } }
{ age: { $ne: 25 } }
{ age: { $gt: 18, $lt: 65 } }
{ age: { $gte: 18, $lte: 65 } }
{ status: { $in: ['active', 'pending'] } }
{ status: { $nin: ['deleted'] } }

// Logical
{ $and: [{ age: { $gte: 18 } }, { active: true }] }
{ $or: [{ status: 'active' }, { admin: true }] }
{ $not: { status: 'deleted' } }

// Element
{ email: { $exists: true } }
{ type: { $type: 'string' } }

// Array
{ tags: { $all: ['mongodb', 'database'] } }
{ scores: { $elemMatch: { $gte: 80, $lt: 90 } } }
{ tags: { $size: 3 } }
```

#### 🔹 Update Operators
```typescript
{ $set: { name: 'New Name' } }
{ $unset: { tempField: '' } }
{ $inc: { count: 1 } }
{ $mul: { price: 1.1 } }
{ $min: { lowScore: 50 } }
{ $max: { highScore: 100 } }
{ $push: { tags: 'new-tag' } }
{ $pull: { tags: 'old-tag' } }
{ $addToSet: { tags: 'unique-tag' } }
{ $pop: { queue: 1 } }  // Remove last
{ $pop: { queue: -1 } } // Remove first
```

#### 🔹 Aggregation Pipeline
```typescript
const results = await collection.aggregate([
  { $match: { status: 'active' } },
  { $group: { _id: '$category', total: { $sum: '$amount' } } },
  { $sort: { total: -1 } },
  { $limit: 10 },
  { $project: { category: '$_id', total: 1, _id: 0 } }
]).toArray();
```

Supported stages: `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, and more.

#### 🔹 Index Operations
```typescript
await collection.createIndex({ email: 1 }, { unique: true });
await collection.createIndex({ name: 1, age: -1 });
const indexes = await collection.listIndexes().toArray();
await collection.dropIndex('email_1');
```

#### 🔹 Database Operations
```typescript
// List databases
const dbs = await client.db().admin().listDatabases();

// List collections
const collections = await db.listCollections().toArray();

// Create/drop collections
await db.createCollection('newcollection');
await db.dropCollection('oldcollection');

// Drop database
await db.dropDatabase();
```

#### 🔹 Count & Distinct
```typescript
// Count documents
const total = await collection.countDocuments({});
const active = await collection.countDocuments({ status: 'active' });
const estimated = await collection.estimatedDocumentCount();

// Distinct values
const departments = await collection.distinct('department');
const activeDepts = await collection.distinct('department', { status: 'active' });
```

#### 🔹 Bulk Operations
```typescript
const result = await collection.bulkWrite([
  { insertOne: { document: { name: 'Bulk1' } } },
  { updateOne: { filter: { name: 'John' }, update: { $set: { bulk: true } } } },
  { deleteOne: { filter: { name: 'Expired' } } },
  { replaceOne: { filter: { _id: id }, replacement: { name: 'Replaced' } } }
]);

console.log(result.insertedCount);  // 1
console.log(result.modifiedCount);  // 1
console.log(result.deletedCount);   // 1
```

### Storage Adapters

TsmDB supports pluggable storage with data integrity features:

```typescript
// In-memory (default) - fast, data lost on stop
const server = new tsmdb.TsmdbServer({ storage: 'memory' });

// In-memory with persistence - periodic snapshots to disk
const server = new tsmdb.TsmdbServer({
  storage: 'memory',
  persistPath: './data/snapshot.json',
  persistIntervalMs: 30000  // Save every 30 seconds
});

// File-based - persistent storage with optional checksums
import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';

const adapter = new FileStorageAdapter('./data/tsmdb', {
  enableChecksums: true,   // CRC32 checksums for data integrity
  strictChecksums: false   // Log warnings vs throw on mismatch
});
```

## ⚡ Performance & Reliability Features

TsmDB includes enterprise-grade features for robustness:

### 🔍 Index-Accelerated Queries

Indexes are automatically used to accelerate queries. Instead of scanning all documents, TsmDB uses:

- **Hash indexes** for equality queries (`$eq`, `$in`)
- **B-tree indexes** for range queries (`$gt`, `$gte`, `$lt`, `$lte`)

```typescript
// Create an index
await collection.createIndex({ email: 1 });
await collection.createIndex({ age: 1 });

// These queries will use the index (fast!)
await collection.findOne({ email: 'alice@example.com' }); // Uses hash lookup
await collection.find({ age: { $gte: 18, $lt: 65 } });    // Uses B-tree range scan
```

### 📊 Query Planner

TsmDB includes a query planner that analyzes queries and selects optimal execution strategies:

```typescript
import { tsmdb } from '@push.rocks/smartmongo';

// For debugging, you can access the query planner
const planner = new tsmdb.QueryPlanner(indexEngine);
const plan = planner.createPlan(filter);

console.log(plan);
// {
//   type: 'IXSCAN',           // or 'IXSCAN_RANGE', 'COLLSCAN'
//   indexName: 'email_1',
//   estimatedCost: 1,
//   selectivity: 0.001
// }
```

### 📝 Write-Ahead Logging (WAL)

For durability, TsmDB supports write-ahead logging:

```typescript
import { tsmdb } from '@push.rocks/smartmongo';

const wal = new tsmdb.WAL('./data/wal.log');
await wal.initialize();

// WAL entries include:
// - LSN (Log Sequence Number)
// - Timestamp
// - Operation type (insert, update, delete, checkpoint)
// - Document data (BSON serialized)
// - CRC32 checksum

// Recovery support
const entries = await wal.getEntriesAfter(lastCheckpointLsn);
```

### 🔐 Session Management

TsmDB tracks client sessions with automatic timeout and transaction linking:

```typescript
// Sessions are automatically managed when using the MongoDB driver
const session = client.startSession();

try {
  session.startTransaction();
  await collection.insertOne({ name: 'Alice' }, { session });
  await collection.updateOne({ name: 'Bob' }, { $inc: { balance: 100 } }, { session });
  await session.commitTransaction();
} catch (error) {
  await session.abortTransaction();
} finally {
  session.endSession();
}

// Session features:
// - Automatic session timeout (30 minutes default)
// - Transaction auto-abort on session expiry
// - Session activity tracking
```

### ✅ Data Integrity Checksums

File-based storage supports CRC32 checksums to detect corruption:

```typescript
import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';

const adapter = new FileStorageAdapter('./data', {
  enableChecksums: true,
  strictChecksums: true  // Throw error on corruption (vs warning)
});

// Documents are checksummed on write, verified on read
// Checksums are automatically stripped before returning to client
```

### 📋 Supported Wire Protocol Commands

| Category | Commands |
|----------|----------|
| **Handshake** | `hello`, `isMaster` |
| **CRUD** | `find`, `insert`, `update`, `delete`, `findAndModify`, `getMore`, `killCursors` |
| **Aggregation** | `aggregate`, `count`, `distinct` |
| **Indexes** | `createIndexes`, `dropIndexes`, `listIndexes` |
| **Transactions** | `startTransaction`, `commitTransaction`, `abortTransaction` |
| **Sessions** | `startSession`, `endSessions` |
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats` |

TsmDB supports MongoDB wire protocol versions 0-21, compatible with MongoDB 3.6 through 7.0 drivers.

## 🧪 Testing Examples

### Jest/Mocha with TsmDB

```typescript
import { tsmdb } from '@push.rocks/smartmongo';
import { MongoClient, Db } from 'mongodb';

let server: tsmdb.TsmdbServer;
let client: MongoClient;
let db: Db;

beforeAll(async () => {
  server = new tsmdb.TsmdbServer({ port: 27117 });
  await server.start();

  client = new MongoClient('mongodb://127.0.0.1:27117');
  await client.connect();
  db = client.db('test');
});

afterAll(async () => {
  await client.close();
  await server.stop();
});

beforeEach(async () => {
  // Clean slate for each test
  await db.dropDatabase();
});

test('should insert and find user', async () => {
  const users = db.collection('users');
  await users.insertOne({ name: 'Alice', email: 'alice@example.com' });

  const user = await users.findOne({ name: 'Alice' });
  expect(user?.email).toBe('alice@example.com');
});
```

### With @push.rocks/tapbundle

```typescript
import { expect, tap } from '@git.zone/tstest/tapbundle';
import { tsmdb } from '@push.rocks/smartmongo';
import { MongoClient } from 'mongodb';

let server: tsmdb.TsmdbServer;
let client: MongoClient;

tap.test('setup', async () => {
  server = new tsmdb.TsmdbServer({ port: 27117 });
  await server.start();
  client = new MongoClient('mongodb://127.0.0.1:27117');
  await client.connect();
});

tap.test('should perform CRUD operations', async () => {
  const db = client.db('test');
  const col = db.collection('items');

  // Create
  const result = await col.insertOne({ name: 'Widget', price: 9.99 });
  expect(result.insertedId).toBeTruthy();

  // Read
  const item = await col.findOne({ name: 'Widget' });
  expect(item?.price).toEqual(9.99);

  // Update
  await col.updateOne({ name: 'Widget' }, { $set: { price: 12.99 } });
  const updated = await col.findOne({ name: 'Widget' });
  expect(updated?.price).toEqual(12.99);

  // Delete
  await col.deleteOne({ name: 'Widget' });
  const deleted = await col.findOne({ name: 'Widget' });
  expect(deleted).toBeNull();
});

tap.test('teardown', async () => {
  await client.close();
  await server.stop();
});

export default tap.start();
```

## 🏗️ Architecture

### TsmDB Wire Protocol Stack

```
┌─────────────────────────────────────────────────────────────┐
│                   Official MongoDB Driver                    │
│                      (mongodb npm)                          │
└─────────────────────────┬───────────────────────────────────┘
                          │ TCP + OP_MSG/BSON
                          ▼
┌─────────────────────────────────────────────────────────────┐
│                      TsmdbServer                            │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │ WireProtocol │→ │CommandRouter │→ │     Handlers     │  │
│  │  (OP_MSG)    │  │              │  │ (Find, Insert..) │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│                       Engines                               │
│  ┌─────────┐ ┌────────┐ ┌───────────┐ ┌───────┐ ┌───────┐ │
│  │ Query   │ │ Update │ │Aggregation│ │ Index │ │Session│ │
│  │ Planner │ │ Engine │ │  Engine   │ │Engine │ │Engine │ │
│  └─────────┘ └────────┘ └───────────┘ └───────┘ └───────┘ │
│                   ┌──────────────────────┐                  │
│                   │ Transaction Engine   │                  │
│                   └──────────────────────┘                  │
└─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────────┐
│                   Storage Layer                              │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────┐  │
│  │  MemoryStorage   │  │   FileStorage    │  │   WAL    │  │
│  │                  │  │  (+ Checksums)   │  │          │  │
│  └──────────────────┘  └──────────────────┘  └──────────┘  │
└─────────────────────────────────────────────────────────────┘
```

### Key Components

| Component | Description |
|-----------|-------------|
| **WireProtocol** | Parses MongoDB OP_MSG binary protocol |
| **CommandRouter** | Routes commands to appropriate handlers |
| **QueryPlanner** | Analyzes queries and selects execution strategy |
| **IndexEngine** | Manages B-tree and hash indexes |
| **SessionEngine** | Tracks client sessions and timeouts |
| **TransactionEngine** | Handles ACID transaction semantics |
| **WAL** | Write-ahead logging for durability |

## License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.

**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

### Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

### Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.