push.rocks/smartdb
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

BREAKING CHANGE(core): replace the TypeScript database engine with a Rust-backed embedded server and bridge

Browse Source
This commit is contained in:
Jürgen Kunz
2026-03-26 19:48:27 +00:00
parent 8ec2046908
commit e23a951dbe
106 changed files with 11567 additions and 10678 deletions
Download Patch File Download Diff File Expand all files Collapse all files

257
readme.md
View File

@@ -1,6 +1,6 @@
# @push.rocks/smartdb
A pure TypeScript MongoDB wire-protocol-compatible database server. Zero binary dependencies, instant startup, pluggable storage — use the official MongoDB driver and it just works. ⚡
A MongoDB-compatible embedded database server powered by Rust 🦀⚡ — use the official `mongodb` driver and it just works. No binary downloads, instant startup, zero config.
## Install
@@ -16,22 +16,67 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community
## What It Does
`@push.rocks/smartdb` is a **real database server** written entirely in TypeScript that speaks the MongoDB binary wire protocol. Connect with the official `mongodb` Node.js driver — no mocks, no stubs, no MongoDB binary required.
`@push.rocks/smartdb` is a **real database server** that speaks the wire protocol used by MongoDB drivers. The core engine is written in Rust for high performance, with a thin TypeScript orchestration layer. Connect with the standard `mongodb` Node.js driver — no mocks, no stubs, no external binaries required.
### Why SmartDB?
| | SmartDB | Real MongoDB |
| | SmartDB | External DB Server |
|---|---|---|
| **Startup time** | ~5ms | ~2-5s |
| **Binary download** | None | ~200MB |
| **Node.js only** | ✅ | ❌ |
| **Startup time** | ~30ms | ~2-5s |
| **Binary download** | Bundled (~7MB) | ~200MB+ |
| **Install** | `pnpm add` | System package / Docker |
| **Persistence** | Memory or file-based | Full disk engine |
| **Perfect for** | Unit tests, CI/CD, prototyping, local dev | Production |
| **Perfect for** | Unit tests, CI/CD, prototyping, local dev, embedded | Production at scale |
### Two Ways to Use It
- 🏗️ **`SmartdbServer`** — Full control. Configure port, host, storage backend, Unix sockets. Great for test fixtures or custom setups.
- 🎯 **`LocalSmartDb`** — Zero-config convenience. Give it a folder path, get a persistent MongoDB-compatible database over a Unix socket. Done.
- 🎯 **`LocalSmartDb`** — Zero-config convenience. Give it a folder path, get a persistent database over a Unix socket. Done.
### Architecture: TypeScript + Rust 🦀
SmartDB uses the same **sidecar binary** pattern as [@push.rocks/smartproxy](https://code.foss.global/push.rocks/smartproxy):
```
┌──────────────────────────────────────────────────────────┐
│ Your Application │
│ (TypeScript / Node.js) │
│ ┌─────────────────┐ ┌───────────────────────────┐ │
│ │ SmartdbServer │────▶│ RustDbBridge (IPC) │ │
│ │ or LocalSmartDb │ │ @push.rocks/smartrust │ │
│ └─────────────────┘ └───────────┬───────────────┘ │
└──────────────────────────────────────┼───────────────────┘
│ spawn + JSON IPC
▼
┌──────────────────────────────────────────────────────────┐
│ rustdb binary 🦀 │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ Wire Protocol│→ │Command Router│→ │ Handlers │ │
│ │ (OP_MSG) │ │ (40 cmds) │ │ Find,Insert.. │ │
│ └──────────────┘ └──────────────┘ └───────┬───────┘ │
│ │ │
│ ┌─────────┐ ┌────────┐ ┌───────────┐ ┌──────┴──────┐ │
│ │ Query │ │ Update │ │Aggregation│ │ Index │ │
│ │ Matcher │ │ Engine │ │ Engine │ │ Engine │ │
│ └─────────┘ └────────┘ └───────────┘ └─────────────┘ │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ MemoryStorage │ │ FileStorage │ │
│ └──────────────────┘ └──────────────────┘ │
└──────────────────────────────────────────────────────────┘
▲
│ TCP / Unix Socket (wire protocol)
│
┌─────────────┴────────────────────────────────────────────┐
│ MongoClient (mongodb npm driver) │
│ Connects directly to Rust binary │
└──────────────────────────────────────────────────────────┘
```
The TypeScript layer handles **lifecycle only** (start/stop/configure via IPC). All database operations flow directly from the `MongoClient` to the Rust binary over TCP or Unix sockets — **zero per-query IPC overhead**.
---
## Quick Start
@@ -47,11 +92,11 @@ import { MongoClient } from 'mongodb';
const db = new LocalSmartDb({ folderPath: './my-data' });
const { connectionUri } = await db.start();
// Connect with the standard MongoDB driver
// Connect with the standard driver
const client = new MongoClient(connectionUri, { directConnection: true });
await client.connect();
// Use exactly like MongoDB
// Use it like any wire-protocol-compatible database
const users = client.db('myapp').collection('users');
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
const user = await users.findOne({ name: 'Alice' });
@@ -89,7 +134,7 @@ await server.stop();
### SmartdbServer
The core server class. Speaks MongoDB wire protocol over TCP or Unix sockets.
The core server class. Manages the Rust database engine and exposes connection details.
#### Constructor Options (`ISmartdbServerOptions`)
@@ -123,15 +168,13 @@ const server = new SmartdbServer({
| Method / Property | Type | Description |
|---|---|---|
| `start()` | `Promise<void>` | Start the server |
| `start()` | `Promise<void>` | Start the server (spawns Rust binary) |
| `stop()` | `Promise<void>` | Stop the server and clean up |
| `getConnectionUri()` | `string` | Get the MongoDB connection URI |
| `getConnectionUri()` | `string` | Get the `mongodb://` connection URI |
| `running` | `boolean` | Whether the server is currently running |
| `port` | `number` | Bound port (TCP mode) |
| `host` | `string` | Bound host (TCP mode) |
| `socketPath` | `string` | Socket path (socket mode) |
| `getUptime()` | `number` | Seconds since start |
| `getConnectionCount()` | `number` | Active client connections |
| `port` | `number` | Configured port (TCP mode) |
| `host` | `string` | Configured host (TCP mode) |
| `socketPath` | `string \| undefined` | Socket path (socket mode) |
### LocalSmartDb
@@ -155,7 +198,7 @@ const db = new LocalSmartDb({
| `start()` | `Promise<ILocalSmartDbConnectionInfo>` | Start and return connection info |
| `stop()` | `Promise<void>` | Stop the server |
| `getConnectionInfo()` | `ILocalSmartDbConnectionInfo` | Get current connection info |
| `getConnectionUri()` | `string` | Get the MongoDB URI |
| `getConnectionUri()` | `string` | Get the connection URI |
| `getServer()` | `SmartdbServer` | Access the underlying server |
| `running` | `boolean` | Whether the server is running |
@@ -170,9 +213,9 @@ interface ILocalSmartDbConnectionInfo {
---
## Supported MongoDB Operations
## Supported Operations
SmartDB supports the core MongoDB operations through the wire protocol. Use the standard `mongodb` driver — these all work:
SmartDB supports the core operations through the wire protocol. Use the standard `mongodb` driver — these all work:
### CRUD
@@ -258,7 +301,7 @@ const results = await collection.aggregate([
]).toArray();
```
**Supported stages:** `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, `$replaceRoot`, `$set`, `$unset`
**Supported stages:** `$match`, `$project`, `$group`, `$sort`, `$limit`, `$skip`, `$unwind`, `$lookup`, `$addFields`, `$count`, `$facet`, `$replaceRoot`, `$set`, `$unionWith`, `$out`, `$merge`
**Group accumulators:** `$sum`, `$avg`, `$min`, `$max`, `$first`, `$last`, `$push`, `$addToSet`, `$count`
@@ -316,148 +359,34 @@ const names = await collection.distinct('name');
| **CRUD** | `find`, `insert`, `update`, `delete`, `findAndModify`, `getMore`, `killCursors` |
| **Aggregation** | `aggregate`, `count`, `distinct` |
| **Indexes** | `createIndexes`, `dropIndexes`, `listIndexes` |
| **Transactions** | `startTransaction`, `commitTransaction`, `abortTransaction` |
| **Sessions** | `startSession`, `endSessions`, `refreshSessions` |
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats`, `connectionStatus`, `currentOp`, `collMod`, `renameCollection` |
| **Sessions** | `startSession`, `endSessions` |
| **Transactions** | `commitTransaction`, `abortTransaction` |
| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats`, `connectionStatus`, `currentOp`, `renameCollection` |
Compatible with MongoDB wire protocol versions 0–21 (MongoDB 3.6 through 7.0 drivers).
Compatible with wire protocol versions 0–21 (driver versions 3.6 through 7.0).
---
## Architecture
## Rust Crate Architecture 🦀
```
┌─────────────────────────────────────────────────────────────┐
│ Official MongoDB Driver │
│ (mongodb npm) │
└─────────────────────────┬───────────────────────────────────┘
│ TCP / Unix Socket + OP_MSG / BSON
▼
┌─────────────────────────────────────────────────────────────┐
│ SmartdbServer │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ │ WireProtocol │→ │CommandRouter │→ │ Handlers │ │
│ │ (OP_MSG) │ │ │ │ (Find, Insert..) │ │
│ └──────────────┘ └──────────────┘ └──────────────────┘ │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Engines │
│ ┌─────────┐ ┌────────┐ ┌───────────┐ ┌───────┐ ┌───────┐ │
│ │ Query │ │ Update │ │Aggregation│ │ Index │ │Session│ │
│ │ Planner │ │ Engine │ │ Engine │ │Engine │ │Engine │ │
│ └─────────┘ └────────┘ └───────────┘ └───────┘ └───────┘ │
│ ┌──────────────────────┐ │
│ │ Transaction Engine │ │
│ └──────────────────────┘ │
└─────────────────────────┬───────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Storage Layer │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────┐ │
│ │ MemoryStorage │ │ FileStorage │ │ WAL │ │
│ │ │ │ (+ Checksums) │ │ │ │
│ └──────────────────┘ └──────────────────┘ └──────────┘ │
└─────────────────────────────────────────────────────────────┘
```
The Rust engine is organized as a Cargo workspace with 8 focused crates:
### Key Components
| Component | What It Does |
| Crate | Purpose |
|---|---|
| **WireProtocol** | Parses/encodes MongoDB OP_MSG binary frames |
| **CommandRouter** | Routes parsed commands to the right handler |
| **QueryPlanner** | Picks COLLSCAN vs IXSCAN based on available indexes |
| **QueryEngine** | Filter matching powered by [mingo](https://github.com/kofrasa/mingo) |
| **UpdateEngine** | Processes `$set`, `$inc`, `$push`, and all update operators |
| **AggregationEngine** | Runs aggregation pipelines via mingo |
| **IndexEngine** | B-tree (range) and hash (equality) indexes |
| **TransactionEngine** | ACID transactions with snapshot isolation |
| **SessionEngine** | Client session tracking with automatic timeouts |
| **WAL** | Write-ahead logging with CRC32 checksums for crash recovery |
| `rustdb` | Binary entry point: TCP/Unix listener, management IPC, CLI |
| `rustdb-config` | Server configuration types (serde, camelCase JSON) |
| `rustdb-wire` | Wire protocol parser/encoder (OP_MSG, OP_QUERY, OP_REPLY) |
| `rustdb-query` | Query matcher, update engine, aggregation, sort, projection |
| `rustdb-storage` | Storage backends (memory, file) + WAL + OpLog |
| `rustdb-index` | B-tree/hash indexes, query planner (IXSCAN/COLLSCAN) |
| `rustdb-txn` | Transaction + session management with snapshot isolation |
| `rustdb-commands` | 40 command handlers wiring everything together |
Cross-compiled for `linux_amd64` and `linux_arm64` via [@git.zone/tsrust](https://www.npmjs.com/package/@git.zone/tsrust).
---
## Advanced Usage
### Storage Adapters
```typescript
import { SmartdbServer } from '@push.rocks/smartdb';
// In-memory (default) — fast, data lost on stop
const server = new SmartdbServer({ storage: 'memory' });
// In-memory with periodic persistence
const server = new SmartdbServer({
storage: 'memory',
persistPath: './data/snapshot.json',
persistIntervalMs: 30000,
});
// File-based — persistent storage with CRC32 checksums
const server = new SmartdbServer({
storage: 'file',
storagePath: './data/smartdb',
});
```
### Query Planner (Debugging)
```typescript
import { QueryPlanner, IndexEngine, MemoryStorageAdapter } from '@push.rocks/smartdb';
const storage = new MemoryStorageAdapter();
await storage.initialize();
const indexEngine = new IndexEngine('mydb', 'mycoll', storage);
const planner = new QueryPlanner(indexEngine);
const plan = await planner.plan({ age: { $gte: 18 } });
console.log(plan);
// { type: 'IXSCAN_RANGE', indexName: 'age_1', selectivity: 0.3, usesRange: true, ... }
const explain = await planner.explain({ age: 18 });
// Returns winning plan, rejected plans, and detailed analysis
```
### Data Integrity Checksums
```typescript
import { calculateDocumentChecksum, addChecksum, verifyChecksum, removeChecksum } from '@push.rocks/smartdb';
const doc = { name: 'Alice', age: 30 };
const protected = addChecksum(doc); // Adds _checksum field
const valid = verifyChecksum(protected); // true
protected.age = 31; // Tamper!
const still = verifyChecksum(protected); // false
const clean = removeChecksum(protected); // Removes _checksum
```
### Write-Ahead Logging
```typescript
import { WAL } from '@push.rocks/smartdb';
const wal = new WAL('./data/wal.log', { checkpointInterval: 100 });
await wal.initialize();
// Entries include: LSN, timestamp, operation, BSON data, CRC32 checksum
const lsn = await wal.logInsert('mydb', 'users', doc);
const entries = wal.getEntriesAfter(lastCheckpoint);
const recovered = wal.recoverDocument(entry);
await wal.checkpoint();
await wal.close();
```
---
## Testing Examples
### Unit Tests with @git.zone/tstest
## Testing Example
```typescript
import { expect, tap } from '@git.zone/tstest/tapbundle';
@@ -489,27 +418,11 @@ tap.test('teardown', async () => {
export default tap.start();
```
### With LocalSmartDb (Persistent Tests)
```typescript
import { LocalSmartDb } from '@push.rocks/smartdb';
import { MongoClient } from 'mongodb';
const db = new LocalSmartDb({ folderPath: './test-data' });
const { connectionUri } = await db.start();
const client = new MongoClient(connectionUri, { directConnection: true });
await client.connect();
// Tests here — data persists between test runs!
await client.close();
await db.stop();
```
---
## License and Legal Information
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.