push.rocks/smartdb
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
160 Commits 1 Branch 70 Tags
main
T
Clone
Open with VS Code Open with VSCodium Open with Intellij IDEA
Download ZIP Download TAR.GZ Download BUNDLE

readme.md

@push.rocks/smartdb

A MongoDB-wire-compatible embedded database server powered by Rust 🦀⚡. It supports the documented command surface through the official mongodb driver without an external MongoDB server. No binary downloads, instant startup, zero config. Features a built-in operation log with point-in-time revert and a web-based debug dashboard.

Install

pnpm add @push.rocks/smartdb
# or
npm install @push.rocks/smartdb

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

What It Does

@push.rocks/smartdb is a real database server that speaks the wire protocol used by MongoDB drivers. The core engine is written in Rust for high performance, with a thin TypeScript orchestration layer. Connect with the standard mongodb Node.js driver — no mocks, no stubs, no external binaries required.

Why SmartDB?

SmartDB External DB Server
Startup time ~30ms ~2-5s
Binary download Bundled (~7MB) ~200MB+
Install pnpm add System package / Docker
Persistence Memory or file-based Full disk engine
Debug UI Built-in 🖥️ External tooling
Point-in-time revert Built-in ⏪ Requires oplog tailing
Perfect for Unit tests, CI/CD, prototyping, local dev, embedded Production at scale

Three Ways to Use It

  • 🎯 LocalSmartDb — Zero-config convenience. Give it a folder path, get a persistent database over a Unix socket. Done.
  • 🏗️ SmartdbServer — Full control. Configure port, host, storage backend, Unix sockets. Great for test fixtures or custom setups.
  • 🖥️ SmartdbDebugServer — Launch a web dashboard to visually browse collections, inspect the operation log, and revert to any point in time.

Architecture: TypeScript + Rust 🦀

SmartDB uses a sidecar binary pattern — TypeScript handles lifecycle, Rust handles all database operations:

┌──────────────────────────────────────────────────────────────┐
│                   Your Application                           │
│                  (TypeScript / Node.js)                      │
│  ┌──────────────────┐      ┌───────────────────────────┐     │
│  │  SmartdbServer   │─────▶│  RustDbBridge (IPC)       │     │
│  │  or LocalSmartDb │      │  @push.rocks/smartrust    │     │
│  └──────────────────┘      └───────────┬───────────────┘     │
└────────────────────────────────────────┼─────────────────────┘
                                         │ spawn + JSON IPC
                                         ▼
┌──────────────────────────────────────────────────────────────┐
│                    rustdb binary                             │
│                                                              │
│  ┌──────────────┐  ┌──────────────┐  ┌───────────────┐       │
│  │ Wire Protocol│→ │Command Router│→ │   Handlers    │       │
│  │  (OP_MSG)    │  │  (40+ cmds)  │  │ Find,Insert.. │       │
│  └──────────────┘  └──────────────┘  └───────┬───────┘       │
│                                              │               │
│  ┌─────────┐ ┌────────┐ ┌───────────┐ ┌──────┴──────┐        │
│  │  Query  │ │ Update │ │Aggregation│ │   Index     │        │
│  │ Matcher │ │ Engine │ │  Engine   │ │   Engine    │        │
│  └─────────┘ └────────┘ └───────────┘ └─────────────┘        │
│                                                              │
│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────┐    │
│  │  MemoryStorage   │  │   FileStorage    │  │  OpLog   │    │
│  └──────────────────┘  └──────────────────┘  └──────────┘    │
└──────────────────────────────────────────────────────────────┘
              ▲
              │ TCP / Unix Socket (wire protocol)
              │
┌─────────────┴────────────────────────────────────────────────┐
│              MongoClient (mongodb npm driver)                │
│              Connects directly to Rust binary                │
└──────────────────────────────────────────────────────────────┘

The TypeScript layer handles lifecycle only (start/stop/configure via IPC). All database operations flow directly from the MongoClient to the Rust binary over TCP or Unix sockets — zero per-query IPC overhead.

Quick Start

Option 1: LocalSmartDb (Zero Config) 🎯

The fastest way to get a persistent local database:

import { LocalSmartDb } from '@push.rocks/smartdb';
import { MongoClient } from 'mongodb';

// Point it at a folder — that's it
const db = new LocalSmartDb({ folderPath: './my-data' });
const { connectionUri } = await db.start();

// Connect with the standard driver
const client = new MongoClient(connectionUri, { directConnection: true });
await client.connect();

// Use it like any wire-protocol-compatible database
const users = client.db('myapp').collection('users');
await users.insertOne({ name: 'Alice', email: 'alice@example.com' });
const user = await users.findOne({ name: 'Alice' });
console.log(user); // { _id: ObjectId(...), name: 'Alice', email: 'alice@example.com' }

// Data persists to disk automatically — survives restarts!
await client.close();
await db.stop();

Option 2: SmartdbServer (Full Control) 🏗️

import { SmartdbServer } from '@push.rocks/smartdb';
import { MongoClient } from 'mongodb';

// TCP mode
const server = new SmartdbServer({ port: 27017 });
await server.start();

const client = new MongoClient('mongodb://127.0.0.1:27017');
await client.connect();

const db = client.db('myapp');
await db.collection('users').insertOne({ name: 'Alice', age: 30 });
const user = await db.collection('users').findOne({ name: 'Alice' });

await client.close();
await server.stop();

Option 3: Debug Server (Visual Dashboard) 🖥️

Launch a web-based dashboard to inspect your database in real time:

debugserver and debugui are optional subpath exports. Install their debug dependencies only when you use them:

pnpm add '@api.global/typedserver@^8' '@design.estate/dees-element@^2'

import { SmartdbServer } from '@push.rocks/smartdb';
import { SmartdbDebugServer } from '@push.rocks/smartdb/debugserver';

const server = new SmartdbServer({ storage: 'memory' });
await server.start();

const debugServer = new SmartdbDebugServer(server, { port: 4000 });
await debugServer.start();
// Open http://localhost:4000 in your browser 🚀

The debug dashboard gives you:

  • 📊 Dashboard — server status, uptime, database/collection counts, operation breakdown
  • 📁 Collection Browser — browse databases, collections, and documents interactively
  • 📝 OpLog Timeline — every insert, update, and delete with expandable field-level diffs
  • ⏪ Point-in-Time Revert — select any oplog sequence, preview what will be undone, and execute

📝 Operation Log & Point-in-Time Revert

Every write operation (insert, update, delete) is automatically recorded in an in-memory operation log (OpLog) with full before/after document snapshots. The OpLog lives in RAM and resets on restart — it covers the current session only, and retention is bounded (default: 10,000 entries or 64 MiB, whichever is hit first; configurable via the oplog server option). Once a limit is exceeded, the oldest entries are evicted. This enables:

  • Change tracking — see exactly what changed, when, and in which collection
  • Field-level diffs — compare previous and new document states
  • Point-in-time revert — undo operations back to any retained sequence number
  • Dry-run preview — see what would be reverted before executing

Programmatic OpLog API

import { SmartdbServer } from '@push.rocks/smartdb';

const server = new SmartdbServer({ port: 27017 });
await server.start();

// ... perform some CRUD operations via MongoClient ...

// Get oplog entries
const oplog = await server.getOpLog({ limit: 50 });
console.log(oplog.entries);
// [{ seq: 1, op: 'insert', db: 'myapp', collection: 'users', document: {...}, previousDocument: null }, ...]

// Get aggregate stats
const stats = await server.getOpLogStats();
console.log(stats);
// { currentSeq: 42, totalEntries: 42, oldestSeq: 1, approxBytes: 18342, entriesByOp: { insert: 20, update: 15, delete: 7 } }

// Preview a revert (dry run)
const preview = await server.revertToSeq(30, true);
console.log(`Would undo ${preview.reverted} operations`);

// Execute the revert — undoes all operations after seq 30
const result = await server.revertToSeq(30, false);
console.log(`Reverted ${result.reverted} operations`);

// Reverts can only reach back as far as retained oplog history —
// revertToSeq returns an error if the target sequence is older than the
// oldest retained entry (evicted by the retention limits).

// Browse collections programmatically
const collections = await server.getCollections();
const docs = await server.getDocuments('myapp', 'users', 50, 0);

OpLog Entry Structure

Each entry contains:

Field Type Description
seq number Monotonically increasing sequence number
timestampMs number Unix timestamp in milliseconds
op 'insert' | 'update' | 'delete' Operation type
db string Database name
collection string Collection name
documentId string Document _id as hex string
document object | null New document state (null for deletes)
previousDocument object | null Previous document state (null for inserts)

✋ No-Op Write Detection

The engine detects document rewrites that change nothing and skips them entirely — no storage write, no WAL append, no index update, and no oplog entry. A skipped rewrite still counts as matched (matchedCount: 1) but reports modifiedCount: 0.

Two classes are skipped:

  • Identical — the post-image equals the stored document byte for byte.
  • Volatile-only — the post-image differs only in top-level volatile metadata fields (currently _updatedAt, the field ODM layers such as @push.rocks/smartdata restamp on every save). The stored document is kept as-is, including its existing _updatedAt, so the timestamp means "last real change" rather than "last save call".

This makes periodic reconcile loops that re-save unchanged documents cost nothing at the engine level. A caller that needs a document to actually change must change a non-volatile field.

Counters are exposed through serverStatus:

const status = await db.command({ serverStatus: 1 });
console.log(status.writes);
// {
//   updatesWritten: 42,        // rewrites that reached storage, index, and oplog
//   noopSkipped: {
//     identical: 1337,         // post-image equal to the stored document
//     volatileOnly: 271,       // only volatile metadata differed
//   },
// }

API Reference

SmartdbServer

The core server class. Manages the Rust database engine and exposes connection details.

Constructor Options (ISmartdbServerOptions)

import { SmartdbServer } from '@push.rocks/smartdb';

// TCP mode (default)
const server = new SmartdbServer({
  port: 0,                  // Default: 27017; 0 requests an OS-assigned port
  host: '127.0.0.1',        // Default: 127.0.0.1
  storage: 'memory',        // 'memory' or 'file' (default: 'memory')
  storagePath: './data',     // Required when storage is 'file'
});
const startupController = new AbortController();
await server.start({
  signal: startupController.signal,
  timeoutMs: 30_000,
});
console.log(server.port); // Actual bound port while running
console.log(server.getConnectionUri()); // mongodb://127.0.0.1:<actual-port>

// Unix socket mode — no port conflicts!
const server = new SmartdbServer({
  socketPath: '/tmp/smartdb.sock',
  storage: 'file',
  storagePath: './data',
});

// Memory storage with periodic persistence
const server = new SmartdbServer({
  storage: 'memory',
  persistPath: './data/snapshot.json',
  persistIntervalMs: 30000, // Save every 30s
});

// Bounded in-memory oplog retention
const server = new SmartdbServer({
  port: 27017,
  oplog: {
    maxEntries: 50000,            // Default: 10000
    maxBytes: 128 * 1024 * 1024,  // Default: 64 MiB
  },
});

// TLS transport for TCP mode
const tlsServer = new SmartdbServer({
  port: 27017,
  tls: {
    enabled: true,
    certPath: './certs/server.pem',
    keyPath: './certs/server.key',
    // caPath: './certs/client-ca.pem',
    // requireClientCert: true, // Enables mTLS client certificate checks
  },
});

// SCRAM-SHA-256 authentication
const secureServer = new SmartdbServer({
  port: 27017,
  auth: {
    enabled: true,
    usersPath: './data/smartdb-users.json', // Optional: persists derived SCRAM credentials
    users: [
      {
        username: 'root',
        password: 'change-me',
        database: 'admin',
        roles: ['root'],
      },
    ],
  },
});

When auth.enabled is true, protected commands require successful SCRAM-SHA-256 authentication through the official MongoDB driver:

const client = new MongoClient('mongodb://root:change-me@127.0.0.1:27017/admin?authSource=admin', {
  directConnection: true,
});
await client.connect();

TLS is available for TCP listeners. getConnectionUri() includes ?tls=true when TLS is enabled; pass the trusted CA to the MongoDB driver with tlsCAFile, ca, or secureContext.

Authentication verifies SCRAM credentials, denies unauthenticated commands, and enforces command-level built-in roles for supported operations. connectionStatus reports the authenticated users and roles for the current socket.

Supported built-in role names are root, read, readWrite, dbAdmin, userAdmin, clusterMonitor, plus readAnyDatabase, readWriteAnyDatabase, dbAdminAnyDatabase, and userAdminAnyDatabase. When usersPath is set, SmartDB persists SCRAM credential material atomically and does not store plaintext passwords. Auth metadata version 3 contains protected decoy material, an immutable persisted SCRAM profile, allocation read-grant principals, and bounded anti-replay state. auth.scramIterations must match the profile already established by that usersPath on every later startup and in every attached process; SmartDB fails closed on a mismatch rather than attempting to rederive credentials without plaintext passwords. SmartDB 4.0.0 startup persistently rewrites exact version 2 owner metadata to version 3 without changing existing owner SCRAM material, principal identities, generations, or roles. This is a one-way major-version compatibility boundary: after migration, SmartDB 3.x must not be restarted against that usersPath. Unsupported versions and malformed auth or grant metadata fail closed.

On Linux, startup performs one narrow compatibility migration before the Rust engine opens usersPath: an effective-user-owned, single-link regular file with the exact legacy mode 0644 is changed to 0600. Current 0600 files are left unchanged. Symlinks, hard links, unexpected modes or ownership, unsafe parent directories, cross-device targets, and identity changes remain fail-closed errors.

Legacy v0 JSON collections are converted into hidden sibling staging directories. SmartDB fsyncs every generated file and the staging directory before atomically publishing a complete v1 collection, while the v0 files remain unchanged. Cancellation removes unpublished staging and preserves the v0 input. Startup fails closed if it finds staging left by an unclean process exit or an incomplete published target, so neither state is mistaken for a complete migration.

Persisted users also carry a random principal identity and a monotonic generation. SmartDB reloads and resolves that identity for every authenticated command, so password or role changes take effect immediately and stale sockets are rejected. Deleting and recreating the same username creates a different principal; an old connection cannot inherit the replacement user's authority. Cross-process user updates are serialized through the persisted users-file lock.

Single-node transactions are supported through official MongoDB driver sessions. Writes with startTransaction and autocommit: false are buffered per logical session, reads inside the transaction see the buffered overlay, commitTransaction applies the write set with conflict checks, and abortTransaction discards it. Live logical sessions remain resumable across socket disconnects. Bounded background cleanup aborts expired transactions, removes expired sessions, and releases publication leases; explicit endSessions and killSessions do the same for their active transactions.

Durable Database Publication Holds

File-backed SmartDB can keep a replaced or deleted database closed after the local mutation is durable and until a downstream coordinator confirms its own fsync. Check the explicit health contract before using this flow:

const health = await server.getHealth();
if (
  health.publicationHoldVersion !== 1 ||
  !health.publicationHoldSupported ||
  health.publicationHoldRequiresExternalDrain
) {
  throw new Error('SmartDB publication holds are unavailable');
}

Set holdPublication: true on an exact fenced importDatabase() or deleteDatabaseTenant() operation. The result contains a held resourceFence with a one-time publicationCapability. Treat that capability as a secret: do not log it or persist it outside protected control-plane state. After downstream state is durable, submit the exact receipt to commitDatabasePublication():

import type { ISmartDbHeldPublicationReceipt } from '@push.rocks/smartdb';

const held = await server.importDatabase({
  databaseName: 'tenant_a',
  source: snapshot,
  username: 'tenant_a_user',
  fence: {
    version: 1,
    scopeId: 'corestore-node-1.tenant-a',
    token: 42,
    mutationId: 'restore-42',
    payloadSha256: controlPlanePayloadSha256,
  },
  holdPublication: true,
});

const receipt = held.resourceFence as ISmartDbHeldPublicationReceipt;
await persistAndFsyncDownstreamState(receipt);
const released = await server.commitDatabasePublication({
  databaseName: 'tenant_a',
  resourceFence: receipt,
});

The held barrier survives restart and blocks wire commands, transactions, startup recovery, compaction, index restoration, and close-time hint writes for that database while unrelated databases remain available. Commit is exact and idempotent. A successful commit removes the raw capability from durable state and retains only protected verification material. Provider/root identity, durable fenced-mode markers, and bounded startup validation make copied, missing, corrupt, or legacy-active publication state fail closed. A higher fencing token compacts resolved older receipts, so long-lived coordinators do not exhaust receipt capacity.

Coordinators that do not yet have their own durable record can call getDatabaseResourceFenceState({ databaseName }) before creating it. File-backed SmartDB returns the durable scope, highest token, current publication phase, and allocation lifecycle/identity when the name is allocation-managed, or null only when both fence state and its durable marker are absent. The inspection never returns mutation IDs, mutation payload digests, publication receipts, or publication capabilities. Treat any active phase, identity mismatch, corrupt state, unsupported storage, or unsafe token as a hard stop rather than creating coordinator state.

Authoritative Database Allocation

allocateDatabaseTenant() is the create-only control-plane API for permanently allocation-managed database names. It is available only when allocationFencingVersion === 1, allocationFencingSupported === true, and allocationFencingRequiresDrain === true. The request requires literal expectedAbsent: true, an ISmartDbResourceFence, and the tenant database, username, and password. roles is optional and defaults to ['readWrite', 'dbAdmin']. SmartDB proves database-root and auth-principal absence while holding the database resource lock, maintenance write gate, and a bounded cross-process reservation on the durable auth store, then returns an ISmartDbAllocateDatabaseTenantResult; its .allocation field is the durable ISmartDbDatabaseAllocationIdentity. Exact fence replay returns the same allocation ID, generation, principal ID, and receipt digests.

Pass the returned allocation identity to every later ensureDatabaseTenant(), importDatabase(), and deleteDatabaseTenant() request for that name. Managed import and delete requests also require the exact allocation username; the identity alone is insufficient. Each distinct managed mutation requires a strictly newer fence token; the current token is accepted only for exact receipt replay or continuation, and older receipts become explicitly stale after advancement. Held publication receipts carry the allocation into commitDatabasePublication(). Managed requests without it and unmanaged requests with it fail closed. Deprovision persists deprovisioning, durably deletes the exact marked root, atomically removes only the exact principal, then persists a permanent deprovisioned tombstone. A later allocation requires a newer fence token, increments generation, and receives a fresh allocation and principal identity. Wire data access is available only while active and only to the current allocation principal; database/user ownership-changing wire commands are rejected.

The provider marker .__rustdb_allocation.json is created and preserved by SmartDB, never accepted from an import payload, and never included in logical exports. The resource lock uses the distinct smartdb-resource-allocation-managed-v1 marker. Pre-allocation binaries, including 2.18.1, interpret that lock marker as invalid and fail closed. Consequently, N-1 rollback of a storage root after allocation fencing has touched a name is intentionally unavailable; restore the allocation-aware binary rather than removing or editing provider metadata.

Read-Only Tenant Attestation

attestDatabaseTenant() verifies that an existing database is exclusively owned by the exact username, role set, and candidate password. It returns only { matches: boolean }; it never returns a URI, credential, principal identity, or mismatch detail. Missing databases or users, conflicting ownership, role differences, allocation-principal drift, and password mismatch all return false through the same constant-work credential path.

Attestation always holds read-only local and maintenance guards. File-backed storage independently supplies the cross-process publication guard. A durable usersPath independently supplies a shared auth guard while current durable auth state is read. Memory storage and auth without a usersPath use their configured in-process providers. Attestation does not refresh process caches or call tenant ensure, create, rotate, import, delete, fence repair, publication-epoch observation, or cache invalidation. Corrupt, held, deprovisioned, disabled, or busy provider state fails as an operational error instead of returning a match result.

const result = await server.attestDatabaseTenant({
  databaseName: 'tenant_a',
  username: 'tenant_a_user',
  roles: ['readWrite'],
  password: candidatePassword,
});

if (!result.matches) {
  throw new Error('Existing tenant authority does not match');
}

Allocation-Bound Read Access Grants

Scratch diagnostics can use issueDatabaseAllocationReadAccessGrant() to create one auxiliary read principal for an exact active allocation. The capability is advertised only when databaseAllocationReadAccessGrantVersion === 1 and databaseAllocationReadAccessGrantSupported === true; this requires file storage, enabled authentication, a durable usersPath bound to the storage root, and ready allocation fencing. The caller supplies databaseName, a bounded grantId, a bounded unique username, password, and the complete ISmartDbDatabaseAllocationIdentity. Roles and expiry are not caller-controlled. SmartDB always assigns exactly ['read'] and a fixed, non-renewing 12-minute expiry.

const grant = await server.issueDatabaseAllocationReadAccessGrant({
  databaseName: allocation.databaseName,
  grantId: 'scratch-diagnostic-2026-08-08',
  username: 'scratch_diagnostic_reader',
  password: generatedSecret,
  allocation,
});

const diagnosticClient = new MongoClient(grant.mongodbUri!, {
  directConnection: true,
});
await diagnosticClient.connect();

Exact issue replay validates the candidate password against the persisted SCRAM credential and returns the original principal generation, issuedAt, and expiresAt only while that grant remains active and unexpired; it never renews the grant. Expired or revoked grantId values are rejected. SmartDB permits at most one active read grant per allocation. Revoked and expired grantId values remain in a per-allocation anti-replay set capped at 1,024 entries, with no eviction while the allocation exists. Capacity exhaustion fails closed. Allocation deprovision uses a two-phase fail-closed sequence: any active grant is first durably retired and tombstoned, then the database is durably deleted, and finally the exact owner plus that allocation's replay state are removed in one auth rewrite. Interruption before completion leaves the grant retired rather than usable.

Grant-specific rejections from the issue and revoke methods are normalized automatically as SmartDbAllocationReadAccessGrantError with stable EGRANT_INVALID_REQUEST, EGRANT_CREDENTIAL_MISMATCH, EGRANT_CONFLICT, EGRANT_REPLAY_CAPACITY, or EGRANT_STATE_UNAVAILABLE codes. The companion exports TSmartDbAllocationReadAccessGrantErrorCode, smartDbAllocationReadAccessGrantErrorCodes, and normalizeSmartDbAllocationReadAccessGrantError(error, code) support type-safe handling and explicit normalization of structured bridge errors. Allocation and resource-fence failures retain their existing SmartDbResourceFenceError codes. Callers should classify errors by these exported classes and codes rather than by message text.

Active allocation read grants never survive a SmartDB engine restart or a new attachment to the durable auth store. Startup durably retires and tombstones them before accepting work; diagnostics must issue a new grantId, username, and credential after restart.

Grant principals can read only their exact allocation database. Reads, collection/index metadata, and read-only transactions are supported. Writes, DDL, user/admin operations, writing aggregates ($out and $merge), cross-database targets, and allocation ownership changes are denied. Revocation, expiry cleanup, and allocation deprovision abort matching logical sessions and transactions, release retained database permits, and remove matching cursors. Stale sockets fail their next command.

revokeDatabaseAllocationReadAccessGrant() takes databaseName, grantId, and the complete allocation identity, returns { revoked: true } without a secret, and is idempotent. Revoke-before-issue records anti-replay state only while a read permit proves that exact allocation is current. A delayed revoke after deprovision returns success without recreating auth metadata.

exportDatabase() drains live MongoDB wire commands for the selected database and holds an exclusive local and cross-process database lease while producing the snapshot. The export is therefore consistent across all collections. It enforces server ceilings while visiting documents instead of first materializing an unbounded database; callers can request lower maxEncodedBytes, maxCollections, maxDocuments, and maxIndexes limits. The result emits canonical MongoDB Extended JSON so every BSON type, including 64-bit integers, survives the JSON management channel exactly. importDatabase() accepts canonical or relaxed Extended JSON. Database migration clients should preserve the exported objects as JSON values and must not coerce Extended JSON numeric wrappers into JavaScript numbers.

getDatabaseContentDigest() computes a bounded, database-name-independent SHA-256 over collection names, exact BSON document bytes, and persisted index specifications. Collection, document, and index enumeration order is normalized; BSON field order and compound-index key order remain significant. ISmartDbGetDatabaseContentDigestInput accepts databaseName and optional ISmartDbDatabaseContentDigestLimits fields maxScannedBsonBytes, maxCollections, maxDocuments, and maxIndexes. Callers may lower the server ceilings of 96 MiB scanned BSON, 10,000 collections, 1,000,000 documents, and 100,000 indexes. The ISmartDbDatabaseContentDigest result reports format smartdb.database.content-digest.v1, algorithm sha256, the lowercase digest, and exact scan counters.

SmartdbServer.start() and the lifecycle-critical health, fence-state, digest, export, import, publication-commit, tenant-allocation, tenant-attestation, tenant-ensure, tenant-delete, read-grant issue, and read-grant revoke methods accept optional ISmartDbManagementOperationOptions with signal?: AbortSignal and timeoutMs?: number. timeoutMs, when provided, must be a positive safe integer no greater than 2,147,483,647. Startup applies both cancellation and one deadline across legacy storage migration, sidecar spawn, auth metadata migration, and database readiness. Cancellation and deadlines are fail-stop once the sidecar is owned: SmartDB attempts to terminate the Rust engine before rejecting. Once termination is confirmed, the operation cannot continue after the caller releases ownership. If termination itself fails, the rejection retains bridge ownership and the service owner must retry stop() until it succeeds. After cancellation or deadline termination, restart SmartDB before accepting more traffic.

const controller = new AbortController();
const exported = await server.exportDatabase(
  { databaseName: 'myapp' },
  { signal: controller.signal, timeoutMs: 5 * 60_000 },
);

Basic user management commands are available for authenticated users with root or userAdmin privileges:

await client.db('admin').command({
  createUser: 'reader',
  pwd: 'readpass',
  roles: [{ role: 'read', db: 'myapp' }],
});

await client.db('admin').command({ usersInfo: 'reader' });

Methods & Properties

Method / Property Type Description
start(options?) Promise<void> Start under one optional cancellation/deadline budget; join partial sidecars before rejection or retain ownership for stop() retry if termination fails
stop() Promise<void> Wait for active startup work, then stop the server and confirm Rust bridge cleanup, including partial startup cleanup
getConnectionUri() string Get the active mongodb:// URI; before start and after stop, port 0 remains unresolved
running boolean Whether the server is currently running
port number Actual bound port while running; otherwise the configured port (TCP mode)
host string Configured host (TCP mode)
socketPath string | undefined Socket path (socket mode)
getMetrics() Promise<ISmartDbMetrics> Server metrics (db/collection counts, sessions, transactions, auth, uptime)
getOpLog(params?) Promise<IOpLogResult> Query oplog entries with optional filters
getOpLogStats() Promise<IOpLogStats> Aggregate oplog statistics
revertToSeq(seq, dryRun?) Promise<IRevertResult> Revert to a specific oplog sequence (must be within retained oplog history)
getCollections(db?) Promise<ICollectionInfo[]> List all collections with counts
getDocuments(db, coll, limit?, skip?) Promise<IDocumentsResult> Browse documents with pagination
getHealth(options?) Promise<ISmartDbHealth> Read readiness and explicit publication-hold capability fields with optional fail-stop cancellation/deadline ownership
allocateDatabaseTenant(params, options?) Promise<ISmartDbAllocateDatabaseTenantResult> Authoritatively allocate an absent file-backed tenant and return its durable allocation identity; supports fail-stop cancellation and deadlines
attestDatabaseTenant(params, options?) Promise<ISmartDbAttestDatabaseTenantResult> Verify exact existing tenant ownership, roles, and candidate credential without mutating provider, auth, fence, or runtime cache state
issueDatabaseAllocationReadAccessGrant(params, options?) Promise<ISmartDbIssueDatabaseAllocationReadAccessGrantResult> Issue or exactly replay one fixed 12-minute read grant bound to the complete active allocation identity
revokeDatabaseAllocationReadAccessGrant(params, options?) Promise<ISmartDbRevokeDatabaseAllocationReadAccessGrantResult> Idempotently revoke/tombstone one exact allocation read grant and invalidate its live runtime state
ensureDatabaseTenant(params, options?) Promise<ISmartDbEnsureDatabaseTenantResult> Idempotently ensure an exact fenced tenant; supports fail-stop cancellation and deadlines
getDatabaseResourceFenceState(params, options?) Promise<ISmartDbDatabaseResourceFenceState | null> Safely inspect a file-backed database fence high-water mark and publication phase without exposing receipts or capability material; supports fail-stop cancellation and deadlines
getDatabaseContentDigest(params, options?) Promise<ISmartDbDatabaseContentDigest> Compute a bounded, database-name-independent digest over exact BSON documents and persisted index specifications; supports fail-stop cancellation and deadlines
exportDatabase(params, options?) Promise<ISmartDbDatabaseExport> Export one database as lossless canonical MongoDB Extended JSON; supports fail-stop cancellation and deadlines
importDatabase(params, options?) Promise<ISmartDbImportDatabaseResult> Durably replace one database from canonical or relaxed MongoDB Extended JSON, optionally leaving publication held; supports fail-stop cancellation and deadlines
deleteDatabaseTenant(params, options?) Promise<ISmartDbDeleteDatabaseTenantResult> Durably delete an exact tenant database/user, optionally leaving publication held; supports fail-stop cancellation and deadlines
commitDatabasePublication(params, options?) Promise<TSmartDbCommitDatabasePublicationResult> Idempotently release an exact held publication receipt; supports fail-stop cancellation and deadlines

LocalSmartDb

Zero-config wrapper around SmartdbServer. Uses Unix sockets and file-based persistence.

Constructor Options (ILocalSmartDbOptions)

import { LocalSmartDb } from '@push.rocks/smartdb';

const db = new LocalSmartDb({
  folderPath: './data',                  // Required: data storage directory
  socketPath: '/tmp/custom.sock',        // Optional: custom socket (default: auto-generated)
});

Methods & Properties

Method / Property Type Description
start() Promise<ILocalSmartDbConnectionInfo> Start and return connection info
stop() Promise<void> Stop the server
getConnectionInfo() ILocalSmartDbConnectionInfo Get current connection info
getConnectionUri() string Get the connection URI
getServer() SmartdbServer Access the underlying server
running boolean Whether the server is running
LocalSmartDb.inspectOfflineStringValue(input, options?) Promise<TLocalSmartDbOfflineStringValueInspectionResult> Read one exact top-level string value from stopped file storage without starting or mutating the engine

Offline String-Value Inspection

inspectOfflineStringValue() is a Linux-only operational API for reading one metadata value while the owning LocalSmartDb engine is stopped. Other platforms reject the call because the reader requires Linux openat2 descriptor traversal. It starts only the Rust management sidecar: it does not start a database listener, run storage migrations, repair tails, replay or truncate the WAL, compact data, or persist hints.

import { LocalSmartDb } from '@push.rocks/smartdb';

const result = await LocalSmartDb.inspectOfflineStringValue({
  folderPath: '/var/lib/myapp/smartdb',
  databaseName: 'myapp',
  collectionName: 'MetadataDoc',
  match: {
    field: 'key',
    value: 'migrationStateV1',
  },
  valueField: 'value',
  limits: {
    maximumDataFileBytes: 16 * 1024 * 1024,
    maximumWalFileBytes: 16 * 1024 * 1024,
    maximumRecords: 100_000,
    maximumRecordBytes: 1024 * 1024,
    maximumResultBytes: 1024 * 1024,
  },
}, {
  timeoutMs: 5000,
});

if (result.status === 'found') {
  console.log(result.value);
}

The result is only { status: 'not-found' } or { status: 'found', value: string }. No other document fields cross the management boundary. The reader validates descriptor-safe paths, the current data/WAL format, CRCs and live-record semantics, applies any bounded uncommitted WAL overlay in memory, and rejects ambiguous matches, corruption, exceeded limits, symlinks, an engine-owned storage root, or files that change during inspection.

The TypeScript API enforces these bounds before creating the sidecar or serializing the IPC request:

Input Enforced range
folderPath 1 to 4096 UTF-8 bytes before and after absolute resolution; no control characters
databaseName, collectionName 1 to 255 UTF-8 bytes; one canonical path component; no controls, slash, or backslash
match.field, valueField 1 to 255 UTF-8 bytes; no control characters
match.value 0 to 1,048,576 UTF-8 bytes and no larger than maximumRecordBytes
Complete serialized request At most 2,097,152 bytes
timeoutMs 1 to 2,147,483,647 milliseconds
maximumDataFileBytes 64 to 268,435,456 bytes
maximumWalFileBytes 64 to 67,108,864 bytes
maximumRecords 1 to 1,000,000 data/WAL items
maximumRecordBytes 22 to 17,825,792 bytes
maximumResultBytes 1 to 16,777,216 bytes

All numeric limits must be positive safe integers. The caller must stop and retain external ownership of the LocalSmartDb daemon for the full call. Current storage roots additionally enforce this through SmartDB's storage-owner lock. Roots created before that lock existed rely on the caller's external single-owner coordination plus descriptor and stability validation. One timeoutMs budget covers sidecar lookup/spawn, readiness, and the command. Cancellation or timeout kills and reaps the disposable sidecar before the call rejects.

SmartdbDebugServer

Web-based debug dashboard served via @api.global/typedserver. Import from the debugserver subpath:

import { SmartdbDebugServer } from '@push.rocks/smartdb/debugserver';

const debugServer = new SmartdbDebugServer(server, { port: 4000 });
await debugServer.start();
// Dashboard at http://localhost:4000

await debugServer.stop();

The UI is bundled as base64-encoded content (via @git.zone/tsbundle) and served from memory — no static file directory needed.

SmartdbDebugUi (Web Component)

For embedding the debug UI directly into your own web application, import the <smartdb-debugui> web component:

import { SmartdbDebugUi } from '@push.rocks/smartdb/debugui';

// In your HTML/lit template:
// <smartdb-debugui .server=${mySmartdbServer}></smartdb-debugui>
//
// Or in HTTP mode (when served by SmartdbDebugServer):
// <smartdb-debugui apiBaseUrl=""></smartdb-debugui>

Supported Operations

SmartDB supports the core operations through the wire protocol. Use the standard mongodb driver — these all work:

CRUD

// Insert
await collection.insertOne({ name: 'Bob' });
await collection.insertMany([{ a: 1 }, { a: 2 }]);

// Find
const doc = await collection.findOne({ name: 'Bob' });
const docs = await collection.find({ age: { $gte: 18 } }).toArray();

// Update
await collection.updateOne({ name: 'Bob' }, { $set: { age: 25 } });
await collection.updateMany({ active: false }, { $set: { archived: true } });

// Delete
await collection.deleteOne({ name: 'Bob' });
await collection.deleteMany({ archived: true });

// Replace
await collection.replaceOne({ _id: id }, { name: 'New Bob', age: 30 });

// Find and Modify
await collection.findOneAndUpdate({ name: 'Bob' }, { $inc: { visits: 1 } }, { returnDocument: 'after' });
await collection.findOneAndDelete({ expired: true });
await collection.findOneAndReplace({ _id: id }, { name: 'Replaced' }, { returnDocument: 'after' });

Query Operators

// Comparison
{ age: { $eq: 25 } }     { age: { $ne: 25 } }
{ age: { $gt: 18 } }     { age: { $lt: 65 } }
{ age: { $gte: 18 } }    { age: { $lte: 65 } }
{ status: { $in: ['active', 'pending'] } }
{ status: { $nin: ['deleted'] } }

// Logical
{ $and: [{ age: { $gte: 18 } }, { active: true }] }
{ $or: [{ status: 'active' }, { admin: true }] }
{ $not: { status: 'deleted' } }

// Element
{ email: { $exists: true } }
{ type: { $type: 'string' } }

// Array
{ tags: { $all: ['mongodb', 'database'] } }
{ scores: { $elemMatch: { $gte: 80, $lt: 90 } } }
{ tags: { $size: 3 } }

// Regex
{ name: { $regex: /^Al/i } }

Update Operators

{ $set: { name: 'New Name' } }
{ $unset: { tempField: '' } }
{ $inc: { count: 1 } }
{ $mul: { price: 1.1 } }
{ $min: { low: 50 } }        { $max: { high: 100 } }
{ $push: { tags: 'new' } }   { $pull: { tags: 'old' } }
{ $addToSet: { tags: 'unique' } }
{ $pop: { queue: 1 } }       // Remove last
{ $pop: { queue: -1 } }      // Remove first
{ $rename: { old: 'new' } }
{ $currentDate: { lastModified: true } }

Aggregation Pipeline

const results = await collection.aggregate([
  { $match: { status: 'active' } },
  { $group: { _id: '$category', total: { $sum: '$amount' } } },
  { $sort: { total: -1 } },
  { $limit: 10 },
  { $project: { category: '$_id', total: 1, _id: 0 } },
]).toArray();

Supported stages: $match, $project, $group, $sort, $limit, $skip, $unwind, $lookup, $addFields, $count, $facet, $replaceRoot, $set, $unionWith, $out, $merge

$lookup supports the equality form and a bounded correlated pipeline form with let, an optional $match using $expr/$and/$eq, and an optional following $limit. Correlated string equality predicates can use a full-key or subset foreign equality index to reduce candidates when available. Candidates are paged, the complete expression remains authoritative, and $limit counts only exact expression matches. Other BSON value shapes use the bounded scan path. Mixed localField/foreignField plus pipeline lookups and other inner stages are rejected explicitly. Complete filter trees are validated before data is read, and query nesting plus aggregation work are bounded.

Group accumulators: $sum, $avg, $min, $max, $first, $last, $push, $addToSet, $count

Indexes

await collection.createIndex({ email: 1 }, { unique: true });
await collection.createIndex({ name: 1, age: -1 });    // compound
await collection.createIndex({ field: 1 }, { sparse: true });
const indexes = await collection.listIndexes().toArray();
await collection.dropIndex('email_1');
await collection.dropIndexes();  // drop all except _id

🛡️ Unique indexes are enforced at the engine level. Duplicate values are rejected with a DuplicateKey error (code 11000) before the document is written to disk — on insertOne, updateOne, findAndModify, and upserts. Index definitions are persisted to indexes.json and automatically restored on restart.

SmartDB supports ascending and descending index keys with name, unique, sparse, and expireAfterSeconds options. It accepts background as a validated no-op and index version v: 2. Unsupported options, including partialFilterExpression, collation, and hidden, are rejected before any index in the request is created. TTL metadata is retained in the catalog; automatic TTL expiry is not implemented.

Database & Admin

await db.listCollections().toArray();
await db.createCollection('new');
await db.dropCollection('old');
await db.dropDatabase();
await db.stats();

const admin = client.db().admin();
await admin.listDatabases();
await admin.ping();
await admin.serverStatus();

SmartDB creates ordinary collections only. Unsupported collection semantics, including views, capped collections, validators, collation, time-series collections, clustered indexes, and encrypted fields, are rejected before the database or namespace is created.

Bulk Operations

const result = await collection.bulkWrite([
  { insertOne: { document: { name: 'Bulk1' } } },
  { updateOne: { filter: { name: 'X' }, update: { $set: { bulk: true } } } },
  { deleteOne: { filter: { name: 'Expired' } } },
]);

Count & Distinct

const count = await collection.countDocuments({ status: 'active' });
const estimated = await collection.estimatedDocumentCount();
const names = await collection.distinct('name');

Wire Protocol Commands

Category Commands
Handshake hello, isMaster, ismaster
CRUD find, insert, update, delete, findAndModify, getMore, killCursors
Aggregation aggregate, count, distinct
Indexes createIndexes, dropIndexes, listIndexes
Sessions startSession, endSessions
Transactions startTransaction, commitTransaction, abortTransaction through driver sessions
Admin ping, listDatabases, listCollections, drop, dropDatabase, create, serverStatus, buildInfo, dbStats, collStats, connectionStatus, currentOp, renameCollection

Advertises MongoDB wire protocol versions 0–21. The documented command surface is tested with the official mongodb Node.js driver version 7.5.x.

Rust Crate Architecture 🦀

The Rust engine is organized as a Cargo workspace with 9 focused crates:

Crate Purpose
rustdb Binary entry point: TCP/Unix listener, management IPC, CLI
rustdb-config Server configuration types (serde, camelCase JSON)
rustdb-wire Wire protocol parser/encoder (OP_MSG, OP_QUERY, OP_REPLY)
rustdb-query Query matcher, update engine, aggregation, sort, projection
rustdb-storage Storage backends (memory, file), OpLog with point-in-time replay
rustdb-index B-tree/hash indexes, query planner (IXSCAN/COLLSCAN)
rustdb-txn Transaction + session management with snapshot isolation
rustdb-auth SCRAM-SHA-256 credential handling, user metadata persistence, RBAC checks
rustdb-commands 40+ command handlers wiring everything together

Cross-compiled for linux_amd64 and linux_arm64 via @git.zone/tsrust.

Storage Engine Reliability 🔒

The Bitcask-style file storage engine includes several reliability features:

  • Write-ahead log (WAL) — every write is logged before being applied, with crash recovery on restart; once committed entries grow past 16 MiB the WAL is checkpointed and truncated at runtime, so wal.rdb does not grow unbounded
  • CRC32 checksums — every record is integrity-checked on read
  • Automatic compaction — dead records are reclaimed when they exceed 50% of file size, runs on startup and after every write
  • Hint file staleness detection — the hint file records the data file size at write time; if data.rdb changed since (e.g. crash after a delete), the engine falls back to a full scan to ensure tombstones are not lost
  • Torn-tail repair — startup scans data.rdb to the last valid record, truncates invalid trailing bytes, and preserves all verified records after interrupted writes
  • Stale socket cleanup — orphaned /tmp/smartdb-*.sock files from crashed instances are automatically cleaned up on startup

Data Integrity CLI 🔍

The Rust binary includes an offline integrity checker:

# Check all collections in a data directory
./dist_rust/rustdb_linux_amd64 --validate-data /path/to/data

# Output:
# === SmartDB Data Integrity Report ===
#
# Database: mydb
#   Collection: users
#     Header:       OK
#     Records:      1,234 (1,200 live, 34 tombstones)
#     Data size:    2.1 MB
#     Duplicates:   0
#     CRC errors:   0
#     Hint file:    OK

Checks file headers, record CRC32 checksums, duplicate _id entries, and hint file consistency. Exit code 1 if any errors are found.

Testing Example

import { expect, tap } from '@git.zone/tstest/tapbundle';
import { SmartdbServer } from '@push.rocks/smartdb';
import { MongoClient } from 'mongodb';

let server: SmartdbServer;
let client: MongoClient;

tap.test('setup', async () => {
  server = new SmartdbServer({ port: 27117 });
  await server.start();
  client = new MongoClient('mongodb://127.0.0.1:27117', { directConnection: true });
  await client.connect();
});

tap.test('should insert and find', async () => {
  const col = client.db('test').collection('items');
  await col.insertOne({ name: 'Widget', price: 9.99 });
  const item = await col.findOne({ name: 'Widget' });
  expect(item?.price).toEqual(9.99);
});

tap.test('should track changes in oplog', async () => {
  const oplog = await server.getOpLog();
  expect(oplog.entries.length).toBeGreaterThan(0);
  expect(oplog.entries[0].op).toEqual('insert');
});

tap.test('teardown', async () => {
  await client.close();
  await server.stop();
});

export default tap.start();

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

Company Information

Task Venture Capital GmbH
Registered at District Court Bremen HRB 35230 HB, Germany

For any legal inquiries or further information, please contact us via email at hello@task.vc.

By using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.

Reference in New Issue View Git Blame Copy Permalink
S
Description
A MongoDB-compatible embedded database server powered by Rust 🦀⚡
rusttypescript
Readme
5 MiB
Languages
Rust 82.1%
TypeScript 17.9%