feat(readme): expand README with storage integrity, WAL, query planner, session & transaction docs; update test script to enable verbose logging and increase timeout

2026-02-01 16:05:00 +00:00
parent 6932059965
commit 316af45b5e
4 changed files with 151 additions and 19 deletions
--- a/changelog.md
+++ b/changelog.md
@@ -1,5 +1,14 @@
 # Changelog

+## 2026-02-01 - 4.1.0 - feat(readme)
+expand README with storage integrity, WAL, query planner, session & transaction docs; update test script to enable verbose logging and increase timeout
+
+- Updated npm test script to run tstest with --verbose, --logfile and --timeout 60 to improve test output and avoid timeouts.
+- Extensive README additions: file storage adapter examples with checksum options, write-ahead logging (WAL) usage and recovery, query planner examples, index and query execution details, session and transaction examples and features.
+- Wire protocol / features table updated to include Transactions and Sessions and added admin commands (dbStats, collStats).
+- Architecture diagram and component list updated to include QueryPlanner, SessionEngine, TransactionEngine and WAL; storage layer annotated with checksums and WAL.
+- Minor example import tweak: MongoClient import now includes Db type in test examples.
+
 ## 2026-02-01 - 4.0.0 - BREAKING CHANGE(storage,engine,server)
 add session & transaction management, index/query planner, WAL and checksum support; integrate index-accelerated queries and update storage API (findByIds) to enable index optimizations

--- a/package.json
+++ b/package.json
@@ -9,7 +9,7 @@
  "author": "Lossless GmbH",
  "license": "MIT",
  "scripts": {
-    "test": "(tstest test/)",
+    "test": "(tstest test/. --verbose --logfile --timeout 60)",
    "build": "(tsbuild --web)",
    "buildDocs": "tsdoc"
  },
--- a/readme.md
+++ b/readme.md
@@ -134,8 +134,8 @@ await server.start();
 console.log(server.getConnectionUri()); // mongodb://127.0.0.1:27017

 // Server properties
-console.log(server.running);           // true
-console.log(server.getUptime());       // seconds
+console.log(server.running);              // true
+console.log(server.getUptime());          // seconds
 console.log(server.getConnectionCount()); // active connections

 await server.stop();
@@ -279,7 +279,7 @@ console.log(result.deletedCount);   // 1

 ### Storage Adapters

-TsmDB supports pluggable storage:
+TsmDB supports pluggable storage with data integrity features:

 ```typescript
 // In-memory (default) - fast, data lost on stop
@@ -292,13 +292,118 @@ const server = new tsmdb.TsmdbServer({
  persistIntervalMs: 30000  // Save every 30 seconds
 });

-// File-based - persistent storage
-const server = new tsmdb.TsmdbServer({
-  storage: 'file',
-  storagePath: './data/tsmdb'
+// File-based - persistent storage with optional checksums
+import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';
+
+const adapter = new FileStorageAdapter('./data/tsmdb', {
+  enableChecksums: true,   // CRC32 checksums for data integrity
+  strictChecksums: false   // Log warnings vs throw on mismatch
 });
 ```

+## ⚡ Performance & Reliability Features
+
+TsmDB includes enterprise-grade features for robustness:
+
+### 🔍 Index-Accelerated Queries
+
+Indexes are automatically used to accelerate queries. Instead of scanning all documents, TsmDB uses:
+
+- **Hash indexes** for equality queries (`$eq`, `$in`)
+- **B-tree indexes** for range queries (`$gt`, `$gte`, `$lt`, `$lte`)
+
+```typescript
+// Create an index
+await collection.createIndex({ email: 1 });
+await collection.createIndex({ age: 1 });
+
+// These queries will use the index (fast!)
+await collection.findOne({ email: 'alice@example.com' }); // Uses hash lookup
+await collection.find({ age: { $gte: 18, $lt: 65 } });    // Uses B-tree range scan
+```
+
+### 📊 Query Planner
+
+TsmDB includes a query planner that analyzes queries and selects optimal execution strategies:
+
+```typescript
+import { tsmdb } from '@push.rocks/smartmongo';
+
+// For debugging, you can access the query planner
+const planner = new tsmdb.QueryPlanner(indexEngine);
+const plan = planner.createPlan(filter);
+
+console.log(plan);
+// {
+//   type: 'IXSCAN',           // or 'IXSCAN_RANGE', 'COLLSCAN'
+//   indexName: 'email_1',
+//   estimatedCost: 1,
+//   selectivity: 0.001
+// }
+```
+
+### 📝 Write-Ahead Logging (WAL)
+
+For durability, TsmDB supports write-ahead logging:
+
+```typescript
+import { tsmdb } from '@push.rocks/smartmongo';
+
+const wal = new tsmdb.WAL('./data/wal.log');
+await wal.initialize();
+
+// WAL entries include:
+// - LSN (Log Sequence Number)
+// - Timestamp
+// - Operation type (insert, update, delete, checkpoint)
+// - Document data (BSON serialized)
+// - CRC32 checksum
+
+// Recovery support
+const entries = await wal.getEntriesAfter(lastCheckpointLsn);
+```
+
+### 🔐 Session Management
+
+TsmDB tracks client sessions with automatic timeout and transaction linking:
+
+```typescript
+// Sessions are automatically managed when using the MongoDB driver
+const session = client.startSession();
+
+try {
+  session.startTransaction();
+  await collection.insertOne({ name: 'Alice' }, { session });
+  await collection.updateOne({ name: 'Bob' }, { $inc: { balance: 100 } }, { session });
+  await session.commitTransaction();
+} catch (error) {
+  await session.abortTransaction();
+} finally {
+  session.endSession();
+}
+
+// Session features:
+// - Automatic session timeout (30 minutes default)
+// - Transaction auto-abort on session expiry
+// - Session activity tracking
+```
+
+### ✅ Data Integrity Checksums
+
+File-based storage supports CRC32 checksums to detect corruption:
+
+```typescript
+import { FileStorageAdapter } from '@push.rocks/smartmongo/tsmdb';
+
+const adapter = new FileStorageAdapter('./data', {
+  enableChecksums: true,
+  strictChecksums: true  // Throw error on corruption (vs warning)
+});
+
+// Documents are checksummed on write, verified on read
+// Checksums are automatically stripped before returning to client
+```
+
 ### 📋 Supported Wire Protocol Commands

 | Category | Commands |
@@ -307,7 +412,9 @@ const server = new tsmdb.TsmdbServer({
 | **CRUD** | `find`, `insert`, `update`, `delete`, `findAndModify`, `getMore`, `killCursors` |
 | **Aggregation** | `aggregate`, `count`, `distinct` |
 | **Indexes** | `createIndexes`, `dropIndexes`, `listIndexes` |
-| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo` |
+| **Transactions** | `startTransaction`, `commitTransaction`, `abortTransaction` |
+| **Sessions** | `startSession`, `endSessions` |
+| **Admin** | `ping`, `listDatabases`, `listCollections`, `drop`, `dropDatabase`, `create`, `serverStatus`, `buildInfo`, `dbStats`, `collStats` |

 TsmDB supports MongoDB wire protocol versions 0-21, compatible with MongoDB 3.6 through 7.0 drivers.

@@ -317,7 +424,7 @@ TsmDB supports MongoDB wire protocol versions 0-21, compatible with MongoDB 3.6

 ```typescript
 import { tsmdb } from '@push.rocks/smartmongo';
-import { MongoClient } from 'mongodb';
+import { MongoClient, Db } from 'mongodb';

 let server: tsmdb.TsmdbServer;
 let client: MongoClient;
@@ -421,21 +528,37 @@ export default tap.start();
                          ▼
 ┌─────────────────────────────────────────────────────────────┐
 │                       Engines                               │
-│  ┌───────────┐ ┌────────────┐ ┌────────────┐ ┌───────────┐ │
-│  │  Query    │ │   Update   │ │Aggregation │ │   Index   │ │
-│  │  Engine   │ │   Engine   │ │  Engine    │ │  Engine   │ │
-│  └───────────┘ └────────────┘ └────────────┘ └───────────┘ │
+│  ┌─────────┐ ┌────────┐ ┌───────────┐ ┌───────┐ ┌───────┐ │
+│  │ Query   │ │ Update │ │Aggregation│ │ Index │ │Session│ │
+│  │ Planner │ │ Engine │ │  Engine   │ │Engine │ │Engine │ │
+│  └─────────┘ └────────┘ └───────────┘ └───────┘ └───────┘ │
+│                   ┌──────────────────────┐                  │
+│                   │ Transaction Engine   │                  │
+│                   └──────────────────────┘                  │
 └─────────────────────────┬───────────────────────────────────┘
                          │
                          ▼
 ┌─────────────────────────────────────────────────────────────┐
-│                   Storage Adapters                          │
-│       ┌──────────────────┐  ┌──────────────────┐           │
-│       │  MemoryStorage   │  │   FileStorage    │           │
-│       └──────────────────┘  └──────────────────┘           │
+│                   Storage Layer                              │
+│  ┌──────────────────┐  ┌──────────────────┐  ┌──────────┐  │
+│  │  MemoryStorage   │  │   FileStorage    │  │   WAL    │  │
+│  │                  │  │  (+ Checksums)   │  │          │  │
+│  └──────────────────┘  └──────────────────┘  └──────────┘  │
 └─────────────────────────────────────────────────────────────┘
 ```

+### Key Components
+
+| Component | Description |
+|-----------|-------------|
+| **WireProtocol** | Parses MongoDB OP_MSG binary protocol |
+| **CommandRouter** | Routes commands to appropriate handlers |
+| **QueryPlanner** | Analyzes queries and selects execution strategy |
+| **IndexEngine** | Manages B-tree and hash indexes |
+| **SessionEngine** | Tracks client sessions and timeouts |
+| **TransactionEngine** | Handles ACID transaction semantics |
+| **WAL** | Write-ahead logging for durability |
+
 ## License and Legal Information

 This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.
--- a/ts/00_commitinfo_data.ts
+++ b/ts/00_commitinfo_data.ts
@@ -3,6 +3,6 @@
 */
 export const commitinfo = {
  name: '@push.rocks/smartmongo',
-  version: '4.0.0',
+  version: '4.1.0',
  description: 'A module for creating and managing a local MongoDB instance for testing purposes.'
 }