push.rocks/smartdb
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(readme): document index enforcement, storage reliability, and data integrity validation features

Browse Source
This commit is contained in:
Jürgen Kunz
2026-04-05 12:38:46 +00:00
parent 02ad9a29a7
commit d37b444dd5
3 changed files with 62 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
changelog.md
View File

@@ -1,5 +1,12 @@
# Changelog
## 2026-04-05 - 2.6.0 - feat(readme)
document index enforcement, storage reliability, and data integrity validation features
- Add documentation for engine-level unique index enforcement and duplicate key behavior
- Describe storage engine reliability features including WAL, CRC32 checks, compaction, hint file staleness detection, and stale socket cleanup
- Add usage documentation for the offline data integrity validation CLI
## 2026-04-05 - 2.5.9 - fix(rustdb-storage)
run collection compaction during file storage initialization after crashes

73
readme.md
View File

@@ -44,38 +44,38 @@ SmartDB uses a **sidecar binary** pattern — TypeScript handles lifecycle, Rust
```
┌──────────────────────────────────────────────────────────────┐
│ Your Application │
│ (TypeScript / Node.js) │
│ Your Application │
│ (TypeScript / Node.js) │
│ ┌─────────────────┐ ┌───────────────────────────┐ │
│ │ SmartdbServer │────▶│ RustDbBridge (IPC) │ │
│ │ or LocalSmartDb │ │ @push.rocks/smartrust │ │
│ │ SmartdbServer │────▶│ RustDbBridge (IPC) │ │
│ │ or LocalSmartDb │ │ @push.rocks/smartrust │ │
│ └─────────────────┘ └───────────┬───────────────┘ │
└──────────────────────────────────────┼───────────────────────┘
│ spawn + JSON IPC
▼
┌──────────────────────────────────────────────────────────────┐
│ rustdb binary 🦀 │
│ rustdb binary │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ Wire Protocol│→ │Command Router│→ │ Handlers │ │
│ │ (OP_MSG) │ │ (40+ cmds) │ │ Find,Insert.. │ │
│ └──────────────┘ └──────────────┘ └───────┬───────┘ │
│ ┌──────────────┐ ┌──────────────┐ ┌───────────────┐ │
│ │ Wire Protocol│→ │Command Router│→ │ Handlers │ │
│ │ (OP_MSG) │ │ (40+ cmds) │ │ Find,Insert.. │ │
│ └──────────────┘ └──────────────┘ └───────┬───────┘ │
│ │ │
│ ┌─────────┐ ┌────────┐ ┌───────────┐ ┌──────┴──────┐ │
│ │ Query │ │ Update │ │Aggregation│ │ Index │ │
│ │ Matcher │ │ Engine │ │ Engine │ │ Engine │ │
│ └─────────┘ └────────┘ └───────────┘ └─────────────┘ │
│ ┌─────────┐ ┌────────┐ ┌───────────┐ ┌──────┴──────┐ │
│ │ Query │ │ Update │ │Aggregation│ │ Index │ │
│ │ Matcher │ │ Engine │ │ Engine │ │ Engine │ │
│ └─────────┘ └────────┘ └───────────┘ └─────────────┘ │
│ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────┐ │
│ │ MemoryStorage │ │ FileStorage │ │ OpLog │ │
│ └──────────────────┘ └──────────────────┘ └──────────┘ │
│ ┌──────────────────┐ ┌──────────────────┐ ┌──────────┐ │
│ │ MemoryStorage │ │ FileStorage │ │ OpLog │ │
│ └──────────────────┘ └──────────────────┘ └──────────┘ │
└──────────────────────────────────────────────────────────────┘
▲
│ TCP / Unix Socket (wire protocol)
│
┌─────────────┴────────────────────────────────────────────────┐
│ MongoClient (mongodb npm driver) │
│ Connects directly to Rust binary │
│ MongoClient (mongodb npm driver) │
│ Connects directly to Rust binary │
└──────────────────────────────────────────────────────────────┘
```
@@ -429,6 +429,8 @@ await collection.dropIndex('email_1');
await collection.dropIndexes(); // drop all except _id
```
> 🛡️ **Unique indexes are enforced at the engine level.** Duplicate values are rejected with a `DuplicateKey` error (code 11000) *before* the document is written to disk — on `insertOne`, `updateOne`, `findAndModify`, and upserts. Index definitions are persisted to `indexes.json` and automatically restored on restart.
### Database & Admin
```typescript
@@ -497,6 +499,39 @@ The Rust engine is organized as a Cargo workspace with 8 focused crates:
Cross-compiled for `linux_amd64` and `linux_arm64` via [@git.zone/tsrust](https://www.npmjs.com/package/@git.zone/tsrust).
### Storage Engine Reliability 🔒
The Bitcask-style file storage engine includes several reliability features:
- **Write-ahead log (WAL)** — every write is logged before being applied, with crash recovery on restart
- **CRC32 checksums** — every record is integrity-checked on read
- **Automatic compaction** — dead records are reclaimed when they exceed 50% of file size, runs on startup and after every write
- **Hint file staleness detection** — the hint file records the data file size at write time; if data.rdb changed since (e.g. crash after a delete), the engine falls back to a full scan to ensure tombstones are not lost
- **Stale socket cleanup** — orphaned `/tmp/smartdb-*.sock` files from crashed instances are automatically cleaned up on startup
### Data Integrity CLI 🔍
The Rust binary includes an offline integrity checker:
```bash
# Check all collections in a data directory
./dist_rust/rustdb_linux_amd64 --validate-data /path/to/data
# Output:
# === SmartDB Data Integrity Report ===
#
# Database: mydb
# Collection: users
# Header: OK
# Records: 1,234 (1,200 live, 34 tombstones)
# Data size: 2.1 MB
# Duplicates: 0
# CRC errors: 0
# Hint file: OK
```
Checks file headers, record CRC32 checksums, duplicate `_id` entries, and hint file consistency. Exit code 1 if any errors are found.
---
## Testing Example
@@ -541,7 +576,7 @@ export default tap.start();
## License and Legal Information
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [LICENSE](./LICENSE) file.
This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the [license](./license) file.
**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

2
ts/00_commitinfo_data.ts
View File

@@ -3,6 +3,6 @@
*/
export const commitinfo = {
name: '@push.rocks/smartdb',
version: '2.5.9',
version: '2.6.0',
description: 'A MongoDB-compatible embedded database server with wire protocol support, backed by a high-performance Rust engine.'
}