push.rocks/smartmigration
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(migration): add lock heartbeats, predictive dry-run planning, and stricter ledger option validation

Browse Source
This commit is contained in:
Jürgen Kunz
2026-04-14 12:31:34 +00:00
parent 19ebdee31a
commit 1b4358aca5
17 changed files with 695 additions and 180 deletions
Download Patch File Download Diff File Expand all files Collapse all files
readme.hints.md
+12 -6
View File
@@ -4,20 +4,26 @@ Internal development notes for `@push.rocks/smartmigration`. This file is not pu
## Architecture
- The ledger is the source of truth. Two backends: mongo (via smartdata `EasyStore`) and S3 (via `bucket.fastPut`/`fastGet`).
- The ledger is the source of truth. Two backends: mongo (a single document in smartdata's `SmartdataEasyStore` collection) and S3 (via `bucket.fastPut`/`fastGet`).
- Step execution is **sequential and registration-ordered**. The `from`/`to` fields exist for chain validation, not for DAG resolution.
- Drivers (`mongodb.Db`, `S3Client`) are exposed via `ctx`, not wrapped. smartmigration writes no SQL or S3 wrappers of its own.
## Locking caveats
- The mongo lock uses a read-modify-write pattern over `EasyStore` (which writes the entire document each time). This is a last-writer-wins CAS, not a true atomic CAS. It's adequate for v1; v2 may move to `findOneAndUpdate` against the underlying mongo collection.
- The mongo lock uses atomic `findOneAndUpdate` / `updateOne` operations against the `SmartdataEasyStore` collection and a background heartbeat that extends `lock.expiresAt` while steps are running.
- The S3 lock is best-effort (S3 has no conditional writes without versioning). S3-only deployments should not run multiple instances against the same ledger without external coordination.
## Why EasyStore (and not a custom collection)?
## Why reuse SmartdataEasyStore?
- Already exists, lazy collection setup, single-document storage that fits the entire ledger naturally.
- Avoids polluting the user's DB with smartmigration-internal classes.
- See `/mnt/data/lossless/push.rocks/smartdata/ts/classes.easystore.ts` for the implementation.
- The collection already exists in the smartdata ecosystem and is a natural fit for singleton blobs keyed by `nameId`.
- Reusing it avoids introducing a second smartmigration-specific collection just to store one document per ledger.
- The lock path talks directly to the collection for atomic updates; the persisted document shape still mirrors EasyStore's `{ nameId, data }` layout.
- See `/mnt/data/lossless/push.rocks/smartdata/ts/classes.easystore.ts` for the reference document shape.
## Runtime notes
- `plan()` / `dryRun` resolve the same fresh-install shortcut as `run()`, so their `currentVersionAfter` value is predictive rather than just echoing the current ledger version.
- Successful resumable steps clear `checkpoints[stepId]`; failed steps leave checkpoints intact so the next run can resume.
## Testing notes