smartmigration/readme.plan.md

@push.rocks/smartmigration — Plan

reread /home/philkunz/.claude/CLAUDE.md before working from this plan

Context

@push.rocks/smartmigration is now implemented. Its purpose is to unify migrations across MongoDB and S3 with a small, builder-style API designed to be invoked on SaaS app startup: it inspects the current data version, computes the sequential chain of steps required to reach the app's target version, executes them safely, and stamps progress into a ledger so re-runs are no-ops.

Why this exists. Across the push.rocks ecosystem (smartdata, smartbucket, smartdb, mongodump, smartversion) there was no migration tooling at the time this plan was written. SaaS apps that ship multiple deploys per week need a deterministic way to evolve persistent state in lockstep with code, and they need it to "just work" when the app boots — not as a separate operator-driven process. Both smartdata and smartbucket already expose their underlying drivers (SmartdataDb.mongoDb / SmartdataDb.mongoDbClient, SmartBucket.storageClient), so smartmigration only needs to provide the runner, ledger, and context plumbing — it does not need to wrap mongo or S3 itself.

Intended outcome. A SaaS app can do this at startup and forget about it:

import { SmartMigration } from '@push.rocks/smartmigration';
import { commitinfo } from './00_commitinfo_data.js';

const migration = new SmartMigration({
  targetVersion: commitinfo.version, // app version drives data version
  db,                                 // optional SmartdataDb
  bucket,                             // optional SmartBucket Bucket
});

migration
  .step('lowercase-emails')
    .from('1.0.0').to('1.1.0')
    .description('Lowercase all user emails')
    .up(async (ctx) => {
      await ctx.mongo.collection('users').updateMany(
        {},
        [{ $set: { email: { $toLower: '$email' } } }],
      );
    })
  .step('reorganize-uploads')
    .from('1.1.0').to('2.0.0')
    .description('Move uploads/ to media/')
    .resumable()
    .up(async (ctx) => {
      const cursor = ctx.bucket.createCursor('uploads/');
      let token = await ctx.checkpoint.read<string>('cursorToken');
      if (token) cursor.setToken(token);
      while (await cursor.hasMore()) {
        for (const key of await cursor.next()) {
          await ctx.bucket.fastMove({
            sourcePath: key,
            destinationPath: 'media/' + key.slice('uploads/'.length),
          });
        }
        await ctx.checkpoint.write('cursorToken', cursor.getToken());
      }
    });

await migration.run(); // fast no-op if already at target

---

Confirmed design decisions

These were chosen during planning and are locked in:

• Step ordering: registration order, with from/to validation. Steps execute in the order they were registered. The runner verifies each step's from matches the previous step's to (or the current ledger version) and errors out on gaps/overlaps. No DAG / topological sort.
• Rollback: up-only for v1. No .down() API. Forward-only migrations are simpler and safer; users restore from backup if a migration goes wrong. May be revisited in v2.

---

Design at a glance

Core principles

1. One unified data version. A single semver string represents the combined state of mongo + S3. Steps transition from → to. (Tracking mongo and S3 versions independently is rejected because it explodes step typing for marginal value — the app's data version is what users actually reason about.)
2. Builder-style step definition, single options object for the runner. The constructor takes a plain options object (new SmartMigration({...})); steps are added via a fluent chain (migration.step('id').from('1.0.0').to('1.1.0').up(async ctx => {...})). The chain returns the parent SmartMigration after .up() so multiple steps chain naturally.
3. Drivers are exposed via context, not wrapped. Migration up functions receive a MigrationContext that hands them both high-level (ctx.db, ctx.bucket) and raw (ctx.mongo, ctx.s3) handles. smartmigration writes no SQL or S3 wrappers of its own.
4. Idempotent, restartable, lockable. Re-running migration.run() on already-applied data is a no-op. Steps marked .resumable() get a per-step checkpoint store. A mongo-backed lock prevents concurrent SaaS instances from racing on the same migration.
5. Fast on the happy path. When currentVersion === targetVersion, run() performs one read against the ledger and returns. No driver calls beyond that.
6. Order is registration order; from/to is for validation. Steps execute in the order they were registered. The runner verifies that each step's from matches the previous step's to (or the current ledger version) and errors out on gaps/overlaps. This avoids the complexity of computing a DAG path while still catching mistakes.

Public API surface

// ts/index.ts
export { SmartMigration } from './classes.smartmigration.js';
export type {
  ISmartMigrationOptions,
  IMigrationStepDefinition,
  IMigrationContext,
  IMigrationCheckpoint,
  IMigrationRunResult,
  IMigrationStepResult,
  IMigrationLedgerEntry,
} from './interfaces.js';
export type { TMigrationStatus, TLedgerBackend } from './types.js';
export { SmartMigrationError } from './classes.smartmigration.js';

ISmartMigrationOptions

export interface ISmartMigrationOptions {
  /** Target version for the data. Typically the app's package.json version. */
  targetVersion: string;

  /** Optional smartdata instance. Required if any step uses ctx.db / ctx.mongo. */
  db?: plugins.smartdata.SmartdataDb;

  /** Optional smartbucket Bucket. Required if any step uses ctx.bucket / ctx.s3. */
  bucket?: plugins.smartbucket.Bucket;

  /** Logical name for this migration ledger. Defaults to "smartmigration". */
  ledgerName?: string;

  /** Where to persist the ledger. Defaults to "mongo" if db provided, otherwise "s3". */
  ledgerBackend?: TLedgerBackend; // 'mongo' | 's3'

  /**
   * For a fresh install (no ledger AND no app data), jump straight to this version
   * instead of running every step from the earliest. Defaults to undefined,
   * which means "run every step from earliest from-version".
   */
  freshInstallVersion?: string;

  /** How long (ms) to wait for a stale lock from another instance. Default 60_000. */
  lockWaitMs?: number;

  /** How long (ms) before this instance's own lock auto-expires. Default 600_000. */
  lockTtlMs?: number;

  /** If true, run() returns the plan without executing anything. Default false. */
  dryRun?: boolean;

  /** Custom logger. Defaults to module logger. */
  logger?: plugins.smartlog.Smartlog;
}

IMigrationContext

export interface IMigrationContext {
  // High-level
  db?: plugins.smartdata.SmartdataDb;
  bucket?: plugins.smartbucket.Bucket;

  // Raw drivers
  mongo?: plugins.mongodb.Db;     // db.mongoDb
  s3?: plugins.awsSdk.S3Client;   // bucket.parentSmartBucket.storageClient

  // Step metadata
  step: {
    id: string;
    fromVersion: string;
    toVersion: string;
    description?: string;
    isResumable: boolean;
  };

  // Convenience
  log: plugins.smartlog.Smartlog;
  isDryRun: boolean;

  /** Only present when step.isResumable === true. */
  checkpoint?: IMigrationCheckpoint;

  /** Convenience for transactional mongo migrations. Throws if no db configured. */
  startSession(): plugins.mongodb.ClientSession;
}

export interface IMigrationCheckpoint {
  read<T = unknown>(key: string): Promise<T | undefined>;
  write<T = unknown>(key: string, value: T): Promise<void>;
  clear(): Promise<void>;
}

MigrationStepBuilder (chained, terminal .up() returns parent SmartMigration)

class MigrationStepBuilder {
  from(version: string): this;
  to(version: string): this;
  description(text: string): this;
  resumable(): this;                                    // enables ctx.checkpoint
  up(handler: (ctx: IMigrationContext) => Promise<void>): SmartMigration;
}

SmartMigration

class SmartMigration {
  public settings: ISmartMigrationOptions;

  constructor(options: ISmartMigrationOptions);

  /** Begin defining a step. Returns a chainable builder. */
  step(id: string): MigrationStepBuilder;

  /**
   * The startup entry point.
   *  1. Acquires the migration lock
   *  2. Reads current ledger version (treats null as fresh install)
   *  3. Validates the chain of registered steps
   *  4. Computes the plan (which steps to run)
   *  5. Executes them sequentially, checkpointing each
   *  6. Releases the lock
   * Returns a result describing what was applied/skipped.
   */
  run(): Promise<IMigrationRunResult>;

  /** Returns the plan without executing. Useful for `--dry-run` style probes. */
  plan(): Promise<IMigrationRunResult>;

  /** Returns the current data version from the ledger, or null if uninitialised. */
  getCurrentVersion(): Promise<string | null>;
}

IMigrationRunResult

export interface IMigrationRunResult {
  currentVersionBefore: string | null;
  currentVersionAfter: string;
  targetVersion: string;
  wasUpToDate: boolean;
  wasFreshInstall: boolean;
  stepsApplied: IMigrationStepResult[];
  stepsSkipped: IMigrationStepResult[];   // populated only on dry-run / when out of range
  totalDurationMs: number;
}

export interface IMigrationStepResult {
  id: string;
  fromVersion: string;
  toVersion: string;
  status: TMigrationStatus; // 'applied' | 'skipped' | 'failed'
  durationMs: number;
  startedAt: string;        // ISO
  finishedAt: string;       // ISO
  error?: { message: string; stack?: string };
}

---

Ledger model

The ledger is the source of truth. There are two backends:

Mongo ledger (default when db is provided)

Backed by an EasyStore<TSmartMigrationLedgerData> (smartdata's existing EasyStore, see /mnt/data/lossless/push.rocks/smartdata/ts/classes.easystore.ts:35-101). The nameId is smartmigration:<ledgerName>. Schema of the stored data:

interface ISmartMigrationLedgerData {
  currentVersion: string | null;
  steps: Record<string, IMigrationLedgerEntry>; // keyed by step id
  lock: {
    holder: string | null;     // random instance UUID
    acquiredAt: string | null; // ISO
    expiresAt: string | null;  // ISO
  };
  checkpoints: Record<string, Record<string, unknown>>; // stepId -> { key: value }
}

Why EasyStore over a custom collection? EasyStore already exists, is designed for exactly this kind of singleton-config-blob use case, handles its own collection setup lazily, and avoids polluting the user's DB with smartmigration-internal classes. The whole ledger fits in one document, which makes the lock CAS trivial.

Locking implementation. Acquire by calling easyStore.readAll(), checking lock.holder === null || lock.expiresAt < now, then easyStore.writeAll({ lock: { holder: instanceId, ... } }). Re-read after writing to confirm we won the race (last-writer-wins is fine here because we re-check). Loop with backoff until lockWaitMs elapsed. This is admittedly not a true CAS — for v1 it's adequate; v2 can move to findOneAndUpdate against the underlying mongo collection if races become a problem.

S3 ledger (default when only bucket is provided)

A single object at <bucket>/.smartmigration/<ledgerName>.json containing the same ISmartMigrationLedgerData shape. Reads use bucket.fastGet, writes use bucket.fastPut with overwrite: true. Locking is best-effort (S3 has no CAS without conditional writes); we set lock.expiresAt and re-read to detect races. Documented limitation: S3-only deployments should not run multiple SaaS instances against the same ledger simultaneously without external coordination — when both mongo and S3 are present (the common SaaS case), the mongo ledger is used and the lock works correctly.

Selection logic

const backend =
  options.ledgerBackend ??
  (options.db ? 'mongo' : options.bucket ? 's3' : null);
if (!backend) throw new SmartMigrationError('Either db or bucket must be provided');

---

Run algorithm

run():
  steps = registeredSteps  // array, in registration order
  validateStepChain(steps)   // checks unique ids, no gaps in version chain

  acquireLock()
  try:
    ledger = readLedger()
    currentVersion = ledger.currentVersion

    if currentVersion === null:
      if isFreshInstall() and freshInstallVersion is set:
        currentVersion = freshInstallVersion
        writeLedger({ currentVersion })
      else:
        currentVersion = steps[0].from  // start from earliest

    if compareSemver(currentVersion, targetVersion) === 0:
      return { wasUpToDate: true, ... }

    plan = computePlan(steps, currentVersion, targetVersion)
    if plan.length === 0:
      throw "no migration path from X to Y"

    for step in plan:
      if dryRun:
        skipped.push(step); continue
      ctx = buildContext(step)
      try:
        await step.handler(ctx)
        ledger.steps[step.id] = { ...result }
        ledger.currentVersion = step.toVersion
        writeLedger(ledger)
        applied.push(step)
      catch err:
        ledger.steps[step.id] = { ...result, status: 'failed', error }
        writeLedger(ledger)
        throw err

  finally:
    releaseLock()

isFreshInstall()

• Mongo: db.mongoDb.listCollections({}, {nameOnly:true}).toArray() → if every collection name starts with smartmigration's reserved prefix or matches the EasyStore class name, fresh.
• S3: open a bucket.createCursor('') and ask for one batch — if empty (after excluding .smartmigration/ prefix), fresh.

validateStepChain(steps)

• Each step.id must be unique
• Each step.from and step.to must be valid semver
• For consecutive steps a, b: a.to === b.from (strict equality, not semver-compare — forces explicit chains)
• compareSemver(step.from, step.to) < 0 for every step

computePlan(steps, current, target)

• Find the step where from === current. Take it and all subsequent steps until one has to === target. Return that slice.
• If current doesn't match any step's from, throw with a clear message naming the registered version chain.
• If we walk past target without matching, throw.

---

File layout

Following the flat-classes pattern used by most push.rocks modules (smartproxy's nested layout is the exception, justified only by its size). The new module's ts/ will look like:

ts/
├── 00_commitinfo_data.ts             // auto-generated by commitinfo on release
├── index.ts                          // re-exports the public surface
├── plugins.ts                        // central import barrel
├── interfaces.ts                     // I-prefixed public interfaces
├── types.ts                          // T-prefixed public type aliases
├── logger.ts                         // module-scoped Smartlog singleton
├── classes.smartmigration.ts         // SmartMigration class + SmartMigrationError
├── classes.migrationstep.ts          // MigrationStepBuilder + internal MigrationStep
├── classes.migrationcontext.ts       // buildContext() factory + checkpoint impl
├── classes.versionresolver.ts        // semver-based plan computation + validation
└── ledgers/
    ├── classes.ledger.ts             // abstract Ledger base
    ├── classes.mongoledger.ts        // EasyStore-backed implementation
    └── classes.s3ledger.ts           // bucket.fastPut/fastGet-backed implementation

ts/plugins.ts content

// node native scope
import { randomUUID } from 'node:crypto';
export { randomUUID };

// pushrocks scope
import * as smartdata from '@push.rocks/smartdata';
import * as smartbucket from '@push.rocks/smartbucket';
import * as smartlog from '@push.rocks/smartlog';
import * as smartlogDestinationLocal from '@push.rocks/smartlog/destination-local';
import * as smartversion from '@push.rocks/smartversion';
import * as smarttime from '@push.rocks/smarttime';
import * as smartpromise from '@push.rocks/smartpromise';
export {
  smartdata, smartbucket, smartlog, smartlogDestinationLocal,
  smartversion, smarttime, smartpromise,
};

// third-party scope (driver re-exports for type access)
import type * as mongodb from 'mongodb';
import type * as awsSdk from '@aws-sdk/client-s3';
export type { mongodb, awsSdk };

smartdata and smartbucket are peerDependencies (not direct dependencies) — users will already have one or both, and we don't want to duplicate them. Listed in dependencies: @push.rocks/smartlog, @push.rocks/smartversion, @push.rocks/smarttime, @push.rocks/smartpromise. Listed in peerDependencies (optional): @push.rocks/smartdata, @push.rocks/smartbucket. Listed in devDependencies for tests: both peers + @push.rocks/smartmongo (in-memory mongo).

---

Project scaffolding files

These mirror the smartproxy conventions, with smartproxy-specific Rust bits removed.

package.json

{
  "name": "@push.rocks/smartmigration",
  "version": "1.0.0",
  "private": false,
  "description": "Unified migration runner for MongoDB (smartdata) and S3 (smartbucket) — designed to be invoked at SaaS app startup, with semver-based version tracking, sequential step execution, idempotent re-runs, and per-step resumable checkpoints.",
  "main": "dist_ts/index.js",
  "typings": "dist_ts/index.d.ts",
  "type": "module",
  "author": "Lossless GmbH",
  "license": "MIT",
  "scripts": {
    "test": "(tstest test/**/test*.ts --verbose --timeout 120 --logfile)",
    "build": "(tsbuild tsfolders --allowimplicitany)",
    "format": "(gitzone format)",
    "buildDocs": "tsdoc"
  },
  "devDependencies": {
    "@git.zone/tsbuild": "^4.4.0",
    "@git.zone/tsrun": "^2.0.2",
    "@git.zone/tstest": "^3.6.0",
    "@push.rocks/smartdata": "^7.1.6",
    "@push.rocks/smartbucket": "^4.5.1",
    "@push.rocks/smartmongo": "^7.0.0",
    "@types/node": "^25.5.0",
    "typescript": "^6.0.2"
  },
  "dependencies": {
    "@push.rocks/smartlog": "^3.2.1",
    "@push.rocks/smartpromise": "^4.2.3",
    "@push.rocks/smarttime": "^4.1.1",
    "@push.rocks/smartversion": "^3.0.5"
  },
  "peerDependencies": {
    "@push.rocks/smartdata": "^7.1.6",
    "@push.rocks/smartbucket": "^4.5.1"
  },
  "peerDependenciesMeta": {
    "@push.rocks/smartdata": { "optional": true },
    "@push.rocks/smartbucket": { "optional": true }
  },
  "files": [
    "ts/**/*",
    "dist/**/*",
    "dist_*/**/*",
    "dist_ts/**/*",
    ".smartconfig.json",
    "readme.md",
    "changelog.md"
  ],
  "keywords": [
    "migration", "schema migration", "data migration",
    "mongodb", "s3", "smartdata", "smartbucket",
    "saas", "startup migration", "semver", "idempotent",
    "ledger", "rolling deploy"
  ],
  "homepage": "https://code.foss.global/push.rocks/smartmigration#readme",
  "repository": {
    "type": "git",
    "url": "https://code.foss.global/push.rocks/smartmigration.git"
  },
  "bugs": {
    "url": "https://code.foss.global/push.rocks/smartmigration/issues"
  },
  "pnpm": {
    "overrides": {},
    "onlyBuiltDependencies": ["mongodb-memory-server"]
  }
}

tsconfig.json (verbatim from smartproxy)

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "esModuleInterop": true,
    "verbatimModuleSyntax": true
  },
  "exclude": ["dist_*/**/*.d.ts"]
}

.smartconfig.json — same shape as smartproxy's, with name/scope/repo updated and the @git.zone/tsrust block omitted. Description and keywords from package.json above.

.gitignore — verbatim from smartproxy minus the rust/target line.

license — MIT, copy from smartproxy.

changelog.md — single entry for 1.0.0 - <today> - Initial release.

readme.hints.md — empty stub.

---

Test plan

Tests live in test/ and follow the smartproxy/tstest conventions: every file ends with export default tap.start(); and uses expect/tap from @git.zone/tstest/tapbundle. Use @push.rocks/smartmongo to spin up an in-memory mongo for tests; for S3 tests, mock against a small in-process fake (or skip and only test mongo paths in v1).

Files to create:

File	What it covers
test/test.basic.ts	Constructor validates options; throws if neither db nor bucket given; default ledger backend selection
test/test.builder.ts	Step builder chains correctly, .from().to().up() registers a step, validation catches duplicate ids and gaps
test/test.versionresolver.ts	computePlan returns the right slice for various current/target combinations; throws on missing chain
test/test.mongoledger.ts	Read/write/lock against a real mongo via smartmongo; lock expiry; concurrent acquire retries
test/test.run.mongo.ts	End-to-end: define 3 steps, run from scratch, verify all applied; re-run is no-op; mid-step failure leaves ledger consistent
test/test.run.checkpoint.ts	Resumable step that crashes mid-way; second run resumes from checkpoint
test/test.freshinstall.ts	Empty db + freshInstallVersion set → jumps to that version, runs no steps
test/test.dryrun.ts	dryRun: true returns plan but does not write

Critical assertions for the end-to-end mongo test (test/test.run.mongo.ts):

const db = await smartmongo.SmartMongo.createAndStart();
const smartmig = new SmartMigration({ targetVersion: '2.0.0', db: db.smartdataDb });
const log: string[] = [];
smartmig
  .step('a').from('1.0.0').to('1.1.0').up(async () => { log.push('a'); })
  .step('b').from('1.1.0').to('1.5.0').up(async () => { log.push('b'); })
  .step('c').from('1.5.0').to('2.0.0').up(async () => { log.push('c'); });

const r1 = await smartmig.run();
expect(r1.stepsApplied).toHaveLength(3);
expect(log).toEqual(['a', 'b', 'c']);
expect(r1.currentVersionAfter).toEqual('2.0.0');

const r2 = await smartmig.run();
expect(r2.wasUpToDate).toBeTrue();
expect(r2.stepsApplied).toHaveLength(0);
expect(log).toEqual(['a', 'b', 'c']); // unchanged

End-to-end verification path (manual smoke after implementation): write a tiny .nogit/debug/saas-startup.ts script that constructs a SmartdataDb against a local mongo, registers two no-op migrations, runs smartmig.run() twice, and prints the result both times — the second run must report wasUpToDate: true.

---

Critical files to create

Source (under /mnt/data/lossless/push.rocks/smartmigration/):

• package.json
• tsconfig.json
• .smartconfig.json
• .gitignore
• license
• changelog.md
• readme.md (full structured README following the smartproxy section template — installation, what is it, quick start, core concepts, common use cases, API reference, troubleshooting, best practices, license)
• readme.hints.md (stub)
• ts/00_commitinfo_data.ts
• ts/index.ts
• ts/plugins.ts
• ts/interfaces.ts
• ts/types.ts
• ts/logger.ts
• ts/classes.smartmigration.ts
• ts/classes.migrationstep.ts
• ts/classes.migrationcontext.ts
• ts/classes.versionresolver.ts
• ts/ledgers/classes.ledger.ts
• ts/ledgers/classes.mongoledger.ts
• ts/ledgers/classes.s3ledger.ts
• test/test.basic.ts
• test/test.builder.ts
• test/test.versionresolver.ts
• test/test.mongoledger.ts
• test/test.run.mongo.ts
• test/test.run.checkpoint.ts
• test/test.freshinstall.ts
• test/test.dryrun.ts

Reused from existing modules (no need to create — these are already in dependencies):

• EasyStore<T> from smartdata (/mnt/data/lossless/push.rocks/smartdata/ts/classes.easystore.ts:9-121) — backs the mongo ledger
• SmartdataDb.mongoDb (raw Db) and SmartdataDb.mongoDbClient (raw MongoClient) for ctx.mongo and transactional sessions
• SmartdataDb.startSession() (/mnt/data/lossless/push.rocks/smartdata/ts/classes.db.ts) for ctx.startSession()
• Bucket.fastGet / Bucket.fastPut for the S3 ledger backend
• Bucket.createCursor (resumable token-based pagination, /mnt/data/lossless/push.rocks/smartbucket/ts/classes.listcursor.ts) — the canonical pattern for restartable S3 migrations, referenced in the readme example
• SmartBucket.storageClient for ctx.s3
• Smartlog from @push.rocks/smartlog for the module logger
• compareVersions from @push.rocks/smartversion for semver ordering