A unified migration runner for MongoDB (via @lossless.org/client/nosqldb) and S3 (via @lossless.org/client/objectstorage) — designed to be invoked at SaaS app startup.

Issue Reporting and Security

For reporting bugs, issues, or security vulnerabilities, please visit community.foss.global/. This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a code.foss.global/ account to submit Pull Requests directly.

Installation

pnpm add @push.rocks/smartmigration
pnpm add @lossless.org/client
pnpm add @aws-sdk/client-s3 # only when your migrations use a bucket

@lossless.org/client is an optional peer dependency, optional in the package-manager sense only — nothing installs it for you. It covers both resources: @lossless.org/client/nosqldb for the database and @lossless.org/client/objectstorage for the bucket. Install it when your migrations touch either. The database handed to db is the SmartdataDb from @lossless.org/client/nosqldb (a NoSqlConnection obtained from LosslessOrgClient.nosqldb() is one, too). The bucket handed to bucket is the Bucket from @lossless.org/client/objectstorage, opened through a SmartBucket (an ObjectStorageConnection obtained from LosslessOrgClient.objectstorage() is one, too). The supported peer range is >=1.4.1 <3.0.0 (the client's breaking line is its major, so the range widens deliberately at each client major); development and the repository lockfile use 2.4.0, and the peer verification runs the whole test suite on the 1.x floor.

@aws-sdk/client-s3 is the S3 driver the objectstorage family runs on, and the client declares it an optional peer of its own. Install it when your migrations use a bucket. A document-only install needs no SDK: this package's declarations never import @lossless.org/client/objectstorage. They type the bucket by IMigrationBucket, the five methods the runner calls, which every supported client's Bucket satisfies, so a consumer that only passes db type-checks and runs without @aws-sdk/client-s3.

A custom logger is any object with a log(level, message, data?) method, the IMigrationLogger contract, and ctx.log in a handler is that logger. A Smartlog from @push.rocks/smartlog of any major satisfies it, as does a plain object, so your logger never has to come from the same smartlog copy this package depends on.

What is smartmigration?

smartmigration is the missing piece for SaaS apps in the push.rocks ecosystem: a deterministic, idempotent way to evolve persistent data in lockstep with code releases. You define a chain of migration steps with from/to semver versions, hand the runner your nosqldb SmartdataDb and/or Bucket, and call run() from your app's startup path. The runner figures out which steps need to execute, runs them sequentially, and stamps progress into a ledger so subsequent boots are fast no-ops.

Key features

Feature	Description
Builder-style API	Fluent step definition: `step('id').from('1.0.0').to('1.1.0').up(handler)`
Unified mongo + S3	A single semver represents your combined data version. One step can touch both.
Drivers exposed via context	`ctx.db`, `ctx.mongo`, `ctx.bucket`, `ctx.s3` — write any operation you can write directly
Idempotent	`run()` is a no-op when data is at `targetVersion`. Costs one ledger read on the happy path.
Resumable	Mark a step `.resumable()` and it gets `ctx.checkpoint.read/write/clear` for restartable bulk operations; successful runs clear the step checkpoint automatically
Lockable	Mongo-backed lock uses atomic updates plus TTL heartbeats to serialize concurrent SaaS instances — safe for rolling deploys
Fresh-install fast path	Configure `freshInstallVersion` to skip migrations on a brand-new database
Target bridging	Optionally bridge schema/data version to the app version without hand-written no-op steps
Reversible	Give a step `.down(handler)` and a target behind the ledger walks the chain back. A path crossing a step without one fails closed before anything runs
Interrupted-step safety	A step whose handler ran but whose stamp never reached the ledger is marked in the ledger; the next run finishes it in the same direction and refuses the opposite one by name
Transactional steps	Mark a step `.transactional()` and its data change and its ledger stamp commit in one database transaction, or not at all
Deletable history	Declare `oldestSupportedVersion` and released steps can be deleted: a ledger older than the floor is refused by name instead of being skipped forward
Dry-run	`dryRun: true` or `.plan()` returns the execution plan without writing anything
Safe operational errors	Unknown handler, dependency, and driver failures cross one fail-closed sanitizer and become `SmartMigrationError` instances with stable boundary codes

Downgrades

A run is entirely one direction: the target is either ahead of the ledger or behind it. When targetVersion is behind the current data version, the runner walks the chain backwards and calls each step's .down() handler, newest step first, stamping the ledger to each step's fromVersion as it goes. Steps come back with status reverted.

migration
  .step('lowercase-emails')
  .from('1.0.0')
  .to('1.1.0')
  .down(async (ctx) => {
    // undo just enough that 1.0.0 code is correct again
  })
  .up(async (ctx) => {
    // ...
  });

.down() is optional and must precede .up(), which is terminal. Three rules make a downgrade safe to reach for:

Irreversible by default. A step without .down() cannot be walked back. If a downgrade path crosses one, the whole run fails with DOWNGRADE_STEP_IRREVERSIBLE before any handler runs, so you never end up half way down a chain. Auto-generated bridge steps do nothing and so need no handler.
The target must be a step boundary. Rewinding to a version no step starts at fails with TARGET_NOT_REACHABLE rather than guessing which half of a step to undo.
No skip-forward in reverse. A forward run tolerates a ledger sitting between two steps by re-running an idempotent handler. Reverse has no equivalent — nothing describes how to undo a partially applied version — so that state fails closed.

Two more behaviours worth knowing:

Bridge mode walks back through the bridge. With targetVersionStrategy: 'bridge' the ledger normally sits above the last step, at the app version. The generated bridge step is reversible by definition (it does nothing), so a rewind crosses it and continues into the real steps.
Fresh installs never run down handlers. A database that never had a step applied has nothing to undo, so a target below freshInstallVersion is simply stamped.
An interrupted down step is finished downwards. A down handler that committed before its stamp reached the ledger leaves the ledger at the step's toVersion with an in-progress marker. The next forward start is refused with INTERRUPTED_STEP_OPPOSITE_DIRECTION instead of skipping the step over reverted data; the next down run re-runs the same .down() handler and stamps it. So a down handler must be idempotent, exactly as an up handler is. See Interrupted steps, and Transactional steps for steps whose revert commits together with the stamp.

plan() and dryRun report a downgrade exactly as they report an upgrade, without executing or writing anything.

Quick start

import { SmartdataDb } from '@lossless.org/client/nosqldb';
import { SmartBucket } from '@lossless.org/client/objectstorage';
import { SmartMigration } from '@push.rocks/smartmigration';
import { commitinfo } from './00_commitinfo_data.js';

// 1. set up your resources as you normally would
const db = new SmartdataDb({
  mongoDbUrl: process.env.MONGODB_URL!,
  mongoDbName: 'myapp',
});
await db.init();

const sb = new SmartBucket({
  accessKey: process.env.S3_ACCESSKEY!,
  accessSecret: process.env.S3_SECRETKEY!,
  endpoint: process.env.S3_ENDPOINT!,
  region: 'us-east-1',
});
const bucket = await sb.getBucketByName('myapp-uploads');

// 2. construct the runner with your app's target version
const migration = new SmartMigration({
  targetVersion: commitinfo.version,
  db,
  bucket,
});

// 3. register migrations in execution order, with from/to validating the chain
migration
  .step('lowercase-emails')
    .from('1.0.0').to('1.1.0')
    .description('Lowercase all user emails')
    .up(async (ctx) => {
      await ctx.mongo!.collection('users').updateMany(
        {},
        [{ $set: { email: { $toLower: '$email' } } }],
      );
    })
  .step('reorganize-uploads')
    .from('1.1.0').to('2.0.0')
    .description('Move uploads/ to media/')
    .resumable()
    .up(async (ctx) => {
      const cursor = ctx.bucket!.createCursor('uploads/');
      const startToken = await ctx.checkpoint!.read<string>('cursorToken');
      if (startToken) cursor.setToken(startToken);
      while (cursor.hasMore()) {
        const { keys } = await cursor.next();
        for (const key of keys) {
          await ctx.bucket!.fastMove({
            sourcePath: key,
            destinationPath: 'media/' + key.slice('uploads/'.length),
            overwrite: true,
          });
        }
        await ctx.checkpoint!.write('cursorToken', cursor.getToken());
      }
    });

// 4. run on startup — fast no-op once the data is at targetVersion
const result = await migration.run();
console.log(`smartmigration: ${result.currentVersionBefore ?? 'fresh'} → ${result.currentVersionAfter} (${result.stepsApplied.length} steps in ${result.totalDurationMs}ms)`);

Core concepts

The data version

A single semver string represents the combined state of your mongo and S3 data. It is not the same as your app version (though you usually want them to match): the app version is what's running, the data version is what's on disk. The runner's job is to bring the data version up to the app version.

Steps and the chain

Each migration is a step with:

A unique id (string) — used as the ledger key
A from semver — the data version this step expects to start from
A to semver — the data version this step produces
An up handler — the actual migration logic
An optional down handler — reverses the step, making it downgradable
An optional description, resumable flag

Steps execute in registration order. The runner validates that the chain is contiguous: step[N].to === step[N+1].from. This catches gaps and overlaps at definition time, before any handler runs.

Resume modes

When run() reads the ledger and finds a current version, it computes a plan: the subset of steps needed to advance from currentVersion to targetVersion. Two resume modes are supported:

Exact resume — currentVersion === step.fromVersion for some step. The normal case, where the ledger sits exactly at a step's starting point (because the previous step's to was written to the ledger when it completed).
Skip-forward resume — currentVersion > step.fromVersion but currentVersion < step.toVersion. The orphan case: the ledger was stamped to an intermediate version that no registered step starts at. This typically happens when an app configures freshInstallVersion: targetVersion across several releases that didn't add any migrations — fresh installs get stamped to whatever commitinfo.version was at install time, not to the last step's to. When a migration is finally added, those installs have a ledger value that doesn't match any step's from.

In skip-forward mode, the planner picks the first step whose toVersion > currentVersion and runs it (and all subsequent steps) normally. The step's handler is being invoked against data that may already be partially in the target shape, so step handlers must be idempotent (use $set over $inc, check existence before insert, filter-based updateMany over cursor iteration where possible). A log line at INFO level announces when a step runs in skip-forward mode.

If no step's toVersion is greater than currentVersion (the ledger is past the end of the chain), the runner throws TARGET_NOT_REACHABLE.

A third case is the interrupted step: a step whose handler started and whose stamp never reached the ledger. The run that meets it finishes that step first, in the direction it was interrupted in; see Interrupted steps.

Interrupted steps

A step's handler commits its data, and the ledger is stamped after the handler returns. Between the two the process can die. Before an ordinary step's handler runs, the runner therefore writes an in-progress marker to the ledger, inProgress: { stepId, direction, fromVersion, toVersion, startedAt }, and the stamp clears it in the same write that records the step. The marker stays when the process dies, when the handler throws and when the lock is lost; it is absent from a ledger at rest, which keeps the four fields every earlier release wrote. Generated bridge steps do nothing and are never marked, and .transactional() steps need no marker because their stamp commits with their data.

The next run reads the marker under the lock and:

finishes the step in the same direction. A run whose target lies at or beyond the step's far end in that direction (at or above toVersion after an interrupted up, at or below fromVersion after an interrupted down) starts with that step and runs the same handler again, so every handler must be idempotent, .down() handlers included. The stamp then clears the marker.
refuses every other target with INTERRUPTED_STEP_OPPOSITE_DIRECTION: the opposite direction, the step's own start version, or a version inside its span. A forward start after an interrupted downgrade would otherwise skip the step over reverted data, and a rollback after an interrupted upgrade would start old code on half-migrated data. A ledger at the target with a marker is not taken for up to date: the fast path is bypassed.
refuses a chain that no longer starts with the step with INTERRUPTED_STEP_NOT_PLANNED, for instance when a release renamed it, deleted it or changed its versions.

plan() and dryRun refuse the same way, without taking the lock.

The marker and the stamp are fenced: they change only the fields they name, and only while the document's lock.holder is this run. A write that matches nothing means another instance took the lock, and the step fails with LOCK_LOST without writing anything more. On the S3 backend the fence is as good as that backend's lock: the runner reads the sidecar, checks the holder and writes it back, and a concurrent writer can interleave with that.

Abandoning an interrupted step

When the step cannot be finished in its direction — its handler is broken and the release has to be rolled back — an operator who has made the step's data consistent by other means clears the marker:

await migration.abandonInterruptedStep({
  stepId: 'reshape-orders',
  reason: 'orders restored from the 02:00 snapshot, see OPS-412',
});

It takes the migration lock (LOCK_TIMEOUT when another instance holds it), requires stepId to name the interrupted step exactly (INTERRUPTED_STEP_MISMATCH otherwise, NO_INTERRUPTED_STEP when there is none), clears only the marker and logs reason at warn level. The ledger version never moves: the next run plans from it as if the step had never started. From that moment the operator owns the consistency of the step's data; smartmigration no longer knows that the data may be part way through the step.

Rolling back below the fixed version

The marker exists from 2.1.0 on the maintenance/2 line and from 4.0.0 on; every release before 2.1.0 and every 3.x release lacks it. Such a runner rebuilds the ledger from the four fields it knows, and its next ledger write removes inProgress. Rolling a consumer back to one of those releases while a step is interrupted drops the marker, and with it the protection above. Finish or abandon an interrupted step before rolling back below the fixed version.

Transactional steps

For a step whose data lives in the migration database, .transactional() closes the window between the commit and the stamp altogether: the runner opens a session, runs the handler in a transaction and writes the step's ledger stamp in that same transaction, so the data change and the stamp commit together or not at all. It applies to both .up() and .down():

migration
  .step('backfill-account-tier')
  .from('1.5.0').to('2.0.0')
  .transactional()
  .down(async (ctx) => {
    await ctx.mongo!.collection('users').updateMany(
      { tier: { $exists: true } },
      { $unset: { tier: '' } },
      { session: ctx.session },
    );
  })
  .up(async (ctx) => {
    await ctx.mongo!.collection('users').updateMany(
      { tier: { $exists: false } },
      { $set: { tier: 'free' } },
      { session: ctx.session },
    );
  });

The rules that come with it:

Write only through ctx.session. The driver's withTransaction may run the handler more than once: it retries an attempt the database aborted as transient (a write conflict, for instance) and rolls the aborted attempt back. Only operations given { session: ctx.session } belong to the transaction; anything else — an operation without the session, an S3 write, a call to another service — commits on its own and runs again on every retry.
Leave the transaction to the runner. ctx.session is a facade of the runner's driver session. Pass it to driver operations as { session: ctx.session }; the driver takes it for the session itself. Its transaction controls, startTransaction, commitTransaction, abortTransaction, endSession and withTransaction, throw TRANSACTION_CONTROL_RESERVED before anything reaches the driver, and the step then fails with that error and nothing committed, even when the handler catches it: a handler that wants to abort the step throws. As a backstop, a transaction that ended some other way before the stamp, past the facade, fails the step with TRANSACTION_ENDED_BY_HANDLER, and the stamp is not written outside it. Do not swallow the error of an operation on the session either: the driver needs it to abort or retry the attempt.
No DDL. Create collections and indexes in an earlier, ordinary step: engines refuse DDL inside a transaction, and recent @lossless.org/client releases refuse a model whose collection is not prepared yet inside one.
Model writes on an observed database. @lossless.org/client releases with write observers refuse model writes in a transaction on a raw driver session while the database has write observers (observeWrites) or open read sets. ctx.session is such a raw session, because the ledger is written through the raw driver and cannot join a db.createSession() transaction. Use the raw driver (ctx.mongo) for those writes, or run the migration before the observers start.
Not resumable, mongo ledger only. .transactional() with .resumable() is refused with TRANSACTIONAL_STEP_NOT_RESUMABLE: a transaction leaves no partial progress to resume. The stamp must live in the transaction's database, so a runner without db and the mongo ledger refuses the step with TRANSACTIONAL_STEP_REQUIRES_MONGO_LEDGER, when it is registered.
An engine with transactions. NoSQLDB runs them on a single node; MongoDB needs a replica set or a sharded cluster. On a standalone mongod the step fails with nothing committed.
The lease. A renewal of the lock during the transaction would conflict with the stamp written to the same ledger document, so the heartbeat pauses while the transaction runs. The lock is renewed once just before withTransaction(), and that lease has to cover the driver's 120-second retry window, in which it starts new attempts, the lifetime the server allows the last attempt, and a 30-second margin: lockTtlMs must be at least 120000 + transactionLifetimeLimitSeconds × 1000 + 30000, which is 210 000 ms with the default lifetime of 60 seconds. The runner reads transactionLifetimeLimitSeconds from the server with getParameter where the engine exposes it, and otherwise uses MongoDB's documented default of 60 seconds; NoSQLDB does not implement getParameter, and a MongoDB user without the getParameter privilege cannot read it either. The log names the source. With the default lockTtlMs of 600 000 ms that is satisfied; a shorter lease refuses the step with TRANSACTION_LEASE_TOO_SHORT before it starts. The stamp is fenced on the lock holder as well, so a transaction that outlives its lease and meets another holder rolls back with LOCK_LOST.
An unconfirmed commit. When the driver rejects after sending the commit, the runner re-reads the ledger on the primary with majority read concern. If it holds this attempt's stamp (same step id, status and startedAt), the transaction committed and the step counts as applied.

Target version strategies

By default, smartmigration is strict: the registered chain must contain a real step whose toVersion exactly matches targetVersion. This is best when you maintain an explicit schema/data version.

If your app uses its package version as targetVersion, releases without schema changes can leave the migration chain behind the app version. In that case, enable bridge mode:

const migration = new SmartMigration({
  targetVersion: commitinfo.version,
  db,
  freshInstallVersion: commitinfo.version,
  targetVersionStrategy: 'bridge',
});

Bridge mode appends an internal no-op step from the last registered migration version to targetVersion when the chain ends before the target. It stamps the ledger to the app version without requiring hand-written no-op migrations for every release.

Important behavior:

Existing real migrations still run first.
If the ledger is already past the last registered migration but below targetVersion, only the internal bridge runs.
Future real migrations still work because skip-forward resume runs the first real step whose toVersion is above the ledger version.
Downgrades and steps that overshoot the target still fail.

Deleting released steps

Steps accumulate. Once every environment has passed a migration, its handler is dead weight — often a large cutover engine nobody wants to keep compiling. Deleting it is safe only if no database can still be behind it, and by default nothing checks that:

skip-forward resume enters the first step whose toVersion is above the ledger. Delete the oldest steps and a database still sitting below the remaining chain is not refused — it silently runs the truncated remainder and ends up stamped at targetVersion as if the deleted work had happened.

oldestSupportedVersion closes that hole. It is the oldest data version this release can still migrate:

const migration = new SmartMigration({
  targetVersion: '32.0.0',
  db,
  freshInstallVersion: '32.0.0',
  oldestSupportedVersion: '30.0.0', // steps below 30.0.0 were deleted in this release
});
migration
  .step('split-tenant-index').from('30.0.0').to('31.0.0').up(async (ctx) => { /* ... */ })
  .step('backfill-usage').from('31.0.0').to('32.0.0').up(async (ctx) => { /* ... */ });

The floor is the start of the chain this release ships — the fromVersion of the oldest step still registered. A ledger below it is refused by name on the first read: before the lock is taken, before anything is written, before any step runs. A ledger at or above it plans exactly as it did without a floor.

The floor judges the ledger, and only the ledger. A resource with no ledger at all has no recorded version to judge: an empty one is bootstrapped as it always was (through freshInstallVersion when set, otherwise to the chain start), and a populated one is assumed to sit at the chain start — which is the floor — and migrates forward from there. Pinning the floor to the chain start is what makes that safe: every version a bootstrap can stamp is at or above the floor, so an interrupted first run is resumed on the next start rather than refused by the release that wrote it.

Deleting steps safely

Pick the boundary to keep, and confirm with getCurrentVersion() that every environment's ledger is at or above it.
Delete the steps below that boundary. The remaining steps must still form a contiguous chain.
Set oldestSupportedVersion to the fromVersion of the oldest remaining step — the boundary you just picked. A floor above it would declare a step this release still ships as unsupported; a floor below it, or inside a step's span, would admit data the remaining chain cannot start from. With targetVersionStrategy: 'bridge' and the chain emptied entirely, the floor is the start of the generated bridge: the version the previous release stamped.
Keep the last release that still contains the deleted steps deployable. That is the intermediate release operators are told to run.

The configuration is refused at startup, not at the first unlucky database:

Code	Cause
`INVALID_VERSION`	`oldestSupportedVersion` is not valid semver
`OLDEST_SUPPORTED_VERSION_ABOVE_TARGET`	the floor is ahead of `targetVersion`
`OLDEST_SUPPORTED_VERSION_ABOVE_FRESH_INSTALL`	the floor is ahead of `freshInstallVersion`, so this release would refuse the database it just created
`OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START`	the floor is not the `fromVersion` of the oldest registered step: above it, steps this release still ships are declared unsupported; below it or inside a step's span, data on the floor would be skip-forward resumed — the truncation the floor exists to refuse

What operators see

A database that is too old fails startup with LEDGER_BELOW_OLDEST_SUPPORTED_VERSION and a message that names the ledger version, the floor, and the way out:

The migration ledger is at "29.2.0", which is below the oldest supported version "30.0.0".
The steps for versions below "30.0.0" are no longer part of this release. Run an intermediate
release that migrates this data to at least "30.0.0", then run this release again.

error.details carries currentVersion (the ledger version that was refused), oldestSupportedVersion and targetVersion.

plan() and dryRun raise the same refusal, and every successful result echoes the configured floor as oldestSupportedVersion (null when unset), so a CI probe can see both.

The ledger

The ledger is the source of truth for "what data version are we at, what steps have been applied, who holds the lock right now." It is persisted in one of two backends:

mongo (default when db is provided) — stored as a single raw document in the nosqldb SmartdataEasyStore collection. Lock acquisition / renewal uses atomic mongo updates and works safely across multiple SaaS instances. Recommended.
s3 (default when only bucket is provided) — a single JSON object at <bucket>/.smartmigration/<ledgerName>.json. Lock is best-effort because S3 has no atomic CAS without additional infrastructure; do not use for multi-instance deployments without external coordination.

If you pass both db and bucket, mongo is used.

The Mongo backend talks directly to the raw collection so lock operations can use atomic filters. Its stored shape is:

{
  _id: ObjectId, // assigned by MongoDB
  nameId: `smartmigration:${ledgerName}`,
  data: {
    currentVersion: string | null,
    steps: Record<string, IMigrationLedgerEntry>,
    lock: {
      holder: string | null,
      acquiredAt: string | null,
      expiresAt: string | null,
    },
    checkpoints: Record<string, Record<string, unknown>>,
    // Only while a step is unfinished; see "Interrupted steps".
    inProgress?: { stepId, direction: 'up' | 'down', fromVersion, toVersion, startedAt },
  },
}

Unlike ordinary documents created through SmartdataDb.createEasyStore(), the migration document is not expected to contain EasyStore model metadata such as ephemeral or lastEdit. Both forms intentionally coexist in SmartdataEasyStore; nameId is unique and the migration key is smartmigration:<ledgerName>.

The migration context

Each up handler receives a IMigrationContext with:

interface IMigrationContext<TBucket extends IMigrationBucket = IMigrationBucket> {
  db?: SmartdataDb;            // high-level nosqldb database
  bucket?: TBucket;            // the bucket you passed: an objectstorage Bucket
  mongo?: mongodb.Db;          // raw mongo Db (db.mongoDb)
  s3?: ReturnType<TBucket['getStorageClient']>; // raw AWS SDK v3 S3Client for a Bucket
  step: { id, fromVersion, toVersion, description?, isResumable };
  log: IMigrationLogger;       // the run's logger: log(level, message, data?)
  isDryRun: boolean;
  checkpoint?: IMigrationCheckpoint; // only when step is .resumable()
  session?: mongodb.ClientSession;    // only when step is .transactional(): pass it to every operation; refuses the transaction controls
  startSession(): mongodb.ClientSession; // a session of the handler's own; throws if no db
}

Use whichever level fits your migration:

High-level (ctx.db, ctx.bucket) for working with nosqldb models / objectstorage files
Raw drivers (ctx.mongo, ctx.s3) for any operation the client's families don't wrap (renameCollection, dropIndex, S3 lifecycle policies, etc.). ctx.s3 is the S3Client behind the bucket, reached through Bucket.getStorageClient(), the escape hatch @lossless.org/client/objectstorage retains
ctx.session in a .transactional() step, a facade of the session whose transaction also commits the ledger stamp; it refuses the transaction controls, which stay with the runner; see Transactional steps. startSession() opens a session of the handler's own, whose commit is not tied to the stamp

The bucket's type travels from the constructor. new SmartMigration({ bucket }) with an objectstorage Bucket is a SmartMigration<Bucket>, so a handler written inline in .up() sees ctx.bucket as that Bucket and ctx.s3 as its S3Client. A standalone handler that uses the bucket names the type itself:

import type { Bucket } from '@lossless.org/client/objectstorage';
import type { IMigrationContext } from '@push.rocks/smartmigration';

export const reprefixAttachments = async (ctx: IMigrationContext<Bucket>) => {
  for await (const key of ctx.bucket!.listAllObjects('attachments/legacy/')) {
    await ctx.bucket!.fastMove({
      sourcePath: key,
      destinationPath: 'attachments/' + key.slice('attachments/legacy/'.length),
      overwrite: true,
    });
  }
};

Without the type argument, ctx.bucket is the IMigrationBucket contract and ctx.s3 is unknown, which is all a document-only handler needs, and its declaration never loads the S3 SDK. A SmartMigration<Bucket>, its steps and its builder still go wherever the unparameterized types are expected.

Error sanitization

Every unknown failure crossing a package-owned operational boundary passes through one synchronous sanitization function. Covered boundaries are ledger initialization/read/write, fresh-install and version planning, lock acquisition/renewal/release, migration run and context setup, and migration step handlers. Without an errorSanitizer, any smartmigration-generated text derived from such a failure uses only this generic record:

{ message: 'An operation failed; error details were redacted.' }

Configure errorSanitizer only when an application has approved, non-sensitive error text to expose:

const migration = new SmartMigration({
  targetVersion: '2.0.0',
  db,
  errorSanitizer: (_error, context) => ({
    message: context.stepId
      ? `Migration step ${context.stepId} failed.`
      : `Migration ${context.phase} operation failed.`,
  }),
});

The callback receives the original unknown throw and an IErrorSanitizerContext. Its phase is one of 'ledger-initialization', 'ledger-read', 'ledger-write', 'planning', 'lock-acquire', 'lock-renew', 'lock-release', 'run-setup', or 'migration-step'; step-related phases also include stepId. It is invoked once for each unknown failure at the first matching boundary. An existing SmartMigrationError is already safe and passes through without another sanitizer call.

The sanitizer output must be a plain synchronous { message: string; stack?: string } record with a non-empty message no longer than ERROR_SANITIZER_MAX_MESSAGE_LENGTH (1,024 code units) and an optional stack no longer than ERROR_SANITIZER_MAX_STACK_LENGTH (8,192 code units). Thrown callbacks, Promises, thenables, accessors, malformed records, and oversized values fail closed to GENERIC_SANITIZED_ERROR.

Smartmigration never reads getters or calls toString() while normalizing an unknown throw. For thrown wrappers, the original value is retained only as the non-enumerable cause of SmartMigrationError; it is excluded from the wrapper message, details, generated stack, logs, persisted ledger records, and JSON serialization. Ordinary package validation and planning errors are already-safe SmartMigrationError instances and retain their specific code and message. This guarantee covers smartmigration-produced output; a migration handler or custom logger that independently emits the raw value remains application-owned.

A lock-release failure is sanitized and logged at warn, but it never changes a successful migration into a failure and never masks an earlier failure. The unreleased lock remains protected by its TTL and can be recovered manually after confirming that no migration runner is active.

Common use cases

Mongo schema rename

migration
  .step('rename-user-email-field')
  .from('1.0.0').to('1.1.0')
  .up(async (ctx) => {
    await ctx.mongo!.collection('users').updateMany(
      { emailAddress: { $exists: true } },
      { $rename: { emailAddress: 'email' } },
    );
  });

Backfilling a derived field with a transaction

migration
  .step('backfill-account-tier')
  .from('1.5.0').to('2.0.0')
  .transactional()
  .up(async (ctx) => {
    const users = ctx.mongo!.collection('users');
    await users.updateMany(
      { tier: { $exists: false }, plan: 'free' },
      { $set: { tier: 'free' } },
      { session: ctx.session },
    );
    await users.updateMany(
      { tier: { $exists: false }, plan: { $in: ['pro', 'enterprise'] } },
      { $set: { tier: 'paid' } },
      { session: ctx.session },
    );
  });

Both updates and the step's ledger stamp commit in one transaction. A transaction the handler opens itself through ctx.startSession() commits before the stamp, which leaves the step's window open.

Resumable S3 reprefix

migration
  .step('move-attachments')
  .from('2.0.0').to('2.1.0')
  .resumable()
  .up(async (ctx) => {
    const cursor = ctx.bucket!.createCursor('attachments/legacy/', { pageSize: 200 });
    const start = await ctx.checkpoint!.read<string>('cursor');
    if (start) cursor.setToken(start);
    while (cursor.hasMore()) {
      const { keys } = await cursor.next();
      for (const key of keys) {
        await ctx.bucket!.fastMove({
          sourcePath: key,
          destinationPath: 'attachments/' + key.slice('attachments/legacy/'.length),
          overwrite: true,
        });
      }
      await ctx.checkpoint!.write('cursor', cursor.getToken());
    }
  });

If the process crashes mid-migration, the next call to run() will resume from the last persisted cursor token. If the step completes successfully, smartmigration clears that step's checkpoint bag automatically.

Mongo + S3 in lockstep

migration
  .step('split-avatars-out-of-users')
  .from('2.1.0').to('2.5.0')
  .up(async (ctx) => {
    // 1. for every user with an inline avatarBytes field, write the bytes to S3
    const users = ctx.mongo!.collection('users');
    const cursor = users.find({ avatarBytes: { $exists: true } });
    for await (const user of cursor) {
      const buf = Buffer.from(user.avatarBytes.buffer);
      await ctx.bucket!.fastPut({
        path: `avatars/${user._id}.bin`,
        contents: buf,
        overwrite: true,
      });
      await users.updateOne(
        { _id: user._id },
        { $set: { avatarKey: `avatars/${user._id}.bin` }, $unset: { avatarBytes: '' } },
      );
    }
  });

Fresh-install fast path

const migration = new SmartMigration({
  targetVersion: '5.0.0',
  db,
  freshInstallVersion: '5.0.0', // brand-new DB jumps straight to 5.0.0
});
// ... register your full chain of steps ...
await migration.run(); // for a fresh DB, runs zero steps and stamps version 5.0.0

A resource is a fresh install when its ledger has no version and it holds no data. Data is judged per backend:

MongoDB — a document in any collection except views, system.* namespaces and SmartdataEasyStore. In SmartdataEasyStore, a document is data when it is not a ledger document (its nameId does not start with smartmigration:) and its data holds at least one key; the empty document an EasyStore read creates is not.
S3 — any object outside the .smartmigration/ prefix.

With both db and bucket, the install is fresh only when neither holds data. Collections and indexes are structure, not data: a database that model preparation or a first read has given collections and indexes, or whose documents were all deleted, is still fresh. Detection stops at the first document that is data.

Call run() before preparing models, and before anything writes a document. Steps may rename collections or reshape indexes that preparation would otherwise install first, and a document written before run() — seeded settings, an EasyStore value, a bootstrap record — is data, so it turns an empty database into one that walks the chain from its first step.

Dry run / planning

const planned = await migration.plan();
console.log(`would apply ${planned.stepsSkipped.length} steps:`,
  planned.stepsSkipped.map((s) => `${s.id}(${s.fromVersion}→${s.toVersion})`).join(' → '));
console.log(`predicted version after run: ${planned.currentVersionAfter}`);

// or — same thing, via dryRun option
const m = new SmartMigration({ targetVersion: '2.0.0', db, dryRun: true });
// ...
const result = await m.run(); // returns plan, doesn't write

plan() / dryRun resolve the same fresh-install shortcut that run() would use, so the returned currentVersionAfter is the predicted post-run version. For S3-backed ledgers, planning does not create the .smartmigration/<ledger>.json sidecar.

API reference

`new SmartMigration(options: ISmartMigrationOptions)`

Option	Type	Default	Description
`targetVersion`	`string`	— (required)	Semver representing the data version this code expects
`db`	`SmartdataDb`	undefined	Required if any step uses `ctx.db`/`ctx.mongo` or for the mongo ledger
`bucket`	`Bucket`	undefined	Required if any step uses `ctx.bucket`/`ctx.s3` or for the S3 ledger. Its type becomes the runner's bucket type, `SmartMigration<Bucket>`
`ledgerName`	`string`	`"smartmigration"`	Logical name; lets multiple migrations coexist on the same db/bucket
`ledgerBackend`	`'mongo'` \| `'s3'`	mongo if db, else s3	Where to persist the ledger
`freshInstallVersion`	`string`	undefined	When no resource holds data, jump straight to this version. See Fresh-install fast path
`oldestSupportedVersion`	`string`	undefined	Oldest data version this release can still migrate; must be the `fromVersion` of the oldest registered step. See Deleting released steps
`targetVersionStrategy`	`'strict'` \| `'bridge'`	`'strict'`	Require the chain to end exactly at targetVersion, or auto-bridge to it
`lockWaitMs`	`number`	`60_000`	How long to wait for a stale lock from another instance
`lockTtlMs`	`number`	`600_000`	How long this instance's own lock auto-expires after
`dryRun`	`boolean`	`false`	If true, `run()` returns the plan without executing or locking
`logger`	`IMigrationLogger`	module logger	Custom logger: any object with `log(level, message, data?)`, a `Smartlog` of any major included; defaults to a Smartlog with a local destination
`errorSanitizer`	`IErrorSanitizer`	generic fail-closed record	Synchronous sanitizer for unknown failures at package-owned operational boundaries

The constructor throws SmartMigrationError with one of these codes on bad input:

INVALID_OPTIONS — options object missing
MISSING_TARGET_VERSION — targetVersion not set
INVALID_VERSION — targetVersion, freshInstallVersion or oldestSupportedVersion is not a valid semver
NO_RESOURCES — neither db nor bucket provided
LEDGER_BACKEND_MISMATCH — explicit ledgerBackend doesn't match the resources you provided
INVALID_LEDGER_NAME — ledgerName is blank or not a string
INVALID_LOCK_WAIT_MS — lockWaitMs is not an integer >= 0
INVALID_LOCK_TTL_MS — lockTtlMs is not an integer >= 1
INVALID_TARGET_VERSION_STRATEGY — targetVersionStrategy is not strict or bridge
INVALID_ERROR_SANITIZER — errorSanitizer is present but is not a function
OLDEST_SUPPORTED_VERSION_ABOVE_TARGET — the floor is ahead of targetVersion
OLDEST_SUPPORTED_VERSION_ABOVE_FRESH_INSTALL — the floor is ahead of freshInstallVersion

`migration.step(id: string).from(v).to(v).[description(t)].[resumable() | transactional()].[down(handler)].up(handler)`

The step builder. .from(), .to(), and .up() are required; .description(), .resumable(), .transactional() and .down() are optional. .up() is the terminal call: it commits the step to the parent runner and returns the runner so you can chain .step('next'). Registering a step throws TRANSACTIONAL_STEP_NOT_RESUMABLE for a step that is both .transactional() and .resumable(), and TRANSACTIONAL_STEP_REQUIRES_MONGO_LEDGER for a .transactional() step on a runner without the mongo ledger.

`migration.abandonInterruptedStep({ stepId, reason }): Promise<IMigrationInProgressStep>`

Clears the in-progress marker of an interrupted step without running it, and returns the marker. It takes the migration lock, requires stepId to name the interrupted step exactly, never moves the data version and logs reason at warn level; the operator owns the consistency of the step's data from then on. Throws INVALID_ABANDON_OPTIONS, LOCK_TIMEOUT, NO_INTERRUPTED_STEP or INTERRUPTED_STEP_MISMATCH. See Abandoning an interrupted step.

`migration.run(): Promise<IMigrationRunResult>`

The startup entry point. Acquires a lock, computes the plan, executes pending steps in order, and releases the lock. Returns:

interface IMigrationRunResult {
  currentVersionBefore: string | null;
  currentVersionAfter: string;
  targetVersion: string;
  oldestSupportedVersion: string | null; // the configured floor, null when unset
  wasUpToDate: boolean;        // true if no steps ran
  wasFreshInstall: boolean;    // true if startup took the fresh-install shortcut
  stepsApplied: IMigrationStepResult[];
  stepsSkipped: IMigrationStepResult[];
  totalDurationMs: number;
}

Throws SmartMigrationError with these codes:

LEDGER_INITIALIZATION_FAILED — ledger backend initialization failed
LEDGER_READ_FAILED / LEDGER_WRITE_FAILED — a raw ledger operation failed
PLANNING_FAILED — an unknown dependency/driver failure occurred while planning; ordinary planning errors retain their specific code
RUN_SETUP_FAILED — an unknown failure occurred while setting up the run or a step context
LOCK_ACQUIRE_FAILED — the lock backend failed while attempting acquisition
LOCK_TIMEOUT — could not acquire lock within lockWaitMs
LOCK_LOST — the runner lost its lock while a migration was still in flight
STEP_FAILED — a step's handler threw; the failure is persisted to the ledger
CHAIN_*, DUPLICATE_STEP_ID, NON_INCREASING_STEP, TARGET_NOT_REACHABLE — chain validation / planning errors
OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START — the configured floor is not the start of the registered chain; raised before the ledger is read
LEDGER_BELOW_OLDEST_SUPPORTED_VERSION — the ledger is older than oldestSupportedVersion; raised on the first ledger read, before the lock and before anything is written
DOWNGRADE_STEP_IRREVERSIBLE — a reverse plan crosses a step with no .down() handler; raised during planning, so nothing has run
DOWNGRADE_NOT_SUPPORTED, DOWNGRADE_PLAN_INVALID (legacy) — a plan was computed in the wrong direction for the versions given. Neither is reachable through the public API since the runner picks the direction itself; they guard direct VersionResolver use

Lock-release failures do not throw a public error code. They produce a sanitized warning and preserve the run's existing success or failure outcome.

`migration.plan(): Promise<IMigrationRunResult>`

Same as run() but does not acquire the lock or execute anything. Useful for --dry-run style probes in CI. stepsSkipped are the steps that would run, and currentVersionAfter is the predicted post-run version after fresh-install resolution.

`migration.getCurrentVersion(): Promise<string | null>`

Returns the current data version from the ledger, or null if the ledger has never been initialized.

Troubleshooting

`LOCK_TIMEOUT` on every startup

Another instance crashed while holding the lock. Wait for lockTtlMs (default 10 minutes) for the lock to expire, or manually clear the lock field on the ledger document.

For Mongo, inspect SmartdataEasyStore for { nameId: 'smartmigration:<ledgerName>' }. Only after confirming that the recorded holder is no longer running, clear the exact observed holder atomically:

db.SmartdataEasyStore.updateOne(
  {
    nameId: 'smartmigration:smartmigration',
    'data.lock.holder': '<observed-holder-id>',
  },
  {
    $set: {
      'data.lock': { holder: null, acquiredAt: null, expiresAt: null },
    },
  },
)

Replace the second smartmigration with a custom ledgerName when configured. For S3, make the equivalent edit to .smartmigration/<ledgerName>.json. Prefer waiting for expiresAt; manual clearing can allow two runners to overlap if the holder is still active.

`LOCK_LOST` during a migration

The runner stopped renewing its lock or another instance replaced the lock document while a step was still running. The heartbeat reports it, and so does every fenced write — the in-progress marker, the step stamp, the stamp of a .transactional() step — that finds another holder; the run then writes nothing more to the ledger. Check for long-running steps, make sure lockTtlMs is comfortably larger than the expected renewal cadence, and investigate other processes touching the same ledger. A non-transactional step that lost its lock keeps its marker, so the next run finishes it.

`INTERRUPTED_STEP_OPPOSITE_DIRECTION` at startup

A previous run started the named step and never stamped it: the process died, the handler threw or the lock was lost. Its data may be part way, so only a run that moves through the step in the direction named in the message can continue. Start the release whose target finishes the step in that direction (the message names the version), let it finish, then deploy the release you wanted. If the step cannot be finished that way, make its data consistent by hand and call abandonInterruptedStep().

`INTERRUPTED_STEP_NOT_PLANNED` at startup

The direction is right, but this release's chain does not start with the interrupted step under the id and versions it ran with: it was renamed, deleted or re-versioned. Restore the step as it was in the release that started it, or make the data consistent by hand and call abandonInterruptedStep().

`TRANSACTION_LEASE_TOO_SHORT`

A .transactional() step runs on the lease renewed just before its transaction, which must cover the driver's 120-second withTransaction retry window, the server's transaction lifetime limit and a 30-second margin: 210 000 ms with the default limit of 60 seconds. Set lockTtlMs to at least the value the message names. See Transactional steps.

`CHAIN_GAP` at startup

Two adjacent steps have mismatched versions: step[N].to !== step[N+1].from. Steps must form a contiguous chain in registration order. Fix the version on the offending step.

`CHAIN_NOT_AT_CURRENT` (legacy)

Retained in the error vocabulary for backward compatibility with downstream consumers that previously branched on it, but no longer thrown by computePlan in normal operation. Prior versions of smartmigration required an exact fromVersion === currentVersion match when resolving the plan; the current planner supports skip-forward resume and handles intermediate-version ledger stamps transparently.

`TARGET_NOT_REACHABLE`

Forward, either (a) a step in the plan upgrades to a version past targetVersion without any step ending exactly at targetVersion, or (b) the ledger's currentVersion is past the end of the registered chain but has not reached targetVersion. Case (a) means the chain has a mid-step that overshoots — add the missing final step or adjust targetVersion. Case (b) means the chain needs a new step extending it toward targetVersion.

In reverse the causes are different, and extending the chain is the wrong response to all of them: the target is not a step boundary (pick a version some step starts at), or the ledger sits between two steps so no step describes how to undo it. With targetVersionStrategy: 'bridge' a ledger above the chain end is walked back through the generated bridge, so that case plans normally.

`LEDGER_BELOW_OLDEST_SUPPORTED_VERSION` at startup

This database is older than the oldest version this release still carries steps for, because those steps were deleted. Deploy the intermediate release named in the message (the last one that still contains them), let it finish migrating, then deploy this release again. The refusal lands on the first read of the ledger, before the lock is taken and before anything is written, so the database is exactly as it was before startup — the one exception is another instance stamping a sub-floor version while this run waits for the lock, where the refusal comes from the re-read under the lock and the lock document exists. Lowering oldestSupportedVersion is not a fix unless the deleted steps come back with it.

`OLDEST_SUPPORTED_VERSION_NOT_THE_CHAIN_START` at startup

The configured floor is not the start of the registered chain. Move it to the fromVersion of the oldest remaining step: a floor above that declares a step this release still ships as unsupported, and a floor below it, or inside a step's span, admits data the chain cannot start from. The error message prints the registered chain.

S3-only deployments and concurrent instances

The S3 ledger's lock is best-effort. If you run multiple SaaS instances against the same S3 bucket, use external coordination (e.g. Redis lock, leader election) before calling run(). The fenced writes of the S3 ledger (the in-progress marker and the step stamp) share that limit: they read the sidecar, check the lock holder and write it back, which a concurrent writer can interleave with. The mongo backend has no such limitation.

Client peer verification

Start the repository's configured MongoDB and object-storage services, then verify the declared peer floor:

gitzone services start
pnpm run verify:client:peer # the 1.x floor, 1.4.1
node scripts/verify-client-peer.mjs 2.0.0 # any other exact version inside the declared peer range

pnpm test runs the suite on the development client (2.4.0); pnpm run verify:client:peer runs the same suite on the 1.x floor, so both client lines are tested whole. The command takes one exact @lossless.org/client version and refuses anything the declared peer range in package.json does not contain, so the range and the verifier can never drift apart. It creates an isolated temporary copy, installs the selected version from the configured registry without file:, link:, or workspace links, builds the package, type-checks every test file, and runs the whole suite against the services. The suite includes the compatibility probe, which verifies SmartdataDb.mongoDb, raw sessions, the nosqldb EasyStore, the atomic smartmigration ledger document, and coexistence of both document types in SmartdataEasyStore; it also checks at compile time that the installed client's Bucket satisfies IMigrationBucket, and the root-types test proves a document-only consumer type-checks without the S3 SDK. The temporary copy carries qenv.yml, whose required: list qenv resolves while the test helpers construct their Qenv, and it reads those names from .nogit/env.json when available or from the equivalent service environment variables in CI; it is removed after success, failure, SIGINT, or SIGTERM. Child commands have a 10-minute default timeout, receive forwarded termination signals as one process group, and escalate to SIGKILL after five seconds. Override those bounds with positive integer SMARTMIGRATION_VERIFY_TIMEOUT_MS and SMARTMIGRATION_VERIFY_TERMINATION_GRACE_MS values.

Best practices

Write idempotent handlers, in both directions. Even with the lock, machines crash, networks partition, and an interrupted step is finished by running the same handler again — .down() handlers included. Use $set over $inc, check existence before insert, etc.
Use .resumable() for any step that processes more than a few hundred records. Crashes happen; resumability avoids redoing work.
Mark data steps in the migration database .transactional() and pass ctx.session to every operation: the data change and the ledger stamp then commit together. See Transactional steps.
Don't mix S3 and Mongo writes inside a single conceptual operation — S3 has no transactions, so design migrations so each S3 write is observably idempotent on its own.
Keep the targetVersion linked to your app's package.json version via the autocreated 00_commitinfo_data.ts. That way every release automatically signals the data version it expects.
Freeze applied migrations in source control — don't edit a step that has been applied to production data, even if the new version is "more correct." Add a new step instead.
Declare oldestSupportedVersion whenever you delete released steps. Without it, a database that never ran the deleted steps is skipped forward in silence. See Deleting released steps.

License and Legal Information

This repository contains open-source code licensed under the MIT License. A copy of the license can be found in the repository license file.

Please note: The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.

Trademarks

This project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH or third parties, and are not included within the scope of the MIT license granted herein.

Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines or the guidelines of the respective third-party owners, and any usage must be approved in writing. Third-party trademarks used herein are the property of their respective owners and used only in a descriptive manner, e.g. for an implementation of an API or similar.

readme.md Unescape Escape

@push.rocks/smartmigration