feat(core): Introduce per-invocation TsDockerSession and session-aware local registry and build orchestration; stream and parse buildx output for improved logging and visibility; detect Docker topology and add CI-safe cleanup; update README with multi-arch, parallel-build, caching, and local registry usage and new CLI flags.

2026-02-07 10:30:52 +00:00
parent 63078139ec
commit 101c4286c1
9 changed files with 500 additions and 167 deletions
--- a/readme.md
+++ b/readme.md
@@ -1,6 +1,6 @@
 # @git.zone/tsdocker

-> 🐳 The ultimate Docker development toolkit for TypeScript projects — build, test, and ship containerized applications with ease.
+> 🐳 The ultimate Docker development toolkit for TypeScript projects — build, test, and ship multi-arch containerized applications with zero friction.

 ## Issue Reporting and Security

@@ -8,15 +8,18 @@ For reporting bugs, issues, or security vulnerabilities, please visit [community

 ## What is tsdocker?

-**tsdocker** is a comprehensive Docker development and building tool that handles everything from testing npm packages in clean environments to building and pushing multi-architecture Docker images across multiple registries.
+**tsdocker** is a comprehensive Docker development and build tool that handles everything from testing npm packages in clean environments to building and pushing multi-architecture Docker images across multiple registries — all from a single CLI.

 ### 🎯 Key Capabilities

 - 🧪 **Containerized Testing** — Run your tests in pristine Docker environments
 - 🏗️ **Smart Docker Builds** — Automatically discover, sort, and build Dockerfiles by dependency
- 🚀 **Multi-Registry Push** — Ship to Docker Hub, GitLab, GitHub Container Registry, and more
- 🔧 **Multi-Architecture** — Build for `amd64` and `arm64` with Docker Buildx
- ⚡ **Zero Config Start** — Works out of the box, scales with your needs
+- 🌍 **True Multi-Architecture** — Build for `amd64` and `arm64` simultaneously with Docker Buildx
+- 🚀 **Multi-Registry Push** — Ship to Docker Hub, GitLab, GitHub Container Registry, and more via OCI Distribution API
+- ⚡ **Parallel Builds** — Level-based parallel builds with configurable concurrency
+- 🗄️ **Persistent Local Registry** — All images flow through a local OCI registry with persistent storage
+- 📦 **Build Caching** — Skip unchanged Dockerfiles with content-hash caching
+- 🔧 **Zero Config Start** — Works out of the box, scales with your needs

 ## Installation

@@ -53,6 +56,7 @@ tsdocker will:
 2. 📊 Analyze `FROM` dependencies between them
 3. 🔄 Sort them topologically
 4. 🏗️ Build each image in the correct order
+5. 📦 Push every image to a persistent local registry (`.nogit/docker-registry/`)

 ### 📤 Push to Registries

@@ -63,33 +67,52 @@ Ship your images to one or all configured registries:
 tsdocker push

 # Push to a specific registry
-tsdocker push registry.gitlab.com
+tsdocker push --registry=registry.gitlab.com
 ```

+Under the hood, `tsdocker push` uses the **OCI Distribution API** to copy images directly from the local registry to remote registries. This means multi-arch manifest lists are preserved end-to-end — no more single-platform-only pushes.
+
 ## CLI Commands

 | Command | Description |
 |---------|-------------|
-| `tsdocker` | Run tests in a fresh Docker container |
+| `tsdocker` | Run tests in a fresh Docker container (legacy mode) |
 | `tsdocker build` | Build all Dockerfiles with dependency ordering |
-| `tsdocker push [registry]` | Push images to configured registries |
+| `tsdocker push` | Build + push images to configured registries |
 | `tsdocker pull <registry>` | Pull images from a specific registry |
-| `tsdocker test` | Run container test scripts (test_*.sh) |
+| `tsdocker test` | Build + run container test scripts (`test_*.sh`) |
 | `tsdocker login` | Authenticate with configured registries |
 | `tsdocker list` | Display discovered Dockerfiles and their dependencies |
-| `tsdocker clean --all` | ⚠️ Aggressively clean Docker environment |
+| `tsdocker clean` | Interactively clean Docker environment |
 | `tsdocker vscode` | Launch containerized VS Code in browser |

+### Build Flags
+
+| Flag | Description |
+|------|-------------|
+| `--platform=linux/arm64` | Override build platform for a single architecture |
+| `--timeout=600` | Build timeout in seconds |
+| `--no-cache` | Force rebuild without Docker layer cache |
+| `--cached` | Skip unchanged Dockerfiles (content-hash based) |
+| `--verbose` | Stream raw `docker build` output |
+| `--parallel` | Enable level-based parallel builds (default concurrency: 4) |
+| `--parallel=8` | Parallel builds with custom concurrency |
+| `--context=mycontext` | Use a specific Docker context |
+
+### Clean Flags
+
+| Flag | Description |
+|------|-------------|
+| `--all` | Include all images and volumes (not just dangling) |
+| `-y` | Auto-confirm all prompts |
+
 ## Configuration

-Configure tsdocker in your `package.json` or `npmextra.json`:
+Configure tsdocker in your `package.json` or `npmextra.json` under the `@git.zone/tsdocker` key:

 ```json
 {
  "@git.zone/tsdocker": {
-    "baseImage": "node:20",
-    "command": "npm test",
-    "dockerSock": false,
    "registries": ["registry.gitlab.com", "docker.io"],
    "registryRepoMap": {
      "registry.gitlab.com": "myorg/myproject"
@@ -98,7 +121,6 @@ Configure tsdocker in your `package.json` or `npmextra.json`:
      "NODE_VERSION": "NODE_VERSION"
    },
    "platforms": ["linux/amd64", "linux/arm64"],
-    "push": false,
    "testDir": "./test"
  }
 }
@@ -106,24 +128,73 @@ Configure tsdocker in your `package.json` or `npmextra.json`:

 ### Configuration Options

-#### Testing Options (Legacy)
-
-| Option | Type | Description |
-|--------|------|-------------|
-| `baseImage` | `string` | Docker image for test environment (default: `hosttoday/ht-docker-node:npmdocker`) |
-| `command` | `string` | Command to run inside container (default: `npmci npm test`) |
-| `dockerSock` | `boolean` | Mount Docker socket for DinD scenarios (default: `false`) |
-
 #### Build & Push Options

-| Option | Type | Description |
-|--------|------|-------------|
-| `registries` | `string[]` | Registry URLs to push to |
-| `registryRepoMap` | `object` | Map registries to different repository paths |
-| `buildArgEnvMap` | `object` | Map Docker build ARGs to environment variables |
-| `platforms` | `string[]` | Target architectures (default: `["linux/amd64"]`) |
-| `push` | `boolean` | Auto-push after build (default: `false`) |
-| `testDir` | `string` | Directory containing test scripts |
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `registries` | `string[]` | `[]` | Registry URLs to push to |
+| `registryRepoMap` | `object` | `{}` | Map registries to different repository paths |
+| `buildArgEnvMap` | `object` | `{}` | Map Docker build ARGs to environment variables |
+| `platforms` | `string[]` | `["linux/amd64"]` | Target architectures for multi-arch builds |
+| `testDir` | `string` | `./test` | Directory containing test scripts |
+
+#### Legacy Testing Options
+
+These options configure the `tsdocker` default command (containerized test runner):
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `baseImage` | `string` | `hosttoday/ht-docker-node:npmdocker` | Docker image for test environment |
+| `command` | `string` | `npmci npm test` | Command to run inside the container |
+| `dockerSock` | `boolean` | `false` | Mount Docker socket for DinD scenarios |
+
+## Architecture: How tsdocker Works
+
+tsdocker uses a **local OCI registry** as the canonical store for all built images. This design solves fundamental problems with Docker's local daemon, which cannot hold multi-architecture manifest lists.
+
+### 📐 Build Flow
+
+```
+┌─────────────────────────────────────────────────┐
+│  tsdocker build                                 │
+│                                                 │
+│  1. Start local registry (localhost:5234)        │
+│     └── Persistent volume: .nogit/docker-registry/
+│                                                 │
+│  2. For each Dockerfile (topological order):     │
+│     ├── Multi-platform: buildx --push → registry │
+│     └── Single-platform: docker build → registry │
+│                                                 │
+│  3. Stop local registry (data persists on disk)  │
+└─────────────────────────────────────────────────┘
+```
+
+### 📤 Push Flow
+
+```
+┌──────────────────────────────────────────────────┐
+│  tsdocker push                                   │
+│                                                  │
+│  1. Start local registry (loads persisted data)   │
+│                                                  │
+│  2. For each image × each remote registry:        │
+│     └── OCI Distribution API copy:               │
+│         ├── Fetch manifest (single or multi-arch) │
+│         ├── Copy blobs (skip if already exist)    │
+│         └── Push manifest with destination tag    │
+│                                                  │
+│  3. Stop local registry                          │
+└──────────────────────────────────────────────────┘
+```
+
+### 🔑 Why a Local Registry?
+
+| Problem | Solution |
+|---------|----------|
+| `docker buildx --load` fails for multi-arch images | `buildx --push` to local registry works for any number of platforms |
+| `docker push` only pushes single-platform manifests | OCI API copy preserves full manifest lists (multi-arch) |
+| Images lost between build and push phases | Persistent storage at `.nogit/docker-registry/` survives restarts |
+| Redundant blob uploads on incremental pushes | HEAD checks skip blobs that already exist on the remote |

 ## Registry Authentication

@@ -140,13 +211,17 @@ export DOCKER_REGISTRY_USER="username"
 export DOCKER_REGISTRY_PASSWORD="password"
 ```

+### Docker Config Fallback
+
+When pushing, tsdocker will also read credentials from `~/.docker/config.json` if no explicit credentials are provided via environment variables. This means `docker login` credentials work automatically.
+
 ### Login Command

 ```bash
 tsdocker login
 ```

-Authenticates with all configured registries.
+Authenticates with all configured registries using the provided environment variables.

 ## Advanced Usage

@@ -162,7 +237,27 @@ Build for multiple platforms using Docker Buildx:
 }
 ```

-tsdocker automatically sets up a Buildx builder when multiple platforms are specified.
+tsdocker automatically:
+- Sets up a Buildx builder with `--driver-opt network=host` (so buildx can reach the local registry)
+- Pushes multi-platform images to the local registry via `buildx --push`
+- Copies the full manifest list (including all platform variants) to remote registries on `tsdocker push`
+
+### ⚡ Parallel Builds
+
+Speed up builds by building independent images concurrently:
+
+```bash
+# Default concurrency (4 workers)
+tsdocker build --parallel
+
+# Custom concurrency
+tsdocker build --parallel=8
+
+# Works with caching too
+tsdocker build --parallel --cached
+```
+
+tsdocker groups Dockerfiles into **dependency levels** using topological analysis. Images within the same level have no dependencies on each other and build in parallel. Each level completes before the next begins.

 ### 📦 Dockerfile Naming Conventions

@@ -190,7 +285,7 @@ COPY . .
 RUN npm run build
 ```

-tsdocker automatically detects that `Dockerfile_app` depends on `Dockerfile_base` and builds them in the correct order.
+tsdocker automatically detects that `Dockerfile_app` depends on `Dockerfile_base`, builds them in the correct order, and makes the base image available to dependent builds via the local registry (using `--build-context` for buildx).

 ### 🧪 Container Test Scripts

@@ -210,6 +305,8 @@ Run with:
 tsdocker test
 ```

+This builds all images, starts the local registry (so multi-arch images can be pulled), and runs each matching test script inside a container.
+
 ### 🔧 Build Args from Environment

 Pass environment variables as Docker build arguments:
@@ -232,6 +329,24 @@ FROM node:${NODE_VERSION}
 RUN echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
 ```

+### 🗺️ Registry Repo Mapping
+
+Use different repository names for different registries:
+
+```json
+{
+  "@git.zone/tsdocker": {
+    "registries": ["registry.gitlab.com", "docker.io"],
+    "registryRepoMap": {
+      "registry.gitlab.com": "mygroup/myproject",
+      "docker.io": "myuser/myproject"
+    }
+  }
+}
+```
+
+When pushing, tsdocker maps the local repo name to the registry-specific path. For example, a locally built `myproject:latest` becomes `registry.gitlab.com/mygroup/myproject:latest` and `docker.io/myuser/myproject:latest`.
+
 ### 🐳 Docker-in-Docker Testing

 Test Docker-related tools by mounting the Docker socket:
@@ -259,68 +374,40 @@ Output:
 Discovered Dockerfiles:
 ========================

-1. Dockerfile_base
+1. /path/to/Dockerfile_base
   Tag: myproject:base
   Base Image: node:20-alpine
   Version: base

-2. Dockerfile_app
+2. /path/to/Dockerfile_app
   Tag: myproject:app
   Base Image: myproject:base
   Version: app
   Depends on: myproject:base
 ```

-### 🗺️ Registry Repo Mapping
-
-Use different repository names for different registries:
-
-```json
-{
-  "@git.zone/tsdocker": {
-    "registries": ["registry.gitlab.com", "docker.io"],
-    "registryRepoMap": {
-      "registry.gitlab.com": "mygroup/myproject",
-      "docker.io": "myuser/myproject"
-    }
-  }
-}
-```
-
-## Environment Variables
-
-### qenv Integration
-
-tsdocker automatically loads environment variables from `qenv.yml`:
-
-```yaml
-# qenv.yml
-API_KEY: your-api-key
-DATABASE_URL: postgres://localhost/test
-```
-
-These are injected into your test container automatically.
-
 ## Examples

-### Basic Test Configuration
+### Minimal Build & Push

 ```json
 {
  "@git.zone/tsdocker": {
-    "baseImage": "node:20",
-    "command": "npm test"
+    "registries": ["docker.io"],
+    "platforms": ["linux/amd64"]
  }
 }
 ```

+```bash
+tsdocker push
+```
+
 ### Full Production Setup

 ```json
 {
  "@git.zone/tsdocker": {
-    "baseImage": "node:20-alpine",
-    "command": "pnpm test",
    "registries": ["registry.gitlab.com", "ghcr.io", "docker.io"],
    "registryRepoMap": {
      "registry.gitlab.com": "myorg/myapp",
@@ -338,57 +425,37 @@ These are injected into your test container automatically.

 ### CI/CD Integration

+**GitLab CI:**
+
 ```yaml
-# .gitlab-ci.yml
-build:
+build-and-push:
  stage: build
  script:
    - npm install -g @git.zone/tsdocker
-    - tsdocker build
    - tsdocker push
+  variables:
+    DOCKER_REGISTRY_1: "registry.gitlab.com|$CI_REGISTRY_USER|$CI_REGISTRY_PASSWORD"
+```

-# GitHub Actions
+**GitHub Actions:**
+
+```yaml
 - name: Build and Push
  run: |
    npm install -g @git.zone/tsdocker
    tsdocker login
-    tsdocker build
    tsdocker push
  env:
    DOCKER_REGISTRY_1: "ghcr.io|${{ github.actor }}|${{ secrets.GITHUB_TOKEN }}"
 ```

-## Requirements
-
- **Docker** — Docker Engine or Docker Desktop must be installed
- **Node.js** — Version 18 or higher (ESM support required)
- **Docker Buildx** — Required for multi-architecture builds (included in Docker Desktop)
-
-## Why tsdocker?
-
-### 🎯 The Problem
-
-Managing Docker workflows manually is tedious:
- Remembering build order for dependent images
- Pushing to multiple registries with different credentials
- Setting up Buildx for multi-arch builds
- Ensuring consistent test environments
-
-### ✨ The Solution
-
-tsdocker automates the entire workflow:
- **One command** to build all images in dependency order
- **One command** to push to all registries
- **Automatic** Buildx setup for multi-platform builds
- **Consistent** containerized test environments
-
 ## TypeScript API

-tsdocker exposes its types for programmatic use:
+tsdocker can also be used programmatically:

 ```typescript
-import type { ITsDockerConfig } from '@git.zone/tsdocker/dist_ts/interfaces/index.js';
 import { TsDockerManager } from '@git.zone/tsdocker/dist_ts/classes.tsdockermanager.js';
+import type { ITsDockerConfig } from '@git.zone/tsdocker/dist_ts/interfaces/index.js';

 const config: ITsDockerConfig = {
  baseImage: 'node:20',
@@ -396,15 +463,21 @@ const config: ITsDockerConfig = {
  dockerSock: false,
  keyValueObject: {},
  registries: ['docker.io'],
-  platforms: ['linux/amd64'],
+  platforms: ['linux/amd64', 'linux/arm64'],
 };

 const manager = new TsDockerManager(config);
 await manager.prepare();
-await manager.build();
+await manager.build({ parallel: true });
 await manager.push();
 ```

+## Requirements
+
+- **Docker** — Docker Engine 20+ or Docker Desktop
+- **Node.js** — Version 18 or higher (for native `fetch` and ESM support)
+- **Docker Buildx** — Required for multi-architecture builds (included in Docker Desktop)
+
 ## Troubleshooting

 ### "docker not found"
@@ -417,11 +490,10 @@ docker --version

 ### Multi-arch build fails

-Make sure Docker Buildx is available:
+Make sure Docker Buildx is available. tsdocker will set up the builder automatically, but you can verify:

 ```bash
 docker buildx version
-docker buildx create --use
 ```

 ### Registry authentication fails
@@ -433,19 +505,22 @@ echo $DOCKER_REGISTRY_1
 tsdocker login
 ```

+tsdocker also falls back to `~/.docker/config.json` — ensure you've run `docker login` for your target registries.
+
 ### Circular dependency detected

 Review your Dockerfiles' `FROM` statements — you have images depending on each other in a loop.

-## Performance Tips
+### Build context too large

-🚀 **Use specific tags**: `node:20-alpine` is smaller and faster than `node:latest`
+Use a `.dockerignore` file to exclude `node_modules`, `.git`, `.nogit`, and other large directories:

-🚀 **Leverage caching**: Docker layers are cached — your builds get faster over time
-
-🚀 **Prune regularly**: `docker system prune` reclaims disk space
-
-🚀 **Use .dockerignore**: Exclude `node_modules`, `.git`, etc. from build context
+```
+node_modules
+.git
+.nogit
+dist_ts
+```

 ## Migration from Legacy