v2.0.0

BREAKING CHANGE(aidoc): migrate agent orchestration to new runAgent API and filesystem tools; refactor model handling and update README and tests
v1.12.0
2026-03-11 18:58:49 +00:00 · 2026-03-11 18:58:48 +00:00 · 2026-01-04 15:20:27 +00:00 · 2026-01-04 15:20:27 +00:00 · 2025-12-16 10:07:47 +00:00 · 2025-12-16 10:07:47 +00:00
21 changed files with 8546 additions and 4698 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -17,4 +17,5 @@ node_modules/
 dist/
 dist_*/
-# custom
+# custom
 **/.claude/settings.local.json
--- a/changelog.md
+++ b/changelog.md
@@ -1,5 +1,191 @@
 # Changelog
 ## 2026-03-11 - 2.0.0 - BREAKING CHANGE(aidoc)
 migrate agent orchestration to new runAgent API and filesystem tools; refactor model handling and update README and tests
 - Replace DualAgentOrchestrator with plugins.smartagent.runAgent and scoped filesystem tools
 - Introduce smartagentTools export and use filesystemTool for agents
 - Replace smartAiInstance with model via plugins.smartai.getModel() and remove previous lifecycle methods (breaking API change)
 - Normalize agent output property from result to text and standardize log messages (removed emojis)
 - Update changelog/README/description generation flows to use new agent interface
 - Bump several devDependencies and dependencies (tsbuild, tstest, @types/node, tspublish, push.rocks packages, typedoc, typescript)
 - Change test entry to export default tap.start()
 - Revise README content and structure
 ## 2026-01-04 - 1.12.0 - feat(commit)
 add token budgeting and dynamic diff token calculation to avoid OpenAI context limit issues
 - Introduce TOKEN_BUDGET constants and calculateMaxDiffTokens() in ts/aidocs_classes/commit.ts
 - Use dynamic maxDiffTokens for DiffProcessor and validate/log warnings when estimated tokens approach limits
 - Add token budgeting notes to readme.hints.md (guidance for splitting large commits and adjusting overhead)
 - Bump dependencies/devDependencies: @git.zone/tstest ^3.1.4, @types/node ^25.0.3, @git.zone/tspublish ^1.11.0, @push.rocks/smartfs ^1.3.1
 ## 2025-12-16 - 1.11.4 - fix(aidocs_classes)
 clarify recommendedNextVersionMessage field to require only the description body without the type(scope) prefix
 - Updated inline documentation in ts/aidocs_classes/commit.ts to explicitly state that recommendedNextVersionMessage must be only the description body (example: 'bump dependency to ^1.2.6') and not include the type(scope) prefix.
 - Removes ambiguity in the example text and improves guidance for commit message generation.
 ## 2025-12-15 - 1.11.0 - feat(commit)
 Integrate DualAgentOrchestrator for commit message generation and improve diff/context handling
 - Add @push.rocks/smartagent dependency and export it from plugins
 - Use DualAgentOrchestrator to generate and guardian-validate commit messages
 - Use DualAgentOrchestrator for changelog generation with guardian validation
 - Switch commit flow to TaskContextFactory and DiffProcessor for token-efficient context
 - Expose getOpenaiToken() and wire orchestrator with the project OpenAI token
 - Enhance iterative context builder and context components to better manage token budgets and sampling
 - Update npmextra.json with release config for @git.zone/cli and reference local smartagent package in package.json
 ## 2025-12-02 - 1.10.0 - feat(diff-processor)
 Improve diff sampling and file prioritization: increase inclusion thresholds, expand sampled context, and boost priority for interface/type and entry-point files
 - Raise small/medium file thresholds used by DiffProcessor (smallFileLines 50 -> 300, mediumFileLines 200 -> 800) so more source files are included fully or summarized rather than treated as large metadata-only files
 - Increase sample window for medium files (sampleHeadLines/sampleTailLines 20 -> 75) to provide more context when summarizing diffs
 - Boost importance scoring for interfaces/type files and entry points (adds +20 for interfaces/.types and +15 for index/mod entry files) to prioritize critical API surface in diff processing
 - Keep other prioritization rules intact (source/test/config/docs/build heuristics), and align the aidoc commit DiffProcessor usage with the new defaults
 ## 2025-11-04 - 1.9.2 - fix(deps)
 Update dependencies and devDependencies to newer versions (bump multiple packages)
 - Bumped devDependencies: @git.zone/tsbuild 2.6.8 -> 2.7.1, @git.zone/tsrun 1.2.46 -> 1.6.2, @git.zone/tstest 2.3.6 -> 2.7.0
 - Bumped runtime dependencies: @push.rocks/smartai 0.5.11 -> 0.8.0, @push.rocks/smartcli 4.0.11 -> 4.0.19, @push.rocks/smartgit 3.2.1 -> 3.3.1, @push.rocks/smartlog 3.1.9 -> 3.1.10, gpt-tokenizer 3.0.1 -> 3.2.0, typedoc 0.28.12 -> 0.28.14, typescript 5.9.2 -> 5.9.3
 - No source code changes in this commit; dependency-only updates. Run the test suite and CI to verify compatibility.
 ## 2025-11-04 - 1.9.1 - fix(iterative-context-builder)
 Rely on DiffProcessor for git diff pre-processing; remove raw char truncation, raise diff token safety, and improve logging
 - Removed raw character-based truncation of additionalContext — diffs are expected to be pre-processed by DiffProcessor instead of blind substring truncation.
 - Now validates pre-processed diff token count only and treats DiffProcessor as the primary sampler (DiffProcessor typically uses a ~100k token budget).
 - Increased MAX_DIFF_TOKENS safety net to 200,000 to cover edge cases and avoid false positives; updated logs to reflect pre-processed diffs.
 - Improved error messaging to indicate a likely DiffProcessor misconfiguration when pre-processed diffs exceed the safety limit.
 - Updated informational logs to state that a pre-processed git diff was added to context.
 ## 2025-11-04 - 1.9.0 - feat(context)
 Add intelligent DiffProcessor to summarize and prioritize git diffs and integrate it into the commit context pipeline
 - Add DiffProcessor (ts/context/diff-processor.ts) to intelligently process git diffs: include small files fully, summarize medium files (head/tail sampling), and mark very large files as metadata-only to stay within token budgets.
 - Integrate DiffProcessor into commit workflow (ts/aidocs_classes/commit.ts): preprocess raw diffs, emit processed diff statistics, and pass a token-efficient diff section into the TaskContextFactory for commit context generation.
 - Export DiffProcessor and its types through the context index and types (ts/context/index.ts, ts/context/types.ts) so other context components can reuse it.
 - Add comprehensive tests for the DiffProcessor behavior and integration (test/test.diffprocessor.node.ts) covering small/medium/large diffs, added/deleted files, prioritization, token budgets, and formatting for context.
 - Minor adjustments across context/task factories and builders to accept and propagate processed diff strings rather than raw diffs, reducing risk of token overflows during iterative context building.
 ## 2025-11-04 - 1.8.3 - fix(context)
 Prevent enormous git diffs and OOM during context building by adding exclusion patterns, truncation, and diagnostic logging
 - Add comprehensive git diff exclusion globs (locks, build artifacts, maps, bundles, IDE folders, logs, caches) when collecting uncommitted diffs to avoid noisy/huge diffs
 - Pass glob patterns directly to smartgit.getUncommittedDiff for efficient server-side matching
 - Emit diagnostic statistics for diffs (files changed, total characters, estimated tokens, number of exclusion patterns) and warn on unusually large diffs
 - Introduce pre-tokenization safety checks in iterative context builder: truncate raw diff text if it exceeds MAX_DIFF_CHARS and throw a clear error if token count still exceeds MAX_DIFF_TOKENS
 - Format and log token counts using locale-aware formatting for clarity
 - Improve robustness of commit context generation to reduce risk of OOM / model-limit overruns
 ## 2025-11-03 - 1.8.0 - feat(context)
 Wire OpenAI provider through task context factory and add git-diff support to iterative context builder
 - Pass AiDoc.openaiInstance through TaskContextFactory into IterativeContextBuilder to reuse the same OpenAI provider and avoid reinitialization.
 - IterativeContextBuilder now accepts an optional OpenAiProvider and an additionalContext string; when provided, git diffs (or other extra context) are prepended to the AI context and token counts are updated.
 - createContextForCommit now forwards the git diff into the iterative builder so commit-specific context includes the diff.
 - Updated aidocs_classes (commit, description, readme) to supply the existing openaiInstance when creating the TaskContextFactory.
 ## 2025-11-03 - 1.7.0 - feat(IterativeContextBuilder)
 Add iterative AI-driven context builder and integrate into task factory; add tests and iterative configuration
 - Introduce IterativeContextBuilder: iterative, token-aware context construction that asks the AI which files to load and evaluates context sufficiency.
 - Switch TaskContextFactory to use IterativeContextBuilder for readme, description and commit tasks (replaces earlier EnhancedContext flow for these tasks).
 - Add iterative configuration options (maxIterations, firstPassFileLimit, subsequentPassFileLimit, temperature, model) in types and ConfigManager and merge support for user config.
 - Update CLI (tokens and aidoc flows) to use the iterative context factory and improve task handling and messaging.
 - Add test coverage: test/test.iterativecontextbuilder.node.ts to validate initialization, iterative builds, token budget respect and multiple task types.
 - Enhance ContextCache, LazyFileLoader, ContextAnalyzer and ContextTrimmer to support the iterative pipeline and smarter prioritization/prompts.
 ## 2025-11-03 - 1.6.1 - fix(context)
 Improve context building, caching and test robustness
 - EnhancedContext: refactored smart context building to use the analyzer and TaskContextFactory by default; taskType now defaults to 'description' and task-specific modes are applied.
 - ConfigManager: simplified analyzer configuration (removed enabled flag) and fixed getAnalyzerConfig fallback shape.
 - ContextCache: more robust mtime handling and persistence; tests updated to use real file mtimes so cache validation works reliably.
 - LazyFileLoader: adjusted token estimation tolerance and improved metadata caching behavior.
 - ContextAnalyzer & trimming pipeline: improved prioritization and trimming integration to better enforce token budgets.
 - Tests: relaxed strict timing/boolean checks and made assertions more tolerant (toEqual vs toBe) to reduce false negatives.
 ## 2025-11-02 - 1.6.0 - feat(context)
 Introduce smart context system: analyzer, lazy loader, cache and README/docs improvements
 - Add ContextAnalyzer for dependency-based file scoring and prioritization (PageRank-like centrality, relevance, efficiency, recency)
 - Add LazyFileLoader to scan metadata and load files in parallel with lightweight token estimates
 - Add ContextCache for persistent file content/token caching with TTL and max-size eviction
 - Enhance ContextTrimmer with tier-based trimming and configurable light/aggressive levels
 - Integrate new components into EnhancedContext and TaskContextFactory to build task-aware, token-optimized contexts
 - Extend ConfigManager and types to support cache, analyzer, prioritization weights and tier configs (npmextra.json driven)
 - Add comprehensive unit tests for ContextAnalyzer, ContextCache and LazyFileLoader
 - Update README with Smart Context Building docs, examples, configuration options and CI workflow snippet
 ## 2025-09-07 - 1.5.2 - fix(package)
 Bump dependencies, refine test script and imports, and overhaul README and docs
 - Bumped multiple dependencies and devDependencies (including @git.zone/tspublish, @git.zone/tsbuild, @git.zone/tstest, @push.rocks/npmextra, @push.rocks/qenv, @push.rocks/smartfile, @push.rocks/smartlog, @push.rocks/smartshell, gpt-tokenizer, typedoc, etc.).
 - Updated test script to run tstest with verbose, logfile and increased timeout; adjusted testCli script invocation.
 - Fixed test import in test/test.aidoc.nonci.ts to use @git.zone/tstest tapbundle.
 - Large README rewrite: reorganized and expanded content, added quick start, CLI commands, examples, configuration, troubleshooting and usage sections.
 - Minor clarification added to commit prompt in ts/aidocs_classes/commit.ts (text cleanup and guidance).
 ## 2025-08-16 - 1.5.1 - fix(aidoc)
 Bump dependencies, add pnpm workspace config, and add AiDoc.stop()
 - Bumped multiple dependencies and devDependencies in package.json (notable upgrades: @git.zone/tsbuild, @git.zone/tspublish, @push.rocks/npmextra, @push.rocks/qenv, @push.rocks/smartai, @push.rocks/smartfile, @push.rocks/smartgit, @push.rocks/smartlog, @push.rocks/smartpath, @push.rocks/smartshell, typedoc, typescript).
 - Added pnpm-workspace.yaml with onlyBuiltDependencies (esbuild, mongodb-memory-server, puppeteer, sharp).
 - Added AiDoc.stop() to properly stop the OpenAI provider (resource/client shutdown).
 - Updated packageManager field in package.json to a newer pnpm version/hash.
 ## 2025-05-14 - 1.5.0 - feat(docs)
 Update project metadata and documentation to reflect comprehensive AI-enhanced features and improved installation and usage instructions
 - Revised descriptions in package.json and npmextra.json to emphasize comprehensive documentation capabilities
 - Expanded README with detailed installation options and extended usage examples for both CLI and API-like integrations
 - Added new dependency (gpt-tokenizer) to support token counting for AI context building
 - Adjusted keywords to better reflect project functionalities such as commit message automation and context trimming
 ## 2025-05-13 - 1.4.5 - fix(dependencies)
 Upgrade various dependency versions and update package manager configuration
 - Bump @git.zone/tsbuild from ^2.1.80 to ^2.3.2
 - Upgrade @push.rocks/tapbundle from ^5.0.23 to ^6.0.3
 - Update @types/node from ^22.8.1 to ^22.15.17
 - Bump @push.rocks/smartai from ^0.4.2 to ^0.5.4
 - Upgrade @push.rocks/smartlog from ^3.0.7 to ^3.0.9
 - Update typedoc from ^0.27.9 to ^0.28.4
 - Bump typescript from ^5.5.2 to ^5.8.3
 - Add packageManager field with pnpm@10.10.0 specification
 ## 2025-02-25 - 1.4.4 - fix(dependencies)
 Update dependencies to latest versions
 - Updated '@push.rocks/smartai' from '^0.0.17' to '^0.4.2'
 - Updated 'typedoc' from '^0.26.1' to '^0.27.9'
 ## 2025-01-14 - 1.4.3 - fix(aidocs_classes)
 Improve readme generation instructions to ensure correct markdown formatting.
 - Added guidance to avoid using backticks at the beginning and end of readme generation to prevent markdown issues.
 - Clarified that the output is directly written to readme.md and backticks should only be used for code blocks.
 ## 2024-10-28 - 1.4.2 - fix(cli)
 Ensure async completion for aidoc readme and description generation
 - Added await statements for asynchronous methods buildReadme and buildDescription in the aidoc command.
 ## 2024-10-28 - 1.4.1 - fix(readme)
 Correct async call to getModuleSubDirs in readme generation.
 - Fixed an issue with asynchronous handling in readme generation for submodules.
 - Ensured that getModuleSubDirs function is called with await to handle promises properly.
 ## 2024-10-28 - 1.4.0 - feat(aidocs)
 Added support for building readmes for sub-modules in aidocs
 - Updated the `Readme` class to handle monorepo projects by generating readmes for sub-modules.
 - Integrated `tspublish` to identify sub-modules for readme generation.
 ## 2024-06-24 - 1.3.12 - fix(aidocs)
 Fix changelog generation by handling leading newlines
--- a/npmextra.json
+++ b/npmextra.json
@@ -9,18 +9,19 @@
      "npmPackagename": "@git.zone/tsdoc",
      "license": "MIT",
      "projectDomain": "git.zone",
-      "description": "An advanced TypeScript documentation tool using AI to generate and enhance documentation for TypeScript projects.",
+      "description": "A comprehensive TypeScript documentation tool that leverages AI to generate and enhance project documentation, including dynamic README creation, API docs via TypeDoc, and smart commit message generation.",
      "keywords": [
        "TypeScript",
-        "documentation generation",
+        "documentation",
-        "AI-enhanced documentation",
+        "AI",
-        "CLI tool",
+        "CLI",
        "README",
        "TypeDoc",
        "commit messages",
        "automation",
        "code analysis",
-        "automated documentation",
+        "context trimming",
-        "developer tools",
+        "developer tools"
        "API documentation",
        "technical writing",
        "code quality improvement"
      ]
    }
  },
@@ -30,5 +31,14 @@
  },
  "tsdoc": {
    "legal": "\n## License and Legal Information\n\nThis repository contains open-source code that is licensed under the MIT License. A copy of the MIT License can be found in the [license](license) file within this repository. \n\n**Please note:** The MIT License does not grant permission to use the trade names, trademarks, service marks, or product names of the project, except as required for reasonable and customary use in describing the origin of the work and reproducing the content of the NOTICE file.\n\n### Trademarks\n\nThis project is owned and maintained by Task Venture Capital GmbH. The names and logos associated with Task Venture Capital GmbH and any related products or services are trademarks of Task Venture Capital GmbH and are not included within the scope of the MIT license granted herein. Use of these trademarks must comply with Task Venture Capital GmbH's Trademark Guidelines, and any usage must be approved in writing by Task Venture Capital GmbH.\n\n### Company Information\n\nTask Venture Capital GmbH  \nRegistered at District court Bremen HRB 35230 HB, Germany\n\nFor any legal inquiries or if you require further information, please contact us via email at hello@task.vc.\n\nBy using this repository, you acknowledge that you have read this section, agree to comply with its terms, and understand that the licensing of the code does not imply endorsement by Task Venture Capital GmbH of any derivative works.\n"
  },
  "@git.zone/cli": {
    "release": {
      "registries": [
        "https://verdaccio.lossless.digital",
        "https://registry.npmjs.org"
      ],
      "accessLevel": "public"
    }
  }
 }
--- a/package.json
+++ b/package.json
@@ -1,8 +1,8 @@
 {
  "name": "@git.zone/tsdoc",
-  "version": "1.3.12",
+  "version": "2.0.0",
  "private": false,
-  "description": "An advanced TypeScript documentation tool using AI to generate and enhance documentation for TypeScript projects.",
+  "description": "A comprehensive TypeScript documentation tool that leverages AI to generate and enhance project documentation, including dynamic README creation, API docs via TypeDoc, and smart commit message generation.",
  "type": "module",
  "exports": {
    ".": "./dist_ts/index.js"
@@ -13,35 +13,37 @@
    "tsdoc": "cli.js"
  },
  "scripts": {
-    "test": "(tstest test/) && npm run testCli",
+    "test": "(tstest test/ --verbose --logfile --timeout 600) && npm run testCli",
    "testCli": "(node ./cli.ts.js) && (node ./cli.ts.js aidocs)",
    "build": "(tsbuild --web --allowimplicitany)",
    "buildDocs": "tsdoc"
  },
  "devDependencies": {
-    "@git.zone/tsbuild": "^2.1.80",
+    "@git.zone/tsbuild": "^4.3.0",
-    "@git.zone/tsrun": "^1.2.46",
+    "@git.zone/tsrun": "^2.0.1",
-    "@git.zone/tstest": "^1.0.90",
+    "@git.zone/tstest": "^3.3.2",
-    "@push.rocks/tapbundle": "^5.0.23",
+    "@types/node": "^25.4.0"
    "@types/node": "^20.14.8"
  },
  "dependencies": {
-    "@push.rocks/early": "^4.0.3",
+    "@git.zone/tspublish": "^1.11.2",
-    "@push.rocks/npmextra": "^5.0.23",
+    "@push.rocks/early": "^4.0.4",
-    "@push.rocks/qenv": "^6.0.5",
+    "@push.rocks/npmextra": "^5.3.3",
-    "@push.rocks/smartai": "^0.0.17",
+    "@push.rocks/qenv": "^6.1.3",
-    "@push.rocks/smartcli": "^4.0.11",
+    "@push.rocks/smartagent": "^3.0.2",
    "@push.rocks/smartai": "^2.0.0",
    "@push.rocks/smartcli": "^4.0.20",
    "@push.rocks/smartdelay": "^3.0.5",
-    "@push.rocks/smartfile": "^11.0.20",
+    "@push.rocks/smartfile": "^13.1.2",
-    "@push.rocks/smartgit": "^3.1.0",
+    "@push.rocks/smartfs": "^1.5.0",
-    "@push.rocks/smartinteract": "^2.0.15",
+    "@push.rocks/smartgit": "^3.3.1",
-    "@push.rocks/smartlog": "^3.0.7",
+    "@push.rocks/smartinteract": "^2.0.16",
    "@push.rocks/smartlog": "^3.2.1",
    "@push.rocks/smartlog-destination-local": "^9.0.2",
-    "@push.rocks/smartpath": "^5.0.18",
+    "@push.rocks/smartpath": "^6.0.0",
-    "@push.rocks/smartshell": "^3.0.5",
+    "@push.rocks/smartshell": "^3.3.7",
-    "@push.rocks/smarttime": "^4.0.6",
+    "@push.rocks/smarttime": "^4.2.3",
-    "typedoc": "^0.26.1",
+    "typedoc": "^0.28.17",
-    "typescript": "^5.5.2"
+    "typescript": "^5.9.3"
  },
  "files": [
    "ts/**/*",
@@ -60,15 +62,16 @@
  ],
  "keywords": [
    "TypeScript",
-    "documentation generation",
+    "documentation",
-    "AI-enhanced documentation",
+    "AI",
-    "CLI tool",
+    "CLI",
    "README",
    "TypeDoc",
    "commit messages",
    "automation",
    "code analysis",
-    "automated documentation",
+    "context trimming",
-    "developer tools",
+    "developer tools"
    "API documentation",
    "technical writing",
    "code quality improvement"
  ],
  "repository": {
    "type": "git",
@@ -77,5 +80,6 @@
  "bugs": {
    "url": "https://gitlab.com/gitzone/tsdoc/issues"
  },
-  "homepage": "https://gitlab.com/gitzone/tsdoc#readme"
+  "homepage": "https://gitlab.com/gitzone/tsdoc#readme",
  "packageManager": "pnpm@10.14.0+sha512.ad27a79641b49c3e481a16a805baa71817a04bbe06a38d17e60e2eaee83f6a146c6a688125f5792e48dd5ba30e7da52a5cda4c3992b9ccf333f9ce223af84748"
 }
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
--- a/readme.hints.md
+++ b/readme.hints.md
@@ -2,4 +2,13 @@
 * alternatively can be used through npx, if installed locally
 * cli parameters are concluded from ./ts/cli.ts
 * this module is not intended for API use.
-* Read carefully through the TypeScript files. Don't make stuff up.
+* Read carefully through the TypeScript files. Don't make stuff up.
 ## Token Budgeting (commit.ts)
 * OpenAI has a 272,000 token context limit
 * The smartagent infrastructure adds ~180,000 tokens of overhead (system messages, tool descriptions, conversation history)
 * TOKEN_BUDGET constants in commit.ts control the available tokens for diff content
 * Dynamic calculation: 272K - 10K (safety) - 180K (overhead) - 2K (prompt) = 80K tokens for diff
 * If token limit errors occur, consider:
  - Splitting large commits into smaller ones
  - Adjusting SMARTAGENT_OVERHEAD if actual overhead is different
--- a/readme.md
+++ b/readme.md
@@ -1,312 +1,215 @@
 # @git.zone/tsdoc
-An advanced TypeScript documentation tool using AI to generate and enhance documentation for TypeScript projects.
+AI-Powered Documentation for TypeScript Projects
 ## Issue Reporting and Security
 For reporting bugs, issues, or security vulnerabilities, please visit [community.foss.global/](https://community.foss.global/). This is the central community hub for all issue reporting. Developers who sign and comply with our contribution agreement and go through identification can also get a [code.foss.global/](https://code.foss.global/) account to submit Pull Requests directly.
 ## Install
 To install `@git.zone/tsdoc`, you can either install it globally or use it with `npx`.
 ### Global Installation
 You can install `@git.zone/tsdoc` globally on your system using npm:
 ```bash
-npm install -g @git.zone/tsdoc
+# Global installation (recommended for CLI usage)
-```
+pnpm add -g @git.zone/tsdoc
-### Usage with npx
+# Or use with npx
 npx @git.zone/tsdoc
-If you prefer not to install it globally, you can use it with `npx`:
+# Or install locally as a dependency
-
+pnpm add @git.zone/tsdoc
 ```bash
 npx @git.zone/tsdoc <command>
 ```
 ## Usage
-`@git.zone/tsdoc` provides a comprehensive CLI tool and advanced AI-enhanced features to generate and enhance documentation for your TypeScript projects.
+`@git.zone/tsdoc` is a comprehensive TypeScript documentation tool that combines traditional TypeDoc API documentation generation with AI-powered documentation workflows. It uses OpenAI models via the Vercel AI SDK to generate READMEs, project descriptions, keywords, and semantic commit messages by intelligently exploring your project with agentic tool use.
-### Using the CLI
+### CLI Commands
-The primary interface for `@git.zone/tsdoc` is through its command-line tool. Below, we'll explore the different commands available and provide examples of how to use them.
+| Command | Description |
 |---------|-------------|
 | `tsdoc` | Auto-detects project type and runs TypeDoc |
 | `tsdoc aidoc` | Generates AI-powered README + description/keywords |
 | `tsdoc readme` | Generates AI-powered README only |
 | `tsdoc description` | Generates AI-powered description and keywords only |
 | `tsdoc commit` | Generates a semantic commit message from uncommitted changes |
 | `tsdoc typedoc` | Generates traditional TypeDoc API documentation |
-### Commands
+### Generating AI-Powered Documentation
-#### `tsdoc` - Auto-detect Documentation Format
+The `aidoc` command combines README generation and description/keyword generation in one step:
 The standard command will automatically detect the documentation format of your project and generate the appropriate documentation.
 ```bash
 tsdoc
 ```
 ### Example
 ```typescript
 // In a TypeScript project, run the above command.
 ```
 #### `tsdoc typedoc` - Generate Documentation using TypeDoc
 The `typedoc` command will generate documentation compliant with the TypeDoc format.
 ```bash
 tsdoc typedoc --publicSubdir docs
 ```
 ### Example
 ```typescript
 import * as plugins from '@push.rocks/smartfile';
 const tsconfigPath = plugins.path.join(__dirname, 'tsconfig.json');
 const outputPath = plugins.path.join(__dirname, 'docs');
 await plugins.smartshellInstance.exec(
  `typedoc --tsconfig ${tsconfigPath} --out ${outputPath} index.ts`,
 );
 ```
 #### `tsdoc aidoc` - Generate AI-Enhanced Documentation
 The `aidoc` command will use AI to generate an enhanced README and update your project description.
 ```bash
 # In your project root
 tsdoc aidoc
 ```
-### Example
+This will:
 1. Analyze your codebase using an AI agent with filesystem access
 2. Generate a comprehensive `readme.md`
 3. Update `package.json` and `npmextra.json` with an AI-generated description and keywords
-```typescript
+You can also run these separately:
 import { AiDoc } from '@git.zone/tsdoc';
 (async () => {
  const aidoc = new AiDoc();
  await aidoc.start();
  await aidoc.buildReadme('./');
  await aidoc.buildDescription('./');
 })();
 ```
 #### `tsdoc test` - Test Your Documentation Setup
 The `test` command will test your current documentation setup, ensuring everything is configured correctly.
 ```bash
-tsdoc test
+# Generate only the README
 tsdoc readme
 # Generate only the description and keywords
 tsdoc description
 ```
-### Example
+### Generating Commit Messages
-```typescript
+The `commit` command analyzes your uncommitted changes and produces a structured commit object:
 import * as plugins from '@git.zone/tsdoc';
-await plugins.runCli().catch((err) => {
+```bash
-  console.error(err);
+tsdoc commit
  process.exit(1);
 });
 ```
-## Features in Depth
+Output is a JSON object following conventional commits:
-### Using Plugins
+```json
-
+{
-`@git.zone/tsdoc` extensively uses plugins to extend its capabilities.
+  "recommendedNextVersionLevel": "feat",
-
+  "recommendedNextVersionScope": "core",
-### Available Plugins
+  "recommendedNextVersionMessage": "add new feature for better documentation",
-
+  "recommendedNextVersionDetails": [
- **npmextra**: Manage npm project-related configurations.
+    "implemented X",
- **qenv**: Environment variable management.
+    "refactored Y"
- **smartai**: AI integration.
+  ],
- **smartcli**: CLI helper.
+  "recommendedNextVersion": "1.13.0",
- **smartdelay**: Simple delay utility.
+  "changelog": "# Changelog\n\n## 2026-03-11 - 1.13.0 - core\n..."
- **smartfile**: File system utilities.
+}
 - **smartinteract**: Interaction helper.
 - **smartlog**: Logging utility.
 - **smartlogDestinationLocal**: Local file destination for logging.
 - **smartpath**: Path utilities.
 - **smartshell**: Shell command execution.
 - **typedoc**: Documentation generation.
 ### Example Usage of Plugins
 #### Path Management
 ```typescript
 import * as path from 'path';
 const packageDir = path.join(__dirname, '../');
 const cwd = process.cwd();
 const binDir = path.join(packageDir, './node_modules/.bin');
 const assetsDir = path.join(packageDir, './assets');
 const publicDir = path.join(cwd, './public');
 const tsDir = path.join(cwd, './ts');
 const tsconfigFile = path.join(assetsDir, './tsconfig.json');
 const typedocOptionsFile = path.join(assetsDir, './typedoc.json');
 ```
-#### Logging
+The commit command includes intelligent diff processing that:
 - Excludes lock files, build artifacts, IDE directories, and caches from the diff
 - Prioritizes source files over build artifacts
 - Samples large diffs with head/tail extraction to stay within token budgets
 - Automatically generates or updates the changelog
-```typescript
+### Generating TypeDoc
 import * as plugins from './plugins';
-const logger = new plugins.smartlog.Smartlog({
+For traditional API documentation via TypeDoc:
  logContext: {
    company: 'Some Company',
    companyunit: 'Some CompanyUnit',
    containerName: 'Some Containername',
    environment: 'local',
    runtime: 'node',
    zone: 'gitzone',
  },
  minimumLogLevel: 'silly',
 });
-logger.addLogDestination(new plugins.smartlogDestinationLocal.DestinationLocal());
+```bash
 # Generate to default ./public directory
 tsdoc typedoc
 # Generate to a specific subdirectory
 tsdoc typedoc --publicSubdir docs
 ```
-## Advanced Usage
+### Monorepo Support
-### Using `TypeDoc` Class
+When generating READMEs, tsdoc automatically detects monorepo submodules via `@git.zone/tspublish` conventions. Each submodule directory with a `tspublish.json` gets its own generated README.
-The `TypeDoc` class provides utility methods to compile TypeScript documentation.
+### Programmatic API
-```typescript
+You can also use tsdoc programmatically:
 import { TypeDoc } from '@git.zone/tsdoc/classes.typedoc';
 const typeDocInstance = new TypeDoc(paths.cwd);
 await typeDocInstance.compile({
  publicSubdir: 'docs',
 });
 ```
 ### Using `AiDoc` Class
 The `AiDoc` class integrates with AI services to enhance your documentation.
 #### Initializing and Using AI
 ```typescript
 import { AiDoc } from '@git.zone/tsdoc';
-const aiDoc = new AiDoc({ OPENAI_TOKEN: 'your-openai-token' });
+const aidoc = new AiDoc();
 await aidoc.start(); // Initializes the AI model (prompts for OpenAI token if needed)
-await aiDoc.start();
+// Generate a README
-await aiDoc.buildReadme('./');
+await aidoc.buildReadme('/path/to/project');
-await aiDoc.buildDescription('./');
+
 // Generate description and keywords
 await aidoc.buildDescription('/path/to/project');
 // Generate a commit message object
 const commitObj = await aidoc.buildNextCommitObject('/path/to/project');
 console.log(commitObj);
 // Get project context information
 const context = await aidoc.getProjectContext('/path/to/project');
 // Get token count for a project
 const tokenCount = await aidoc.getProjectContextTokenCount('/path/to/project');
 // Estimate tokens in arbitrary text
 const tokens = aidoc.countTokens('some text here');
 await aidoc.stop();
 ```
-#### Retrieving AI Tokens
+## Configuration
-```typescript
+### OpenAI Token
 import * as plugins from '@git.zone/tsdoc/plugins';
-const qenv = new plugins.qenv.Qenv();
+An OpenAI API key is required for all AI features. It can be provided in three ways:
 const openaiToken = await qenv.getEnvVarOnDemand('OPENAI_TOKEN');
 ```
-### Testing
+1. **Environment variable**: `OPENAI_TOKEN`
 2. **Interactive prompt**: On first run, tsdoc will prompt for the token and persist it
 3. **Constructor argument**: Pass `{ OPENAI_TOKEN: 'sk-...' }` to `new AiDoc()`
-The provided tests demonstrate how to verify the functionality of the `@git.zone/tsdoc` tool.
+The token is persisted at `~/.npmextra/kv/@git.zone/tsdoc.json` for subsequent runs.
-#### Example Test Script
+### npmextra.json
-```typescript
+tsdoc uses `npmextra.json` for project metadata. The `tsdoc` key holds legal information that gets appended to generated READMEs:
 import { expect, tap } from '@push.rocks/tapbundle';
 import * as tsdoc from '../ts/index';
-tap.test('should create AiDoc instance', async () => {
+```json
-  const aidoc = new tsdoc.AiDoc({ OPENAI_TOKEN: 'dummy-token' });
+{
-  expect(aidoc).toBeInstanceOf(tsdoc.AiDoc);
+  "tsdoc": {
-});
+    "legal": "\n## License and Legal Information\n\n..."
-
+  },
-tap.test('should start AiDoc', async () => {
+  "gitzone": {
-  const aidoc = new tsdoc.AiDoc({ OPENAI_TOKEN: 'dummy-token' });
+    "module": {
-  await aidoc.start();
+      "githost": "gitlab.com",
-  await aidoc.buildReadme('./');
+      "gitscope": "gitzone",
-  await aidoc.buildDescription('./');
+      "gitrepo": "tsdoc",
-});
+      "npmPackagename": "@git.zone/tsdoc",
-
+      "description": "...",
-tap.start();
+      "keywords": ["..."]
 ```
 ### Internals
 The `@git.zone/tsdoc` consists of several internal classes and utilities that streamline its functionality.
 #### File Paths Management
 Located in `ts/paths.ts`, the file defines various directories and file paths used by the tool.
 ```typescript
 import * as plugins from './plugins';
 export const packageDir = plugins.path.join(
  plugins.smartpath.get.dirnameFromImportMetaUrl(import.meta.url),
  '../',
 );
 export const cwd = process.cwd();
 export const binDir = plugins.path.join(packageDir, './node_modules/.bin');
 export const assetsDir = plugins.path.join(packageDir, './assets');
 export const publicDir = plugins.path.join(cwd, './public');
 export const tsDir = plugins.path.join(cwd, './ts');
 export const tsconfigFile = plugins.path.join(assetsDir, './tsconfig.json');
 export const typedocOptionsFile = plugins.path.join(assetsDir, './typedoc.json');
 ```
 #### Utility Commands
 Define utility commands that streamline various processes.
 ##### Example: SmartCLI Usage
 ```typescript
 // Import required modules and plugins
 import * as plugins from './plugins';
 import * as paths from './paths';
 // TypeDoc and AiDoc classes
 import { TypeDoc } from './classes.typedoc';
 import { AiDoc } from './classes.aidoc';
 // Export a run function
 export const run = async () => {
  const tsdocCli = new plugins.smartcli.Smartcli();
  tsdocCli.standardCommand().subscribe(async (argvArg) => {
    switch (true) {
      case await TypeDoc.isTypeDocDir(paths.cwd):
        tsdocCli.triggerCommand('typedoc', argvArg);
        break;
      default:
        console.error(`Cannot determine docs format at ${paths.cwd}`);
    }
-  });
+  }
-
+}
  tsdocCli.addCommand('typedoc').subscribe(async (argvArg) => {
    const typeDocInstance = new TypeDoc(paths.cwd);
    await typeDocInstance.compile({
      publicSubdir: argvArg.publicSubdir,
    });
  });
  tsdocCli.addCommand('aidoc').subscribe(async (argvArg) => {
    const aidocInstance = new AiDoc(argvArg);
    await aidocInstance.start();
    await aidocInstance.buildReadme(paths.cwd);
    await aidocInstance.buildDescription(paths.cwd);
  });
  tsdocCli.startParse();
 };
 ```
-By leveraging these functionalities, you can configure and extend `@git.zone/tsdoc` to fit your specific documentation generation needs.
+## Architecture
-### Further Enhancements
+### Core Components
-The `@git.zone/tsdoc` tool is designed to be extensible. Explore the source files and plugins to add more functionality or integrate with other tools as needed. This README provides an extensive overview of the commands and features but it's always beneficial to dive into the source code to understand the intricacies and potential customizations. Happy documenting!
+```
@git.zone/tsdoc
 ├── AiDoc                 # Main orchestrator - manages AI model and delegates to task classes
 ├── TypeDoc               # Traditional TypeDoc API documentation generation
 ├── ProjectContext        # Gathers project files for context (package.json, ts/, test/)
 ├── DiffProcessor         # Intelligent git diff processing with prioritization and sampling
 ├── Readme                # AI-powered README generation using runAgent with filesystem tools
 ├── Commit                # AI-powered commit message generation with diff analysis
 ├── Description           # AI-powered description and keyword generation
 └── CLI                   # Command-line interface built on @push.rocks/smartcli
 ```
 ### AI Agent Architecture
 tsdoc uses `@push.rocks/smartagent`'s `runAgent()` function with `@push.rocks/smartai`'s `getModel()` for all AI tasks. Each documentation task (readme, commit, description) runs an autonomous AI agent that:
 1. Receives a system prompt defining its role and constraints
 2. Gets access to scoped filesystem tools (read-only, limited to project directory)
 3. Explores the project structure autonomously using tool calls
 4. Produces the final output (README markdown, commit JSON, or description JSON)
 ### Diff Processing
 The `DiffProcessor` class handles large git diffs intelligently:
 - **Small files** (< 300 lines): Included in full
 - **Medium files** (< 800 lines): Head/tail sampling with context
 - **Large files**: Metadata only (filepath, lines added/removed)
 - Files are prioritized by importance: source > test > config > docs > build artifacts
 - Token budget is enforced dynamically based on OpenAI's context limits
 ## Requirements
 - **Node.js** >= 18.0.0
 - **TypeScript** project with `ts/` source directory
 - **OpenAI API key** (for AI features)
 ## License and Legal Information
@@ -320,7 +223,7 @@ This project is owned and maintained by Task Venture Capital GmbH. The names and
 ### Company Information
-Task Venture Capital GmbH  
+Task Venture Capital GmbH
 Registered at District court Bremen HRB 35230 HB, Germany
 For any legal inquiries or if you require further information, please contact us via email at hello@task.vc.
--- a/readme.plan.md
+++ b/readme.plan.md
@@ -0,0 +1,314 @@
 # TSDocs Context Optimization Plan
 ## Problem Statement
 For large TypeScript projects, the context generated for AI-based documentation creation becomes too large, potentially exceeding even o4-mini's 200K token limit. This affects the ability to effectively generate:
 - Project documentation (README.md)
 - API descriptions and keywords
 - Commit messages and changelogs
 Current implementation simply includes all TypeScript files and key project files, but lacks intelligent selection, prioritization, or content reduction mechanisms.
 ## Analysis of Approaches
 ### 1. Smart Content Selection
 **Description:** Intelligently select only files that are necessary for the specific task being performed, using heuristic rules.
 **Advantages:**
 - Simple to implement
 - Predictable behavior
 - Can be fine-tuned for different operations
 **Disadvantages:**
 - Requires manual tuning of rules
 - May miss important context in complex projects
 - Static approach lacks adaptability
 **Implementation Complexity:** Medium
 ### 2. File Prioritization
 **Description:** Rank files by relevance using git history, file size, import/export analysis, and relationship to the current task.
 **Advantages:**
 - Adaptively includes the most relevant files first
 - Maintains context for frequently changed or central files
 - Can leverage git history for additional signals
 **Disadvantages:**
 - Complexity in determining accurate relevance scores
 - Requires analyzing project structure
 - May require scanning imports/exports for dependency analysis
 **Implementation Complexity:** High
 ### 3. Chunking Strategy
 **Description:** Process the project in logical segments, generating intermediate results that are then combined to create the final output.
 **Advantages:**
 - Can handle projects of any size
 - Focused context for each specific part
 - May improve quality by focusing on specific areas deeply
 **Disadvantages:**
 - Complex orchestration of multiple AI calls
 - Challenge in maintaining consistency across chunks
 - May increase time and cost for processing
 **Implementation Complexity:** High
 ### 4. Dynamic Context Trimming
 **Description:** Automatically reduce context by removing non-essential code while preserving structure. Techniques include:
 - Removing implementation details but keeping interfaces and type definitions
 - Truncating large functions while keeping signatures
 - Removing comments and whitespace (except JSDoc)
 - Keeping only imports/exports for context files
 **Advantages:**
 - Preserves full project structure
 - Flexible token usage based on importance
 - Good balance between completeness and token efficiency
 **Disadvantages:**
 - Potential to remove important implementation details
 - Risk of missing context needed for specific tasks
 - Complex rules for what to trim vs keep
 **Implementation Complexity:** Medium
 ### 5. Embeddings-Based Retrieval
 **Description:** Create vector embeddings of project files and retrieve only the most relevant ones for a specific task using semantic similarity.
 **Advantages:**
 - Highly adaptive to different types of requests
 - Leverages semantic understanding of content
 - Can scale to extremely large projects
 **Disadvantages:**
 - Requires setting up and managing embeddings database
 - Added complexity of running vector similarity searches
 - Higher resource requirements for maintaining embeddings
 **Implementation Complexity:** Very High
 ### 6. Task-Specific Contexts
 **Description:** Create separate optimized contexts for different tasks (readme, commit messages, etc.) with distinct file selection and processing strategies.
 **Advantages:**
 - Highly optimized for each specific task
 - Efficient token usage for each operation
 - Improved quality through task-focused contexts
 **Disadvantages:**
 - Maintenance of multiple context building strategies
 - More complex configuration
 - Potential duplication in implementation
 **Implementation Complexity:** Medium
 ### 7. Recursive Summarization
 **Description:** Summarize larger files first, then include these summaries in the final context along with smaller files included in full.
 **Advantages:**
 - Can handle arbitrary project sizes
 - Preserves essential information from all files
 - Balanced approach to token usage
 **Disadvantages:**
 - Quality loss from summarization
 - Increased processing time from multiple AI calls
 - Complex orchestration logic
 **Implementation Complexity:** High
 ## Implementation Strategy
 We propose a phased implementation approach, starting with the most impactful and straightforward approaches, then building toward more complex solutions as needed:
 ### Phase 1: Foundation (1-2 weeks)
 1. **Implement Dynamic Context Trimming**
   - Create a `ContextProcessor` class that takes SmartFile objects and applies trimming rules
   - Implement configurable trimming rules (remove implementations, keep signatures)
   - Add a configuration option to control trimming aggressiveness
   - Support preserving JSDoc comments while removing other comments
 2. **Enhance Token Monitoring**
   - Track token usage per file to identify problematic files
   - Implement token budgeting to stay within limits
   - Add detailed token reporting for optimization
 ### Phase 2: Smart Selection (2-3 weeks)
 3. **Implement Task-Specific Contexts**
   - Create specialized context builders for readme, commit messages, and descriptions
   - Customize file selection rules for each task
   - Add configuration options for task-specific settings
 4. **Add Smart Content Selection**
   - Implement heuristic rules for file importance
   - Create configuration for inclusion/exclusion patterns
   - Add ability to focus on specific directories or modules
 ### Phase 3: Advanced Techniques (3-4 weeks)
 5. **Implement File Prioritization**
   - Add git history analysis to identify frequently changed files
   - Implement dependency analysis to identify central files
   - Create a scoring system for file relevance
 6. **Add Optional Recursive Summarization**
   - Implement file summarization for large files
   - Create a hybrid approach that mixes full files and summaries
   - Add configuration to control summarization thresholds
 ### Phase 4: Research-Based Approaches (Future Consideration)
 7. **Research and Evaluate Embeddings-Based Retrieval**
   - Prototype embeddings creation for TypeScript files
   - Evaluate performance and accuracy
   - Implement if benefits justify the complexity
 8. **Explore Chunking Strategies**
   - Research effective chunking approaches for documentation
   - Prototype and evaluate performance
   - Implement if benefits justify the complexity
 ## Technical Design
 ### Core Components
 1. **ContextBuilder** - Enhanced version of current ProjectContext
   ```typescript
   interface IContextBuilder {
     buildContext(): Promise<string>;
     getTokenCount(): number;
     setContextMode(mode: 'normal' | 'trimmed' | 'summarized'): void;
     setTokenBudget(maxTokens: number): void;
     setPrioritizationStrategy(strategy: IPrioritizationStrategy): void;
   }
   ```
 2. **FileProcessor** - Handles per-file processing and trimming
   ```typescript
   interface IFileProcessor {
     processFile(file: SmartFile): Promise<string>;
     setProcessingMode(mode: 'full' | 'trim' | 'summarize'): void;
     getTokenCount(): number;
   }
   ```
 3. **PrioritizationStrategy** - Ranks files by importance
   ```typescript
   interface IPrioritizationStrategy {
     rankFiles(files: SmartFile[], context: string): Promise<SmartFile[]>;
     setImportanceMetrics(metrics: IImportanceMetrics): void;
   }
   ```
 4. **TaskContextFactory** - Creates optimized contexts for specific tasks
   ```typescript
   interface ITaskContextFactory {
     createContextForReadme(projectDir: string): Promise<string>;
     createContextForCommit(projectDir: string, diff: string): Promise<string>;
     createContextForDescription(projectDir: string): Promise<string>;
   }
   ```
 ### Configuration Options
 The system will support configuration via a new section in `npmextra.json`:
 ```json
 {
  "tsdoc": {
    "context": {
      "maxTokens": 190000,
      "defaultMode": "dynamic",
      "taskSpecificSettings": {
        "readme": {
          "mode": "full",
          "includePaths": ["src/", "lib/"],
          "excludePaths": ["test/", "examples/"]
        },
        "commit": {
          "mode": "trimmed",
          "focusOnChangedFiles": true
        },
        "description": {
          "mode": "summarized",
          "includePackageInfo": true
        }
      },
      "trimming": {
        "removeImplementations": true,
        "preserveInterfaces": true,
        "preserveTypeDefs": true,
        "preserveJSDoc": true,
        "maxFunctionLines": 5
      }
    }
  }
 }
 ```
 ## Cost-Benefit Analysis
 ### Cost Considerations
 1. **Development costs**
   - Initial implementation of foundational components (~30-40 hours)
   - Testing and validation across different project sizes (~10-15 hours)
   - Documentation and configuration examples (~5 hours)
 2. **Operational costs**
   - Potential increased processing time for context preparation
   - Additional API calls for summarization or embeddings approaches
   - Monitoring and maintenance of the system
 ### Benefits
 1. **Scalability**
   - Support for projects of any size, up to and beyond o4-mini's 200K token limit
   - Future-proof design that can adapt to different models and token limits
 2. **Quality improvements**
   - More focused contexts lead to better AI outputs
   - Task-specific optimization improves relevance
   - Consistent performance regardless of project size
 3. **User experience**
   - Predictable behavior for all project sizes
   - Transparent token usage reporting
   - Configuration options for different usage patterns
 ## First Deliverable
 For immediate improvements, we recommend implementing Dynamic Context Trimming and Task-Specific Contexts first, as these offer the best balance of impact and implementation complexity.
 ### Implementation Plan for Dynamic Context Trimming
 1. Create a basic `ContextTrimmer` class that processes TypeScript files:
   - Remove function bodies but keep signatures
   - Preserve interface and type definitions
   - Keep imports and exports
   - Preserve JSDoc comments
 2. Integrate with the existing ProjectContext class:
   - Add a trimming mode option
   - Apply trimming during the context building process
   - Track and report token savings
 3. Modify the CLI to support trimming options:
   - Add a `--trim` flag to enable trimming
   - Add a `--trim-level` option for controlling aggressiveness
   - Show token usage with and without trimming
 This approach could reduce token usage by 40-70% while preserving the essential structure of the codebase, making it suitable for large projects while maintaining high-quality AI outputs.
--- a/test/test.aidoc.nonci.ts
+++ b/test/test.aidoc.nonci.ts
@@ -1,4 +1,4 @@
-import { tap, expect } from '@push.rocks/tapbundle';
+import { tap, expect } from '@git.zone/tstest/tapbundle';
 import * as qenv from '@push.rocks/qenv';
 let testQenv = new qenv.Qenv('./', '.nogit/');
@@ -33,7 +33,10 @@ tap.test('should build commit object', async () => {
  expect(commitObject).toHaveProperty('recommendedNextVersionLevel');
  expect(commitObject).toHaveProperty('recommendedNextVersionScope');
  expect(commitObject).toHaveProperty('recommendedNextVersionMessage');
 });
-})
+tap.test('should stop AIdocs', async () => {
  await aidocs.stop();
 });
-tap.start();
+export default tap.start();
--- a/test/test.diffprocessor.node.ts
+++ b/test/test.diffprocessor.node.ts
@@ -0,0 +1,304 @@
 import { tap, expect } from '@git.zone/tstest/tapbundle';
 import { DiffProcessor } from '../ts/classes.diffprocessor.js';
 // Sample diff strings for testing
 const createSmallDiff = (filepath: string, addedLines = 5, removedLines = 3): string => {
  const lines: string[] = [];
  lines.push(`--- a/${filepath}`);
  lines.push(`+++ b/${filepath}`);
  lines.push(`@@ -1,10 +1,12 @@`);
  for (let i = 0; i < removedLines; i++) {
    lines.push(`-removed line ${i + 1}`);
  }
  for (let i = 0; i < addedLines; i++) {
    lines.push(`+added line ${i + 1}`);
  }
  lines.push(' unchanged line');
  return lines.join('\n');
 };
 const createMediumDiff = (filepath: string): string => {
  const lines: string[] = [];
  lines.push(`--- a/${filepath}`);
  lines.push(`+++ b/${filepath}`);
  lines.push(`@@ -1,100 +1,150 @@`);
  // 150 lines of changes
  for (let i = 0; i < 75; i++) {
    lines.push(`+added line ${i + 1}`);
  }
  for (let i = 0; i < 75; i++) {
    lines.push(`-removed line ${i + 1}`);
  }
  return lines.join('\n');
 };
 const createLargeDiff = (filepath: string): string => {
  const lines: string[] = [];
  lines.push(`--- a/${filepath}`);
  lines.push(`+++ b/${filepath}`);
  lines.push(`@@ -1,1000 +1,1500 @@`);
  // 2500 lines of changes
  for (let i = 0; i < 1250; i++) {
    lines.push(`+added line ${i + 1}`);
  }
  for (let i = 0; i < 1250; i++) {
    lines.push(`-removed line ${i + 1}`);
  }
  return lines.join('\n');
 };
 const createDeletedFileDiff = (filepath: string): string => {
  return `--- a/${filepath}
 +++ /dev/null
@@ -1,5 +0,0 @@
 -deleted line 1
 -deleted line 2
 -deleted line 3
 -deleted line 4
 -deleted line 5`;
 };
 const createAddedFileDiff = (filepath: string): string => {
  return `--- /dev/null
 +++ b/${filepath}
@@ -0,0 +1,5 @@
 +added line 1
 +added line 2
 +added line 3
 +added line 4
 +added line 5`;
 };
 tap.test('DiffProcessor should parse small diff correctly', async () => {
  const processor = new DiffProcessor();
  const smallDiff = createSmallDiff('src/test.ts', 5, 3);
  const result = processor.processDiffs([smallDiff]);
  expect(result.totalFiles).toEqual(1);
  expect(result.fullDiffs.length).toEqual(1);
  expect(result.summarizedDiffs.length).toEqual(0);
  expect(result.metadataOnly.length).toEqual(0);
  expect(result.totalTokens).toBeGreaterThan(0);
 });
 tap.test('DiffProcessor should summarize medium diff', async () => {
  const processor = new DiffProcessor();
  const mediumDiff = createMediumDiff('src/medium-file.ts');
  const result = processor.processDiffs([mediumDiff]);
  expect(result.totalFiles).toEqual(1);
  expect(result.fullDiffs.length).toEqual(0);
  expect(result.summarizedDiffs.length).toEqual(1);
  expect(result.metadataOnly.length).toEqual(0);
  // Verify the summarized diff contains the sample
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('SUMMARIZED DIFFS');
  expect(formatted).toInclude('lines omitted');
 });
 tap.test('DiffProcessor should handle large diff as metadata only', async () => {
  const processor = new DiffProcessor();
  const largeDiff = createLargeDiff('dist/bundle.js');
  const result = processor.processDiffs([largeDiff]);
  expect(result.totalFiles).toEqual(1);
  expect(result.fullDiffs.length).toEqual(0);
  expect(result.summarizedDiffs.length).toEqual(0);
  expect(result.metadataOnly.length).toEqual(1);
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('METADATA ONLY');
  expect(formatted).toInclude('dist/bundle.js');
 });
 tap.test('DiffProcessor should prioritize source files over build artifacts', async () => {
  const processor = new DiffProcessor();
  const diffs = [
    createSmallDiff('dist/bundle.js'),
    createSmallDiff('src/important.ts'),
    createSmallDiff('build/output.js'),
    createSmallDiff('src/core.ts'),
  ];
  const result = processor.processDiffs(diffs);
  expect(result.totalFiles).toEqual(4);
  // Source files should be included fully first
  const formatted = processor.formatForContext(result);
  const srcImportantIndex = formatted.indexOf('src/important.ts');
  const srcCoreIndex = formatted.indexOf('src/core.ts');
  const distBundleIndex = formatted.indexOf('dist/bundle.js');
  const buildOutputIndex = formatted.indexOf('build/output.js');
  // Source files should appear before build artifacts
  expect(srcImportantIndex).toBeLessThan(distBundleIndex);
  expect(srcCoreIndex).toBeLessThan(buildOutputIndex);
 });
 tap.test('DiffProcessor should respect token budget', async () => {
  const processor = new DiffProcessor({
    maxDiffTokens: 500, // Very small budget to force metadata-only
  });
  // Create multiple large diffs that will exceed budget
  const diffs = [
    createLargeDiff('src/file1.ts'),
    createLargeDiff('src/file2.ts'),
    createLargeDiff('src/file3.ts'),
    createLargeDiff('src/file4.ts'),
  ];
  const result = processor.processDiffs(diffs);
  expect(result.totalTokens).toBeLessThanOrEqual(500);
  // With such a small budget and large files, most should be metadata only
  expect(result.metadataOnly.length).toBeGreaterThanOrEqual(2);
 });
 tap.test('DiffProcessor should handle deleted files', async () => {
  const processor = new DiffProcessor();
  const deletedDiff = createDeletedFileDiff('src/old-file.ts');
  const result = processor.processDiffs([deletedDiff]);
  expect(result.totalFiles).toEqual(1);
  // Small deleted file should be included fully
  expect(result.fullDiffs.length).toEqual(1);
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('src/old-file.ts');
  // Verify the file appears in the output
  expect(formatted).toInclude('FULL DIFFS');
 });
 tap.test('DiffProcessor should handle added files', async () => {
  const processor = new DiffProcessor();
  const addedDiff = createAddedFileDiff('src/new-file.ts');
  const result = processor.processDiffs([addedDiff]);
  expect(result.totalFiles).toEqual(1);
  // Small added file should be included fully
  expect(result.fullDiffs.length).toEqual(1);
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('src/new-file.ts');
  // Verify the file appears in the output
  expect(formatted).toInclude('FULL DIFFS');
 });
 tap.test('DiffProcessor should handle mixed file sizes', async () => {
  const processor = new DiffProcessor();
  const diffs = [
    createSmallDiff('src/small.ts'),
    createMediumDiff('src/medium.ts'),
    createLargeDiff('dist/large.js'),
  ];
  const result = processor.processDiffs(diffs);
  expect(result.totalFiles).toEqual(3);
  expect(result.fullDiffs.length).toEqual(1); // small file
  expect(result.summarizedDiffs.length).toEqual(1); // medium file
  expect(result.metadataOnly.length).toEqual(1); // large file
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('FULL DIFFS (1 files)');
  expect(formatted).toInclude('SUMMARIZED DIFFS (1 files)');
  expect(formatted).toInclude('METADATA ONLY (1 files)');
 });
 tap.test('DiffProcessor should handle empty diff array', async () => {
  const processor = new DiffProcessor();
  const result = processor.processDiffs([]);
  expect(result.totalFiles).toEqual(0);
  expect(result.fullDiffs.length).toEqual(0);
  expect(result.summarizedDiffs.length).toEqual(0);
  expect(result.metadataOnly.length).toEqual(0);
  expect(result.totalTokens).toEqual(0);
 });
 tap.test('DiffProcessor should generate comprehensive summary', async () => {
  const processor = new DiffProcessor();
  const diffs = [
    createSmallDiff('src/file1.ts'),
    createSmallDiff('src/file2.ts'),
    createMediumDiff('src/file3.ts'),
    createLargeDiff('dist/bundle.js'),
  ];
  const result = processor.processDiffs(diffs);
  const formatted = processor.formatForContext(result);
  expect(formatted).toInclude('GIT DIFF SUMMARY');
  expect(formatted).toInclude('Files changed: 4 total');
  expect(formatted).toInclude('included in full');
  expect(formatted).toInclude('summarized');
  expect(formatted).toInclude('metadata only');
  expect(formatted).toInclude('Estimated tokens:');
  expect(formatted).toInclude('END OF GIT DIFF');
 });
 tap.test('DiffProcessor should handle custom options', async () => {
  const processor = new DiffProcessor({
    maxDiffTokens: 50000,
    smallFileLines: 30,
    mediumFileLines: 150,
    sampleHeadLines: 10,
    sampleTailLines: 10,
  });
  const mediumDiff = createMediumDiff('src/file.ts'); // 150 lines
  const result = processor.processDiffs([mediumDiff]);
  // With custom settings, this should be summarized (exactly at the mediumFileLines threshold)
  expect(result.summarizedDiffs.length).toEqual(1);
 });
 tap.test('DiffProcessor should prioritize test files appropriately', async () => {
  const processor = new DiffProcessor();
  const diffs = [
    createSmallDiff('src/core.ts'),
    createSmallDiff('test/core.test.ts'),
    createSmallDiff('config.json'),
  ];
  const result = processor.processDiffs(diffs);
  const formatted = processor.formatForContext(result);
  // Source files should come before test files
  const srcIndex = formatted.indexOf('src/core.ts');
  const testIndex = formatted.indexOf('test/core.test.ts');
  expect(srcIndex).toBeLessThan(testIndex);
 });
 tap.test('DiffProcessor should handle files with no changes gracefully', async () => {
  const processor = new DiffProcessor();
  const emptyDiff = `--- a/src/file.ts
 +++ b/src/file.ts
@@ -1,1 +1,1 @@`;
  const result = processor.processDiffs([emptyDiff]);
  expect(result.totalFiles).toEqual(1);
  expect(result.fullDiffs.length).toEqual(1); // Still included as a small file
 });
 export default tap.start();
--- a/test/test.ts
+++ b/test/test.ts
@@ -1,8 +0,0 @@
 import { expect, tap } from '@push.rocks/tapbundle';
 import * as tsdoc from '../ts/index.js';
 tap.test('first test', async () => {
  console.log('test');
 });
 tap.start();
--- a/ts/00_commitinfo_data.ts
+++ b/ts/00_commitinfo_data.ts
@@ -3,6 +3,6 @@
 */
 export const commitinfo = {
  name: '@git.zone/tsdoc',
-  version: '1.3.12',
+  version: '2.0.0',
-  description: 'An advanced TypeScript documentation tool using AI to generate and enhance documentation for TypeScript projects.'
+  description: 'A comprehensive TypeScript documentation tool that leverages AI to generate and enhance project documentation, including dynamic README creation, API docs via TypeDoc, and smart commit message generation.'
 }
--- a/ts/aidocs_classes/commit.ts
+++ b/ts/aidocs_classes/commit.ts
@@ -1,6 +1,27 @@
 import * as plugins from '../plugins.js';
 import { AiDoc } from '../classes.aidoc.js';
 import { ProjectContext } from './projectcontext.js';
 import { DiffProcessor } from '../classes.diffprocessor.js';
 import { logger } from '../logging.js';
 // Token budget configuration for OpenAI API limits
 const TOKEN_BUDGET = {
  OPENAI_CONTEXT_LIMIT: 272000,   // OpenAI's configured limit
  SAFETY_MARGIN: 10000,            // Buffer to avoid hitting exact limit
  SMARTAGENT_OVERHEAD: 180000,     // System msgs, tools, history, formatting
  TASK_PROMPT_OVERHEAD: 2000,      // Task prompt template size
 } as const;
 /**
 * Calculate max tokens available for diff content based on total budget
 */
 function calculateMaxDiffTokens(): number {
  const available = TOKEN_BUDGET.OPENAI_CONTEXT_LIMIT
    - TOKEN_BUDGET.SAFETY_MARGIN
    - TOKEN_BUDGET.SMARTAGENT_OVERHEAD
    - TOKEN_BUDGET.TASK_PROMPT_OVERHEAD;
  return Math.max(available, 30000);
 }
 export interface INextCommitObject {
  recommendedNextVersionLevel: 'fix' | 'feat' | 'BREAKING CHANGE'; // the recommended next version level of the project
@@ -27,93 +48,237 @@ export class Commit {
      smartgitInstance,
      this.projectDir
    );
-    const diffStringArray = await gitRepo.getUncommittedDiff([
+
    // Define comprehensive exclusion patterns
    // smartgit@3.3.0+ supports glob patterns natively
    const excludePatterns = [
      // Lock files
      'pnpm-lock.yaml',
      'package-lock.json',
-    ]);
+      'npm-shrinkwrap.json',
-    const projectContext = new ProjectContext(this.projectDir);
+      'yarn.lock',
-    let contextString = await projectContext.update();
+      'deno.lock',
-    contextString = `
+      'bun.lockb',
 ${contextString}
-Below is the diff of the uncommitted changes. If nothing is changed, there are no changes:
+      // Build artifacts (main culprit for large diffs!)
      'dist/**',
      'dist_*/**',           // dist_ts, dist_web, etc.
      'build/**',
      '.next/**',
      'out/**',
      'public/dist/**',
-${diffStringArray[0] ? diffStringArray.join('\n\n') : 'No changes.'}
+      // Compiled/bundled files
-    `;
+      '**/*.js.map',
      '**/*.d.ts.map',
      '**/*.min.js',
      '**/*.bundle.js',
      '**/*.chunk.js',
-    let result = await this.aiDocsRef.openaiInstance.chat({
+      // IDE/Editor directories
-      systemMessage: `
+      '.claude/**',
-You create a commit message for a git commit.
+      '.cursor/**',
-The commit message should be based on the files in the project.
+      '.vscode/**',
-You should not include any licensing information.
+      '.idea/**',
-You should not include any personal information.
+      '**/*.swp',
      '**/*.swo',
-Important: Answer only in valid JSON.
+      // Logs and caches
      '.nogit/**',
      '**/*.log',
      '.cache/**',
      '.rpt2_cache/**',
      'coverage/**',
      '.nyc_output/**',
    ];
-Your answer should be parseable with JSON.parse() without modifying anything.
+    // Pass glob patterns directly to smartgit - it handles matching internally
    const diffStringArray = await gitRepo.getUncommittedDiff(excludePatterns);
-Here is the structure of the JSON you should return:
+    // Process diffs intelligently using DiffProcessor
    let processedDiffString: string;
-interface {
+    if (diffStringArray.length > 0) {
-  recommendedNextVersionLevel: 'fix' | 'feat' | 'BREAKING CHANGE'; // the recommended next version level of the project
+      // Diagnostic logging for raw diff statistics
-  recommendedNextVersionScope: string; // the recommended scope name of the next version, like "core" or "cli", or specific class names.
+      const totalChars = diffStringArray.join('\n\n').length;
-  recommendedNextVersionMessage: string; // the commit message. Don't put fix() feat() or BREAKING CHANGE in the message. Please just the message itself.
+      const estimatedTokens = Math.ceil(totalChars / 4);
-  recommendedNextVersionDetails: string[]; // detailed bullet points for the changelog
+
-  recommendedNextVersion: string; // the recommended next version of the project, x.x.x
+      console.log(`Raw git diff statistics:`);
      console.log(`   Files changed: ${diffStringArray.length}`);
      console.log(`   Total characters: ${totalChars.toLocaleString()}`);
      console.log(`   Estimated tokens: ${estimatedTokens.toLocaleString()}`);
      console.log(`   Exclusion patterns: ${excludePatterns.length}`);
      // Calculate available tokens for diff based on total budget
      const maxDiffTokens = calculateMaxDiffTokens();
      console.log(`Token budget: ${maxDiffTokens.toLocaleString()} tokens for diff (limit: ${TOKEN_BUDGET.OPENAI_CONTEXT_LIMIT.toLocaleString()}, overhead: ${(TOKEN_BUDGET.SMARTAGENT_OVERHEAD + TOKEN_BUDGET.TASK_PROMPT_OVERHEAD).toLocaleString()})`);
      // Use DiffProcessor to intelligently handle large diffs
      const diffProcessor = new DiffProcessor({
        maxDiffTokens,               // Dynamic based on total budget
        smallFileLines: 300,         // Most source files are under 300 lines
        mediumFileLines: 800,        // Only very large files get head/tail treatment
        sampleHeadLines: 75,         // When sampling, show more context
        sampleTailLines: 75,         // When sampling, show more context
      });
      const processedDiff = diffProcessor.processDiffs(diffStringArray);
      processedDiffString = diffProcessor.formatForContext(processedDiff);
      console.log(`Processed diff statistics:`);
      console.log(`   Full diffs: ${processedDiff.fullDiffs.length} files`);
      console.log(`   Summarized: ${processedDiff.summarizedDiffs.length} files`);
      console.log(`   Metadata only: ${processedDiff.metadataOnly.length} files`);
      console.log(`   Final tokens: ${processedDiff.totalTokens.toLocaleString()}`);
      if (estimatedTokens > 50000) {
        console.log(`DiffProcessor reduced token usage: ${estimatedTokens.toLocaleString()} -> ${processedDiff.totalTokens.toLocaleString()}`);
      }
      // Validate total tokens won't exceed limit
      const totalEstimatedTokens = processedDiff.totalTokens
        + TOKEN_BUDGET.SMARTAGENT_OVERHEAD
        + TOKEN_BUDGET.TASK_PROMPT_OVERHEAD;
      if (totalEstimatedTokens > TOKEN_BUDGET.OPENAI_CONTEXT_LIMIT - TOKEN_BUDGET.SAFETY_MARGIN) {
        console.log(`Warning: Estimated tokens (${totalEstimatedTokens.toLocaleString()}) approaching limit`);
        console.log(`   Consider splitting into smaller commits`);
      }
    } else {
      processedDiffString = 'No changes.';
    }
    // Use runAgent for commit message generation with filesystem tool
    const fsTools = plugins.smartagentTools.filesystemTool({ rootDir: this.projectDir });
    const commitSystemPrompt = `
 You create commit messages for git commits following semantic versioning conventions.
 You have access to filesystem tools to explore the project if needed.
 IMPORTANT RULES:
 - Only READ files (package.json, source files) for context
 - Do NOT write, delete, or modify any files
 - Version level (fix/feat/BREAKING CHANGE) must match the scope of changes
 - Commit message must be clear, professional, and follow conventional commit conventions
 - Do NOT include personal information, licensing details, or AI mentions (Claude/Codex)
 - JSON structure must be valid with all required fields
 - Scope must accurately reflect the changed modules/files
 `;
    const commitTaskPrompt = `
 Project directory: ${this.projectDir}
 You have access to filesystem tools to explore the project if needed:
 - Use list_directory to see project structure
 - Use read_file to read package.json or source files for context
 Analyze the git diff below to understand what changed and generate a commit message.
 You should not include any licensing information or personal information.
 Never mention CLAUDE code, or codex.
 Your final response must be ONLY valid JSON - the raw JSON object, nothing else.
 No explanations, no summaries, no markdown - just the JSON object that can be parsed with JSON.parse().
 Here is the structure of the JSON you must return:
 {
  "recommendedNextVersionLevel": "fix" | "feat" | "BREAKING CHANGE",
  "recommendedNextVersionScope": "string",
  "recommendedNextVersionMessage": "string (ONLY the description body WITHOUT the type(scope): prefix - e.g. 'bump dependency to ^1.2.6' NOT 'fix(deps): bump dependency to ^1.2.6')",
  "recommendedNextVersionDetails": ["string"],
  "recommendedNextVersion": "x.x.x"
 }
-For the recommendedNextVersionDetails, please only add a detail entries to the array if it has an obvious value to the reader.
+For recommendedNextVersionDetails, only add entries that have obvious value to the reader.
-You are being given the files of the project. You should use them to create the commit message.
+Here is the git diff showing what changed:
 Also you are given a diff
-`,
+${processedDiffString}
-      messageHistory: [],
+
-      userMessage: contextString,
+Analyze these changes and output the JSON commit message object.
 `;
    logger.log('info', 'Starting commit message generation with agent...');
    const commitResult = await plugins.smartagent.runAgent({
      model: this.aiDocsRef.model,
      prompt: commitTaskPrompt,
      system: commitSystemPrompt,
      tools: fsTools,
      maxSteps: 10,
      onToolCall: (toolName) => logger.log('info', `[Commit] Tool call: ${toolName}`),
    });
-    // console.log(result.message);
+    // Extract JSON from result - handle cases where AI adds text around it
-    const resultObject: INextCommitObject = JSON.parse(
+    let jsonString = commitResult.text
-      result.message.replace('```json', '').replace('```', '')
+      .replace(/```json\n?/gi, '')
-    );
+      .replace(/```\n?/gi, '');
    // Try to find JSON object in the result
    const jsonMatch = jsonString.match(/\{[\s\S]*\}/);
    if (!jsonMatch) {
      throw new Error(`Could not find JSON object in result: ${jsonString.substring(0, 100)}...`);
    }
    jsonString = jsonMatch[0];
    const resultObject: INextCommitObject = JSON.parse(jsonString);
    const previousChangelogPath = plugins.path.join(this.projectDir, 'changelog.md');
    let previousChangelog: plugins.smartfile.SmartFile;
-    if (await plugins.smartfile.fs.fileExists(previousChangelogPath)) {
+    if (await plugins.fsInstance.file(previousChangelogPath).exists()) {
-      previousChangelog = await plugins.smartfile.SmartFile.fromFilePath(previousChangelogPath);
+      previousChangelog = await plugins.smartfileFactory.fromFilePath(previousChangelogPath);
    }
    if (!previousChangelog) {
      // lets build the changelog based on that
      const commitMessages = await gitRepo.getAllCommitMessages();
      console.log(JSON.stringify(commitMessages, null, 2));
-      let result2 = await this.aiDocsRef.openaiInstance.chat({
+
-        messageHistory: [],
+      const changelogSystemPrompt = `
-        systemMessage: `
+You generate changelog.md files for software projects.
 RULES:
 - Changelog must follow proper markdown format with ## headers for each version
 - Entries must be chronologically ordered (newest first)
 - Version ranges for trivial commits should be properly summarized
 - No duplicate or empty entries
 - Format: ## yyyy-mm-dd - x.x.x - scope
 `;
      const changelogTaskPrompt = `
 You are building a changelog.md file for the project.
 Omit commits and versions that lack relevant changes, but make sure to mention them as a range with a summarizing message instead.
 A changelog entry should look like this:
    ## yyyy-mm-dd - x.x.x - scope here
-    main descriptiom here
+    main description here
    - detailed bullet points follow
 You are given:
 * the commit messages of the project
-Only return the changelog file, so it can be written directly to changelog.md`,
+Only return the changelog file content, so it can be written directly to changelog.md.
-        userMessage: `
+
 Here are the commit messages:
 ${JSON.stringify(commitMessages, null, 2)}
-  `,
+`;
      const changelogResult = await plugins.smartagent.runAgent({
        model: this.aiDocsRef.model,
        prompt: changelogTaskPrompt,
        system: changelogSystemPrompt,
        maxSteps: 1,
        onToolCall: (toolName) => logger.log('info', `[Changelog] Tool call: ${toolName}`),
      });
-      previousChangelog = await plugins.smartfile.SmartFile.fromString(
+      previousChangelog = plugins.smartfileFactory.fromString(
        previousChangelogPath,
-        result2.message.replaceAll('```markdown', '').replaceAll('```', ''),
+        changelogResult.text.replaceAll('```markdown', '').replaceAll('```', ''),
        'utf8'
      );
    }
--- a/ts/aidocs_classes/description.ts
+++ b/ts/aidocs_classes/description.ts
@@ -1,6 +1,7 @@
 import type { AiDoc } from '../classes.aidoc.js';
 import * as plugins from '../plugins.js';
 import { ProjectContext } from './projectcontext.js';
 import { logger } from '../logging.js';
 interface IDescriptionInterface {
  description: string;
@@ -18,47 +19,78 @@ export class Description {
  }
  public async build() {
-    // we can now assemble the directory structure.
+    // Use runAgent with filesystem tool for agent-driven exploration
-    const projectContext = new ProjectContext(this.projectDir);
+    const fsTools = plugins.smartagentTools.filesystemTool({ rootDir: this.projectDir });
    const contextString = await projectContext.update();
-    let result = await this.aiDocsRef.openaiInstance.chat({
+    const descriptionSystemPrompt = `
-      systemMessage: `
+You create project descriptions and keywords for npm packages.
-You create a json adhering the following interface:
+
 You have access to filesystem tools to explore the project.
 IMPORTANT RULES:
 - Only READ files (package.json, npmextra.json, source files in ts/)
 - Do NOT write, delete, or modify any files
 - Your final response must be valid JSON only
 - Description must be a clear, concise one-sentence summary
 - Keywords must be relevant to the project's use cases
 - Both description and keywords fields must be present
 - Do NOT wrap JSON in markdown code blocks
 `;
    const descriptionTaskPrompt = `
 PROJECT DIRECTORY: ${this.projectDir}
 Use the filesystem tools to explore the project and understand what it does:
 1. First, use list_directory to see the project structure
 2. Read package.json to understand the package name and current description
 3. Read npmextra.json if it exists for additional metadata
 4. Read key source files in ts/ directory to understand the implementation
 Then generate a description and keywords based on your exploration.
 Your FINAL response must be valid JSON adhering to this interface:
 {
  description: string; // a sensible short, one sentence description of the project
-  keywords: string[]; // an array of tags that describe the project
+  keywords: string[]; // an array of tags that describe the project based on use cases
 }
 The description should be based on what you understand from the project's files.
 The keywords should be based on use cases you see from the files.
 Don't be cheap about the way you think.
 Important: Answer only in valid JSON.
-You answer should be parseable with JSON.parse() without modifying anything.
+Your answer should be parseable with JSON.parse() without modifying anything.
 Don't wrap the JSON in \`\`\`json\`\`\` - just return the raw JSON object.
 `;
-Don't wrap the JSON in three ticks json!!!
+    logger.log('info', 'Starting description generation with agent...');
-    `,
+
-      messageHistory: [],
+    const descriptionResult = await plugins.smartagent.runAgent({
-      userMessage: contextString,
+      model: this.aiDocsRef.model,
      prompt: descriptionTaskPrompt,
      system: descriptionSystemPrompt,
      tools: fsTools,
      maxSteps: 15,
      onToolCall: (toolName) => logger.log('info', `[Description] Tool call: ${toolName}`),
    });
-    console.log(result.message);
+    console.log(descriptionResult.text);
    const resultObject: IDescriptionInterface = JSON.parse(
-      result.message.replace('```json', '').replace('```', ''),
+      descriptionResult.text.replace('```json', '').replace('```', ''),
    );
-    const npmextraJson = (await projectContext.gatherFiles()).smartfilesNpmextraJSON;
+    // Use ProjectContext to get file handles for writing
    const projectContext = new ProjectContext(this.projectDir);
    const files = await projectContext.gatherFiles();
    // Update npmextra.json
    const npmextraJson = files.smartfilesNpmextraJSON;
    const npmextraJsonContent = JSON.parse(npmextraJson.contents.toString());
-    npmextraJsonContent.gitzone.module.description = resultObject.description;
+    npmextraJsonContent['@git.zone/cli'].module.description = resultObject.description;
-    npmextraJsonContent.gitzone.module.keywords = resultObject.keywords;
+    npmextraJsonContent['@git.zone/cli'].module.keywords = resultObject.keywords;
    npmextraJson.contents = Buffer.from(JSON.stringify(npmextraJsonContent, null, 2));
    await npmextraJson.write();
-    // do the same with packageJson
+    // Update package.json
-    const packageJson = (await projectContext.gatherFiles()).smartfilePackageJSON;
+    const packageJson = files.smartfilePackageJSON;
    const packageJsonContent = JSON.parse(packageJson.contents.toString());
    packageJsonContent.description = resultObject.description;
    packageJsonContent.keywords = resultObject.keywords;
@@ -68,6 +100,6 @@ Don't wrap the JSON in three ticks json!!!
    console.log(`\n======================\n`);
    console.log(JSON.stringify(resultObject, null, 2));
    console.log(`\n======================\n`);
-    return result.message;
+    return descriptionResult.text;
  }
 }
--- a/ts/aidocs_classes/projectcontext.ts
+++ b/ts/aidocs_classes/projectcontext.ts
@@ -5,37 +5,37 @@ export class ProjectContext {
  // INSTANCE
  public projectDir: string;
  private tokenCount: number = 0;
  private contextString: string = '';
  constructor(projectDirArg: string) {
    this.projectDir = projectDirArg;
  }
  public async gatherFiles() {
-    const smartfilePackageJSON = await plugins.smartfile.SmartFile.fromFilePath(
+    const smartfilePackageJSON = await plugins.smartfileFactory.fromFilePath(
      plugins.path.join(this.projectDir, 'package.json'),
      this.projectDir,
    );
-    const smartfilesReadme = await plugins.smartfile.SmartFile.fromFilePath(
+    const smartfilesReadme = await plugins.smartfileFactory.fromFilePath(
      plugins.path.join(this.projectDir, 'readme.md'),
      this.projectDir,
    );
-    const smartfilesReadmeHints = await plugins.smartfile.SmartFile.fromFilePath(
+    const smartfilesReadmeHints = await plugins.smartfileFactory.fromFilePath(
      plugins.path.join(this.projectDir, 'readme.hints.md'),
      this.projectDir,
    );
-    const smartfilesNpmextraJSON = await plugins.smartfile.SmartFile.fromFilePath(
+    const smartfilesNpmextraJSON = await plugins.smartfileFactory.fromFilePath(
      plugins.path.join(this.projectDir, 'npmextra.json'),
      this.projectDir,
    );
-    const smartfilesMod = await plugins.smartfile.fs.fileTreeToObject(
+    const smartfilesMod = await plugins.smartfileFactory.virtualDirectoryFromPath(
      this.projectDir,
-      'ts*/**/*.ts',
+    ).then(vd => vd.filter(f => f.relative.startsWith('ts') && f.relative.endsWith('.ts')).listFiles());
-    );
+    const smartfilesTest = await plugins.smartfileFactory.virtualDirectoryFromPath(
    const smartfilesTest = await plugins.smartfile.fs.fileTreeToObject(
      this.projectDir,
-      'test/**/*.ts',
+    ).then(vd => vd.filter(f => f.relative.startsWith('test/') && f.relative.endsWith('.ts')).listFiles());
    );
    return {
      smartfilePackageJSON,
      smartfilesReadme,
@@ -63,6 +63,17 @@ ${smartfile.contents.toString()}
      .join('\n');
  }
  /**
   * Estimate token count for a string
   * Uses a rough estimate of 4 characters per token
   * @param text The text to estimate tokens for
   * @returns Estimated number of tokens
   */
  public countTokens(text: string): number {
    // Rough estimate: ~4 characters per token for English text
    return Math.ceil(text.length / 4);
  }
  private async buildContext(dirArg: string) {
    const files = await this.gatherFiles();
    let context = await this.convertFilesToContext([
@@ -73,10 +84,33 @@ ${smartfile.contents.toString()}
      ...files.smartfilesMod,
      ...files.smartfilesTest,
    ]);
    // Count tokens in the context
    this.contextString = context;
    this.tokenCount = this.countTokens(context);
    // console.log(context);
    return context;
  }
  /**
   * Get the token count for the current context
   * @returns The number of tokens in the context
   */
  public getTokenCount(): number {
    return this.tokenCount;
  }
  /**
   * Get both the context string and its token count
   * @returns An object containing the context string and token count
   */
  public getContextWithTokenCount(): { context: string; tokenCount: number } {
    return {
      context: this.contextString,
      tokenCount: this.tokenCount
    };
  }
  public async update() {
    const result = await this.buildContext(this.projectDir);
    return result;
--- a/ts/aidocs_classes/readme.ts
+++ b/ts/aidocs_classes/readme.ts
@@ -1,6 +1,8 @@
 import type { AiDoc } from '../classes.aidoc.js';
 import * as plugins from '../plugins.js';
 import * as paths from '../paths.js';
 import { ProjectContext } from './projectcontext.js';
 import { logger } from '../logging.js';
 export class Readme {
  // INSTANCE
@@ -15,70 +17,175 @@ export class Readme {
  public async build() {
    let finalReadmeString = ``;
-    // we can now assemble the directory structure.
+    // First check legal info before introducing any cost
    const projectContext = new ProjectContext(this.projectDir);
    const contextString = await projectContext.update();
    // lets first check legal before introducung any cost
    const npmExtraJson = JSON.parse(
-      (await projectContext.gatherFiles()).smartfilesNpmextraJSON.contents.toString(),
+      (await projectContext.gatherFiles()).smartfilesNpmextraJSON.contents.toString()
    );
-    const legalInfo = npmExtraJson?.tsdoc?.legal;
+    const legalInfo = npmExtraJson?.['@git.zone/tsdoc']?.legal;
    if (!legalInfo) {
      const error = new Error(`No legal information found in npmextra.json`);
      console.log(error);
    }
-    let result = await this.aiDocsRef.openaiInstance.chat({
+    // Use runAgent with filesystem tool for agent-driven exploration
-      systemMessage: `
+    const fsTools = plugins.smartagentTools.filesystemTool({ rootDir: this.projectDir });
 You create markdown readmes for npm projects. You only output the markdown readme.
-The Readme should follow the following template:
+    const readmeSystemPrompt = `
 You create markdown READMEs for npm projects. You only output the markdown readme.
 You have access to filesystem tools to explore the project. Use them to understand the codebase.
 IMPORTANT RULES:
 - Only READ files within the project directory
 - Do NOT write, delete, or modify any files
 - README must follow proper markdown format
 - Must contain Install and Usage sections
 - Code examples must use correct TypeScript/ESM syntax
 - Documentation must be comprehensive and helpful
 - Do NOT include licensing information (added separately)
 - Do NOT use CommonJS syntax - only ESM
 - Do NOT include "in conclusion" or similar filler
 `;
    const readmeTaskPrompt = `
 PROJECT DIRECTORY: ${this.projectDir}
 Use the filesystem tools to explore the project and understand what it does:
 1. First, use list_directory to see the project structure
 2. Read package.json to understand the package name, description, and dependencies
 3. Read the existing readme.md if it exists (use it as a base, improve and expand)
 4. Read readme.hints.md if it exists (contains hints for documentation)
 5. Read key source files in ts/ directory to understand the API and implementation
 6. Focus on exported classes, interfaces, and functions
 Then generate a comprehensive README following this template:
 # Project Name
-[
+[The name from package.json and description]
  The name is the module name of package.json
  The description is in the description field of package.json
 ]
 ## Install
-[
+[Short text on how to install the project]
  Write a short text on how to install the project
 ]
 ## Usage
-[ 
+[
  Give code examples here.
  Construct sensible scenarios for the user.
  Make sure to show a complete set of features of the module.
  Don't omit use cases.
  It does not matter how much time you need.
  ALWAYS USE ESM SYNTAX AND TYPESCRIPT.
-  DON'T CHICKEN OUT. Write at least 4000 words. More if necessary.
+  Write at least 4000 words. More if necessary.
-  If there is already a readme, take the Usage section as base. Remove outdated content, and expand and improve upon the valid parts.
+  If there is already a readme, take the Usage section as base. Remove outdated content, expand and improve.
-  Super important: Check for completenes.
+  Check for completeness.
-  Don't include any licensing information. This will be added in a later step.
+  Don't include any licensing information. This will be added later.
-  Avoid "in conclusions".
+  Avoid "in conclusion" statements.
  Good to know:
  * npmextra.json contains overall module information.
  * readme.hints.md provides valuable hints about module ideas.
 ]
-            `,
+`;
-      messageHistory: [],
+
-      userMessage: contextString,
+    logger.log('info', 'Starting README generation with agent...');
    const readmeResult = await plugins.smartagent.runAgent({
      model: this.aiDocsRef.model,
      prompt: readmeTaskPrompt,
      system: readmeSystemPrompt,
      tools: fsTools,
      maxSteps: 25,
      onToolCall: (toolName) => logger.log('info', `[README] Tool call: ${toolName}`),
    });
-    finalReadmeString += result.message + '\n' + legalInfo;
+    // Clean up markdown formatting if wrapped in code blocks
    let resultMessage = readmeResult.text
      .replace(/^```markdown\n?/i, '')
      .replace(/\n?```$/i, '');
    finalReadmeString += resultMessage + '\n' + legalInfo;
    console.log(`\n======================\n`);
-    console.log(result.message);
+    console.log(resultMessage);
    console.log(`\n======================\n`);
    const readme = (await projectContext.gatherFiles()).smartfilesReadme;
    readme.contents = Buffer.from(finalReadmeString);
    await readme.write();
-    return result.message;
+    // lets care about monorepo aspects
    const tsPublishInstance = new plugins.tspublish.TsPublish();
    const subModules = await tsPublishInstance.getModuleSubDirs(paths.cwd);
    logger.log('info', `Found ${Object.keys(subModules).length} sub modules`);
    for (const subModule of Object.keys(subModules)) {
      logger.log('info', `Building readme for ${subModule}`);
      const subModulePath = plugins.path.join(paths.cwd, subModule);
      const tspublishData = await plugins.fsInstance
        .file(plugins.path.join(subModulePath, 'tspublish.json'))
        .encoding('utf8')
        .read();
      const subModuleFsTools = plugins.smartagentTools.filesystemTool({ rootDir: subModulePath });
      const subModuleSystemPrompt = `
 You create markdown READMEs for npm projects. You only output the markdown readme.
 IMPORTANT RULES:
 - Only READ files within the submodule directory
 - Do NOT write, delete, or modify any files
 - README must be comprehensive, well-formatted markdown with ESM TypeScript examples
 - Do NOT include licensing information (added separately)
 `;
      const subModulePrompt = `
 SUB MODULE: ${subModule}
 SUB MODULE DIRECTORY: ${subModulePath}
 IMPORTANT: YOU ARE CREATING THE README FOR THIS SUB MODULE: ${subModule}
 The Sub Module will be published with:
 ${JSON.stringify(tspublishData, null, 2)}
 Use the filesystem tools to explore the submodule:
 1. Use list_directory to see the submodule structure
 2. Read package.json to understand the submodule
 3. Read source files in ts/ directory to understand the implementation
 Generate a README following the template:
 # Project Name
 [name and description from package.json]
 ## Install
 [installation instructions]
 ## Usage
 [
  Code examples with complete features.
  ESM TypeScript syntax only.
  Write at least 4000 words.
  No licensing information.
  No "in conclusion".
 ]
 Don't use \`\`\` at the beginning or end. Only for code blocks.
 `;
      const subModuleResult = await plugins.smartagent.runAgent({
        model: this.aiDocsRef.model,
        prompt: subModulePrompt,
        system: subModuleSystemPrompt,
        tools: subModuleFsTools,
        maxSteps: 20,
        onToolCall: (toolName) => logger.log('info', `[README:${subModule}] Tool call: ${toolName}`),
      });
      const subModuleReadmeString = subModuleResult.text
        .replace(/^```markdown\n?/i, '')
        .replace(/\n?```$/i, '') + '\n' + legalInfo;
      await plugins.fsInstance
        .file(plugins.path.join(subModulePath, 'readme.md'))
        .encoding('utf8')
        .write(subModuleReadmeString);
      logger.log('success', `Built readme for ${subModule}`);
    }
    return resultMessage;
  }
 }
--- a/ts/classes.aidoc.ts
+++ b/ts/classes.aidoc.ts
@@ -8,7 +8,7 @@ export class AiDoc {
  public npmextraKV: plugins.npmextra.KeyValueStore;
  public qenvInstance: plugins.qenv.Qenv;
  public aidocInteract: plugins.smartinteract.SmartInteract;
-  public openaiInstance: plugins.smartai.OpenAiProvider;
+  public model: plugins.smartai.LanguageModelV3;
  argvArg: any;
@@ -36,9 +36,25 @@ export class AiDoc {
    this.aidocInteract = new plugins.smartinteract.SmartInteract();
    this.qenvInstance = new plugins.qenv.Qenv();
    if (!(await this.qenvInstance.getEnvVarOnDemand('OPENAI_TOKEN'))) {
      // Migrate old KV store path to new path if needed
      const homeDir = plugins.smartpath.get.home();
      const oldKvPath = plugins.path.join(homeDir, '.npmextra/kv/tsdoc.json');
      const newKvDir = plugins.path.join(homeDir, '.npmextra/kv/@git.zone');
      const newKvPath = plugins.path.join(newKvDir, 'tsdoc.json');
      if (
        await plugins.fsInstance.file(oldKvPath).exists() &&
        !(await plugins.fsInstance.file(newKvPath).exists())
      ) {
        console.log('Migrating tsdoc KeyValueStore to @git.zone/tsdoc...');
        await plugins.fsInstance.directory(newKvDir).recursive().create();
        await plugins.fsInstance.file(oldKvPath).copy(newKvPath);
        await plugins.fsInstance.file(oldKvPath).delete();
        console.log('Migration complete: tsdoc.json -> @git.zone/tsdoc.json');
      }
      this.npmextraKV = new plugins.npmextra.KeyValueStore({
        typeArg: 'userHomeDir',
-        identityArg: 'tsdoc',
+        identityArg: '@git.zone/tsdoc',
        mandatoryKeys: ['OPENAI_TOKEN'],
      });
@@ -64,15 +80,24 @@ export class AiDoc {
        await this.npmextraKV.writeKey('OPENAI_TOKEN', this.openaiToken);
      }
    }
-    if (!this.openaiToken) {
+    if (!this.openaiToken && this.npmextraKV) {
      this.openaiToken = await this.npmextraKV.readKey('OPENAI_TOKEN');
    }
-    // lets assume we have an OPENAI_Token now
+    // Create model using getModel()
-    this.openaiInstance = new plugins.smartai.OpenAiProvider({
+    this.model = plugins.smartai.getModel({
-      openaiToken: this.openaiToken,
+      provider: 'openai',
      model: 'gpt-5.4',
      apiKey: this.openaiToken,
    });
-    await this.openaiInstance.start();
+  }
  public async stop() {
    // No lifecycle management needed with getModel() API
  }
  public getOpenaiToken(): string {
    return this.openaiToken;
  }
  public async buildReadme(projectDirArg: string) {
@@ -94,4 +119,36 @@ export class AiDoc {
    const projectContextInstance = new aiDocsClasses.ProjectContext(projectDirArg);
    return await projectContextInstance.gatherFiles();
  }
  /**
   * Get the context with token count information
   * @param projectDirArg The path to the project directory
   * @returns An object containing the context string and its token count
   */
  public async getProjectContextWithTokenCount(projectDirArg: string) {
    const projectContextInstance = new aiDocsClasses.ProjectContext(projectDirArg);
    await projectContextInstance.update();
    return projectContextInstance.getContextWithTokenCount();
  }
  /**
   * Get just the token count for a project's context
   * @param projectDirArg The path to the project directory
   * @returns The number of tokens in the project context
   */
  public async getProjectContextTokenCount(projectDirArg: string) {
    const projectContextInstance = new aiDocsClasses.ProjectContext(projectDirArg);
    await projectContextInstance.update();
    return projectContextInstance.getTokenCount();
  }
  /**
   * Estimate token count in a text string
   * @param text The text to estimate tokens for
   * @returns Estimated number of tokens
   */
  public countTokens(text: string): number {
    const projectContextInstance = new aiDocsClasses.ProjectContext('');
    return projectContextInstance.countTokens(text);
  }
 }
--- a/ts/classes.diffprocessor.ts
+++ b/ts/classes.diffprocessor.ts
@@ -0,0 +1,353 @@
 /**
 * Intelligent git diff processor that handles large diffs by sampling and prioritization
 * instead of blind truncation.
 */
 export interface IDiffFileInfo {
  filepath: string;
  status: 'added' | 'modified' | 'deleted';
  linesAdded: number;
  linesRemoved: number;
  totalLines: number;
  estimatedTokens: number;
  diffContent: string;
 }
 export interface IProcessedDiff {
  summary: string;          // Human-readable overview
  fullDiffs: string[];      // Small files included fully
  summarizedDiffs: string[]; // Medium files with head/tail
  metadataOnly: string[];   // Large files, just stats
  totalFiles: number;
  totalTokens: number;
 }
 export interface IDiffProcessorOptions {
  maxDiffTokens?: number;      // Maximum tokens for entire diff section (default: 100000)
  smallFileLines?: number;     // Files <= this are included fully (default: 50)
  mediumFileLines?: number;    // Files <= this are summarized (default: 200)
  sampleHeadLines?: number;    // Lines to show at start of medium files (default: 20)
  sampleTailLines?: number;    // Lines to show at end of medium files (default: 20)
 }
 export class DiffProcessor {
  private options: Required<IDiffProcessorOptions>;
  constructor(options: IDiffProcessorOptions = {}) {
    this.options = {
      maxDiffTokens: options.maxDiffTokens ?? 100000,
      smallFileLines: options.smallFileLines ?? 50,
      mediumFileLines: options.mediumFileLines ?? 200,
      sampleHeadLines: options.sampleHeadLines ?? 20,
      sampleTailLines: options.sampleTailLines ?? 20,
    };
  }
  /**
   * Process an array of git diffs into a structured, token-efficient format
   */
  public processDiffs(diffStringArray: string[]): IProcessedDiff {
    // Parse all diffs into file info objects
    const fileInfos: IDiffFileInfo[] = diffStringArray
      .map(diffString => this.parseDiffFile(diffString))
      .filter(info => info !== null) as IDiffFileInfo[];
    // Prioritize files (source files first, build artifacts last)
    const prioritized = this.prioritizeFiles(fileInfos);
    const result: IProcessedDiff = {
      summary: '',
      fullDiffs: [],
      summarizedDiffs: [],
      metadataOnly: [],
      totalFiles: prioritized.length,
      totalTokens: 0,
    };
    let tokensUsed = 0;
    const tokenBudget = this.options.maxDiffTokens;
    // Categorize and include files based on size and token budget
    for (const fileInfo of prioritized) {
      const remainingBudget = tokenBudget - tokensUsed;
      if (remainingBudget <= 0) {
        // Budget exhausted - rest are metadata only
        result.metadataOnly.push(this.formatMetadataOnly(fileInfo));
        continue;
      }
      if (fileInfo.totalLines <= this.options.smallFileLines) {
        // Small file - include fully if budget allows
        if (fileInfo.estimatedTokens <= remainingBudget) {
          const statusPrefix = this.getFileStatusPrefix(fileInfo);
          result.fullDiffs.push(`${statusPrefix}${fileInfo.diffContent}`);
          tokensUsed += fileInfo.estimatedTokens;
        } else {
          result.metadataOnly.push(this.formatMetadataOnly(fileInfo));
        }
      } else if (fileInfo.totalLines <= this.options.mediumFileLines) {
        // Medium file - try to include summary with head/tail
        const summary = this.extractDiffSample(
          fileInfo,
          this.options.sampleHeadLines,
          this.options.sampleTailLines
        );
        const summaryTokens = Math.ceil(summary.length / 4); // Rough estimate
        if (summaryTokens <= remainingBudget) {
          result.summarizedDiffs.push(summary);
          tokensUsed += summaryTokens;
        } else {
          result.metadataOnly.push(this.formatMetadataOnly(fileInfo));
        }
      } else {
        // Large file - metadata only
        result.metadataOnly.push(this.formatMetadataOnly(fileInfo));
      }
    }
    result.totalTokens = tokensUsed;
    result.summary = this.generateSummary(result);
    return result;
  }
  /**
   * Format the processed diff for inclusion in context
   */
  public formatForContext(processed: IProcessedDiff): string {
    const sections: string[] = [];
    // Summary section
    sections.push('====== GIT DIFF SUMMARY ======');
    sections.push(processed.summary);
    sections.push('');
    // Full diffs section
    if (processed.fullDiffs.length > 0) {
      sections.push(`====== FULL DIFFS (${processed.fullDiffs.length} files) ======`);
      sections.push(processed.fullDiffs.join('\n\n'));
      sections.push('');
    }
    // Summarized diffs section
    if (processed.summarizedDiffs.length > 0) {
      sections.push(`====== SUMMARIZED DIFFS (${processed.summarizedDiffs.length} files) ======`);
      sections.push(processed.summarizedDiffs.join('\n\n'));
      sections.push('');
    }
    // Metadata only section
    if (processed.metadataOnly.length > 0) {
      sections.push(`====== METADATA ONLY (${processed.metadataOnly.length} files) ======`);
      sections.push(processed.metadataOnly.join('\n'));
      sections.push('');
    }
    sections.push('====== END OF GIT DIFF ======');
    return sections.join('\n');
  }
  /**
   * Parse a single git diff string into file information
   */
  private parseDiffFile(diffString: string): IDiffFileInfo | null {
    if (!diffString || diffString.trim().length === 0) {
      return null;
    }
    const lines = diffString.split('\n');
    let filepath = '';
    let status: 'added' | 'modified' | 'deleted' = 'modified';
    let linesAdded = 0;
    let linesRemoved = 0;
    // Parse diff header to extract filepath and status
    for (const line of lines) {
      if (line.startsWith('--- a/')) {
        filepath = line.substring(6);
      } else if (line.startsWith('+++ b/')) {
        const newPath = line.substring(6);
        if (newPath === '/dev/null') {
          status = 'deleted';
        } else if (filepath === '/dev/null') {
          status = 'added';
          filepath = newPath;
        } else {
          filepath = newPath;
        }
      } else if (line.startsWith('+') && !line.startsWith('+++')) {
        linesAdded++;
      } else if (line.startsWith('-') && !line.startsWith('---')) {
        linesRemoved++;
      }
    }
    const totalLines = linesAdded + linesRemoved;
    const estimatedTokens = Math.ceil(diffString.length / 4);
    return {
      filepath,
      status,
      linesAdded,
      linesRemoved,
      totalLines,
      estimatedTokens,
      diffContent: diffString,
    };
  }
  /**
   * Prioritize files by importance (source files before build artifacts)
   */
  private prioritizeFiles(files: IDiffFileInfo[]): IDiffFileInfo[] {
    return files.sort((a, b) => {
      const scoreA = this.getFileImportanceScore(a.filepath);
      const scoreB = this.getFileImportanceScore(b.filepath);
      return scoreB - scoreA; // Higher score first
    });
  }
  /**
   * Calculate importance score for a file path
   */
  private getFileImportanceScore(filepath: string): number {
    // Source files - highest priority
    if (filepath.match(/^(src|lib|app|components|pages|api)\//)) {
      return 100;
    }
    // Test files - high priority
    if (filepath.match(/\.(test|spec)\.(ts|js|tsx|jsx)$/) || filepath.startsWith('test/')) {
      return 80;
    }
    // Configuration files - medium-high priority
    if (filepath.match(/\.(json|yaml|yml|toml|config\.(ts|js))$/)) {
      return 60;
    }
    // Documentation - medium priority
    if (filepath.match(/\.(md|txt|rst)$/)) {
      return 40;
    }
    // Build artifacts - low priority
    if (filepath.match(/^(dist|build|out|\.next|public\/dist)\//)) {
      return 10;
    }
    // Start with default priority
    let score = 50;
    // Boost interface/type files - they're usually small but critical
    if (filepath.includes('interfaces/') || filepath.includes('.types.')) {
      score += 20;
    }
    // Boost entry points
    if (filepath.endsWith('index.ts') || filepath.endsWith('mod.ts')) {
      score += 15;
    }
    return score;
  }
  /**
   * Extract head and tail lines from a diff, omitting the middle
   */
  private extractDiffSample(fileInfo: IDiffFileInfo, headLines: number, tailLines: number): string {
    const lines = fileInfo.diffContent.split('\n');
    const totalLines = lines.length;
    if (totalLines <= headLines + tailLines) {
      // File is small enough to include fully
      return fileInfo.diffContent;
    }
    // Extract file metadata from diff header
    const headerLines: string[] = [];
    let bodyStartIndex = 0;
    for (let i = 0; i < lines.length; i++) {
      if (lines[i].startsWith('@@')) {
        headerLines.push(...lines.slice(0, i + 1));
        bodyStartIndex = i + 1;
        break;
      }
    }
    const bodyLines = lines.slice(bodyStartIndex);
    const head = bodyLines.slice(0, headLines);
    const tail = bodyLines.slice(-tailLines);
    const omittedLines = bodyLines.length - headLines - tailLines;
    const statusEmoji = fileInfo.status === 'added' ? '➕' :
                       fileInfo.status === 'deleted' ? '➖' : '📝';
    const parts: string[] = [];
    parts.push(`${statusEmoji} FILE: ${fileInfo.filepath}`);
    parts.push(`CHANGES: +${fileInfo.linesAdded} lines, -${fileInfo.linesRemoved} lines (${fileInfo.totalLines} total)`);
    parts.push('');
    parts.push(...headerLines);
    parts.push(...head);
    parts.push('');
    parts.push(`[... ${omittedLines} lines omitted - use Read tool to see full file ...]`);
    parts.push('');
    parts.push(...tail);
    return parts.join('\n');
  }
  /**
   * Get file status prefix with emoji
   */
  private getFileStatusPrefix(fileInfo: IDiffFileInfo): string {
    const statusEmoji = fileInfo.status === 'added' ? '➕' :
                       fileInfo.status === 'deleted' ? '➖' : '📝';
    return `${statusEmoji} `;
  }
  /**
   * Extract filepath from diff content
   */
  private extractFilepathFromDiff(diffContent: string): string {
    const lines = diffContent.split('\n');
    for (const line of lines) {
      if (line.startsWith('+++ b/')) {
        return line.substring(6);
      }
    }
    return 'unknown';
  }
  /**
   * Format file info as metadata only
   */
  private formatMetadataOnly(fileInfo: IDiffFileInfo): string {
    const statusEmoji = fileInfo.status === 'added' ? '➕' :
                       fileInfo.status === 'deleted' ? '➖' : '📝';
    return `${statusEmoji} ${fileInfo.filepath} (+${fileInfo.linesAdded}, -${fileInfo.linesRemoved})`;
  }
  /**
   * Generate human-readable summary of processed diff
   */
  private generateSummary(result: IProcessedDiff): string {
    const parts: string[] = [];
    parts.push(`Files changed: ${result.totalFiles} total`);
    parts.push(`- ${result.fullDiffs.length} included in full`);
    parts.push(`- ${result.summarizedDiffs.length} summarized (head/tail shown)`);
    parts.push(`- ${result.metadataOnly.length} metadata only`);
    parts.push(`Estimated tokens: ~${result.totalTokens.toLocaleString()}`);
    if (result.metadataOnly.length > 0) {
      parts.push('');
      parts.push('NOTE: Some files excluded to stay within token budget.');
      parts.push('Use Read tool with specific file paths to see full content.');
    }
    return parts.join('\n');
  }
 }
--- a/ts/classes.typedoc.ts
+++ b/ts/classes.typedoc.ts
@@ -33,19 +33,19 @@ export class TypeDoc {
      include: [],
    };
    let startDirectory = '';
-    if (plugins.smartfile.fs.isDirectory(plugins.path.join(paths.cwd, './ts'))) {
+    if (await plugins.fsInstance.directory(plugins.path.join(paths.cwd, './ts')).exists()) {
      data.include.push(plugins.path.join(paths.cwd, './ts/**/*'));
      startDirectory = 'ts';
    }
-    if (plugins.smartfile.fs.isDirectory(plugins.path.join(paths.cwd, './ts_web'))) {
+    if (await plugins.fsInstance.directory(plugins.path.join(paths.cwd, './ts_web')).exists()) {
      data.include.push(plugins.path.join(paths.cwd, './ts_web/**/*'));
      if (!startDirectory) {
        startDirectory = 'ts_web';
      }
    }
-    await plugins.smartfile.memory.toFs(JSON.stringify(data), paths.tsconfigFile);
+    await plugins.fsInstance.file(paths.tsconfigFile).encoding('utf8').write(JSON.stringify(data));
    let targetDir = paths.publicDir;
    if (options?.publicSubdir) {
      targetDir = plugins.path.join(targetDir, options.publicSubdir);
@@ -53,6 +53,6 @@ export class TypeDoc {
    await this.smartshellInstance.exec(
      `typedoc --tsconfig ${paths.tsconfigFile} --out ${targetDir} ${startDirectory}/index.ts`,
    );
-    plugins.smartfile.fs.remove(paths.tsconfigFile);
+    await plugins.fsInstance.file(paths.tsconfigFile).delete();
  }
 }
--- a/ts/cli.ts
+++ b/ts/cli.ts
@@ -28,20 +28,51 @@ export const run = async () => {
  });
  tsdocCli.addCommand('aidoc').subscribe(async (argvArg) => {
    logger.log('info', `Generating new readme...`);
    logger.log('info', `This may take some time...`);
    const aidocInstance = new AiDoc();
    await aidocInstance.start();
-    aidocInstance.buildReadme(paths.cwd);
+
    logger.log('info', `Generating new readme...`);
    logger.log('info', `This may take some time...`);
    await aidocInstance.buildReadme(paths.cwd);
    logger.log('info', `Generating new keywords...`);
    logger.log('info', `This may take some time...`);
-    aidocInstance.buildDescription(paths.cwd);
+    await aidocInstance.buildDescription(paths.cwd);
  });
  tsdocCli.addCommand('readme').subscribe(async (argvArg) => {
    const aidocInstance = new AiDoc();
    await aidocInstance.start();
    logger.log('info', `Generating new readme...`);
    logger.log('info', `This may take some time...`);
    await aidocInstance.buildReadme(paths.cwd);
  });
  tsdocCli.addCommand('description').subscribe(async (argvArg) => {
    const aidocInstance = new AiDoc();
    await aidocInstance.start();
    logger.log('info', `Generating new description and keywords...`);
    logger.log('info', `This may take some time...`);
    await aidocInstance.buildDescription(paths.cwd);
  });
  tsdocCli.addCommand('commit').subscribe(async (argvArg) => {
    const aidocInstance = new AiDoc();
    await aidocInstance.start();
    logger.log('info', `Generating commit message...`);
    logger.log('info', `This may take some time...`);
    const commitObject = await aidocInstance.buildNextCommitObject(paths.cwd);
    logger.log('ok', `Commit message generated:`);
    console.log(JSON.stringify(commitObject, null, 2));
  });
  tsdocCli.addCommand('test').subscribe((argvArg) => {
    tsdocCli.triggerCommand('typedoc', argvArg);
    process.on('exit', async () => {
-      await plugins.smartfile.fs.remove(paths.publicDir);
+      await plugins.fsInstance.directory(paths.publicDir).recursive().delete();
    });
  });
--- a/ts/plugins.ts
+++ b/ts/plugins.ts
@@ -6,10 +6,13 @@ export { path };
 // pushrocks scope
 import * as npmextra from '@push.rocks/npmextra';
 import * as qenv from '@push.rocks/qenv';
 import * as smartagent from '@push.rocks/smartagent';
 import * as smartagentTools from '@push.rocks/smartagent/tools';
 import * as smartai from '@push.rocks/smartai';
 import * as smartcli from '@push.rocks/smartcli';
 import * as smartdelay from '@push.rocks/smartdelay';
 import * as smartfile from '@push.rocks/smartfile';
 import * as smartfs from '@push.rocks/smartfs';
 import * as smartgit from '@push.rocks/smartgit';
 import * as smartinteract from '@push.rocks/smartinteract';
 import * as smartlog from '@push.rocks/smartlog';
@@ -21,10 +24,13 @@ import * as smarttime from '@push.rocks/smarttime';
 export {
  npmextra,
  qenv,
  smartagent,
  smartagentTools,
  smartai,
  smartcli,
  smartdelay,
  smartfile,
  smartfs,
  smartgit,
  smartinteract,
  smartlog,
@@ -34,6 +40,18 @@ export {
  smarttime,
 };
 // Create a shared SmartFs instance for filesystem operations
 const smartFsNodeProvider = new smartfs.SmartFsProviderNode();
 export const fsInstance = new smartfs.SmartFs(smartFsNodeProvider);
 // Create a shared SmartFileFactory for in-memory file operations
 export const smartfileFactory = smartfile.SmartFileFactory.nodeFs();
 // @git.zone scope
 import * as tspublish from '@git.zone/tspublish';
 export { tspublish };
 // third party scope
 import * as typedoc from 'typedoc';
Author	SHA1	Message	Date
Juergen Kunz	fc85f28f69	v2.0.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2026-03-11 18:58:49 +00:00
Juergen Kunz	6b2957b272	BREAKING CHANGE(aidoc): migrate agent orchestration to new runAgent API and filesystem tools; refactor model handling and update README and tests	2026-03-11 18:58:48 +00:00
Juergen Kunz	883985dbc0	v1.12.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2026-01-04 15:20:27 +00:00
Juergen Kunz	21006b41d0	feat(commit): add token budgeting and dynamic diff token calculation to avoid OpenAI context limit issues	2026-01-04 15:20:27 +00:00
Juergen Kunz	5d0411a5ba	v1.11.4 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-16 10:07:47 +00:00
Juergen Kunz	39f5410b76	fix(aidocs_classes): clarify recommendedNextVersionMessage field to require only the description body without the type(scope) prefix	2025-12-16 10:07:47 +00:00
Juergen Kunz	1a517fdd1b	1.11.3 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-15 15:54:45 +00:00
Juergen Kunz	90af6eb1b1	update	2025-12-15 15:54:25 +00:00
Juergen Kunz	3485392979	1.11.2 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-15 15:22:45 +00:00
Juergen Kunz	89adae2cff	update	2025-12-15 15:22:35 +00:00
Juergen Kunz	3451ab7456	update	2025-12-15 15:14:16 +00:00
Juergen Kunz	bcded1eafa	update	2025-12-15 14:34:02 +00:00
Juergen Kunz	9cae46e2fe	update	2025-12-15 14:33:58 +00:00
Juergen Kunz	65c1df30da	1.11.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-15 12:02:16 +00:00
Juergen Kunz	e8f2add812	fix(dependencies): update	2025-12-15 12:02:13 +00:00
Juergen Kunz	8fcc304ee3	v1.11.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-15 11:36:03 +00:00
Juergen Kunz	69802b46b6	feat(commit): Integrate DualAgentOrchestrator for commit message generation and improve diff/context handling	2025-12-15 11:36:03 +00:00
Juergen Kunz	e500455557	1.10.2 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-13 22:50:26 +00:00
Juergen Kunz	4029691ccd	1.10.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-13 11:42:43 +00:00
Juergen Kunz	3b1c84d7e8	fix(npmextra): update to new format	2025-12-13 11:42:39 +00:00
Juergen Kunz	f8d0895aab	v1.10.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-12-02 12:17:10 +00:00
Juergen Kunz	d7ec2220a1	feat(diff-processor): Improve diff sampling and file prioritization: increase inclusion thresholds, expand sampled context, and boost priority for interface/type and entry-point files	2025-12-02 12:17:10 +00:00
Juergen Kunz	c24ce31b1f	1.9.2 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-04 03:43:27 +00:00
Juergen Kunz	fec2017cc6	fix(deps): Update dependencies and devDependencies to newer versions (bump multiple packages)	2025-11-04 03:43:27 +00:00
Juergen Kunz	88fac91c79	1.9.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-04 02:28:55 +00:00
Juergen Kunz	ce4da89da9	fix(iterative-context-builder): Rely on DiffProcessor for git diff pre-processing; remove raw char truncation, raise diff token safety, and improve logging	2025-11-04 02:28:55 +00:00
Juergen Kunz	6524adea18	1.9.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-04 02:19:57 +00:00
Juergen Kunz	4bf0c02618	feat(context): Add intelligent DiffProcessor to summarize and prioritize git diffs and integrate it into the commit context pipeline	2025-11-04 02:19:57 +00:00
Juergen Kunz	f84a65217d	1.8.3 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-04 01:37:15 +00:00
Juergen Kunz	3f22fc91ae	fix(context): Prevent enormous git diffs and OOM during context building by adding exclusion patterns, truncation, and diagnostic logging	2025-11-04 01:37:15 +00:00
Juergen Kunz	11e65b92ec	1.8.2 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-03 17:53:03 +00:00
Juergen Kunz	0a3080518f	1.8.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-03 17:50:09 +00:00
Juergen Kunz	d0a4ddbb4b	fix(git diff): improve git diff	2025-11-03 17:49:35 +00:00
Juergen Kunz	481339d3cb	1.8.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-03 13:37:16 +00:00
Juergen Kunz	ebc3d760af	feat(context): Wire OpenAI provider through task context factory and add git-diff support to iterative context builder	2025-11-03 13:37:16 +00:00
Juergen Kunz	a6d678e36c	1.7.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-03 13:19:29 +00:00
Juergen Kunz	8c3e16a4f2	feat(IterativeContextBuilder): Add iterative AI-driven context builder and integrate into task factory; add tests and iterative configuration	2025-11-03 13:19:29 +00:00
Juergen Kunz	2276fb0c0c	1.6.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-03 11:04:21 +00:00
Juergen Kunz	0a9d535df4	fix(context): Improve context building, caching and test robustness	2025-11-03 11:04:21 +00:00
Juergen Kunz	d46fd1590e	1.6.0 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 0s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-11-02 23:07:59 +00:00
Juergen Kunz	1d7317f063	feat(context): Introduce smart context system: analyzer, lazy loader, cache and README/docs improvements	2025-11-02 23:07:59 +00:00
Juergen Kunz	fe5121ec9c	1.5.2 Some checks failed Default (tags) / security (push) Failing after 1s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-09-07 07:54:04 +00:00
Juergen Kunz	c084b20390	fix(package): Bump dependencies, refine test script and imports, and overhaul README and docs	2025-09-07 07:54:04 +00:00
Juergen Kunz	6f024536a8	1.5.1 Some checks failed Default (tags) / security (push) Failing after 0s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-08-16 11:20:39 +00:00
Juergen Kunz	2405fb3370	fix(aidoc): Bump dependencies, add pnpm workspace config, and add AiDoc.stop()	2025-08-16 11:20:39 +00:00
Philipp Kunz	8561940b8c	1.5.0 Some checks failed Default (tags) / security (push) Failing after 1s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-05-14 11:27:38 +00:00
Philipp Kunz	ab273ea75c	feat(docs): Update project metadata and documentation to reflect comprehensive AI-enhanced features and improved installation and usage instructions	2025-05-14 11:27:38 +00:00
Philipp Kunz	620737566f	1.4.5 Some checks failed Default (tags) / security (push) Failing after 1s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-05-13 21:20:10 +00:00
Philipp Kunz	23453bf16b	fix(dependencies): Upgrade various dependency versions and update package manager configuration	2025-05-13 21:20:10 +00:00
Philipp Kunz	84947cfb80	1.4.4 Some checks failed Default (tags) / security (push) Failing after 1s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-02-25 18:33:51 +00:00
Philipp Kunz	1a9ac9091d	fix(dependencies): Update dependencies to latest versions	2025-02-25 18:33:51 +00:00
Philipp Kunz	88b93b8b83	1.4.3 Some checks failed Default (tags) / security (push) Failing after 2s Details Default (tags) / test (push) Failing after 1s Details Default (tags) / release (push) Has been skipped Details Default (tags) / metadata (push) Has been skipped Details	2025-01-14 17:42:19 +01:00
Philipp Kunz	77279a9135	fix(aidocs_classes): Improve readme generation instructions to ensure correct markdown formatting.	2025-01-14 17:42:19 +01:00
Philipp Kunz	7426addbdd	1.4.2	2024-10-28 21:45:28 +01:00
Philipp Kunz	58d060d729	fix(cli): Ensure async completion for aidoc readme and description generation	2024-10-28 21:45:28 +01:00
Philipp Kunz	370cbfe6f3	1.4.1	2024-10-28 21:20:51 +01:00
Philipp Kunz	2adb4e8cb0	fix(readme): Correct async call to getModuleSubDirs in readme generation.	2024-10-28 21:20:50 +01:00
Philipp Kunz	e8608b1cae	1.4.0	2024-10-28 21:16:00 +01:00
Philipp Kunz	33fa7fa337	feat(aidocs): Added support for building readmes for sub-modules in aidocs	2024-10-28 21:15:59 +01:00