git.zone/tsdoc
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(commit): add token budgeting and dynamic diff token calculation to avoid OpenAI context limit issues

Browse Source
This commit is contained in:
Jürgen Kunz
2026-01-04 15:20:27 +00:00
parent 5d0411a5ba
commit 21006b41d0
6 changed files with 1270 additions and 926 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
readme.hints.md
View File

@@ -2,4 +2,13 @@
* alternatively can be used through npx, if installed locally
* cli parameters are concluded from ./ts/cli.ts
* this module is not intended for API use.
* Read carefully through the TypeScript files. Don't make stuff up.
* Read carefully through the TypeScript files. Don't make stuff up.
## Token Budgeting (commit.ts)
* OpenAI has a 272,000 token context limit
* The smartagent infrastructure adds ~180,000 tokens of overhead (system messages, tool descriptions, conversation history)
* TOKEN_BUDGET constants in commit.ts control the available tokens for diff content
* Dynamic calculation: 272K - 10K (safety) - 180K (overhead) - 2K (prompt) = 80K tokens for diff
* If token limit errors occur, consider:
- Splitting large commits into smaller ones
- Adjusting SMARTAGENT_OVERHEAD if actual overhead is different