feat(package): Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.
This commit is contained in:
		| @@ -1,5 +1,12 @@ | |||||||
| # Changelog | # Changelog | ||||||
|  |  | ||||||
|  | ## 2025-05-15 - 1.6.0 - feat(package) | ||||||
|  | Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements. | ||||||
|  |  | ||||||
|  | - Replaced 'main' and 'typings' in package.json with explicit exports for improved module resolution. | ||||||
|  | - Added .claude/settings.local.json to configure permissions for bash commands and web fetches. | ||||||
|  | - Updated readme.plan.md with a comprehensive roadmap covering enhanced error reporting, rich test metadata, nested test suites, and advanced test features. | ||||||
|  |  | ||||||
| ## 2025-05-15 - 1.5.0 - feat(cli) | ## 2025-05-15 - 1.5.0 - feat(cli) | ||||||
| Improve test runner configuration: update test scripts, reorganize test directories, update dependencies and add local settings for command permissions. | Improve test runner configuration: update test scripts, reorganize test directories, update dependencies and add local settings for command permissions. | ||||||
|  |  | ||||||
|   | |||||||
| @@ -3,8 +3,11 @@ | |||||||
|   "version": "1.5.0", |   "version": "1.5.0", | ||||||
|   "private": false, |   "private": false, | ||||||
|   "description": "a test utility to run tests that match test/**/*.ts", |   "description": "a test utility to run tests that match test/**/*.ts", | ||||||
|   "main": "dist_ts/index.js", |   "exports": { | ||||||
|   "typings": "dist_ts/index.d.ts", |     ".": "./dist_ts/index.js", | ||||||
|  |     "./tapbundle": "./dist_ts_tapbundle/index.js", | ||||||
|  |     "./tapbundle_node": "./dist_ts_tapbundle_node/index.js" | ||||||
|  |   }, | ||||||
|   "type": "module", |   "type": "module", | ||||||
|   "author": "Lossless GmbH", |   "author": "Lossless GmbH", | ||||||
|   "license": "MIT", |   "license": "MIT", | ||||||
|   | |||||||
							
								
								
									
										275
									
								
								readme.plan.md
									
									
									
									
									
								
							
							
						
						
									
										275
									
								
								readme.plan.md
									
									
									
									
									
								
							| @@ -1,41 +1,256 @@ | |||||||
| # Plan for showing logs for failed tests | # Improvement Plan for tstest and tapbundle | ||||||
|  |  | ||||||
| !! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !! | !! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !! | ||||||
|  |  | ||||||
| ## Goal | ## 1. Enhanced Communication Between tapbundle and tstest | ||||||
| When a test fails, we want to display all the console logs from that failed test in the terminal, even without the --verbose flag. This makes debugging failed tests much easier. |  | ||||||
|  |  | ||||||
| ## Current Behavior | ### 1.1 Real-time Test Progress API | ||||||
| - Default mode: Only shows test results, no console logs | - Create a bidirectional communication channel between tapbundle and tstest | ||||||
| - Verbose mode: Shows all console logs from all tests | - Emit events for test lifecycle stages (start, progress, completion) | ||||||
| - When a test fails: Only shows the error message | - Allow tstest to subscribe to tapbundle events for better progress reporting | ||||||
|  | - Implement a standardized message format for test metadata | ||||||
|  |  | ||||||
| ## Desired Behavior | ### 1.2 Rich Error Reporting | ||||||
| - Default mode: Shows test results, and IF a test fails, shows all console logs from that failed test | - Pass structured error objects from tapbundle to tstest | ||||||
| - Verbose mode: Shows all console logs from all tests (unchanged) | - Include stack traces, code snippets, and contextual information | ||||||
| - When a test fails: Shows all console logs from that test plus the error | - Support for error categorization (assertion failures, timeouts, uncaught exceptions) | ||||||
|  | - Visual diff output for failed assertions | ||||||
|  |  | ||||||
| ## Implementation Plan | ## 2. Enhanced toolsArg Functionality | ||||||
|  |  | ||||||
| ### 1. Update TapParser | ### 2.1 Test Flow Control | ||||||
| - Store console logs for each test temporarily | ```typescript | ||||||
| - When a test fails, mark that its logs should be shown | tap.test('conditional test', async (toolsArg) => { | ||||||
|  |   const result = await someOperation(); | ||||||
|    |    | ||||||
| ### 2. Update TsTestLogger |   // Skip the rest of the test | ||||||
| - Add a new method to handle failed test logs |   if (!result) { | ||||||
| - Modify testConsoleOutput to buffer logs when not in verbose mode |     return toolsArg.skip('Precondition not met'); | ||||||
| - When a test fails, flush the buffered logs for that test |   } | ||||||
|    |    | ||||||
| ### 3. Update test result handling |   // Conditional skipping | ||||||
| - When a test fails, trigger display of all buffered logs for that test |   await toolsArg.skipIf(condition, 'Reason for skipping'); | ||||||
| - Clear logs after each test completes successfully |  | ||||||
|    |    | ||||||
| ## Code Changes Needed |   // Mark test as todo | ||||||
| 1. Add log buffering to TapParser |   await toolsArg.todo('Not implemented yet'); | ||||||
| 2. Update TsTestLogger to handle failed test logs | }); | ||||||
| 3. Modify test result processing to show logs on failure | ``` | ||||||
|  |  | ||||||
| ## Files to Modify | ### 2.2 Test Metadata and Configuration | ||||||
| - `ts/tstest.classes.tap.parser.ts` - Add log buffering | ```typescript | ||||||
| - `ts/tstest.logging.ts` - Add failed test log handling | tap.test('configurable test', async (toolsArg) => { | ||||||
| - `ts/tstest.classes.tap.testresult.ts` - May need to store logs |   // Set custom timeout | ||||||
|  |   toolsArg.timeout(5000); | ||||||
|  |    | ||||||
|  |   // Retry on failure | ||||||
|  |   toolsArg.retry(3); | ||||||
|  |    | ||||||
|  |   // Add tags for filtering | ||||||
|  |   toolsArg.tags(['slow', 'integration']); | ||||||
|  |    | ||||||
|  |   // Set test priority | ||||||
|  |   toolsArg.priority('high'); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ### 2.3 Test Data and Context Sharing | ||||||
|  | ```typescript | ||||||
|  | tap.test('data-driven test', async (toolsArg) => { | ||||||
|  |   // Access shared context | ||||||
|  |   const sharedData = toolsArg.context.get('sharedData'); | ||||||
|  |    | ||||||
|  |   // Set data for other tests | ||||||
|  |   toolsArg.context.set('resultData', computedValue); | ||||||
|  |    | ||||||
|  |   // Parameterized test data | ||||||
|  |   const testData = toolsArg.data<TestInput>(); | ||||||
|  |   expect(processData(testData)).toEqual(expected); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ## 3. Nested Tests and Test Suites | ||||||
|  |  | ||||||
|  | ### 3.1 Test Grouping with describe() | ||||||
|  | ```typescript | ||||||
|  | tap.describe('User Authentication', () => { | ||||||
|  |   tap.beforeEach(async (toolsArg) => { | ||||||
|  |     // Setup for each test in this suite | ||||||
|  |     await toolsArg.context.set('db', await createTestDatabase()); | ||||||
|  |   }); | ||||||
|  |    | ||||||
|  |   tap.afterEach(async (toolsArg) => { | ||||||
|  |     // Cleanup after each test | ||||||
|  |     await toolsArg.context.get('db').cleanup(); | ||||||
|  |   }); | ||||||
|  |    | ||||||
|  |   tap.test('should login with valid credentials', async (toolsArg) => { | ||||||
|  |     // Test implementation | ||||||
|  |   }); | ||||||
|  |    | ||||||
|  |   tap.describe('Password Reset', () => { | ||||||
|  |     tap.test('should send reset email', async (toolsArg) => { | ||||||
|  |       // Nested test | ||||||
|  |     }); | ||||||
|  |   }); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ### 3.2 Hierarchical Test Organization | ||||||
|  | - Support for multiple levels of nesting | ||||||
|  | - Inherited context and configuration from parent suites | ||||||
|  | - Aggregated reporting for test suites | ||||||
|  | - Suite-level lifecycle hooks | ||||||
|  |  | ||||||
|  | ## 4. Advanced Test Features | ||||||
|  |  | ||||||
|  | ### 4.1 Snapshot Testing | ||||||
|  | ```typescript | ||||||
|  | tap.test('component render', async (toolsArg) => { | ||||||
|  |   const output = renderComponent(props); | ||||||
|  |    | ||||||
|  |   // Compare with stored snapshot | ||||||
|  |   await toolsArg.matchSnapshot(output, 'component-output'); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ### 4.2 Performance Benchmarking | ||||||
|  | ```typescript | ||||||
|  | tap.test('performance test', async (toolsArg) => { | ||||||
|  |   const benchmark = toolsArg.benchmark(); | ||||||
|  |    | ||||||
|  |   // Run operation | ||||||
|  |   await expensiveOperation(); | ||||||
|  |    | ||||||
|  |   // Assert performance constraints | ||||||
|  |   benchmark.expect({ | ||||||
|  |     maxDuration: 1000, | ||||||
|  |     maxMemory: '100MB' | ||||||
|  |   }); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ### 4.3 Test Fixtures and Factories | ||||||
|  | ```typescript | ||||||
|  | tap.test('with fixtures', async (toolsArg) => { | ||||||
|  |   // Create test fixtures | ||||||
|  |   const user = await toolsArg.fixture('user', { name: 'Test User' }); | ||||||
|  |   const post = await toolsArg.fixture('post', { author: user }); | ||||||
|  |    | ||||||
|  |   // Use factory functions | ||||||
|  |   const users = await toolsArg.factory('user').createMany(5); | ||||||
|  | }); | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ## 5. Test Execution Improvements | ||||||
|  |  | ||||||
|  | ### 5.1 Parallel Test Execution | ||||||
|  | - Run independent tests concurrently | ||||||
|  | - Configurable concurrency limits | ||||||
|  | - Resource pooling for shared resources | ||||||
|  | - Proper isolation between parallel tests | ||||||
|  |  | ||||||
|  | ### 5.2 Watch Mode | ||||||
|  | - Automatically re-run tests on file changes | ||||||
|  | - Intelligent test selection based on changed files | ||||||
|  | - Fast feedback loop for development | ||||||
|  | - Integration with IDE/editor plugins | ||||||
|  |  | ||||||
|  | ### 5.3 Advanced Test Filtering | ||||||
|  | ```typescript | ||||||
|  | // Run tests by tags | ||||||
|  | tstest --tags "unit,fast" | ||||||
|  |  | ||||||
|  | // Exclude tests by pattern | ||||||
|  | tstest --exclude "**/slow/**" | ||||||
|  |  | ||||||
|  | // Run only failed tests from last run | ||||||
|  | tstest --failed | ||||||
|  |  | ||||||
|  | // Run tests modified in git | ||||||
|  | tstest --changed | ||||||
|  | ``` | ||||||
|  |  | ||||||
|  | ## 6. Reporting and Analytics | ||||||
|  |  | ||||||
|  | ### 6.1 Custom Reporters | ||||||
|  | - Plugin architecture for custom reporters | ||||||
|  | - Built-in reporters: JSON, JUnit, HTML, Markdown | ||||||
|  | - Real-time streaming reporters | ||||||
|  | - Aggregated test metrics and trends | ||||||
|  |  | ||||||
|  | ### 6.2 Coverage Integration | ||||||
|  | - Built-in code coverage collection | ||||||
|  | - Coverage thresholds and enforcement | ||||||
|  | - Coverage trending over time | ||||||
|  | - Integration with CI/CD pipelines | ||||||
|  |  | ||||||
|  | ### 6.3 Test Analytics Dashboard | ||||||
|  | - Web-based dashboard for test results | ||||||
|  | - Historical test performance data | ||||||
|  | - Flaky test detection | ||||||
|  | - Test impact analysis | ||||||
|  |  | ||||||
|  | ## 7. Developer Experience | ||||||
|  |  | ||||||
|  | ### 7.1 Better Error Messages | ||||||
|  | - Clear, actionable error messages | ||||||
|  | - Suggestions for common issues | ||||||
|  | - Links to documentation | ||||||
|  | - Code examples in error output | ||||||
|  |  | ||||||
|  | ### 7.2 Interactive Mode | ||||||
|  | - REPL for exploring test failures | ||||||
|  | - Debugging integration | ||||||
|  | - Step-through test execution | ||||||
|  | - Interactive test data manipulation | ||||||
|  |  | ||||||
|  | ### 7.3 VS Code Extension | ||||||
|  | - Test explorer integration | ||||||
|  | - Inline test results | ||||||
|  | - CodeLens for running individual tests | ||||||
|  | - Debugging support | ||||||
|  |  | ||||||
|  | ## Implementation Phases | ||||||
|  |  | ||||||
|  | ### Phase 1: Core Enhancements (Priority: High) | ||||||
|  | 1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry) | ||||||
|  | 2. Add basic test grouping with describe() | ||||||
|  | 3. Improve error reporting between tapbundle and tstest | ||||||
|  |  | ||||||
|  | ### Phase 2: Advanced Features (Priority: Medium) | ||||||
|  | 1. Implement nested test suites | ||||||
|  | 2. Add snapshot testing | ||||||
|  | 3. Create test fixture system | ||||||
|  | 4. Implement parallel test execution | ||||||
|  |  | ||||||
|  | ### Phase 3: Developer Experience (Priority: Medium) | ||||||
|  | 1. Add watch mode | ||||||
|  | 2. Implement custom reporters | ||||||
|  | 3. Create VS Code extension | ||||||
|  | 4. Add interactive debugging | ||||||
|  |  | ||||||
|  | ### Phase 4: Analytics and Performance (Priority: Low) | ||||||
|  | 1. Build test analytics dashboard | ||||||
|  | 2. Add performance benchmarking | ||||||
|  | 3. Implement coverage integration | ||||||
|  | 4. Create trend analysis tools | ||||||
|  |  | ||||||
|  | ## Technical Considerations | ||||||
|  |  | ||||||
|  | ### API Design Principles | ||||||
|  | - Maintain backward compatibility | ||||||
|  | - Progressive enhancement approach | ||||||
|  | - Opt-in features to avoid breaking changes | ||||||
|  | - Clear migration paths for new features | ||||||
|  |  | ||||||
|  | ### Performance Goals | ||||||
|  | - Minimal overhead for test execution | ||||||
|  | - Efficient parallel execution | ||||||
|  | - Fast test discovery | ||||||
|  | - Optimized browser test bundling | ||||||
|  |  | ||||||
|  | ### Integration Points | ||||||
|  | - Clean interfaces between tstest and tapbundle | ||||||
|  | - Extensible plugin architecture | ||||||
|  | - Standard test result format | ||||||
|  | - Compatible with existing CI/CD tools | ||||||
| @@ -3,6 +3,6 @@ | |||||||
|  */ |  */ | ||||||
| export const commitinfo = { | export const commitinfo = { | ||||||
|   name: '@git.zone/tstest', |   name: '@git.zone/tstest', | ||||||
|   version: '1.5.0', |   version: '1.6.0', | ||||||
|   description: 'a test utility to run tests that match test/**/*.ts' |   description: 'a test utility to run tests that match test/**/*.ts' | ||||||
| } | } | ||||||
|   | |||||||
		Reference in New Issue
	
	Block a user