tstest/readme.plan.md

Improvement Plan for tstest and tapbundle

!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!

1. Enhanced Communication Between tapbundle and tstest

1.1 Real-time Test Progress API

• Create a bidirectional communication channel between tapbundle and tstest
• Emit events for test lifecycle stages (start, progress, completion)
• Allow tstest to subscribe to tapbundle events for better progress reporting
• Implement a standardized message format for test metadata

1.2 Rich Error Reporting

• Pass structured error objects from tapbundle to tstest
• Include stack traces, code snippets, and contextual information
• Support for error categorization (assertion failures, timeouts, uncaught exceptions)
• Visual diff output for failed assertions

2. Enhanced toolsArg Functionality

2.1 Test Flow Control ✅

tap.test('conditional test', async (toolsArg) => {
  const result = await someOperation();
  
  // Skip the rest of the test
  if (!result) {
    return toolsArg.skip('Precondition not met');
  }
  
  // Conditional skipping
  await toolsArg.skipIf(condition, 'Reason for skipping');
  
  // Mark test as todo
  await toolsArg.todo('Not implemented yet');
});

2.2 Test Metadata and Configuration ✅

// Fluent syntax ✅
tap.tags('slow', 'integration')
  .priority('high')
  .timeout(5000)
  .retry(3)
  .test('configurable test', async (toolsArg) => {
    // Test implementation
  });

2.3 Test Data and Context Sharing ✅

tap.test('data-driven test', async (toolsArg) => {
  // Access shared context ✅
  const sharedData = toolsArg.context.get('sharedData');
  
  // Set data for other tests ✅
  toolsArg.context.set('resultData', computedValue);
  
  // Parameterized test data (not yet implemented)
  const testData = toolsArg.data<TestInput>();
  expect(processData(testData)).toEqual(expected);
});

3. Nested Tests and Test Suites

3.1 Test Grouping with describe() ✅

tap.describe('User Authentication', () => {
  tap.beforeEach(async (toolsArg) => {
    // Setup for each test in this suite
    await toolsArg.context.set('db', await createTestDatabase());
  });
  
  tap.afterEach(async (toolsArg) => {
    // Cleanup after each test
    await toolsArg.context.get('db').cleanup();
  });
  
  tap.test('should login with valid credentials', async (toolsArg) => {
    // Test implementation
  });
  
  tap.describe('Password Reset', () => {
    tap.test('should send reset email', async (toolsArg) => {
      // Nested test
    });
  });
});

3.2 Hierarchical Test Organization

• Support for multiple levels of nesting
• Inherited context and configuration from parent suites
• Aggregated reporting for test suites
• Suite-level lifecycle hooks

4. Advanced Test Features

4.1 Snapshot Testing

tap.test('component render', async (toolsArg) => {
  const output = renderComponent(props);
  
  // Compare with stored snapshot
  await toolsArg.matchSnapshot(output, 'component-output');
});

4.2 Performance Benchmarking

tap.test('performance test', async (toolsArg) => {
  const benchmark = toolsArg.benchmark();
  
  // Run operation
  await expensiveOperation();
  
  // Assert performance constraints
  benchmark.expect({
    maxDuration: 1000,
    maxMemory: '100MB'
  });
});

4.3 Test Fixtures and Factories ✅

tap.test('with fixtures', async (toolsArg) => {
  // Create test fixtures
  const user = await toolsArg.fixture('user', { name: 'Test User' });
  const post = await toolsArg.fixture('post', { author: user });
  
  // Use factory functions
  const users = await toolsArg.factory('user').createMany(5);
});

5. Test Execution Improvements

5.1 Parallel Test Execution ✅

• Run independent tests concurrently ✅
• Configurable concurrency limits (via file naming convention)
• Resource pooling for shared resources
• Proper isolation between parallel tests ✅

Implementation:

• Tests with para__<groupNumber> in filename run in parallel
• Different groups run sequentially
• Tests without para__ run serially

5.2 Watch Mode

• Automatically re-run tests on file changes
• Intelligent test selection based on changed files
• Fast feedback loop for development
• Integration with IDE/editor plugins

5.3 Advanced Test Filtering ✅ (partially)

// Run tests by tags ✅
tstest --tags "unit,fast"

// Exclude tests by pattern (not yet implemented)
tstest --exclude "**/slow/**"

// Run only failed tests from last run (not yet implemented)
tstest --failed

// Run tests modified in git (not yet implemented)
tstest --changed

6. Reporting and Analytics

6.1 Custom Reporters

• Plugin architecture for custom reporters
• Built-in reporters: JSON, JUnit, HTML, Markdown
• Real-time streaming reporters
• Aggregated test metrics and trends

6.2 Coverage Integration

• Built-in code coverage collection
• Coverage thresholds and enforcement
• Coverage trending over time
• Integration with CI/CD pipelines

6.3 Test Analytics Dashboard

• Web-based dashboard for test results
• Historical test performance data
• Flaky test detection
• Test impact analysis

7. Developer Experience

7.1 Better Error Messages

• Clear, actionable error messages
• Suggestions for common issues
• Links to documentation
• Code examples in error output

7.2 Interactive Mode (Needs Detailed Specification)

• REPL for exploring test failures
  • Need to define: How to enter interactive mode? When tests fail?
  • What commands/features should be available in the REPL?
• Debugging integration
  • Node.js inspector protocol integration?
  • Breakpoint support?
• Step-through test execution
  • Pause between tests?
  • Step into/over/out functionality?
• Interactive test data manipulation
  • Modify test inputs on the fly?
  • Inspect intermediate values?

7.3 VS Code Extension (Scratched)

• Test explorer integration
• Inline test results
• CodeLens for running individual tests
• Debugging support

Implementation Phases

Phase 1: Core Enhancements (Priority: High) ✅

1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry) ✅
2. Add basic test grouping with describe() ✅
3. Improve error reporting between tapbundle and tstest ✅

Phase 2: Advanced Features (Priority: Medium)

1. Implement nested test suites ✅ (basic describe support)
2. Add snapshot testing ✅
3. Create test fixture system ✅
4. Implement parallel test execution ✅

Phase 3: Developer Experience (Priority: Medium)

1. Add watch mode
2. Implement custom reporters
3. Create VS Code extension (Scratched)
4. Add interactive debugging (Needs detailed spec first)

Phase 4: Analytics and Performance (Priority: Low)

1. Build test analytics dashboard
2. Add performance benchmarking
3. Implement coverage integration
4. Create trend analysis tools

Technical Considerations

API Design Principles

• Maintain backward compatibility
• Progressive enhancement approach
• Opt-in features to avoid breaking changes
• Clear migration paths for new features

Performance Goals

• Minimal overhead for test execution
• Efficient parallel execution
• Fast test discovery
• Optimized browser test bundling

Integration Points

• Clean interfaces between tstest and tapbundle
• Extensible plugin architecture
• Standard test result format
• Compatible with existing CI/CD tools