tstest/readme.plan.md

Improvement Plan for tstest and tapbundle

!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!

Improved Internal Protocol (NEW - Critical)

Current Issues

• TAP protocol uses # for metadata which conflicts with test descriptions containing #
• Fragile regex parsing that breaks with special characters
• Limited extensibility for new metadata types

Proposed Solution: Protocol V2

• Use Unicode delimiters ⟦TSTEST:META:{}⟧ that won't appear in test names
• Structured JSON metadata format
• Separate protocol blocks for complex data (errors, snapshots)
• Backwards compatible with gradual migration

Implementation

• Phase 1: Add protocol v2 parser alongside v1
• Phase 2: Generate v2 by default with --legacy flag for v1
• Phase 3: Full migration to v2 in next major version

See readme.protocol.md for detailed specification.

Test Configuration System (NEW)

Global Test Configuration via 00init.ts

• Discovery: Check for test/00init.ts before running tests
• Execution: Import and execute before any test files if found
• Purpose: Define project-wide default test settings

tap.settings() API

interface TapSettings {
  // Timing
  timeout?: number;              // Default timeout for all tests (ms)
  slowThreshold?: number;        // Mark tests as slow if they exceed this (ms)
  
  // Execution Control
  bail?: boolean;                // Stop on first test failure
  retries?: number;              // Number of retries for failed tests
  retryDelay?: number;           // Delay between retries (ms)
  
  // Output Control
  suppressConsole?: boolean;     // Suppress console output in passing tests
  verboseErrors?: boolean;       // Show full stack traces
  showTestDuration?: boolean;    // Show duration for each test
  
  // Parallel Execution
  maxConcurrency?: number;       // Max parallel tests (for .para files)
  isolateTests?: boolean;        // Run each test in fresh context
  
  // Lifecycle Hooks
  beforeAll?: () => Promise<void> | void;
  afterAll?: () => Promise<void> | void;
  beforeEach?: (testName: string) => Promise<void> | void;
  afterEach?: (testName: string, passed: boolean) => Promise<void> | void;
  
  // Environment
  env?: Record<string, string>;  // Additional environment variables
  
  // Features
  enableSnapshots?: boolean;     // Enable snapshot testing
  snapshotDirectory?: string;    // Custom snapshot directory
  updateSnapshots?: boolean;     // Update snapshots instead of comparing
}

Settings Inheritance

• Global (00init.ts) → File level → Test level
• More specific settings override less specific ones
• Arrays/objects are merged, primitives are replaced

Implementation Phases

1. Core Infrastructure: Settings storage and merge logic
2. Discovery: 00init.ts loading mechanism
3. Application: Apply settings to test execution
4. Advanced: Parallel execution and snapshot configuration

1. Enhanced Communication Between tapbundle and tstest

1.1 Real-time Test Progress API

• Create a bidirectional communication channel between tapbundle and tstest
• Emit events for test lifecycle stages (start, progress, completion)
• Allow tstest to subscribe to tapbundle events for better progress reporting
• Implement a standardized message format for test metadata

1.2 Rich Error Reporting

• Pass structured error objects from tapbundle to tstest
• Include stack traces, code snippets, and contextual information
• Support for error categorization (assertion failures, timeouts, uncaught exceptions)
• Visual diff output for failed assertions

2. Enhanced toolsArg Functionality

2.3 Test Data and Context Sharing (Partial)

tap.test('data-driven test', async (toolsArg) => {
  // Parameterized test data (not yet implemented)
  const testData = toolsArg.data<TestInput>();
  expect(processData(testData)).toEqual(expected);
});

3. Nested Tests and Test Suites

3.2 Hierarchical Test Organization (Not yet implemented)

• Support for multiple levels of nesting
• Inherited context and configuration from parent suites
• Aggregated reporting for test suites
• Suite-level lifecycle hooks

4. Advanced Test Features

4.1 Snapshot Testing ✅ (Basic implementation complete)

4.2 Performance Benchmarking

tap.test('performance test', async (toolsArg) => {
  const benchmark = toolsArg.benchmark();
  
  // Run operation
  await expensiveOperation();
  
  // Assert performance constraints
  benchmark.expect({
    maxDuration: 1000,
    maxMemory: '100MB'
  });
});

5. Test Execution Improvements

5.2 Watch Mode

• Automatically re-run tests on file changes
• Intelligent test selection based on changed files
• Fast feedback loop for development
• Integration with IDE/editor plugins

5.3 Advanced Test Filtering (Partial)

// Exclude tests by pattern (not yet implemented)
tstest --exclude "**/slow/**"

// Run only failed tests from last run (not yet implemented)
tstest --failed

// Run tests modified in git (not yet implemented)
tstest --changed

6. Reporting and Analytics

6.1 Custom Reporters

• Plugin architecture for custom reporters
• Built-in reporters: JSON, JUnit, HTML, Markdown
• Real-time streaming reporters
• Aggregated test metrics and trends

6.2 Coverage Integration

• Built-in code coverage collection
• Coverage thresholds and enforcement
• Coverage trending over time
• Integration with CI/CD pipelines

6.3 Test Analytics Dashboard

• Web-based dashboard for test results
• Historical test performance data
• Flaky test detection
• Test impact analysis

7. Developer Experience

7.1 Better Error Messages

• Clear, actionable error messages
• Suggestions for common issues
• Links to documentation
• Code examples in error output

Implementation Phases

Phase 1: Improved Internal Protocol (Priority: Critical) (NEW)

1. Implement Protocol V2 parser in tstest
2. Add protocol version negotiation
3. Update tapbundle to generate V2 format with feature flag
4. Test with real-world test suites containing special characters

Phase 2: Test Configuration System (Priority: High)

1. Implement tap.settings() API with TypeScript interfaces
2. Add 00init.ts discovery and loading mechanism
3. Implement settings inheritance and merge logic
4. Apply settings to test execution (timeouts, retries, etc.)

Phase 3: Enhanced Communication (Priority: High)

1. Build on Protocol V2 for richer communication
2. Implement real-time test progress API
3. Add structured error reporting with diffs and traces

Phase 4: Developer Experience (Priority: Medium)

1. Add watch mode
2. Implement custom reporters
3. Complete advanced test filtering options
4. Add performance benchmarking API

Phase 5: Analytics and Performance (Priority: Low)

1. Build test analytics dashboard
2. Implement coverage integration
3. Create trend analysis tools
4. Add test impact analysis

Technical Considerations

API Design Principles

• Maintain backward compatibility
• Progressive enhancement approach
• Opt-in features to avoid breaking changes
• Clear migration paths for new features

Performance Goals

• Minimal overhead for test execution
• Efficient parallel execution
• Fast test discovery
• Optimized browser test bundling

Integration Points

• Clean interfaces between tstest and tapbundle
• Extensible plugin architecture
• Standard test result format
• Compatible with existing CI/CD tools