tstest/readme.plan.md

# Improvement Plan for tstest and tapbundle

!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!

## 1. Enhanced Communication Between tapbundle and tstest

### 1.1 Real-time Test Progress API
- Create a bidirectional communication channel between tapbundle and tstest
- Emit events for test lifecycle stages (start, progress, completion)
- Allow tstest to subscribe to tapbundle events for better progress reporting
- Implement a standardized message format for test metadata

### 1.2 Rich Error Reporting
- Pass structured error objects from tapbundle to tstest
- Include stack traces, code snippets, and contextual information
- Support for error categorization (assertion failures, timeouts, uncaught exceptions)
- Visual diff output for failed assertions

## 2. Enhanced toolsArg Functionality

### 2.1 Test Flow Control ✅
```typescript
tap.test('conditional test', async (toolsArg) => {
  const result = await someOperation();
  
  // Skip the rest of the test
  if (!result) {
    return toolsArg.skip('Precondition not met');
  }
  
  // Conditional skipping
  await toolsArg.skipIf(condition, 'Reason for skipping');
  
  // Mark test as todo
  await toolsArg.todo('Not implemented yet');
});
```

### 2.2 Test Metadata and Configuration ✅
```typescript
// Fluent syntax ✅
tap.tags('slow', 'integration')
  .priority('high')
  .timeout(5000)
  .retry(3)
  .test('configurable test', async (toolsArg) => {
    // Test implementation
  });
```

### 2.3 Test Data and Context Sharing ✅
```typescript
tap.test('data-driven test', async (toolsArg) => {
  // Access shared context ✅
  const sharedData = toolsArg.context.get('sharedData');
  
  // Set data for other tests ✅
  toolsArg.context.set('resultData', computedValue);
  
  // Parameterized test data (not yet implemented)
  const testData = toolsArg.data<TestInput>();
  expect(processData(testData)).toEqual(expected);
});
```

## 3. Nested Tests and Test Suites

### 3.1 Test Grouping with describe() ✅
```typescript
tap.describe('User Authentication', () => {
  tap.beforeEach(async (toolsArg) => {
    // Setup for each test in this suite
    await toolsArg.context.set('db', await createTestDatabase());
  });
  
  tap.afterEach(async (toolsArg) => {
    // Cleanup after each test
    await toolsArg.context.get('db').cleanup();
  });
  
  tap.test('should login with valid credentials', async (toolsArg) => {
    // Test implementation
  });
  
  tap.describe('Password Reset', () => {
    tap.test('should send reset email', async (toolsArg) => {
      // Nested test
    });
  });
});
```

### 3.2 Hierarchical Test Organization
- Support for multiple levels of nesting
- Inherited context and configuration from parent suites
- Aggregated reporting for test suites
- Suite-level lifecycle hooks

## 4. Advanced Test Features

### 4.1 Snapshot Testing
```typescript
tap.test('component render', async (toolsArg) => {
  const output = renderComponent(props);
  
  // Compare with stored snapshot
  await toolsArg.matchSnapshot(output, 'component-output');
});
```

### 4.2 Performance Benchmarking
```typescript
tap.test('performance test', async (toolsArg) => {
  const benchmark = toolsArg.benchmark();
  
  // Run operation
  await expensiveOperation();
  
  // Assert performance constraints
  benchmark.expect({
    maxDuration: 1000,
    maxMemory: '100MB'
  });
});
```

### 4.3 Test Fixtures and Factories ✅
```typescript
tap.test('with fixtures', async (toolsArg) => {
  // Create test fixtures
  const user = await toolsArg.fixture('user', { name: 'Test User' });
  const post = await toolsArg.fixture('post', { author: user });
  
  // Use factory functions
  const users = await toolsArg.factory('user').createMany(5);
});
```

## 5. Test Execution Improvements

### 5.1 Parallel Test Execution ✅
- Run independent tests concurrently ✅
- Configurable concurrency limits (via file naming convention)
- Resource pooling for shared resources
- Proper isolation between parallel tests ✅

Implementation:
- Tests with `para__<groupNumber>` in filename run in parallel
- Different groups run sequentially
- Tests without `para__` run serially

### 5.2 Watch Mode
- Automatically re-run tests on file changes
- Intelligent test selection based on changed files
- Fast feedback loop for development
- Integration with IDE/editor plugins

### 5.3 Advanced Test Filtering ✅ (partially)
```typescript
// Run tests by tags ✅
tstest --tags "unit,fast"

// Exclude tests by pattern (not yet implemented)
tstest --exclude "**/slow/**"

// Run only failed tests from last run (not yet implemented)
tstest --failed

// Run tests modified in git (not yet implemented)
tstest --changed
```

## 6. Reporting and Analytics

### 6.1 Custom Reporters
- Plugin architecture for custom reporters
- Built-in reporters: JSON, JUnit, HTML, Markdown
- Real-time streaming reporters
- Aggregated test metrics and trends

### 6.2 Coverage Integration
- Built-in code coverage collection
- Coverage thresholds and enforcement
- Coverage trending over time
- Integration with CI/CD pipelines

### 6.3 Test Analytics Dashboard
- Web-based dashboard for test results
- Historical test performance data
- Flaky test detection
- Test impact analysis

## 7. Developer Experience

### 7.1 Better Error Messages
- Clear, actionable error messages
- Suggestions for common issues
- Links to documentation
- Code examples in error output

### 7.2 Interactive Mode (Needs Detailed Specification)
- REPL for exploring test failures
  - Need to define: How to enter interactive mode? When tests fail?
  - What commands/features should be available in the REPL?
- Debugging integration  
  - Node.js inspector protocol integration?
  - Breakpoint support?
- Step-through test execution
  - Pause between tests? 
  - Step into/over/out functionality?
- Interactive test data manipulation
  - Modify test inputs on the fly?
  - Inspect intermediate values?

### 7.3 ~~VS Code Extension~~ (Scratched)
- ~~Test explorer integration~~
- ~~Inline test results~~
- ~~CodeLens for running individual tests~~
- ~~Debugging support~~

## Implementation Phases

### Phase 1: Core Enhancements (Priority: High) ✅
1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry) ✅
2. Add basic test grouping with describe() ✅
3. Improve error reporting between tapbundle and tstest ✅

### Phase 2: Advanced Features (Priority: Medium)
1. Implement nested test suites ✅ (basic describe support)
2. Add snapshot testing ✅
3. Create test fixture system ✅
4. Implement parallel test execution ✅

### Phase 3: Developer Experience (Priority: Medium)
1. Add watch mode
2. Implement custom reporters
3. ~~Create VS Code extension~~ (Scratched)
4. Add interactive debugging (Needs detailed spec first)

### Phase 4: Analytics and Performance (Priority: Low)
1. Build test analytics dashboard
2. Add performance benchmarking
3. Implement coverage integration
4. Create trend analysis tools

## Technical Considerations

### API Design Principles
- Maintain backward compatibility
- Progressive enhancement approach
- Opt-in features to avoid breaking changes
- Clear migration paths for new features

### Performance Goals
- Minimal overhead for test execution
- Efficient parallel execution
- Fast test discovery
- Optimized browser test bundling

### Integration Points
- Clean interfaces between tstest and tapbundle
- Extensible plugin architecture
- Standard test result format
- Compatible with existing CI/CD tools