git.zone/tstest
3
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(package): Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.

Browse Source
This commit is contained in:
Philipp Kunz
2025-05-15 20:48:26 +00:00
parent cc388f1408
commit b28e2eace3
4 changed files with 258 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

275
readme.plan.md
View File

@@ -1,41 +1,256 @@
# Plan for showing logs for failed tests
# Improvement Plan for tstest and tapbundle
!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!
## Goal
When a test fails, we want to display all the console logs from that failed test in the terminal, even without the --verbose flag. This makes debugging failed tests much easier.
## 1. Enhanced Communication Between tapbundle and tstest
## Current Behavior
- Default mode: Only shows test results, no console logs
- Verbose mode: Shows all console logs from all tests
- When a test fails: Only shows the error message
### 1.1 Real-time Test Progress API
- Create a bidirectional communication channel between tapbundle and tstest
- Emit events for test lifecycle stages (start, progress, completion)
- Allow tstest to subscribe to tapbundle events for better progress reporting
- Implement a standardized message format for test metadata
## Desired Behavior
- Default mode: Shows test results, and IF a test fails, shows all console logs from that failed test
- Verbose mode: Shows all console logs from all tests (unchanged)
- When a test fails: Shows all console logs from that test plus the error
### 1.2 Rich Error Reporting
- Pass structured error objects from tapbundle to tstest
- Include stack traces, code snippets, and contextual information
- Support for error categorization (assertion failures, timeouts, uncaught exceptions)
- Visual diff output for failed assertions
## Implementation Plan
## 2. Enhanced toolsArg Functionality
### 1. Update TapParser
- Store console logs for each test temporarily
- When a test fails, mark that its logs should be shown
### 2.1 Test Flow Control
```typescript
tap.test('conditional test', async (toolsArg) => {
const result = await someOperation();
// Skip the rest of the test
if (!result) {
return toolsArg.skip('Precondition not met');
}
// Conditional skipping
await toolsArg.skipIf(condition, 'Reason for skipping');
// Mark test as todo
await toolsArg.todo('Not implemented yet');
});
```
### 2. Update TsTestLogger
- Add a new method to handle failed test logs
- Modify testConsoleOutput to buffer logs when not in verbose mode
- When a test fails, flush the buffered logs for that test
### 2.2 Test Metadata and Configuration
```typescript
tap.test('configurable test', async (toolsArg) => {
// Set custom timeout
toolsArg.timeout(5000);
// Retry on failure
toolsArg.retry(3);
// Add tags for filtering
toolsArg.tags(['slow', 'integration']);
// Set test priority
toolsArg.priority('high');
});
```
### 3. Update test result handling
- When a test fails, trigger display of all buffered logs for that test
- Clear logs after each test completes successfully
### 2.3 Test Data and Context Sharing
```typescript
tap.test('data-driven test', async (toolsArg) => {
// Access shared context
const sharedData = toolsArg.context.get('sharedData');
// Set data for other tests
toolsArg.context.set('resultData', computedValue);
// Parameterized test data
const testData = toolsArg.data<TestInput>();
expect(processData(testData)).toEqual(expected);
});
```
## Code Changes Needed
1. Add log buffering to TapParser
2. Update TsTestLogger to handle failed test logs
3. Modify test result processing to show logs on failure
## 3. Nested Tests and Test Suites
## Files to Modify
- `ts/tstest.classes.tap.parser.ts` - Add log buffering
- `ts/tstest.logging.ts` - Add failed test log handling
- `ts/tstest.classes.tap.testresult.ts` - May need to store logs
### 3.1 Test Grouping with describe()
```typescript
tap.describe('User Authentication', () => {
tap.beforeEach(async (toolsArg) => {
// Setup for each test in this suite
await toolsArg.context.set('db', await createTestDatabase());
});
tap.afterEach(async (toolsArg) => {
// Cleanup after each test
await toolsArg.context.get('db').cleanup();
});
tap.test('should login with valid credentials', async (toolsArg) => {
// Test implementation
});
tap.describe('Password Reset', () => {
tap.test('should send reset email', async (toolsArg) => {
// Nested test
});
});
});
```
### 3.2 Hierarchical Test Organization
- Support for multiple levels of nesting
- Inherited context and configuration from parent suites
- Aggregated reporting for test suites
- Suite-level lifecycle hooks
## 4. Advanced Test Features
### 4.1 Snapshot Testing
```typescript
tap.test('component render', async (toolsArg) => {
const output = renderComponent(props);
// Compare with stored snapshot
await toolsArg.matchSnapshot(output, 'component-output');
});
```
### 4.2 Performance Benchmarking
```typescript
tap.test('performance test', async (toolsArg) => {
const benchmark = toolsArg.benchmark();
// Run operation
await expensiveOperation();
// Assert performance constraints
benchmark.expect({
maxDuration: 1000,
maxMemory: '100MB'
});
});
```
### 4.3 Test Fixtures and Factories
```typescript
tap.test('with fixtures', async (toolsArg) => {
// Create test fixtures
const user = await toolsArg.fixture('user', { name: 'Test User' });
const post = await toolsArg.fixture('post', { author: user });
// Use factory functions
const users = await toolsArg.factory('user').createMany(5);
});
```
## 5. Test Execution Improvements
### 5.1 Parallel Test Execution
- Run independent tests concurrently
- Configurable concurrency limits
- Resource pooling for shared resources
- Proper isolation between parallel tests
### 5.2 Watch Mode
- Automatically re-run tests on file changes
- Intelligent test selection based on changed files
- Fast feedback loop for development
- Integration with IDE/editor plugins
### 5.3 Advanced Test Filtering
```typescript
// Run tests by tags
tstest --tags "unit,fast"
// Exclude tests by pattern
tstest --exclude "**/slow/**"
// Run only failed tests from last run
tstest --failed
// Run tests modified in git
tstest --changed
```
## 6. Reporting and Analytics
### 6.1 Custom Reporters
- Plugin architecture for custom reporters
- Built-in reporters: JSON, JUnit, HTML, Markdown
- Real-time streaming reporters
- Aggregated test metrics and trends
### 6.2 Coverage Integration
- Built-in code coverage collection
- Coverage thresholds and enforcement
- Coverage trending over time
- Integration with CI/CD pipelines
### 6.3 Test Analytics Dashboard
- Web-based dashboard for test results
- Historical test performance data
- Flaky test detection
- Test impact analysis
## 7. Developer Experience
### 7.1 Better Error Messages
- Clear, actionable error messages
- Suggestions for common issues
- Links to documentation
- Code examples in error output
### 7.2 Interactive Mode
- REPL for exploring test failures
- Debugging integration
- Step-through test execution
- Interactive test data manipulation
### 7.3 VS Code Extension
- Test explorer integration
- Inline test results
- CodeLens for running individual tests
- Debugging support
## Implementation Phases
### Phase 1: Core Enhancements (Priority: High)
1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry)
2. Add basic test grouping with describe()
3. Improve error reporting between tapbundle and tstest
### Phase 2: Advanced Features (Priority: Medium)
1. Implement nested test suites
2. Add snapshot testing
3. Create test fixture system
4. Implement parallel test execution
### Phase 3: Developer Experience (Priority: Medium)
1. Add watch mode
2. Implement custom reporters
3. Create VS Code extension
4. Add interactive debugging
### Phase 4: Analytics and Performance (Priority: Low)
1. Build test analytics dashboard
2. Add performance benchmarking
3. Implement coverage integration
4. Create trend analysis tools
## Technical Considerations
### API Design Principles
- Maintain backward compatibility
- Progressive enhancement approach
- Opt-in features to avoid breaking changes
- Clear migration paths for new features
### Performance Goals
- Minimal overhead for test execution
- Efficient parallel execution
- Fast test discovery
- Optimized browser test bundling
### Integration Points
- Clean interfaces between tstest and tapbundle
- Extensible plugin architecture
- Standard test result format
- Compatible with existing CI/CD tools