feat(package): Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.
This commit is contained in:
parent
cc388f1408
commit
b28e2eace3
@ -1,5 +1,12 @@
|
||||
# Changelog
|
||||
|
||||
## 2025-05-15 - 1.6.0 - feat(package)
|
||||
Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.
|
||||
|
||||
- Replaced 'main' and 'typings' in package.json with explicit exports for improved module resolution.
|
||||
- Added .claude/settings.local.json to configure permissions for bash commands and web fetches.
|
||||
- Updated readme.plan.md with a comprehensive roadmap covering enhanced error reporting, rich test metadata, nested test suites, and advanced test features.
|
||||
|
||||
## 2025-05-15 - 1.5.0 - feat(cli)
|
||||
Improve test runner configuration: update test scripts, reorganize test directories, update dependencies and add local settings for command permissions.
|
||||
|
||||
|
@ -3,8 +3,11 @@
|
||||
"version": "1.5.0",
|
||||
"private": false,
|
||||
"description": "a test utility to run tests that match test/**/*.ts",
|
||||
"main": "dist_ts/index.js",
|
||||
"typings": "dist_ts/index.d.ts",
|
||||
"exports": {
|
||||
".": "./dist_ts/index.js",
|
||||
"./tapbundle": "./dist_ts_tapbundle/index.js",
|
||||
"./tapbundle_node": "./dist_ts_tapbundle_node/index.js"
|
||||
},
|
||||
"type": "module",
|
||||
"author": "Lossless GmbH",
|
||||
"license": "MIT",
|
||||
|
275
readme.plan.md
275
readme.plan.md
@ -1,41 +1,256 @@
|
||||
# Plan for showing logs for failed tests
|
||||
# Improvement Plan for tstest and tapbundle
|
||||
|
||||
!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!
|
||||
|
||||
## Goal
|
||||
When a test fails, we want to display all the console logs from that failed test in the terminal, even without the --verbose flag. This makes debugging failed tests much easier.
|
||||
## 1. Enhanced Communication Between tapbundle and tstest
|
||||
|
||||
## Current Behavior
|
||||
- Default mode: Only shows test results, no console logs
|
||||
- Verbose mode: Shows all console logs from all tests
|
||||
- When a test fails: Only shows the error message
|
||||
### 1.1 Real-time Test Progress API
|
||||
- Create a bidirectional communication channel between tapbundle and tstest
|
||||
- Emit events for test lifecycle stages (start, progress, completion)
|
||||
- Allow tstest to subscribe to tapbundle events for better progress reporting
|
||||
- Implement a standardized message format for test metadata
|
||||
|
||||
## Desired Behavior
|
||||
- Default mode: Shows test results, and IF a test fails, shows all console logs from that failed test
|
||||
- Verbose mode: Shows all console logs from all tests (unchanged)
|
||||
- When a test fails: Shows all console logs from that test plus the error
|
||||
### 1.2 Rich Error Reporting
|
||||
- Pass structured error objects from tapbundle to tstest
|
||||
- Include stack traces, code snippets, and contextual information
|
||||
- Support for error categorization (assertion failures, timeouts, uncaught exceptions)
|
||||
- Visual diff output for failed assertions
|
||||
|
||||
## Implementation Plan
|
||||
## 2. Enhanced toolsArg Functionality
|
||||
|
||||
### 1. Update TapParser
|
||||
- Store console logs for each test temporarily
|
||||
- When a test fails, mark that its logs should be shown
|
||||
### 2.1 Test Flow Control
|
||||
```typescript
|
||||
tap.test('conditional test', async (toolsArg) => {
|
||||
const result = await someOperation();
|
||||
|
||||
// Skip the rest of the test
|
||||
if (!result) {
|
||||
return toolsArg.skip('Precondition not met');
|
||||
}
|
||||
|
||||
// Conditional skipping
|
||||
await toolsArg.skipIf(condition, 'Reason for skipping');
|
||||
|
||||
// Mark test as todo
|
||||
await toolsArg.todo('Not implemented yet');
|
||||
});
|
||||
```
|
||||
|
||||
### 2. Update TsTestLogger
|
||||
- Add a new method to handle failed test logs
|
||||
- Modify testConsoleOutput to buffer logs when not in verbose mode
|
||||
- When a test fails, flush the buffered logs for that test
|
||||
### 2.2 Test Metadata and Configuration
|
||||
```typescript
|
||||
tap.test('configurable test', async (toolsArg) => {
|
||||
// Set custom timeout
|
||||
toolsArg.timeout(5000);
|
||||
|
||||
// Retry on failure
|
||||
toolsArg.retry(3);
|
||||
|
||||
// Add tags for filtering
|
||||
toolsArg.tags(['slow', 'integration']);
|
||||
|
||||
// Set test priority
|
||||
toolsArg.priority('high');
|
||||
});
|
||||
```
|
||||
|
||||
### 3. Update test result handling
|
||||
- When a test fails, trigger display of all buffered logs for that test
|
||||
- Clear logs after each test completes successfully
|
||||
### 2.3 Test Data and Context Sharing
|
||||
```typescript
|
||||
tap.test('data-driven test', async (toolsArg) => {
|
||||
// Access shared context
|
||||
const sharedData = toolsArg.context.get('sharedData');
|
||||
|
||||
// Set data for other tests
|
||||
toolsArg.context.set('resultData', computedValue);
|
||||
|
||||
// Parameterized test data
|
||||
const testData = toolsArg.data<TestInput>();
|
||||
expect(processData(testData)).toEqual(expected);
|
||||
});
|
||||
```
|
||||
|
||||
## Code Changes Needed
|
||||
1. Add log buffering to TapParser
|
||||
2. Update TsTestLogger to handle failed test logs
|
||||
3. Modify test result processing to show logs on failure
|
||||
## 3. Nested Tests and Test Suites
|
||||
|
||||
## Files to Modify
|
||||
- `ts/tstest.classes.tap.parser.ts` - Add log buffering
|
||||
- `ts/tstest.logging.ts` - Add failed test log handling
|
||||
- `ts/tstest.classes.tap.testresult.ts` - May need to store logs
|
||||
### 3.1 Test Grouping with describe()
|
||||
```typescript
|
||||
tap.describe('User Authentication', () => {
|
||||
tap.beforeEach(async (toolsArg) => {
|
||||
// Setup for each test in this suite
|
||||
await toolsArg.context.set('db', await createTestDatabase());
|
||||
});
|
||||
|
||||
tap.afterEach(async (toolsArg) => {
|
||||
// Cleanup after each test
|
||||
await toolsArg.context.get('db').cleanup();
|
||||
});
|
||||
|
||||
tap.test('should login with valid credentials', async (toolsArg) => {
|
||||
// Test implementation
|
||||
});
|
||||
|
||||
tap.describe('Password Reset', () => {
|
||||
tap.test('should send reset email', async (toolsArg) => {
|
||||
// Nested test
|
||||
});
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### 3.2 Hierarchical Test Organization
|
||||
- Support for multiple levels of nesting
|
||||
- Inherited context and configuration from parent suites
|
||||
- Aggregated reporting for test suites
|
||||
- Suite-level lifecycle hooks
|
||||
|
||||
## 4. Advanced Test Features
|
||||
|
||||
### 4.1 Snapshot Testing
|
||||
```typescript
|
||||
tap.test('component render', async (toolsArg) => {
|
||||
const output = renderComponent(props);
|
||||
|
||||
// Compare with stored snapshot
|
||||
await toolsArg.matchSnapshot(output, 'component-output');
|
||||
});
|
||||
```
|
||||
|
||||
### 4.2 Performance Benchmarking
|
||||
```typescript
|
||||
tap.test('performance test', async (toolsArg) => {
|
||||
const benchmark = toolsArg.benchmark();
|
||||
|
||||
// Run operation
|
||||
await expensiveOperation();
|
||||
|
||||
// Assert performance constraints
|
||||
benchmark.expect({
|
||||
maxDuration: 1000,
|
||||
maxMemory: '100MB'
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
### 4.3 Test Fixtures and Factories
|
||||
```typescript
|
||||
tap.test('with fixtures', async (toolsArg) => {
|
||||
// Create test fixtures
|
||||
const user = await toolsArg.fixture('user', { name: 'Test User' });
|
||||
const post = await toolsArg.fixture('post', { author: user });
|
||||
|
||||
// Use factory functions
|
||||
const users = await toolsArg.factory('user').createMany(5);
|
||||
});
|
||||
```
|
||||
|
||||
## 5. Test Execution Improvements
|
||||
|
||||
### 5.1 Parallel Test Execution
|
||||
- Run independent tests concurrently
|
||||
- Configurable concurrency limits
|
||||
- Resource pooling for shared resources
|
||||
- Proper isolation between parallel tests
|
||||
|
||||
### 5.2 Watch Mode
|
||||
- Automatically re-run tests on file changes
|
||||
- Intelligent test selection based on changed files
|
||||
- Fast feedback loop for development
|
||||
- Integration with IDE/editor plugins
|
||||
|
||||
### 5.3 Advanced Test Filtering
|
||||
```typescript
|
||||
// Run tests by tags
|
||||
tstest --tags "unit,fast"
|
||||
|
||||
// Exclude tests by pattern
|
||||
tstest --exclude "**/slow/**"
|
||||
|
||||
// Run only failed tests from last run
|
||||
tstest --failed
|
||||
|
||||
// Run tests modified in git
|
||||
tstest --changed
|
||||
```
|
||||
|
||||
## 6. Reporting and Analytics
|
||||
|
||||
### 6.1 Custom Reporters
|
||||
- Plugin architecture for custom reporters
|
||||
- Built-in reporters: JSON, JUnit, HTML, Markdown
|
||||
- Real-time streaming reporters
|
||||
- Aggregated test metrics and trends
|
||||
|
||||
### 6.2 Coverage Integration
|
||||
- Built-in code coverage collection
|
||||
- Coverage thresholds and enforcement
|
||||
- Coverage trending over time
|
||||
- Integration with CI/CD pipelines
|
||||
|
||||
### 6.3 Test Analytics Dashboard
|
||||
- Web-based dashboard for test results
|
||||
- Historical test performance data
|
||||
- Flaky test detection
|
||||
- Test impact analysis
|
||||
|
||||
## 7. Developer Experience
|
||||
|
||||
### 7.1 Better Error Messages
|
||||
- Clear, actionable error messages
|
||||
- Suggestions for common issues
|
||||
- Links to documentation
|
||||
- Code examples in error output
|
||||
|
||||
### 7.2 Interactive Mode
|
||||
- REPL for exploring test failures
|
||||
- Debugging integration
|
||||
- Step-through test execution
|
||||
- Interactive test data manipulation
|
||||
|
||||
### 7.3 VS Code Extension
|
||||
- Test explorer integration
|
||||
- Inline test results
|
||||
- CodeLens for running individual tests
|
||||
- Debugging support
|
||||
|
||||
## Implementation Phases
|
||||
|
||||
### Phase 1: Core Enhancements (Priority: High)
|
||||
1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry)
|
||||
2. Add basic test grouping with describe()
|
||||
3. Improve error reporting between tapbundle and tstest
|
||||
|
||||
### Phase 2: Advanced Features (Priority: Medium)
|
||||
1. Implement nested test suites
|
||||
2. Add snapshot testing
|
||||
3. Create test fixture system
|
||||
4. Implement parallel test execution
|
||||
|
||||
### Phase 3: Developer Experience (Priority: Medium)
|
||||
1. Add watch mode
|
||||
2. Implement custom reporters
|
||||
3. Create VS Code extension
|
||||
4. Add interactive debugging
|
||||
|
||||
### Phase 4: Analytics and Performance (Priority: Low)
|
||||
1. Build test analytics dashboard
|
||||
2. Add performance benchmarking
|
||||
3. Implement coverage integration
|
||||
4. Create trend analysis tools
|
||||
|
||||
## Technical Considerations
|
||||
|
||||
### API Design Principles
|
||||
- Maintain backward compatibility
|
||||
- Progressive enhancement approach
|
||||
- Opt-in features to avoid breaking changes
|
||||
- Clear migration paths for new features
|
||||
|
||||
### Performance Goals
|
||||
- Minimal overhead for test execution
|
||||
- Efficient parallel execution
|
||||
- Fast test discovery
|
||||
- Optimized browser test bundling
|
||||
|
||||
### Integration Points
|
||||
- Clean interfaces between tstest and tapbundle
|
||||
- Extensible plugin architecture
|
||||
- Standard test result format
|
||||
- Compatible with existing CI/CD tools
|
@ -3,6 +3,6 @@
|
||||
*/
|
||||
export const commitinfo = {
|
||||
name: '@git.zone/tstest',
|
||||
version: '1.5.0',
|
||||
version: '1.6.0',
|
||||
description: 'a test utility to run tests that match test/**/*.ts'
|
||||
}
|
||||
|
Loading…
x
Reference in New Issue
Block a user