feat(package): Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.
This commit is contained in:
parent
cc388f1408
commit
b28e2eace3
@ -1,5 +1,12 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## 2025-05-15 - 1.6.0 - feat(package)
|
||||||
|
Revamp package exports and update permissions with an extensive improvement plan for test runner enhancements.
|
||||||
|
|
||||||
|
- Replaced 'main' and 'typings' in package.json with explicit exports for improved module resolution.
|
||||||
|
- Added .claude/settings.local.json to configure permissions for bash commands and web fetches.
|
||||||
|
- Updated readme.plan.md with a comprehensive roadmap covering enhanced error reporting, rich test metadata, nested test suites, and advanced test features.
|
||||||
|
|
||||||
## 2025-05-15 - 1.5.0 - feat(cli)
|
## 2025-05-15 - 1.5.0 - feat(cli)
|
||||||
Improve test runner configuration: update test scripts, reorganize test directories, update dependencies and add local settings for command permissions.
|
Improve test runner configuration: update test scripts, reorganize test directories, update dependencies and add local settings for command permissions.
|
||||||
|
|
||||||
|
@ -3,8 +3,11 @@
|
|||||||
"version": "1.5.0",
|
"version": "1.5.0",
|
||||||
"private": false,
|
"private": false,
|
||||||
"description": "a test utility to run tests that match test/**/*.ts",
|
"description": "a test utility to run tests that match test/**/*.ts",
|
||||||
"main": "dist_ts/index.js",
|
"exports": {
|
||||||
"typings": "dist_ts/index.d.ts",
|
".": "./dist_ts/index.js",
|
||||||
|
"./tapbundle": "./dist_ts_tapbundle/index.js",
|
||||||
|
"./tapbundle_node": "./dist_ts_tapbundle_node/index.js"
|
||||||
|
},
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"author": "Lossless GmbH",
|
"author": "Lossless GmbH",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
|
275
readme.plan.md
275
readme.plan.md
@ -1,41 +1,256 @@
|
|||||||
# Plan for showing logs for failed tests
|
# Improvement Plan for tstest and tapbundle
|
||||||
|
|
||||||
!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!
|
!! FIRST: Reread /home/philkunz/.claude/CLAUDE.md to ensure following all guidelines !!
|
||||||
|
|
||||||
## Goal
|
## 1. Enhanced Communication Between tapbundle and tstest
|
||||||
When a test fails, we want to display all the console logs from that failed test in the terminal, even without the --verbose flag. This makes debugging failed tests much easier.
|
|
||||||
|
|
||||||
## Current Behavior
|
### 1.1 Real-time Test Progress API
|
||||||
- Default mode: Only shows test results, no console logs
|
- Create a bidirectional communication channel between tapbundle and tstest
|
||||||
- Verbose mode: Shows all console logs from all tests
|
- Emit events for test lifecycle stages (start, progress, completion)
|
||||||
- When a test fails: Only shows the error message
|
- Allow tstest to subscribe to tapbundle events for better progress reporting
|
||||||
|
- Implement a standardized message format for test metadata
|
||||||
|
|
||||||
## Desired Behavior
|
### 1.2 Rich Error Reporting
|
||||||
- Default mode: Shows test results, and IF a test fails, shows all console logs from that failed test
|
- Pass structured error objects from tapbundle to tstest
|
||||||
- Verbose mode: Shows all console logs from all tests (unchanged)
|
- Include stack traces, code snippets, and contextual information
|
||||||
- When a test fails: Shows all console logs from that test plus the error
|
- Support for error categorization (assertion failures, timeouts, uncaught exceptions)
|
||||||
|
- Visual diff output for failed assertions
|
||||||
|
|
||||||
## Implementation Plan
|
## 2. Enhanced toolsArg Functionality
|
||||||
|
|
||||||
### 1. Update TapParser
|
### 2.1 Test Flow Control
|
||||||
- Store console logs for each test temporarily
|
```typescript
|
||||||
- When a test fails, mark that its logs should be shown
|
tap.test('conditional test', async (toolsArg) => {
|
||||||
|
const result = await someOperation();
|
||||||
|
|
||||||
### 2. Update TsTestLogger
|
// Skip the rest of the test
|
||||||
- Add a new method to handle failed test logs
|
if (!result) {
|
||||||
- Modify testConsoleOutput to buffer logs when not in verbose mode
|
return toolsArg.skip('Precondition not met');
|
||||||
- When a test fails, flush the buffered logs for that test
|
}
|
||||||
|
|
||||||
### 3. Update test result handling
|
// Conditional skipping
|
||||||
- When a test fails, trigger display of all buffered logs for that test
|
await toolsArg.skipIf(condition, 'Reason for skipping');
|
||||||
- Clear logs after each test completes successfully
|
|
||||||
|
|
||||||
## Code Changes Needed
|
// Mark test as todo
|
||||||
1. Add log buffering to TapParser
|
await toolsArg.todo('Not implemented yet');
|
||||||
2. Update TsTestLogger to handle failed test logs
|
});
|
||||||
3. Modify test result processing to show logs on failure
|
```
|
||||||
|
|
||||||
## Files to Modify
|
### 2.2 Test Metadata and Configuration
|
||||||
- `ts/tstest.classes.tap.parser.ts` - Add log buffering
|
```typescript
|
||||||
- `ts/tstest.logging.ts` - Add failed test log handling
|
tap.test('configurable test', async (toolsArg) => {
|
||||||
- `ts/tstest.classes.tap.testresult.ts` - May need to store logs
|
// Set custom timeout
|
||||||
|
toolsArg.timeout(5000);
|
||||||
|
|
||||||
|
// Retry on failure
|
||||||
|
toolsArg.retry(3);
|
||||||
|
|
||||||
|
// Add tags for filtering
|
||||||
|
toolsArg.tags(['slow', 'integration']);
|
||||||
|
|
||||||
|
// Set test priority
|
||||||
|
toolsArg.priority('high');
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.3 Test Data and Context Sharing
|
||||||
|
```typescript
|
||||||
|
tap.test('data-driven test', async (toolsArg) => {
|
||||||
|
// Access shared context
|
||||||
|
const sharedData = toolsArg.context.get('sharedData');
|
||||||
|
|
||||||
|
// Set data for other tests
|
||||||
|
toolsArg.context.set('resultData', computedValue);
|
||||||
|
|
||||||
|
// Parameterized test data
|
||||||
|
const testData = toolsArg.data<TestInput>();
|
||||||
|
expect(processData(testData)).toEqual(expected);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## 3. Nested Tests and Test Suites
|
||||||
|
|
||||||
|
### 3.1 Test Grouping with describe()
|
||||||
|
```typescript
|
||||||
|
tap.describe('User Authentication', () => {
|
||||||
|
tap.beforeEach(async (toolsArg) => {
|
||||||
|
// Setup for each test in this suite
|
||||||
|
await toolsArg.context.set('db', await createTestDatabase());
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.afterEach(async (toolsArg) => {
|
||||||
|
// Cleanup after each test
|
||||||
|
await toolsArg.context.get('db').cleanup();
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.test('should login with valid credentials', async (toolsArg) => {
|
||||||
|
// Test implementation
|
||||||
|
});
|
||||||
|
|
||||||
|
tap.describe('Password Reset', () => {
|
||||||
|
tap.test('should send reset email', async (toolsArg) => {
|
||||||
|
// Nested test
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3.2 Hierarchical Test Organization
|
||||||
|
- Support for multiple levels of nesting
|
||||||
|
- Inherited context and configuration from parent suites
|
||||||
|
- Aggregated reporting for test suites
|
||||||
|
- Suite-level lifecycle hooks
|
||||||
|
|
||||||
|
## 4. Advanced Test Features
|
||||||
|
|
||||||
|
### 4.1 Snapshot Testing
|
||||||
|
```typescript
|
||||||
|
tap.test('component render', async (toolsArg) => {
|
||||||
|
const output = renderComponent(props);
|
||||||
|
|
||||||
|
// Compare with stored snapshot
|
||||||
|
await toolsArg.matchSnapshot(output, 'component-output');
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.2 Performance Benchmarking
|
||||||
|
```typescript
|
||||||
|
tap.test('performance test', async (toolsArg) => {
|
||||||
|
const benchmark = toolsArg.benchmark();
|
||||||
|
|
||||||
|
// Run operation
|
||||||
|
await expensiveOperation();
|
||||||
|
|
||||||
|
// Assert performance constraints
|
||||||
|
benchmark.expect({
|
||||||
|
maxDuration: 1000,
|
||||||
|
maxMemory: '100MB'
|
||||||
|
});
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4.3 Test Fixtures and Factories
|
||||||
|
```typescript
|
||||||
|
tap.test('with fixtures', async (toolsArg) => {
|
||||||
|
// Create test fixtures
|
||||||
|
const user = await toolsArg.fixture('user', { name: 'Test User' });
|
||||||
|
const post = await toolsArg.fixture('post', { author: user });
|
||||||
|
|
||||||
|
// Use factory functions
|
||||||
|
const users = await toolsArg.factory('user').createMany(5);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5. Test Execution Improvements
|
||||||
|
|
||||||
|
### 5.1 Parallel Test Execution
|
||||||
|
- Run independent tests concurrently
|
||||||
|
- Configurable concurrency limits
|
||||||
|
- Resource pooling for shared resources
|
||||||
|
- Proper isolation between parallel tests
|
||||||
|
|
||||||
|
### 5.2 Watch Mode
|
||||||
|
- Automatically re-run tests on file changes
|
||||||
|
- Intelligent test selection based on changed files
|
||||||
|
- Fast feedback loop for development
|
||||||
|
- Integration with IDE/editor plugins
|
||||||
|
|
||||||
|
### 5.3 Advanced Test Filtering
|
||||||
|
```typescript
|
||||||
|
// Run tests by tags
|
||||||
|
tstest --tags "unit,fast"
|
||||||
|
|
||||||
|
// Exclude tests by pattern
|
||||||
|
tstest --exclude "**/slow/**"
|
||||||
|
|
||||||
|
// Run only failed tests from last run
|
||||||
|
tstest --failed
|
||||||
|
|
||||||
|
// Run tests modified in git
|
||||||
|
tstest --changed
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Reporting and Analytics
|
||||||
|
|
||||||
|
### 6.1 Custom Reporters
|
||||||
|
- Plugin architecture for custom reporters
|
||||||
|
- Built-in reporters: JSON, JUnit, HTML, Markdown
|
||||||
|
- Real-time streaming reporters
|
||||||
|
- Aggregated test metrics and trends
|
||||||
|
|
||||||
|
### 6.2 Coverage Integration
|
||||||
|
- Built-in code coverage collection
|
||||||
|
- Coverage thresholds and enforcement
|
||||||
|
- Coverage trending over time
|
||||||
|
- Integration with CI/CD pipelines
|
||||||
|
|
||||||
|
### 6.3 Test Analytics Dashboard
|
||||||
|
- Web-based dashboard for test results
|
||||||
|
- Historical test performance data
|
||||||
|
- Flaky test detection
|
||||||
|
- Test impact analysis
|
||||||
|
|
||||||
|
## 7. Developer Experience
|
||||||
|
|
||||||
|
### 7.1 Better Error Messages
|
||||||
|
- Clear, actionable error messages
|
||||||
|
- Suggestions for common issues
|
||||||
|
- Links to documentation
|
||||||
|
- Code examples in error output
|
||||||
|
|
||||||
|
### 7.2 Interactive Mode
|
||||||
|
- REPL for exploring test failures
|
||||||
|
- Debugging integration
|
||||||
|
- Step-through test execution
|
||||||
|
- Interactive test data manipulation
|
||||||
|
|
||||||
|
### 7.3 VS Code Extension
|
||||||
|
- Test explorer integration
|
||||||
|
- Inline test results
|
||||||
|
- CodeLens for running individual tests
|
||||||
|
- Debugging support
|
||||||
|
|
||||||
|
## Implementation Phases
|
||||||
|
|
||||||
|
### Phase 1: Core Enhancements (Priority: High)
|
||||||
|
1. Implement enhanced toolsArg methods (skip, skipIf, timeout, retry)
|
||||||
|
2. Add basic test grouping with describe()
|
||||||
|
3. Improve error reporting between tapbundle and tstest
|
||||||
|
|
||||||
|
### Phase 2: Advanced Features (Priority: Medium)
|
||||||
|
1. Implement nested test suites
|
||||||
|
2. Add snapshot testing
|
||||||
|
3. Create test fixture system
|
||||||
|
4. Implement parallel test execution
|
||||||
|
|
||||||
|
### Phase 3: Developer Experience (Priority: Medium)
|
||||||
|
1. Add watch mode
|
||||||
|
2. Implement custom reporters
|
||||||
|
3. Create VS Code extension
|
||||||
|
4. Add interactive debugging
|
||||||
|
|
||||||
|
### Phase 4: Analytics and Performance (Priority: Low)
|
||||||
|
1. Build test analytics dashboard
|
||||||
|
2. Add performance benchmarking
|
||||||
|
3. Implement coverage integration
|
||||||
|
4. Create trend analysis tools
|
||||||
|
|
||||||
|
## Technical Considerations
|
||||||
|
|
||||||
|
### API Design Principles
|
||||||
|
- Maintain backward compatibility
|
||||||
|
- Progressive enhancement approach
|
||||||
|
- Opt-in features to avoid breaking changes
|
||||||
|
- Clear migration paths for new features
|
||||||
|
|
||||||
|
### Performance Goals
|
||||||
|
- Minimal overhead for test execution
|
||||||
|
- Efficient parallel execution
|
||||||
|
- Fast test discovery
|
||||||
|
- Optimized browser test bundling
|
||||||
|
|
||||||
|
### Integration Points
|
||||||
|
- Clean interfaces between tstest and tapbundle
|
||||||
|
- Extensible plugin architecture
|
||||||
|
- Standard test result format
|
||||||
|
- Compatible with existing CI/CD tools
|
@ -3,6 +3,6 @@
|
|||||||
*/
|
*/
|
||||||
export const commitinfo = {
|
export const commitinfo = {
|
||||||
name: '@git.zone/tstest',
|
name: '@git.zone/tstest',
|
||||||
version: '1.5.0',
|
version: '1.6.0',
|
||||||
description: 'a test utility to run tests that match test/**/*.ts'
|
description: 'a test utility to run tests that match test/**/*.ts'
|
||||||
}
|
}
|
||||||
|
Loading…
x
Reference in New Issue
Block a user