fix(tstest): Fix test timing display issue and update TAP protocol documentation

2025-05-23 22:27:12 +00:00
parent 02aeb8195e
commit c48887a820
9 changed files with 1041 additions and 211 deletions
--- a/readme.protocol.md
+++ b/readme.protocol.md
@ -0,0 +1,287 @@
+# Improved Internal Protocol Design
+
+## Current Issues with TAP Protocol
+
+1. **Delimiter Conflict**: Using `#` for metadata conflicts with test descriptions containing `#`
+2. **Ambiguous Parsing**: No clear boundary between test name and metadata
+3. **Limited Extensibility**: Adding new metadata requires regex changes
+4. **Mixed Concerns**: Protocol data mixed with human-readable output
+
+## Proposed Internal Protocol v2
+
+### Design Principles
+
+1. **Clear Separation**: Protocol data must be unambiguously separated from user content
+2. **Extensibility**: Easy to add new metadata without breaking parsers
+3. **Backwards Compatible**: Can coexist with standard TAP for gradual migration
+4. **Machine Readable**: Structured format for reliable parsing
+5. **Human Friendly**: Still readable in raw form
+
+### Protocol Options
+
+#### Option 1: Special Delimiters
+```
+ok 1 - test description ::TSTEST:: {"time":123,"retry":0}
+not ok 2 - another test ::TSTEST:: {"time":45,"error":"timeout"}
+ok 3 - skipped test ::TSTEST:: {"time":0,"skip":"not ready"}
+```
+
+**Pros**: 
+- Simple to implement
+- Backwards compatible with TAP parsers (they ignore the suffix)
+- Easy to parse with split()
+
+**Cons**: 
+- Still could conflict if test name contains `::TSTEST::`
+- Not standard TAP
+
+#### Option 2: Separate Metadata Lines
+```
+ok 1 - test description
+::METADATA:: {"test":1,"time":123,"retry":0}
+not ok 2 - another test  
+::METADATA:: {"test":2,"time":45,"error":"timeout"}
+```
+
+**Pros**:
+- Complete separation of concerns
+- No chance of conflicts
+- Can include arbitrary metadata
+
+**Cons**:
+- Requires correlation between lines
+- More complex parsing
+
+#### Option 3: YAML Blocks (TAP 13 Compatible)
+```
+ok 1 - test description
+  ---
+  time: 123
+  retry: 0
+  ...
+not ok 2 - another test
+  ---
+  time: 45
+  error: timeout
+  stack: |
+    Error: timeout
+      at Test.run (test.js:10:5)
+  ...
+```
+
+**Pros**:
+- Standard TAP 13 feature
+- Structured data format
+- Human readable
+- Extensible
+
+**Cons**:
+- More verbose
+- YAML parsing overhead
+
+#### Option 4: Binary Protocol Markers (Recommended)
+```
+ok 1 - test description
+␛[TSTEST:eyJ0aW1lIjoxMjMsInJldHJ5IjowfQ==]␛
+not ok 2 - another test
+␛[TSTEST:eyJ0aW1lIjo0NSwiZXJyb3IiOiJ0aW1lb3V0In0=]␛
+```
+
+Using ASCII escape character (␛ = \x1B) with base64 encoded JSON.
+
+**Pros**:
+- Zero chance of accidental conflicts
+- Compact
+- Fast to parse
+- Invisible in most terminals
+
+**Cons**:
+- Not human readable in raw form
+- Requires base64 encoding/decoding
+
+### Recommended Implementation: Hybrid Approach
+
+Use multiple strategies based on context:
+
+1. **For timing and basic metadata**: Use structured delimiters
+   ```
+   ok 1 - test name ⟦time:123,retry:0⟧
+   ```
+
+2. **For complex data (errors, snapshots)**: Use separate protocol lines
+   ```
+   ok 1 - test failed
+   ⟦TSTEST:ERROR⟧
+   {"message":"Assertion failed","stack":"...","diff":"..."}
+   ⟦/TSTEST:ERROR⟧
+   ```
+
+3. **For human-readable output**: Keep standard TAP comments
+   ```
+   # Test suite: User Authentication
+   ok 1 - should login
+   ```
+
+### Implementation Plan
+
+#### Phase 1: Parser Enhancement
+1. Add new protocol parser alongside existing TAP parser
+2. Support both old and new formats during transition
+3. Add protocol version negotiation
+
+#### Phase 2: Metadata Structure
+```typescript
+interface TestMetadata {
+  // Timing
+  time: number;           // milliseconds
+  startTime?: number;     // Unix timestamp
+  endTime?: number;       // Unix timestamp
+  
+  // Status
+  skip?: string;          // skip reason
+  todo?: string;          // todo reason
+  retry?: number;         // retry attempt
+  maxRetries?: number;    // max retries allowed
+  
+  // Error details
+  error?: {
+    message: string;
+    stack?: string;
+    diff?: string;
+    actual?: any;
+    expected?: any;
+  };
+  
+  // Test context
+  file?: string;          // source file
+  line?: number;          // line number
+  column?: number;        // column number
+  
+  // Custom data
+  tags?: string[];        // test tags
+  custom?: Record<string, any>;
+}
+```
+
+#### Phase 3: Protocol Messages
+
+##### Success Message
+```
+ok 1 - user authentication works
+⟦TSTEST:META:{"time":123,"tags":["auth","unit"]}⟧
+```
+
+##### Failure Message
+```
+not ok 2 - login fails with invalid password
+⟦TSTEST:META:{"time":45,"retry":1,"maxRetries":3}⟧
+⟦TSTEST:ERROR⟧
+{
+  "message": "Expected 401 but got 500",
+  "stack": "Error: Expected 401 but got 500\n    at Test.run (auth.test.ts:25:10)",
+  "actual": 500,
+  "expected": 401
+}
+⟦/TSTEST:ERROR⟧
+```
+
+##### Skip Message
+```
+ok 3 - database integration test ⟦TSTEST:SKIP:No database connection⟧
+```
+
+##### Snapshot Communication
+```
+⟦TSTEST:SNAPSHOT:user-profile⟧
+{
+  "name": "John Doe",
+  "email": "john@example.com",
+  "roles": ["user", "admin"]
+}
+⟦/TSTEST:SNAPSHOT⟧
+```
+
+### Migration Strategy
+
+1. **Version Detection**: First line indicates protocol version
+   ```
+   ⟦TSTEST:PROTOCOL:2.0⟧
+   TAP version 13
+   ```
+
+2. **Gradual Rollout**:
+   - v1.10: Add protocol v2 parser, keep v1 generator
+   - v1.11: Generate v2 by default, v1 with --legacy flag  
+   - v2.0: Remove v1 support
+
+3. **Feature Flags**:
+   ```typescript
+   tap.settings({
+     protocol: 'v2',        // or 'v1', 'auto'
+     protocolFeatures: {
+       structuredErrors: true,
+       enhancedTiming: true,
+       binaryMarkers: false
+     }
+   });
+   ```
+
+### Benefits of New Protocol
+
+1. **Reliability**: No more regex fragility or description conflicts
+2. **Performance**: Faster parsing with clear boundaries
+3. **Extensibility**: Easy to add new metadata fields
+4. **Debugging**: Rich error information with stack traces and diffs
+5. **Integration**: Better IDE and CI/CD tool integration
+6. **Forward Compatible**: Room for future enhancements
+
+### Example Parser Implementation
+
+```typescript
+class ProtocolV2Parser {
+  private readonly MARKER_START = '⟦TSTEST:';
+  private readonly MARKER_END = '⟧';
+  
+  parseMetadata(line: string): TestMetadata | null {
+    const start = line.lastIndexOf(this.MARKER_START);
+    if (start === -1) return null;
+    
+    const end = line.indexOf(this.MARKER_END, start);
+    if (end === -1) return null;
+    
+    const content = line.substring(start + this.MARKER_START.length, end);
+    const [type, data] = content.split(':', 2);
+    
+    switch (type) {
+      case 'META':
+        return JSON.parse(data);
+      case 'SKIP':
+        return { skip: data };
+      case 'TODO':
+        return { todo: data };
+      default:
+        return null;
+    }
+  }
+  
+  parseTestLine(line: string): ParsedTest {
+    // First extract any metadata
+    const metadata = this.parseMetadata(line);
+    
+    // Then parse the TAP part (without metadata)
+    const cleanLine = this.removeMetadata(line);
+    const tapResult = this.parseTAP(cleanLine);
+    
+    return { ...tapResult, metadata };
+  }
+}
+```
+
+### Next Steps
+
+1. Implement proof of concept with basic metadata support
+2. Test with real-world test suites for edge cases
+3. Benchmark parsing performance
+4. Get feedback from users
+5. Finalize protocol specification
+6. Implement in both tapbundle and tstest