tstest/readme.protocol.md

Improved Internal Protocol Design

Current Issues with TAP Protocol

1. Delimiter Conflict: Using # for metadata conflicts with test descriptions containing #
2. Ambiguous Parsing: No clear boundary between test name and metadata
3. Limited Extensibility: Adding new metadata requires regex changes
4. Mixed Concerns: Protocol data mixed with human-readable output

Proposed Internal Protocol v2

Design Principles

1. Clear Separation: Protocol data must be unambiguously separated from user content
2. Extensibility: Easy to add new metadata without breaking parsers
3. Backwards Compatible: Can coexist with standard TAP for gradual migration
4. Machine Readable: Structured format for reliable parsing
5. Human Friendly: Still readable in raw form

Protocol Options

Option 1: Special Delimiters

ok 1 - test description ::TSTEST:: {"time":123,"retry":0}
not ok 2 - another test ::TSTEST:: {"time":45,"error":"timeout"}
ok 3 - skipped test ::TSTEST:: {"time":0,"skip":"not ready"}

Pros:

• Simple to implement
• Backwards compatible with TAP parsers (they ignore the suffix)
• Easy to parse with split()

Cons:

• Still could conflict if test name contains ::TSTEST::
• Not standard TAP

Option 2: Separate Metadata Lines

ok 1 - test description
::METADATA:: {"test":1,"time":123,"retry":0}
not ok 2 - another test  
::METADATA:: {"test":2,"time":45,"error":"timeout"}

Pros:

• Complete separation of concerns
• No chance of conflicts
• Can include arbitrary metadata

Cons:

• Requires correlation between lines
• More complex parsing

Option 3: YAML Blocks (TAP 13 Compatible)

ok 1 - test description
  ---
  time: 123
  retry: 0
  ...
not ok 2 - another test
  ---
  time: 45
  error: timeout
  stack: |
    Error: timeout
      at Test.run (test.js:10:5)
  ...

Pros:

• Standard TAP 13 feature
• Structured data format
• Human readable
• Extensible

Cons:

• More verbose
• YAML parsing overhead

Option 4: Binary Protocol Markers (Recommended)

ok 1 - test description
␛[TSTEST:eyJ0aW1lIjoxMjMsInJldHJ5IjowfQ==]␛
not ok 2 - another test
␛[TSTEST:eyJ0aW1lIjo0NSwiZXJyb3IiOiJ0aW1lb3V0In0=]␛

Using ASCII escape character (␛ = \x1B) with base64 encoded JSON.

Pros:

• Zero chance of accidental conflicts
• Compact
• Fast to parse
• Invisible in most terminals

Cons:

• Not human readable in raw form
• Requires base64 encoding/decoding

Recommended Implementation: Hybrid Approach

Use multiple strategies based on context:

1. For timing and basic metadata: Use structured delimiters

   ok 1 - test name ⟦time:123,retry:0⟧
   

2. For complex data (errors, snapshots): Use separate protocol lines

   ok 1 - test failed
   ⟦TSTEST:ERROR⟧
   {"message":"Assertion failed","stack":"...","diff":"..."}
   ⟦/TSTEST:ERROR⟧
   

3. For human-readable output: Keep standard TAP comments

   # Test suite: User Authentication
   ok 1 - should login
   

Implementation Plan

Phase 1: Parser Enhancement

1. Add new protocol parser alongside existing TAP parser
2. Support both old and new formats during transition
3. Add protocol version negotiation

Phase 2: Metadata Structure

interface TestMetadata {
  // Timing
  time: number;           // milliseconds
  startTime?: number;     // Unix timestamp
  endTime?: number;       // Unix timestamp
  
  // Status
  skip?: string;          // skip reason
  todo?: string;          // todo reason
  retry?: number;         // retry attempt
  maxRetries?: number;    // max retries allowed
  
  // Error details
  error?: {
    message: string;
    stack?: string;
    diff?: string;
    actual?: any;
    expected?: any;
  };
  
  // Test context
  file?: string;          // source file
  line?: number;          // line number
  column?: number;        // column number
  
  // Custom data
  tags?: string[];        // test tags
  custom?: Record<string, any>;
}

Phase 3: Protocol Messages

Success Message

ok 1 - user authentication works
⟦TSTEST:META:{"time":123,"tags":["auth","unit"]}⟧

Failure Message

not ok 2 - login fails with invalid password
⟦TSTEST:META:{"time":45,"retry":1,"maxRetries":3}⟧
⟦TSTEST:ERROR⟧
{
  "message": "Expected 401 but got 500",
  "stack": "Error: Expected 401 but got 500\n    at Test.run (auth.test.ts:25:10)",
  "actual": 500,
  "expected": 401
}
⟦/TSTEST:ERROR⟧

Skip Message

ok 3 - database integration test ⟦TSTEST:SKIP:No database connection⟧

Snapshot Communication

⟦TSTEST:SNAPSHOT:user-profile⟧
{
  "name": "John Doe",
  "email": "john@example.com",
  "roles": ["user", "admin"]
}
⟦/TSTEST:SNAPSHOT⟧

Migration Strategy

1. Version Detection: First line indicates protocol version

   ⟦TSTEST:PROTOCOL:2.0⟧
   TAP version 13
   

2. Gradual Rollout:

   • v1.10: Add protocol v2 parser, keep v1 generator
   • v1.11: Generate v2 by default, v1 with --legacy flag
   • v2.0: Remove v1 support

3. Feature Flags:

   tap.settings({
     protocol: 'v2',        // or 'v1', 'auto'
     protocolFeatures: {
       structuredErrors: true,
       enhancedTiming: true,
       binaryMarkers: false
     }
   });
   

Benefits of New Protocol

1. Reliability: No more regex fragility or description conflicts
2. Performance: Faster parsing with clear boundaries
3. Extensibility: Easy to add new metadata fields
4. Debugging: Rich error information with stack traces and diffs
5. Integration: Better IDE and CI/CD tool integration
6. Forward Compatible: Room for future enhancements

Example Parser Implementation

class ProtocolV2Parser {
  private readonly MARKER_START = '⟦TSTEST:';
  private readonly MARKER_END = '⟧';
  
  parseMetadata(line: string): TestMetadata | null {
    const start = line.lastIndexOf(this.MARKER_START);
    if (start === -1) return null;
    
    const end = line.indexOf(this.MARKER_END, start);
    if (end === -1) return null;
    
    const content = line.substring(start + this.MARKER_START.length, end);
    const [type, data] = content.split(':', 2);
    
    switch (type) {
      case 'META':
        return JSON.parse(data);
      case 'SKIP':
        return { skip: data };
      case 'TODO':
        return { todo: data };
      default:
        return null;
    }
  }
  
  parseTestLine(line: string): ParsedTest {
    // First extract any metadata
    const metadata = this.parseMetadata(line);
    
    // Then parse the TAP part (without metadata)
    const cleanLine = this.removeMetadata(line);
    const tapResult = this.parseTAP(cleanLine);
    
    return { ...tapResult, metadata };
  }
}

Next Steps

1. Implement proof of concept with basic metadata support
2. Test with real-world test suites for edge cases
3. Benchmark parsing performance
4. Get feedback from users
5. Finalize protocol specification
6. Implement in both tapbundle and tstest