tstest/readme.protocol.md

# Improved Internal Protocol Design

## Current Issues with TAP Protocol

1. **Delimiter Conflict**: Using `#` for metadata conflicts with test descriptions containing `#`
2. **Ambiguous Parsing**: No clear boundary between test name and metadata
3. **Limited Extensibility**: Adding new metadata requires regex changes
4. **Mixed Concerns**: Protocol data mixed with human-readable output

## Proposed Internal Protocol v2

### Design Principles

1. **Clear Separation**: Protocol data must be unambiguously separated from user content
2. **Extensibility**: Easy to add new metadata without breaking parsers
3. **Backwards Compatible**: Can coexist with standard TAP for gradual migration
4. **Machine Readable**: Structured format for reliable parsing
5. **Human Friendly**: Still readable in raw form

### Protocol Options

#### Option 1: Special Delimiters
```
ok 1 - test description ::TSTEST:: {"time":123,"retry":0}
not ok 2 - another test ::TSTEST:: {"time":45,"error":"timeout"}
ok 3 - skipped test ::TSTEST:: {"time":0,"skip":"not ready"}
```

**Pros**: 
- Simple to implement
- Backwards compatible with TAP parsers (they ignore the suffix)
- Easy to parse with split()

**Cons**: 
- Still could conflict if test name contains `::TSTEST::`
- Not standard TAP

#### Option 2: Separate Metadata Lines
```
ok 1 - test description
::METADATA:: {"test":1,"time":123,"retry":0}
not ok 2 - another test  
::METADATA:: {"test":2,"time":45,"error":"timeout"}
```

**Pros**:
- Complete separation of concerns
- No chance of conflicts
- Can include arbitrary metadata

**Cons**:
- Requires correlation between lines
- More complex parsing

#### Option 3: YAML Blocks (TAP 13 Compatible)
```
ok 1 - test description
  ---
  time: 123
  retry: 0
  ...
not ok 2 - another test
  ---
  time: 45
  error: timeout
  stack: |
    Error: timeout
      at Test.run (test.js:10:5)
  ...
```

**Pros**:
- Standard TAP 13 feature
- Structured data format
- Human readable
- Extensible

**Cons**:
- More verbose
- YAML parsing overhead

#### Option 4: Binary Protocol Markers (Recommended)
```
ok 1 - test description
␛[TSTEST:eyJ0aW1lIjoxMjMsInJldHJ5IjowfQ==]␛
not ok 2 - another test
␛[TSTEST:eyJ0aW1lIjo0NSwiZXJyb3IiOiJ0aW1lb3V0In0=]␛
```

Using ASCII escape character (␛ = \x1B) with base64 encoded JSON.

**Pros**:
- Zero chance of accidental conflicts
- Compact
- Fast to parse
- Invisible in most terminals

**Cons**:
- Not human readable in raw form
- Requires base64 encoding/decoding

### Recommended Implementation: Hybrid Approach

Use multiple strategies based on context:

1. **For timing and basic metadata**: Use structured delimiters
   ```
   ok 1 - test name ⟦time:123,retry:0⟧
   ```

2. **For complex data (errors, snapshots)**: Use separate protocol lines
   ```
   ok 1 - test failed
   ⟦TSTEST:ERROR⟧
   {"message":"Assertion failed","stack":"...","diff":"..."}
   ⟦/TSTEST:ERROR⟧
   ```

3. **For human-readable output**: Keep standard TAP comments
   ```
   # Test suite: User Authentication
   ok 1 - should login
   ```

### Implementation Plan

#### Phase 1: Parser Enhancement
1. Add new protocol parser alongside existing TAP parser
2. Support both old and new formats during transition
3. Add protocol version negotiation

#### Phase 2: Metadata Structure
```typescript
interface TestMetadata {
  // Timing
  time: number;           // milliseconds
  startTime?: number;     // Unix timestamp
  endTime?: number;       // Unix timestamp
  
  // Status
  skip?: string;          // skip reason
  todo?: string;          // todo reason
  retry?: number;         // retry attempt
  maxRetries?: number;    // max retries allowed
  
  // Error details
  error?: {
    message: string;
    stack?: string;
    diff?: string;
    actual?: any;
    expected?: any;
  };
  
  // Test context
  file?: string;          // source file
  line?: number;          // line number
  column?: number;        // column number
  
  // Custom data
  tags?: string[];        // test tags
  custom?: Record<string, any>;
}
```

#### Phase 3: Protocol Messages

##### Success Message
```
ok 1 - user authentication works
⟦TSTEST:META:{"time":123,"tags":["auth","unit"]}⟧
```

##### Failure Message
```
not ok 2 - login fails with invalid password
⟦TSTEST:META:{"time":45,"retry":1,"maxRetries":3}⟧
⟦TSTEST:ERROR⟧
{
  "message": "Expected 401 but got 500",
  "stack": "Error: Expected 401 but got 500\n    at Test.run (auth.test.ts:25:10)",
  "actual": 500,
  "expected": 401
}
⟦/TSTEST:ERROR⟧
```

##### Skip Message
```
ok 3 - database integration test ⟦TSTEST:SKIP:No database connection⟧
```

##### Snapshot Communication
```
⟦TSTEST:SNAPSHOT:user-profile⟧
{
  "name": "John Doe",
  "email": "john@example.com",
  "roles": ["user", "admin"]
}
⟦/TSTEST:SNAPSHOT⟧
```

### Migration Strategy

1. **Version Detection**: First line indicates protocol version
   ```
   ⟦TSTEST:PROTOCOL:2.0⟧
   TAP version 13
   ```

2. **Gradual Rollout**:
   - v1.10: Add protocol v2 parser, keep v1 generator
   - v1.11: Generate v2 by default, v1 with --legacy flag  
   - v2.0: Remove v1 support

3. **Feature Flags**:
   ```typescript
   tap.settings({
     protocol: 'v2',        // or 'v1', 'auto'
     protocolFeatures: {
       structuredErrors: true,
       enhancedTiming: true,
       binaryMarkers: false
     }
   });
   ```

### Benefits of New Protocol

1. **Reliability**: No more regex fragility or description conflicts
2. **Performance**: Faster parsing with clear boundaries
3. **Extensibility**: Easy to add new metadata fields
4. **Debugging**: Rich error information with stack traces and diffs
5. **Integration**: Better IDE and CI/CD tool integration
6. **Forward Compatible**: Room for future enhancements

### Example Parser Implementation

```typescript
class ProtocolV2Parser {
  private readonly MARKER_START = '⟦TSTEST:';
  private readonly MARKER_END = '⟧';
  
  parseMetadata(line: string): TestMetadata | null {
    const start = line.lastIndexOf(this.MARKER_START);
    if (start === -1) return null;
    
    const end = line.indexOf(this.MARKER_END, start);
    if (end === -1) return null;
    
    const content = line.substring(start + this.MARKER_START.length, end);
    const [type, data] = content.split(':', 2);
    
    switch (type) {
      case 'META':
        return JSON.parse(data);
      case 'SKIP':
        return { skip: data };
      case 'TODO':
        return { todo: data };
      default:
        return null;
    }
  }
  
  parseTestLine(line: string): ParsedTest {
    // First extract any metadata
    const metadata = this.parseMetadata(line);
    
    // Then parse the TAP part (without metadata)
    const cleanLine = this.removeMetadata(line);
    const tapResult = this.parseTAP(cleanLine);
    
    return { ...tapResult, metadata };
  }
}
```

### Next Steps

1. Implement proof of concept with basic metadata support
2. Test with real-world test suites for edge cases
3. Benchmark parsing performance
4. Get feedback from users
5. Finalize protocol specification
6. Implement in both tapbundle and tstest