RFC: Electric Client Conformance Test Suite

[erik-the-implementer]

RFC: Electric Client Conformance Test Suite

Summary

Introduce a language-agnostic conformance test suite that verifies behavioral parity between Electric's TypeScript and Elixir client implementations. The suite would run on CI and catch behavioral divergences before they ship to users.

Motivation

Recent experience has shown that bugs fixed in one client can go undetected in the other for extended periods. In the span of a few days, four separate issues were discovered in the Elixir client that had already been identified and fixed in the TypeScript client:

Bug	TS Fix	Elixir Fix	Gap
Stale CDN response causes infinite loop	#4015	#4027	Weeks
No fast-loop detection for repeated requests at same offset	(existed)	#4028	Unknown
Handle -next suffix accumulates on repeated 409s	(existed)	#4029	Unknown
Retry backoff uses wrong jitter strategy; missing response header validation	(existed)	#4031	Unknown

All four bugs share the same pattern: a behavior was correctly implemented in the TS client but was either missing or implemented differently in the Elixir client. A conformance test suite that exercises both clients against the same expected behaviors would have caught each of these at the time the TS fix was merged.

Design

Architecture

Inspired by the durable-streams client conformance tests, the suite uses a language-agnostic architecture where a central test runner communicates with client-specific adapters:

┌─────────────────────────────────────────────────────────────────┐
│                    Test Runner (Node.js/Vitest)                 │
│  - Defines test scenarios                                      │
│  - Manages mock Electric server lifecycle                      │
│  - Orchestrates client adapter processes                       │
│  - Compares results against expected behavior                  │
└────────────────────────┬────────────────────────────────────────┘
                         │ stdin/stdout (JSON lines)
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│              Client Adapter (language-specific)                 │
│  - Thin wrapper around the native client library               │
│  - Reads commands from stdin, writes results to stdout         │
│  - TypeScript adapter: uses @electric-sql/client directly      │
│  - Elixir adapter: uses Electric.Client via a Mix script       │
└─────────────────────────────────────────────────────────────────┘
                         │ HTTP
                         ▼
┌─────────────────────────────────────────────────────────────────┐
│              Mock Electric Server (TypeScript)                  │
│  - Simulates Electric sync service HTTP responses              │
│  - Programmable: returns specific headers, status codes,       │
│    stale responses, 409s, etc. per test scenario               │
│  - No database required                                        │
└─────────────────────────────────────────────────────────────────┘

Why stdin/stdout JSON lines?

• Language-agnostic: No need for FFI, shared libraries, or HTTP bridges between the test runner and the client under test
• Process isolation: Each test gets a fresh client process — no leaked state between tests
• Simple to implement: The adapter is a thin wrapper (typically <200 lines) around the native client
• Debuggable: JSON lines can be captured and replayed

Package location

packages/
  client-conformance-tests/
    package.json              # Node.js package (test runner)
    src/
      runner.ts               # Test orchestration
      mock-server.ts          # Programmable mock Electric server
      protocol.ts             # Command/result type definitions
      adapters/
        typescript.ts         # Built-in TS adapter (in-process, no stdin/stdout needed)
    test-cases/
      stale-cdn/              # Stale CDN response handling
      error-recovery/         # Retry, backoff, error handling
      shape-lifecycle/        # Shape creation, rotation, 409 handling
      state-machine/          # State transition conformance
      header-validation/      # Response header requirements
    adapters/
      elixir/                 # Elixir adapter (Mix script)
        mix.exs
        lib/adapter.ex

Mock Electric server

Rather than running a full Electric sync service (which requires PostgreSQL), the conformance tests use a lightweight mock server that simulates Electric's HTTP API. Each test scenario programs the mock to return specific responses:

// Example: test for stale CDN response handling
const scenario = mockServer.scenario()
  .onRequest({ path: "/v1/shape", method: "GET" })
  .respondWith({
    status: 200,
    headers: {
      "electric-handle": "expired-handle-1",  // stale/expired handle
      "electric-offset": "0_0",
    },
    body: [],
  })
  .thenOnRetry()  // client should retry with cache buster
  .respondWith({
    status: 200,
    headers: {
      "electric-handle": "valid-handle-2",
      "electric-offset": "0_0",
    },
    body: [/* messages */],
  });

Client adapter protocol

Commands sent from runner to adapter (stdin):

// Initialize adapter with server URL
{ "type": "init", "serverUrl": "http://localhost:3456", "config": { ... } }

// Subscribe to a shape
{ "type": "subscribe", "table": "items", "where": "...", "options": { "subscribe": true } }

// Wait for the client to reach a specific state
{ "type": "await_state", "state": "up-to-date", "timeoutMs": 5000 }

// Check current client state
{ "type": "get_state" }

// Disconnect/cleanup
{ "type": "shutdown" }

Results sent from adapter to runner (stdout):

// Subscription created
{ "type": "subscribed", "shapeId": "..." }

// State reached
{ "type": "state_reached", "state": "up-to-date", "metadata": { ... } }

// Error occurred
{ "type": "error", "code": "STALE_CDN_INFINITE_LOOP", "message": "..." }

// Messages received
{ "type": "messages", "count": 5, "offset": "123_4" }

// HTTP request made (for observability)
{ "type": "http_request", "url": "...", "method": "GET", "attempt": 3 }

The http_request event is key — it lets the test runner verify behaviors like:

• Cache busters are added on stale responses
• Requests don't repeat the same URL (fast-loop detection)
• Retry backoff timing follows the expected strategy

Test case categories

Based on the recent bugs and the existing SPEC.md, the initial test suite should cover:

1. Stale CDN response handling

• Client adds cache buster when server returns stale/expired handle
• Client does not enter infinite loop on stale responses
• Client respects max stale retry count
• Cache buster makes each retry URL unique

2. Fast-loop detection

• Client detects N+ requests at same offset within a time window
• First detection clears caches and resets
• Subsequent detections apply exponential backoff
• Client raises error after max consecutive detections

3. Shape rotation (409 handling)

• Client handles 409 with new handle correctly
• Client handles 409 without new handle (appends -next suffix)
• Handle -next suffix does not accumulate on repeated 409s
• Client resets offset to -1 on shape rotation

4. Retry and backoff

• Client uses full jitter strategy (random between 1 and delay)
• Max retry delay is bounded (32s)
• Backoff resets after successful request

5. Response header validation

• Client validates presence of required Electric headers (electric-handle, electric-offset, electric-schema)
• Missing headers produce clear error about proxy/CDN misconfiguration

6. State machine transitions

• Derived from SPEC.md's state transition table
• Key invariants: isUpToDate semantics, pause/resume round-trip, error/retry identity

CI integration

# .github/workflows/client_conformance_tests.yml
name: client-conformance-tests
on:
  push:
    branches: ['main']
    paths:
      - 'packages/typescript-client/**'
      - 'packages/elixir-client/**'
      - 'packages/client-conformance-tests/**'
  pull_request:
    paths:
      - 'packages/typescript-client/**'
      - 'packages/elixir-client/**'
      - 'packages/client-conformance-tests/**'

jobs:
  conformance:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        client: [typescript, elixir]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - uses: erlef/setup-beam@v1  # for Elixir adapter
      - run: pnpm install
      - run: pnpm --filter client-conformance-tests test -- --adapter ${{ matrix.client }}

Key: the workflow triggers on changes to either client package, ensuring that a fix in one client immediately runs conformance checks against the other.

Scope and phasing

Phase 1: Core infrastructure + stale CDN tests

• Mock Electric server with programmable responses
• Stdin/stdout adapter protocol
• TypeScript adapter (in-process for speed)
• Elixir adapter (Mix script)
• Test cases for stale CDN handling (the most impactful recent bug)
• CI workflow

Phase 2: Error recovery and retry behavior

• Fast-loop detection tests
• Retry backoff strategy conformance
• Response header validation tests
• Shape rotation (409) handling tests

Phase 3: State machine conformance

• Port key invariants from SPEC.md to conformance tests
• State transition coverage for critical paths
• Pause/resume behavior

Future considerations

• Additional client implementations (Rust, Python, etc.) only need to implement the adapter protocol
• Performance benchmarks (response latency, memory usage) could use the same infrastructure
• The mock server could be extended to test edge cases that are hard to reproduce with a real Electric instance

Alternatives considered

1. Integration tests against a real Electric instance

Both clients already have integration tests that run against a real Electric sync service + PostgreSQL. These are valuable but:

• Slow (require database setup)
• Don't cover edge cases like stale CDN responses or malformed headers
• Can't precisely control timing for fast-loop detection tests

The conformance suite complements, not replaces, integration tests.

2. Shared test definitions in a neutral format (e.g. YAML)

The durable-streams project uses YAML test case files. This is appealing for its simplicity but:

• Electric's client behavior is more complex (state machine with 7 states, 10 events)
• Many test scenarios require precise control over response timing and ordering
• TypeScript test code (with the mock server API) is more expressive for these scenarios

We could adopt YAML for simpler test cases in the future while keeping TypeScript for complex scenarios.

3. Port TS tests directly to Elixir

This would catch divergences but:

• Duplicates effort for every new test
• Tests can diverge just like the implementations
• No single source of truth

Open questions

1. Elixir adapter implementation: Should the Elixir adapter be a Mix escriptized binary, or an Elixir script that requires Mix to run? The former is more portable; the latter is simpler and sufficient for CI.

2. Timing-sensitive tests: Fast-loop detection depends on a 500ms sliding window. How do we make these tests reliable without being flaky? Options: mock time in the adapter, use generous tolerances, or test the detection logic separately.

3. TS adapter in-process vs out-of-process: Running the TS adapter in-process (importing the client directly) is faster and simpler. But it means the TS client isn't tested through the same stdin/stdout path as other clients. Is this acceptable?

4. Scope of "conformance": Should the suite test only behaviors that both clients must share (HTTP-level protocol conformance), or also internal behaviors like state machine transitions? The former is more practical; the latter is more thorough.

References

• durable-streams/client-conformance-tests — inspiration for the architecture
• TypeScript client SPEC.md — formal state machine specification
• Recent Elixir client fixes: fix(elixir-client): Fix stale CDN infinite loop #4027, fix(elixir-client): Add fast-loop detection #4028, fix(elixir-client): Remove handle -next suffix accumulation #4029, fix(elixir-client): Improve elixir client parity #4031
• Corresponding TypeScript client fix: fix(typescript-client): fix infinite loop on stale CDN responses #4015