Skip to content

Add co-activation quality testing for way matching #27

@aaronsb

Description

@aaronsb

Context

The way matching system uses additive triggering — multiple ways can fire on a single prompt when they share relevant vocabulary. This is by design: a prompt like "migrate the database schema" legitimately needs guidance from both delivery/migrations (schema change mechanics) and architecture/design (target architecture).

Currently the test framework (fixture + integration) measures single-expected-way accuracy. It validates that the right way fires, but doesn't measure whether the set of co-activated ways is appropriate.

What's needed

A test layer that validates co-activation quality:

  • Prompts designed to trigger 2+ related ways, with expected sets
  • Measurement of whether co-activated ways provide complementary (not redundant) context
  • Detection of unwanted co-activation (unrelated ways bleeding in)

The activation test (step 5) now tests co-activation live, but the automated harness doesn't yet support multi-expected-way fixtures.

Suggested approach

Extend test-fixtures.jsonl format to support "expected": ["way-a", "way-b"] arrays alongside the existing single-string format. Add co-activation test cases for known overlap areas: schema/design, error/debugging, security/deps.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions