diff --git a/README.md b/README.md index add15c2..a2b4889 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,7 @@ Agent Skills are modular, text-based playbooks that teach an agent how to perfor | Skill | Description | |-------|-------------| | `experiments/launchdarkly-experiment-setup` | Set up experiments with metrics, treatments, and data collection | +| `experiments/launchdarkly-experiment-hypothesis-builder` | Coach a strong, testable hypothesis and hand off a pre-resolved config to experiment setup (draft) | ### Metrics diff --git a/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml new file mode 100644 index 0000000..bfa9f71 --- /dev/null +++ b/evals/launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml @@ -0,0 +1,181 @@ +# yaml-language-server: $schema=https://promptfoo.dev/config-schema.json +# +# Evaluates launchdarkly-experiment-hypothesis-builder — an advisory skill that +# coaches a hypothesis and hands off, never writing. Assertions read the tool-call +# trajectory plus llm-rubrics; it runs read-only via mcp_tool_allowlist. +# Run: promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml +description: "End-to-end evaluation of the launchdarkly-experiment-hypothesis-builder skill" + +prompts: + - file://../../skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md + +providers: + - id: file://../providers/claude-skill-agent-sdk.js + label: claude-skill-agent-sdk + config: + skill_slug: launchdarkly-experiment-hypothesis-builder + expose_mcp_tools: true + expose_ask_question: true + force_skill_invocation: true # force-load the skill so the eval tests it, not base Claude + # Advisory / handoff-only: expose read tools only so it cannot write. + mcp_tool_allowlist: + - list-flags + - get-flag + - list-feature-flags + - list-metrics + - get-metric + - list-metric-events + - get-project + +tests: + # Golden path: coach a strong draft to a good hypothesis, no writes. + - description: "Golden path: coach a strong draft to a good hypothesis, no writes" + vars: + max_turns: 25 + mock_ask_question_answers: + - ["~5%"] + user_request: > + In project "my-app", I want to test moving our primary navigation to the + left rail. I expect it to increase signup conversion, because it reduces + cognitive load, while login success rate stays flat. + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes (handoff/advisory only)':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: javascript + value: | + const asks = (output.trajectory||[]).filter(t=>t.tool==='ask-question').length; + const pass = asks <= 2; + return { pass, score: pass?1:(asks<=3?0.5:0), reason:`ask-question calls: ${asks}` }; + metric: does_not_over_interrogate + weight: 1 + - type: llm-rubric + value: | + Evaluate the hypothesis the agent produced. Score 1.0 if all met, deduct 0.25 each: + 1. One clear if/then/because sentence naming the intervention (left-rail nav) and the primary metric (signup conversion) with a direction (increase). + 2. Single-variable — only the nav position changes between control and treatment. + 3. Falsifiable — a result could prove it wrong. + 4. It surfaces the flag and metric it would use so the experiment can be built downstream. + metric: hypothesis_quality + weight: 2 + + # Weak input: rebuild a vague goal via questions, no writes. + - description: "Fork (weak): a vague goal is rebuilt via questions, no writes" + vars: + max_turns: 20 + user_request: "I want to increase revenue. Project is my-app." + assert: + - type: javascript + value: | + const asks = (output.trajectory||[]).filter(t=>t.tool==='ask-question'); + const text = (asks.map(a=>a.arguments&&a.arguments.prompt||'').join(' ') + ' ' + String(output.response||'')).toLowerCase(); + const asked = asks.length>0 || /\?/.test(String(output.response||'')); + const onTopic = /(metric|measur|what.*change|specific change|which|area)/.test(text); + const score = (asked?0.5:0)+(onTopic?0.5:0); + return { pass: asked && onTopic, score, reason:`asks=${asks.length} onTopic=${onTopic}` }; + metric: coaches_missing_element + weight: 3 + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + Score 1.0 if all met, deduct 0.33 each: + 1. The agent treats "increase revenue" as too vague to build from (a goal, not a hypothesis). + 2. It asks for the specific change to test and a concrete measurable primary metric. + 3. It does not fabricate an intervention, metric, or magnitude the user never provided. + metric: weak_input_handled + weight: 2 + + # Cost of being wrong: resolve an existing metric without a verbatim search + # (LD search is literal substring, so a raw phrase misses "Signup completed"). + - description: "Cost of wrong: resolve the existing metric without a verbatim search, no writes" + vars: + max_turns: 25 + user_request: > + In project "my-app", test a one-click signup button. I expect it to raise + signup completion. + assert: + - type: javascript + value: | + const traj = output.trajectory || []; + const qs = traj.filter(t=>t.tool==='list-metrics').map(t=>String((t.arguments&&t.arguments.query)||'').toLowerCase()); + const searched = qs.length>0; + const verbatim = qs.some(q=>q==='signup completion'); + const pass = searched && !verbatim; + return { pass, score: pass?1:(searched?0.4:0), reason:`list-metrics queries=[${qs.join(', ')}]` }; + metric: searched_not_verbatim + weight: 2 + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + The project already has a "Signup completed" metric. Score 1.0 if both met, deduct 0.5 each: + 1. The agent surfaces the existing "Signup completed" metric rather than proposing a brand-new duplicate. + 2. It confirms the metric pick with the user rather than silently assuming it. + metric: resolution_quality + weight: 2 + + # Cost of being wrong: catch a metric/outcome mismatch (predicts engagement, + # proposes measuring revenue). + - description: "Cost of wrong: catch the metric<->outcome mismatch, no writes" + vars: + max_turns: 25 + user_request: > + In project "my-app", I want to test a new onboarding checklist. I think it + will boost engagement — let's measure it by revenue. + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 2 + - type: llm-rubric + value: | + The user predicts ENGAGEMENT will move but proposes measuring REVENUE. Score 1.0 if all met, deduct 0.33 each: + 1. The agent catches the mismatch and does not silently accept revenue as the primary metric for an engagement prediction. + 2. It aligns the primary metric to the predicted outcome (an engagement metric), or asks the user to reconcile the two. + 3. It does not invent a magnitude the user never gave. + metric: mismatch_caught + weight: 3 + + # Safety: a platform self-test (A/A) is not built into an experiment. + - description: "Safety: a platform self-test is not built into an experiment, no writes" + vars: + max_turns: 15 + user_request: "A/A test to validate bucketing. Project my-app." + assert: + - type: javascript + value: | + const tools = output.tools_called || []; + const writes = ['create-flag','create-feature-flag','update-feature-flag','create-metric','create-experiment','start-experiment-iteration'].filter(w=>tools.includes(w)); + const pass = writes.length===0; + return { pass, score: pass?1:0, reason: pass?'No writes':'Wrote: '+writes.join(', ') }; + metric: no_writes + weight: 3 + - type: llm-rubric + value: | + Score 1.0 if both met, deduct 0.5 each: + 1. The agent recognizes this as a platform/self-test (an A/A bucketing check), not a real experiment idea. + 2. It gently confirms intent rather than coaching it into a full hypothesis and configuration. + metric: nonreal_handled + weight: 2 diff --git a/evals/mocks/tool-responses.json b/evals/mocks/tool-responses.json index 1824785..ccf23b5 100644 --- a/evals/mocks/tool-responses.json +++ b/evals/mocks/tool-responses.json @@ -1,4 +1,18 @@ { + "list-metrics": { + "metrics": [ + { "key": "checkout-conversion", "name": "Checkout conversion", "kind": "custom", "measureType": "occurrence", "successCriteria": "HigherThanBaseline", "tags": ["growth"] }, + { "key": "signup-completed", "name": "Signup completed", "kind": "custom", "measureType": "occurrence", "successCriteria": "HigherThanBaseline", "tags": ["growth"] }, + { "key": "page-load-time", "name": "Page load time", "kind": "custom", "measureType": "value", "unit": "ms", "successCriteria": "LowerThanBaseline", "tags": ["performance"] }, + { "key": "error-rate", "name": "Error rate", "kind": "custom", "measureType": "occurrence", "successCriteria": "LowerThanBaseline", "tags": ["guardrail"] } + ], + "totalCount": 4, + "pageInfo": { "limit": 20, "offset": 0 } + }, + "list-metric-events": { + "events": [{ "eventKey": "{{eventKey}}", "count": 1240, "lastSeen": "2026-07-01T00:00:00Z" }], + "totalCount": 1 + }, "list-flags": { "flags": [ { diff --git a/evals/package.json b/evals/package.json index 7a1c744..6e0b949 100644 --- a/evals/package.json +++ b/evals/package.json @@ -15,6 +15,8 @@ "eval:flag-create:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-create/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:flag-command": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-command/promptfooconfig.yaml --env-file .env --no-cache -o launchdarkly-flag-command/results.json", "eval:flag-command:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-flag-command/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", + "eval:hypothesis-builder": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml --env-file .env --no-cache -o launchdarkly-experiment-hypothesis-builder/results.json", + "eval:hypothesis-builder:single": "promptfoo eval -c shared/defaults.yaml -c launchdarkly-experiment-hypothesis-builder/promptfooconfig.yaml --env-file .env --no-cache --filter-first-n 1", "eval:all": "node scripts/aggregate.js --run", "eval:aggregate": "node scripts/aggregate.js", "eval:diff": "node scripts/diff-changed-skills.js", diff --git a/evals/providers/claude-skill-agent-sdk.js b/evals/providers/claude-skill-agent-sdk.js index 90d5205..9e0f47b 100644 --- a/evals/providers/claude-skill-agent-sdk.js +++ b/evals/providers/claude-skill-agent-sdk.js @@ -23,6 +23,11 @@ * (Read/Grep/Glob/Bash/Edit/Write/...). Default false. * expose_mcp_tools - Default true. Set false for skills that should never * call LaunchDarkly MCP tools (routing/advisory skills). + * mcp_tool_allowlist - Optional array of tool names. When set, expose ONLY + * these LaunchDarkly tools — e.g. read-only lookups + * (list and get tools) for an advisory/handoff skill + * that must never write, so it physically cannot + * mutate state. Null/unset exposes all tools. * force_skill_invocation - Default false. When true, set initialPrompt to * `/` to explicitly invoke the skill via * slash command. Use for routing/advisory skills whose @@ -162,6 +167,11 @@ class ClaudeSkillAgentSdk { this.exposeMcpTools = config.expose_mcp_tools !== false; this.forceSkillInvocation = Boolean(config.force_skill_invocation); this.exposeAskQuestion = Boolean(config.expose_ask_question); + // When set, expose ONLY these LaunchDarkly tools (by name) — e.g. read-only + // tools for an advisory/handoff skill that must never write. Null = expose all. + this.mcpToolAllowlist = Array.isArray(config.mcp_tool_allowlist) + ? config.mcp_tool_allowlist + : null; const source = resolveSkillSource(this.skillSlug); if (!source) { @@ -211,8 +221,11 @@ class ClaudeSkillAgentSdk { let currentTurn = 0; const mockState = createMockState(); + const exposedToolDefs = this.mcpToolAllowlist + ? toolDefs.filter((def) => this.mcpToolAllowlist.includes(def.name)) + : toolDefs; const mcpTools = this.exposeMcpTools - ? toolDefs.map((def) => + ? exposedToolDefs.map((def) => tool( def.name, def.description, @@ -298,7 +311,7 @@ class ClaudeSkillAgentSdk { const allowedMcpToolNames = []; if (this.exposeMcpTools) { - for (const def of toolDefs) { + for (const def of exposedToolDefs) { allowedMcpToolNames.push(`mcp__launchdarkly-mocks__${def.name}`); } } diff --git a/evals/scripts/_manifest.js b/evals/scripts/_manifest.js index 5ff7a55..53e6b8b 100644 --- a/evals/scripts/_manifest.js +++ b/evals/scripts/_manifest.js @@ -49,6 +49,12 @@ const SUITES = [ skillDir: "skills/feature-flags/launchdarkly-flag-command", readme: "skills/feature-flags/launchdarkly-flag-command/README.md", }, + { + suite: "launchdarkly-experiment-hypothesis-builder", + skillKey: "experiments/launchdarkly-experiment-hypothesis-builder", + skillDir: "skills/experiments/launchdarkly-experiment-hypothesis-builder", + readme: "skills/experiments/launchdarkly-experiment-hypothesis-builder/README.md", + }, ]; /** diff --git a/evals/tools/definitions.json b/evals/tools/definitions.json index bb33eab..32dd220 100644 --- a/evals/tools/definitions.json +++ b/evals/tools/definitions.json @@ -509,5 +509,31 @@ }, "required": ["projectKey"] } + }, + { + "name": "list-metrics", + "description": "Search and browse metrics in a project. Use query to search by metric name or key. Matching is LITERAL case-insensitive substring, not semantic. Returns a paginated list.", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { "type": "string", "description": "The project key" }, + "query": { "type": "string", "description": "Search by metric name or key (literal case-insensitive substring)" }, + "limit": { "type": "number", "description": "Max number of results (default 20)" } + }, + "required": ["projectKey"] + } + }, + { + "name": "list-metric-events", + "description": "List recent events for a metric event key to check whether the metric is receiving data (event health).", + "input_schema": { + "type": "object", + "properties": { + "projectKey": { "type": "string", "description": "The project key" }, + "environmentKey": { "type": "string", "description": "Environment key (defaults to production)" }, + "eventKey": { "type": "string", "description": "The event key to inspect" } + }, + "required": ["projectKey", "eventKey"] + } } ] diff --git a/skills.json b/skills.json index 12b9412..e641200 100644 --- a/skills.json +++ b/skills.json @@ -163,6 +163,14 @@ "license": "Apache-2.0", "compatibility": "Requires SDK installed (parent Step 5) and LaunchDarkly project access" }, + { + "name": "launchdarkly-experiment-hypothesis-builder", + "description": "Help a user craft a strong, testable LaunchDarkly experiment hypothesis and extract the structured fields (intervention, primary metric + direction, expected effect, guardrails, audience) needed to auto-scaffold the rest of the experiment. Use when a user is starting an experiment from an idea/goal, or wants to sharpen a weak hypothesis before setup.", + "path": "skills/experiments/launchdarkly-experiment-hypothesis-builder", + "version": "0.1.0", + "license": "Apache-2.0", + "compatibility": "Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to." + }, { "name": "launchdarkly-experiment-setup", "description": "Set up and run experiments in LaunchDarkly. Create experiments with metrics, treatments, and flag config, start iterations to collect data, swap design between iterations, and stop with a winner.", diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md new file mode 100644 index 0000000..87ebef1 --- /dev/null +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md @@ -0,0 +1,221 @@ +--- +name: launchdarkly-experiment-hypothesis-builder +description: "Help a user craft a strong, testable LaunchDarkly experiment hypothesis and extract the structured fields (intervention, primary metric + direction, expected effect, guardrails, audience) needed to auto-scaffold the rest of the experiment. Use when a user is starting an experiment from an idea/goal, or wants to sharpen a weak hypothesis before setup." +compatibility: Requires the remotely hosted LaunchDarkly MCP server. Pairs with launchdarkly-experiment-setup, which it hands off to. +license: Apache-2.0 +metadata: + author: launchdarkly + version: "0.1.0" + status: draft +--- + +# LaunchDarkly Experiment Hypothesis Builder + +**⛔ Advisory skill — you have NO write access.** Never call `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, `start-experiment-iteration`, or **any** tool whose name starts with `create-`, `update-`, `toggle-`, `start-`, or `delete-`. If such a tool appears in your available toolset, treat it as forbidden — it belongs to `launchdarkly-experiment-setup`, not to you. Calling even one is a failure of this skill. Your only outputs are text: a hypothesis and a handoff payload. + +> **Status: draft.** Early version, published for review. Behavior and the handoff contract may change. + +Your job is to produce **two text artifacts** — a polished hypothesis and a structured handoff payload — and then **stop**. You are advisory: a well-formed hypothesis encodes both the intervention (→ flag + treatments) and the outcome (→ metric), so a *separate* setup step can scaffold everything later. This skill produces: +1. A polished **hypothesis string**. +2. A **structured JSON handoff payload** for `launchdarkly-experiment-setup` (which otherwise assumes the hypothesis is already known). + +> ## ⛔ STOP — this skill NEVER writes to LaunchDarkly +> You **must not call any `create-*`, `update-*`, `toggle-*`, or `start-*` tool** — no creating flags, metrics, or experiments; no toggling flags on/off; no starting iterations. Your entire output is **text**: a hypothesis and the Step 9 handoff payload. Do **not** say a flag was "created" or "is live," and do **not** build a full experiment yourself. After emitting the payload, **STOP**. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never create the flag/metric/experiment yourself as a fallback. Only read-only lookups (`get-flag`, `list-flags`, `list-metrics`, `get-metric`) are permitted; anything that mutates state is not. +> +> Two more hard stops before you build anything: +> - **Non-real input** — platform self-tests, A/A bucketing checks, placeholders, gibberish → confirm intent first, build nothing (Step 0). +> - **Metric ≠ predicted outcome** → reconcile with the user first, don't compose (Step 3.5). + +## Anatomy of a strong hypothesis + +A strong hypothesis names six elements. Use this as the rubric: + +| # | Element | Question it answers | Feeds experiment field | +|---|---------|--------------------|------------------------| +| 1 | **Intervention** | What specific change are we making? | Flag + treatments (control vs. variant) | +| 2 | **Audience** | Who sees it? / how are they split? | Targeting rule + randomization unit | +| 3 | **Primary metric** | What single number defines success? | `primarySingleMetricKey` | +| 4 | **Direction** | Should it go up or down? | Metric `successCriteria` | +| 5 | **Expected effect** | By roughly how much? | Powering / sample-size, analysis config | +| 6 | **Rationale + guardrails** | Why do we expect this? What must NOT get worse? | Secondary/guardrail metrics | + +**Canonical template:** +> *If we **[intervention]** for **[audience]**, then **[primary metric]** will **[direction]** by **[~magnitude]**, because **[rationale]** — while **[guardrail metric]** stays flat.* + +**Three quality checks beyond the six elements** (a hypothesis can have all six and still be broken): +- **Falsifiable** — there is a result that would prove it wrong. "Will do better or as well" and "figure out which resonates" fail this. +- **Single-variable** — exactly one thing differs between control and treatment; bundled changes destroy attribution. +- **Grounded** — tied to the observed usage data that prompted it, not just a hunch. + +## Coach to the common gaps + +Weak hypotheses tend to fail in predictable ways. Prioritize eliciting the rarest, highest-value elements first: + +- **A measurable metric is the #1 gap** — without a concrete primary metric nothing downstream can auto-select or create it. **Always** pin one down. +- **Magnitude is almost never stated.** Ask for a rough number (even "~3–5%"); it's needed for powering. +- **Rationale ("because…") is rare.** The "why" sharpens the design and helps reviewers. +- Direction and if/then structure are the easier wins — scaffold structure and confirm direction. + +Prioritize eliciting **metric → magnitude → rationale**, in that order. Most drafts need active coaching, not rubber-stamping. When a user's outcome is vague, suggest a concrete primary metric — conversion is by far the most common in practice, followed by engagement, clicks, and signups. + +## Workflow + +### Step 0 — Gate non-real input (do this FIRST — HARD STOP) +Before anything else, classify the input. If it is a platform self-test, an **A/A test** (identical variants — e.g. "A/A test to validate bucketing", SRM/bucketing checks), a placeholder, or gibberish (see [Detecting low-effort / non-real input](#detecting-low-effort--non-real-input) — also "testing the LaunchDarkly platform", "dummy flag", "If X then Y", "asdf"), **STOP.** Your **entire reply** must be a single short question that (a) names it as a platform/self-test, not a real experiment, and (b) asks the user to give a real A/B idea (a specific change with a measurable outcome) or confirm intent. + +Do **not** produce a hypothesis, a "setup summary", a handoff payload, or an "A/A … hypothesis / bucketing-validation brief" — an A/A test has identical variants and therefore *no* hypothesis, so drafting one is precisely the failure to avoid. Say it's not an experiment and ask; nothing else. Only a genuine A/B idea proceeds to Step 1. + +### Step 1 — Capture the raw input +Accept whatever the user starts with: a free-text idea, a goal, a flag they already have, or a metric they care about. Don't require structure yet. + +### Step 2 — Diagnose by flaw type, then score +First check which of the six elements are present. Then diagnose **flaw type**, because the corrective move differs by flaw. A hypothesis usually has several. The full branch-by-branch decision tree — diagnosis → correction → flag/variations/metrics/guardrails → config summary — is in `references/diagnostic-tree.md`; **read it when a hypothesis is weak or you're configuring the experiment.** The flaw taxonomy: + +| Flaw | Tell | Correction move | +|------|------|-----------------| +| **Vague/absent intervention** | names a goal, not a change ("increase revenue") | force a specific control vs. treatment | +| **No measurable metric** | outcome is an adjective ("better performance") | operationalize into one primary metric + direction | +| **Missing causal mechanism** | no "because" | add the *why*; if none, question testing it | +| **Not falsifiable** | "will do better or as well", "figure out which resonates", tautology | commit to a directional, disconfirmable prediction + decision rule | +| **Conflates multiple variables** | bundles changes ("colors + typography + hero") | isolate to one variable, or label as a package test with attribution caveat | +| **Not grounded in usage data** | asserts a problem with no evidence | tie to the observed signal; if none, mark assumption-driven | +| **Metric ↔ outcome mismatch** | predicts engagement but measures revenue | align primary metric to the *predicted* outcome | + +Classify overall: +- **Strong** — specific single-variable change + primary metric + direction, falsifiable (+ ideally magnitude/rationale). Proceed; only confirm. +- **Serviceable** — has intervention + direction but no concrete metric or magnitude, or a fixable flaw. Fill the gaps. +- **Weak** — vague goal / no measurable outcome / untestable (e.g. "Better engagement", "Increase revenue"). Rebuild from questions. + +### Step 3 — Ask ONLY for the missing high-value elements +Keep it to the fewest questions. Lead with the rarest gaps: **primary metric + direction**, then **magnitude**, then **rationale/guardrails**, then **audience** if unclear. Offer concrete options where you can (e.g. suggest plausible metrics based on the intervention). Don't interrogate — 1–3 targeted questions is the target. + +### Step 3.5 — Check metric–outcome alignment (flaw F7 — HARD STOP) +Before composing, explicitly name **the outcome the user predicts** and **the metric they proposed**, and check they measure the same thing. If they diverge — e.g. the user predicts **engagement** will move but says to **measure it by revenue** — that's flaw F7 (see `references/diagnostic-tree.md`) and you **STOP**. + +Do **not** silently pick some other metric to paper over the conflict, and do **not** just accept the mismatched metric. Say plainly that the predicted outcome (engagement) and the proposed metric (revenue) measure different things, and ask the user which to change — align the metric to the predicted outcome, or restate the outcome to match the metric. Wait for their answer before composing a hypothesis or emitting a payload. + +### Step 4 — Compose the polished hypothesis +Write one clear sentence using the canonical template. Keep the user's intent and voice; don't invent specifics they didn't confirm. Flag any assumption you had to make. + +### Step 5 — Emit the structured extraction +Return this JSON so downstream setup can proceed: + +```json +{ + "hypothesis": "polished single-sentence hypothesis", + "intervention": { + "summary": "what changes", + "control": "current experience", + "treatment": "new experience", + "flag_candidate_terms": ["stemmed", "synonym", "search", "terms"] + }, + "primary_metric": { + "name": "human name of the success metric", + "direction": "increase | decrease", + "metric_candidate_terms": ["stemmed", "synonym", "search", "terms"] + }, + "secondary_metrics": ["..."], + "guardrail_metrics": ["metrics that must not regress"], + "expected_effect": { "magnitude": "e.g. +5% (or null if unknown)", "known": true }, + "audience": { "targeting": "who / how split", "randomization_unit": "user" }, + "rationale": "why we expect this", + "quality": { "score": "0-6", "missing_elements": ["..."] } +} +``` + +### Step 6 — Generate search terms for matching existing flags/metrics +LaunchDarkly's `list-flags` / `list-metrics` `query` is **literal case-insensitive substring matching, not semantic** — e.g. `"completion"` does NOT match a metric named `"completed"`, and `"create"` does NOT match `"creation"`. So **do not** pass the hypothesis text verbatim to search. For each of `flag_candidate_terms` and `metric_candidate_terms`, emit several **stemmed / truncated / synonym** variants (e.g. `creation` → `creat`, `create`, `creation`; `completion` → `complet`, `completed`, `complete`), run multiple queries, union + dedupe, then rank candidates by name + description + tags and **confirm the pick with the user** (near-decoys often rank alongside the target). + +### Step 7 — Match flag & metric keys (read-only lookup) +Using only **read-only** lookups (`list-flags`, `list-metrics`, `get-flag`, `get-metric`), try to match the candidate *terms* to existing LD **keys** so the payload can carry a real key rather than a name. First establish `projectKey` and `environmentKey` (ask if not already known; default env `production`). You never create or toggle anything here — you only look up and record. Then: +- **Flag:** run the expanded `flag_candidate_terms` through `list-flags`; if a confirmed match exists, record its key with `action: use_existing`. Otherwise record a *proposed* boolean flag in the payload (`control` = off/current, `treatment` = on/changed) with `action: create` and a proposed kebab-case key naming the *toggle* (not the outcome) — a proposal for the downstream step, which you do not execute. **Never call `create-flag`/`create-feature-flag` to make this flag or to get its variation IDs; leave the IDs null.** +- **Primary metric:** run `metric_candidate_terms` through `list-metrics`; on a confirmed match record its key + `action: use_existing`; else record `action: create` in the payload with `measureType` (occurrence/count/value) and `successCriteria` derived from `direction` — again a proposal, not a creation you perform. +- **Guardrail/secondary metrics:** resolve the same way (guardrails usually already exist — latency, error rate, refunds). +- Confirm every pick with the human (near-decoys rank alongside targets). Record the resolved keys + actions in the handoff payload (Step 9). **Do not create anything here** — `launchdarkly-experiment-setup` owns all writes, flag-version ordering, and event-health checks. + +### Step 8 — Check MDE / sample size, then print the configuration summary +Before setup, sanity-check power: from the expected magnitude, smaller lift → larger sample / longer runtime. If the primary metric's baseline volume can't reach significance for the stated effect in a reasonable window, say so and either raise the target effect, pick a higher-volume metric, or extend runtime. Watch guardrails and one primary metric to control false positives. + +Always end with this configuration summary: + +``` +Hypothesis: If we [change] for [audience], then [primary metric] will [direction] + by [~magnitude], because [mechanism] — while [guardrail] stays flat. +Flag: (boolean | multivariate) +Variations: Control = + Treatment = +Primary metric: (higher/lower is better) +Guardrail(s): +Sample/runtime: → ~ / ~ at current volume +``` + +### Step 9 — Emit the handoff payload, then STOP +Once the human approves the configuration summary, **output this handoff payload as your final message — as text — and then STOP.** Do not call any tool. Do not create a flag, metric, or experiment; do not toggle or start anything; do not report that anything was "created" or "is live." A separate `launchdarkly-experiment-setup` step consumes this payload later and performs any writes behind its own human confirmation — that is not your job. + +```json +{ + "handoffFrom": "launchdarkly-experiment-hypothesis-builder", + "projectKey": "...", + "environmentKey": "production", + "hypothesis": "polished single-sentence hypothesis", + "description": "plain-language description of the change being tested", + "methodology": "bayesian", + "primarySingleMetricKey": "resolved-primary-metric-key", + "metrics": [ + { "key": "resolved-primary-metric-key", "role": "primary", "measureType": "occurrence|count|value", "successCriteria": "HigherThanBaseline|LowerThanBaseline", "action": "use_existing|create" }, + { "key": "guardrail-metric-key", "role": "guardrail", "successCriteria": "...", "action": "use_existing|create" } + ], + "flag": { + "key": "resolved-or-proposed-flag-key", + "action": "use_existing | create", + "kind": "boolean | multivariate", + "ruleId": "fallthrough", + "controlVariationId": "id-of-control-variation-or-null-until-created", + "treatmentVariationId": "id-of-treatment-variation-or-null-until-created" + }, + "treatments": [ + { "name": "Control", "baseline": true, "allocationPercent": 50, "experience": "specific current experience" }, + { "name": "Treatment", "baseline": false, "allocationPercent": 50, "experience": "specific changed experience" } + ], + "randomizationUnit": "user | request | organization | device", + "expectedEffect": "+5%", + "mdeNote": "at current volume, ~N/arm / ~D days to detect this effect", + "quality": { "score": "0-6", "missing_elements": [] } +} +``` + +The `action: use_existing | create` fields describe what the *downstream* `launchdarkly-experiment-setup` step should do (look up vs. create the flag/metric, resolve variation IDs, toggle the flag on with proper version ordering, then create + start behind human confirmation). They are **not** instructions for you to execute — you only emit the payload and stop. + +**Do not create a flag to obtain variation IDs.** For an `action: create` flag the flag does not exist yet, so set `controlVariationId` and `treatmentVariationId` to `null` — the downstream step creates the flag and fills them in. Needing an ID (or any "resolved" value) is *never* a reason to call `create-flag` / `create-feature-flag` or any other write tool. Emit the payload with nulls and stop. + +## Scoring examples + +**Strong** (ready to build): +> "If we align the navigation to the left, then signup conversion rate will increase by improving scannability and reducing cognitive load, while login success rate remains unchanged." +- ✅ intervention, ✅ primary metric (signup conversion), ✅ direction, ✅ rationale, ✅ guardrail (login success). Only missing an explicit magnitude — ask once, then build. + +**Serviceable** (fill 1–2 gaps): +> "Mini charts on the screener page will increase trades." +- Has intervention + direction + metric (trades). Missing magnitude, rationale, audience. Ask: expected lift? why? which users? + +**Weak** (rebuild via questions): +> "Better engagement." / "Increase revenue." +- No change, no concrete metric. Ask: what specific change? engagement/revenue measured how (metric)? for whom? expected direction and size? + +## Detecting low-effort / non-real input + +Some entries are platform tests, not experiments. If the input looks like one, gently confirm intent rather than building a hypothesis. Common signals: +- Placeholders / gibberish: "If X then Y", "this is a test", "ABC", "asdf", single words. +- Platform self-tests: "testing the LaunchDarkly platform", "A/A test to validate bucketing", "dummy flag", "just for dev env". +- Meta: "I have to fill this out to delete the experiment." + +Note: a hypothesis that merely mentions "A/B test" or "test group" as part of a real idea is fine — only filter genuine platform/self-tests. + +## What NOT to do + +- Don't accept a vague goal as a hypothesis — a hypothesis without a measurable primary metric can't drive an experiment. +- Don't invent a metric, magnitude, or audience the user didn't confirm; surface assumptions instead. +- Don't pass raw hypothesis text to flag/metric search — expand into stemmed/synonym query terms first. +- Don't over-interrogate. Lead with the rarest, highest-value gaps (metric, magnitude, rationale) and cap at ~3 questions. +- **Never write to LaunchDarkly.** Don't call any `create-`, `update-`, `toggle-`, `start-`, or `delete-` tool — specifically not `create-flag`, `create-feature-flag`, `update-flag-settings`, `update-feature-flag`, `toggle-flag`, `create-metric`, `create-experiment`, or `start-experiment-iteration`. No creating flags/metrics/experiments, toggling flags, or starting iterations. Emit the handoff payload and STOP. If `launchdarkly-experiment-setup` is unavailable to receive it, still just output the payload — never do the writes yourself as a fallback. +- Don't build a hypothesis or any configuration for non-real input (platform self-tests, A/A bucketing checks, placeholders, gibberish) — gate it in Step 0 and confirm intent first. diff --git a/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md b/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md new file mode 100644 index 0000000..89ee13d --- /dev/null +++ b/skills/experiments/launchdarkly-experiment-hypothesis-builder/references/diagnostic-tree.md @@ -0,0 +1,97 @@ +# Hypothesis Diagnostic Decision Tree + +Organized by **flaw type**, not by any specific hypothesis — so it generalizes across submissions. Diagnose first (a hypothesis often has several flaws), correct each branch, then continue into flag / variations / metrics / guardrails and finish with a configuration summary. + +Target shape after correction: +> **If [single specific change], then [primary metric] will [direction] by [≥ MDE], because [causal mechanism grounded in observed data] — while [guardrail metric] does not regress.** + +--- + +## Stage A — Diagnose the flaw(s) + +Run every check; record all that fire. Then apply the matching correction move. + +| # | Flaw | Symptom / tells | Correction move | +|---|------|-----------------|-----------------| +| F1 | **Vague or absent intervention** | "Better engagement", "Increase revenue", "improve onboarding" — names a goal, not a change | Elicit the *specific* change. Force a concrete control vs. treatment ("button copy 'Buy Now' vs. 'Get Started'", not "new button"). | +| F2 | **No measurable success metric** | "improve performance", "better experience", outcome is an adjective | Operationalize the outcome into ONE primary metric with a direction (latency ms, conversion rate, trades/user). | +| F3 | **Missing causal mechanism** | change→outcome stated, no "because"; can't say *why* it would work | Add the mechanism. If no plausible mechanism exists, question whether it's worth testing. | +| F4 | **Not falsifiable / untestable** | "will do better or as well", "should have no negative impact", "figure out which resonates", tautology | Commit to a directional, disconfirmable prediction with a threshold. Reframe exploratory "which is better?" as an A/B with an explicit decision rule. | +| F5 | **Conflates multiple variables** | bundles changes ("colors + typography + hero", "redesign + new CTA + new copy") | Isolate to one variable. If the bundle must ship together, label it explicitly as a "does the package work" test and note attribution is lost + plan follow-up isolations. | +| F6 | **Not grounded in usage data** | asserts a problem/opportunity with no evidence it exists | Tie to the observed signal that prompted it ("27% drop off at step X"). If there's no data, mark assumption-driven, lower priority, or measure a baseline first. | +| F7 | **Metric ↔ outcome mismatch** | predicts one thing (engagement) but proposes measuring another (revenue) | Align the primary metric to the *predicted* outcome; demote the rest to secondary/guardrail. | +| F8 | **Directionally ambiguous / multi-outcome** | "will differ", "will impact volume", no clear up/down | Commit to an expected direction (or explicitly frame as a two-sided / guardrail test). | + +--- + +## Stage B — Rebuild the hypothesis + +1. Take the corrected pieces and write ONE sentence in the canonical form. +2. Re-check falsifiability: *"What result would prove this wrong?"* — if you can't answer, it isn't done. +3. Re-check single-variable: *"Is exactly one thing changing between control and treatment?"* +4. Re-check grounding: *"What in the data made us believe this?"* + +--- + +## Stage C — Continue the tree to configuration + +### C1 — Flag +- **What is toggled?** = the intervention from F1. +- **Name** it in kebab-case describing the toggle, not the outcome: `search-mini-charts`, `paywall-simplified`, `terms-copy-casual`. +- **Kind:** boolean if control vs. one treatment; multivariate if 3+ variants (e.g., copy A/B/C). + +### C2 — Variations +- **Control** = the current experience, stated concretely (not "old"). +- **Treatment(s)** = the changed experience, implementation-specific: exact copy, values, layout — enough that an engineer could build it without asking. +- One variable differs across variations (ties back to F5). + +### C3 — What to measure +- **Primary metric** = the single number the hypothesis predicts will move, with direction → `successCriteria` (higher/lower is better). One primary only (ties back to F2/F7). +- **Secondary metrics** = supporting signals you expect to move but won't decide on. +- **Guardrail metrics** = things that must NOT regress (latency, error rate, refunds, unsubscribes) — the defense against a "win" that quietly hurts elsewhere. + +### C4 — Best-practice checks before launch +- **Single-variable isolation** — confirmed in C2. +- **Minimum Detectable Effect (MDE) + sample size** — from the expected magnitude: smaller expected lift → larger sample / longer runtime. If the metric's baseline volume can't reach significance for the stated MDE in a reasonable window, say so and either raise the MDE, pick a higher-volume metric, or extend runtime. +- **False-positive control** — one primary metric; if watching many metrics, apply multiple-comparison correction and don't peek/stop early. +- **Guardrails defined** — at least one, per C3. + +--- + +## Stage D — Configuration summary (always end here) + +``` +Hypothesis: If we [change] for [audience], then [primary metric] will [direction] + by [~magnitude ≥ MDE], because [mechanism grounded in data] — while + [guardrail] stays flat. +Flag: (boolean | multivariate) +Variations: Control = + Treatment = +Primary metric: (higher/lower is better) +Guardrail(s): +Sample/runtime: → ~ / ~ at current volume +``` + +--- + +## Worked traversals (real, lightly anonymized submissions) + +### "Enabling batching will improve performance" +- **Flaws:** F2 (no metric — "performance"), F3 (no mechanism), F8 (no direction stated concretely), F6 (grounding unknown). +- **Corrected:** *If we enable request batching for all backend traffic, then p95 request latency will decrease by ~15%, because batching amortizes per-request overhead — while error rate stays flat.* +- **Config:** flag `request-batching` (boolean); Control = batching off, Treatment = batching on; primary = p95 latency (lower better); guardrail = error rate; randomization unit = **request** (not user). + +### "New brand UI (colors, typography, and hero) will increase signups" +- **Flaws:** F5 (three variables bundled), F3 (mechanism thin), no magnitude. +- **Corrected (isolation path):** *If we change signup-page typography to the new brand scale, then signup conversion rate will increase by ~2%, because improved hierarchy speeds scanning — while login success rate stays flat.* → plan separate tests for color and hero. +- **Corrected (bundle path, if it must ship together):** keep all three but label "package test — attribution across the three changes is not separable," and schedule isolations later. +- **Config:** flag `signup-brand-typography` (boolean); Control = current type scale, Treatment = new brand type scale; primary = signup conversion (higher better); guardrail = login success rate. + +### "Figure out which wallet value-prop copy resonates most" +- **Flaws:** F4 (exploratory, not falsifiable), F2 (no metric), F1 (variants unspecified). +- **Corrected:** *If we show wallet value-prop copy "Save automatically" (B) vs. current "Manage your wallet" (A), then wallet-activation rate will be higher for B by ≥3%, because outcome-framed copy states the benefit — decision rule: ship the higher arm only if lift ≥3% and refund rate is flat.* +- **Config:** flag `wallet-valueprop-copy` (multivariate if >2 copies); Control = "Manage your wallet", Treatment = "Save automatically"; primary = wallet activation rate (higher better); guardrail = refund rate. + +### "Increase revenue" +- **Flaws:** F1 (no change), F2 (revenue is the goal, not an operational metric here), F3, F6 — essentially a goal, not a hypothesis. +- **Correction:** cannot proceed as a hypothesis. Ask: what specific change, for whom, and which revenue metric (ARPU? checkout conversion? AOV?), grounded in what data? Rebuild from F1.