Skip to content

fix(claude-sdk): workspace skills not discovered by Claude Agent SDK provider #876

@christso

Description

@christso

Problem

The claude-sdk provider does not discover workspace-level skills from .claude/skills/. When running skill-trigger evals with provider: claude-sdk, positive test cases always fail because the agent doesn't know the skill exists.

Evidence

Running multi-provider-skill-trigger.EVAL.yaml (from examples/features/agent-skills-evals/) with --target claude-sdk:

Test claude-sdk result
should-trigger-direct-request (should_trigger: true) FAIL — skill not found
should-trigger-casual-phrasing (should_trigger: true) FAIL — skill not found
should-not-trigger-unrelated (should_trigger: false) PASS
should-not-trigger-near-miss (should_trigger: false) PASS

The same eval passes all positive cases with claude-cli (claude -p subprocess).

claude-sdk session init — NO workspace skills or plugins:

"skills": ["keybindings-help", "debug", "claude-developer-platform"],
"plugins": []

claude-cli session init — workspace skills discovered automatically:

"skills": ["keybindings-help", "debug", "claude-developer-platform", "csv-analyzer", ...],
"plugins": [{"name": "frontend-design", ...}, ...]

Root Cause

claude -p CLI auto-discovers .claude/skills/ from the workspace cwd. The Claude Agent SDK's query() does not — it needs explicit configuration.

The AgentV ClaudeSdkProvider (in packages/core/src/evaluation/providers/claude-sdk.ts) passes only: model, cwd, systemPrompt, maxTurns, maxBudgetUsd, env. It does NOT pass skills or plugin config.

SDK Capabilities (already available)

The SDK (@anthropic-ai/claude-agent-sdk v0.2.49, types in sdk.d.ts) supports:

  • Options.plugins?: SdkPluginConfig[] — load local plugins: [{ type: 'local', path: './my-plugin' }]
    • BUT .claude/skills/ are bare skill files (SKILL.md), not plugins with plugin.json manifests
  • AgentDefinition.skills?: string[] — preload skill names into agent context
  • Options.allowedTools?: string[] / Options.disallowedTools?: string[] — tool control

Files to Modify

File Change
packages/core/src/evaluation/providers/claude-sdk.ts Pass skills/plugins to SDK query() options
packages/core/src/evaluation/providers/targets.ts Extend ClaudeResolvedConfig if new target fields needed
examples/features/.agentv/targets.yaml Add claude-sdk target definition

Design Question

.claude/skills/ directories are bare SKILL.md files, not structured plugins. The SDK's plugins option expects { type: 'local', path } pointing to a plugin with plugin.json. Options to bridge this:

  1. Auto-detect and inject via system prompt — read SKILL.md files from workspace .claude/skills/ and append to systemPrompt. Pragmatic, works today, no SDK changes needed.
  2. Wrap as temporary plugin — create an ephemeral plugin manifest pointing to the workspace skills. More complex but uses the SDK's native plugin system.
  3. Check if SDK auto-discovers from cwd — verify whether setting cwd to the workspace root triggers built-in skill discovery in newer SDK versions. If so, this might be a version issue.

Acceptance Criteria

  • multi-provider-skill-trigger.EVAL.yaml passes all 4 tests with --target claude-sdk
  • Workspace .claude/skills/ are available to the SDK session
  • No manual skill configuration required in targets.yaml for workspace skills

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions