Improve AI Configs

[sattensil]

AI Config variations: Cannot read or write tools, model, customParameters, judgeConfiguration

TL;DR

The MCP server's OpenAPI schema is missing critical fields. Users cannot configure AI agent variations via MCP—tools are silently dropped, model parameters ignored, and these fields don't appear in GET responses even when set via UI.

Result: MCP is unusable for AI Config agent workflows. Users must fall back to the UI or direct REST API.

---

Reproduction

1. Try to create a variation with tools

MCP call: create-ai-config-variation
{
  "projectKey": "my-project",
  "configKey": "my-agent",
  "key": "v1",
  "name": "Agent V1",
  "instructions": "You are a helpful assistant",
  "tools": [{"key": "search", "version": 1}],
  "model": {
    "modelName": "claude-3-5-sonnet",
    "parameters": {"temperature": 0.7, "max_tokens": 4096}
  }
}

Expected: Variation created with tools and model settings
Actual: Variation created, but tools missing and model is {}

2. Try to read a variation configured via UI

MCP call: get-ai-config-variation
{
  "projectKey": "my-project", 
  "configKey": "my-agent",
  "variationId": "abc123"
}

Expected: Response includes tools, model, customParameters
Actual: Response shows model: {}, no tools field—even though tools are visible in the LaunchDarkly UI

---

Affected Fields

Field	Write	Read	Purpose
tools	❌	❌	Attach tool definitions to agent variations
toolKeys	❌	❌	Simplified tool attachment (latest versions)
model.modelName	❌	❌	Model identifier
model.parameters	❌	❌	temperature, max_tokens, etc.
customParameters	❌	❌	Custom key-value pairs
judgeConfiguration	❌	❌	Online evaluation settings

# AI Config variations: Cannot read or write `tools`, `model`, `customParameters`, `judgeConfiguration`

Root Cause

The bundled OpenAPI spec (schemas/openapi.json) defines incomplete schemas that Speakeasy generates into Zod validators.

POST/PATCH Schema

File: src/models/components/aiconfigvariationpost.ts

// model is an empty object - no modelName, no parameters
export type AIConfigVariationPostModel = {};
export const AIConfigVariationPostModel$inboundSchema = z.object({});

// Main schema missing tools, toolKeys, customParameters, judgeConfiguration
export const AIConfigVariationPost$inboundSchema = z.object({
  key: z.string(),
  name: z.string(),
  instructions: z.string().optional(),
  messages: z.array(Message$inboundSchema),
  model: z.lazy(() => AIConfigVariationPostModel$inboundSchema).optional(),
  modelConfigKey: z.string().optional(),
  // ❌ tools, toolKeys, customParameters, judgeConfiguration not defined
});

GET Response Schema

File: src/models/components/aiconfigvariation.ts

// Same problem - tools stripped from API response before reaching client
export const AIConfigVariation$inboundSchema = z.object({
  key: z.string(),
  model: z.lazy(() => AIConfigVariationModel$inboundSchema),  // Empty {}
  // ❌ tools not defined - stripped even if API returns them
});

Validation Blocks Requests

File: src/funcs/aiConfigsCreateVariation.ts

const parsed = safeParse(
  request,
  (value) => operations.PostAIConfigVariationRequest$outboundSchema.parse(value),
  "Input validation failed",
);
if (!parsed.ok) {
  return [parsed, { status: "invalid" }];  // Request never reaches API
}

---

Evidence: REST API Works

The same payload succeeds when sent directly to the REST API:

# This works - bypassing MCP
payload = {
    "key": "v1",
    "name": "Agent V1", 
    "instructions": "You are a helpful assistant",
    "tools": [{"key": "search", "version": 1}],
    "modelConfigKey": "Anthropic.claude-3-5-sonnet",
    "customParameters": {"temperature": 0.7}
}
requests.post(f"{base}/api/v2/projects/{proj}/ai-configs/{config}/variations", 
              json=payload, headers=headers)
# ✅ Tools attached, customParameters stored

[Official API docs](https://launchdarkly.com/docs/api/ai-configs/post-ai-config-variation) confirm these fields are supported.

---

Additional Gap: Missing /ai-tools Endpoints

The MCP server has no tools for managing AI tool definitions:

Missing MCP Tool	REST Endpoint
create-ai-tool	POST /api/v2/projects/{project}/ai-tools
list-ai-tools	GET /api/v2/projects/{project}/ai-tools
get-ai-tool	GET /api/v2/projects/{project}/ai-tools/{key}
update-ai-tool	PATCH /api/v2/projects/{project}/ai-tools/{key}
delete-ai-tool	DELETE /api/v2/projects/{project}/ai-tools/{key}

Without these, users cannot create the tool definitions that variations reference.

---

Suggested Fix

Since the SDK is auto-generated by Speakeasy, update the source OpenAPI specification with the missing fields:

AIConfigVariationPost:
  properties:
    tools:
      type: array
      items:
        type: object
        properties:
          key: { type: string }
          version: { type: integer }
        required: [key, version]
    toolKeys:
      type: array
      items: { type: string }
    model:
      type: object
      properties:
        modelName: { type: string }
        parameters: { type: object, additionalProperties: true }
    customParameters:
      type: object
      additionalProperties: true
    judgeConfiguration:
      type: object
      properties:
        judges:
          type: array
          items:
            type: object
            properties:
              judgeConfigKey: { type: string }
              samplingRate: { type: number }

Then regenerate the SDK and add MCP tools for the /ai-tools endpoints.

---

Workaround

Until fixed, users must:

1. Use MCP to create the AI Config and variation skeleton
2. Complete configuration (tools, model, customParameters) in the LaunchDarkly UI
3. Or bypass MCP entirely and use direct REST API calls

---

Environment

• MCP Server: @launchdarkly/mcp-server@0.4.2
• Generator: Speakeasy
• Tested with: Cursor, Claude Desktop

Files Affected

File	Role
schemas/openapi.json	Source spec (root cause)
src/models/components/aiconfigvariationpost.ts	POST schema
src/models/components/aiconfigvariationpatch.ts	PATCH schema
src/models/components/aiconfigvariation.ts	GET response schema
src/mcp-server/tools/	Missing ai-tools endpoints