feat(mcp): implement enterprise resilience pipeline (circuit breaker, schema validation)

Author: Shanyu-Dabbiru

apps/sim/lib/mcp/resilience/ARCHITECTURE.md

@@ -0,0 +1,79 @@
+# MCP Resilience Pipeline 
+
+**Objective:** Upgrade the core `executeTool` engine from a naive proxy to an Enterprise-Grade Resilience Pipeline, ensuring our AI workflows never suffer cascading failures from downstream server instability or LLM hallucinations.
+
+---
+
+## 1. The "Thundering Herd" Problem (Circuit Breaker)
+
+**Before:**
+When a downstream provider (e.g., a database or API) experienced latency or went down, our workflow engine would continuously retry. If 1,000 agents hit a struggling server simultaneously, they would overwhelm it (a DDOS-like "thundering herd"), crash our workflow executor, and severely degrade user experience across the platform.
+
+**After (`CircuitBreakerMiddleware`):**
+We implemented an intelligent State Machine with a **HALF-OPEN Concurrency Semaphore**.
+- **The Trip:** If a server fails 3 times, we cut the circuit (`OPEN` state). All subsequent requests instantly *fast-fail* locally (0ms latency), protecting the downstream server from being hammered.
+- **The Elegant Recovery:** After a cooldown, we allow exactly **one** probe request through (`HALF-OPEN`). If it succeeds, the circuit closes. If it fails, it trips again.
+
+#### Live Demo Output
+
+```mermaid
+sequenceDiagram
+    participant Agent
+    participant Pipeline
+    participant TargetServer
+    
+    Agent->>Pipeline: executeTool (Server Down)
+    Pipeline--xTargetServer: ❌ Fails (Attempt 1-3)
+    Note over Pipeline: 🔴 Tripped to OPEN
+    Agent->>Pipeline: executeTool
+    Pipeline-->>Agent: 🛑 Fast-Fail (0ms latency) - Target Protected
+    Note over Pipeline: ⏳ Cooldown... 🟡 HALF-OPEN
+    Agent->>Pipeline: executeTool (Probe)
+    Pipeline-->>TargetServer: Exact 1 request allowed
+    TargetServer-->>Pipeline: ✅ Success
+    Note over Pipeline: 🟢 Reset to CLOSED
+    Agent->>Pipeline: executeTool
+    Pipeline-->>TargetServer: Resume normal traffic
+```
+
+---
+
+## 2. LLM Hallucinated Arguments (Schema Validator)
+
+**Before:**
+If an LLM hallucinated arguments that didn't match a tool's JSON schema, the downstream server or our proxy would throw a fatal exception. The workflow would crash, requiring user intervention, and wasting the compute/tokens already spent.
+
+**After (`SchemaValidatorMiddleware`):**
+We implemented high-performance **Zod Schema Caching**. 
+- We intercept the tool call *before* it leaves our system.
+- If the schema is invalid, we do *not* crash. Instead, we return a gracefully formatted, native MCP error: `{ isError: true, content: "Schema validation failed: [Zod Error Details]" }`.
+- **The Magic:** The LLM receives this error, realizes its mistake, and natively **self-corrects** on the next turn, achieving autonomous self-healing without dropping the user's workflow.
+
+---
+
+## 3. The "Black Box" Problem (Telemetry)
+
+**Before:**
+If a tool execution tool 10 seconds or failed, we had no granular visibility into *why*. Was it a network timeout? A validation error? A 500 from the target?
+
+**After (`TelemetryMiddleware`):**
+Every single tool execution now generates rich metadata:
+- `latency_ms`
+- Exact `failure_reason` (e.g., `TIMEOUT`, `VALIDATION_ERROR`, `API_500`)
+- `serverId` and `workspaceId`
+
+This allows us to build real-time monitoring dashboards to detect struggling third-party integrations before our users even report them.
+
+---
+
+## Architectural Impact: The Composable Pipeline
+
+Perhaps the most significant engineering achievement is the **Architecture Shift**. We moved away from a brittle, monolithic proxy to a modern **Chain of Responsibility**.
+
+```typescript
+// The new elegant implementation in McpService
+this.pipeline = new ResiliencePipeline()
+  .use(this.telemetry)
+  .use(this.schemaValidator)
+  .use(this.circuitBreaker)
+```

apps/sim/lib/mcp/resilience/README.md

@@ -1,15 +1,3 @@
-# Goal Description
-Implement a "Day 2" reliability layer for the simstudioai/sim workflow engine by building a composable Resilience Interceptor/Middleware Pipeline for the MCP `executeTool` logic. This pipeline ensures enterprise-grade stability by introducing a Circuit Breaker State Machine, Zod-based Schema Enforcement for LLM outputs, and detailed Telemetry for latency and failure analysis, while addressing high-concurrency Node/TS environments.
-
-## User Review Required
-- Please confirm if `apps/sim/lib/mcp/service.ts` is the correct core injection point for wrapping `executeTool`.
-- Note on file path: `apps/sim/lib/workflow/executor.ts` was not found. Instead, `apps/sim/executor/execution/executor.ts` and `apps/sim/tools/workflow/executor.ts` were analyzed. Ensure intercepting `McpService`'s `executeTool` serves your architectural needs.
-- Please confirm the schema enforcement approach: we will compile and cache JSON Schemas to Zod validators upon MCP server discovery or lazily, instead of parsing dynamically per request.
-
-## Proposed Changes
-
-We will split the implementation into discrete PRs / Commits to maintain structure.
-
 ### Part 1: Telemetry Hooks
 Implement the foundation for tracking.
 *(Change Rationale: Transitioning to a middleware pattern instead of a monolithic proxy, allowing telemetry to be composed easily).*