From 4b01055786c6019f4c0ce222926d1d622502545b Mon Sep 17 00:00:00 2001 From: kawofong <14829553+kawofong@users.noreply.github.com> Date: Thu, 9 Apr 2026 17:49:22 -0400 Subject: [PATCH] package as a agent skill --- README.md | 26 +++++++- SKILL.md | 65 +++++++++++++++++++ .../graph-patterns.md | 2 +- .../sdk-request-failures.md | 2 +- 4 files changed, 91 insertions(+), 4 deletions(-) create mode 100644 SKILL.md rename GRAPH_PATTERNS.md => references/graph-patterns.md (98%) rename SDK_METRICS_REQUEST_FAILURES.md => references/sdk-request-failures.md (99%) diff --git a/README.md b/README.md index 7040504..7515bbc 100644 --- a/README.md +++ b/README.md @@ -11,16 +11,38 @@ This isn't a metrics glossary. It's a troubleshooting guide built around how pro - **It connects the dots.** Rather than explaining metrics in isolation, it guides you across related signals so you can build a complete picture of what's happening — not just identify a single data point. - **It covers real impact.** Every entry is grounded in what the failure actually means for your system and your users, not just what the metric technically represents. - **It's practical beyond incidents.** The same knowledge applies to cluster and database sizing, capacity planning, and understanding system behavior under load testing. -- **It works as a Claude skill.** The content is structured to be used directly with Claude for interactive troubleshooting. +- **It works as a Claude skill.** The content is structured to be used directly with Claude for interactive troubleshooting. See [Installation](#installation) below. This is a living document. It will keep growing as I work through new patterns and scenarios. --- +## Installation + +This repository is packaged as a Claude Code Agent Skill. Install it to give Claude access to this knowledge base during interactive troubleshooting sessions. + +### Option 1: Manual clone + +```bash +mkdir -p ~/.claude/skills +git clone https://github.com/tsurdilo/temporal-metrics ~/.claude/skills/temporal-metrics +``` + +### Option 2: npx skills + +```bash +npx skills add tsurdilo/temporal-metrics +``` + +Once installed, Claude will automatically activate the skill when you ask about Temporal SDK metrics, rate limiting, throttling, gRPC status codes, or production incident troubleshooting. + +--- + Useful dashboards to use alongside this knowledge base: - https://github.com/tsurdilo/my-temporal-dockercompose - https://github.com/tsurdilo/my-temporal-dockercompose/tree/main/deployment/grafana/dashboards ## Table of Contents -1. [Troubleshooting SDK Request Failure Metrics](./SDK_METRICS_REQUEST_FAILURES.md) \ No newline at end of file +1. [Troubleshooting SDK Request Failure Metrics](./references/sdk-request-failures.md) +2. [Graph Pattern Reference](./references/graph-patterns.md) diff --git a/SKILL.md b/SKILL.md new file mode 100644 index 0000000..54b068e --- /dev/null +++ b/SKILL.md @@ -0,0 +1,65 @@ +--- +name: temporal-metrics +description: > + This skill should be used when the user asks about "temporal_request_failure", + "temporal_long_request_failure", "ResourceExhausted", "SDK metrics", + "rate limiting", "throttling", "gRPC status codes", "namespace RPS", + "execution bucket", "visibility bucket", "poll throttling", + "schedule to start latency", "workflow task timeout", "activity timeout", + "heartbeat timeout", "resource exhausted cause", "server dashboard", + "metric troubleshooting", "Temporal production incident", + "request failure metrics", "graph patterns", "RPS quota", + "frontend.namespaceRPS", or asks to diagnose Temporal metrics or + troubleshoot Temporal SDK errors. +version: 0.1.0 +--- + +# Temporal Metrics Troubleshooting Skill + +You now have access to a production-focused troubleshooting knowledge base for Temporal SDK request failure metrics. This knowledge comes from 5+ years of real-world experience across hundreds of Temporal implementations — production incidents, load tests, and hard-won debugging sessions. It is not a metrics glossary. It connects dots across related signals so you can build a complete picture of what is happening. + +## What This Skill Covers + +- **`temporal_request_failure`** and **`temporal_long_request_failure`** — the two most useful SDK metrics for troubleshooting Temporal applications +- How the SDK distinguishes the two metrics (based on `LongPollContextKey`) +- SDK gRPC retry behavior — which status codes are retried, which are not, and what that means for business impact +- Alert severity tiers — which failures warrant immediate alerting vs. passive monitoring +- Diagnostic tables mapping every operation + status code combination to what it means operationally +- Server-side rate limit buckets (Execution, Visibility, NamespaceReplicationInducing) and priority levels +- How to correlate SDK metrics with server dashboard panels +- Visual graph pattern interpretation for RPS failure graphs + +## How to Use the References + +Read the appropriate reference file based on what the user needs help with: + +### SDK Request Failure Metrics (the main guide) + +**Read `references/sdk-request-failures.md`** for: +- Understanding what `temporal_request_failure` or `temporal_long_request_failure` means +- Diagnosing a specific operation + status code combination +- Understanding alert severity and business impact +- Server-side throttling theory (rate limit buckets, priorities, dynamic config keys) +- Mapping SDK metrics to server dashboard panels +- Understanding gRPC retry behavior + +### Graph Pattern Reference + +**Read `references/graph-patterns.md`** for: +- Interpreting visual patterns in RPS failure graphs +- Identifying root causes from graph shapes (e.g., sustained plateau with periodic drops) +- Distinguishing between causes using correlated signals + +## Key Concepts + +- **`temporal_request_failure`** covers all standard client and worker operations (starts, signals, completions, heartbeats, visibility queries) +- **`temporal_long_request_failure`** covers long-poll operations (`PollWorkflowTaskQueue`, `PollActivityTaskQueue`, `PollWorkflowExecutionUpdate`, `GetWorkflowExecutionHistory` with `wait_new_event=true`) +- The server has **three independent rate limit buckets** — Execution, Visibility, and NamespaceReplicationInducing — each with its own RPS quota and priority queue +- Within each bucket, lower-priority operations are shed first under load (Priority 0 = highest, 5 = lowest) +- Not all `ResourceExhausted` errors are throttling — can also be gRPC message size violations or `BusyWorkflow` throttle + +## Companion Dashboards + +These dashboards are referenced throughout the knowledge base: +- https://github.com/tsurdilo/my-temporal-dockercompose +- https://github.com/tsurdilo/my-temporal-dockercompose/tree/main/deployment/grafana/dashboards diff --git a/GRAPH_PATTERNS.md b/references/graph-patterns.md similarity index 98% rename from GRAPH_PATTERNS.md rename to references/graph-patterns.md index 7fb6b73..3394c78 100644 --- a/GRAPH_PATTERNS.md +++ b/references/graph-patterns.md @@ -2,7 +2,7 @@ This is a living playbook for interpreting common visual patterns in `temporal_request_failure` and `temporal_long_request_failure` RPS graphs. Each entry maps a visual shape to its likely causes, business impact, and how to distinguish between root causes using correlated signals. -For background on the metrics, operations, and rate limit buckets referenced here, see the main [README](https://github.com/tsurdilo/temporal-metrics/blob/main/README.md). +For background on the metrics, operations, and rate limit buckets referenced here, see the main [SDK Request Failures guide](./sdk-request-failures.md). --- diff --git a/SDK_METRICS_REQUEST_FAILURES.md b/references/sdk-request-failures.md similarity index 99% rename from SDK_METRICS_REQUEST_FAILURES.md rename to references/sdk-request-failures.md index 7c9a72d..6af4ab8 100644 --- a/SDK_METRICS_REQUEST_FAILURES.md +++ b/references/sdk-request-failures.md @@ -772,7 +772,7 @@ All at Priority 1 within the visibility bucket. --- ## Graph Pattern Reference -For a visual pattern playbook — common shapes you'll see in `temporal_request_failure` and `temporal_long_request_failure` RPS graphs, what they mean, their business impact, and how to triage them — see the separate **[Graph Pattern Reference](./GRAPH_PATTERNS.md)**. +For a visual pattern playbook — common shapes you'll see in `temporal_request_failure` and `temporal_long_request_failure` RPS graphs, what they mean, their business impact, and how to triage them — see the separate **[Graph Pattern Reference](./graph-patterns.md)**. This is a living document that grows as new patterns are documented.