RFC: pup vet — Observability health checks

[jakedgy]

RFC: pup vet — Observability health checks

Moved from #129

---

The Problem

Observability configurations degrade silently. Monitors lose their notification channels. Traces break across service boundaries. Logs stop correlating with APM. These aren't matters of preference or org-specific style — they're genuinely broken configurations that are invisible until something goes wrong.

No one finds these without a dedicated audit, and no one does a dedicated audit until it's too late.

The Idea

pup vet — a health check for your Datadog setup. Surfaces things that are universally broken or misconfigured, without assuming how your org has chosen to structure Datadog.

The name fits: take your pup to the vet for a checkup.

$ pup vet

CRITICAL: silent-monitors (2 found)
  Monitors with no notification channels
  - #12345 "High CPU on web-api"
  - #67890 "Disk usage > 90%"
  -> Add notification channels so alerts reach on-call responders

WARNING: muted-forgotten (1 found)
  Monitors muted for >30 days
  - #11111 "Memory pressure on auth-service" (muted 45 days)
  -> Review and unmute, or delete if no longer needed

PASSED: stale-monitors, untagged-monitors, no-recovery-threshold

Summary: 1 critical, 1 warning, 3 passed

Design Principles

Universally useful, not opinionated. Every check should surface something that's broken regardless of how an org uses Datadog. Silent monitors are broken everywhere. Broken trace correlation is broken everywhere. We don't tell you how to tag or organize — we find things that aren't working as intended.

Agent-first, human-readable. In agent mode, output is structured JSON with severity, affected resources, and actionable recommendations. In human mode, a clear terminal summary. An AI agent can run pup vet as a starting point for any observability task.

Scoped or full. pup vet audits everything. pup vet --service=web-api or pup vet --tags=team:platform narrows the scope.

Initial Checks

Starting with things that are universally broken — no assumptions about org structure:

Check	What it catches
silent-monitors	Monitors with no notification channels — alerts fire into the void
stale-monitors	Monitors stuck in "No Data" for extended periods — abandoned or misconfigured
untagged-monitors	Monitors without any tags — can't be filtered, routed, or grouped
muted-forgotten	Monitors muted for 30+ days — meant to be temporary, now permanent
no-recovery-threshold	Monitors without recovery thresholds — flapping risk

Natural expansions as we learn what's valuable: broken trace/log correlation, USM gaps, SLOs without error budget alerts, services without any monitoring.

Where This Goes

pup vet is the foundation. Once you can audit what exists and what's broken, the next natural question during an incident becomes: "what's actually happening right now across all my data?" That's a cross-domain investigation capability that builds on top of vet — but it's a separate proposal, and one that needs more careful design to avoid being org-specific.

Design Detail

A more detailed design doc (shared types, output schemas, phasing) is available at docs/plans/2026-02-27-vet-investigate-design.md.

Open Questions

1. What universally broken configurations do you encounter most in Datadog?
2. What checks would you run first if pup vet existed today?
3. Are there categories of misconfiguration that are hard for humans to spot but easy to detect programmatically?

---

This design was developed collaboratively with Claude Code.

[platinummonkey]

this is a great idea!

Some other interesting thoughts popped in my mind about this:

• Doing it a development time as well and having the agent check for coverage on the new feature/change. This would need some sort of static analysis but look for what new metric/log/trace needs to be added to ensure you have a higher chance of release without issue.
• Automatic chore cleanup tasks and cleaning up monitoring sprawl
• pup vet --check=on-call-health how's a team's on-call burden?
• Unused telemetry - clean it up or add a monitor on it?

[jakedgy]

Really like these — especially the on-call health angle. A monitor that triggers and recovers 47 times a month isn't a matter of opinion, it's just burning out whoever's on call. That clears the "universally useful" bar easily.

The sprawl/cleanup idea is interesting because it's the flip side of vet — vet finds things that are broken or missing, cleanup finds things that shouldn't be there anymore. Monitors that haven't changed state in 6 months, duplicates with near-identical queries, etc.

Re: dev-time coverage — I'm actually not sure static analysis is the right approach there. If an agent is reviewing the code, it already understands the change. What it doesn't have is the observability picture. So pup vet --service=web-api gives the agent that context — here's what monitoring exists, here are the gaps — and the agent connects the dots. Pup doesn't need to understand code, it just needs to clearly surface what exists and what's missing.

And I like the framing on unused telemetry — not just "delete this waste" but "you're collecting this for a reason, are you actually using it?"

Are there specific ones you'd want to run first if vet existed today?

[platinummonkey]

prototyped something on #142 but it also depends on DataDog/datadog-api-client-rust#1292

[jakedgy]

I'll try to take a look at this today. I'm really excited about the potential for this. I've been using pup to audit and improve our monitoring. Clade Code makes it easy to knock out the annoyances and gaps that tend to just pile up, especially in a smaller organization without dedicated o11y resources.

[esummers-DD]

Posting an update following a discussion between @platinummonkey and the teams involved here at Datadog:

We've agreed that we're not planning to implement/ship pup vet as part of the pup CLI. We want to keep pup focused and avoid positioning it as the home for functionality that's better delivered through more dedicated platform surfaces.

That said, we do intend to carry forward elements of this proposal. You should expect parts of the approach to be integrated into other Datadog features and/or APIs where they can be maintained and supported responsibly.

We'll use this discussion as a place to collect feedback and requirements so they're easy to track and route to the right owners.