Skip to content

Epic: complete self-host observability — host/container/Redis/Qdrant/stack self-monitoring + remaining exporter gaps #5365

Description

@JSONbored

Problem: Following the Reviews & PRs dashboard / AI-usage dashboard consolidation audit (#5234, #5247, #5259, #5267, #5302, #5310, #5342, #5349), a full sweep of every integrated backend found several pieces of the self-host stack have zero Prometheus/Grafana visibility today:

  • No host-level metrics (CPU/memory/disk/network of the machine itself) — docker stats-only, invisible in Grafana.
  • No per-container resource metrics (which container is actually using the CPU/memory quota it's been given).
  • Redis — always-on, required — has zero metrics despite every other core backend (Postgres, GPU, backups) already having one.
  • Qdrant already exposes a native Prometheus /metrics endpoint (:6333/metrics, confirmed live) that is simply never scraped — a free win.
  • Prometheus, Alertmanager, Loki, Tempo, and Grafana all expose their own native /metrics (confirmed live against edge-nl-01) but none are self-scraped — no visibility into whether the observability stack itself is healthy, ingesting, or falling behind.
  • The OTEL collector's own internal telemetry port (:8888) isn't enabled/scraped — no visibility into dropped spans/metrics or collector health.
  • REES (review-enrichment/, first-party TypeScript) has no /metrics endpoint at all (confirmed: GET /metrics → 404) — analyzer success/failure/duration/timeout rates are invisible.
  • browserless (visual-review screenshot capture) exposes a /metrics endpoint, but it returns JSON, not Prometheus text format (confirmed live) — not scrapeable as-is.
  • Ollama has no native Prometheus endpoint and no official community exporter exists (confirmed: GET /metrics → 404) — this stays a documented, unactionable limitation, not a sub-issue below.
  • Sentry (sentry.io) is currently only reachable via a plain hyperlink from the Grafana resource-hub dashboard — no in-Grafana visualization of error/issue data (distinct from Epic: consolidate Sentry + Alertmanager alerting into one real-time notification path #5007, which is about alerting/notification routing, not visualization).

Area: Self-host observability (docker-compose.yml, prometheus/, grafana/dashboards/, review-enrichment/)

Goal (verbatim from the requesting conversation): "The whole stack should be able to be properly observed/logged/metrics and analysis directly from grafana, so observability is built in and complete from end to end, ideally."

Deliverables — tracked as independently-shippable sub-issues:

Acceptance criteria (epic-level):

  • Every backend this stack runs (app, Postgres, Redis, Qdrant, GPU, backups, REES, browserless, Ollama-documented-exception) has either real Prometheus metrics in Grafana or an explicit, documented reason it can't.
  • The observability stack itself (Prometheus/Alertmanager/Loki/Tempo/Grafana/OTEL collector) is self-monitored, not just monitoring everything else.
  • A self-hoster can answer "is any part of my stack using resources sub-optimally or behaving abnormally" from Grafana alone, without shelling into the host.

Boundaries:

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.roadmapOn the Wave-2 agent-layer roadmap board (project 9)

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions