Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions apps/price_pusher/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ pnpm run dev evm --config config.evm.mainnet.json --metrics-port 9091

### Running Locally with Docker

You can run the monitoring stack (Prometheus and Grafana) using the provided docker-compose configuration:
You can run the monitoring stack (Prometheus, Loki, Promtail, and Grafana) using the provided docker-compose configuration:

1. Use the sample docker-compose file for metrics:

Expand All @@ -303,9 +303,11 @@ docker-compose -f docker-compose.metrics.sample.yaml up

This will start:
- Prometheus server on port 9090 with the alerts configured in alerts.sample.yml
- Loki server on port 3100 (log aggregation)
- Promtail, configured by promtail.sample.yml to ship every Docker container's logs into Loki and tag them with `namespace=<container_name>`
- Grafana server on port 3000 with default credentials (admin/admin)

The docker-compose.metrics.sample.yaml file includes a pre-configured Grafana dashboard (see the [Dashboard](#dashboard) section below) that displays all the metrics mentioned above. This dashboard provides monitoring of your price pusher operations with panels for configured feeds, active feeds, wallet balance, update statistics, and error tracking. The dashboard is automatically provisioned when you start the stack with docker-compose.
The docker-compose.metrics.sample.yaml file includes a pre-configured Grafana dashboard (see the [Dashboard](#dashboard) section below) that displays all the metrics mentioned above plus log panels (Tx Hash, All Logs, Error Logs) backed by Loki. The dashboard is automatically provisioned when you start the stack with docker-compose; the `chain` template variable lists every container name Promtail has seen, so running one pusher container per chain gives you per-chain log views out of the box.

### Example Grafana Queries

Expand Down
12 changes: 12 additions & 0 deletions apps/price_pusher/datasource.sample.yml
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Prometheus metrics lack namespace label but dashboard queries filter on it

The Grafana dashboard (pre-existing, not changed in this PR) filters all Prometheus queries with namespace="$chain", e.g. pyth_price_feeds_total{namespace="$chain"}. However, the price pusher's metrics code (src/metrics.ts:35-89) does not include a namespace label on any metric, and prometheus.sample.yml doesn't add one via relabeling. This means in the sample docker-compose setup, Prometheus metric panels will show no data when a specific chain is selected. This is a pre-existing issue — likely the dashboard was designed for a Kubernetes environment where Prometheus adds namespace labels from service discovery. The chain template variable is now sourced from Loki's namespace label (container names), which further highlights the disconnect. Users following the sample setup should be aware that metric panels may not populate without adding a matching namespace label to their Prometheus scrape config.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
@@ -1,8 +1,20 @@
apiVersion: 1

datasources:
# Both UIDs below are referenced from grafana-dashboard.sample.json (the
# Prometheus UID by every metric panel, the Loki UID by Tx Hash / All Logs
# / Error Logs and the `chain` template variable). Without explicit UIDs
# Grafana auto-generates per-provision random values and the dashboard's
# hardcoded references fail to resolve. Keep them in sync if the dashboard
# is regenerated.
- name: Prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
uid: edryyydtht14wa
isDefault: true
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.
- name: Loki
type: loki
access: proxy
url: http://loki:3100
uid: ads9ouz3jh4hsa
23 changes: 23 additions & 0 deletions apps/price_pusher/docker-compose.metrics.sample.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,28 @@ services:
networks:
- monitoring

loki:
image: grafana/loki:latest
container_name: loki
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- monitoring

promtail:
image: grafana/promtail:latest
container_name: promtail
volumes:
- ./promtail.sample.yml:/etc/promtail/config.yml
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
command: -config.file=/etc/promtail/config.yml
depends_on:
- loki
networks:
- monitoring

grafana:
image: grafana/grafana:latest
container_name: grafana
Expand All @@ -33,6 +55,7 @@ services:
- GF_USERS_ALLOW_SIGN_UP=false
depends_on:
- prometheus
- loki
networks:
- monitoring

Expand Down
47 changes: 47 additions & 0 deletions apps/price_pusher/promtail.sample.yml
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚩 Dashboard log queries use | logfmt on JSON-formatted pino output

The pre-existing grafana-dashboard.sample.json uses | logfmt as the first parser in several Loki queries (e.g. the All Logs panel at line 1148: {namespace=~"$chain"} | logfmt, and the Tx Hash panel at line 1098 which chains | logfmt | json). Since pino outputs JSON (not logfmt), the logfmt parser will fail to extract any fields. In the Tx Hash panel the subsequent | json stage recovers by parsing the original JSON line, so it still works — but the logfmt step is redundant. In the All Logs panel, | logfmt alone means no structured fields are extracted, though log lines still display. This is a pre-existing issue in the dashboard (not introduced by this PR), but since this PR adds the infrastructure (Loki/Promtail) that makes these panels actually functional for the first time, it may be worth addressing the dashboard queries in a follow-up.

Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Sample Promtail config that ships Docker container logs to the local Loki
# instance defined in docker-compose.metrics.sample.yaml. It tags every line
# with `namespace=<container_name>` so the `chain` template variable in
# grafana-dashboard.sample.json (which filters `{namespace=~"$chain"}`) picks
# up one entry per pusher container you run alongside this stack.
#
# Run one pusher container per chain (e.g. `price-pusher-evm`,
# `price-pusher-sui`) and the dashboard's chain selector will list them.

server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:
- url: http://loki:3100/loki/api/v1/push

scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
# Use the container name (without the leading slash Docker prepends) as
# the `namespace` label that the Grafana dashboard filters on.
- source_labels: ["__meta_docker_container_name"]
regex: "/(.*)"
target_label: namespace
- source_labels: ["__meta_docker_container_id"]
target_label: container_id
pipeline_stages:
# Surface pino's standard `level` field so the Error Logs panel
# (`detected_level = error`) can detect log levels emitted as JSON.
# Pino emits numeric severities by default (10=trace, 20=debug,
# 30=info, 40=warn, 50=error, 60=fatal) and Loki's detected_level
# only recognises the string forms, so translate before labelling.
- json:
expressions:
level: level
msg: msg
- template:
source: level
template: '{{ if eq .Value "10" }}trace{{ else if eq .Value "20" }}debug{{ else if eq .Value "30" }}info{{ else if eq .Value "40" }}warn{{ else if eq .Value "50" }}error{{ else if eq .Value "60" }}fatal{{ else }}{{ .Value }}{{ end }}'
- labels:
level:
Comment thread
devin-ai-integration[bot] marked this conversation as resolved.