runtime status

Runtime Status

You will learn what the admin heartbeat shows, what is available through the admin status API, and how to compare snapshots to find resource waste.

Heartbeat strip

The top heartbeat strip answers "is anything on fire?" without opening a detail view.

The first row shows:

• Daemon — RUNNING while an engine run is active, otherwise Idle.
• Healthy, delayed, risky, unavailable, archived, empty, unmaintained — feed-health counts. These tiles are clickable and filter the feed table.

The second row shows compact runtime vitals:

• Uptime — daemon uptime and configured feed count.
• Config — successful reload count or the latest reload error.
• Heap — Go heap allocation and total Go runtime memory.
• Goroutines — current goroutine count and GC count.
• Disk free — free space on the runtime data volume.
• Integrity — current pipeline-integrity finding count, with startup repair status when repairs were deferred.

The four live queues are shown in the current-run panel below the heartbeat. See Live Queues.

Admin status API

The admin UI polls GET /api/v1/admin/status for high-frequency heartbeat updates; mode=light is an equivalent explicit spelling. That response is intentionally compact so the page keeps responding while heavy engine work is running. Full GET /api/v1/admin/status?mode=full responses include more fields than the heartbeat renders and are intended for lower-frequency diagnostics.

Important top-level blocks:

Block	Meaning
system	Go runtime, process, disk, CPU, I/O, and file-descriptor snapshots.
engine	Engine running state, current phase, engine-lane active/waiting work, integrity-cache summaries, background tasks, config reload state, and lifetime counters/timings.
scheduler	Scheduler item snapshot: enabled state, next due time, last check, frequency, failures, and scheduler detail per item.
queues	Waiting, active, deferred, and pending queue items.
metrics	Scheduler counters and latest batch timings.
feeds	Complete feed health and visibility summary used by the heartbeat, including enabled/disabled counts, health buckets, hidden feeds, entry totals, and unique-IP totals.
artifacts	Artifact parent state shown by the artifact inventory.

Telemetry counters and timings

The daemon exposes monotonic operation counters that increment over the process lifetime:

• Download counts — scheduler download counters under metrics, plus downloader status counters in engine.lifetime_metrics.counters.
• Processing counts — scheduler processing counters under metrics, engine phases under engine.current_metrics and engine.last_metrics, and operation timings in engine.lifetime_metrics.operations.
• Pairwise comparison — candidate pairs checked, overlap computations, skip reasons.
• iprange operations — loads, saves, merges, compares, diffs, searches (text and binary).
• Entity work — sidecar reads/writes, country/ASN patch counts, range attribution operations.
• Admin polling — request counts, response bytes, build timings for admin API endpoints.

These counters are cumulative. To see what happened in the last hour, sample the status endpoint twice and compute the delta.

Process resources

The heartbeat shows the compact resource fields operators need most often: heap, goroutines, disk free, and integrity. The admin status API also exposes process-level resource snapshots under system:

• CPU — user, system, and total seconds since start.
• Memory — Go heap counters from the application, plus host monitoring for RSS and I/O pressure.
• File descriptors — use host monitoring for open FD count.
• I/O — use host monitoring for process read/write bytes and syscall counters.

Snapshot-diff workflow

To identify what consumed resources between two points in time:

1. Record the status endpoint output at time A.
2. Record the status endpoint output at time B.
3. Compare counter deltas against elapsed time and CPU deltas.

This tells you which operations moved the most. For example, if iprange.compare.ops jumped by 50,000 in five minutes, pairwise comparisons dominated that period.

The counters use stable names that work across daemon restarts and OpenTelemetry export.

Engine lane and integrity cache

The engine.engine_lane block shows broad engine-owned work that does not belong to the four feed queue lists. It includes the lane limit, active count, waiting count, and the active/waiting work labels shown in Background Work.

The engine.pipeline_integrity_cache and engine.entity_integrity_cache blocks summarize the last settled integrity state. They expose cache state, queued/running flags, timestamps, and finding counts. The status endpoint does not embed full integrity findings; use the integrity panels or integrity API endpoints for the finding rows.

See also

• Live Queues — the four queue panels below the status strip
• Background Work — deferred work not shown in the queues
• Telemetry Reference — field and metric names