Export BEAM + cgroup/OS host metrics for reconciliation#4678
Draft
alco wants to merge 11 commits into
Draft
Conversation
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
alco
force-pushed
the
alco--host-cgroup-metrics
branch
from
7248e1b to
9357ea7
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4678 +/- ##
==========================================
+ Coverage 60.03% 60.34% +0.31%
==========================================
Files 395 414 +19
Lines 43763 44613 +850
Branches 12587 12588 +1
==========================================
+ Hits 26271 26923 +652
- Misses 17411 17608 +197
- Partials 81 82 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Scaffold ElectricTelemetry.SystemMetrics, a module of poller-invoked measurement functions complementing ApplicationTelemetry's VM stats. This first piece delivers: - Cached platform/cgroup detection (system_info/0) stored in :persistent_term. Detects OS via :os.type() and cgroup version (v1/v2/ none) by stat-ing /sys/fs/cgroup. Later tasks (cgroup/host readers) build on this. - recon_alloc aggregate metrics (every 5s tick): vm.alloc.allocated, vm.alloc.used, vm.alloc.unused (allocated - used), and vm.alloc.carrier_usage (used/allocated ratio). - Per-allocator fragmentation (vm.alloc.fragmentation.unused, tagged by allocator), gated via a :persistent_term tick counter to run roughly once a minute since :recon_alloc.fragmentation/1 is O(carriers). Registers the measurements in ApplicationTelemetry and adds the corresponding Telemetry.Metrics definitions. Adds recon ~> 2.5 dep.
Bumping a :persistent_term tick counter on every 5s poll triggered a global process-heap GC scan per tick. Replace it with an atomic :counters ref created once and cached in :persistent_term (one put per gate key, ever), bumped via :counters.add/3 on the hot path. Same fire-every-12th-tick (~60s) semantics; :force/:gate_key affordances preserved.
Adds ElectricTelemetry.SystemMetrics.Cgroup, a defensive reader for cgroup
v1 and v2 accounting files, plus a thin cgroup_measurement/2 delegator in
SystemMetrics registered as a builtin periodic measurement.
Emits cgroup.* memory (current/anon/file/working_set/max), cpu
(usage_usec/nr_throttled/throttled_usec) and io (rbytes/wbytes) metrics,
plus PSI full avg10 for memory and cpu on v2. v2 reads /sys/fs/cgroup/*
directly; v1 reads per-controller subdirs. The cgroup version comes from the
cached system_info/0 detection; :none (incl. non-Linux) no-ops cleanly.
memory.max is skipped when unlimited ("max" on v2, the huge sentinel on v1);
working_set is skipped when inactive_file is missing; cpu times are converted
ns->us on v1 for a consistent microsecond unit. All reads are exhaustively
defensive: missing/malformed files skip the metric rather than crash. IO is
v2-only (v1 blkio is awkward and absent on our hosts). The cgroup root is
configurable via opts for fixture-based tests.
Add ElectricTelemetry.SystemMetrics.Proc, a defensive Linux /proc reader that emits host.mem.* (from /proc/meminfo) and host.proc.beam.* (from /proc/<beam_pid>/status and /proc/<beam_pid>/io). This bridges the cgroup plane and the BEAM allocator plane: per-process anon/file/shmem RSS shows cgroup anon tracking the BEAM footprint, and host meminfo gives the host-RAM ceiling/cache picture (one task per host on EC2, so host ~= container). Parsing of the flat /proc text files lives in a shared ElectricTelemetry.SystemMetrics.ProcfsParse helper. All reads are exhaustively defensive: missing files, EACCES on /proc/<pid>/io, missing keys, and malformed values skip the affected metric rather than crash. meminfo and status Rss*/VmRSS values (kB) are converted to bytes; io read_bytes/write_bytes are already bytes and left unscaled. The reader no-ops cleanly on non-Linux. It is wired in as a thin proc_measurement/2 delegator on SystemMetrics, registered in ApplicationTelemetry.builtin_periodic_measurements/1 (read every ~5s tick) with last_value metric defs in metrics/1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cgroup had byte-identical private copies of read_raw_file/1 and parse_int/1, now public on the shared ProcfsParse helper. Delegate the two format-agnostic primitives to ProcfsParse and delete the duplicates, leaving a single source of truth. The two stat parsers stay separate: ProcfsParse.parse_keyed_kv (colon-delimited "key: value unit") and Cgroup.read_stat_file (space-delimited "key value") parse different formats. Cgroup.parse_float/1 has no ProcfsParse equivalent and stays. Also tighten a ProcfsParse comment to describe the trim-then-space-split the code actually does. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend ElectricTelemetry.DiskUsage to optionally compute per-directory
subtotals during the existing periodic disk walk (a single traversal,
no second pass and no new timer).
Disk.recursive_usage_grouped/3 returns {total, %{dir_name => bytes}}
where directories are bucketed by name at a configurable depth. The
returned total is byte-identical to recursive_usage/2, including the
legacy reset-to-0 behaviour on an unreadable directory; the bucket map
is a best-effort tally that only grows on regular files.
DiskUsage gains :group_depth (default nil = no bucketing, preserving
current behaviour) and :top_n (default 10) options, caches the top-N
largest buckets in ETS, and exposes them via current_dirs/1.
StackTelemetry wires group_depth: 4 (the electric shape storage layout
<storage_dir>/<stack_id>/<p1>/<p2>/<shape_handle>) and defines the new
electric.storage.dir.bytes metric tagged [:stack_id, :shape].
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
report_disk_usage/2 now also reads DiskUsage.current_dirs/1 and emits electric.storage.dir.bytes (tagged by shape handle) for the top-N largest shapes, reusing the existing periodic disk-usage walk and its emission cadence. Cardinality is bounded to the top-N buckets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shape-handle directory depth relative to the disk-usage walk root differs by deployment (standalone walks the bare storage dir with shapes nested under shapes/; cloud walks the per-tenant shapes dir directly), so a hard-coded constant in the generic telemetry package would be wrong for one of them. StackTelemetry now reads :shape_dir_group_depth from its opts (supplied by the electric-side caller) instead of hard-coding 4, and passes nil through to disable grouping for backends without an on-disk shape layout. Also document the transient full-map + O(n log n) sort in top_n/2 as a known tradeoff that is fine at ~10k shapes per stack, and use a realistic two-level-sharded fixture layout in the tests with a regression guard asserting a too-shallow depth buckets by the hash-shard prefix instead of the shape handle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes an off-by-one in the per-shape electric.storage.dir.bytes metric: DiskUsage walks the bare config.storage_dir, but shapes are stored one level deeper under shapes/<stack_id>/<p1>/<p2>/<shape_handle> in the standalone deployment, so the previous fixed depth of 4 bucketed by the 2-char hash-shard prefix (aggregating many shapes) instead of by shape handle. Compute the depth from the path that PureFileStorage.shape_data_dir/3 actually produces for a sample handle, measured relative to the walk root. This yields 5 for standalone and 4 for the cloud per-tenant layout (which walks <root>/<tenant_id>/shapes directly), and stays correct if the sharding scheme changes. Returns nil to disable grouping for non-file storage backends. Cross-stack attribution is a non-issue: each stack's walk root is already its own subtree (per-tenant shapes dir in cloud; single stack standalone), so a stack only ever buckets its own shapes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
recon_alloc's used/allocated ratio can momentarily sample slightly above 1.0 (observed ~1.003) because the underlying mbcs/sbcs stats are read non-atomically. Use a tolerant sanity ceiling (1.5) instead of a hard 1.0 so the test isn't flaky. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the otel-export.lux test to assert that the new SystemMetrics families reach the collector end-to-end: vm.alloc.* (allocator carriers), host.mem.* / host.proc.beam.* (/proc), and cgroup.memory.current. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
alco
force-pushed
the
alco--host-cgroup-metrics
branch
from
2a943f8 to
1e1c98e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
In-app telemetry poller that exports a complete, reconcilable memory/CPU/disk picture for Electric — the BEAM plane (what the VM thinks it uses) alongside the cgroup/OS plane (what the kernel charges), in one event stream with identical resource attributes.
What's included
New
ElectricTelemetry.SystemMetricspoller (+Cgroup,Proc,ProcfsParsesubmodules) reading/procand/sys/fs/cgroupdirectly and emitting OTel gauges:vm.alloc.{allocated,used,unused,carrier_usage}(5s), plus gated per-allocatorvm.alloc.fragmentation.unused{allocator}(~60s, O(carriers), gated via:counters).allocated − usedis the carrier-fragmentation / OOM-risk signal.cgroup.memory.*,cgroup.cpu.*,cgroup.io.*, and PSI*.pressure.full.avg10(v2). Version detected once at boot from cachedsystem_info/0; stable metric names regardless of host cgroup version.host.mem.*andhost.proc.beam.*(incl.io.{read,write}_bytes) to bridge BEAM ↔ cgroup accounting.electric.storage.dir.bytes{stack_id,shape}(top-N), by extending the existingDiskUsage60s walk (single traversal, no second walk, no hot-path counters). Group depth is derived fromPureFileStorage.shape_data_dir/3, so it's correct for both standalone (depth 5) and cloud (depth 4) layouts. Namedelectric.storage.*to sit next toelectric.storage.used.bytesinStackTelemetry.Not included (deferred follow-ups)
app.fs.capacity/used/avail(statvfs filesystem-capacity gauges)Why it matters
Lets us answer, from one Honeycomb query, whether memory growth is real (BEAM heap/binaries), reclaimable (page cache), or fragmentation (carriers held from the OS) — and the equivalent for CPU saturation and disk fill. Without the cgroup/OS plane next to the BEAM plane, there's no way to tell a genuine leak from reclaimable page cache, or to attribute CPU to real scheduler work vs. kernel accounting.
🤖 Generated with Claude Code