feat(open_feature): emit server-side EVP flagevaluation#5896
Conversation
- Aggregator spec: canonical_context_key type-differentiation, pruning 256/256 - Two-tier aggregation: full/degraded caps, drop-counted overflow - runtime_default from nil variant, first/last min/max - Degraded tier omits targeting_key/context (schema omitempty) - No Digest::MD5 assertion (reviewer concern #3)
…ations transport spec - aggregator.rb: two-tier full/degraded aggregation with canonical_context_key (sorted type-tagged length-delimited, not a hash digest; reviewer concern #3) - Context pruning: 256 fields / 256 chars before keying (reviewer concern #1) - Caps: globalCap=131072/perFlagCap=10000/degradedCap=32768 with drop-counted overflow (reviewer concern #8); eval-time min/max (reviewer concern #4) - Runtime default from nil variant (reviewer concern #5) - writer.rb: background Thread/SizedQueue(4096) flush every 10s to transport - transport.rb: FlagevaluationsSpec + build_flagevaluations + send_flag_evaluations posting to /evp_proxy/v2/api/v2/flagevaluations with EVP subdomain header
- hooks/flag_eval_evp_hook.rb: finally-stage hook, cheap capture + non-blocking enqueue only (reviewer concern #7); reads __dd_allocation_key + dd.eval.timestamp_ms - component.rb: create_flag_eval_evp_hook wired behind DD_FLAGGING_EVALUATION_COUNTS_ENABLED killswitch (default on); builds flagevaluations transport + FlagEvaluation::Writer - provider.rb: hooks() returns both OTel flag_eval_hook and EVP flag_eval_evp_hook - flag_eval_hook.rb + flag_eval_metrics.rb: byte-for-byte unchanged (PRES-01) - provider_spec.rb: updated to stub flag_eval_evp_hook; added killswitch test
…hook lifecycle gap) The Ruby openfeature-sdk (through at least 0.5.x) does not invoke provider hooks during flag evaluation — Client#fetch_details calls the provider directly and returns without running any hook lifecycle. FlagEvalEVPHook#finally was therefore never called, so no events reached the EVP writer queue and count=0 at mock-intake. Fix: call FlagEvalEVPHook#finally directly from Provider#evaluate after engine.fetch_value returns, using lightweight duck-typed wrappers (EvpEvalContext, HookContext, HookDetails) that satisfy the hook's interface without depending on SDK-provided hook context objects. The hook remains registered via Provider#hooks for forward compatibility with future SDK versions that do implement the full hook lifecycle. call_evp_hook is idempotent and best-effort: if the component is absent or the killswitch is off, it is a no-op; any error is logged at debug level and does not propagate into the evaluation hot path. EvaluationContext#attributes gap: ::OpenFeature::SDK::EvaluationContext exposes #fields (all context fields) and #targeting_key but NOT #attributes (which the hook reads). EvpEvalContext adapts fields -> attributes, excluding the targeting_key field to match the hook's expected shape.
…that silently killed EVP emission Two bugs in FlagEvaluation::Writer prevented the EVP flagevaluation component from ever loading, causing flag_eval_evp_hook to be nil and no events to reach mock-intake: 1. rescue inside until...end without begin (SyntaxError on Ruby 3.x): Ruby only allows `rescue` inside `until...end` if it is wrapped in `begin...end`. The file failed to parse on every require, raising SyntaxError which propagated through create_flag_eval_evp_hook's `rescue LoadError` guard (which does not catch ScriptError subclasses) and crashed the entire OpenFeature component. 2. Wrong require_relative depth for core/utils/time (LoadError if syntax were fixed): `require_relative '../../../core/utils/time'` from lib/datadog/open_feature/flagevaluation/writer.rb resolves to lib/core/utils/time (3 levels up = lib/), not lib/datadog/core/utils/time. The exposures worker uses the correct `../../core/utils/time` (2 levels up = lib/datadog/), so writer.rb is fixed to match. Both bugs together meant: SyntaxError on parse -> component init crash -> evp hook nil -> call_evp_hook guard returns immediately -> zero events queued -> zero EVP payloads. Add writer_spec.rb with a syntax-load regression guard and an enqueue+flush integration test (uses allow_any_instance_of to prevent background thread start in tests).
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 597ec11 | Docs | Datadog PR Page | Give us feedback! |
BenchmarksBenchmark execution time: 2026-06-27 20:58:58 Comparing candidate commit 240a3b8 in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.
|
|
👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description. If changes need to be present in CHANGELOG.md you can state it this way **Change log entry**
Yes. A brief summary to be placed into the CHANGELOG.md(possible answers Yes/Yep/Yeah) Or you can opt out like that **Change log entry**
None.(possible answers No/Nope/None) Visited at: 2026-06-24 02:05:59 UTC |
Typing analysisNote: Ignored files are excluded from the next sections. Untyped methodsThis PR introduces 32 partially typed methods, and clears 2 partially typed methods. It decreases the percentage of typed methods from 65.81% to 65.32% (-0.49%). Partially typed methods (+32-2)❌ Introduced:Untyped other declarationsThis PR introduces 2 partially typed other declarations. It increases the percentage of typed other declarations from 83.0% to 83.46% (+0.46%). Partially typed other declarations (+2-0)❌ Introduced:If you believe a method or an attribute is rightfully untyped or partially typed, you can add |
Resolve steep typecheck failures: - Declare Provider's EvpEvalContext/HookContext/HookDetails struct constants - Add EvaluationContext::TARGETING_KEY to the vendored openfeature-sdk stub - Add evaluation_counts_enabled to the _OpenFeature settings interface - Annotate writer build_event hash and ignore the unverifiable record splat
…2446-evp-flagevaluation-ruby
|
@codex review and find edge-cases, concurrency issues and bugs |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 03c721fc17
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@leoromanovsky Probably you know that, but to explain review delays - it's very hard to review massive PR's with lots of new functionality generated by AI. Recommendations: smaller context and smaller PR's, we don't have restriction on deliverable and that makes partial-PR's possible. |
…2446-evp-flagevaluation-ruby
@Strech Let's discuss that above privately. I made some improvements to the prefork and shutdown behavior that codex raised concerns about. The first fix prevents the background writer from spending an entire cycle draining an always-growing queue. Normal drain cycles are now bounded, so the writer can still reach its flush cadence under sustained traffic. Shutdown still uses an exhaustive final drain, and that remains protected by the existing 5-second shutdown timeout. The second fix snapshots the evaluation context attributes when the event is enqueued. Without this, nested hashes, arrays, or mutable strings could be changed by application code after the flag evaluation, and the background writer could emit the later mutated values instead of the values that existed at evaluation time. Ran them locally against new system tests (DataDog/system-tests#7226) that I will publish after this lands. To address the codex feedback, I added benchmarking test coverage, similar to the implementation in the Go tracer (https://github.com/DataDog/dd-trace-go/blob/main/openfeature/flageval_logging_scale_test.go). |
Strech
left a comment
There was a problem hiding this comment.
LGTM
P.S I'm not convinced tho about naming convention violation, that is not enforced.
TonyCTHsu
left a comment
There was a problem hiding this comment.
I need more time to review this
|
Thanks for the work here — and apologies my initial change-request was terse. To be concrete: I'm holding this because a few things don't meet our quality/maintainability bar yet. This isn't about timing or risk — it's that the Ruby guild reads and maintains this code long after merge, so readability/consistency are part of "done." Several of these were already raised by @Strech / Codex and are still open. Tests
Dead code
Typing
Naming
Conventions
None of this is about being a bottleneck — the path to merge is resolving these, and I'll re-review promptly once they're addressed. |
@TonyCTHsu I'll make these changes to advance this PR. We spoke privately about your team adding an |
TonyCTHsu
left a comment
There was a problem hiding this comment.
Thanks for working through all of this — the cleanup is a real improvement and I appreciate the turnaround.
Approving. One small non-blocking nits for whenever you next touch this — no need to gate the merge on them:
start_background_thread is still a one-line delegator to perform. It now serves as a clear named seam (init / after-fork / lazy-start, and the mock point in specs), so I'm happy to keep it — just noting it in case you'd intended to inline it.
Nice work getting this over the line.
|
Final e2e verification before merging, thanks for the help! |
Motivation
Customers need Ruby services to report the same server-side feature-flag evaluation signal as the other SDKs, and APM needs a backend-verifiable rollout path instead of SDK-specific local proof. This Ruby contribution adds bounded EVP
flagevaluationdelivery while preserving the existing OTel metric and exposure paths, which supports cross-SDK approval on smoothness and time-to-land.The Datadog Ruby tracer continues to support older Ruby runtimes for APM. Datadog Feature Flags through OpenFeature now requires
openfeature-sdk0.5.1or later, which makes Ruby 3.1 the practical runtime floor for this integration because the upstream OpenFeature Ruby SDK provider hook surface is not available on Ruby 2.5.Changes
FlagEvalEVPHookas a provider hook so the OpenFeature SDK invokes EVP capture with SDK-finalEvaluationDetailsafter hook failures and type validation.FlagEvalEVPHook/flag_eval_evp_hookso it is distinct fromFlagEvalMetricsHook.0.3.1to0.5.1for Ruby 3.1+ appraisal gemfiles.batchedflagevaluations.jsonschema fixture and avoids validating against duplicated backend schema in this repo.reasonout of the EVP enqueue path, writer, aggregation keys, payload shape, tests, and RBS declarations.error.message, preserves visible variant/allocation in degraded rows, and posts batches to/evp_proxy/v2/api/v2/flagevaluation.flagevaluationpayloads by encoded uncompressed JSON bytes under the 5 MiB EVP limit.targeting_keyandcontext; drops and logs only if the degraded row still cannot fit.reasonwriter, aggregator, and provider hook contract.Decisions
reasonis intentionally not part of EVP payloads or aggregate keys because the worker schema does not accept it.dd.eval.timestamp_msintoflag_metadataat evaluation entry; the EVP hook uses that timestamp later to preserve first/last-evaluation bounds after SDK finalization.openfeature-sdk0.5.1is the minimum OpenFeature dependency for this integration because it is the first line we are willing to support for the provider hook and telemetry surface.{context, flagEvaluations}wrapper.flowchart TD A[drained aggregated rows] --> B[serialize candidate batch as JSON] B --> C{batch <= 5 MiB?} C -- yes --> D[post asynchronously through EVP proxy] C -- no --> E{current row fits degraded?} E -- yes --> F[omit targeting_key and context] F --> B E -- no --> G[drop, log, count]Related PRs
Validation Evidence
Dogfooding App
ffe-dogfoodingapp-rubywas run with localdd-trace-rbartifacts and usedDatadog::OpenFeature::NativeEvaluator.ffe-dogfooding-string-flagthrough the Ruby dogfooding app with public-safe targeting keys:ruby-batch-evp-20260623T014728Z-alpharuby-batch-evp-20260623T014728Z-bravoruby-batch-evp-20260623T014728Z-charlievariant_2, and the backend rows below storevariant_2for the aggregated events.System Tests
binaries/dd-trace-rbtree:./run.sh +v FEATURE_FLAGGING_AND_EXPERIMENTATION --library ruby -k "Test_FFE_EVP_Flagevaluation"8 passed, 2630 deselected in 50.74son Ruby2.36.0-dev,rails72.202for/api/v2/flagevaluationpayloads.evp-count-flagproduced one aggregated bucket fortargeting_key=evp-count-userwithvariant.key=on,allocation.key=default-allocation, andevaluation_count=5.evp-burst-aggregation-flagproduced one aggregated bucket fortargeting_key=evp-burst-userwithvariant.key=on,allocation.key=default-allocation, andevaluation_count=512.evp-degradation-flagproduced a degraded aggregated bucket with notargeting_key,variant.key=on,allocation.key=default-allocation, andevaluation_count=2000.Staging End-To-End
--datacenter us1.staging.dog --customer-auth=skipagainst theflagevaluationtrack for the exact targeting keys above.flag.key=ffe-dogfooding-string-flag,variant.key=variant_2,allocation.key=allocation-override-392dd7c149f8, andevaluation_count=12:ruby-batch-evp-20260623T014728Z-alpha:first_evaluation=1782179248384,last_evaluation=1782179248487,timestamp=1782179256172,evaluation_count=12ruby-batch-evp-20260623T014728Z-bravo:first_evaluation=1782179248498,last_evaluation=1782179248595,timestamp=1782179256172,evaluation_count=12ruby-batch-evp-20260623T014728Z-charlie:first_evaluation=1782179248607,last_evaluation=1782179248705,timestamp=1782179256172,evaluation_count=12