feat(openfeature): emit server-side EVP flagevaluation

[leoromanovsky]

Motivation

Server-side feature-flag evaluations need to be observable in the same backend track as the rest of the cross-SDK rollout so APM can judge rollout smoothness and time-to-land from real evaluation counts, not only app-local behavior or metric-side approximations. This contribution gives Python services backend-visible flagevaluation telemetry with bounded cardinality and batching, so customers and internal dogfooding can verify which flags evaluated, for which targeting keys, and how often, while preserving the existing feature_flag.evaluations metric path.

Changes

• Adds a Python EVP flagevaluation writer behind DD_FLAGGING_EVALUATION_COUNTS_ENABLED; existing OTel feature_flag.evaluations metrics remain unchanged.
• Registers an OpenFeature finally_after logging hook that captures a cheap evaluation snapshot and hands it to a background writer.
• Renames the capture hook to FlagEvalEVPHook / _flag_eval_evp_hook.py so it is distinct from FlagEvalMetricsHook.
• Posts batched payloads to the Agent EVP proxy route /evp_proxy/v2/api/v2/flagevaluation with X-Datadog-EVP-Subdomain: event-platform-intake.
• Aggregates evaluations by schema-visible dimensions: flag key, variant key, allocation key, runtime-default state, error message, targeting key, and pruned evaluation context.
• Flattens and prunes evaluation context before enqueue with deterministic 256-field / 256-character bounds.
• Splits encoded flagEvaluations request bodies under the shared EVP payload limit before POSTing.
• Degrades a single oversized full-tier row by dropping targeting_key and context; if the degraded row still cannot fit, the writer drops and logs it.
• Uses evaluation-time metadata to populate first_evaluation and last_evaluation rather than flush time.
• Keeps accepting the legacy DataDogProvider(initialization_timeout=...) keyword for compatibility, while async initialization no longer uses it.

Evaluation Capture And Flush

flowchart TD
  Eval["OpenFeature evaluation"] --> Metrics["FlagEvalMetricsHook\nexisting OTel metric"]
  Eval --> Logging["FlagEvalEVPHook\nfinally_after"]
  Logging --> Snapshot["Cheap scalar snapshot\nflag, variant, allocation,\ntargeting_key, context, eval time"]
  Snapshot --> Queue["Bounded non-blocking queue\nQueue size 4096"]
  Queue --> Drain["Background drain worker\n0.1s queue drain"]
  Drain --> Aggregate["FlagEvaluationWriter aggregation\nfull tier then degraded tier"]
  Aggregate --> Flush["Periodic flush\n10s interval"]
  Flush --> EVP["Agent EVP proxy\n/evp_proxy/v2/api/v2/flagevaluation"]

Loading

Payload Limit Handling

flowchart TD
  Events["Aggregated flagEvaluations"] --> Encode["Encode each event with compact JSON"]
  Encode --> FitsEvent{"Single event fits\ninside shared EVP limit?"}
  FitsEvent -- "yes" --> Batch["Add to current payload"]
  FitsEvent -- "no" --> Degrade["Drop targeting_key and context\nfor that event only"]
  Degrade --> FitsDegraded{"Degraded event fits?"}
  FitsDegraded -- "yes" --> Batch
  FitsDegraded -- "no" --> Drop["Drop and count\npayload_limit"]
  Batch --> FitsPayload{"Payload still <=\nDEFAULT_EVP_PAYLOAD_SIZE_LIMIT?"}
  FitsPayload -- "yes" --> Continue["Continue batching"]
  FitsPayload -- "no" --> Split["Close current payload\nstart next payload"]
  Split --> Continue
  Continue --> Post["POST each payload sequentially"]

Loading

Decisions

• Keep this as a dual-path implementation: the EVP logging writer is additive and separately gated, while the existing OTel metric hook remains registered when the provider is enabled.
• Use the existing Agent EVP proxy transport and the canonical flagevaluation backend route instead of adding a Python-specific transport or schema.
• Keep the evaluation logging hook non-blocking on application evaluation threads: queueing is best-effort, bounded, and drop-counted under backpressure.
• Do aggregation, JSON serialization, payload splitting, and POSTs only in the background writer path.
• Split flagEvaluations payloads by encoded byte size using DEFAULT_EVP_PAYLOAD_SIZE_LIMIT; send split payloads sequentially from the writer thread.
• Use a two-tier aggregation model: full buckets retain targeting_key and pruned context; degraded buckets retain only lower-cardinality schema dimensions.
• Store targeting_key in the dedicated EVP field and remove targetingKey / targeting_key from context before building context keys or payloads.
• Do not emit OpenFeature reason because it is not accepted by the canonical worker schema, and do not populate targeting_rule.key until Python exposes real targeting-rule metadata.
• Keep openfeature-sdk as an application/test/lint dependency rather than adding it as a runtime dependency of ddtrace; the dogfooding app dependency floor is being tightened separately in https://github.com/DataDog/ffe-dogfooding/pull/88.

Validation Evidence

Dogfooding App

• ffe-dogfooding app-python was run with local dd-trace-py artifacts and reached PROVIDER_READY.
• The running dogfooding app reported openfeature-sdk=0.9.0.
• Evaluated ffe-dogfooding-string-flag through the Python dogfooding app with 5 evaluations per targeting key over one persistent connection:
  • python-batch-keepalive-20260623T004034Z-alpha
  • python-batch-keepalive-20260623T004034Z-bravo
  • python-batch-keepalive-20260623T004034Z-charlie
• App-side result: all evaluations returned variant_2 with allocation allocation-override-392dd7c149f8.

System Tests

• Companion draft PR: Enable EVP flagevaluation system tests for Python system-tests#7183

Staging End-To-End

• Dogfooding ran without the local mock-intake EVP tee/proxy, so the Agent sent EVP traffic through the normal backend path.
• Retriever staging query returned 3 backend flagevaluation rows for the exact targeting keys above.
• Each targeting key returned one row with evaluation_count=5:
  • python-batch-keepalive-20260623T004034Z-alpha
  • python-batch-keepalive-20260623T004034Z-bravo
  • python-batch-keepalive-20260623T004034Z-charlie
• All rows had flag.key=ffe-dogfooding-string-flag, variant.key=variant_2, and allocation.key=allocation-override-392dd7c149f8.

[datadog-datadog-prod-us1-2]

 

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 9 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | build linux serverless: [amd64, cp315-cp315, v113741238-d2b8243-manylinux2014_x86_64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp315-cp315, v113741589-d2b8243-musllinux_1_2_aarch64, 1]     

DataDog/apm-reliability/dd-trace-py | build linux: [amd64, cp315-cp315, v113741491-d2b8243-musllinux_1_2_x86_64]     

View all 9 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 2 jobs - 2 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e6c25dc | Docs | Datadog PR Page | Give us feedback!}

[cit-pr-commenter-54b7da]

Codeowners resolved as

tests/openfeature/test_provider_status.py                               @DataDog/feature-flagging-and-experimentation-sdk

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-07-06 04:56:15

Comparing candidate commit e6c25dc in PR branch leo.romanovsky/ffl-2446-evp-flagevaluation-python with baseline commit c12bb9d in branch main.

Found 0 performance improvements and 5 performance regressions! Performance is the same for 591 metrics, 10 unstable metrics.

scenario:iast_aspects-re_expand_aspect

• 🟥 execution_time [+289.127µs; +341.719µs] or [+8.055%; +9.521%]

scenario:iast_aspects-re_search_aspect

• 🟥 execution_time [+21.266µs; +29.215µs] or [+7.336%; +10.078%]

scenario:iastaspects-add_aspect

• 🟥 execution_time [+8.228µs; +10.975µs] or [+7.820%; +10.432%]

scenario:iastaspectsospath-ospathbasename_aspect

• 🟥 execution_time [+91.512µs; +102.000µs] or [+21.474%; +23.935%]

scenario:span-start

• 🟥 execution_time [+1.253ms; +1.403ms] or [+8.267%; +9.258%]

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 4974 circular imports that already exist on the base branch and have not been changed by this PR.

Show first 5 (shortest cycles)

ddtrace.internal.datastreams -> ddtrace.internal.datastreams.google_cloud_pubsub -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.botocore -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.aiokafka -> ddtrace.internal.datastreams
ddtrace.internal.datastreams -> ddtrace.internal.datastreams.kombu -> ddtrace.internal.datastreams
ddtrace.internal.core -> ddtrace._trace.span -> ddtrace.internal.core

To see all existing cycles, download the cycles-base.json and cycles-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/cycles.py compare cycles-base.json cycles-pr.json