Enable EVP flagevaluation system tests for Ruby#7184
Conversation
|
|
🎉 All green!🧪 All tests passed 🔗 Commit SHA: 684efe6 | Docs | Datadog PR Page | Give us feedback! |
…p-flagevaluation-enable-ruby-clean # Conflicts: # tests/ffe/test_flag_eval_evp.py
…p-flagevaluation-enable-ruby-clean
…p-flagevaluation-enable-ruby-clean
leoromanovsky
requested review from
dd-oleksii and
sameerank
and removed request for
a team
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 684efe67c3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| anchor_response = evaluate_flag(anchor_flag_key, targeting_key="evp-degradation-window-anchor") | ||
| assert anchor_response.status_code == 200, f"Window anchor request failed: {anchor_response.text}" | ||
| wait_for_evp_flagevaluation_event(anchor_flag_key) |
There was a problem hiding this comment.
In the FEATURE_FLAGGING_AND_EXPERIMENTATION end-to-end scenario, this setup can now fail before pytest reaches the matching test: the response assertion and wait_for_evp_flagevaluation_event() both run inside setup_, and that helper asserts after 30s. .cursor/rules/pr-review.mdc says setup_* methods must not fail because a setup failure aborts the whole scenario; if the anchor request returns non-200 or the agent payload is delayed, unrelated FFE tests in the scenario will not run. Store the anchor result and assert/wait from test_ffe_evp_flagevaluation_degradation instead.
Useful? React with 👍 / 👎.
| ) | ||
| ] | ||
| if batch_start + EVP_DEGRADATION_BATCH_SIZE < self.eval_count: | ||
| time.sleep(EVP_DEGRADATION_BATCH_PAUSE_SECONDS) |
There was a problem hiding this comment.
This fixed sleep runs in setup_ffe_evp_flagevaluation_degradation between every batch in an end-to-end scenario. .cursor/rules/pr-review.mdc explicitly forbids time.sleep() or explicit waits inside setup_ methods because setup is only for trace generation and timing should be handled by scenario/interface timeouts; this adds fixed delay to the scenario and makes the result depend on pacing in setup. Please move the waiting/pacing into validation or scenario configuration.
Useful? React with 👍 / 👎.
4 participants
Motivation
Ruby's SDK support for server-side EVP
flagevaluationis merged in DataDog/dd-trace-rb#5896. This PR enables the shared system-tests contract for the Rubyrails72weblog so reviewers can validate that Ruby evaluations produce the same Agent-visible EVP signal as the other SDKs.This is primarily a manifest activation. The one shared test adjustment is included because the
maindegradation workload does not reliably exercise the degradation contract for Ruby: with only the manifest enabled,mainsends10,000 + 2,000 = 12,000targeting keys in one/fferequest, the weblog returns200, but the test times out waiting for>= 12,000EVP evaluations. In the local failure, the Agent captured7,168evaluations and0degraded evaluations, so the test never crossed the production10,000full-detail cap.Changes
tests/ffe/test_flag_eval_evp.pyfor the validated Rubyrails72weblog inmanifests/ruby.ymlatv2.37.0-dev; other Ruby weblogs remainirrelevant.12,000evaluations in one request to10,050evaluations sent as bounded batches:10 x 1,000plus1 x 50.10,000full-detail evaluations prove the production cap, and the50overflow evaluations must land in degraded buckets./api/v2/flagevaluation.Decisions
rails72for EVPflagevaluation; broader non-metric OpenFeature evaluation coverage for older Ruby weblogs remains separate.Validation
tests/ffe/test_flag_eval_evp.pyrestored toorigin/main:TEST_LIBRARY=ruby WEBLOG_VARIANT=rails72 ./run.sh +v +l ruby FEATURE_FLAGGING_AND_EXPERIMENTATION ++ -k "Test_FFE_EVP_Flagevaluation_Degradation"Failed: timed out waiting for
>= 12000; captured payload summary wastotal_evaluations=7168,degraded_evaluations=0.TEST_LIBRARY=ruby WEBLOG_VARIANT=rails72 ./run.sh +v +l ruby FEATURE_FLAGGING_AND_EXPERIMENTATION ++ -k "Test_FFE_EVP_Flagevaluation_Degradation"Passed:
1 passed, 2648 deselected in 34.04s.TEST_LIBRARY=ruby WEBLOG_VARIANT=rails72 ./run.sh +v +l ruby FEATURE_FLAGGING_AND_EXPERIMENTATION ++ -k "Test_FFE_EVP_Flagevaluation"Passed:
8 passed, 2641 deselected in 62.83s; captured payload summary for degradation wastotal_evaluations=10050,visible_evaluations=10000,degraded_evaluations=50.