Skip to content

fix(openfeature): map invalid FFE configs to parse errors#5948

Merged
lloeki merged 1 commit into
lloeki/bump-libdatadog-36from
leo.romanovsky/fix-libdatadog-v36-ffe-parse-error-rb
Jun 24, 2026
Merged

fix(openfeature): map invalid FFE configs to parse errors#5948
lloeki merged 1 commit into
lloeki/bump-libdatadog-36from
leo.romanovsky/fix-libdatadog-v36-ffe-parse-error-rb

Conversation

@leoromanovsky

@leoromanovsky leoromanovsky commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Motivation

The libdatadog v36 bump in #5928 changes invalid per-flag FFE configuration failures to surface through the native result as caller-default behavior. Ruby then reports OpenFeature evaluation metrics with feature_flag.result.reason:default and no error.type, which breaks the existing system-test contract for invalid regex and variant type mismatch cases.

Changes

This adds a narrow compatibility shim in the OpenFeature native evaluator. When libdatadog returns the v36 invalid-flag sentinel (DEFAULT, no error code, and flag configuration is invalid or unsupported), Ruby converts it into the existing OpenFeature parse-error resolution using the caller default value.

The change also adds the missing OpenFeature constants and RBS updates needed for that mapping, plus a regression spec for the invalid-configuration case and a guard that ordinary default fallthrough still remains default behavior.

Decisions

The mapping stays at the OpenFeature evaluator boundary instead of metrics code, so provider resolution, hooks, exposure reporting, and metric tagging all see the same ERROR / PARSE_ERROR result. The check is intentionally narrow to avoid changing disabled flags, default allocations, or normal no-match fallthrough from libdatadog v36.

Change log entry

Yes. Fix OpenFeature invalid FFE flag configurations to report parse errors instead of default evaluations.

Validation

  • bundle install
  • bundle exec rake test:open_feature 2>&1 | tee /tmp/full_rspec.log | grep -E 'Pending:|Failures:|Finished' -A 99 (ruby_3.2_openfeature_latest: 100 examples, 0 failures; ruby_3.2_openfeature_min: 100 examples, 0 failures)
  • STANDARDOPTS="lib/datadog/open_feature/ext.rb lib/datadog/open_feature/native_evaluator.rb spec/datadog/open_feature/native_evaluator_spec.rb" bundle exec rake standard
  • bundle exec rake steep:check

@dd-octo-sts

dd-octo-sts Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hey @DataDog/ruby-guild, please fill "Change log entry" section in the pull request description.

If changes need to be present in CHANGELOG.md you can state it this way

**Change log entry**

Yes. A brief summary to be placed into the CHANGELOG.md

(possible answers Yes/Yep/Yeah)

Or you can opt out like that

**Change log entry**

None.

(possible answers No/Nope/None)

Visited at: 2026-06-24 10:22:26 UTC

@leoromanovsky leoromanovsky marked this pull request as ready for review June 24, 2026 10:40
@leoromanovsky leoromanovsky requested review from a team as code owners June 24, 2026 10:40
@leoromanovsky leoromanovsky requested review from dd-oleksii, lloeki and sameerank and removed request for a team June 24, 2026 10:40
@pr-commenter

pr-commenter Bot commented Jun 24, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-06-24 10:43:25

Comparing candidate commit 8727217 in PR branch leo.romanovsky/fix-libdatadog-v36-ffe-parse-error-rb with baseline commit c6cac23 in branch lloeki/bump-libdatadog-36.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@lloeki lloeki merged commit cb494a1 into lloeki/bump-libdatadog-36 Jun 24, 2026
169 checks passed
@lloeki lloeki deleted the leo.romanovsky/fix-libdatadog-v36-ffe-parse-error-rb branch June 24, 2026 11:39
@dd-octo-sts dd-octo-sts Bot added this to the 2.36.0 milestone Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants