Skip to content

Pin injector and system-tests to Ruby 4 dev refs#5981

Draft
lloeki wants to merge 1 commit into
masterfrom
lloeki/ruby4-bundler4
Draft

Pin injector and system-tests to Ruby 4 dev refs#5981
lloeki wants to merge 1 commit into
masterfrom
lloeki/ruby4-bundler4

Conversation

@lloeki

@lloeki lloeki commented Jul 2, 2026

Copy link
Copy Markdown
Member

What does this PR do?
Temporary integration branch to validate Ruby 4.0 / RubyGems 4.0 / Bundler 4.0
single-step instrumentation (SSI) end-to-end before either dependency is released.

  • .gitlab/prepare-oci-package.sh: build the OCI package from the injector
    development branch lloeki/ruby4-bundler4 (datadog-injector-rb#65) instead of
    v1.3.0.
  • .gitlab-ci.yml: run system-tests at the Ruby 4 / Bundler 4 branch commit
    (system-tests#7249) instead of the automated ref.

Motivation:
The injector guard change (datadog-injector-rb#65) and the new Rails 8 / Ruby 4 /
Bundler 4 SSI weblog (system-tests#7249) need to be exercised together against this
repo\s OCI packaging and system-tests pipeline. The tracer already supports Ruby
4.0 (MAXIMUM_RUBY_VERSION = 4.1), so injection into a Ruby 4 app should succeed.

Change log entry
None. CI-only, temporary integration branch.

Additional Notes:
Both pins are marked TEMPORARY in-line and MUST be reverted to released/automated
refs before merge:

  • injector_ref back to a released injector tag.
  • SYSTEM_TESTS_REF back to the value managed by
    .github/workflows/update-system-tests.yml.

Also depends on the images-rb Ruby 4 / Bundler 4 toolchain rollout.

How to test the change?
Runs the docker_ssi and lib-injection system-tests scenario groups in CI against
the pinned refs; the new rails8-app* (RB40) weblog should build and inject.

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Jul 2, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI

⚠️ Warnings

🚦 16 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-rb | rails8-app-vendored-mode.amd64.DOC: [public.ecr.aws/lts/ubuntu:22.04, linux/amd64, 4.0.5]   View in Datadog   GitLab

🧪 2 Tests failed

test_install_supported_runtime from system_tests_suite   View in Datadog
AssertionError: No traces found for request QZRHTVZMYBLXHDQZISZEZTSZLUOHJTWMKQTG
assert None

self = <tests.docker_ssi.test_docker_ssi.TestDockerSSIFeatures object at 0x7f8840c40fb0>

    @features.ssi_guardrails
    @bug(condition="centos-7" in context.weblog_variant and context.library == "java", reason="APMON-1490")
    @irrelevant(context.library == "python" and context.installed_language_runtime < "3.8.0")
    @irrelevant(context.library == "java" and context.installed_language_runtime < "1.8.0_0")
    @irrelevant(context.library == "php" and context.installed_language_runtime < "7.0")
...
test_telemetry from system_tests_suite   View in Datadog
AssertionError: No telemetry data found for library_entrypoint.complete
assert False

self = <tests.docker_ssi.test_docker_ssi.TestDockerSSIFeatures object at 0x7f8840c411c0>

    @features.ssi_guardrails
    @bug(
        condition="centos-7" in context.weblog_variant and context.library == "java",
        reason="APMON-1490",
    )
...

DataDog/apm-reliability/dd-trace-rb | rails8-app.amd64.DOA9: [public.ecr.aws/lts/ubuntu:22.04, linux/amd64, 4.0.5]   View in Datadog   GitLab

🧪 1 Test failed

test_telemetry_source_ssi from system_tests_suite   View in Datadog
AssertionError: No traces found for request CXIDACJMQQZHGMEGJXYAONGJKVSZALIEXTRN
assert None

self = <tests.docker_ssi.test_docker_ssi_appsec.TestDockerSSIAppsecFeatures object at 0x7fcf16fd2840>

    @irrelevant(context.library == "java" and context.installed_language_runtime < "1.8.0_0")
    @irrelevant(context.library == "php" and context.installed_language_runtime < "7.1")
    @irrelevant(context.library == "nodejs" and context.installed_language_runtime < "17.0")
    @irrelevant(context.library >= "python@4.0.0.dev" and context.installed_language_runtime < "3.9.0")
    @irrelevant(context.library < "python@4.0.0.dev" and context.installed_language_runtime < "3.8.0")
...

DataDog/apm-reliability/dd-trace-rb | rails8-app-deployment-mode.amd64.DOA9: [public.ecr.aws/lts/ubuntu:22.04, linux/amd64, 4.0.5]   View in Datadog   GitLab

View all 16 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog auto-retried 4 jobs - 0 passed on retry View in Datadog

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 445d8eb | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Jul 2, 2026

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-07-03 08:07:12

Comparing candidate commit 445d8eb in PR branch lloeki/ruby4-bundler4 with baseline commit 3ab60d6 in branch master.

Found 0 performance improvements and 1 performance regressions! Performance is the same for 47 metrics, 1 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:profiling - Allocations ()

  • 🟥 throughput [-233512.155op/s; -189421.125op/s] or [-7.155%; -5.804%]

@lloeki lloeki force-pushed the lloeki/ruby4-bundler4 branch 2 times, most recently from d1fa7df to 28a8271 Compare July 2, 2026 15:50
Temporary integration branch to validate Ruby 4.0 / Bundler 4.0 SSI
end-to-end before either dependency is released.

 - prepare-oci-package.sh: build the OCI package from the injector
   development branch lloeki/ruby4-bundler4 (datadog-injector-rb#65)
   instead of v1.3.0.
 - .gitlab-ci.yml: run system-tests at the Ruby 4 / Bundler 4 branch
   commit (system-tests#7249) instead of the automated ref.

Both pins must be reverted to released/automated refs before merge.
@lloeki lloeki force-pushed the lloeki/ruby4-bundler4 branch from 28a8271 to 445d8eb Compare July 3, 2026 07:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant