test(debugger): ensure span error budgets are reasonable

[tylfin]

Motivation

Add a test to verify that span decoration probes with improper expressions (e.g., "Cannot dereference field") have their error logs rate-limited to at most once per second. Currently marked as a bug for Java since span decoration probes lack probe budgets, producing 150 error snapshots for 150 requests instead of the expected ~1 per second.

Changes

• Added a new test method test_span_probe_expression_budgets in tests/debugger/test_debugger_probe_budgets.py to verify that error logs from span probe expression failures are rate-limited to at most 1 per second, with appropriate assertions and handling for language-specific limitations.
• Introduced a helper method setup_span_probe_expression_budgets to configure and trigger the span probe expression budget scenario in the same test file.
• Added a new probe configuration file probe_span_method_budgets_expression.json under tests/debugger/utils/probes/ to define a span probe with an intentionally failing expression, used for testing error logging and rate limiting.
• Imported the bug decorator from utils in tests/debugger/test_debugger_probe_budgets.py to allow marking tests with known bugs, specifically for Java.

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Test logic is modified? -> Get a review from RFC owner.
   • Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

• Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
• A docker base image is modified?
  • the relevant build-XXX-image label is present
• A scenario is added, removed or renamed?
  • Get a review from R&P team

[github-actions]

CODEOWNERS have been resolved as:

tests/debugger/utils/probes/probe_span_method_budgets_expression.json   @DataDog/debugger @DataDog/system-tests-core
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
tests/debugger/test_debugger_probe_budgets.py                           @DataDog/debugger @DataDog/system-tests-core

[nccatoni]

LGTM for @DataDog/system-tests-core, you should also get a review from someone familiar with the feature

[datadog-official]

⚠️ Tests

✨ Fix all issues with Cursor

⚠️ Warnings

❄️ 3 New flaky tests detected

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_decoration_method_snapshot[uds] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace decor454-d7eb-451d-9af4-069515281276 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7f9dcbffa810>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_decoration_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:126: 
...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_method_snapshot[uds] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace span9710-d3ee-4b06-85df-745e14f431a6 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7f9dcbffa6f0>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:112: 
...

tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots.test_span_method_snapshot[uds-spring-boot] from system_tests_suite (Datadog) (Fix with Cursor)

ValueError: Trace span3aeb-8fd1-41a5-87b8-ea08b13f0867 was not received.

self = <tests.debugger.test_debugger_probe_snapshot.Test_Debugger_Method_Probe_Snaphots object at 0x7f06883d3a40>

    @missing_feature(context.library == "golang", reason="Not yet implemented", force_skip=True)
    def test_span_method_snapshot(self):
        self._assert()
>       self._validate_spans()

tests/debugger/test_debugger_probe_snapshot.py:112: 
...

🧪 31 Tests failed

tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets.test_span_probe_expression_budgets[apache-mod-7.0] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Expected at least 1 error entry to verify the span probe expression error was logged, got 0
assert 0 >= 1

self = <tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets object at 0x7f05b774f620>

    def test_span_probe_expression_budgets(self):
        self.collect()
        self.assert_setup_ok()
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok()
...

tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets.test_span_probe_expression_budgets[apache-mod-7.0-zts] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Expected at least 1 error entry to verify the span probe expression error was logged, got 0
assert 0 >= 1

self = <tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets object at 0x7f8476c7ed50>

    def test_span_probe_expression_budgets(self):
        self.collect()
        self.assert_setup_ok()
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok()
...

tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets.test_span_probe_expression_budgets[apache-mod-7.1] from system_tests_suite (Datadog) (Fix with Cursor)

AssertionError: Expected at least 1 error entry to verify the span probe expression error was logged, got 0
assert 0 >= 1

self = <tests.debugger.test_debugger_probe_budgets.Test_Debugger_Probe_Budgets object at 0x7f563d94d130>

    def test_span_probe_expression_budgets(self):
        self.collect()
        self.assert_setup_ok()
        self.assert_rc_state_not_error()
        self.assert_all_weblog_responses_ok()
...

View all

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: df28155 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}