Add Data Streams Monitoring support to ActiveJob#5961
Conversation
Inject and extract the DSM pathway context through the serialized job payload so produce (enqueue) and consume (perform) checkpoints connect across the enqueue/perform process boundary, regardless of queue adapter.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f37bfbf454
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Strech
left a comment
Strech
left a comment
There was a problem hiding this comment.
I have few minor suggestions to reduce nesting and increase simplicity. Of course Codex brought a valid point too
|
Thanks! I'll get those fixed. |
Require datadog/data_streams so the enabled? facade is defined under the tracing-only entrypoint, avoiding a NoMethodError on every serialize. Collapse the two single-method patches into one DataStreams module and add a regression test for the unconfigured-DSM path.
|
@Strech can you approve the CI run? |
vpellan
left a comment
vpellan
left a comment
There was a problem hiding this comment.
Hey! Thanks for your contribution! As with the racecar PR, I'll also ask for a review from the DSM team
|
@dasch Done ✅ |
This comment has been minimized.
This comment has been minimized.
ActiveJob 4.2 deserializes jobs via a class method, not the instance method, so a prepended instance deserialize has no super to call and cannot receive the queue. Gate DSM injection to ActiveJob 5.0+ and skip the DSM specs below that version. Replace untyped with any in the RBS and add a module self-type so steep can resolve queue_name on the host job class.
|
🤖 Heads up on a behavioral note beyond the RBS fixes: CI caught that ActiveJob 4.2 (Rails 4) deserializes jobs through a class method, not the instance method, so a prepended instance |
|
🤖 CI note for maintainers — the two red checks are a fork-PR limitation, not the diff. This PR is from a fork (
The Everything that does run is green, including the two jobs that caught real issues in the previous round:
To get a fully green run, a maintainer would need to re-run CI from a branch on |
4 participants
What does this PR do?
Adds Data Streams Monitoring (DSM) support to the ActiveJob integration. When DSM is enabled, the integration sets a produce checkpoint on enqueue and a consume checkpoint on perform, and propagates the DSM pathway context from the enqueueing process to the worker process.
The pathway context is carried through
ActiveJob::Core#serialize/#deserialize: the serialized job hash is the payload every queue adapter persists and transmits, and the only thing that actually crosses the enqueue → perform process boundary. The ActiveSupport notification events (enqueue.active_job/perform.active_job) only expose the in-memory job object, so they cannot carry context to the worker. Hooking serialize/deserialize works uniformly across all queue adapters (Sidekiq, SQS, Resque, inline, …) and every supported Rails version (4.2+).Motivation:
ActiveJob is a common messaging boundary in Rails apps, but DSM could not trace messages through it. This connects the enqueue and perform sides into a single pathway so latency and throughput across job queues show up in Data Streams Monitoring.
Change log entry
Yes. Data Streams Monitoring now traces messages through ActiveJob enqueue and perform.
Additional Notes:
DSM checkpoints are tagged
type:active_jobregardless of the underlying adapter, keeping the pathway self-documenting and consistent. Checkpoints are set only whenDatadog::DataStreams.enabled?, mirroring the existing Kafka/WaterDrop integrations. Any error raised while setting a checkpoint is swallowed and logged at debug level, so DSM can never break a customer's job from enqueuing or running.Closes #5976
How to test the change?
Unit tests added in
spec/datadog/tracing/contrib/rails/rails_active_job_spec.rbcovering: produce checkpoint + pathway injection on serialize, consume checkpoint + pathway extraction on deserialize, no-ops when DSM is disabled, and that a raising checkpoint is swallowed so serialize/deserialize still succeed. Verified green on Ruby 3.2 / Rails 6.1 and Ruby 3.4 / Rails 8.