[Bug] Trace disconnection between RabbitMQ producer/consumer & incomplete consumer trace stack (unresolved historical issues)

[liuhaolong10]

Search before asking

• I had searched in the issues and found no similar issues.

Apache SkyWalking Component

Java Agent (apache/skywalking-java)

What happened

Problem Description
When using the skywalking-java agent to collect trace data for RabbitMQ message production and consumption, two key issues persist:

1. TraceId disconnection between producer and consumer services
The trace links of RabbitMQ message production (by producer service) and consumption (by consumer service) cannot be connected through a single traceId.
Example scenario:

- Java Service A (producer) sends a RabbitMQ message.

- Java Service B (consumer) consumes this message.
   Expected behavior: The entire call chain (Service A → RabbitMQ → Service B) should be connected via one unified traceId.
  Actual issue: The consumer thread in Service B fails to retrieve the skywalking-java traceId when consuming the RabbitMQ message, leading to disconnected trace links between Service A and Service B.

2. Incomplete consumer trace stack
The collected trace stack for the RabbitMQ consumer only shows a single layer named "rabbitmq", and does not attach the details of subsequent business operations (e.g., MySQL queries, Redis operations) associated with message consumption. (As shown in the attached screenshot below)

Related Historical Issues
I searched the issue list and found two similar unresolved issues regarding RabbitMQ trace collection:

• [Bug] In springboot+rabbitmq projects, rabbitmq listener did not generates traceId automatically. #11300

• skywalking plugin apm-rabbitmq-5.x-plugin-8.5.0 cannot trace customer  #7018

Solution Status
I have developed a complete solution for the above issues, and it has been fully tested to effectively resolve all the mentioned problems. (As shown in the attached screenshot below)

Request for Guidance
Could you please guide me on the official process to submit the code patch for this fix? Specifically, I would like to know:

• The target branch to submit the PR to (e.g., master, release-x.x)

• Any coding standards or PR template requirements to follow

• The code review process and key points to note

What you expected to happen

After troubleshooting the code of the RabbitMQ collection plugin, I found that the method intercepted by the instrumentation is executed in a thread pool dedicated to RabbitMQ message processing, which is not the same thread as the one where the consumer actually executes the business logic code. This thread inconsistency leads to the loss of trace information.

How to reproduce

Steps to Reproduce the Issue

1. In a Java service, write code to send a RabbitMQ message.

2. Then write code to consume the RabbitMQ message, and add a database query operation in the consumption logic.

This is a 100% reproducible issue (consistently reproducible) and can be easily reproduced.

Anything else

No response

Are you willing to submit a pull request to fix on your own?

• Yes I am willing to submit a pull request on my own!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

As you mentioned, you will submit a pull request, have you already found the cause?

[liuhaolong10]

Yes, I have identified the root cause.
The original instrumentation method in the RabbitMQ plugin does not work correctly.
I have replaced it with a new instrumentation method, which can now resolve the problem.

[wu-sheng]

Note, we already have plugin tests for it. So, the old one maybe work for some cases, but not the cases you used. You need to update test-scenarios of plugin when you submit a PR.

[liuhaolong10]

After troubleshooting, I found that the old RabbitMQ instrumentation is executed in a dedicated thread pool for RabbitMQ message processing, which is not the same thread where the consumer runs the real business logic.

Because of this, the old instrumentation can only capture that a message was consumed, but cannot continue tracing the subsequent business traces such as database or Redis operations.

In my opinion, this old instrumentation is no longer meaningful.

I suggest removing the old instrumentation code and its related unit tests, and replacing them with the new instrumentation logic I have implemented.

I have already written the unit tests for the new instrumentation.

Since I have not maintained the SkyWalking Java agent before, I would appreciate it if you could review the code I will submit.

[wu-sheng]

• https://github.com/apache/skywalking-java/tree/main/test/plugin/scenarios/rocketmq-scenario
• https://github.com/apache/skywalking-java/tree/main/test/plugin/scenarios/rocketmq-5-grpc-scenario

Two rocketMQ cases exist. They are not stable, maybe you found the reason. You could learn how to run this test via this doc, https://skywalking.apache.org/docs/skywalking-java/next/en/setup/service-agent/java-agent/plugin-test/

[liuhaolong10]

Through my tests, I found that the current method instrumentation in the RabbitMQ plugin can collect messages consumed by the native amqp-client,
but cannot collect messages consumed by spring-rabbit using the @RabbitListener annotation.
Therefore, I have added a new spring-rabbit plugin,
and modified the code of the original RabbitMQ plugin to avoid data collection conflicts between these two plugins.
I will submit the code following the official plugin development guidelines later.