Skip to content

[propagator-xray] X-Amzn-Trace-Id inject key is mixed-case, breaking gRPC metadata usage #2129

@gonsuke

Description

@gonsuke

Description of the bug

When OpenTelemetry::Propagator::XRay::TextMapPropagator is used as a propagator and OpenTelemetry.propagation.inject(metadata) is called with a gRPC metadata hash, the key X-Amzn-Trace-Id (mixed-case) is injected. However, gRPC requires metadata keys to match [a-z0-9-_.] (lowercase only), this causes a runtime error.

The relevant line in text_map_propagator.rb

XRAY_CONTEXT_KEY = 'X-Amzn-Trace-Id'  # line 20
# ...
setter.set(carrier, XRAY_CONTEXT_KEY, xray_value)  # line 98

Share details about your runtime

  • Operating system details: Linux Debian bookworm, macOS 26.3.1
  • Ruby 3.4.4
  • opentelemetry-propagator-xray: 0.26.1
  • opentelemetry-sdk: 1.10.0
  • grpc gem (Ruby gRPC)

Share a simplified reproduction if possible

require 'opentelemetry/sdk'
require 'opentelemetry-propagator-xray'
require 'grpc'

OpenTelemetry::SDK.configure do |c|
  c.propagators = [
    OpenTelemetry::Trace::Propagation::TraceContext.text_map_propagator,
    OpenTelemetry::Propagator::XRay::TextMapPropagator.new
  ]
  c.service_name = 'repro'
end

metadata = {}
tracer = OpenTelemetry.tracer_provider.tracer('repro')
tracer.in_span('test') do
  OpenTelemetry.propagation.inject(metadata)
end

puts "Injected keys : #{metadata.keys.inspect}"

begin
  chan = GRPC::Core::Channel.new('localhost:9999', nil, :this_channel_is_insecure)
  call = chan.create_call(nil, nil, '/blah/method', nil, Time.now + 1)
  call.run_batch(GRPC::Core::CallOps::SEND_INITIAL_METADATA => metadata)
rescue GRPC::BadStatus, GRPC::Core::CallError => e
  puts "gRPC error     : #{e.class}: #{e.message}"
rescue StandardError => e
  puts "Other error    : #{e.class}: #{e.message}"
end

Output:

Injected keys : ["traceparent", "X-Amzn-Trace-Id"]
Other error    : ArgumentError: 'X-Amzn-Trace-Id' is an invalid header key, must match [a-z0-9-_.]+

Workaround

I worked around this by wrapping the gRPC metadata hash in a carrier that normalizes keys to lowercase before writing:

class LowercaseCarrier
  def initialize(hash) = @hash = hash
  def []=(key, value)  = @hash[key.downcase] = value
  def [](key)          = @hash[key.downcase]
  def keys             = @hash.keys
end

OpenTelemetry.propagation.inject(LowercaseCarrier.new(metadata))

The AWS X-Ray header name is case-insensitive at the HTTP level (per RFC 7230), so I'd imagine lowercase would be fine... but I wanted to double check here before assuming anything.

Thanks for maintaining these gems. It's been a solid building block for our tracing setup on AWS.

Tip: React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more in our end user docs.

Metadata

Metadata

Assignees

Labels

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions