instance_id tag always 0 exporting traces using OTEL

[jriguera]

We are using OTEL collector to receive telemetry from loggregator. For metrics and logs, the instace_id tag is the applicacion index (as stated in the docs) but when a span is generated on a gorouter (from an Envelope_Timer) the instance_id is always 0.

Eg:

Otel collector span dump:

    Trace ID       : 572d882b982442f17038cef2b6a7a7e2
    Parent ID      :
    ID             : 7038cef2b6a7a7e2
    Name           : /
    Kind           : Server
    Start time     : 2025-04-23 08:46:18.756613328 +0000 UTC
    End time       : 2025-04-23 08:46:18.76391097 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> instance_id: Str(0)
     -> source_id: Str(0967492b-e013-4c4c-8607-e423b5e67ca4)
     -> component: Str(route-emitter)
     -> index: Str(09b2820c-76fc-4e22-be43-ceeb8b54b477)
     -> space_name: Str(test)
     -> space_id: Str(937d13ff-dbce-4740-bfdd-494d64727202)
     -> uri: Str(http://go-hello.t.cf.domain/)
     -> organization_id: Str(c1bc891b-1ec2-4131-a599-c6a18b51f90d)
     -> process_id: Str(0967492b-e013-4c4c-8607-e423b5e67ca4)
     -> custom_tag: Str(some_value)
     -> process_instance_id: Str(d5334cfb-a204-47c2-7b9f-8d42)
     -> request_id: Str(572d882b-9824-42f1-7038-cef2b6a7a7e2)
     -> remote_address: Str(x.x.x.16:44156)
     -> app_id: Str(0967492b-e013-4c4c-8607-e423b5e67ca4)
     -> process_type: Str(web)
     -> organization_name: Str(engineering-enablement)
     -> status_code: Str(200)
     -> deployment: Str(cf-test)
     -> method: Str(GET)
     -> app_name: Str(go-hello)
     -> user_agent: Str(curl/7.88.1)
     -> forwarded: Str(x.x.x.39 x.x.x.97)
     -> job: Str(router)
     -> content_length: Str(58)
     -> peer_type: Str(Server)
     -> ip: Str(x.x.x.179)
     -> origin: Str(gorouter)
     -> routing_instance_id: Str()

RTR Access log:

go-hello.t.cf.domain - [2025-04-23T08:46:18.756541129Z] "GET / HTTP/1.1" 200 0 58 "-" "curl/7.88.1" "x.x.x.16:44156" "x.x.x.92:61018" x_forwarded_for:"x.x.x.39, x.x.x.97, x.x.x.16" x_forwarded_proto:"http" vcap_request_id:"572d882b-9824-42f1-7038-cef2b6a7a7e2" response_time:0.007161 gorouter_time:0.000173 app_id:"0967492b-e013-4c4c-8607-e423b5e67ca4" app_index:"2" instance_id:"d5334cfb-a204-47c2-7b9f-8d42" x_cf_routererror:"-" x_forwarded_host:"-" traceparent:"00-572d882b982442f17038cef2b6a7a7e2-7038cef2b6a7a7e2-01" tracestate:"gorouter=7038cef2b6a7a7e2"

I will try to investigate where is the bug.

[jriguera]

It seems to me, the root of the issue is because the field InstanceId is never populated on the HttpStartStopEvent generated in the Gorouter (see https://github.com/cloudfoundry/gorouter/blob/b5be7589592b9cf21f69dd74055bdcc77157e369/handlers/httpstartstop.go#L60-L80) but the application index value is added as a tag instance_id (probably in envelopeTags function but I have not check).

When the envelope is processed in OTEL code, it has the field InstanceId unset, but there is a tag instance_id in the envelope tags map which is skipped in this line if k == "instance_id" ... continue

So, the problem is we are assigning instance_id from the envelope InstanceId field in using e.GetInstanceId(), which is always unset (set to blank "") in HttpStartStopEvents (so eventually coerced 0) , and later we skip the instance_id in the tags (most likely because it was assumed they have the same value).

I am not sure what is the best approach to fix this issue ... maybe just get always the instance_id from the tags, where the rest of application/org/space labels are defined?? or is it a gorouter issue?

[chombium]

Hi @jriguera,

Afaik this behavior is known and expected as the Gorouter only gets ip addresses of healthy app instances and doesn't know anything about instances ids or indexes.

Here are the two things I can think of that can be done:

1. check and trace the vcap request id. I think it is set in the HttpStartStop requestId and corelate the applicationId and instanceIndex fields
2. Check what meta data there is in the traces If you have Zipkin tracing activated.

You can also ask the colleagues from the CF Routing group on CF Slack, how are the RTR logs generated and why the app index is always 0.

[chombium]

Thanks for looking deeper into this @jriguera .

It seems to me, the root of the issue is because the field InstanceId is never populated on the HttpStartStopEvent generated in the Gorouter (see https://github.com/cloudfoundry/gorouter/blob/b5be7589592b9cf21f69dd74055bdcc77157e369/handlers/httpstartstop.go#L60-L80) but the application index value is added as a tag instance_id (probably in envelopeTags function but I have not check).

I guess we'll have to the routing guys to understand why are the headers removed here and what does the ProxyRoundTripper do.

When the envelope is processed in OTEL code, it has the field InstanceId unset, but there is a tag instance_id in the envelope tags map which is skipped in this line if k == "instance_id" ... continue

IMO we have to fix this. Now we have Otel semantic conventions for CF and we can map these fields properly.

[chombium]

Hi @jriguera,

I've just spoken with the colleagues from the routing team who have confirmed that the route-emitter simply registers and unregisters routes and doesn't know anything about app instances. That's also documented in the routiung architecture docs. I guess this is expected and we should discuss if we need this data in the gorouter spans we can open an issue. The only reliable thing to correlate the data are the is the request_id (vcap request id) fields.

Can you please write a bit more on what are you trying to do? Maybe there is another way to get what you want to (at least until we clarify and fix this).

[jriguera]

Sure, first of all, I do not think is an issue in the Gorouter (having the field InstanceId unset is allowed but imho it is unexpected).

I think the problem is about (in)consistency. All the information about the application instance which handles the request comes in the envelope tags (which is filled by the gorouter): application_name, org_name, space_name, all the IDs are there, even the request_id itself comes from the envelope tags. The only missing piece of information is the instance_id which we do not take same source of all the rest (from the tags), but from the InstanceId field. That's simple because there are two fields with similar names InstanceId vs instance_id. I think, we should process instance_id from the tags like the others and maybe renamed to cloudfoundry.app.index just according with the new semantic conventions for CF (although this attribute is not defined).

Why do we need this? Because it’s useful to generate metrics and statistics for requests grouped by route, instance, and availability zone (AZ). We're generating these metrics from spans, which originate from the HttpStartStop loggregator envelope event. At this stage, extracting this data from other fields like requestId isn’t practical and it would require significant processing overhead.

And I completely agree we should switch to the new OTEL semantic conventions for CF!