feat: include Tempo MCP server in set of exposed Tempo APIs

[andreasgerstmayr]

Include Tempo MCP server in set of exposed Tempo APIs

[saswatamcode]

Will these APIs be tenanted? (or even need to be tenanted?)

[andreasgerstmayr]

Yes, the MCP endpoint is handled like a regular Tempo API endpoint.
It requires the X-Scope-OrgID header (specifying the tenant) to be set (if the Tempo instance has multi-tenancy enabled).

[squat]

I'm concerned that this MCP endpoint has no authorization on it. This mean that a user/account who lacks the permissions to read/write traces for a tenant will be able to side step that permissions issue by issuing commands to the MCP server and having the agent do it for them directly on Tempo, where Observatorium API can't check permissions.

[andreasgerstmayr]

I'm concerned that this MCP endpoint has no authorization on it. This mean that a user/account who lacks the permissions to read/write traces for a tenant will be able to side step that permissions issue by issuing commands to the MCP server and having the agent do it for them directly on Tempo, where Observatorium API can't check permissions.

The MCP endpoint has the same authorization configured which we use for accessing the other Tempo API methods:

api/main.go

Lines 877 to 878 in 9e931b5

	tracesv1.WithTempoMiddleware(authorization.WithAuthorizers(authorizers, rbac.Read, "traces")),
	tracesv1.WithTempoMiddleware(logsv1.WithEnforceAuthorizationLabels()),

The only issue I see is that the MCP server may gain tools which should require write permissions in the future. Currently it only exposes read-only tools, i.e. tools which query trace data. I can't think of any MCP tool requiring write access to Tempo, I don't think the MCP server ever will have a tool to ingest traces for example.

[squat]

The only issue I see is that the MCP server may gain tools which should require write permissions in the future

This is exactly what I am driving at: users could escalate permissions via the MCP server. If you have read permissions on the Tempo API via Observation RBAC and the MCP server allows other functionality, you could escalate your permissions.

I'm not sure how we want to deal with this, since we aren't in control of the Tempo project and what they choose to put into the MCP server. I would say that RBAC is simply insufficient for MCP and we need fine grained authorization for every tool resource and prompt the MCP server exposes.

It would be a different story if the MCP server was a client of the Observatorium API, rather than the upstream.

[andreasgerstmayr]

I'm not sure how we want to deal with this, since we aren't in control of the Tempo project and what they choose to put into the MCP server. I would say that RBAC is simply insufficient for MCP and we need fine grained authorization for every tool resource and prompt the MCP server exposes.

I think RBAC with the current read/write permissions can still work in the context of MCP:

• tools support a "read-only hint" annotation
• resources are always read-only / non destructive afaics

I can proxy all requests to the MCP server in observatorium and reject tool calls based on their "read-only hint" annotation. Would that work? If so, I'll update this PR.

It would be a different story if the MCP server was a client of the Observatorium API, rather than the upstream.

I did propose to extract the MCP server in upstream, i.e. to change the request flow to Tempo MCP server --> optional proxy (in our case, observatorium) --> Tempo HTTP API, but that idea was rejected.

For reference, the Tempo MCP server currently exposes the following:

Resources

Resource	URI	Description
TraceQL Basic Docs	docs://traceql/basic	Basic TraceQL documentation covering intrinsics, operators, and attribute syntaxes
TraceQL Aggregates Docs	docs://traceql/aggregates	Aggregates documentation covering count, sum, and other aggregation functions
TraceQL Structural Docs	docs://traceql/structural	Structural queries documentation covering advanced query patterns
TraceQL Metrics Docs	docs://traceql/metrics	Documentation on generating metrics from tracing data
TraceQL Query Docs	docs://traceql/query	Full TraceQL search documentation with examples

Tools

Tool	Description
docs-traceql	Documentation on TraceQL search - covers basic attributes, aggregates, pipelining, structural queries, and metrics
get-attribute-names	Get available attribute names for TraceQL queries, optionally filtered by scope (span, resource, event, link, instrumentation)
get-attribute-values	Get values for a specific attribute (e.g., resource.service.name to find all services)
get-trace	Retrieve a specific trace by ID
traceql-search	Search for traces using TraceQL queries
traceql-metrics-instant	Get a single metric value at the current instant from a TraceQL metrics query
traceql-metrics-range	Get a metric time series from a TraceQL metrics query

[squat]

I think RBAC with the current read/write permissions can still work in the context of MCP:

• tools support a "read-only hint" annotation
• resources are always read-only / non destructive afaics

I can proxy all requests to the MCP server in observatorium and reject tool calls based on their "read-only hint" annotation. Would that work? If so, I'll update this PR.

I think this should be good. This means that any tool without the read-only hint annotation won't be callable, which works for us. It moves the burden onto Tempo to correctly annotate tools if they want them to work in truly read-only contexts.