[arch] Design discussion: shared pluggable service for cross-agent state (feedback, sessions, traces)

[rdwj]

Background

The v0.12.0 enterprise feature track added several stateful surfaces to BaseAgent's server layer:

• POST/GET /v1/sessions — conversation persistence
• GET /v1/traces — trace inspection
• POST/GET/PATCH /v1/feedback — user feedback collection
• GET /metrics — Prometheus

All four follow the same shape: BaseAgent owns a pluggable store (Null / SQLite / Postgres), the server layer exposes REST endpoints, and the gateway-template proxies through. This was the right call when most deployments had one or two agents.

In multi-agent deployments (eg 10 agents fronted by a single gateway and UI) the per-agent ownership starts to chafe:

• Duplication — 10 Postgres pools, 10 schema migrations, 10 housekeeping loops, all writing the same tables
• Fan-out for cross-agent queries — "show me all thumbs-down feedback this week" requires the dashboard to hit 10 endpoints and merge client-side, OR query the shared Postgres directly out-of-band (the schema becomes a de-facto API)
• Schema becomes a contract — once N agents write the same table, schema changes need coordinated rollouts
• No auth boundary between agents sharing storage
• Sessions are conceptually cross-agent — a user talks to "the system," not to agent base-agent: Implement prompt loader (prompts.py) #4. Today there's no clean way to follow a conversation that gets routed to different agents

What to design

Open question: what does the shape of a shared 'agent platform' service look like, and which surfaces move there?

Initial options to discuss (not a decision, a starting point):

1. Status quo + documentation. Document that multi-agent deployments should point all agents at the same Postgres and treat the shared schema as a stable join point. Cheapest, but the rough edges remain.

2. Full extraction. A new FastAPI service (working name: `fipsagents-platform`) owns sessions + traces + feedback. BaseAgent becomes a thin client. Gateway routes `/v1/sessions`, `/v1/traces`, `/v1/feedback` to the platform service rather than fanning out to per-agent endpoints. One Postgres pool, one REST surface, one dashboard backend.

3. Partial extraction. Move feedback + sessions (genuinely cross-agent) but leave traces in BaseAgent shipping to an Otel collector (industry-standard answer, already partially done via `OTELTraceStore`). Less moving parts, addresses the highest-value duplication.

4. Something else. Maybe BaseAgent keeps everything but grows a 'remote store' adapter for each — same code, configurable backend (in-process vs HTTP). Lets a deployer choose per-feature without forcing a topology.

Things to think about during the discussion

• Migration story. The longer we wait, the more deployments depend on the per-agent endpoints. Cheap to do now while there's effectively one production user; observable migration later
• Memory is intentionally NOT in this list — `self.memory` is per-agent by design, and MemoryHub already provides the centralized option
• Metrics is also separate — Prometheus scrape targets are inherently per-pod, that's fine
• Auth — if multiple agents share a backend, who's allowed to write what? Today there's no model for this
• Deployment friction — every service we extract is another Helm chart, another readiness probe, another thing for ops to think about. Worth it iff the cross-agent benefits land
• Pluggability shape — same `FeedbackStore`/`SessionStore`/`TraceStore` ABCs we have today, just running in a different process? Or a different abstraction entirely?

Out of scope for this issue

This is a design discussion issue, not an implementation. The goal is to come out with a written architecture decision (in `docs/architecture.md` or similar) that we can point at when implementing.

Captured during the v0.12.0 feedback feature track. Conversation context: the per-agent ownership felt fine for one or two agents but the smell got louder once we considered the 10-agent case.

[rdwj]

Decision: Option 4 — remote-store adapter, with a sibling platform service

Written into docs/architecture.md (new section: "Cross-Agent Platform Service"). Summary below.

Shape

We extend the existing pluggable-store pattern with HTTP implementations rather than changing where state ownership lives.

• HttpFeedbackStore, HttpSessionStore, HttpTraceStore added to fipsagents.server alongside the Null/SQLite/Postgres implementations. Same ABCs, HTTP transport.
• fips-agents/fipsagents-platform — new sibling repo. FastAPI service, one Postgres pool, one schema, one migration loop, re-exposes the store semantics as REST. Implemented on top of the SQLite/Postgres ABCs (not a parallel implementation).
• gateway-template gets a routing mode. When the platform service is deployed, gateway routes /v1/sessions/*, /v1/feedback/*, /v1/traces/* directly to it rather than fanning out to per-agent endpoints.
• BaseAgent's HTTP endpoints stay for backward compatibility. When an agent is configured with an HttpXStore, those endpoints become pass-throughs (or are turned off via config). Existing v0.12.0 deployments do not break.

Why Option 4 over Options 2/3

• Option 4 vs Option 3 (partial extraction + OTEL for traces): Option 3 is cleaner conceptually but forces every multi-agent deployment to also adopt an OTEL collector in step one. Option 4 lets the platform's TraceStore ship to OTEL internally as a later iteration without changing topology.
• Option 4 vs Option 2 (full extraction): Option 2 forces the platform service on every deployment, even single-agent ones. Option 4 is opt-in via config — small deployments stay simple.
• Option 4 vs Option 1 (status quo): Status quo leaves every multi-agent rough edge in place (10 Postgres pools, schema-as-de-facto-API, no auth boundary).

Why this is right for Cost Tracking (#104)

#104 explicitly states "usage data attaches to session records — no new storage backend needed." That means token-usage data flows through SessionStore.update(). Whether the session store is SqliteSessionStore or HttpSessionStore, BaseAgent-side code is identical — the data lands wherever sessions land. No re-architecture needed when the platform service exists.

Tradeoffs flagged in the architecture doc

• "Schema as contract" doesn't disappear — moves from raw Postgres to the platform's HTTP API (better, but needs an API stability policy).
• New ops surface (Helm chart, readiness probe, Postgres dependency). Worth it iff deployment runs ≥2 agents.
• Test infrastructure doubles — every ABC needs in-process and HTTP-roundtrip tests.

Out of scope for the initial extraction

• Auth boundary between agents writing shared storage — needed if multi-tenant is in scope; otherwise punt.
• Cross-agent session continuity — protocol question, not a storage question.
• Migrating traces to OTEL — already partially solved by OTELTraceStore; can adopt later without a topology change.

Next step

Implementation will be tracked in follow-up issues across fipsagents-platform (new repo bootstrap), agent-template (HttpXStore implementations), and gateway-template (routing mode). To be filed at the start of the next session.

[rdwj]

Implementation issues filed (next-session plan from NEXT_SESSION.md):

• agent-template#114 — HttpFeedbackStore / HttpSessionStore / HttpTraceStore (this repo)
• gateway-template#30 — routing mode for /v1/sessions/* /v1/feedback/* /v1/traces/*
• fipsagents-platform#1 (new repo) — bootstrap commit (feedback proof point shipped, sessions + traces queued)

The new sibling repo: https://github.com/fips-agents/fipsagents-platform

Bootstrap commit ships:

• FastAPI service skeleton (healthz/readyz, sessions and traces stubbed at 501).
• Feedback proof point — POST/GET/PATCH/list/stats — reusing fipsagents.server.feedback.FeedbackStore over SQLite/Postgres rather than reimplementing persistence.
• Bearer-token auth (none / keycloak modes; same realm as the gateway).
• Helm chart (UBI9, ConfigMap-driven, optional PVC for SQLite, optional Route).
• 13/13 pytest passing end-to-end.

Sessions and traces follow once the pattern proves out. Tracking continues at fipsagents-platform#1.

[rdwj]

Closing — decision implemented end-to-end

The architecture decision (option 2: full extraction) was recorded in docs/architecture.md § "Cross-Agent Platform Service" and has now landed across all three repos.

What shipped

fipsagents-platform (new sibling repo)

• v0.1.0 — initial REST surface (FastAPI, sqlite/postgres backends, Keycloak auth) — closes fipsagents-platform#1
• v0.2.0 — sessions + traces proof points, wire shape extended for HTTP store parity
• v0.2.1 — GET /v1/sessions/{id}/cost_data read companion (fipsagents-platform#5)

fipsagents (agent-side BaseAgent + server)

• 0.13.0 — HttpSessionStore / HttpTraceStore / HttpFeedbackStore and per-store backend override (feat: HTTP store implementations for Cross-Agent Platform Service (#112 impl) #114, feat(server): HTTP-backed FeedbackStore/SessionStore/TraceStore #115)
• 0.14.0–0.14.2 — SessionStore.update(), per-turn cost accumulator, cumulative HTTP read path (feat(server): SessionStore.update() + per-turn cost tracking #117, fix(llm): stream_options.include_usage so cost tracking actually fires #119, feat(server): HttpSessionStore.get_cost_data reads from platform #120)
• 0.15.0 — Cost Tracking v2: PricingConfig, BudgetEnforcer, tenant_id/session_id Prometheus labels, OTEL GenAI semantic conventions (feat(server): PricingConfig + GET /v1/sessions/{id}/usage #121, feat(server): tenant_id + session_id labels on agent_tokens_total #122, feat(server): BudgetEnforcer with per-session + per-tenant limits #123, feat(server): OTEL GenAI semantic conventions on trace spans #124, release in release: fipsagents 0.15.0 — Cost Tracking v2 #125)

gateway-template

• chart 0.5.0 — platform.url routing mode for /v1/feedback*, /v1/sessions/*, /v1/traces/* with per-prefix opt-out, agent carve-out for GET /v1/sessions/{id}/usage (gateway-template#31, tag v0.5.0)

Verified end-to-end

Cluster-smoked against fipsagents-platform 0.2.1 + an agent on fipsagents 0.15.0 + RedHatAI/gpt-oss-20b on RHPDS in namespace fipsagents-cost-smoke. Cumulative cost_data accumulation, /v1/sessions/{id}/usage dollar aggregation (cost_usd=$0.1207 from a 3-turn convo, math exact), agent_tokens_total{tenant_id="cost-smoke"} Prometheus labels, and BudgetEnforcer soft-warnings at both per-session and per-tenant scopes all confirmed against real vLLM traffic.

Not covered here

Two follow-ups deliberately outside this issue's scope, tracked separately:

• Multi-replica per-tenant cost aggregation — current BudgetEnforcer has "this agent process's view" semantics; a cross-agent aggregator is a future capability on fipsagents-platform.
• Auth model for "who's allowed to write what" between agents sharing the platform — current model is "valid Keycloak token + same realm". Per-agent service-account scoping would be a future iteration.

Closing as implemented.