Encapsulate vMCP assembly behind a config-driven public API

[tgrunnagle]

Summary / Motivation

vMCP's assembly — turning configuration into a running server — lives in
pkg/vmcp/cli/serve.go (~700 lines). It builds backend discovery (the dynamic
BackendRegistry + Kubernetes watcher, or static/inline backends), the aggregator,
router, backend client, health monitor, optimizer, session-manager config,
incoming-auth and rate-limit middleware, the workflow definitions, and finally the
server.

That wiring is not packaged. An embedder (e.g. the enterprise gateway) that wants
a vMCP server must re-implement the same sequence by hand and honor assembly rules
that today exist only in doc comments — and it breaks whenever OSS changes how vMCP
assembles itself. The cost of keeping that copy in sync belongs in the OSS.

#5541 (merged via #5542) encapsulated one seam — the Kubernetes-backed
BackendRegistry + watcher — so embedders no longer hand-wire it. This issue finishes
the job for the assembly as a whole: a caller hands over config and gets back a
VMCP core (to decorate) plus the transport config to serve. The OSS owns the
assembly; embedders compose vMCP instead of reimplementing it.

Raised as a blocker on #5542:
#5542 (comment)

vMCP should encapsulate its own assembly. An embedder should hand over config and
get a server back — not wire up vMCP's internals or honor assembly rules that live
only in doc comments. […] which backends exist, and how they're discovered, is a
config choice — not a dependency the embedder constructs and injects.

This extends the direction of epic #5419 / RFC THV-0076. As written, THV-0076 keeps
assembly at the composition root and treats discovery as an injected dependency, so
this adds new public surface beyond the RFC — flagging for @jerm-dro and a possible RFC
follow-up before/with implementation.

Goal

Config in, server out — with decoration preserved (the RFC's supported extension
mechanism):

// Shared stateful collaborators (registry+watcher, telemetry, health) are built once
// and passed to BOTH derivations via opts; every other field each derives from vmcpCfg.
opts := []app.Option{app.WithVersion(v) /*, app.WithBackendRegistry(reg, watcher), ... */}

core, coreCleanup, err := app.BuildCore(ctx, vmcpCfg, opts...)
defer coreCleanup()

// the transport config is derived from vmcpCfg directly — NOT returned by BuildCore
serverCfg, srvCleanup, err := app.BuildServerConfig(ctx, vmcpCfg, opts...)
defer srvCleanup()

// decoration is the embedder's business, invisible to the server
srv, err := server.Serve(ctx, userScoped(core), serverCfg)

Dependency injection via core.New(&core.Config{...}) + server.Serve(ctx, v, &server.ServerConfig{...}) stays for tests and advanced embedders — both coexist.

Design (proposed)

• New OSS package — NOT root pkg/vmcp (it would import-cycle with core/server).
  Proposed pkg/vmcp/app (name TBD).
• Input is the existing serialized vmcpconfig.Config — already the single source
  of truth (Group, Backends, OutgoingAuth.Source: inline|discovered, aggregation,
  optimizer, incoming/outgoing auth, telemetry, audit, rate limiting, composite tools).
  No new parallel config type (avoids the go-style "parallel types that drift"
  anti-pattern). The assembly OWNS interpreting it — including choosing the discovery
  mechanism from OutgoingAuth.Source + Backends/Group — so the embedder no longer
  encodes those rules.
• Options carry the non-serialized runtime injectables that are CLI / composition-
  root concerns today: REST config (default in-cluster, reusing Public constructor for a Kubernetes-backed dynamic BackendRegistry (no pkg/vmcp/k8s import for embedders) #5541's
  backendregistry.WithRESTConfig), container factory (TEI optimizer), embedded auth
  server, version string, host/port, and an injected *slog.Logger (cf. vmcp: accept injected *slog.Logger via Config (avoid sharing slog.Default with the embedder) #5381).
• Two independent derivations from vmcpconfig.Config, mirroring the existing
  deriveCoreConfig/deriveServerConfig split (P3.1 deriveCoreConfig/deriveServerConfig config split #5444):
  • BuildCore derives core.Config from vmcpConfig (+ opts) and calls core.New,
    returning only the VMCP (for decoration). It does not return a
    ServerConfig — the transport config is not a byproduct of core construction.
  • BuildServerConfig derives *server.ServerConfig from vmcpConfig (+ opts)
    directly — an independent projection of the same input.
    Both reuse the existing deriveCoreConfig / deriveServerConfig / WithDefaults
    helpers (pkg/vmcp/server/derive.go, promoted to exported or relocated), so there is
    exactly one config-split path, shared with P3.2 Reduce server.New body to the wrapper #5445's reduced server.New.
• Shared stateful collaborators are built once and threaded into both derivations,
  never built twice: the TelemetryProvider (else duplicate OTEL pipelines), the
  *health.Monitor (else double health-check traffic / a divergent view), and — in
  dynamic mode — the BackendRegistry+watcher (else two informer caches). These are the
  override seams (Options): the caller (or the one-shot app.Serve) builds them once,
  passes the same opts to both calls, and owns their lifecycle/cleanup. Each Build*
  cleanup releases only what it acquired. This mirrors how deriveCoreConfig/
  deriveServerConfig already take backendRegistry/healthMon as explicit shared args.
• Reuse Public constructor for a Kubernetes-backed dynamic BackendRegistry (no pkg/vmcp/k8s import for embedders) #5541 for the discovered path; build the static/inline registry for
  Backends.

Scope

• app.BuildCore(ctx, *vmcpconfig.Config, ...Option) (core.VMCP, cleanup func(), error)
  — derives core.Config from vmcpConfig (+ opts) and calls core.New. Returns
  only the core (no ServerConfig). Reuses the P3.1 deriveCoreConfig/deriveServerConfig config split #5444 derive helpers and Public constructor for a Kubernetes-backed dynamic BackendRegistry (no pkg/vmcp/k8s import for embedders) #5541's
  registry constructor.
• app.BuildServerConfig(ctx, *vmcpconfig.Config, ...Option) (*server.ServerConfig, cleanup func(), error)
  — derives the transport config directly from vmcpConfig (+ opts), reusing the
  P3.1 deriveCoreConfig/deriveServerConfig config split #5444 derive helpers.
• Functional Options for the runtime injectables and the shared stateful
  collaborators (REST config, container factory, embedded auth server, telemetry
  provider, health monitor, backend registry+watcher, version, host/port, logger) —
  so the same opts passed to both Build* calls share one instance.
• Runnable example embedder: BuildCore + BuildServerConfig → decorate →
  server.Serve — extending the decorator example from P4.2 Runnable decorator example embedder #5447.
• Unit tests: each derivation yields a working core / ServerConfig for both
  discovered and inline discovery; options are applied; a shared collaborator
  passed to both is built once; each cleanup releases only what it acquired.
• (optional) one-shot convenience app.Serve(ctx, *vmcpconfig.Config, ...Option) (*server.Server, error)
  that builds the shared collaborators once, derives both configs, and serves — for
  the no-decoration case.
• Migrate pkg/vmcp/cli/serve.go to call the new assembly API (app.BuildCore +
  app.BuildServerConfig), replacing the hand-wired assembly in the OSS production
  serve path with a call through the new package. This is what actually removes the
  OSS duplication and proves the API is sufficient for the real caller. Coordinate
  with P3.2 Reduce server.New body to the wrapper #5445 to ensure a single shared config-split path.

Out of scope (tracked as follow-ups)

• Enterprise gateway adopting the API — separate repo / issue.
• Retiring the legacy server.Config god-object and the deriveCoreConfig /
  deriveServerConfig helpers once both server.New and cli.Serve route through the
  assembly.
• Any change to the serialized vmcpconfig.Config schema / CRD / YAML.
• A new programmatic vmcp.Config sum-type (e.g. Discovery: KubernetesDiscovery | StaticDiscovery): considered and rejected for now in favor of reusing
  vmcpconfig.Config, to avoid maintaining a second config type.

Dependencies / coordination

• Builds on Public constructor for a Kubernetes-backed dynamic BackendRegistry (no pkg/vmcp/k8s import for embedders) #5541 (registry seam) and the P3.1 deriveCoreConfig/deriveServerConfig config split #5444 derive helpers (closed).
• Coordinate with P3.2 Reduce server.New body to the wrapper #5445 (Phase 3 — reduces server.New to a wrapper over the same
  derive helpers): both must share a single config-split path.

References

• Blocker comment: Add public Kubernetes-backed dynamic BackendRegistry constructor #5542 (comment)
• Epic vMCP interface refactor #5419; RFC THV-0076
• Related: Phase 3: Reduce server.New to wrapper + config split #5432, Phase 4: Documentation + runnable decorator example #5433, P3.1 deriveCoreConfig/deriveServerConfig config split #5444, P3.2 Reduce server.New body to the wrapper #5445, P4.2 Runnable decorator example embedder #5447, Public constructor for a Kubernetes-backed dynamic BackendRegistry (no pkg/vmcp/k8s import for embedders) #5541, vmcp: accept injected *slog.Logger via Config (avoid sharing slog.Default with the embedder) #5381