Cosmos

Cosmos is a BGP control plane for Kubernetes. You define instances, sessions, advertisements, and VPCs as Kubernetes resources; the controller reconciles them into running FRR and GoBGP daemons on each node and programs learned routes into the kernel via netlink.

API groups: bgp.miloapis.com/v1alpha1 · providers.bgp.miloapis.com/v1alpha1 · vpc.miloapis.com/v1alpha1

---

How it works

Cosmos runs as a DaemonSet in the bgp-system namespace. Each node runs two BGP planes:

• Underlay — FRR handles IPv6 unicast sessions to top-of-rack switches
• Overlay — GoBGP handles VPNv4/VPNv6 sessions to regional route reflectors

At startup the controller auto-bootstraps a BGPProvider resource for each daemon. These providers are the unit of targeting for all other resources.

Cosmos operates in a multi-cluster model:

1. A management cluster holds BGPSession and BGPExternalPeer resources. The management cluster cosmos resolves peer addresses and writes fully self-contained session specs.
2. Karmada propagates BGPSession resources to POP and infra clusters.
3. On each member cluster, the SessionReconciler generates BGPPeer resources from the propagated sessions, and the PeerReconciler configures the daemons over gRPC.

The controller has no built-in knowledge of datacenter topology — all topology is expressed through CRDs and Karmada propagation policies.

---

Key features

• Dual-plane per node. FRR underlay (IPv6 unicast) and GoBGP overlay (VPNv4/VPNv6) managed independently.
• Provider abstraction. BGPProvider resources decouple topology from daemon type; FRR and GoBGP are both supported.
• Auto-bootstrapped providers. No manual daemon registration; the controller creates BGPProvider resources at startup.
• Multi-cluster propagation. Sessions are written once in the management cluster and distributed by Karmada.
• CNI-independent. Works with any CNI or none at all.
• VPC primitives. VPC and VPCAttachment CRDs model virtual networks and their interface bindings.

---

Prerequisites

• Multi-cluster Kubernetes with IPv6 enabled (nodes need global-scope IPv6 addresses)
• Karmada for resource propagation (management → POP/infra clusters)
• kubectl configured for your clusters
• Container images accessible to your clusters:
  • ghcr.io/milo-os/cosmos:latest — controller
  • An FRR image with northbound gRPC enabled
  • A GoBGP daemon image

---

Quick start

Install the CRDs and deploy the controller on a member cluster:

kubectl apply -k config/crd
kubectl apply -k config/deploy

After the DaemonSet pods become ready, verify that BGPProvider resources were auto-bootstrapped (one per daemon per node):

kubectl get bgpproviders

Create a BGPInstance to configure one BGP plane. The providerSelector targets the auto-bootstrapped providers for that plane:

apiVersion: bgp.miloapis.com/v1alpha1
kind: BGPInstance
metadata:
  name: underlay
spec:
  providerSelector:
    matchLabels:
      bgp.miloapis.com/daemon: frr
  asNumber: 65000
  addressFamilies:
    - afi: IPv6
      safi: Unicast

In the management cluster, create a BGPSession to establish peering. Sessions are propagated to member clusters by Karmada:

apiVersion: bgp.miloapis.com/v1alpha1
kind: BGPSession
metadata:
  name: node-1-to-tor-1
spec:
  fromProviderSelector:
    matchLabels:
      bgp.miloapis.com/node: node-1
      bgp.miloapis.com/daemon: frr
  fromInstanceRef: underlay
  toPeers:
    - address: "2001:db8:fabric::1"
      asNumber: 65000
      instanceRef: underlay
  addressFamilies:
    - afi: IPv6
      safi: Unicast

After Karmada propagation, verify generated peer resources and session state:

kubectl get bgppeers

For a complete walkthrough, see the Getting Started guide.

---

API reference

BGP — bgp.miloapis.com/v1alpha1

Resource	Short name	Description
BGPInstance	bgpi	Speaker identity: AS number, address families, timers. Targets providers by selector.
BGPExternalPeer	bgpep	Registry entry for peers outside the cosmos-managed fleet. Management cluster only.
BGPPeer	bgppr	Per-provider peer configuration. Generated by SessionReconciler. Never written directly.
BGPSession	bgps	Bilateral session intent. Written in management cluster; propagated via Karmada.
BGPAdvertisement	bgpadv	Infrastructure prefix advertisement (loopbacks, SRv6 locators). Not for workload routes.
BGPRoutePolicy	bgprp	Import/export route filtering applied to matched BGPPeer resources.

Providers — providers.bgp.miloapis.com/v1alpha1

Resource	Short name	Description
BGPProvider	bgpp	One BGP daemon instance (FRR or GoBGP). Auto-bootstrapped at startup.

VPC — vpc.miloapis.com/v1alpha1

Resource	Description
VPC	Virtual network with one or more CIDR prefixes.
VPCAttachment	Binds a VPC to a named interface with assigned addresses.

---

Development

Install development tools first:

task tools

Command	Description
task build	Compile all packages
task test	Run unit tests
task lint	Run golangci-lint
task lint-fix	Run golangci-lint and apply auto-fixes
task vet	Run go vet
task fmt	Run go fmt
task generate	Regenerate deepcopy methods
task manifests	Regenerate CRD manifests from API types
task image	Build the container image
task image-push	Build and push the container image
task test-e2e	Create a kind cluster, deploy, run the full E2E suite, and tear down
task ci	Run the full CI pipeline locally (build, vet, unit tests, and e2e)
task clean	Remove build artifacts and temporary files

E2E tests use Chainsaw and run against a kind cluster. The full suite also runs in CI on every pull request.

Build the container image:

docker build -f build/Dockerfile -t ghcr.io/milo-os/cosmos:dev .

---

Documentation

• Getting started — deploy and establish your first BGP sessions
• API reference — full CRD field definitions, conditions, and operational contracts
• Service design — architecture, controller internals, and design decisions
• Enhancements — accepted design proposals

Examples

• BGPProvider (auto-bootstrapped)
• BGPInstance — underlay FRR
• BGPInstance — overlay GoBGP
• BGPInstance — infra route reflector
• BGPExternalPeer — ToR switch
• BGPSession — underlay to ToR
• BGPSession — overlay to route reflectors
• BGPSession — RR client (infra)
• Rejected configurations

---

License

Apache 2.0