Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 27 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# datumctl-inventory

A [datumctl](https://github.com/datum-cloud/datumctl) plugin that provides a
read view over the Datum Cloud physical inventory — providers, regions, sites,
clusters, and nodes — served by the [milo inventory
service](https://github.com/milo-os/inventory) (`inventory.miloapis.com/v1alpha1`).
Once installed it is invoked as `datumctl inventory ...`.
A [datumctl](https://github.com/datum-cloud/datumctl) plugin for the Datum Cloud
inventory, modeled as a **property graph**: typed `Node`s (Region, Site,
Cluster, Provider, Host, …) connected by typed `Edge`s (located-in, member-of,
provided-by, …), each carrying an attribute bag. The available types and their
attributes live in the `NodeType`/`EdgeType` schema registry. Served by the
[milo inventory service](https://github.com/milo-os/inventory)
(`graph.inventory.miloapis.com/v1alpha2`). Invoked as `datumctl inventory ...`.

## Install

Expand All @@ -19,41 +21,40 @@ exposes it as `datumctl inventory`.

| Command | Description |
|---|---|
| `datumctl inventory providers` | List providers |
| `datumctl inventory regions` | List regions |
| `datumctl inventory sites [--region R] [--provider P]` | List sites |
| `datumctl inventory clusters [--region R] [--site S]` | List clusters |
| `datumctl inventory nodes [--region R] [--site S] [--cluster C]` | List nodes |
| `datumctl inventory tree [--region R]` | region → site → node hierarchy |
| `datumctl inventory summary` | Fleet-wide counts |
| `datumctl inventory apply -f FILE [--dry-run=server]` | Create/update objects from a manifest |
| `datumctl inventory get <TYPE>` | List nodes of an asset class; columns derived from the NodeType schema |
| `datumctl inventory get edges [--type T] [--from N] [--to N]` | List edges (relationships) |
| `datumctl inventory types` | List the NodeType/EdgeType schema registry |
| `datumctl inventory neighbors NODE [--edge T] [--direction out\|in\|both]` | Nodes adjacent to NODE |
| `datumctl inventory tree [--edge T] [--root-type T]` | Containment hierarchy from edges (default: `located-in`, rooted at `Region`) |
| `datumctl inventory summary` | Counts per node type and edge type |
| `datumctl inventory apply -f FILE [--dry-run=server]` | Create/update graph objects from a manifest |

The list subcommands accept `-o table|json|yaml` (default `table`).
The `get`, `types`, `summary` commands accept `-o table|json|yaml` (default `table`).

`--region`, `--site`, and `--cluster` filter server-side using the
`topology.inventory.miloapis.com/*` labels the inventory controllers propagate
onto objects. `--provider` filters on the site's `providerRef`.
Relationships that were typed fields in the old model (a site's region, a
node's cluster) are now **edges** — query them with `get edges`, `neighbors`,
or `tree` rather than as columns.

## Populating the inventory

`apply` is an idempotent, declarative upsert for inventory objects — for
loading the inventory from declared configuration, not fleet management:
`apply` is an idempotent, declarative upsert of graph objects — for loading the
inventory from declared configuration, not fleet management:

```sh
# Apply a manifest (objects land in dependency order: provider, region,
# site, cluster, node — regardless of order in the file)
datumctl inventory apply -f fleet.yaml
# Apply a manifest (objects land in dependency order: node/edge types first,
# then nodes, then edges — regardless of order in the file)
datumctl inventory apply -f graph.yaml

# Pipe from a renderer
render-fleet | datumctl inventory apply -f -

# Validate against the server without persisting
datumctl inventory apply -f fleet.yaml --dry-run=server
datumctl inventory apply -f graph.yaml --dry-run=server
```

It uses server-side apply with field manager `datumctl-inventory`, so
re-applying the same manifest makes no changes. Only `Provider`, `Region`,
`Site`, `Cluster`, and `Node` are accepted.
re-applying the same manifest makes no changes. Only `NodeType`, `EdgeType`,
`Node`, and `Edge` are accepted.

Inventory objects are cluster-scoped on the Datum Cloud platform root, so the
plugin talks to the platform API directly and takes no organization or project
Expand All @@ -65,7 +66,7 @@ datumctl injects context via environment variables and execs the plugin. The
plugin reads `DATUM_API_HOST`, fetches a short-lived token through the
credentials helper (`plugin.Token()`), and builds a controller-runtime client
against the platform root using the milo inventory project's published typed
API (`go.miloapis.com/inventory/api/v1alpha1`). See the
API (`go.miloapis.com/inventory/api/v1alpha2`). See the
[datumctl plugin docs](https://github.com/datum-cloud/datumctl/blob/main/docs/developer/plugins.md).

This split keeps Datum's CLI surface in `datum-cloud/` while depending on the
Expand Down
41 changes: 25 additions & 16 deletions apply.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,26 +4,28 @@ package main

import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"sort"
"strings"

"github.com/spf13/cobra"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/serializer"
utilyaml "k8s.io/apimachinery/pkg/util/yaml"
"sigs.k8s.io/controller-runtime/pkg/client"

inventoryv1alpha1 "go.miloapis.com/inventory/api/v1alpha1"
inventoryv1alpha2 "go.miloapis.com/inventory/api/v1alpha2"
)

const fieldManager = "datumctl-inventory"

// applyOrder lists the inventory kinds apply handles, in dependency order:
// parents are applied before the children that reference them.
var applyOrder = []string{"Provider", "Region", "Site", "Cluster", "Node"}
// applyOrder lists the graph kinds apply handles, in dependency order: the
// type registry first, then nodes, then the edges that reference nodes.
var applyOrder = []string{"NodeType", "EdgeType", "Node", "Edge"}

func kindOrder(kind string) (int, bool) {
for i, k := range applyOrder {
Expand All @@ -39,14 +41,14 @@ func newApplyCmd() *cobra.Command {
var dryRun string
cmd := &cobra.Command{
Use: "apply -f FILE",
Short: "Create or update inventory objects from a manifest",
Long: `Create or update inventory objects (providers, regions, sites, clusters,
nodes) from a YAML or JSON manifest.
Short: "Create or update inventory graph objects from a manifest",
Long: `Create or update inventory graph objects (NodeType, EdgeType, Node, Edge)
from a YAML or JSON manifest.

apply is an idempotent, declarative upsert: re-applying the same manifest makes
no changes. Objects are applied in dependency order (providers, then regions,
then sites, then clusters, then nodes) so a single mixed manifest lands cleanly.
It uses server-side apply with field manager "datumctl-inventory".
no changes. Objects are applied in dependency order (node/edge types first, then
nodes, then the edges that reference them) so a single mixed manifest lands
cleanly. It uses server-side apply with field manager "datumctl-inventory".

This is for populating the inventory from declared configuration — not fleet
management. Inventory lives on the Datum Cloud platform root, so apply takes no
Expand Down Expand Up @@ -106,7 +108,7 @@ type applyObj struct {
// kinds apply does not handle.
func readManifests(stdin io.Reader, files []string) ([]applyObj, error) {
scheme := runtime.NewScheme()
if err := inventoryv1alpha1.AddToScheme(scheme); err != nil {
if err := inventoryv1alpha2.AddToScheme(scheme); err != nil {
return nil, fmt.Errorf("build scheme: %w", err)
}
decoder := serializer.NewCodecFactory(scheme).UniversalDeserializer()
Expand All @@ -130,15 +132,22 @@ func readManifests(stdin io.Reader, files []string) ([]applyObj, error) {
if len(raw.Raw) == 0 {
continue
}
obj, gvk, derr := decoder.Decode(raw.Raw, nil, nil)
if derr != nil {
// Check the kind before typed decode so an unsupported kind gets a
// helpful message rather than the scheme's "not registered" error.
var tm metav1.TypeMeta
if derr := json.Unmarshal(raw.Raw, &tm); derr != nil {
closeFn()
return nil, fmt.Errorf("decode %s: %w", f, derr)
return nil, fmt.Errorf("parse %s: %w", f, derr)
}
order, ok := kindOrder(gvk.Kind)
order, ok := kindOrder(tm.Kind)
if !ok {
closeFn()
return nil, fmt.Errorf("unsupported kind %q in %s (apply handles: %s)", gvk.Kind, f, strings.Join(applyOrder, ", "))
return nil, fmt.Errorf("unsupported kind %q in %s (apply handles: %s)", tm.Kind, f, strings.Join(applyOrder, ", "))
}
obj, gvk, derr := decoder.Decode(raw.Raw, nil, nil)
if derr != nil {
closeFn()
return nil, fmt.Errorf("decode %s: %w", f, derr)
}
co, ok := obj.(client.Object)
if !ok {
Expand Down
73 changes: 36 additions & 37 deletions apply_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -9,35 +9,32 @@ import (
)

const sampleManifest = `
apiVersion: inventory.miloapis.com/v1alpha1
kind: Node
apiVersion: graph.inventory.miloapis.com/v1alpha2
kind: Edge
metadata:
name: node-a
name: site-dfw1-in-uc
spec:
siteRef:
name: us-central-1a
hardware:
cpuCores: 8
cpuArchitecture: amd64
memoryBytes: 1073741824
type: located-in
from:
name: site-dfw1
to:
name: region-uc
---
apiVersion: inventory.miloapis.com/v1alpha1
kind: Provider
apiVersion: graph.inventory.miloapis.com/v1alpha2
kind: NodeType
metadata:
name: netactuate
name: Site
spec:
displayName: NetActuate
type: Hosting
displayName: Site
---
apiVersion: inventory.miloapis.com/v1alpha1
kind: Site
apiVersion: graph.inventory.miloapis.com/v1alpha2
kind: Node
metadata:
name: us-central-1a
name: site-dfw1
spec:
displayName: Dallas
type: AvailabilityZone
regionRef:
name: us-central-1
type: Site
attributes:
displayName: Dallas
`

func TestReadManifestsParsesAndOrders(t *testing.T) {
Expand All @@ -49,14 +46,13 @@ func TestReadManifestsParsesAndOrders(t *testing.T) {
t.Fatalf("got %d objects, want 3", len(objs))
}
sort.SliceStable(objs, func(i, j int) bool { return objs[i].order < objs[j].order })
gotKinds := []string{objs[0].kind, objs[1].kind, objs[2].kind}
want := []string{"Provider", "Site", "Node"}
got := []string{objs[0].kind, objs[1].kind, objs[2].kind}
want := []string{"NodeType", "Node", "Edge"}
for i := range want {
if gotKinds[i] != want[i] {
t.Errorf("order[%d] = %s, want %s (full: %v)", i, gotKinds[i], want[i], gotKinds)
if got[i] != want[i] {
t.Errorf("order[%d] = %s, want %s (full: %v)", i, got[i], want[i], got)
}
}
// GVK must be set on each object so server-side apply has apiVersion/kind.
for _, o := range objs {
if o.obj.GetObjectKind().GroupVersionKind().Kind == "" {
t.Errorf("%s/%s missing GVK", o.kind, o.obj.GetName())
Expand All @@ -66,10 +62,10 @@ func TestReadManifestsParsesAndOrders(t *testing.T) {

func TestReadManifestsRejectsUnsupportedKind(t *testing.T) {
const m = `
apiVersion: inventory.miloapis.com/v1alpha1
kind: Rack
apiVersion: graph.inventory.miloapis.com/v1alpha2
kind: Widget
metadata:
name: rack-a
name: w
`
_, err := readManifests(strings.NewReader(m), []string{"-"})
if err == nil || !strings.Contains(err.Error(), "unsupported kind") {
Expand All @@ -88,15 +84,18 @@ func TestReadManifestsEmpty(t *testing.T) {
}

func TestKindOrder(t *testing.T) {
if _, ok := kindOrder("Provider"); !ok {
t.Error("Provider should be ordered")
nt, ok := kindOrder("NodeType")
if !ok {
t.Fatal("NodeType should be ordered")
}
e, ok := kindOrder("Edge")
if !ok {
t.Fatal("Edge should be ordered")
}
p, _ := kindOrder("Provider")
n, _ := kindOrder("Node")
if !(p < n) {
t.Errorf("Provider (%d) should sort before Node (%d)", p, n)
if !(nt < e) {
t.Errorf("NodeType (%d) should sort before Edge (%d)", nt, e)
}
if _, ok := kindOrder("Rack"); ok {
t.Error("Rack should not be applyable")
if _, ok := kindOrder("Widget"); ok {
t.Error("Widget should not be applyable")
}
}
4 changes: 2 additions & 2 deletions client.go
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ import (
"sigs.k8s.io/controller-runtime/pkg/client"

"go.datum.net/datumctl/plugin"
inventoryv1alpha1 "go.miloapis.com/inventory/api/v1alpha1"
inventoryv1alpha2 "go.miloapis.com/inventory/api/v1alpha2"
)

// newClient builds a controller-runtime client against the Datum Cloud platform
Expand All @@ -33,7 +33,7 @@ func newClient() (client.Client, error) {
}

scheme := runtime.NewScheme()
if err := inventoryv1alpha1.AddToScheme(scheme); err != nil {
if err := inventoryv1alpha2.AddToScheme(scheme); err != nil {
return nil, fmt.Errorf("build scheme: %w", err)
}

Expand Down
Loading