Reintroduce bundled database to the chart (#365)

Resolves #358

# Description

Reintroduces a bundled PostgreSQL instance to the Helm chart and
restructures the `database` values key to align with [kagent PR
#1527](kagent-dev/kagent#1527).

**What changed:**

- **Bundled PostgreSQL on by default** — `postgres:18` is deployed
alongside Agent Registry when `database.postgres.bundled.enabled=true`
(the default). Intended for development and evaluation only; a warning
is shown in NOTES.txt.
- **Restructured `database` values** — replaced the old
`database.bundled.*` / `database.external.*` split with a flat
`database.postgres.*` structure:
- `database.postgres.url` / `database.postgres.urlFile` — external
connection string (urlFile takes precedence)
- `database.postgres.bundled.*` — bundled instance config (image,
storage, resources)
- `database.postgres.vectorEnabled` — enables
`AGENT_REGISTRY_EMBEDDINGS_ENABLED` and
`AGENT_REGISTRY_DATABASE_VECTOR_ENABLED` on the app pod
- **Default image is `postgres:18`** (official, plain). Users who want
semantic search can opt in via `database.postgres.vectorEnabled=true`
with their own pgvector-capable PostgreSQL instance.
- **Conditional vector migrations** — base schema no longer requires
pgvector. Vector schema (embedding columns, HNSW indexes) is applied in
a separate migration set, only when `DatabaseVectorEnabled=true`.
- **URL file support** — `AGENT_REGISTRY_DATABASE_URL_FILE` env var /
`database.postgres.urlFile` value reads the database URL from a file at
startup.
- **Separate PostgreSQL Secret** — bundled DB password lives in its own
`{release}-postgresql` Secret, decoupled from the JWT key Secret.
- **Init container only when needed** — the `wait-for-postgres` init
container is skipped when `url` or `urlFile` is set, even if
`bundled.enabled=true`.
- **Simplified helpers** — removed `agentregistry.secretName`,
`agentregistry.passwordSecretName`, and `agentregistry.databaseUrl`;
logic inlined into templates. Removed `global.existingSecret`.
- **Updated docs** — `README.md`, `DEVELOPMENT.md`, chart
`README.md.gotmpl`, and `NOTES.txt` updated for accuracy.
- **Helm unit tests** — new `postgresql_test.yaml` suite;
`deployment_test.yaml`, `secrets_test.yaml`, and `validation_test.yaml`
updated for the new values structure.

# Change Type

/kind feature
/kind install

# Changelog

```release-note
Reintroduces a bundled PostgreSQL database option to the Helm chart with a new database.postgres.* values structure. The bundled database (postgres:18) is enabled by default for development and evaluation. Set database.postgres.bundled.enabled=false and supply database.postgres.url for production.
```

Author: nikolasmatt

.env.example

@@ -4,7 +4,7 @@ AGENT_REGISTRY_SERVER_ADDRESS=:8080
 
 # Database Configuration
 # PostgreSQL connection string
-AGENT_REGISTRY_DATABASE_URL=postgres://localhost:5432/agent-registry?sslmode=disable
+AGENT_REGISTRY_DATABASE_URL=postgres://localhost:5432/agentregistry?sslmode=disable
 
 # Seed Configuration
 # Path to seed data file (optional)

DEVELOPMENT.md

@@ -2,7 +2,7 @@
 
 ## Local Kubernetes Environment
 
-The fastest way to run the full stack locally is with [Kind](https://kind.sigs.k8s.io/). A single `make` target creates the cluster, deploys PostgreSQL/pgvector, builds the server image, and installs AgentRegistry via Helm.
+The fastest way to run the full stack locally is with [Kind](https://kind.sigs.k8s.io/). A single `make` target creates the cluster, builds the server image, and installs AgentRegistry via Helm — a PostgreSQL instance with pgvector is bundled and deployed automatically by the Helm chart for local development.
 
 ### Prerequisites
 
@@ -19,13 +19,12 @@ The fastest way to run the full stack locally is with [Kind](https://kind.sigs.k
 make setup-kind-cluster
 ```
 
-This runs three steps in order:
+This runs two steps in order:
 
 | Step | Target | What it does |
 |------|--------|-------------|
 | 1 | `create-kind-cluster` | Installs `kind` to `./bin/`, creates Kind cluster + local registry (`localhost:5001`) + MetalLB |
-| 2 | `install-postgresql` | Deploys `pgvector/pgvector:0.8.2-pg16` into the `agentregistry` namespace |
-| 3 | `install-agentregistry` | Builds server image, pushes to local registry, Helm installs AgentRegistry |
+| 2 | `install-agentregistry` | Builds server image, pushes to local registry, Helm installs AgentRegistry (bundled PostgreSQL with pgvector override for local dev) |
 
 Each target can also be run independently — useful when iterating on code:
 
@@ -48,9 +47,9 @@ On subsequent runs, `install-agentregistry` reuses the `jwtPrivateKey` already s
 kubectl --context kind-agentregistry port-forward -n agentregistry svc/agentregistry 12121:12121
 # open http://localhost:12121
 
-# PostgreSQL (for direct inspection)
-kubectl --context kind-agentregistry port-forward -n agentregistry svc/postgres-pgvector 5432:5432
-psql -h localhost -U agentregistry -d agent-registry
+# Bundled PostgreSQL (for direct inspection)
+kubectl --context kind-agentregistry port-forward -n agentregistry svc/agentregistry-postgresql 5432:5432
+psql -h localhost -U agentregistry -d agentregistry
 ```
 
 ### Teardown
@@ -111,208 +110,30 @@ The UI is available at `http://localhost:12121`.
 
 # Architecture Overview
 
-### 1. CLI Layer (cmd/)
+**Tech stack:** Go 1.25+ · PostgreSQL + pgvector (pgx) · [Huma](https://huma.rocks/) (OpenAPI) · [Cobra](https://cobra.dev/) (CLI) · Next.js 14 (App Router) · Tailwind CSS · shadcn/ui
 
-Built with [Cobra](https://github.com/spf13/cobra), provides all command-line functionality:
-
-- **Registry Management**: connect, disconnect, refresh
-- **Resource Discovery**: list, search, show
-- **Installation**: install, uninstall
-- **Configuration**: configure clients
-- **UI**: launch web interface
-
-Each command has placeholder implementations ready to be filled with actual logic.
-
-### 2. Data Layer (internal/database/)
-
-Uses **SQLite** for local storage:
-
-**Tables:**
-- `registries` - Connected registries
-- `servers` - MCP servers from registries
-- `skills` - Skills from registries
-- `installations` - Installed resources
-
-**Location:** `~/.arctl/arctl.db`
-
-The schema is based on the MCP Registry JSON schema provided, supporting the full `ServerDetail` structure.
-
-### 3. API Layer (internal/api/)
-
-Built with [Gin](https://github.com/gin-gonic/gin), provides REST API:
-
-**Endpoints:**
-- `GET /api/health` - Health check
-- `GET /api/registries` - List registries
-- `GET /api/servers` - List MCP servers
-- `GET /api/skills` - List skills
-- `GET /api/installations` - List installed resources
-- `GET /*` - Serve embedded UI
-
-**Port:** 8080 (configurable with `--port`)
-
-### 4. UI Layer (ui/)
-
-Built with:
-- **Framework:** Next.js 14 (App Router)
-- **Language:** TypeScript
-- **Styling:** Tailwind CSS
-- **Components:** shadcn/ui
-- **Icons:** Lucide React
-
-**Features:**
-- Dashboard with statistics
-- Resource browser (registries, MCP servers, skills)
-- Real-time data from API
-- Responsive design
-- Installation status indicators
-
-**Build Output:** Static files exported to `internal/registry/api/ui/dist/`
-
-## Data Flow
-
-### CLI Command Execution
-
-```
-User Input
-    ↓
-Cobra Command (cmd/)
-    ↓
-Business Logic (TODO)
-    ↓
-Database Layer (internal/database/)
-    ↓
-SQLite (~/.arctl/arctl.db)
-```
-
-### Web UI Request
-
-```
-Browser Request
-    ↓
-Gin Router (internal/api/)
-    ↓
-API Handler
-    ↓
-Database Query
-    ↓
-JSON Response
-    ↓
-React Component (ui/)
-    ↓
-User Interface
-```
-
-## Embedding Strategy
-
-### How It Works
-
-1. **Build Phase** (`make build-ui`):
-   - Next.js builds static files
-   - Output goes to `internal/registry/api/ui/dist/`
-
-2. **Compile Phase** (`make build-cli`):
-   - Go's `embed` directive includes entire `ui/dist/` directory
-   - Files become part of the binary
-
-3. **Runtime Phase** (`./bin/arctl ui`):
-   - Gin serves files from embedded FS
-   - No external dependencies needed
-
-### Embed Directive
-
-```go
-//go:embed ui/dist/*
-var embeddedUI embed.FS
-```
-
-This embeds all files in `internal/registry/api/ui/dist/` at compile time.
+For a detailed breakdown of layers, conventions, and contribution guidelines see [`AGENTS.md`](AGENTS.md).
 
 ## Build Process
 
-### Development
-
 ```bash
 # UI only (hot reload)
 make dev-ui
 
-# CLI only (quick iteration)
-go build -o bin/arctl main.go
-```
-
-### Production
+# Build CLI binary
+make build-cli
 
-```bash
-# Full build with embedding
-make build
+# Build server binary
+make build-server
 
-# Creates: ./bin/arctl (single binary with UI embedded)
+# Build UI static assets
+make build-ui
 ```
 
-## Extension Points
-
-### Adding a New CLI Command
-
-1. Create `cmd/mycommand.go`
-2. Define the command with Cobra
-3. Add to `rootCmd` in `init()`
-4. Implement logic (call database layer)
-
-### Adding a New API Endpoint
-
-1. Add handler in `internal/api/server.go`
-2. Register route in `StartServer()`
-3. Call database layer
-4. Return JSON response
-
-### Adding a New UI Page
-
-1. Create `ui/app/mypage/page.tsx`
-2. Fetch data from `/api/*` endpoints
-3. Use shadcn components for UI
-4. Rebuild with `make build-ui`
-
-### Adding Database Tables
-
-1. Update schema in `internal/database/database.go`
-2. Add model in `internal/models/models.go`
-3. Add query methods in database package
-4. Database auto-migrates on first run
-
-## Security Considerations
-
-### Database
-
-- Stored in user's home directory (`~/.arctl/`)
-- No network access
-- File permissions: 0755 (directory), default (file)
-
-### API Server
-
-- Localhost only by default
-- CORS not configured (local use)
-- No authentication (local tool)
-
-### Embedded UI
-
-- Static files only
-- No server-side execution
-- Served from memory (embedded)
-
-## Contributing
-
-When adding features:
-
-1. Add placeholder implementations first
-2. Create tests (TODO)
-3. Update documentation
-4. Rebuild with `make build`
-5. Test the binary
-
 ## Resources
 
 - [Cobra Documentation](https://cobra.dev/)
-- [Gin Documentation](https://gin-gonic.com/)
+- [Huma Documentation](https://huma.rocks/)
 - [Next.js Documentation](https://nextjs.org/docs)
 - [shadcn/ui Components](https://ui.shadcn.com/)
 - [MCP Protocol Specification](https://spec.modelcontextprotocol.io/)

Makefile

@@ -343,11 +343,6 @@ kind-debug: ## Shell into Kind control-plane and run btop for resource monitorin
 	docker exec -it $(KIND_CLUSTER_NAME)-control-plane bash -c 'apt-get update -qq && apt-get install -y --no-install-recommends btop htop'
 	docker exec -it $(KIND_CLUSTER_NAME)-control-plane bash -c 'btop --utf-force'
 
-.PHONY: install-postgresql
-install-postgresql: ## Deploy standalone PostgreSQL/pgvector into the Kind cluster
-	kubectl --context $(KIND_CLUSTER_CONTEXT) apply -f examples/postgres-pgvector.yaml
-	kubectl --context $(KIND_CLUSTER_CONTEXT) -n agentregistry wait --for=condition=ready pod -l app=postgres-pgvector --timeout=120s
-
 BUILD ?= true
 
 .PHONY: install-agentregistry
@@ -365,14 +360,15 @@ endif
 	    --create-namespace \
 	    --set image.pullPolicy=Always \
 	    --set image.registry=$(DOCKER_REGISTRY) \
-	    --set image.repository=$(DOCKER_REPO)/server \
+	    --set image.repository=$(DOCKER_REPO) \
 	    --set image.tag=$(VERSION) \
-	    --set database.host=postgres-pgvector.$(KIND_NAMESPACE).svc.cluster.local \
-	    --set database.password=agentregistry \
-	    --set database.sslMode=disable \
 	    --set config.jwtPrivateKey="$$JWT_KEY" \
 	    --set config.enableAnonymousAuth="true" \
 	    --set service.type=LoadBalancer \
+	    --set database.postgres.bundled.image.repository=pgvector \
+	    --set database.postgres.bundled.image.name=pgvector \
+	    --set database.postgres.bundled.image.tag=pg16 \
+	    --set database.postgres.vectorEnabled=true \
 	    --wait \
 	    --timeout=5m;
 
@@ -412,9 +408,9 @@ install-kagent-controller: ## Deploy kagent controller (minimal, no agents/tools
 	  --wait \
 	  --timeout=5m
 
-## Set up a full local K8s dev environment (Kind + PostgreSQL/pgvector + AgentRegistry + kagent).
+## Set up a full local K8s dev environment (Kind + AgentRegistry with bundled PostgreSQL + kagent).
 .PHONY: setup-kind-cluster
-setup-kind-cluster: create-kind-cluster install-postgresql install-kagent install-agentregistry ## Set up the full local Kind development environment
+setup-kind-cluster: create-kind-cluster install-kagent install-agentregistry ## Set up the full local Kind development environment
 
 .PHONY: dump-kind-state
 dump-kind-state: ## Dump Kind cluster state for debugging (pods, events, kagent logs)
@@ -425,7 +421,12 @@ dump-kind-state: ## Dump Kind cluster state for debugging (pods, events, kagent
 	@kubectl get pods -A --context $(KIND_CLUSTER_CONTEXT) 2>/dev/null || true
 	@echo ""
 	@echo "=== Pod describe ==="
-	@kubectl describe pods --context $(KIND_CLUSTER_CONTEXT) 2>/dev/null || true
+	@kubectl describe pods -A --context $(KIND_CLUSTER_CONTEXT) 2>/dev/null || true
+	@echo ""
+	@echo "=== AgentRegistry logs ==="
+	@kubectl logs deployment/agentregistry -n $(KIND_NAMESPACE) --context $(KIND_CLUSTER_CONTEXT) --tail=100 2>/dev/null || true
+	@echo "=== AgentRegistry previous logs ==="
+	@kubectl logs deployment/agentregistry -n $(KIND_NAMESPACE) --context $(KIND_CLUSTER_CONTEXT) --tail=100 --previous 2>/dev/null || true
 	@echo ""
 	@echo "=== Events ==="
 	@kubectl get events -A --sort-by='.lastTimestamp' --context $(KIND_CLUSTER_CONTEXT) 2>/dev/null | tail -50 || true
@@ -538,9 +539,7 @@ charts-render-test: charts-deps ## Render chart templates as a smoke test
 	@echo "Rendering chart templates for $(HELM_CHART_DIR)..."
 	$(HELM) template test-release $(HELM_CHART_DIR) \
 	  --values $(HELM_CHART_DIR)/values.yaml \
-	  --set config.jwtPrivateKey=deadbeef1234567890abcdef12345678 \
-	  --set database.password=ci-password \
-	  --set database.host=postgres.example.com
+	  --set config.jwtPrivateKey=deadbeef1234567890abcdef12345678
 
 # Package the chart into $(HELM_PACKAGE_DIR)/.
 .PHONY: charts-package

README.md

@@ -189,6 +189,8 @@ When you run `arctl deploy`, agentregistry automatically configures the gateway
 | [MCP Go SDK](https://github.com/modelcontextprotocol/go-sdk) | Go SDK for building MCP servers |
 | [Model Context Protocol](https://modelcontextprotocol.io/) | The open standard for AI-to-tool communication |
 
+> **Semantic search** requires a PostgreSQL instance with the pgvector extension. It is disabled by default. To enable it, ensure your database has pgvector support and set `AGENT_REGISTRY_DATABASE_VECTOR_ENABLED=true` (docker-compose / `.env`) or `--set database.postgres.vectorEnabled=true` (Helm).
+
 ---
 
 ## Community

charts/agentregistry/Chart-template.yaml

@@ -26,3 +26,5 @@ annotations:
   artifacthub.io/images: |
     - name: agentregistry
       image: ghcr.io/agentregistry-dev/agentregistry/server:${CHART_VERSION}
+    - name: postgresql
+      image: docker.io/library/postgres:18