feat: local k3d environment and workflow improvements#373
Open
cybersiddhu wants to merge 76 commits intodevelopfrom
Open
feat: local k3d environment and workflow improvements#373cybersiddhu wants to merge 76 commits intodevelopfrom
cybersiddhu wants to merge 76 commits intodevelopfrom
Conversation
Introduce a new Justfile module for managing k3d clusters. This allows for easy creation, deletion, and listing of a local, single-node k3d cluster, including persistent storage mapping for development purposes.
…mmand Allow specifying the k3s image version during cluster creation to control the Kubernetes version used. Add a new command to display cluster information using kubectl, which is useful for debugging and verification.
Add a new recipe to export the kubeconfig file for a specified k3d cluster.
… variables Centralizing configuration variables like cluster name, image, host path, and port forwarding into module variables makes the `k3d.justfile` cleaner and easier to maintain.
Introduce a new Just recipe to dump a remote ArangoDB database. This allows developers to easily create local backups of remote databases by setting up a temporary port-forward and using the official ArangoDB Docker image for the dump operation.
Introduce commands to manage the lifecycle of k3d clusters, allowing users to stop, start, or restart a cluster by name.
Add pulumi-local-setup for filesystem-based state storage and setup-local-stack for creating stacks with passphrase secrets. Includes safe quoting of all interpolated parameters, exact-match stack existence check via JSON output, and pulumi-management group annotations for both recipes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The default path for storing Pulumi local state is changed from "scratch/pulumi-state" to "pulumi-files/local-state" to better organize project files and separate infrastructure state from temporary scratch files.
The default output filename for exporting the kubeconfig is changed from 'kubeconfig.yaml' to 'k3d-dev-cluster.yaml' to provide a more descriptive and specific name for the development cluster configuration.
Prevents local development artifacts, specifically the k3d cluster configuration and local Claude settings, from being accidentally committed to the repository.
…dule The logic for setting up the Pulumi local backend and initializing local stacks is moved from `pulumi.justfile` to a new `local-pulumi.justfile`. This refactoring organizes the build scripts by separating cloud-specific (GCP) Pulumi management from local development Pulumi management, improving modularity and maintainability.
… stack commands The existing commands for managing local Pulumi stacks are renamed and refactored to better align with conventional naming and improve usability. A new `new-stack` command is introduced to create a stack from scratch, securely fetching the passphrase from `pass`. The existing command for copying configuration is renamed to `new-stack-from` and is also updated to use the passphrase securely and explicitly specify the Pulumi project directory in all `pulumi` calls.
Prevents accidental staging and committing of local Pulumi state files, which should remain local to the development environment.
…projects locally The `new-project` command simplifies the creation of a new Pulumi Go project using the local backend, including setting up the passphrase from `pass`. The stack creation commands (`new-stack` and `new-stack-from`) now explicitly check if the target project folder exists before attempting to run Pulumi commands, preventing cryptic errors if the project directory is missing. The `pulumi-local-setup` command is now chained to the new stack/project commands to ensure the local backend is initialized before any Pulumi operations.
Prevents accidental staging and committing of local Pulumi state files which should remain local to the project structure.
…rces Introduce `create-resource` and `remove-resource` commands in `local-pulumi.justfile` to manage Pulumi resources for a local stack. These commands securely retrieve the stack passphrase using `pass` and set it as an environment variable before running `pulumi up` or `pulumi destroy`.
Adds a new Pulumi project named `namespace-bootstrap` written in Go. This project is configured to create a Kubernetes Namespace based on a configuration value read from Pulumi stack configuration.
Introduce a new Pulumi configuration file for local storage provisioning. This allows for easy setup of a local path provisioner with a specific disk type and name when deploying infrastructure locally using Pulumi.
Introduce the necessary files and structure for using Beads, an AI-native issue tracker, directly within the repository. This includes adding a `.gitignore` to exclude generated files, a `README.md` to onboard users, a default `config.yaml`, and initial issue data in `issues.jsonl`.
… files Introduce documentation for agents detailing the required workflow using the 'bd' tool for issue tracking. Configure Git to use a custom 'beads' merge driver for `.beads/issues.jsonl` files to handle potential conflicts in this specific file format correctly.
Generates a temporary kubeconfig via `kops export kubeconfig` into a mktemp path. All kubectl calls use KUBECONFIG= inline (no permanent export) and the temp file is removed in the EXIT trap.
Remove OS-specific logic for determining Docker network flags and endpoint in the `dump-remote-db` recipe. The local port is also changed from 8529 to 9255 to avoid potential conflicts if a local ArangoDB instance is running on the default port.
Introduce a new `list-restic-snapshots` command in the ArangoDB justfile to easily list backup snapshots stored in a GCS repository using restic.
…kups Adds a new recipe to the ArangoDB justfile to automate the restoration of the most recent restic snapshot from a GCS bucket to a specified local directory. This simplifies the process of recovering database backups by handling the fetching of necessary secrets (restic password and GCS credentials) from Kubernetes secrets and setting up the environment for restic.
…tance locally Adds a new `deploy-local-arangodb` target to the ArangoDB justfile. This target automates the deployment of the ArangoDB operator and a single ArangoDB instance onto a local k3d cluster. It handles retrieving necessary secrets (passphrase and root password) from `pass`, setting up the Pulumi stacks by cloning from an existing stack, and configuring specific settings like storage size and root password for the local deployment before running `pulumi create-resource`.
…deploy recipe Updates the `deploy-local-arangodb` recipe to use a dynamic cluster name derived from `K3D_CLUSTER_NAME` or a default, and updates the kubeconfig export command to use this dynamic name.
- Add arm64 to kube-arangodb operator Helm chart architectures so the operator pod can schedule on arm64 k3d nodes - Add spec.architecture: [arm64] to ArangoDeployment to signal arm64 intent - Relabel k3d node as amd64 in deploy-local-arangodb recipe to work around the operator hardcoding amd64-only nodeAffinity for database pods; containerd still pulls the correct arm64 image at runtime - Add deploy-local-arangodb just recipe for full local k3d deployment including stack creation, config, and Pulumi up for both projects
kube-arangodb operator hardcodes amd64 node affinity on database pods. Label the server node with kubernetes.io/arch=amd64 via --k3s-node-label at cluster creation so pods schedule without a post-deploy workaround. containerd still pulls the correct arm64 image at runtime. Remove the manual relabel step from deploy-local-arangodb since the cluster creation recipe now handles it.
…d single instance Introduce local Pulumi configuration files for deploying the ArangoDB operator and a single ArangoDB instance. This enables declarative infrastructure management for these components using Pulumi, setting specific versions, namespaces, and storage configurations.
….justfile Refactor Pulumi configuration setting commands to use `config set-all` for better readability and atomic updates. Update `.beads/issues.jsonl` to reflect the current state of cluster operations, including closing the ArangoDB deployment task and adding new, more detailed tasks for database restoration and user creation. The changes standardize how Pulumi configuration is set, moving from multiple `--path` calls to a single `set-all` call, which is cleaner. The issue file updates reflect the progress made in setting up the local development environment, specifically closing the deployment task for ArangoDB and introducing necessary steps for data restoration and user setup.
Include system collections during database dumps to ensure complete data backups. Comment out the automatic pod restart logic in the restore process to prevent unintended service disruptions and allow for manual verification after data restoration.
Introduce a backup section to the configuration file with git-push disabled by default.
There was a problem hiding this comment.
Pull request overview
This PR adds a local development workflow centered around k3d + Pulumi, including local stack configs for core services (ArangoDB, NATS, MinIO, modware backends) and introduces Beads-based issue tracking artifacts into the repo.
Changes:
- Added new
justmodules for local Pulumi state management, k3d cluster lifecycle, and ArangoDB dump/restore + local deploy workflows. - Added local Pulumi stack configs (
Pulumi.local.yaml) for storage class, NATS, MinIO, ArangoDB operator/single, namespace bootstrap, and backend services. - Bootstrapped Beads configuration and repository files, plus updated gitignore/attributes to support local state and JSONL merging.
Reviewed changes
Copilot reviewed 26 out of 29 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| storage_class/Pulumi.local.yaml | Adds local stack config for a local-path storage class. |
| nats/Pulumi.local.yaml | Adds local stack config for NATS Helm deployment. |
| namespace-bootstrap/main.go | New Pulumi Go program to create a namespace from config. |
| namespace-bootstrap/go.sum | Adds dependency lockfile for the new namespace-bootstrap module. |
| namespace-bootstrap/go.mod | Declares a separate Go module for namespace-bootstrap. |
| namespace-bootstrap/Pulumi.yaml | Defines Pulumi project metadata for namespace-bootstrap. |
| namespace-bootstrap/Pulumi.local.yaml | Adds local stack config for namespace-bootstrap (namespace=dev). |
| modware-stock/Pulumi.local.yaml | Adds local stack config for the stock backend service. |
| modware-annotation/Pulumi.local.yaml | Adds local stack config for the annotation backend service. |
| minio/main.go | Updates Helm values to toggle MinIO console based on webui. |
| minio/Pulumi.local.yaml | Adds local stack config for MinIO (chart/image/storage/secret). |
| just_modules/local-pulumi.justfile | New helper recipes for local Pulumi backend + stack/project lifecycle. |
| just_modules/k3d.justfile | New k3d cluster lifecycle helpers + kubeconfig export + MinIO port-forward. |
| just_modules/arangodb.justfile | Adds ArangoDB remote dump, restic snapshot listing/restore, local restore, and local deploy workflows. |
| create-arangodb-databases/Pulumi.local.yaml | Adds local stack config for provisioning databases/users/secrets in ArangoDB. |
| arangodb-single/main.go | Adds explicit architecture selection to the ArangoDB single spec. |
| arangodb-single/Pulumi.local.yaml | Adds local stack config for arangodb-single (with some duplicated keys). |
| arangodb-operator/main.go | Updates Helm values to include operator architectures list. |
| arangodb-operator/Pulumi.local.yaml | Adds local stack config for arangodb-operator (with some duplicated keys). |
| Justfile | Registers new just modules: local-pulumi, k3d, arangodb. |
| AGENTS.md | Adds repo-local agent workflow instructions (Beads onboarding + session checklist). |
| .gitignore | Ignores local kubeconfig export and local Pulumi backend state directories. |
| .gitattributes | Adds a merge driver mapping for .beads/issues.jsonl. |
| .beads/metadata.json | Adds Beads metadata configuration. |
| .beads/issues.jsonl | Adds initial Beads issues/knowledge-base entries. |
| .beads/interactions.jsonl | Adds Beads interactions log file (present in PR file list). |
| .beads/config.yaml | Adds Beads configuration (including backup git-push setting). |
| .beads/README.md | Adds Beads documentation for contributors. |
| .beads/.gitignore | Adds Beads internal/runtime ignore rules. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+14
to
+21
| arangodb-single:secret: | ||
| name: arangodb-root | ||
| password: | ||
| secure: v1:9VXPvbs6DLKd1/ez:74t+N7wWjthEedvW46TcXDWvzLBQAdIw7BHqn4YZLVLBepMoOaheOBtrDVaVUuj3ej9nxUeSShCLws9cIwRqQdn7CgavOd6p6Hmd6f+W4Q== | ||
| arangodb-single:storage: | ||
| class: dictycr-balanced | ||
| size: 20Gi | ||
| arangodb-single:version: 3.11.6 |
There was a problem hiding this comment.
This Pulumi stack config contains multiple duplicated keys outside of arangodb-single:properties (e.g., arangodb-single:secret, arangodb-single:storage, arangodb-single:version). The program reads only properties via config.TryObject("properties", ...), so the extra keys are unused and can drift out of sync. Consider removing the redundant top-level keys to keep a single source of truth for local config.
Suggested change
| arangodb-single:secret: | |
| name: arangodb-root | |
| password: | |
| secure: v1:9VXPvbs6DLKd1/ez:74t+N7wWjthEedvW46TcXDWvzLBQAdIw7BHqn4YZLVLBepMoOaheOBtrDVaVUuj3ej9nxUeSShCLws9cIwRqQdn7CgavOd6p6Hmd6f+W4Q== | |
| arangodb-single:storage: | |
| class: dictycr-balanced | |
| size: 20Gi | |
| arangodb-single:version: 3.11.6 |
Comment on lines
+3
to
+15
| arangodb-operator:chart: | ||
| name: kube-arangodb | ||
| repository: https://arangodb.github.io/kube-arangodb | ||
| version: 1.3.4 | ||
| arangodb-operator:deploymentReplication: "false" | ||
| arangodb-operator:namespace: dev | ||
| arangodb-operator:properties: | ||
| chart: | ||
| name: kube-arangodb | ||
| repository: https://arangodb.github.io/kube-arangodb | ||
| version: 1.3.4 | ||
| deploymentReplication: false | ||
| namespace: dev |
There was a problem hiding this comment.
This stack config duplicates values both at arangodb-operator:* and under arangodb-operator:properties.*. Since the program reads only properties (config.TryObject("properties", ...)), the top-level keys (including deploymentReplication: "false" as a string) are unused and can become misleading. Consider removing the non-properties keys so the local config can’t diverge.
Comment on lines
+46
to
+47
| # Kill any existing port-forward on this port to avoid conflicts | ||
| lsof -ti:8529 | xargs kill -9 2>/dev/null || true |
There was a problem hiding this comment.
The port-cleanup step is killing processes on port 8529, but this recipe forwards to LOCAL_PORT=9255. This can both leave the intended local port occupied (causing the port-forward to fail) and inadvertently kill unrelated processes that happen to be listening on 8529. Update the lsof/xargs line to target ${LOCAL_PORT} (and consider checking for empty output before calling xargs).
Suggested change
| # Kill any existing port-forward on this port to avoid conflicts | |
| lsof -ti:8529 | xargs kill -9 2>/dev/null || true | |
| # Kill any existing port-forward on this local port to avoid conflicts | |
| PIDS=$(lsof -ti:"$LOCAL_PORT" 2>/dev/null || true) | |
| if [ -n "$PIDS" ]; then | |
| kill -9 $PIDS 2>/dev/null || true | |
| fi |
Comment on lines
+314
to
+317
| --plaintext "arangodb-operator:properties.deploymentReplication=false" \ | ||
| --plaintext "arangodb-operator:properties.chart.name=kube-arangodb" \ | ||
| --plaintext "arangodb-operator:properties.chart.version=1.2.42" \ | ||
| --plaintext "arangodb-operator:properties.chart.repository=https://arangodb.github.io/kube-arangodb" |
There was a problem hiding this comment.
This local deploy helper pins kube-arangodb chart.version to 1.2.42, but this PR upgrades the operator to 1.3.4 (see arangodb-operator/Pulumi.local.yaml) and the PR description calls out the upgrade. Leaving this at 1.2.42 will downgrade/override the stack config during deploy-local-arangodb. Update the version here to match the intended operator version (or read it from the existing stack config instead of hardcoding).
Comment on lines
+106
to
+109
| Architecture: pulumi.StringArray{ | ||
| pulumi.String("arm64"), | ||
| }, | ||
| Single: adp.createSingleSpec(), |
There was a problem hiding this comment.
ArangoDeploymentSpec.Architecture is hardcoded to arm64 here. That makes this Pulumi program unsuitable for amd64 clusters, and it also conflicts with the k3d module’s approach of labeling nodes as kubernetes.io/arch=amd64 to satisfy kube-arangodb affinity. Prefer making architecture configurable (or omitting it by default) and aligning it with the cluster/node-label strategy used elsewhere in this PR.
Suggested change
| Architecture: pulumi.StringArray{ | |
| pulumi.String("arm64"), | |
| }, | |
| Single: adp.createSingleSpec(), | |
| Single: adp.createSingleSpec(), |
Comment on lines
+7
to
+18
| # Create a single-node k3d cluster with persistent storage mapping | ||
| # Node is labeled kubernetes.io/arch=amd64 to satisfy kube-arangodb's hardcoded | ||
| # amd64 node affinity; containerd still pulls the correct arm64 image at runtime. | ||
| [group('k3d')] | ||
| create-cluster name=cluster_name port=port_forward path=host_path image=k3s_image: | ||
| mkdir -p {{ path }} | ||
| k3d cluster create {{ name }} \ | ||
| --servers 1 \ | ||
| --image {{ image }} \ | ||
| --port "{{ port }}" \ | ||
| --volume "{{ path }}:/var/lib/rancher/k3s/storage@all" \ | ||
| --k3s-node-label "kubernetes.io/arch=amd64@server:0" |
There was a problem hiding this comment.
This k3d cluster creation labels the server node as kubernetes.io/arch=amd64, but the ArangoDB single-instance Pulumi program in this PR hardcodes spec.architecture=arm64. These two changes are inconsistent and can lead to scheduling/affinity confusion (and potentially unschedulable pods, depending on how kube-arangodb uses the arch label). Pick one arm64-compat strategy and make the k3d node labeling and ArangoDeployment spec consistent (ideally via configuration).
Ensure that system collections are captured during database backups and included during restoration processes. This change updates the Go backup logic and the justfile scripts to use the correct flag syntax.
Implement multi-platform build support using Docker buildx arguments. Update the build process to dynamically fetch the correct Restic binary based on the target architecture and upgrade the base image to ArangoDB to ensure a more robust runtime environment.
- Add just_modules/docker.justfile for image management - Parameterize go, arango, and restic versions - Support local, single-arch, and multi-arch builds - Wire docker module into root Justfile
cybersiddhu
force-pushed
the
feat/run-local-k3d
branch
from
7db18df to
9c3b8ea
Compare
This commit simplifies temporary file creation in the ArangoDB justfile by using 'mktemp -t' with descriptive prefixes instead of hardcoded paths and extensions. It also updates the database-backup image tag in the experiments stack.
Use the operator JWT secret (arangodb-jwt) to authenticate both arangorestore and the post-restore arangosh password reset, removing the dependency on knowing the current root password. After restoring a prod dump with --include-system-collections, the _system._users collection is overwritten with prod credentials. The recipe now reads the desired password from the arangodb-root k8s secret and resets it via arangosh --server.jwt-secret-keyfile once the restore completes. Drops the root_pass parameter; credentials are sourced entirely from the k3d cluster secrets.
…working Use a single cleanup trap covering all resources (JWT_FILE, lost+found, PF_PID) to fix the trap-override bug. Replace runtime uname check with just os() for Darwin Docker networking, fixing --net=host no-op on Docker Desktop. Apply consistent DOCKER_NET_FLAGS to both docker run calls. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ers per operation Replace single job container running all operations with init containers for ensure-user and ensure-database steps, and main containers for ensure-grant. Update image to ghcr.io/dictybase-docker/arangoadmin@fec50dd in both experiments and local stacks. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Centralize repetitive logic into private helper recipes to improve maintainability and reduce code duplication across the ArangoDB justfile.
Add --password-policy always flag to ensure passwords are always validated during ArangoDB user creation operations.
Use arangodb-pass secret instead of non-existent arangodb-root when retrieving the desired root password during local restore.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR introduces comprehensive improvements to run and manage the cluster locally using k3d, Pulumi, and ArangoDB.
Key Changes
justfilefor cluster lifecycle management, arm64 compatibility fixes, and configuration exports.modware-stock,modware-annotation, and storage classes.beadsfor issue tracking within the repository and improved.gitignorefiltering for local state files (e.g., dolt, pulumi state, backup directories).Testing
Tested by running a local k3d cluster and deploying the relevant resources via Pulumi.