armada-operator

Armada Operator is a Kubernetes-native Operator for simpler installation of Armada. This project introduces CRDs for Armada services and provides a controller to manage the lifecycle of these services.

How it works

This project aims to follow the Kubernetes Operator pattern

It uses Controllers which provides a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster

Installation

Prerequisites

In order to start, make sure you have the following installed:

• kubectl
• helm
• Go v1.21+ - required for the Quickstart make kind-all target

Quickstart

To start immediately with Armada and the Operator, you can run the following make target:

make kind-all

This will run the following make targets:

1. kind-create-cluster - Creates a kind cluster using the configuration from hack/kind-config.yaml.
2. install-and-wait-cert-manager - Installs cert-manager and waits for it to be ready.
3. helm-repos - Adds necessary Helm repositories for external dependencies.
4. helm-install - Installs the latest released Armada Operator using Helm from the gresearch Helm registry.
5. install-armada-deps - Installs required Armada dependencies such as Prometheus, PostgreSQL, Pulsar, and Redis using Helm.
6. wait-for-armada-deps - Waits for all Armada dependencies to be ready.
7. create-armada-namespace - Creates the Armada namespace armada
8. apply-armada-crs - Applies the Custom Resource definitions (CRs) of all Armada components using kubectl.
9. create-armadactl-config - Creates the armadactl configuration (~/.armadactl.yaml) file pointing to localhost:30002` as per the quickstart guide.
10. apply-default-priority-class - Applies the default priority class required by Armada for all jobs.
11. get-armadactl - Downloads the armadactl binary for interacting with the Armada API.

Manual setup

You’ll need a Kubernetes cluster to run the Armada Operator. You can use KIND to run a local cluster for testing, or you can run against a remote cluster.

To install the latest release of Armada Operator in your cluster, run the following commands:

# Add the G-Research Helm repository
helm repo add gresearch https://g-research.github.io/charts
# Install the Armada Operator
helm install armada-operator gresearch/armada-operator --namespace armada-system --create-namespace

The Armada Operator will be installed in the armada-system namespace.

Installing Armada

In order to install Armada, first you will need to install the Armada external dependencies:

• kube-prometheus-stack
• Pulsar
• Redis
• Postgres

You can install the required Armada dependencies either manually or by running the following make target:

make install-armada-deps

Note: You will need to wait for all dependencies to be running before proceeding to the next step.

After that run the following command to install Armada components using the CRs provided in dev/quickstart/armada-crs.yaml

# Create armada namespace
kubectl create namespace armada
# Install Armada components
kubectl apply -n armada -f dev/quickstart/armada-crs.yaml

Note: dev/quickstart/armada-crs.yaml uses NodePort Services for exposing Armada components (Lookout UI @ 30000, Armada gRPC API @ 30001 and Armada REST API @ 30002). This example is created to be used with the kind config defined at hack/kind-config.yaml for demonstration purposes only and should not be used in production.

Which will deploy CRs for each Armada component. Once every Armada service is deployed, you should have a fully functional installation of Armada.

Armada requires a default PriorityClass to be set for all jobs. You can apply the default PriorityClass by running the following command:

kubectl apply -f dev/quickstart/priority-class.yaml

Installing armadactl

To install armadactl, run the following make target:

make get-armadactl

Or download it from the GitHub Release page for your platform.

Create a configuration file for armadactl by running the following command:

make create-armadactl-config

Or create a configuration file manually in ~/.armadactl.yaml with the following content:

currentContext: main
contexts:
  main:
    # URL of the Armada gRPC API
    armadaUrl: localhost:30002 # This uses the NodePort configured in the Quickstart guide

Usage

Create a queue called example by running:

armadactl create queue example

Submit a job to the example queue by running:

armadactl submit dev/quickstart/example-job.yaml

Check the status of your job in the Lookout UI by visiting http://localhost:30000 (assuming Armada was installed via the Quickstart guide and it is exposed via a NodePort service) in your browser.

Migrations

Migrating to v0.11 and beyond

Since v0.11, Armada Scheduler requires permissionGroupsMapping also to be configured.

Make sure the applicationConfig field in the Armada Scheduler CRD includes the permissionGroupsMapping field.

Quickstart example which allows anonymous auth:

auth:
  anonymousAuth: true
  permissionGroupMapping:
    execute_jobs: ["everyone"]

Documentation

For a step-by-step guide on how to install Armada using the Armada Operator and interact with the Armada API, please read the Quickstart guide in the documentation and follow runbooks in the dev/runbooks/ directory.

For more info on Armada, please visit the Armada website and the GitHub repository armadaproject/armada

For understanding the minimal configuration required to deploy Armada services, please refer to the armada-crs.yaml file.

For advanced usage, please refer to the Armada CRD reference docs in dev/crd/ directory.

Each Armada Operator CRD supports a field .spec.applicationConfig which injects configuration into the Armada component. Here is a list of support Armada component configurations:

• Armada Server
• Armada Executor
• Armada Scheduler
• Armada Scheduler Ingester
• Armada Lookout
• Armada Lookout Ingester
• Armada Event Ingester
• Armada Binoculars

Local development

This section assumes you have a kind cluster named armada running on your machine (it will appear as kind-armada in your kubeconfig).

Check out the Makefile for more commands to help you with your development or run make help for a list of available commands.

Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).

Start a Development Cluster

This section assumes you have KIND installed.

If you do not have a Kubernetes cluster to test against, you can start one using the following command:

make kind-create-cluster

Build & install from scratch

Run the following make command:

make kind-deploy

This command will do the following:

• Build the armada-operator binary for linux/amd64
• Build the armada-operator Docker image
• Load the Docker image into the kind cluster
• Install each CRD supported by the armada-operator on the cluster
• Install the armada-operator on the cluster using kustomize

Run locally

In order to run the operator locally, you can use one of the following commands:

# Run the operator locally with webhooks enabled
make run
# Run the operator locally with webhooks disabled
make run-no-webhook

Stop the Development Cluster

To stop the development cluster:

make kind-delete-cluster

This will totally destroy your development Kind cluster.

Contributing

Please feel free to contribute bug-reports or ideas for enhancements via GitHub's issue system.

Code contributions are also welcome. When submitting a pull-request please ensure it references a relevant issue as well as making sure all CI checks pass.

Testing

Please test contributions thoroughly before requesting reviews. At a minimum:

# Lint code using golangci-lint.
make lint
# Run unit tests.
make test-unit
# Run integration tests.
make test-integration

should all succeed without error.

Add and change appropriate unit and integration tests to ensure your changes are covered by automated tests and appear to be correct.

FAQ

kube-prometheus-stack is not installing

If you get the following error:

Error: template: kube-prometheus-stack/templates/prometheus/prometheus.yaml:262:11: executing "kube-prometheus-stack/templates/prometheus/prometheus.yaml" at <ne .Values.prometheus.prometheusSpec.scrapeConfigNamespaceSelector nil>: error calling ne: uncomparable type map[string]interface {}: map[]

Try upgrading your Helm version to v3.16.2 or later.

License

Copyright 2024.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.