diff --git a/src/current/v25.2/cockroachdb-operator-performance.md b/src/current/v25.2/cockroachdb-operator-performance.md index 309b0b44024..5a950c4a380 100644 --- a/src/current/v25.2/cockroachdb-operator-performance.md +++ b/src/current/v25.2/cockroachdb-operator-performance.md @@ -177,10 +177,12 @@ Once you've picked out an amount of CPU and memory to reserve for Cockroach, con ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi ~~~ When you initialize the cluster, check that all the CockroachDB pods are scheduled successfully. If you see any get stuck in the pending state, run `kubectl describe pod {podname}` and check the `Events` for information about why they're still pending. You may need to manually preempt pods on one or more nodes by running `kubectl delete pod` on them to make room for the CockroachDB pods. As long as the pods you delete were created by a higher-level Kubernetes object such as a `Deployment`, they'll be safely recreated on another node. @@ -194,12 +196,14 @@ To set resource limits, in addition to the [resource requests](#resource-request ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi - limits: - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi + limits: + memory: 12300Mi ~~~ Pods will be limited to their reserved resources and are unlikely to be preempted, except in rare cases. This will not improve performance on an underutilized Kubernetes cluster, but provides more predictable performance as other workloads run. @@ -266,27 +270,29 @@ Client applications such as benchmarking applications running on the same machin ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: kubernetes.io/hostname - - weight: 99 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: kubernetes.io/hostname + - weight: 99 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The preceding configuration will first prefer to put the `loadgen` pods on different nodes from each other, which is important for the fault tolerance of the `loadgen` pods themselves. As a secondary priority, it will attempt to put the pods on nodes that do not already have a running `CockroachDB` pod. This will ensure the best possible balance of fault tolerance and performance for the load generator and CockroachDB cluster. diff --git a/src/current/v25.2/configure-cockroachdb-operator.md b/src/current/v25.2/configure-cockroachdb-operator.md index 9029f6bd2fa..79f99233604 100644 --- a/src/current/v25.2/configure-cockroachdb-operator.md +++ b/src/current/v25.2/configure-cockroachdb-operator.md @@ -27,18 +27,20 @@ You can set the CPU and memory resources allocated to the CockroachDB container 1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the [Production Checklist]({% link {{ page.version.version }}/recommended-production-settings.md %}#hardware). {{site.data.alerts.end}} -Specify CPU and memory values in `cockroachdb.crdbCluster.resources.limits` and `cockroachdb.crdbCluster.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - resources: - limits: - cpu: 4000m - memory: 16Gi - requests: - cpu: 4000m - memory: 16Gi + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi ~~~ Apply the new settings to the cluster: diff --git a/src/current/v25.2/deploy-cockroachdb-with-cockroachdb-operator.md b/src/current/v25.2/deploy-cockroachdb-with-cockroachdb-operator.md index 4288dde61a4..44758d974f5 100644 --- a/src/current/v25.2/deploy-cockroachdb-with-cockroachdb-operator.md +++ b/src/current/v25.2/deploy-cockroachdb-with-cockroachdb-operator.md @@ -252,7 +252,21 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st If you intend to deploy CockroachDB nodes across multiple different regions, follow the additional steps described in [Deploy across multiple regions](#deploy-across-multiple-regions). {{site.data.alerts.callout_end}} -1. Uncomment and modify `cockroachdb.crdbCluster.resources` in the values file with the CPU and memory requests and limits for each node to use. The default values are 4vCPU and 16GiB of memory: +1. Modify `cockroachdb.crdbCluster.podTemplate.spec.resources` in the values file with the CPU and memory limits and requests for each node to use. For example, to define default values of 4vCPU and 16GiB of memory: + + ~~~ yaml + cockroachdb: + crdbCluster: + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi + ~~~ For more information on configuring node resource allocation, refer to [Resource management]({% link {{ page.version.version }}/configure-cockroachdb-operator.md %}) @@ -435,13 +449,18 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st cockroach start --locality region=us-central1,zone=us-central1-c,dc=dc2 ~~~ - Optionally, review the `cockroachdb.crdbCluster.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: + Optionally, review the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - topologyKey: topology.kubernetes.io/zone + podTemplate: + spec: + topologySpreadConstraints: + # maxSkew defines the degree to which the pods can be unevenly distributed. + - maxSkew: 1 + # topologyKey defines the key for topology spread. + topologyKey: topology.kubernetes.io/zone ~~~ For more information on localities and topology planning, see the [topology patterns documentation]({% link {{ page.version.version }}/topology-patterns.md %}). diff --git a/src/current/v25.2/override-templates-cockroachdb-operator.md b/src/current/v25.2/override-templates-cockroachdb-operator.md index f98cb7d209c..8355a6aa511 100644 --- a/src/current/v25.2/override-templates-cockroachdb-operator.md +++ b/src/current/v25.2/override-templates-cockroachdb-operator.md @@ -10,7 +10,7 @@ The {{ site.data.products.cockroachdb-operator }} provides abstractions that sim - A default pod specification is used for the CockroachDB Kubernetes pod. - The `values.yaml` configuration maps to a subset of `cockroach start` flags when CockroachDB is initialized. -This page describes configuration options that allow advanced users to manually override the pod template and `cockroach start` flags as needed for deployment. +This page describes configuration options that allow advanced users to manually override pod template metadata and `cockroach start` flags as needed for deployment. {{site.data.alerts.callout_info}} The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}). @@ -24,6 +24,8 @@ This page describes advanced configurations that override the supported default The `cockroachdb.crdbCluster.podTemplate` field allows you to override the default pod metadata and specification configured by the {{ site.data.products.cockroachdb-operator }}. The values in this field are merged with the default pod specification, where settings in `podTemplate` override any values in the default. +The `podTemplate` field includes a `containers` field that specifies the container name and image that the template is applied to. By default, this resolves to the `cockroachdb` container name and can be excluded from modifications to the CockroachDB pod template YAML. If needed, you can provide specific images for the `containers` and `initContainers`. For example, the following `podTemplate` configuration specifies a custom init container: + ~~~ yaml cockroachdb: crdbCluster: diff --git a/src/current/v25.2/schedule-cockroachdb-operator.md b/src/current/v25.2/schedule-cockroachdb-operator.md index 6b3b5563ec6..4630ea6d9d1 100644 --- a/src/current/v25.2/schedule-cockroachdb-operator.md +++ b/src/current/v25.2/schedule-cockroachdb-operator.md @@ -17,16 +17,18 @@ The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ pag A pod with a *node selector* will be scheduled onto a worker node that has matching [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), or key-value pairs. -Specify the labels in `cockroachdb.crdbCluster.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. +Specify the labels in `cockroachdb.crdbCluster.podTemplate.spec.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. The following configuration causes CockroachDB pods to be scheduled onto worker nodes that have *both* the labels `worker-pool-name=crdb-workers` and `kubernetes.io/arch=amd64`: ~~~ yaml cockroachdb: crdbCluster: - nodeSelector: - worker-pool-name: crdb-workers - kubernetes.io/arch: amd64 + podTemplate: + spec: + nodeSelector: + worker-pool-name: crdb-workers + kubernetes.io/arch: amd64 ~~~ For an example of labeling nodes, see [Scheduling CockroachDB onto labeled nodes](#example-scheduling-cockroachdb-onto-labeled-nodes). @@ -46,30 +48,32 @@ For an example, see [Scheduling CockroachDB onto labeled nodes](#example-schedul ### Add a node affinity -Specify node affinities in `cockroachdb.crdbCluster.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify node affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The following configuration requires that CockroachDB pods are scheduled onto worker nodes running a Linux operating system, with a preference against worker nodes in the `us-east4-b` availability zone. ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - preference: - matchExpressions: - - key: topology.kubernetes.io/zone - operator: NotIn - values: - - us-east4-b + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + preference: + matchExpressions: + - key: topology.kubernetes.io/zone + operator: NotIn + values: + - us-east4-b ~~~ The `requiredDuringSchedulingIgnoredDuringExecution` node affinity rule, using the `In` operator, requires CockroachDB pods to be scheduled onto nodes with the matching label `kubernetes.io/os=linux`. It will not evict pods that are already running on nodes that do not match the affinity requirements. @@ -80,7 +84,7 @@ For more context on how these rules work, see the [Kubernetes documentation](htt ### Add a pod affinity or anti-affinity -Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.affinity.podAffinity` and `cockroachdb.crdbCluster.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAffinity` and `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The {{ site.data.products.cockroachdb-operator }} hard-codes the pod template to only allow one pod per Kubernetes node. If you need to override this value, you can [override the pod template]({% link {{ page.version.version }}/override-templates-cockroachdb-operator.md %}#override-the-default-pod). @@ -89,27 +93,29 @@ The following configuration attempts to schedule CockroachDB pods in the same zo ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: topology.kubernetes.io/zone - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: topology.kubernetes.io/zone + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `preferredDuringSchedulingIgnoredDuringExecution` pod affinity rule, using the `In` operator and specified `weight`, encourages (but does not require) CockroachDB pods to be co-located with pods labeled `app=loadgen` already running in the same zone, as specified with `topologyKey`. @@ -161,24 +167,26 @@ In this example, CockroachDB has not yet been deployed to a running Kubernetes c ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node - operator: In - values: - - crdb - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node + operator: In + values: + - crdb + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `nodeAffinity` rule requires CockroachDB pods to be scheduled onto worker nodes with the label `node=crdb`. The `podAntiAffinity` rule requires CockroachDB pods not to be co-located on a worker node, as specified with `topologyKey`. @@ -217,18 +225,20 @@ For an example, see [Evicting CockroachDB from a running worker node](#example-e ### Add a toleration -Specify pod tolerations in the `cockroachdb.crdbCluster.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). +Specify pod tolerations in the `cockroachdb.crdbCluster.podTemplate.spec.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). The following toleration matches a taint with the specified key, value, and `NoSchedule` effect, using the `Equal` operator. A toleration that uses the `Equal` operator must include a `value` field: ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Equal" - value: "example" - effect: "NoSchedule" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Equal" + value: "example" + effect: "NoSchedule" ~~~ A `NoSchedule` taint on a node prevents pods from being scheduled onto the node. The matching toleration allows a pod to be scheduled onto the node. A `NoSchedule` toleration is therefore best included before [deploying the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). @@ -242,11 +252,13 @@ The following toleration matches every taint with the specified key and `NoExecu ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" - tolerationSeconds: 3600 + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 3600 ~~~ A `NoExecute` taint on a node prevents pods from being scheduled onto the node, and evicts pods from the node if they are already running on the node. The matching toleration allows a pod to be scheduled onto the node, and to continue running on the node if `tolerationSeconds` is not specified. If `tolerationSeconds` is specified, the pod is evicted after this number of seconds. @@ -288,10 +300,12 @@ In this example, CockroachDB has already been deployed on a Kubernetes cluster. ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" ~~~ Because no tolerationSeconds is specified, CockroachDB will be evicted immediately from the tainted worker node. @@ -325,20 +339,22 @@ A pod with a *topology spread constraint* must satisfy its conditions when being ### Add a topology spread constraint -Specify pod topology spread constraints in the `cockroachdb.crdbCluster.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. +Specify pod topology spread constraints in the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. The following topology spread constraint ensures that CockroachDB pods deployed with the label `environment=production` will not be unevenly distributed across zones by more than `1` pod: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: DoNotSchedule - labelSelector: - matchLabels: - environment: production + podTemplate: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + environment: production ~~~ The `DoNotSchedule` condition prevents labeled pods from being scheduled onto Kubernetes worker nodes when doing so would fail to meet the spread and topology constraints specified with `maxSkew` and `topologyKey`, respectively. @@ -349,15 +365,17 @@ For more context on how these rules work, see the [Kubernetes documentation](htt To assist in working with your cluster, you can add labels and annotations to your resources. -Specify labels in `cockroachdb.crdbCluster.podLabels` and annotations in `cockroachdb.crdbCluster.podAnnotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify labels in `cockroachdb.crdbCluster.podTemplate.metadata.labels` and annotations in `cockroachdb.crdbCluster.podTemplate.metadata.annotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - podLabels: - app.kubernetes.io/version: v25.1.4 - podAnnotations - operator: https://raw.githubusercontent.com/cockroachdb/helm-charts/refs/heads/master/cockroachdb-parent/charts/cockroachdb/values.yaml + podTemplate: + metadata: + labels: + app.kubernetes.io/name: "cockroachdb" + annotations: + kubernetes.io/description: "This is a CockroachDB pod." ~~~ To verify that the labels and annotations were applied to a pod, for example, run `kubectl describe pod {pod-name}`. diff --git a/src/current/v25.3/cockroachdb-operator-performance.md b/src/current/v25.3/cockroachdb-operator-performance.md index 309b0b44024..5a950c4a380 100644 --- a/src/current/v25.3/cockroachdb-operator-performance.md +++ b/src/current/v25.3/cockroachdb-operator-performance.md @@ -177,10 +177,12 @@ Once you've picked out an amount of CPU and memory to reserve for Cockroach, con ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi ~~~ When you initialize the cluster, check that all the CockroachDB pods are scheduled successfully. If you see any get stuck in the pending state, run `kubectl describe pod {podname}` and check the `Events` for information about why they're still pending. You may need to manually preempt pods on one or more nodes by running `kubectl delete pod` on them to make room for the CockroachDB pods. As long as the pods you delete were created by a higher-level Kubernetes object such as a `Deployment`, they'll be safely recreated on another node. @@ -194,12 +196,14 @@ To set resource limits, in addition to the [resource requests](#resource-request ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi - limits: - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi + limits: + memory: 12300Mi ~~~ Pods will be limited to their reserved resources and are unlikely to be preempted, except in rare cases. This will not improve performance on an underutilized Kubernetes cluster, but provides more predictable performance as other workloads run. @@ -266,27 +270,29 @@ Client applications such as benchmarking applications running on the same machin ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: kubernetes.io/hostname - - weight: 99 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: kubernetes.io/hostname + - weight: 99 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The preceding configuration will first prefer to put the `loadgen` pods on different nodes from each other, which is important for the fault tolerance of the `loadgen` pods themselves. As a secondary priority, it will attempt to put the pods on nodes that do not already have a running `CockroachDB` pod. This will ensure the best possible balance of fault tolerance and performance for the load generator and CockroachDB cluster. diff --git a/src/current/v25.3/configure-cockroachdb-operator.md b/src/current/v25.3/configure-cockroachdb-operator.md index 9029f6bd2fa..79f99233604 100644 --- a/src/current/v25.3/configure-cockroachdb-operator.md +++ b/src/current/v25.3/configure-cockroachdb-operator.md @@ -27,18 +27,20 @@ You can set the CPU and memory resources allocated to the CockroachDB container 1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the [Production Checklist]({% link {{ page.version.version }}/recommended-production-settings.md %}#hardware). {{site.data.alerts.end}} -Specify CPU and memory values in `cockroachdb.crdbCluster.resources.limits` and `cockroachdb.crdbCluster.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - resources: - limits: - cpu: 4000m - memory: 16Gi - requests: - cpu: 4000m - memory: 16Gi + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi ~~~ Apply the new settings to the cluster: diff --git a/src/current/v25.3/deploy-cockroachdb-with-cockroachdb-operator.md b/src/current/v25.3/deploy-cockroachdb-with-cockroachdb-operator.md index bf9fdc62e77..a90fb7fb5a1 100644 --- a/src/current/v25.3/deploy-cockroachdb-with-cockroachdb-operator.md +++ b/src/current/v25.3/deploy-cockroachdb-with-cockroachdb-operator.md @@ -252,7 +252,21 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st If you intend to deploy CockroachDB nodes across multiple different regions, follow the additional steps described in [Deploy across multiple regions](#deploy-across-multiple-regions). {{site.data.alerts.callout_end}} -1. Uncomment and modify `cockroachdb.crdbCluster.resources` in the values file with the CPU and memory requests and limits for each node to use. The default values are 4vCPU and 16GiB of memory: +1. Modify `cockroachdb.crdbCluster.podTemplate.spec.resources` in the values file with the CPU and memory limits and requests for each node to use. For example, to define default values of 4vCPU and 16GiB of memory: + + ~~~ yaml + cockroachdb: + crdbCluster: + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi + ~~~ For more information on configuring node resource allocation, refer to [Resource management]({% link {{ page.version.version }}/configure-cockroachdb-operator.md %}) @@ -435,13 +449,18 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st cockroach start --locality region=us-central1,zone=us-central1-c,dc=dc2 ~~~ - Optionally, review the `cockroachdb.crdbCluster.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: + Optionally, review the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - topologyKey: topology.kubernetes.io/zone + podTemplate: + spec: + topologySpreadConstraints: + # maxSkew defines the degree to which the pods can be unevenly distributed. + - maxSkew: 1 + # topologyKey defines the key for topology spread. + topologyKey: topology.kubernetes.io/zone ~~~ For more information on localities and topology planning, see the [topology patterns documentation]({% link {{ page.version.version }}/topology-patterns.md %}). diff --git a/src/current/v25.3/override-templates-cockroachdb-operator.md b/src/current/v25.3/override-templates-cockroachdb-operator.md index 9b27b3debbb..ee1f4635b6e 100644 --- a/src/current/v25.3/override-templates-cockroachdb-operator.md +++ b/src/current/v25.3/override-templates-cockroachdb-operator.md @@ -10,7 +10,7 @@ The {{ site.data.products.cockroachdb-operator }} provides abstractions that sim - A default pod specification is used for the CockroachDB Kubernetes pod. - The `values.yaml` configuration maps to a subset of `cockroach start` flags when CockroachDB is initialized. -This page describes configuration options that allow advanced users to manually override the pod template and `cockroach start` flags as needed for deployment. +This page describes configuration options that allow advanced users to manually override pod template metadata and `cockroach start` flags as needed for deployment. {{site.data.alerts.callout_info}} The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}). @@ -24,6 +24,8 @@ This page describes advanced configurations that override the supported default The `cockroachdb.crdbCluster.podTemplate` field allows you to override the default pod metadata and specification configured by the {{ site.data.products.cockroachdb-operator }}. The values in this field are merged with the default pod specification, where settings in `podTemplate` override any values in the default. +The `podTemplate` field includes a `containers` field that specifies the container name and image that the template is applied to. By default, this resolves to the `cockroachdb` container name and can be excluded from modifications to the CockroachDB pod template YAML. If needed, you can provide specific images for the `containers` and `initContainers`. For example, the following `podTemplate` configuration specifies a custom init container: + ~~~ yaml cockroachdb: crdbCluster: diff --git a/src/current/v25.3/schedule-cockroachdb-operator.md b/src/current/v25.3/schedule-cockroachdb-operator.md index 6b3b5563ec6..4630ea6d9d1 100644 --- a/src/current/v25.3/schedule-cockroachdb-operator.md +++ b/src/current/v25.3/schedule-cockroachdb-operator.md @@ -17,16 +17,18 @@ The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ pag A pod with a *node selector* will be scheduled onto a worker node that has matching [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), or key-value pairs. -Specify the labels in `cockroachdb.crdbCluster.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. +Specify the labels in `cockroachdb.crdbCluster.podTemplate.spec.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. The following configuration causes CockroachDB pods to be scheduled onto worker nodes that have *both* the labels `worker-pool-name=crdb-workers` and `kubernetes.io/arch=amd64`: ~~~ yaml cockroachdb: crdbCluster: - nodeSelector: - worker-pool-name: crdb-workers - kubernetes.io/arch: amd64 + podTemplate: + spec: + nodeSelector: + worker-pool-name: crdb-workers + kubernetes.io/arch: amd64 ~~~ For an example of labeling nodes, see [Scheduling CockroachDB onto labeled nodes](#example-scheduling-cockroachdb-onto-labeled-nodes). @@ -46,30 +48,32 @@ For an example, see [Scheduling CockroachDB onto labeled nodes](#example-schedul ### Add a node affinity -Specify node affinities in `cockroachdb.crdbCluster.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify node affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The following configuration requires that CockroachDB pods are scheduled onto worker nodes running a Linux operating system, with a preference against worker nodes in the `us-east4-b` availability zone. ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - preference: - matchExpressions: - - key: topology.kubernetes.io/zone - operator: NotIn - values: - - us-east4-b + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + preference: + matchExpressions: + - key: topology.kubernetes.io/zone + operator: NotIn + values: + - us-east4-b ~~~ The `requiredDuringSchedulingIgnoredDuringExecution` node affinity rule, using the `In` operator, requires CockroachDB pods to be scheduled onto nodes with the matching label `kubernetes.io/os=linux`. It will not evict pods that are already running on nodes that do not match the affinity requirements. @@ -80,7 +84,7 @@ For more context on how these rules work, see the [Kubernetes documentation](htt ### Add a pod affinity or anti-affinity -Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.affinity.podAffinity` and `cockroachdb.crdbCluster.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAffinity` and `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The {{ site.data.products.cockroachdb-operator }} hard-codes the pod template to only allow one pod per Kubernetes node. If you need to override this value, you can [override the pod template]({% link {{ page.version.version }}/override-templates-cockroachdb-operator.md %}#override-the-default-pod). @@ -89,27 +93,29 @@ The following configuration attempts to schedule CockroachDB pods in the same zo ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: topology.kubernetes.io/zone - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: topology.kubernetes.io/zone + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `preferredDuringSchedulingIgnoredDuringExecution` pod affinity rule, using the `In` operator and specified `weight`, encourages (but does not require) CockroachDB pods to be co-located with pods labeled `app=loadgen` already running in the same zone, as specified with `topologyKey`. @@ -161,24 +167,26 @@ In this example, CockroachDB has not yet been deployed to a running Kubernetes c ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node - operator: In - values: - - crdb - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node + operator: In + values: + - crdb + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `nodeAffinity` rule requires CockroachDB pods to be scheduled onto worker nodes with the label `node=crdb`. The `podAntiAffinity` rule requires CockroachDB pods not to be co-located on a worker node, as specified with `topologyKey`. @@ -217,18 +225,20 @@ For an example, see [Evicting CockroachDB from a running worker node](#example-e ### Add a toleration -Specify pod tolerations in the `cockroachdb.crdbCluster.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). +Specify pod tolerations in the `cockroachdb.crdbCluster.podTemplate.spec.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). The following toleration matches a taint with the specified key, value, and `NoSchedule` effect, using the `Equal` operator. A toleration that uses the `Equal` operator must include a `value` field: ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Equal" - value: "example" - effect: "NoSchedule" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Equal" + value: "example" + effect: "NoSchedule" ~~~ A `NoSchedule` taint on a node prevents pods from being scheduled onto the node. The matching toleration allows a pod to be scheduled onto the node. A `NoSchedule` toleration is therefore best included before [deploying the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). @@ -242,11 +252,13 @@ The following toleration matches every taint with the specified key and `NoExecu ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" - tolerationSeconds: 3600 + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 3600 ~~~ A `NoExecute` taint on a node prevents pods from being scheduled onto the node, and evicts pods from the node if they are already running on the node. The matching toleration allows a pod to be scheduled onto the node, and to continue running on the node if `tolerationSeconds` is not specified. If `tolerationSeconds` is specified, the pod is evicted after this number of seconds. @@ -288,10 +300,12 @@ In this example, CockroachDB has already been deployed on a Kubernetes cluster. ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" ~~~ Because no tolerationSeconds is specified, CockroachDB will be evicted immediately from the tainted worker node. @@ -325,20 +339,22 @@ A pod with a *topology spread constraint* must satisfy its conditions when being ### Add a topology spread constraint -Specify pod topology spread constraints in the `cockroachdb.crdbCluster.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. +Specify pod topology spread constraints in the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. The following topology spread constraint ensures that CockroachDB pods deployed with the label `environment=production` will not be unevenly distributed across zones by more than `1` pod: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: DoNotSchedule - labelSelector: - matchLabels: - environment: production + podTemplate: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + environment: production ~~~ The `DoNotSchedule` condition prevents labeled pods from being scheduled onto Kubernetes worker nodes when doing so would fail to meet the spread and topology constraints specified with `maxSkew` and `topologyKey`, respectively. @@ -349,15 +365,17 @@ For more context on how these rules work, see the [Kubernetes documentation](htt To assist in working with your cluster, you can add labels and annotations to your resources. -Specify labels in `cockroachdb.crdbCluster.podLabels` and annotations in `cockroachdb.crdbCluster.podAnnotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify labels in `cockroachdb.crdbCluster.podTemplate.metadata.labels` and annotations in `cockroachdb.crdbCluster.podTemplate.metadata.annotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - podLabels: - app.kubernetes.io/version: v25.1.4 - podAnnotations - operator: https://raw.githubusercontent.com/cockroachdb/helm-charts/refs/heads/master/cockroachdb-parent/charts/cockroachdb/values.yaml + podTemplate: + metadata: + labels: + app.kubernetes.io/name: "cockroachdb" + annotations: + kubernetes.io/description: "This is a CockroachDB pod." ~~~ To verify that the labels and annotations were applied to a pod, for example, run `kubectl describe pod {pod-name}`. diff --git a/src/current/v25.4/cockroachdb-operator-performance.md b/src/current/v25.4/cockroachdb-operator-performance.md index 309b0b44024..5a950c4a380 100644 --- a/src/current/v25.4/cockroachdb-operator-performance.md +++ b/src/current/v25.4/cockroachdb-operator-performance.md @@ -177,10 +177,12 @@ Once you've picked out an amount of CPU and memory to reserve for Cockroach, con ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi ~~~ When you initialize the cluster, check that all the CockroachDB pods are scheduled successfully. If you see any get stuck in the pending state, run `kubectl describe pod {podname}` and check the `Events` for information about why they're still pending. You may need to manually preempt pods on one or more nodes by running `kubectl delete pod` on them to make room for the CockroachDB pods. As long as the pods you delete were created by a higher-level Kubernetes object such as a `Deployment`, they'll be safely recreated on another node. @@ -194,12 +196,14 @@ To set resource limits, in addition to the [resource requests](#resource-request ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi - limits: - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi + limits: + memory: 12300Mi ~~~ Pods will be limited to their reserved resources and are unlikely to be preempted, except in rare cases. This will not improve performance on an underutilized Kubernetes cluster, but provides more predictable performance as other workloads run. @@ -266,27 +270,29 @@ Client applications such as benchmarking applications running on the same machin ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: kubernetes.io/hostname - - weight: 99 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: kubernetes.io/hostname + - weight: 99 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The preceding configuration will first prefer to put the `loadgen` pods on different nodes from each other, which is important for the fault tolerance of the `loadgen` pods themselves. As a secondary priority, it will attempt to put the pods on nodes that do not already have a running `CockroachDB` pod. This will ensure the best possible balance of fault tolerance and performance for the load generator and CockroachDB cluster. diff --git a/src/current/v25.4/configure-cockroachdb-operator.md b/src/current/v25.4/configure-cockroachdb-operator.md index 9029f6bd2fa..79f99233604 100644 --- a/src/current/v25.4/configure-cockroachdb-operator.md +++ b/src/current/v25.4/configure-cockroachdb-operator.md @@ -27,18 +27,20 @@ You can set the CPU and memory resources allocated to the CockroachDB container 1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the [Production Checklist]({% link {{ page.version.version }}/recommended-production-settings.md %}#hardware). {{site.data.alerts.end}} -Specify CPU and memory values in `cockroachdb.crdbCluster.resources.limits` and `cockroachdb.crdbCluster.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - resources: - limits: - cpu: 4000m - memory: 16Gi - requests: - cpu: 4000m - memory: 16Gi + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi ~~~ Apply the new settings to the cluster: diff --git a/src/current/v25.4/deploy-cockroachdb-with-cockroachdb-operator.md b/src/current/v25.4/deploy-cockroachdb-with-cockroachdb-operator.md index 98d791d77ba..c946f0a294e 100644 --- a/src/current/v25.4/deploy-cockroachdb-with-cockroachdb-operator.md +++ b/src/current/v25.4/deploy-cockroachdb-with-cockroachdb-operator.md @@ -252,7 +252,21 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st If you intend to deploy CockroachDB nodes across multiple different regions, follow the additional steps described in [Deploy across multiple regions](#deploy-across-multiple-regions). {{site.data.alerts.callout_end}} -1. Uncomment and modify `cockroachdb.crdbCluster.resources` in the values file with the CPU and memory requests and limits for each node to use. The default values are 4vCPU and 16GiB of memory: +1. Modify `cockroachdb.crdbCluster.podTemplate.spec.resources` in the values file with the CPU and memory limits and requests for each node to use. For example, to define default values of 4vCPU and 16GiB of memory: + + ~~~ yaml + cockroachdb: + crdbCluster: + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi + ~~~ For more information on configuring node resource allocation, refer to [Resource management]({% link {{ page.version.version }}/configure-cockroachdb-operator.md %}) @@ -302,13 +316,18 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st cockroach start --locality region=us-central1,zone=us-central1-c,dc=dc2 ~~~ - Optionally, review the `cockroachdb.crdbCluster.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: + Optionally, review the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - topologyKey: topology.kubernetes.io/zone + podTemplate: + spec: + topologySpreadConstraints: + # maxSkew defines the degree to which the pods can be unevenly distributed. + - maxSkew: 1 + # topologyKey defines the key for topology spread. + topologyKey: topology.kubernetes.io/zone ~~~ For more information on localities and topology planning, see the [topology patterns documentation]({% link {{ page.version.version }}/topology-patterns.md %}). diff --git a/src/current/v25.4/override-templates-cockroachdb-operator.md b/src/current/v25.4/override-templates-cockroachdb-operator.md index 482c985f62a..9504949cec7 100644 --- a/src/current/v25.4/override-templates-cockroachdb-operator.md +++ b/src/current/v25.4/override-templates-cockroachdb-operator.md @@ -10,7 +10,7 @@ The {{ site.data.products.cockroachdb-operator }} provides abstractions that sim - A default pod specification is used for the CockroachDB Kubernetes pod. - The `values.yaml` configuration maps to a subset of `cockroach start` flags when CockroachDB is initialized. -This page describes configuration options that allow advanced users to manually override the pod template and `cockroach start` flags as needed for deployment. +This page describes configuration options that allow advanced users to manually override pod template metadata and `cockroach start` flags as needed for deployment. {{site.data.alerts.callout_info}} The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}). @@ -24,6 +24,8 @@ This page describes advanced configurations that override the supported default The `cockroachdb.crdbCluster.podTemplate` field allows you to override the default pod metadata and specification configured by the {{ site.data.products.cockroachdb-operator }}. The values in this field are merged with the default pod specification, where settings in `podTemplate` override any values in the default. +The `podTemplate` field includes a `containers` field that specifies the container name and image that the template is applied to. By default, this resolves to the `cockroachdb` container name and can be excluded from modifications to the CockroachDB pod template YAML. If needed, you can provide specific images for the `containers` and `initContainers`. For example, the following `podTemplate` configuration specifies a custom init container: + ~~~ yaml cockroachdb: crdbCluster: @@ -35,37 +37,7 @@ cockroachdb: # initContainers captures the list of init containers for CockroachDB pods. initContainers: - name : cockroachdb-init - image: us-docker.pkg.dev/cockroach-cloud-images/data-plane/init-container@sha256:c3e4ba851802a429c7f76c639a64b9152d206cebb31162c1760f05e98f7c4254 - # containers captures the list of containers for CockroachDB pods. - containers: - - name: cockroachdb - image: cockroachdb/cockroach:v25.4.0 - # imagePullSecrets captures the secrets for fetching images from private registries. - imagePullSecrets: [] -~~~ - -At least one value for `containers` must be specified if any part of `podTemplate` is being modified. For example, the following `podTemplate` configuration overrides pod anti-affinity behavior and specifies a default `cockroachdb/cockroach:v25.4.0` container image: - -~~~ yaml -cockroachdb: - crdbCluster: - podTemplate: - spec: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app.kubernetes.io/component - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname - containers: - - name: cockroachdb - image: cockroachdb/cockroach:v25.4.0 + image: us-docker.pkg.dev/cockroach-cloud-images/data-plane/init-container@sha256:example1234567890abcdefghijklmnopqrstuvwxyz ~~~ ## Override the default `cockroach start` flags diff --git a/src/current/v25.4/schedule-cockroachdb-operator.md b/src/current/v25.4/schedule-cockroachdb-operator.md index 6b3b5563ec6..4630ea6d9d1 100644 --- a/src/current/v25.4/schedule-cockroachdb-operator.md +++ b/src/current/v25.4/schedule-cockroachdb-operator.md @@ -17,16 +17,18 @@ The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ pag A pod with a *node selector* will be scheduled onto a worker node that has matching [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), or key-value pairs. -Specify the labels in `cockroachdb.crdbCluster.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. +Specify the labels in `cockroachdb.crdbCluster.podTemplate.spec.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. The following configuration causes CockroachDB pods to be scheduled onto worker nodes that have *both* the labels `worker-pool-name=crdb-workers` and `kubernetes.io/arch=amd64`: ~~~ yaml cockroachdb: crdbCluster: - nodeSelector: - worker-pool-name: crdb-workers - kubernetes.io/arch: amd64 + podTemplate: + spec: + nodeSelector: + worker-pool-name: crdb-workers + kubernetes.io/arch: amd64 ~~~ For an example of labeling nodes, see [Scheduling CockroachDB onto labeled nodes](#example-scheduling-cockroachdb-onto-labeled-nodes). @@ -46,30 +48,32 @@ For an example, see [Scheduling CockroachDB onto labeled nodes](#example-schedul ### Add a node affinity -Specify node affinities in `cockroachdb.crdbCluster.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify node affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The following configuration requires that CockroachDB pods are scheduled onto worker nodes running a Linux operating system, with a preference against worker nodes in the `us-east4-b` availability zone. ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - preference: - matchExpressions: - - key: topology.kubernetes.io/zone - operator: NotIn - values: - - us-east4-b + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + preference: + matchExpressions: + - key: topology.kubernetes.io/zone + operator: NotIn + values: + - us-east4-b ~~~ The `requiredDuringSchedulingIgnoredDuringExecution` node affinity rule, using the `In` operator, requires CockroachDB pods to be scheduled onto nodes with the matching label `kubernetes.io/os=linux`. It will not evict pods that are already running on nodes that do not match the affinity requirements. @@ -80,7 +84,7 @@ For more context on how these rules work, see the [Kubernetes documentation](htt ### Add a pod affinity or anti-affinity -Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.affinity.podAffinity` and `cockroachdb.crdbCluster.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAffinity` and `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The {{ site.data.products.cockroachdb-operator }} hard-codes the pod template to only allow one pod per Kubernetes node. If you need to override this value, you can [override the pod template]({% link {{ page.version.version }}/override-templates-cockroachdb-operator.md %}#override-the-default-pod). @@ -89,27 +93,29 @@ The following configuration attempts to schedule CockroachDB pods in the same zo ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: topology.kubernetes.io/zone - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: topology.kubernetes.io/zone + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `preferredDuringSchedulingIgnoredDuringExecution` pod affinity rule, using the `In` operator and specified `weight`, encourages (but does not require) CockroachDB pods to be co-located with pods labeled `app=loadgen` already running in the same zone, as specified with `topologyKey`. @@ -161,24 +167,26 @@ In this example, CockroachDB has not yet been deployed to a running Kubernetes c ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node - operator: In - values: - - crdb - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node + operator: In + values: + - crdb + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `nodeAffinity` rule requires CockroachDB pods to be scheduled onto worker nodes with the label `node=crdb`. The `podAntiAffinity` rule requires CockroachDB pods not to be co-located on a worker node, as specified with `topologyKey`. @@ -217,18 +225,20 @@ For an example, see [Evicting CockroachDB from a running worker node](#example-e ### Add a toleration -Specify pod tolerations in the `cockroachdb.crdbCluster.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). +Specify pod tolerations in the `cockroachdb.crdbCluster.podTemplate.spec.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). The following toleration matches a taint with the specified key, value, and `NoSchedule` effect, using the `Equal` operator. A toleration that uses the `Equal` operator must include a `value` field: ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Equal" - value: "example" - effect: "NoSchedule" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Equal" + value: "example" + effect: "NoSchedule" ~~~ A `NoSchedule` taint on a node prevents pods from being scheduled onto the node. The matching toleration allows a pod to be scheduled onto the node. A `NoSchedule` toleration is therefore best included before [deploying the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). @@ -242,11 +252,13 @@ The following toleration matches every taint with the specified key and `NoExecu ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" - tolerationSeconds: 3600 + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 3600 ~~~ A `NoExecute` taint on a node prevents pods from being scheduled onto the node, and evicts pods from the node if they are already running on the node. The matching toleration allows a pod to be scheduled onto the node, and to continue running on the node if `tolerationSeconds` is not specified. If `tolerationSeconds` is specified, the pod is evicted after this number of seconds. @@ -288,10 +300,12 @@ In this example, CockroachDB has already been deployed on a Kubernetes cluster. ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" ~~~ Because no tolerationSeconds is specified, CockroachDB will be evicted immediately from the tainted worker node. @@ -325,20 +339,22 @@ A pod with a *topology spread constraint* must satisfy its conditions when being ### Add a topology spread constraint -Specify pod topology spread constraints in the `cockroachdb.crdbCluster.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. +Specify pod topology spread constraints in the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. The following topology spread constraint ensures that CockroachDB pods deployed with the label `environment=production` will not be unevenly distributed across zones by more than `1` pod: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: DoNotSchedule - labelSelector: - matchLabels: - environment: production + podTemplate: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + environment: production ~~~ The `DoNotSchedule` condition prevents labeled pods from being scheduled onto Kubernetes worker nodes when doing so would fail to meet the spread and topology constraints specified with `maxSkew` and `topologyKey`, respectively. @@ -349,15 +365,17 @@ For more context on how these rules work, see the [Kubernetes documentation](htt To assist in working with your cluster, you can add labels and annotations to your resources. -Specify labels in `cockroachdb.crdbCluster.podLabels` and annotations in `cockroachdb.crdbCluster.podAnnotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify labels in `cockroachdb.crdbCluster.podTemplate.metadata.labels` and annotations in `cockroachdb.crdbCluster.podTemplate.metadata.annotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - podLabels: - app.kubernetes.io/version: v25.1.4 - podAnnotations - operator: https://raw.githubusercontent.com/cockroachdb/helm-charts/refs/heads/master/cockroachdb-parent/charts/cockroachdb/values.yaml + podTemplate: + metadata: + labels: + app.kubernetes.io/name: "cockroachdb" + annotations: + kubernetes.io/description: "This is a CockroachDB pod." ~~~ To verify that the labels and annotations were applied to a pod, for example, run `kubectl describe pod {pod-name}`. diff --git a/src/current/v26.1/cockroachdb-operator-performance.md b/src/current/v26.1/cockroachdb-operator-performance.md index 309b0b44024..5a950c4a380 100644 --- a/src/current/v26.1/cockroachdb-operator-performance.md +++ b/src/current/v26.1/cockroachdb-operator-performance.md @@ -177,10 +177,12 @@ Once you've picked out an amount of CPU and memory to reserve for Cockroach, con ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi ~~~ When you initialize the cluster, check that all the CockroachDB pods are scheduled successfully. If you see any get stuck in the pending state, run `kubectl describe pod {podname}` and check the `Events` for information about why they're still pending. You may need to manually preempt pods on one or more nodes by running `kubectl delete pod` on them to make room for the CockroachDB pods. As long as the pods you delete were created by a higher-level Kubernetes object such as a `Deployment`, they'll be safely recreated on another node. @@ -194,12 +196,14 @@ To set resource limits, in addition to the [resource requests](#resource-request ~~~ yaml cockroachdb: crdbCluster: - resources: - requests: - cpu: 3500m - memory: 12300Mi - limits: - memory: 12300Mi + podTemplate: + spec: + resources: + requests: + cpu: 3500m + memory: 12300Mi + limits: + memory: 12300Mi ~~~ Pods will be limited to their reserved resources and are unlikely to be preempted, except in rare cases. This will not improve performance on an underutilized Kubernetes cluster, but provides more predictable performance as other workloads run. @@ -266,27 +270,29 @@ Client applications such as benchmarking applications running on the same machin ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: kubernetes.io/hostname - - weight: 99 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAntiAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: kubernetes.io/hostname + - weight: 99 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The preceding configuration will first prefer to put the `loadgen` pods on different nodes from each other, which is important for the fault tolerance of the `loadgen` pods themselves. As a secondary priority, it will attempt to put the pods on nodes that do not already have a running `CockroachDB` pod. This will ensure the best possible balance of fault tolerance and performance for the load generator and CockroachDB cluster. diff --git a/src/current/v26.1/configure-cockroachdb-operator.md b/src/current/v26.1/configure-cockroachdb-operator.md index 9029f6bd2fa..79f99233604 100644 --- a/src/current/v26.1/configure-cockroachdb-operator.md +++ b/src/current/v26.1/configure-cockroachdb-operator.md @@ -27,18 +27,20 @@ You can set the CPU and memory resources allocated to the CockroachDB container 1 CPU in Kubernetes is equivalent to 1 vCPU or 1 hyperthread. For best practices on provisioning CPU and memory for CockroachDB, refer to the [Production Checklist]({% link {{ page.version.version }}/recommended-production-settings.md %}#hardware). {{site.data.alerts.end}} -Specify CPU and memory values in `cockroachdb.crdbCluster.resources.limits` and `cockroachdb.crdbCluster.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify CPU and memory values in `cockroachdb.crdbCluster.podTemplate.spec.resources.limits` and `cockroachdb.crdbCluster.podTemplate.spec.resources.requests` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - resources: - limits: - cpu: 4000m - memory: 16Gi - requests: - cpu: 4000m - memory: 16Gi + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi ~~~ Apply the new settings to the cluster: diff --git a/src/current/v26.1/deploy-cockroachdb-with-cockroachdb-operator.md b/src/current/v26.1/deploy-cockroachdb-with-cockroachdb-operator.md index 98d791d77ba..c946f0a294e 100644 --- a/src/current/v26.1/deploy-cockroachdb-with-cockroachdb-operator.md +++ b/src/current/v26.1/deploy-cockroachdb-with-cockroachdb-operator.md @@ -252,7 +252,21 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st If you intend to deploy CockroachDB nodes across multiple different regions, follow the additional steps described in [Deploy across multiple regions](#deploy-across-multiple-regions). {{site.data.alerts.callout_end}} -1. Uncomment and modify `cockroachdb.crdbCluster.resources` in the values file with the CPU and memory requests and limits for each node to use. The default values are 4vCPU and 16GiB of memory: +1. Modify `cockroachdb.crdbCluster.podTemplate.spec.resources` in the values file with the CPU and memory limits and requests for each node to use. For example, to define default values of 4vCPU and 16GiB of memory: + + ~~~ yaml + cockroachdb: + crdbCluster: + podTemplate: + spec: + resources: + limits: + cpu: 4000m + memory: 16Gi + requests: + cpu: 4000m + memory: 16gi + ~~~ For more information on configuring node resource allocation, refer to [Resource management]({% link {{ page.version.version }}/configure-cockroachdb-operator.md %}) @@ -302,13 +316,18 @@ For bare metal deployments, the specific Kubernetes infrastructure deployment st cockroach start --locality region=us-central1,zone=us-central1-c,dc=dc2 ~~~ - Optionally, review the `cockroachdb.crdbCluster.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: + Optionally, review the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` configuration and set `topologyKey` to the `nodeLabel` value of a locality level that has distinct values for each node. By default the lowest locality level is `zone`, so the following configuration sets that value as the `topologyKey`: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - topologyKey: topology.kubernetes.io/zone + podTemplate: + spec: + topologySpreadConstraints: + # maxSkew defines the degree to which the pods can be unevenly distributed. + - maxSkew: 1 + # topologyKey defines the key for topology spread. + topologyKey: topology.kubernetes.io/zone ~~~ For more information on localities and topology planning, see the [topology patterns documentation]({% link {{ page.version.version }}/topology-patterns.md %}). diff --git a/src/current/v26.1/override-templates-cockroachdb-operator.md b/src/current/v26.1/override-templates-cockroachdb-operator.md index 482c985f62a..9504949cec7 100644 --- a/src/current/v26.1/override-templates-cockroachdb-operator.md +++ b/src/current/v26.1/override-templates-cockroachdb-operator.md @@ -10,7 +10,7 @@ The {{ site.data.products.cockroachdb-operator }} provides abstractions that sim - A default pod specification is used for the CockroachDB Kubernetes pod. - The `values.yaml` configuration maps to a subset of `cockroach start` flags when CockroachDB is initialized. -This page describes configuration options that allow advanced users to manually override the pod template and `cockroach start` flags as needed for deployment. +This page describes configuration options that allow advanced users to manually override pod template metadata and `cockroach start` flags as needed for deployment. {{site.data.alerts.callout_info}} The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ page.version.version }}/cockroachdb-feature-availability.md %}). @@ -24,6 +24,8 @@ This page describes advanced configurations that override the supported default The `cockroachdb.crdbCluster.podTemplate` field allows you to override the default pod metadata and specification configured by the {{ site.data.products.cockroachdb-operator }}. The values in this field are merged with the default pod specification, where settings in `podTemplate` override any values in the default. +The `podTemplate` field includes a `containers` field that specifies the container name and image that the template is applied to. By default, this resolves to the `cockroachdb` container name and can be excluded from modifications to the CockroachDB pod template YAML. If needed, you can provide specific images for the `containers` and `initContainers`. For example, the following `podTemplate` configuration specifies a custom init container: + ~~~ yaml cockroachdb: crdbCluster: @@ -35,37 +37,7 @@ cockroachdb: # initContainers captures the list of init containers for CockroachDB pods. initContainers: - name : cockroachdb-init - image: us-docker.pkg.dev/cockroach-cloud-images/data-plane/init-container@sha256:c3e4ba851802a429c7f76c639a64b9152d206cebb31162c1760f05e98f7c4254 - # containers captures the list of containers for CockroachDB pods. - containers: - - name: cockroachdb - image: cockroachdb/cockroach:v25.4.0 - # imagePullSecrets captures the secrets for fetching images from private registries. - imagePullSecrets: [] -~~~ - -At least one value for `containers` must be specified if any part of `podTemplate` is being modified. For example, the following `podTemplate` configuration overrides pod anti-affinity behavior and specifies a default `cockroachdb/cockroach:v25.4.0` container image: - -~~~ yaml -cockroachdb: - crdbCluster: - podTemplate: - spec: - affinity: - podAntiAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app.kubernetes.io/component - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname - containers: - - name: cockroachdb - image: cockroachdb/cockroach:v25.4.0 + image: us-docker.pkg.dev/cockroach-cloud-images/data-plane/init-container@sha256:example1234567890abcdefghijklmnopqrstuvwxyz ~~~ ## Override the default `cockroach start` flags diff --git a/src/current/v26.1/schedule-cockroachdb-operator.md b/src/current/v26.1/schedule-cockroachdb-operator.md index 6b3b5563ec6..4630ea6d9d1 100644 --- a/src/current/v26.1/schedule-cockroachdb-operator.md +++ b/src/current/v26.1/schedule-cockroachdb-operator.md @@ -17,16 +17,18 @@ The {{ site.data.products.cockroachdb-operator }} is in [Preview]({% link {{ pag A pod with a *node selector* will be scheduled onto a worker node that has matching [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/), or key-value pairs. -Specify the labels in `cockroachdb.crdbCluster.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. +Specify the labels in `cockroachdb.crdbCluster.podTemplate.spec.nodeSelector` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `nodeSelector` labels, the node must match all of them. The following configuration causes CockroachDB pods to be scheduled onto worker nodes that have *both* the labels `worker-pool-name=crdb-workers` and `kubernetes.io/arch=amd64`: ~~~ yaml cockroachdb: crdbCluster: - nodeSelector: - worker-pool-name: crdb-workers - kubernetes.io/arch: amd64 + podTemplate: + spec: + nodeSelector: + worker-pool-name: crdb-workers + kubernetes.io/arch: amd64 ~~~ For an example of labeling nodes, see [Scheduling CockroachDB onto labeled nodes](#example-scheduling-cockroachdb-onto-labeled-nodes). @@ -46,30 +48,32 @@ For an example, see [Scheduling CockroachDB onto labeled nodes](#example-schedul ### Add a node affinity -Specify node affinities in `cockroachdb.crdbCluster.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify node affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.nodeAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The following configuration requires that CockroachDB pods are scheduled onto worker nodes running a Linux operating system, with a preference against worker nodes in the `us-east4-b` availability zone. ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: kubernetes.io/os - operator: In - values: - - linux - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - preference: - matchExpressions: - - key: topology.kubernetes.io/zone - operator: NotIn - values: - - us-east4-b + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/os + operator: In + values: + - linux + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + preference: + matchExpressions: + - key: topology.kubernetes.io/zone + operator: NotIn + values: + - us-east4-b ~~~ The `requiredDuringSchedulingIgnoredDuringExecution` node affinity rule, using the `In` operator, requires CockroachDB pods to be scheduled onto nodes with the matching label `kubernetes.io/os=linux`. It will not evict pods that are already running on nodes that do not match the affinity requirements. @@ -80,7 +84,7 @@ For more context on how these rules work, see the [Kubernetes documentation](htt ### Add a pod affinity or anti-affinity -Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.affinity.podAffinity` and `cockroachdb.crdbCluster.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. +Specify pod affinities and node anti-affinities in `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAffinity` and `cockroachdb.crdbCluster.podTemplate.spec.affinity.podAntiAffinity` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `matchExpressions` labels, the node must match all of them. If you specify multiple `values` for a label, the node can match any of the values. The {{ site.data.products.cockroachdb-operator }} hard-codes the pod template to only allow one pod per Kubernetes node. If you need to override this value, you can [override the pod template]({% link {{ page.version.version }}/override-templates-cockroachdb-operator.md %}#override-the-default-pod). @@ -89,27 +93,29 @@ The following configuration attempts to schedule CockroachDB pods in the same zo ~~~ yaml cockroachdb: crdbCluster: - affinity: - podAffinity: - preferredDuringSchedulingIgnoredDuringExecution: - - weight: 100 - podAffinityTerm: - labelSelector: - matchExpressions: - - key: app - operator: In - values: - - loadgen - topologyKey: topology.kubernetes.io/zone - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + podAffinity: + preferredDuringSchedulingIgnoredDuringExecution: + - weight: 100 + podAffinityTerm: + labelSelector: + matchExpressions: + - key: app + operator: In + values: + - loadgen + topologyKey: topology.kubernetes.io/zone + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `preferredDuringSchedulingIgnoredDuringExecution` pod affinity rule, using the `In` operator and specified `weight`, encourages (but does not require) CockroachDB pods to be co-located with pods labeled `app=loadgen` already running in the same zone, as specified with `topologyKey`. @@ -161,24 +167,26 @@ In this example, CockroachDB has not yet been deployed to a running Kubernetes c ~~~ yaml cockroachdb: crdbCluster: - affinity: - nodeAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - nodeSelectorTerms: - - matchExpressions: - - key: node - operator: In - values: - - crdb - podAntiAffinity: - requiredDuringSchedulingIgnoredDuringExecution: - - labelSelector: - matchExpressions: - - key: app.kubernetes.io/instance - operator: In - values: - - cockroachdb - topologyKey: kubernetes.io/hostname + podTemplate: + spec: + affinity: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: node + operator: In + values: + - crdb + podAntiAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + - labelSelector: + matchExpressions: + - key: app.kubernetes.io/instance + operator: In + values: + - cockroachdb + topologyKey: kubernetes.io/hostname ~~~ The `nodeAffinity` rule requires CockroachDB pods to be scheduled onto worker nodes with the label `node=crdb`. The `podAntiAffinity` rule requires CockroachDB pods not to be co-located on a worker node, as specified with `topologyKey`. @@ -217,18 +225,20 @@ For an example, see [Evicting CockroachDB from a running worker node](#example-e ### Add a toleration -Specify pod tolerations in the `cockroachdb.crdbCluster.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). +Specify pod tolerations in the `cockroachdb.crdbCluster.podTemplate.spec.tolerations` object in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). The following toleration matches a taint with the specified key, value, and `NoSchedule` effect, using the `Equal` operator. A toleration that uses the `Equal` operator must include a `value` field: ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Equal" - value: "example" - effect: "NoSchedule" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Equal" + value: "example" + effect: "NoSchedule" ~~~ A `NoSchedule` taint on a node prevents pods from being scheduled onto the node. The matching toleration allows a pod to be scheduled onto the node. A `NoSchedule` toleration is therefore best included before [deploying the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). @@ -242,11 +252,13 @@ The following toleration matches every taint with the specified key and `NoExecu ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" - tolerationSeconds: 3600 + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" + tolerationSeconds: 3600 ~~~ A `NoExecute` taint on a node prevents pods from being scheduled onto the node, and evicts pods from the node if they are already running on the node. The matching toleration allows a pod to be scheduled onto the node, and to continue running on the node if `tolerationSeconds` is not specified. If `tolerationSeconds` is specified, the pod is evicted after this number of seconds. @@ -288,10 +300,12 @@ In this example, CockroachDB has already been deployed on a Kubernetes cluster. ~~~ yaml cockroachdb: crdbCluster: - tolerations: - - key: "test" - operator: "Exists" - effect: "NoExecute" + podTemplate: + spec: + tolerations: + - key: "test" + operator: "Exists" + effect: "NoExecute" ~~~ Because no tolerationSeconds is specified, CockroachDB will be evicted immediately from the tainted worker node. @@ -325,20 +339,22 @@ A pod with a *topology spread constraint* must satisfy its conditions when being ### Add a topology spread constraint -Specify pod topology spread constraints in the `cockroachdb.crdbCluster.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. +Specify pod topology spread constraints in the `cockroachdb.crdbCluster.podTemplate.spec.topologySpreadConstraints` object of the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster). If you specify multiple `topologySpreadConstraints` objects, the matching pods must satisfy all of the constraints. The following topology spread constraint ensures that CockroachDB pods deployed with the label `environment=production` will not be unevenly distributed across zones by more than `1` pod: ~~~ yaml cockroachdb: crdbCluster: - topologySpreadConstraints: - - maxSkew: 1 - topologyKey: topology.kubernetes.io/zone - whenUnsatisfiable: DoNotSchedule - labelSelector: - matchLabels: - environment: production + podTemplate: + spec: + topologySpreadConstraints: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + whenUnsatisfiable: DoNotSchedule + labelSelector: + matchLabels: + environment: production ~~~ The `DoNotSchedule` condition prevents labeled pods from being scheduled onto Kubernetes worker nodes when doing so would fail to meet the spread and topology constraints specified with `maxSkew` and `topologyKey`, respectively. @@ -349,15 +365,17 @@ For more context on how these rules work, see the [Kubernetes documentation](htt To assist in working with your cluster, you can add labels and annotations to your resources. -Specify labels in `cockroachdb.crdbCluster.podLabels` and annotations in `cockroachdb.crdbCluster.podAnnotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): +Specify labels in `cockroachdb.crdbCluster.podTemplate.metadata.labels` and annotations in `cockroachdb.crdbCluster.podTemplate.metadata.annotations` in the values file used to [deploy the cluster]({% link {{ page.version.version }}/deploy-cockroachdb-with-cockroachdb-operator.md %}#initialize-the-cluster): ~~~ yaml cockroachdb: crdbCluster: - podLabels: - app.kubernetes.io/version: v25.1.4 - podAnnotations - operator: https://raw.githubusercontent.com/cockroachdb/helm-charts/refs/heads/master/cockroachdb-parent/charts/cockroachdb/values.yaml + podTemplate: + metadata: + labels: + app.kubernetes.io/name: "cockroachdb" + annotations: + kubernetes.io/description: "This is a CockroachDB pod." ~~~ To verify that the labels and annotations were applied to a pod, for example, run `kubectl describe pod {pod-name}`.