Add priorityClassName to Helm charts and docs

[marcleblanc2]

• Customer's Kubernetes cluster policy blocks pods from starting if they do not have a priorityClassName in their config.
• Added logic so that priorityClassName could be defined once, under the sourcegraph top level key, and / or under each pod's top-level key, which would override the config on the sourcegraph top level key, so the customer could configure:

sourcegraph:
  priorityClassName: p2

pgsql:
  priorityClassName: p1

Checklist

• Follow the manual testing process
• Update changelog
• Update Kubernetes update doc

Test plan

• Tested with Helm template
• Followed the manual testing process
• Will test with a customer

[michaellzc]

can we DRY things up a bit using partial template?

see

deploy-sourcegraph-helm/charts/sourcegraph/templates/gitserver/gitserver.StatefulSet.yaml

Lines 100 to 102 in 604350a

	{{- include "sourcegraph.nodeSelector" (list . "gitserver" ) | trim | nindent 6 }}
	{{- include "sourcegraph.affinity" (list . "gitserver" ) | trim | nindent 6 }}
	{{- include "sourcegraph.tolerations" (list . "gitserver" ) | trim | nindent 6 }}

deploy-sourcegraph-helm/charts/sourcegraph/templates/_helpers.tpl

Lines 109 to 120 in 604350a

	{{- define "sourcegraph.nodeSelector" -}}
	{{- $top := index . 0 }}
	{{- $service := index . 1 }}
	{{- $globalNodeSelector := (index $top.Values "sourcegraph" "nodeSelector") }}
	{{- $serviceNodeSelector := (index $top.Values $service "nodeSelector") }}
	nodeSelector:
	{{- if $serviceNodeSelector }}
	{{- $serviceNodeSelector | toYaml | trim | nindent 2 }}
	{{- else if $globalNodeSelector }}
	{{- $globalNodeSelector | toYaml | trim | nindent 2 }}
	{{- end }}
	{{- end }}

[marcleblanc2]

can we DRY things up a bit using partial template?

@michaellzc Yes, I'm working on a few fixes, starting with #780, and will continue with DRYing up our templates a bit better.

This change is holding up a customer project, so I'll fast follow with the DRY updates.