Skip to content

Document securityContext for OpenShift upgrade hook jobs#2874

Open
pnalubandhu-a11y wants to merge 2 commits into
mainfrom
sa/openshift-upgrade-hook-scc
Open

Document securityContext for OpenShift upgrade hook jobs#2874
pnalubandhu-a11y wants to merge 2 commits into
mainfrom
sa/openshift-upgrade-hook-scc

Conversation

@pnalubandhu-a11y

Copy link
Copy Markdown

What

Adds an "Upgrade hook jobs" note to the self-managed operator page's OpenShift section.

Why

The operator runs pre-upgrade/post-upgrade hook Jobs during upgrades — for example the app-rename hook, which also runs when you enable Weave. Those hook pods default to UID 999 (inherited from wandb-base), so on OpenShift restricted-v2 rejects them. The Helm upgrade then fails and the custom resource can get stuck in Loading, with nothing in the docs explaining why.

Change

Documents setting appRenamePreHook.podSecurityContext and appRenamePostHook.podSecurityContext to the project's UID range — the same fix already used for the main components.

Testing

Reproduced on OCP 4.21: enabling Weave triggered pre-upgrade hooks failed: Job <release>-app-rename-pre-hook not ready, and the Helm release went to failed. Validated the fix with helm template on wandb-base — setting the hook's podSecurityContext renders the Job pod with the given runAsUser/fsGroup (in-range) instead of the default 999.

The pre/post-upgrade hook jobs (e.g. app-rename, which runs on Weave enable)
default to UID 999 and get rejected by restricted-v2, failing the upgrade and
leaving the CR stuck in Loading. Document setting their podSecurityContext to
the project UID range, same as the main components.
@pnalubandhu-a11y pnalubandhu-a11y requested a review from a team as a code owner July 7, 2026 18:33
@pnalubandhu-a11y

pnalubandhu-a11y commented Jul 7, 2026

Copy link
Copy Markdown
Author

Sorry for the broad ping — narrowing this. cc @wandb/delivery-tooling-team 🙂

Context: this is one of four small docs PRs fixing the W&B self-managed on OpenShift install guide, based on a real OCP 4.21 deployment. This one (#2874) documents that the operator's pre/post-upgrade hook jobs default to UID 999, so restricted-v2 rejects them and the Helm upgrade wedges (CR stuck Loading) — the fix is setting their podSecurityContext to the project UID range. Companions: #2869, #2870, #2872. @wandb/docs-team is the reviewer.

@blalor

blalor commented Jul 7, 2026

Copy link
Copy Markdown

Please pick a narrower group for at-mentioning. There are 70 people on platform-group and at least one of us has no idea what this is about. 😂

@mintlify

mintlify Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
wandb 🟢 Ready View Preview Jul 7, 2026, 6:56 PM

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

📚 Mintlify Preview Links

🔗 View Full Preview

📝 Changed (1 total)

📄 Pages (1)

File Preview
platform/hosting/self-managed/operator.mdx Operator

🤖 Generated automatically when Mintlify deployment succeeds
📍 Deployment: f8731ec at 2026-07-09 17:02:45 UTC

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

🔗 Link Checker Results

All links are valid!

No broken links were detected.

Checked against: https://wb-21fd5541-sa-openshift-upgrade-hook-scc.mintlify.app

@pnalubandhu-a11y

Copy link
Copy Markdown
Author

@blalor good call — narrowed it to @wandb/delivery-tooling-team and added a one-line summary so nobody lands on it cold. Did the same across the companion PRs (#2869, #2870, #2872). Thanks!

@johndmulhausen

Copy link
Copy Markdown
Contributor

📋 Validation of technical assertions

Every claim below was checked against the truth on the ground using Glean (internal engineering/support knowledge) and Sourcegraph (the implementing code in wandb/helm-charts and wandb/operator), then put through an adversarial verification pass.

10 assertions · ✅ 8 confirmed · ✅ 2 confirmed (qualified) · 🟡 0 partial · ⚠️ 0 reframed · ⛔ 0 could not validate · ❌ 0 refuted.

Nothing was refuted. Two claims are confirmed with a scope qualifier: the app-rename hook is not Weave-specific (it runs on every upgrade; enabling Weave is one example), and the operator's failure phase is nuanced (Loading is set right before the blocking Helm apply, but a returned apply error transitions the CR to Invalid Config before the next requeue returns it to Loading). Both are accurate as written given the doc's hedged wording ("for example", "can get stuck"). See Scope notes for a pre-existing key-nesting inconsistency on the same page that is outside this PR's diff.

# Assertion in the docs Verdict Evidence (truth on the ground) Source
1 The operator runs pre-upgrade and post-upgrade hook Jobs during upgrades · operator.mdx:103 ✅ Confirmed appRenamePreHook.helmHook.hook: "pre-upgrade" and appRenamePostHook.helmHook.hook: "post-upgrade"; job.yaml renders the helm.sh/hook annotation; the rendered manifest carries "helm.sh/hook": "pre-upgrade" / "post-upgrade" values.yaml#L932-L996; _helpers.tpl#L66-L68; default.snap#L5246-L5262
2 The app-rename hook also runs when you enable Weave · operator.mdx:103 ✅ Confirmed (qualified) The app-rename hooks are pre-upgrade/post-upgrade Helm hooks, so they run on every helm upgrade; enabling Weave is one upgrade that triggers them. Qualifier: not Weave-specific. The doc's "for example … also runs" wording is accurate. Corroborated by an internal support thread. values.yaml#L939-L960
3 These hook pods default to UID 999, like the main components (inherited from wandb-base) · operator.mdx:103 ✅ Confirmed wandb-base default podSecurityContext.runAsUser: 999; the rendered app-rename hook pod shows runAsUser: 999 / runAsGroup: 0 / fsGroup: 0. app, api, console are all wandb-base aliases sharing this default. wandb-base/values.yaml#L101-L105; default.snap#L5314-L5319
4 On OpenShift, restricted-v2 rejects UID-999 hook pods · operator.mdx:103 ✅ Confirmed restricted-v2 uses RunAsUser: MustRunAsRange, drawing UIDs from the project's openshift.io/sa.scc.uid-range. A pod that explicitly sets a UID outside that range is denied: "unable to validate against any security context constraint: [ … runAsUser: Invalid value … must be in the ranges …]". 999 falls outside the default project range. Corroborated by an internal support thread. OpenShift SCC docs; SCC writeup
5 The rejection makes the Helm upgrade fail · operator.mdx:103 ✅ Confirmed A pre-upgrade hook Job whose pod can't be admitted never becomes ready; Helm fails a release whose pre-upgrade hook fails. Corroborated by an internal support thread reporting pre-upgrade/post-upgrade hooks failed: Job <release>-app-rename-*-hook not ready and the release going to failed. values.yaml#L958-L960; Helm hooks
6 A failed upgrade can leave the custom resource stuck in Loading · operator.mdx:103 ✅ Confirmed (qualified) Loading is a real CR status phase; the controller calls Set(status.Loading) immediately before the blocking helm Apply, so the CR sits in Loading while the failing hook is retried. Qualifier: a returned apply error sets phase Invalid Config before the next requeue returns it to Loading, so the CR may also surface as Invalid Config. The doc's "can get stuck in Loading" matches the observed symptom. status.go#L12-L17; weightsandbiases_controller.go#L253-L268
7 Setting appRenamePreHook.podSecurityContext fixes the pre-hook · operator.mdx:106 ✅ Confirmed appRenamePreHook is a top-level wandb-base alias. _pods.tpl renders the pod securityContext from merge (default dict .podData.podSecurityContext) $.root.Values.podSecurityContext, so appRenamePreHook.podSecurityContext flows into the Job pod's securityContext, overriding the default 999. Chart.yaml#L20-L24; _pods.tpl#L59-L60
8 Setting appRenamePostHook.podSecurityContext fixes the post-hook · operator.mdx:111 ✅ Confirmed appRenamePostHook is likewise a top-level wandb-base alias, rendered by the same _pods.tpl pod template. Same override path applies. Chart.yaml#L25-L29; _pods.tpl#L59-L60
9 runAsUser / runAsGroup / fsGroup are valid podSecurityContext sub-keys that render into the pod · operator.mdx:108-115 ✅ Confirmed The wandb-base default podSecurityContext uses exactly runAsUser, runAsGroup, fsGroup (plus runAsNonRoot, fsGroupChangePolicy, seccompProfile); the rendered pod shows all of them. wandb-base/values.yaml#L101-L108; default.snap#L5314-L5321
10 Using an in-range UID (like the main-components fix) resolves the rejection · operator.mdx:103,108 ✅ Confirmed Setting podSecurityContext.runAsUser deep-merges over the wandb-base default 999, so the Job pod renders with the in-range UID; an in-range UID passes restricted-v2's MustRunAsRange. The example 1000810000 is correctly presented as a placeholder ("a value within your project's UID range"). _pods.tpl#L59-L60; OpenShift SCC docs

Every source is a clickable link to where you can verify it. wandb/helm-charts links are pinned to commit 1722a47; wandb/operator links to ff42719.

Open items — recommended next steps

No blocking items — every assertion validated. Two optional considerations (neither requires a change; the doc's hedged wording already covers them):

  • Add artifacts chapter #2 — If you want to remove any chance a reader reads the app-rename hook as Weave-specific, you could say it runs "on every upgrade (for example, when you enable Weave)". Current wording is acceptable.
  • Revert "Added photos to GitHub repo" #6 — Optionally acknowledge the CR may also briefly show Invalid Config on the apply-error path. Not required given "can get stuck in Loading" is a fair description of the observed symptom.

Scope notes

  • Key-nesting inconsistency (pre-existing, outside this diff): this PR sets appRenamePreHook.podSecurityContext / appRenamePostHook.podSecurityContext, which is the key the wandb-base subchart actually consumes (_pods.tpl). The page's existing example above (the api component, unchanged by this PR) nests the setting as api.pod.securityContext — a path that does not appear to be read by the wandb-base or operator-wandb templates. Since this PR describes its fix as "the same … as the main components," reviewers may want to reconcile the two examples so the pattern is literally identical. Flagging as context only; it is not part of this diff.
  • Error string / Job name: the <release>-app-rename-pre-hook / -post-hook Job names in the PR description match the rendered chart ({{ .Release.Name }}-<jobName>), confirming the reported error strings; those strings live in the PR body, not the doc, so they are not table rows.

🤖 Generated by the CoreWeave Docs Team's Validator (beta-validator v1.0.0). Sources are linked inline; this is an accuracy aid, not a substitute for SME review.

@pnalubandhu-a11y

Copy link
Copy Markdown
Author

@wandb/docs-team gentle nudge — this one has no reviewer yet and is CI-green. Could someone pick it up? Part of the self-managed OpenShift docs series (companions #2869 / #2870 / #2872). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants