Document applying config changes on self-managed (restart pods or Reloader) by pnalubandhu-a11y · Pull Request #2872 · wandb/docs

pnalubandhu-a11y · 2026-07-07T16:03:18Z

What

Adds an "Applying configuration changes" section to the self-managed operator page.

Why

After install, changing config in the CR (especially SSO/OIDC) updates the ConfigMaps but doesn't restart the running pods, so the change looks like it had no effect. The most common symptom is "I changed SSO settings but login still behaves the same."

The chart annotates deployments with reloader.stakater.com/auto: "true", but that only works if the Stakater Reloader controller is installed — which isn't the case on a default cluster.

Change

New section with the two ways to apply a config change:

Manually restart the affected deployments (rollout restart app/api)
Install Stakater Reloader so it happens automatically

Testing

On OCP 4.21, an OIDC config change didn't take effect until app/api were restarted. The deployments carried the reloader annotation but no Reloader controller was installed, so nothing restarted them automatically.

…ader) Config/auth changes update the ConfigMaps but don't restart the running pods, so changes (SSO/OIDC especially) look like they didn't take effect. Add a section explaining you need to restart app/api or install Stakater Reloader.

pnalubandhu-a11y · 2026-07-07T16:03:33Z

Narrowing the ping. cc @wandb/delivery-tooling-team 🙂

Context: docs PR in the self-managed OpenShift series. Documents that config/auth changes (e.g. SSO/OIDC) need a pod restart to take effect — the deployments carry the Stakater Reloader annotation but Reloader isn't installed by default, so nothing restarts them. Companions: #2869, #2870, #2874. @wandb/docs-team is the reviewer.

mintlify · 2026-07-07T16:04:11Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
wandb	🟢 Ready	View Preview	Jul 7, 2026, 4:07 PM

github-actions · 2026-07-07T16:08:18Z

📚 Mintlify Preview Links

🔗 View Full Preview

📝 Changed (1 total)

📄 Pages (1)

File	Preview
`platform/hosting/self-managed/operator.mdx`	Operator

🤖 Generated automatically when Mintlify deployment succeeds
📍 Deployment: 462b383 at 2026-07-09 17:09:12 UTC

github-actions · 2026-07-07T16:08:35Z

🔗 Link Checker Results

✅ All links are valid!

No broken links were detected.

Checked against: https://wb-21fd5541-sa-self-managed-config-change-restart.mintlify.app

johndmulhausen · 2026-07-07T18:54:29Z

📋 Validation of technical assertions

Every claim below was checked against the truth on the ground using Sourcegraph (the implementing wandb/helm-charts code) and Glean (internal support/engineering knowledge), then put through an adversarial verification pass. Source links point only to wandb/* and public repos; internal corroboration is described generically rather than linked.
7 assertions · ✅ 6 confirmed · 🟡 1 partial · ⚠️ 0 qualified/reframed · ⛔ 0 could not validate · ❌ 0 refuted.
Nothing was outright refuted, but 4 confirmations carry a scope qualifier the doc should note, and the one partial is the copy-pasteable rollout restart command: its deployment names do not match the operator's default chart (the app deployment renders as <release>-app-bc, and there is no standalone <release>-api deployment).

#	Assertion in the docs	Verdict	Evidence (truth on the ground)	Source
1	Changing config in the CR (SSO/OIDC, env vars) makes the operator update the underlying ConfigMaps, but running pods don't automatically pick up the change · `operator.mdx:243`	✅ Confirmed (qualified)	The `operator-wandb` chart renders config into ConfigMaps (`<release>-app-configmap`, `<release>-api-configmap`, `<release>-bucket-configmap`) consumed via `envFrom`; nothing in the chart restarts pods on a ConfigMap change. Qualifier: SSO/OIDC secrets (e.g. client secret) land in a Secret, not only ConfigMaps — the Reloader bullet's "ConfigMap or Secret" is the accurate scope.	helm-charts:default.snap#L3162-L3168
2	Kubernetes does not restart pods when a mounted ConfigMap changes · `operator.mdx:243`	✅ Confirmed (qualified)	This is exactly why the chart carries the Reloader annotation (a no-op unless something restarts pods). Qualifier: a ConfigMap mounted as a volume has its files updated in place by the kubelet (no restart, but the running process keeps the old values); env-var / `subPath` ConfigMaps aren't updated at all. Either way a restart is required, so the doc's conclusion holds. Also corroborated by internal support cases (not linked).	stakater/Reloader
3	Remedy 1: restart with `kubectl -n <namespace> rollout restart deployment/<release>-app deployment/<release>-api`; `app`/`api` cover most auth/SSO changes · `operator.mdx:247,250`	🟡 Partially confirmed	Approach confirmed (an explicit `rollout restart` is the standard remedy). But the names are wrong for the operator's default chart: the app deployment renders as `<release>-app-bc` (`deploymentPostfix: "bc"` is the default) — there is no `<release>-app` — and there is no standalone `<release>-api` deployment: `api` is a ConfigMap consumed by the `app` (wandb/local) deployment. As written, the command errors with `NotFound` on a default install. Names also vary by install method/version. See Open items.	helm-charts:_helpers.tpl#L26-L29; operator-wandb/values.yaml#L748-L750; default.snap#L2830-L2834
4	The chart annotates the deployments with `reloader.stakater.com/auto: "true"` · `operator.mdx:253`	✅ Confirmed	`charts/wandb-base/templates/deployment.yaml` sets `reloader.stakater.com/auto: "true"` when `.Values.addReloaderAnnotations` is true, and the default is `true`. It renders on the app/console/etc. deployments in the default snapshot.	deployment.yaml#L11-L13; values.yaml#L78; default.snap#L2845-L2846
5	With the Reloader controller running, it restarts affected pods automatically when their ConfigMap or Secret changes · `operator.mdx:253`	✅ Confirmed	This is the documented behavior of the `reloader.stakater.com/auto: "true"` annotation: Reloader watches the ConfigMaps/Secrets a workload references and triggers a rolling restart when they change. Corroborated by internal support notes (not linked).	stakater/Reloader
6	Reloader is not installed by default; on a fresh cluster you must deploy it yourself · `operator.mdx:253`	✅ Confirmed	`operator-wandb` bundles no Reloader: its `charts/` holds only third-party deps (bufstream, clickhouse, etcd, kafka, mysql, otel, prometheus, redis, stackdriver, yace) and no chart renders a Reloader controller Deployment. The chart only annotates; it does not install the controller.	operator-wandb/charts/
7	If SSO/OIDC settings change and login behaves the same, it's almost always because pods still run the old config (the most common symptom) · `operator.mdx:256`	✅ Confirmed (qualified)	Confirmed as a well-known failure mode: SSO/OIDC config changes that don't take effect until the pods are restarted (the chart ships the Reloader annotation specifically to auto-apply such changes). Qualifier: the "almost always / most common" framing is stronger than the evidence independently quantifies; corroboration for the frequency comes from internal support cases (not linked), so treat the superlative as SME-owned.	deployment.yaml#L11-L13

Source links are wandb/* blobs (pinned to commit d122c6c) and the public stakater/Reloader repo. Behavioral claims were also corroborated against internal CoreWeave support knowledge, which is not linked here because this is an external repo.

Open items — recommended next steps

Added How-to guide title, Integrations titles, and W&B logo. Removed … #3 — the rollout restart command's deployment names don't match the operator's default chart, so it ships a command that errors on a default install. Fix the doc: either (a) tell the reader to discover names first with kubectl -n <namespace> get deployments (names carry a -bc suffix, e.g. <release>-app-bc, and there is no separate <release>-api), or (b) hardcode the current default-chart names. Confirm the canonical operator names with the PR author @pnalubandhu-a11y — on the OCP 4.21 test, were the deployments <release>-app/<release>-api or <release>-app-bc? That pins down the version behavior. Quick check on any live install: kubectl -n <namespace> get deploy -o name | grep -E 'app|api'.

Scope notes

The wandb/helm-charts evidence is from the current default render (test-configs/operator-wandb/__snapshots__/default.snap, commit d122c6c); deploymentPostfix: "bc" is the shipped default for the app and console subcharts.
Reloader's controller behavior (claims Add artifacts chapter #2, Added photos to GitHub repo #5) was verified against its public annotation semantics and internal support knowledge; the stakater/Reloader upstream repo is not indexed in our code search, so those rows cite the public repo rather than a specific line.

_{🤖 Generated by the CoreWeave Docs Team's Validator (beta-validator v1.0.0). Sources are linked inline; this is an accuracy aid, not a substitute for SME review.}

johndmulhausen · 2026-07-07T18:54:36Z

+- **Restart the affected deployments manually.** Restart whichever components consume the changed config; `app` and `api` cover most auth/SSO changes:
+
+  ```bash
+  kubectl -n <namespace> rollout restart deployment/<release>-app deployment/<release>-api


🟡 Partially confirmed — the approach (restart the affected deployments) is confirmed, but these deployment names don't match the operator's default chart, so this command errors with NotFound on a default install.

On the current operator-wandb chart the app deployment renders as <release>-app-bc (deploymentPostfix: "bc" is the default), and there is no standalone <release>-api deployment — api is a ConfigMap that the app (wandb/local) deployment consumes, so restarting app applies the API/SSO config too. Names also vary by install method/version.

Suggested change

kubectl -n <namespace> rollout restart deployment/<release>-app deployment/<release>-api

# Deployment names vary by chart version/install. On the operator's default chart the

# app deployment is <release>-app-bc, and there is no separate <release>-api deployment

# (the app deployment serves the API and consumes the SSO/OIDC config). List them first:

kubectl -n <namespace> get deployments

kubectl -n <namespace> rollout restart deployment/<release>-app-bc

Source: operator-wandb/values.yaml#L748-L750, wandb-base/_helpers.tpl#L26-L29, default.snap#L2830-L2834 · Recommended: confirm the canonical operator deployment names with @pnalubandhu-a11y (the OCP 4.21 test — were they <release>-app/<release>-api or -bc?).

🤖 CoreWeave Docs Team's Validator — partially confirmed

Review feedback: hardcoded names (<release>-app / <release>-api) don't match the default chart, where the app deployment carries a -bc suffix and names vary by version/install. List deployments first, then restart the right ones.

pnalubandhu-a11y · 2026-07-07T19:21:40Z

Good catch on #3 — fixed in 6c9470f. Since deployment names vary by version/install, the command now lists deployments first, then restarts by name (noting the -bc suffix, e.g. <release>-app-bc).

To answer the question: on the OCP 4.21 install I tested, the deployments were wandbpk-app-bc and wandbpk-api — so the app carried the -bc suffix and there was a separate api deployment. Since that differs from the default chart snapshot, discover-first is the safest guidance.

pnalubandhu-a11y · 2026-07-10T04:46:08Z

@wandb/docs-team gentle nudge — CI-green and ready for review. Part of the self-managed OpenShift docs series (companions #2869 / #2870 / #2874). Thanks!

pnalubandhu-a11y requested a review from a team as a code owner July 7, 2026 16:03

mintlify Bot deployed to staging July 7, 2026 16:07 View deployment

This was referenced Jul 7, 2026

Document securityContext for OpenShift upgrade hook jobs #2874

Open

Fix OpenShift securityContext example (podSecurityContext + fsGroup) #2869

Open

Add shared/multi-tenant cluster note (oidc-reviewer collision) #2870

Open

johndmulhausen reviewed Jul 7, 2026

View reviewed changes

Make the restart command discover deployment names first

6c9470f

Review feedback: hardcoded names (<release>-app / <release>-api) don't match the default chart, where the app deployment carries a -bc suffix and names vary by version/install. List deployments first, then restart the right ones.

mintlify Bot deployed to staging July 7, 2026 19:26 View deployment

Polish added content per Google Developer Style Guide

462b383

mintlify Bot deployed to staging July 9, 2026 17:08 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Document applying config changes on self-managed (restart pods or Reloader)#2872

Document applying config changes on self-managed (restart pods or Reloader)#2872
pnalubandhu-a11y wants to merge 3 commits into
mainfrom
sa/self-managed-config-change-restart

pnalubandhu-a11y commented Jul 7, 2026

Uh oh!

pnalubandhu-a11y commented Jul 7, 2026 •

edited

Loading

Uh oh!

mintlify Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 7, 2026

Uh oh!

johndmulhausen commented Jul 7, 2026

Uh oh!

johndmulhausen Jul 7, 2026

Uh oh!

pnalubandhu-a11y commented Jul 7, 2026

Uh oh!

pnalubandhu-a11y commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

-  kubectl -n <namespace> rollout restart deployment/<release>-app deployment/<release>-api
+  # Deployment names vary by chart version/install. On the operator's default chart the
+  # app deployment is <release>-app-bc, and there is no separate <release>-api deployment
+  # (the app deployment serves the API and consumes the SSO/OIDC config). List them first:
+  kubectl -n <namespace> get deployments
+  kubectl -n <namespace> rollout restart deployment/<release>-app-bc

Uh oh!

Conversation

pnalubandhu-a11y commented Jul 7, 2026

What

Why

Change

Testing

Uh oh!

pnalubandhu-a11y commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mintlify Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📚 Mintlify Preview Links

📝 Changed (1 total)

📄 Pages (1)

Uh oh!

github-actions Bot commented Jul 7, 2026

🔗 Link Checker Results

Uh oh!

johndmulhausen commented Jul 7, 2026

📋 Validation of technical assertions

Open items — recommended next steps

Scope notes

Uh oh!

johndmulhausen Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

pnalubandhu-a11y commented Jul 7, 2026

Uh oh!

pnalubandhu-a11y commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pnalubandhu-a11y commented Jul 7, 2026 •

edited

Loading

mintlify Bot commented Jul 7, 2026 •

edited

Loading

github-actions Bot commented Jul 7, 2026 •

edited

Loading