feat: migrate to Helm charts, add LLMInferenceService, and support deploy on existing OpenShift by itay1551 · Pull Request #4 · rh-ai-quickstart/private-coding-assistant

itay1551 · 2026-07-02T12:24:14Z

Summary

This PR restructures the entire deployment stack and adds a new deployment mode:

Migrate ArgoCD manifests to Helm charts — both ARO and ROSA deployments now use proper Helm charts (pca-operators, pca-platform-config, pca-ai-serving, pca-devspaces) with templated values, replacing the raw ArgoCD YAML manifests.
Replace InferenceService workaround with LLMInferenceService — migrates from the manual ServingRuntime + InferenceService + Gateway + TLS Job workaround to the native OpenShift AI LLMInferenceService CRD (serving.kserve.io/v1alpha2), which automatically manages vLLM runtime, EPP router, InferencePool, HTTPRoute, and TLS.
Add local deployment on existing OpenShift — new make deploy / make undeploy targets deploy the AI serving stack on any existing OpenShift cluster via Helm, without needing Terraform provisioning or ArgoCD. Values overrides in deploy_existing_openshift/ disable cluster-scoped resources that already exist.
Upgrade RHOAI from 2.19 to 3.4 and GPU Operator from v24.9 to v25.3 for LLMInferenceService support and CUDA 12.9+ driver compatibility.
Add missing operators — NFD, LeaderWorkerSet, and cert-manager subscriptions added to the operators chart.
Fix pre-commit CI — exclude Helm templates/ from yamllint/yamlfmt (Go template syntax is not valid YAML).

Test plan

helm template renders all charts without errors
LLMInferenceService deployed on ROSA cluster via make run
vLLM pod running on GPU node, model loaded and serving inference
/v1/models and /v1/chat/completions return valid responses
Pre-commit hooks pass locally and in CI
Full Gateway routing path (Gateway → EPP → vLLM) once cluster has CPU headroom for the router

Move raw YAML manifests from argocd/ wave directories into Helm chart structures under charts/. Only REPLACE_WITH_* placeholders and hardcoded values are converted to Helm template expressions — all other manifests remain unchanged. Changes: - ROSA: 5 charts (app-of-apps, operators, platform-config, ai-serving, devspaces) - ARO: 6 charts (same + benchmarks) - Escape Go template syntax in llminferenceservice.yaml for Helm compatibility - Update Terraform gitops-bootstrap.tf: directory source -> helm source - Add gitops_charts_path variable to both platforms

LeaderWorkerSetOperator and ClusterPolicy CRs depend on CRDs that are installed by operators in wave 1. Moving them to wave 2 (platform-config) ensures the CRDs exist before the CRs are applied. Also adds required spec.daemonsets field to nvidia ClusterPolicy for GPU operator v24.9+ compatibility.

…adlock - cert-manager and leader-worker-set ArgoCD Applications referenced non-existent repo (llm-d/llm-d-playbook). Replace with OLM Subscriptions from Red Hat Operators catalog. - Move model-cache PVC to sync-wave 2 (same as LLMInferenceService) to avoid deadlock: WaitForFirstConsumer PVC stays Pending in wave 0, blocking the pod that would trigger binding in wave 2.

….io/v1 The Red Hat LWS operator CRD uses operator.openshift.io group, not leaderworkerset.x-k8s.io as originally assumed from the llm-d playbook.

- Add DSCInitialization template with serviceMesh.managementState=Unmanaged. RHOAI 2.19 defaults to Managed which requires SM 2.x, but our cluster uses SM 3.x + Gateway API. Setting Unmanaged unblocks KServe deployment. - Fix HardwareProfile apiVersion from infrastructure.opendatahub.io/v1 to dashboard.opendatahub.io/v1alpha1 (matching the actual CRD).

…hema - RHOAI 2.19 KServe requires serviceMesh.managementState=Managed with SM 2.x. Switch subscription from servicemeshoperator3 to servicemeshoperator. - Revert DSCI to Managed with full SM control plane config. - Fix HardwareProfile: use nodeSelector/tolerations at spec level (v1alpha1 schema has no scheduling field).

The dashboard.opendatahub.io/v1alpha1 HardwareProfile CRD requires spec.displayName and spec.enabled fields.

The HardwareProfile was incorrectly defined for trn1.32xlarge (Trainium) which doesn't exist in Terraform. Replace with g6e.2xlarge matching the GPU machine pool: NVIDIA L40S, taint nvidia.com/gpu, 8 vCPU / 32 GiB.

- Change rhods-operator subscription channel from stable-2.19 to stable-3.4. - Remove Service Mesh 2.x subscription (RHOAI 3.4 requires SM v2 to NOT be installed). - Update DSC to v2 API with wva.managementState=Managed for llm-d workload variant autoscaler. - Set DSCI serviceMesh.managementState=Removed. RHOAI 3.4 includes native LLMInferenceService CRD and Gateway API inference extension.

…mode RHOAI 3.4 with rawDeploymentServiceConfig: Headed uses raw K8s deployments, not KNative. Removing frees ~15 pods of CPU request capacity needed to schedule the RHOAI 3.4 operator.

RHOAI 3.4 provides LLMInferenceService through the modelsAsService component under kserve. Without it the CRD is never registered.

…ctivity Link Red Hat Connectivity Link (Kuadrant/Authorino) is not available on ROSA, blocking LLMInferenceService. Switch to standard KServe InferenceService with raw deployment mode which works immediately. Changes: - Replace LLMInferenceService with InferenceService (v1beta1) - Add vLLM NVIDIA ServingRuntime - Fix HardwareProfile to infrastructure.opendatahub.io/v1 API - Simplify HTTPRoute to point directly to predictor service - Add maas-default-gateway to pca-platform-config - Add TODO.md documenting the Connectivity Link dependency

PostSync prevents the job from blocking sync. BeforeHookCreation deletes the old job before re-creating on next sync.

…wnloads

The NVIDIA GPU operator v24.9 on OpenShift doesn't bundle NFD. Without NFD, nodes aren't labeled with nvidia.com/gpu.present=true and DaemonSets (driver, device-plugin) never deploy.

Migrate from the manual InferenceService + custom ServingRuntime + Gateway workaround to the proper OpenShift AI LLMInferenceService CRD which bundles vLLM runtime, EPP router, and Gateway API routing automatically. Changes: - Add LLMInferenceService (serving.kserve.io/v1alpha2) for Qwen3-Coder-30B - Delete obsolete workaround templates (vllm-nvidia-runtime, inferenceservice, llm-d-gateway, tls-secret-job) - Upgrade GPU Operator subscription channel from v24.9 to v25.3 - Add Makefile deploy/undeploy targets for existing OpenShift clusters - Add deploy_existing_openshift/ values overrides for helm-based deploys - Update TODO.md to reflect deployed status - Update README directory tree Tested: LLMInferenceService deployed on ROSA cluster, vLLM pod running, model loaded and serving inference successfully.

Helm templates contain Go template syntax ({{ }}) which is not valid YAML, causing yamllint and yamlfmt to fail on every CI run.

itay1551 · 2026-07-02T19:35:38Z

@Hadar301 This is ready for review,
This is working on ROSA and existing OpenShift for example I tested it on dev-2 .
Probably has some small bugs; we'll retest it when we're 98% done.

Just read the description to understand what is happending in high level.

Thanks!

Hadar301 · 2026-07-05T11:03:56Z

 .terraform.lock.hcl
 *.tfplan
 tfplan
+scripts/


Should be PCA_Deployment_*/scripts/? in case we'll have scripts/ in the future

I was experimenting with local scripts, if someone needs this in the future the person who needs it will remove it.

Hadar301 · 2026-07-05T11:09:37Z

+		--namespace $(NAMESPACE) --create-namespace \
+		-f $(DEPLOY_VALUES_DIR)/values-platform-config.yaml \
+		--set namespace=$(NAMESPACE) \
+		--set hfToken.data=$$(echo -n '$(HF_TOKEN)' | base64)


Recommendation:
Use the raw token in the Makefile and let the Helm template do the encoding.
Change the Secret template to token: {{ .Values.hfToken.raw | b64enc }} and pass
--set hfToken.raw=$(HF_TOKEN) without the shell base64.

If HF_TOKEN contains characters like +, /, or =, the shell expansion inside $$() may break (apparently it's an edge case, but I still think that the suggestion to convert to base64 in helm would be better)

I agree, good catch, I will update it.
Thanks!

Hadar301 · 2026-07-05T11:10:16Z

Is it an empty file?

Yeah, will be used for future values,
What do you think, you perfer it empty, or should I just remove the file?

We can keep it, maybe add a comment that states that purpose of the file

Hadar301 · 2026-07-05T11:10:42Z

Same here: Is it an empty file?

same as above^

Hadar301 · 2026-07-05T11:15:49Z

The old LLMInferenceService.yaml mounted a 100Gi PVC (model-cache) via kserve-provision-location for model weight caching. This is missing here — without it, the model gets re-downloaded from HuggingFace on every pod restart (7-13 min cold start vs 3-5 min with PVC). Is this intentional?

I added it back, it was causing me some problems for different ArgoCD waves, I think I got it right right now

Hadar301

Looks good! Added a few comments with suggestions.

Restructure PCA_Deployment_ARO from raw ArgoCD YAML manifests into six Helm charts under charts/, matching the pattern from PR rh-ai-quickstart#4. Templatize repo URL, target revision, chart paths, and HF token with Helm values. Update Terraform gitops-bootstrap to pass Helm parameters instead of using directory mode. Operator changes: - Replace dead llm-d-playbook ArgoCD Applications with OLM Subscriptions for cert-manager and LeaderWorkerSet - Remove Service Mesh and Serverless operators (not needed with RHOAI 3.x) - Add Node Feature Discovery operator (GPU Operator dependency) - Remove invalid lws-operator-cr.yaml Platform changes: - Upgrade DataScienceCluster to v2 API with RHOAI 3.x schema - Fix HardwareProfile to use scheduling.node (replaces nodeSelectors) - Remove useOpenShiftDriverToolkit from ClusterPolicy - Prefix DevWorkspace template with underscore (reference only) Root changes: - Add make deploy/undeploy targets for existing OpenShift clusters - Exclude Helm templates from yamllint/yamlfmt - Add deploy_existing_openshift/ values overrides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rename hfToken.data → hfToken.raw across ARO/ROSA charts, Terraform, and Makefile. Templates now apply b64enc so callers pass plaintext instead of pre-encoded base64. Default placeholders changed from CHANGE_ME to __overridden_at_deploy_time__.

…uster deploy Gate HardwareProfiles, Neuron runtime template, MaaS gateway, and NFD instance behind clusterResources (default true). Add a model-cache PVC template and mount it in LLMInferenceService. The existing-cluster values override sets clusterResources: false.

itay1551 · 2026-07-06T11:44:22Z

@Hadar301 I updated the PR with your suggestions,
In addition, I was adding a skill to deploy on an existing OpenShift cluster, the local deployment needs improvments,

That's going to be my next PR.

Restructure PCA_Deployment_ARO from raw ArgoCD YAML manifests into six Helm charts under charts/, matching the pattern from PR rh-ai-quickstart#4. Templatize repo URL, target revision, chart paths, and HF token with Helm values. Update Terraform gitops-bootstrap to pass Helm parameters instead of using directory mode. Operator changes: - Replace dead llm-d-playbook ArgoCD Applications with OLM Subscriptions for cert-manager and LeaderWorkerSet - Remove Service Mesh and Serverless operators (not needed with RHOAI 3.x) - Add Node Feature Discovery operator (GPU Operator dependency) - Remove invalid lws-operator-cr.yaml Platform changes: - Upgrade DataScienceCluster to v2 API with RHOAI 3.x schema - Fix HardwareProfile to use scheduling.node (replaces nodeSelectors) - Remove useOpenShiftDriverToolkit from ClusterPolicy - Prefix DevWorkspace template with underscore (reference only) Root changes: - Add make deploy/undeploy targets for existing OpenShift clusters - Exclude Helm templates from yamllint/yamlfmt - Add deploy_existing_openshift/ values overrides Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

itay1551 marked this pull request as draft July 2, 2026 12:24

itay1551 force-pushed the feat/helm-chart-migration branch from 5b33860 to 7101c4d Compare July 2, 2026 13:27

itay1551 added 5 commits July 2, 2026 16:29

fix: correct LeaderWorkerSetOperator apiVersion to operator.openshift…

77a64ed

….io/v1 The Red Hat LWS operator CRD uses operator.openshift.io group, not leaderworkerset.x-k8s.io as originally assumed from the llm-d playbook.

itay1551 force-pushed the feat/helm-chart-migration branch from 7101c4d to 7fe7804 Compare July 2, 2026 13:31

itay1551 added 16 commits July 2, 2026 16:32

fix: add required displayName and enabled fields to HardwareProfile

ea37411

The dashboard.opendatahub.io/v1alpha1 HardwareProfile CRD requires spec.displayName and spec.enabled fields.

fix: align HardwareProfile with Terraform GPU pool (g6e.2xlarge / L40S)

faf2764

The HardwareProfile was incorrectly defined for trn1.32xlarge (Trainium) which doesn't exist in Terraform. Replace with g6e.2xlarge matching the GPU machine pool: NVIDIA L40S, taint nvidia.com/gpu, 8 vCPU / 32 GiB.

Remove Serverless operator — not needed for RHOAI 3.4 raw deployment …

32a86b7

…mode RHOAI 3.4 with rawDeploymentServiceConfig: Headed uses raw K8s deployments, not KNative. Removing frees ~15 pods of CPU request capacity needed to schedule the RHOAI 3.4 operator.

Enable modelsAsService in DSC for LLMInferenceService CRD

fa77ac4

RHOAI 3.4 provides LLMInferenceService through the modelsAsService component under kserve. Without it the CRD is never registered.

Fix HardwareProfile scheduling type: Node (capital N)

a5635e5

Fix TLS job: use ubi9/ubi (has openssl), increase backoffLimit

97b6598

Add destination.namespace to pca-ai-serving app definition

347ad95

Fix TLS job CN length — use short CN with SAN for full hostname

4f4e457

Move TLS job to PostSync hook, add scripts/ to gitignore

e4bf8e1

PostSync prevents the job from blocking sync. BeforeHookCreation deletes the old job before re-creating on next sync.

Fix InferenceService: remove invalid volumeMounts from predictor level

30d1181

Remove unused model-cache PVC — KServe storage initializer handles do…

cd9c3f4

…wnloads

Add NFD operator — required for GPU detection by NVIDIA operator

01c909d

The NVIDIA GPU operator v24.9 on OpenShift doesn't bundle NFD. Without NFD, nodes aren't labeled with nvidia.com/gpu.present=true and DaemonSets (driver, device-plugin) never deploy.

itay1551 changed the title ~~feat: migrate ArgoCD manifests to Helm charts~~ feat: deploy vLLM via LLMInferenceService (llm-d) with Helm charts Jul 2, 2026

style: remove extra blank lines in values.yaml (yamlfmt)

64375ca

itay1551 changed the title ~~feat: deploy vLLM via LLMInferenceService (llm-d) with Helm charts~~ feat: Migrate to Helm charts and add Openshift deployment without provisioning Jul 2, 2026

fix: exclude Helm templates from yamllint/yamlfmt pre-commit hooks

b0d1926

Helm templates contain Go template syntax ({{ }}) which is not valid YAML, causing yamllint and yamlfmt to fail on every CI run.

itay1551 changed the title ~~feat: Migrate to Helm charts and add Openshift deployment without provisioning~~ feat: migrate to Helm charts, add LLMInferenceService, and support deploy on existing OpenShift Jul 2, 2026

itay1551 marked this pull request as ready for review July 2, 2026 19:34

itay1551 requested a review from Hadar301 July 2, 2026 19:34

Hadar301 reviewed Jul 5, 2026

View reviewed changes

Hadar301 previously approved these changes Jul 5, 2026

View reviewed changes

itay1551 added 4 commits July 6, 2026 14:34

docs: add deploy-existing-openshift Cursor skill

2e78466

chore: remove outdated ROSA TODO.md

d37cf81

itay1551 dismissed Hadar301’s stale review via d37cf81 July 6, 2026 11:34

Hadar301 self-requested a review July 6, 2026 11:47

Hadar301 approved these changes Jul 6, 2026

View reviewed changes

itay1551 merged commit b531d38 into main Jul 6, 2026
1 check passed

itay1551 deleted the feat/helm-chart-migration branch July 6, 2026 11:48

Uh oh!

Conversation

itay1551 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

itay1551 commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hadar301 left a comment

Choose a reason for hiding this comment

Uh oh!

itay1551 commented Jul 6, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

itay1551 commented Jul 2, 2026 •

edited

Loading

itay1551 commented Jul 2, 2026 •

edited

Loading