Skip to content

Commit dacd091

Browse files
authored
docs: node img cache removal troubleshooting section (#14)
Signed-off-by: tarilabs <matteo.mortari@gmail.com>
1 parent 204304d commit dacd091

File tree

1 file changed

+26
-0
lines changed

1 file changed

+26
-0
lines changed

docs/development/openshift-setup.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -755,6 +755,32 @@ curl -k -v -H "Authorization: Bearer $TOKEN" \
755755
oc auth can-i create evaluationjobs.trustyai.opendatahub.io
756756
```
757757

758+
### Removing Cached Images from Nodes
759+
760+
**Symptoms:** After pushing a new version of an image with the same tag (e.g., `:dev`), pods continue using the old cached image.
761+
762+
When using a tag other than `latest`, Kubernetes defaults (if no otherwise specified) to `imagePullPolicy: IfNotPresent`, meaning nodes will reuse a cached image rather than pulling the updated one. If you don't want to change the `imagePullPolicy`, you can force nodes to re-fetch the image by removing it from each node's container runtime cache.
763+
764+
**Remove a cached image from all worker nodes:**
765+
766+
```bash
767+
IMG=quay.io/your-org/eval-hub:dev
768+
oc get nodes -l node-role.kubernetes.io/worker -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' | while read NODE; do
769+
oc debug node/$NODE --quiet -- chroot /host crictl rmi $IMG
770+
done
771+
```
772+
773+
Replace the `IMG` value with the image reference you want to remove.
774+
775+
After removing the cached image, delete the existing pod(s) so that the new pod pulls the updated image:
776+
777+
```bash
778+
oc delete pod -n evalhub-test -l app=eval-hub
779+
```
780+
781+
!!! tip
782+
This approach iterates over all worker nodes, which is typical in OpenShift clusters with multiple nodes. Each node maintains its own image cache, so the image must be removed from every node where it may have been pulled.
783+
758784
### Operator Not Reconciling
759785

760786
**Symptoms:** EvalHub CR created but no pods deployed

0 commit comments

Comments
 (0)