DevWorkspace fails to backup image if internal OpenShift registry is used

[tolusha]

Description

DevWorkspace fails to backup image if internal OpenShift registry is used

How To Reproduce

Test backup following #1530

config:
  routing:
    defaultRoutingClass: basic
  workspace:
    backupCronJob:
      enable: true
      registry:
        path: image-registry.openshift-image-registry.svc.cluster.local
      schedule: '* * * * *'
    imagePullPolicy: Always

Expected behavior

Successful backup

Additional context

Using mounted service account token for registry authentication
Error: failed to validate the credentials for image-registry.openshift-image-registry.svc.cluster.local: Get "https://image-registry.openshift-image-registry.svc.cluster.local/v2/": dial tcp 172.30.17.225:443: i/o timeou

[dkwon17]

On my CRC instance, I noticed that I don't get an error and backup gets pushed successfully if the registry.path is set to the route url: default-route-openshift-image-registry.apps-crc.testing

[rohanKanojia]

@dkwon17 : You're right. I'm able to get backup working when I specify route URL instead of service URL:

config:
  workspace:
    backupCronJob:
      oras:
        extraArgs: '--insecure'
      enable: true
      schedule: '*/1 * * * *'
      registry:
        path: default-route-openshift-image-registry.apps-crc.testing

Log output after updating backup OCP test with route URL (same test fails when we use service URL):

bash backup-test-ocp-registry.sh                                                           
Context "default/api-crc-testing:6443/kubeadmin" modified.
🔍 Getting OpenShift internal registry route...
✅ Registry route: default-route-openshift-image-registry.apps-crc.testing completed.
Will stop workspace to allow backup : test-devworkspace-should-get-backup
Will keep running workspace to avoid backup : test-devworkspace-no-backup

⚙  Enabling backup CronJob with OpenShift internal registry...
DevWorkspaceOperatorConfig exists, patching...
devworkspaceoperatorconfig.controller.devfile.io/devworkspace-operator-config patched (no change)
✅ DevWorkspaceOperatorConfig configured for backup completed.
🚀 Creating DevWorkspaces...
🚀 Applying DevWorkspace manifest...
devworkspace.workspace.devfile.io/test-devworkspace-should-get-backup created
✅ DevWorkspace manifest applied completed.
⏳ Waiting for DevWorkspace 'test-devworkspace-should-get-backup' to become 🟢 *Running*...
devworkspace.workspace.devfile.io/test-devworkspace-should-get-backup condition met
✅ DevWorkspace 'test-devworkspace-should-get-backup' is running completed.
🚀 Applying DevWorkspace manifest...
devworkspace.workspace.devfile.io/test-devworkspace-no-backup created
✅ DevWorkspace manifest applied completed.
⏳ Waiting for DevWorkspace 'test-devworkspace-no-backup' to become 🟢 *Running*...
devworkspace.workspace.devfile.io/test-devworkspace-no-backup condition met
✅ DevWorkspace 'test-devworkspace-no-backup' is running completed.
✅ Both workspaces are running completed.
📝 Modifying README.md in test-devworkspace-should-get-backup...
Defaulted container "dev" out of: dev, project-clone (init), che-code-injector (init)
✅ File modified completed.
🛑 Stopping workspace: test-devworkspace-should-get-backup
devworkspace.workspace.devfile.io/test-devworkspace-should-get-backup patched
✅ Workspace stopped completed.

👀 Monitoring for backup Jobs for 600s...
✅ Backup Job detected for STOPPED workspace: devworkspace-backup-xk969 completed.
⏳ Waiting for backup Job to complete...
✅ Backup Job completed for stopped workspace completed.

📦 Verifying backup artifact (ImageStream) for test-devworkspace-should-get-backup...
✅ ImageStream exists for stopped workspace completed.
📋 ImageStream details:
image-registry.openshift-image-registry.svc:5000/openshift-operators/test-devworkspace-should-get-backup

📦 Verifying NO backup artifact for test-devworkspace-no-backup...
✅ No ImageStream for running workspace completed.

🎉 Backup validation successful
✅ Backup created ONLY for stopped workspace completed.
✅ No backup for running workspace completed.
🗑  Deleting DevWorkspaces...
devworkspace.workspace.devfile.io "test-devworkspace-should-get-backup" deleted
devworkspace.workspace.devfile.io "test-devworkspace-no-backup" deleted
🗑  Deleting ImageStreams...
✅ Cleanup complete completed.

[rohanKanojia]

Okay, regarding #1553 (comment) using route URL doesn't seem to work in case of clusters created via clusterbot.

I see failure with this error, however no failure is observed on CRC cluster following these steps:

❌ Backup Job pod is in Error state. Printing logs...
+ set -e
+ exec /workspace-recovery.sh --backup

Backing up devworkspace 'test-devworkspace-should-get-backup' in namespace 'openshift-operators' to image 'default-route-openshift-image-registry.apps-crc.testing/openshift-operators/test-devworkspace-should-get-backup:latest'
./
./.code-workspace
./web-nodejs-sample/
./web-nodejs-sample/README.md
./web-nodejs-sample/package.json
./web-nodejs-sample/.git/
./web-nodejs-sample/.git/HEAD
./web-nodejs-sample/.git/config
./web-nodejs-sample/.git/hooks/
./web-nodejs-sample/.git/hooks/prepare-commit-msg.sample
./web-nodejs-sample/.git/hooks/update.sample
./web-nodejs-sample/.git/hooks/sendemail-validate.sample
./web-nodejs-sample/.git/hooks/fsmonitor-watchman.sample
./web-nodejs-sample/.git/hooks/post-update.sample
./web-nodejs-sample/.git/hooks/pre-push.sample
./web-nodejs-sample/.git/hooks/pre-receive.sample
./web-nodejs-sample/.git/hooks/commit-msg.sample
./web-nodejs-sample/.git/hooks/pre-rebase.sample
./web-nodejs-sample/.git/hooks/applypatch-msg.sample
./web-nodejs-sample/.git/hooks/pre-applypatch.sample
./web-nodejs-sample/.git/hooks/pre-merge-commit.sample
./web-nodejs-sample/.git/hooks/pre-commit.sample
./web-nodejs-sample/.git/hooks/push-to-checkout.sample
./web-nodejs-sample/.git/description
./web-nodejs-sample/.git/objects/
./web-nodejs-sample/.git/objects/pack/
./web-nodejs-sample/.git/objects/pack/pack-00c9cb529f78a543deaf778dfaffcb012aef3c11.idx
./web-nodejs-sample/.git/objects/pack/pack-00c9cb529f78a543deaf778dfaffcb012aef3c11.rev
./web-nodejs-sample/.git/objects/pack/pack-00c9cb529f78a543deaf778dfaffcb012aef3c11.pack
./web-nodejs-sample/.git/objects/info/
./web-nodejs-sample/.git/logs/
./web-nodejs-sample/.git/logs/HEAD
./web-nodejs-sample/.git/logs/refs/
./web-nodejs-sample/.git/logs/refs/remotes/
./web-nodejs-sample/.git/logs/refs/remotes/origin/
./web-nodejs-sample/.git/logs/refs/remotes/origin/HEAD
./web-nodejs-sample/.git/logs/refs/heads/
./web-nodejs-sample/.git/logs/refs/heads/main
./web-nodejs-sample/.git/index
./web-nodejs-sample/.git/refs/
./web-nodejs-sample/.git/refs/tags/
./web-nodejs-sample/.git/refs/remotes/
./web-nodejs-sample/.git/refs/remotes/origin/
./web-nodejs-sample/.git/refs/remotes/origin/HEAD
./web-nodejs-sample/.git/refs/heads/
./web-nodejs-sample/.git/refs/heads/main
./web-nodejs-sample/.git/packed-refs
./web-nodejs-sample/.git/FETCH_HEAD
./web-nodejs-sample/.git/branches/
./web-nodejs-sample/.git/info/
./web-nodejs-sample/.git/info/exclude
./web-nodejs-sample/LICENSE
./web-nodejs-sample/.gitignore
./web-nodejs-sample/devfile.yaml
./web-nodejs-sample/.vscode/
./web-nodejs-sample/.vscode/launch.json
./web-nodejs-sample/.github/
./web-nodejs-sample/.github/CODEOWNERS
./web-nodejs-sample/.gitattributes
./web-nodejs-sample/package-lock.json
./web-nodejs-sample/app/
./web-nodejs-sample/app/app.js
Using mounted service account token for registry authentication
Error: failed to validate the credentials for default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp: lookup default-route-openshift-image-registry.apps-crc.testing on 172.30.0.10:53: no such host