Long 1h+ running Gcloud Builds: Access Tokens Expire

[catvec]

I have a very long running container build (some R dependencies take a long time). After 1 hour I start getting the error in Skaffold output:

build [redacted-image-name] failed: error getting build status: googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

This error persisted for ~15 minutes (no more output, just this one log line in red). Followed by Skaffold exiting with a more detailed error:

Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "metadata": {
      "method": "google.devtools.cloudbuild.v1.CloudBuild.GetBuild",
      "service": "cloudbuild.googleapis.com"
    },
    "reason": "ACCESS_TOKEN_TYPE_UNSUPPORTED"
  }
]

My skaffold.yaml looks like:

apiVersion: skaffold/v4beta11
kind: Config
metadata:
  name: redacted-name

build:
  artifacts:
    - image: redacted-image-name
      context: .
      docker:
        dockerfile: Dockerfile
  
  googleCloudBuild:
    projectId: redacted-project-name
    timeout: 14400s # 4 hours

profiles:
  - name: dev
    manifests:
      # ...
    deploy:
      # ...

I ran the build with skaffold run -p dev.

I am running skaffold version v2.17.1.

I think what's happening is the auth tokens skaffold is using only last for 1 hour. And after an hour they don't get refreshed internally. If I start another build withskaffold run -p dev it starts the build just fine again.