Skip to content

chore: Migrate gsutil usage to gcloud storage #10002

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ruhi-chaddha merged 1 commit into from
Apr 17, 2026
Merged

chore: Migrate gsutil usage to gcloud storage #10002

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions deploy/setup-secret.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function download_existing_key() {
KEY_IDS=$(gcloud iam service-accounts keys list --iam-account=metrics-writer@${METRICS_PROJECT_ID}.iam.gserviceaccount.com --project=${METRICS_PROJECT_ID} --managed-by=user --format="value(name)")
while read -r KEY_ID
do
if gsutil cp gs://${BUCKET_ID}/${KEY_ID}.json ${KEY_FILE}; then
if gcloud storage cp gs://${BUCKET_ID}/${KEY_ID}.json ${KEY_FILE}; then
echo "Downloaded existing key to ${KEY_FILE}"
return 0
fi
Expand All @@ -52,8 +52,8 @@ function upload_new_key() {
fi
echo "New service account key created."
KEY_ID=$(gcloud iam service-accounts keys list --iam-account=metrics-writer@${METRICS_PROJECT_ID}.iam.gserviceaccount.com --project=${METRICS_PROJECT_ID} --managed-by=user --format="value(name)" --limit=1)
gsutil cp ${KEY_FILE} gs://${BUCKET_ID}/${KEY_ID}.json
gsutil cp ${KEY_FILE} gs://${BUCKET_ID}/${LATEST_GCS_PATH}
gcloud storage cp ${KEY_FILE} gs://${BUCKET_ID}/${KEY_ID}.json
gcloud storage cp ${KEY_FILE} gs://${BUCKET_ID}/${LATEST_GCS_PATH}
echo "New service account key uploaded to GCS."
return 0
}
Expand Down
6 changes: 3 additions & 3 deletions hack/update-lts-dependencies.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ HELM_URL=https://get.helm.sh/helm-${HELM_VERSION}-linux-${ARCH}.tar.gz
wget -q -O helm.tar.gz "${HELM_URL}"

# upload the binary to skaffold gcs bucket
gsutil -q cp helm.tar.gz gs://skaffold/deps/helm/helm-${HELM_VERSION}-linux-amd64.tar.gz
gcloud storage cp helm.tar.gz gs://skaffold/deps/helm/helm-${HELM_VERSION}-linux-amd64.tar.gz
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The original gsutil command used the -q flag for quiet output. The migrated gcloud storage cp command should include the --quiet flag to maintain the same behavior and prevent progress indicators from being printed to stdout.

Suggested change
gcloud storage cp helm.tar.gz gs://skaffold/deps/helm/helm-${HELM_VERSION}-linux-amd64.tar.gz
gcloud storage cp --quiet helm.tar.gz gs://skaffold/deps/helm/helm-${HELM_VERSION}-linux-amd64.tar.gz

Copy link
Copy Markdown
Contributor

@ruhi-chaddha ruhi-chaddha Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any specific reason why the quiet flag is no longer being included?

Copy link
Copy Markdown
Contributor Author

@bhandarivijay-png bhandarivijay-png Mar 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The omission of the --quiet flag was an oversight during the initial script conversion. Since the original gsutil command used -q to suppress progress indicators,which is critical for keeping CI/CD logs clean and preventing stdout clutter. we should definitely include it in the cloud storage version to maintain the same behavior. I'll update the migration logic to ensure this flag is preserved.


# take the shasum and put into correct digests/ dir
sha256sum helm.tar.gz > ${DIGESTS_DIR}/helm.amd64.sha256
Expand All @@ -78,7 +78,7 @@ KUSTOMIZE_URL=https://github.com/kubernetes-sigs/kustomize/releases/download/kus
wget -q -O kustomize.tar.gz "${KUSTOMIZE_URL}"

# upload the binary to skaffold gcs bucket
gsutil -q cp kustomize.tar.gz gs://skaffold/deps/kustomize/v${KUSTOMIZE_VERSION}/kustomize_v${KUSTOMIZE_VERSION}_linux_amd64.tar.gz
gcloud storage cp kustomize.tar.gz gs://skaffold/deps/kustomize/v${KUSTOMIZE_VERSION}/kustomize_v${KUSTOMIZE_VERSION}_linux_amd64.tar.gz
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The original gsutil command used the -q flag for quiet output. The migrated gcloud storage cp command should include the --quiet flag to maintain the same behavior and prevent progress indicators from being printed to stdout.

Suggested change
gcloud storage cp kustomize.tar.gz gs://skaffold/deps/kustomize/v${KUSTOMIZE_VERSION}/kustomize_v${KUSTOMIZE_VERSION}_linux_amd64.tar.gz
gcloud storage cp --quiet kustomize.tar.gz gs://skaffold/deps/kustomize/v${KUSTOMIZE_VERSION}/kustomize_v${KUSTOMIZE_VERSION}_linux_amd64.tar.gz


# take the shasum and put into correct digests/ dir
sha256sum kustomize.tar.gz > ${DIGESTS_DIR}/kustomize.amd64.sha256
Expand All @@ -88,7 +88,7 @@ KPT_URL=https://github.com/GoogleContainerTools/kpt/releases/download/v${KPT_VER
wget -q -O kpt "${KPT_URL}"

# upload the binary to skaffold gcs bucket
gsutil -q cp kpt gs://skaffold/deps/kpt/v${KPT_VERSION}/kpt_linux_amd64
gcloud storage cp kpt gs://skaffold/deps/kpt/v${KPT_VERSION}/kpt_linux_amd64
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot Mar 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The original gsutil command used the -q flag for quiet output. The migrated gcloud storage cp command should include the --quiet flag to maintain the same behavior and prevent progress indicators from being printed to stdout.

Suggested change
gcloud storage cp kpt gs://skaffold/deps/kpt/v${KPT_VERSION}/kpt_linux_amd64
gcloud storage cp --quiet kpt gs://skaffold/deps/kpt/v${KPT_VERSION}/kpt_linux_amd64


# take the shasum and put into correct digests/ dir
sha256sum kpt > ${DIGESTS_DIR}/kpt.amd64.sha256
Expand Down
8 changes: 4 additions & 4 deletions pkg/skaffold/gcs/gsutil.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import (
"github.com/GoogleContainerTools/skaffold/v2/proto/v1"
)

const GsutilExec = "gsutil"
const GsutilExec = "gcloud"

type Gsutil interface {
Copy(ctx context.Context, src, dst string, recursive bool) error
Expand All @@ -49,11 +49,11 @@ func NewGsutil() Gsutil {
return &gsutil{}
}

// Copy calls `gsutil cp [-r] <source_url> <destination_url>
// Copy calls `gcloud storage cp [--recursive] <source_url> <destination_url>
func (g *gsutil) Copy(ctx context.Context, src, dst string, recursive bool) error {
args := []string{"cp"}
args := []string{"storage", "cp"}
if recursive {
args = append(args, "-r")
args = append(args, "--recursive")
}
args = append(args, src, dst)
cmd := exec.CommandContext(ctx, GsutilExec, args...)
Expand Down
6 changes: 3 additions & 3 deletions pkg/skaffold/gcs/gsutil_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,20 +48,20 @@ func TestCopy(t *testing.T) {
description: "copy single file",
src: file,
dst: gcsFile,
commands: testutil.CmdRunOut(fmt.Sprintf("gsutil cp %s %s", file, gcsFile), "logs"),
commands: testutil.CmdRunOut(fmt.Sprintf("gcloud storage cp %s %s", file, gcsFile), "logs"),
},
{
description: "copy recursively",
src: folder,
dst: gcsFolder,
commands: testutil.CmdRunOut(fmt.Sprintf("gsutil cp -r %s %s", folder, gcsFolder), "logs"),
commands: testutil.CmdRunOut(fmt.Sprintf("gcloud storage cp --recursive %s %s", folder, gcsFolder), "logs"),
recursive: true,
},
{
description: "copy failed",
src: file,
dst: gcsFile,
commands: testutil.CmdRunOutErr(fmt.Sprintf("gsutil cp %s %s", file, gcsFile), "logs", fmt.Errorf("file not found")),
commands: testutil.CmdRunOutErr(fmt.Sprintf("gcloud storage cp %s %s", file, gcsFile), "logs", fmt.Errorf("file not found")),
shouldErr: true,
},
}
Expand Down
6 changes: 3 additions & 3 deletions vendor/cloud.google.com/go/CONTRIBUTING.md
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 1 addition & 2 deletions vendor/google.golang.org/api/cloudbuild/v1/cloudbuild-api.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.