Skip to content

Add retry for GCE 403 concurrent operations quota errors#16625

Open
golgeek wants to merge 1 commit intoGoogleCloudPlatform:mainfrom
golgeek:golgeek/gce-403-rate-limit
Open

Add retry for GCE 403 concurrent operations quota errors#16625
golgeek wants to merge 1 commit intoGoogleCloudPlatform:mainfrom
golgeek:golgeek/gce-403-rate-limit

Conversation

@golgeek
Copy link

@golgeek golgeek commented Mar 5, 2026

GCE returns HTTP 403 (not 429) when the concurrent operations quota is exceeded.
The error includes reason CONCURRENT_OPERATIONS_QUOTA_EXCEEDED in the error details.
This is a transient condition that clears as soon as in-flight operations complete, but the provider currently treats it as a hard failure.

This adds a new global retry predicate is403ConcurrentOperationsQuotaError that matches 403 errors with CONCURRENT_OPERATIONS_QUOTA_EXCEEDED in the ErrorInfo details and retries them automatically.

Fixes hashicorp/terraform-provider-google#9207

Release Note Template

transport: Added automatic retry for GCE 403 errors with reason `CONCURRENT_OPERATIONS_QUOTA_EXCEEDED`

Signed-off-by: Ludovic Leroux <ludo.leroux@cockroachlabs.com>
@modular-magician modular-magician added the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Mar 5, 2026
@github-actions github-actions bot requested a review from trodge March 5, 2026 17:27
@github-actions
Copy link

github-actions bot commented Mar 5, 2026

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@trodge, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

@modular-magician modular-magician removed the awaiting-approval Pull requests that need reviewer's approval to run presubmit tests label Mar 5, 2026
@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

google provider: Diff ( 2 files changed, 85 insertions(+))
google-beta provider: Diff ( 2 files changed, 85 insertions(+))
terraform-google-conversion: Diff ( 1 file changed, 34 insertions(+))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add retries for GCE Quota 403s

2 participants