Skip to content

E2E: harness-custom-jwt shard fails — leaked OAuth2 credential providers exhaust account quota #1674

Description

@aidandaly24

Description

The E2E Tests (Full Suite) workflow fails intermittently on the shard that runs e2e-tests/harness-custom-jwt.test.ts. The first test (deploys with CUSTOM_JWT authorizer configuration) fails at deploy with:

{"success":false, ...,
 "error":"The number of agent identity Oauth2 credential providers in this account has reached its limit"}

and the remaining tests in the file then cascade (No deployed targets found, expected 'local-only' to be 'deployed').

Example failed run: https://github.com/aws/agentcore-cli/actions/runs/28463247986/job/84356322175

Root Cause

harness-custom-jwt.test.ts adds a CUSTOM_JWT harness with --client-id/--client-secret, which registers a managed OAuth2 credential provider (<harness>-oauth). That provider is created imperatively pre-deploy, outside the CloudFormation stack, so the test's remove all + teardown deploy destroys the stack but leaves the OAuth2 provider behind.

The E2E globalSetup stale-cleanup only reaps ApiKey credential providers (ListApiKeyCredentialProvidersCommand), not OAuth2 providers, and nothing else deletes them. So every run leaks one E2eHrnsJwt<ts>-oauth provider permanently. The account caps OAuth2 providers at 50 (quota L-431051DC); once full, every CUSTOM_JWT deploy fails with the quota error above.

Confirmed empirically: the E2E account was at exactly 50/50 OAuth2 providers, all named E2eHrnsJwt<ts>-oauth, accumulated over ~1 week. (Leaked providers have been manually reaped to unblock CI in the interim.)

Fix (this PR)

  • Add cleanupStaleOAuth2CredentialProviders to the E2E globalSetup hook so every run auto-reaps leaked E2e-prefixed OAuth2 providers older than 30 min (mirrors the existing ApiKey / recommendation / stack reapers).
  • Add per-test teardown in harness-custom-jwt.test.ts afterAll to delete its own managed OAuth2 provider, so a single run self-cleans.

Acceptance Criteria

  • globalSetup deletes stale E2e-prefixed OAuth2 providers before tests start.
  • harness-custom-jwt.test.ts deletes its <harness>-oauth provider on teardown.
  • The E2E Tests (Full Suite) CUSTOM_JWT shard passes without hitting the OAuth2 quota.

Notes

The underlying product behavior — real users' OAuth2 providers being orphaned on remove all/teardown — is tracked separately in #1673 and is not fixed by this PR (test-side mitigation only).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingci

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions