Skip to content

Use host metadata to determine GCP SA token requirement#1548

Closed
hectorcast-db wants to merge 11 commits intomainfrom
hectorcast-db/stack/port/gcp-sa-from-metadata
Closed

Use host metadata to determine GCP SA token requirement#1548
hectorcast-db wants to merge 11 commits intomainfrom
hectorcast-db/stack/port/gcp-sa-from-metadata

Conversation

@hectorcast-db
Copy link
Contributor

@hectorcast-db hectorcast-db commented Mar 16, 2026 •
edited
Loading

🥞 Stacked PR

Use this link to review incremental changes.

Summary

  • Port of Python SDK PR Use host metadata to determine GCP SA token requirement databricks-sdk-py#1322
  • Add requiresGcpSaAccessToken() that uses host metadata to determine if a GCP SA access token is needed
  • Workspace hosts (metadata has workspace_id) skip SA token entirely
  • Account hosts or when metadata is unavailable fall back to checking AccountID
  • GoogleDefaultCredentials and GoogleCredentials use this metadata-based decision

Test plan

  • TestRequiresGcpSaAccessToken_WorkspaceFromMetadata — workspace host, no SA token needed
  • TestRequiresGcpSaAccessToken_AccountFromMetadata — account host, SA token needed
  • TestRequiresGcpSaAccessToken_MetadataError_FallsBackToAccountID — error with account_id
  • TestRequiresGcpSaAccessToken_MetadataError_NoAccountID — error without account_id
  • TestRequiresGcpSaAccessToken_NoHost_WithAccountID / _NoAccountID — no-host fallback

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.

@hectorcast-db
Call resolveHostMetadata on Config init (non-fatal)
933f570 
Port of Python SDK PR #1318 and discovery URL fix from PR #1332.
Extract applyHostMetadata() from resolveHostMetadata() for reuse.
Call host metadata resolution during EnsureResolved() for unified hosts,
with non-fatal error handling (warns on failure instead of failing).
The OIDC endpoint from metadata is now treated as the OIDC root, with
/.well-known/oauth-authorization-server appended to form the full
discovery URL.

Co-authored-by: Isaac
This was referenced Mar 16, 2026
Merged
Merged
Merged
Merged
Merged
Open
@hectorcast-db
Log warning when fixHostIfNeeded fails during metadata resolution
3ae0219 
Instead of silently ignoring the error from fixHostIfNeeded(), log it
as a warning before skipping host metadata resolution.

Co-authored-by: Isaac
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port/gcp-sa-from-metadata branch from 64e25de to e75b8e4 Compare March 16, 2026 12:07
@hectorcast-db
Refactor: merge applyHostMetadata into resolveHostMetadata, call from…
e3c3308 
… EnsureResolved
@hectorcast-db
Refactor: merge applyHostMetadata into resolveHostMetadata, call from…
20b6cd4 
… EnsureResolved
@hectorcast-db
Resolve TokenAudience from host metadata for account hosts
9893d9c 
Port of Python SDK PR #1321. When host metadata indicates an account host
(no workspace_id) and account_id is present, automatically set TokenAudience
to the account_id if not already configured by the user.

Co-authored-by: Isaac
@hectorcast-db
Make GCP SA token refresh non-blocking with warning on failure
41ef7b4 
Port of Python SDK PR #1330. Add serviceToServiceVisitorWithFallback()
which logs a warning and skips the secondary header when the SA token
source fails, instead of returning an error. GoogleDefaultCredentials
now always attempts to create an SA token source regardless of config
type, falling back gracefully on failure. GoogleCredentials also uses
the fallback visitor.

Co-authored-by: Isaac
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port/gcp-sa-from-metadata branch from e75b8e4 to 7ec6f59 Compare March 16, 2026 13:08
@hectorcast-db
Consolidate serviceToServiceVisitor and serviceToServiceVisitorWithFa…
07e28b7 
…llback into one function
@hectorcast-db
Use TEST_ENVIRONMENT_TYPE for integration test filtering
0da1b0d 
Port of Python SDK PR #1317. Add support for the TEST_ENVIRONMENT_TYPE
environment variable to control which integration tests run. Supported
values: WORKSPACE, UC_WORKSPACE, ACCOUNT, UC_ACCOUNT. Falls back to
the existing DATABRICKS_ACCOUNT_ID-based logic when not set.

Co-authored-by: Isaac
@hectorcast-db
Add integration test for host metadata resolution
e9854aa 
Port of Python SDK PR #1332. Verifies that EnsureResolved with
Experimental_IsUnifiedHost populates account_id and workspace_id
from the host metadata endpoint on workspace environments.

Co-authored-by: Isaac
@hectorcast-db
Remove unified flag usage, rely on host metadata
fae626d 
Port of Python SDK PR #1331. HostType() no longer returns UnifiedHost;
host type is determined solely by URL pattern. EnsureResolved() now
always resolves host metadata when a host is configured (not gated
behind Experimental_IsUnifiedHost). IsAccountClient() no longer panics
on unified hosts. ConfigType() returns WorkspaceConfig for account
hosts with WorkspaceID. buildHostCommand() no longer has a UnifiedHost
case. getOidcEndpoints() and getOAuthArgument() remove UnifiedHost
cases, relying on DiscoveryURL from metadata.

The Experimental_IsUnifiedHost field and UnifiedHost const remain in
the codebase for backward compatibility but are no longer checked.

Note: codegen templates in service/ files still check for UnifiedHost
and need to be updated in the codegen repo to use cfg.WorkspaceID
directly.

Co-authored-by: Isaac
@hectorcast-db
Use host metadata to determine GCP SA token requirement
ecb1dbe 
Port of Python SDK PR #1322. Add requiresGcpSaAccessToken() which
checks host metadata to determine if a GCP SA access token is needed.
For workspace hosts (metadata has workspace_id), the SA token is
skipped. For account hosts or when metadata is unavailable, falls
back to checking AccountID. GoogleDefaultCredentials and
GoogleCredentials now use this metadata-based decision instead of
always attempting SA token creation.

Co-authored-by: Isaac
@hectorcast-db hectorcast-db force-pushed the hectorcast-db/stack/port/gcp-sa-from-metadata branch from 7ec6f59 to ecb1dbe Compare March 16, 2026 13:33
@github-actions
Copy link

github-actions bot commented Mar 16, 2026

If integration tests don't run automatically, an authorized user can run them manually by following the instructions below:

Trigger:
go/deco-tests-run/sdk-go

Inputs:

  • PR number: 1548
  • Commit SHA: ecb1dbeed4ed1990a74895c6ced958c05f16ffef

Checks will be approved automatically on success.

github-merge-queue bot pushed a commit that referenced this pull request Mar 16, 2026
@hectorcast-db
Call resolveHostMetadata on Config init (#1542)
42f84f1 
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-go/pull/1542/files)
to review incremental changes.
-
[**stack/port/resolve-host-metadata-on-init**](#1542)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1542/files)]
-
[stack/port/resolve-token-audience-from-metadata](#1543)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1543/files/20b6cd4abc1a3284d586c88f802c4b7df2678062..9893d9cbbfe8baab7f7aeacb8ce7faf49026c86a)]
-
[stack/port/gcp-sa-token-non-blocking](#1544)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1544/files/9893d9cbbfe8baab7f7aeacb8ce7faf49026c86a..07e28b7aef05ada2f357f87faa749c6990be8173)]
-
[stack/port/test-environment-type](#1545)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1545/files/07e28b7aef05ada2f357f87faa749c6990be8173..0da1b0d546ab8842dffbd50aa55fb136bbeffddf)]
-
[stack/port/host-metadata-integration-test](#1546)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1546/files/0da1b0d546ab8842dffbd50aa55fb136bbeffddf..e9854aad19dc522ffe8def175bef3a3eabface2b)]
-
[stack/port/remove-unified-flag](#1547)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1547/files/e9854aad19dc522ffe8def175bef3a3eabface2b..fae626deb92c4671a0c8aa0f1e3e6bad1f8c5cc6)]
-
[stack/port/gcp-sa-from-metadata](#1548)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1548/files/fae626deb92c4671a0c8aa0f1e3e6bad1f8c5cc6..ecb1dbeed4ed1990a74895c6ced958c05f16ffef)]

---------
## Summary
- Port of Python SDK PR
databricks/databricks-sdk-py#1318 and discovery
URL fix from PR
databricks/databricks-sdk-py#1332
- Extract `applyHostMetadata()` from `resolveHostMetadata()` for reuse
during config init
- Call host metadata resolution during `EnsureResolved()` for unified
hosts (gated behind `Experimental_IsUnifiedHost`), with non-fatal error
handling (warns on failure)
- OIDC endpoint from metadata is now treated as the OIDC root, with
`/.well-known/oauth-authorization-server` appended to form the full
discovery URL

## Test plan
- `TestEnsureResolved_ResolvesHostMetadata_WhenUnifiedHost` — verifies
fields populated from metadata
- `TestEnsureResolved_HostMetadataFailure_NonFatal` — 500 response,
config still resolves
- `TestEnsureResolved_HostMetadata_NoOidcEndpoint_NonFatal` — missing
oidc_endpoint, no error
-
`TestEnsureResolved_HostMetadata_MissingAccountIdWithPlaceholder_Warns`
— template needs account_id but missing
- Existing `resolveHostMetadata` tests updated for new discovery URL
format

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.
github-merge-queue bot pushed a commit that referenced this pull request Mar 17, 2026
@hectorcast-db
Resolve TokenAudience from host metadata for account hosts (#1543)
2a95760 
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-go/pull/1543/files)
to review incremental changes.
-
[**stack/port/resolve-token-audience-from-metadata**](#1543)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1543/files)]
-
[stack/port/gcp-sa-token-non-blocking](#1544)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1544/files/01d8ab7988cf049ece2bf295acc9218f6dd82e07..ab7e3d841888a136a21a2c95549392f137d6b523)]
-
[stack/port/test-environment-type](#1545)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1545/files/ab7e3d841888a136a21a2c95549392f137d6b523..0e20fea87196f5b177313bfb6c3ef5c8ec678bf0)]
-
[stack/port/host-metadata-integration-test](#1546)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1546/files/0e20fea87196f5b177313bfb6c3ef5c8ec678bf0..f9d041dbd73e531215a30dab6b0441d192f4bfb5)]
-
[stack/port/remove-unified-flag](#1547)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1547/files/f9d041dbd73e531215a30dab6b0441d192f4bfb5..086adf0511ffbff5cc9edb9dc5159b5dad9b7299)]
-
[stack/port/gcp-sa-from-metadata](#1548)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1548/files/086adf0511ffbff5cc9edb9dc5159b5dad9b7299..c8a54deafc36e70c86f869231f8b5d09b044f28f)]

---------
## Summary
- Port of Python SDK PR
databricks/databricks-sdk-py#1321
- When host metadata indicates an account host (no workspace_id) and
account_id is present, automatically set `TokenAudience` to the
account_id if not already configured

## Test plan
- `TestApplyHostMetadata_SetsTokenAudienceForAccountHost` — no
workspace_id, has account_id → set
- `TestApplyHostMetadata_NoTokenAudienceForWorkspaceHost` — has
workspace_id → not set
- `TestApplyHostMetadata_DoesNotOverrideExistingTokenAudience` — pre-set
value preserved

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.
github-merge-queue bot pushed a commit that referenced this pull request Mar 17, 2026
@hectorcast-db
Make GCP SA token refresh non-blocking with warning on failure (#1544)
146612b 
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-go/pull/1544/files)
to review incremental changes.
-
[**stack/port/gcp-sa-token-non-blocking**](#1544)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1544/files)]
-
[stack/port/test-environment-type](#1545)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1545/files/bd038478c97820339e8964bfd74457dabfa945ad..2be31bf5c15ab0d33857e90e8504425f34489d63)]
-
[stack/port/host-metadata-integration-test](#1546)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1546/files/2be31bf5c15ab0d33857e90e8504425f34489d63..b950d35820b3aefac7f9f3d12c837ddf5836ed80)]
-
[stack/port/remove-unified-flag](#1547)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1547/files/b950d35820b3aefac7f9f3d12c837ddf5836ed80..221415789356aa846dfb0f346d690b67bd5c2aa4)]
-
[stack/port/gcp-sa-from-metadata](#1548)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1548/files/221415789356aa846dfb0f346d690b67bd5c2aa4..8130b1f2566f65af5c40439ba1063b7f5f477835)]

---------
## Summary
- Port of Python SDK PR
databricks/databricks-sdk-py#1330
- Add `serviceToServiceVisitorWithFallback()` that logs a warning and
skips the secondary header when the SA token source fails, instead of
returning an error
- `GoogleDefaultCredentials` now always attempts to create an SA token
source regardless of config type, falling back gracefully on failure
- `GoogleCredentials` also uses the fallback visitor

## Test plan
- `TestServiceToServiceVisitorWithFallback_BothSucceed`
- `TestServiceToServiceVisitorWithFallback_SecondaryFails_SkipsHeader`
- `TestServiceToServiceVisitorWithFallback_PrimaryFails_ReturnsError`

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.
github-merge-queue bot pushed a commit that referenced this pull request Mar 18, 2026
@hectorcast-db
Use TEST_ENVIRONMENT_TYPE for integration test filtering (#1545)
567e047 
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-go/pull/1545/files)
to review incremental changes.
-
[**stack/port/test-environment-type**](#1545)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1545/files)]
-
[stack/port/host-metadata-integration-test](#1546)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1546/files/c5bb35f157c24cbeb92396330626119a72cb2de7..996e3f148d614483fbd3302e8c5a75fbf2c32afc)]
-
[stack/port/remove-unified-flag](#1547)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1547/files/996e3f148d614483fbd3302e8c5a75fbf2c32afc..5bfb6b121ee68f765a5ed4cb194f58372c9d7bf5)]
-
[stack/port/gcp-sa-from-metadata](#1548)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1548/files/5bfb6b121ee68f765a5ed4cb194f58372c9d7bf5..6bfe5a46df8e98136ae1fa675158f5babc50ede4)]

---------
## Summary
- Port of Python SDK PR
databricks/databricks-sdk-py#1317
- Add `TEST_ENVIRONMENT_TYPE` environment variable support for
integration test filtering
- Supported values: `WORKSPACE`, `UC_WORKSPACE`, `ACCOUNT`, `UC_ACCOUNT`
- Falls back to existing `DATABRICKS_ACCOUNT_ID`-based logic when not
set

## Test plan
- Integration test preludes updated; backward compatible with existing
CI

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.
github-merge-queue bot pushed a commit that referenced this pull request Mar 18, 2026
@hectorcast-db
Add integration test for host metadata resolution (#1546)
beb18fe 
## 🥞 Stacked PR
Use this
[link](https://github.com/databricks/databricks-sdk-go/pull/1546/files)
to review incremental changes.
-
[**stack/port/host-metadata-integration-test**](#1546)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1546/files)]
-
[stack/port/remove-unified-flag](#1547)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1547/files/e715791f64e4510f1646cdfe8987b940d68e16a7..51c58114402bcbff9ec1038f50134dc052c89020)]
-
[stack/port/gcp-sa-from-metadata](#1548)
[[Files
changed](https://github.com/databricks/databricks-sdk-go/pull/1548/files/51c58114402bcbff9ec1038f50134dc052c89020..9fc7a606d80f252123f07bed759c3dbb4e5fd864)]

---------
## Summary
- Port of Python SDK PR
databricks/databricks-sdk-py#1332 (integration
test only)
- Add integration test verifying that `EnsureResolved` populates
`account_id` and `workspace_id` from the host metadata endpoint on
workspace environments

## Test plan
- `TestHostMetadataResolution` — requires workspace cloud env with
`CLOUD_ENV` set

NO_CHANGELOG=true

This pull request was AI-assisted by Isaac.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@hectorcast-db