feat(rest): use vended storage credentials from LoadTable response#2651
Open
plusplusjiajia wants to merge 1 commit into
Open
feat(rest): use vended storage credentials from LoadTable response#2651plusplusjiajia wants to merge 1 commit into
plusplusjiajia wants to merge 1 commit into
Conversation
plusplusjiajia
force-pushed
the
feat/rest-vended-credentials
branch
from
ac46430 to
894d5cf
Compare
xanderbailey
left a comment
xanderbailey
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the PR, I had a quick look and left some comments / questions.
| .client | ||
| .request(Method::GET, context.config.table_endpoint(table_ident)) | ||
| // Opt in to vended storage credentials. | ||
| .header("X-Iceberg-Access-Delegation", "vended-credentials") |
Contributor
There was a problem hiding this comment.
Java will read these headers off the catalog config, should we do the same?
header.X-Iceberg-Access-Delegation
| let file_io = self | ||
| .load_file_io(Some(&response.metadata_location), None) | ||
| .await?; | ||
| .load_table(commit.identifier()) |
Contributor
There was a problem hiding this comment.
A full load table seems heavy to me. Can we not wire in the pre-credentialed FileIO into this method somehow?
| } | ||
|
|
||
| /// FileIO props: server `config`, then vended `storage_credentials` (longest prefix wins), then user props. | ||
| fn table_file_io_config( |
Contributor
There was a problem hiding this comment.
I also notice that Java constructs clientByPrefix Link which keeps credentials separate per prefix. So we can't support multi-location tables here I think? Also it would silently fail in those cases which I don't think is ideal? WDYT?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
load_table / create_table now request vended credentials via the
X-Iceberg-Access-Delegation: vended-credentialsheader and merge the returnedstorage_credentialsinto the FileIO properties.LoadTableResultalreadyexposes
storage_credentials(its doc even says clients should prefer it overconfig), but the field was never used.Why
REST catalogs that vend per-table credentials return them in
storage_credentials. Without requesting delegation and merging them, FileIO is built without credentials and data access fails.Changes
X-Iceberg-Access-Delegation: vended-credentialson load/create.table_file_io_config()merges config + storage_credentials(longest prefix wins) + catalog props.
update_tablereloads after commit to return a credentialed FileIO(CommitTableResponse carries no credentials).