Skip to content

feat(rest): use vended storage credentials from LoadTable response #2651

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
plusplusjiajia wants to merge 1 commit into
base: main
Choose a base branch
from
+30 −13
Open

feat(rest): use vended storage credentials from LoadTable response #2651

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 30 additions & 13 deletions crates/catalog/rest/src/catalog.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ use crate::client::{
use crate::types::{
CatalogConfig, CommitTableRequest, CommitTableResponse, CreateNamespaceRequest,
CreateTableRequest, ListNamespaceResponse, ListTablesResponse, LoadTableResult,
NamespaceResponse, RegisterTableRequest, RenameTableRequest,
NamespaceResponse, RegisterTableRequest, RenameTableRequest, StorageCredential,
};

/// REST catalog URI
Expand Down Expand Up @@ -748,6 +748,7 @@ impl Catalog for RestCatalog {
let request = context
.client
.request(Method::POST, context.config.tables_endpoint(namespace))
.header("X-Iceberg-Access-Delegation", "vended-credentials")
.json(&CreateTableRequest {
name: creation.name,
location: creation.location,
Expand Down Expand Up @@ -791,11 +792,7 @@ impl Catalog for RestCatalog {
"Metadata location missing in `create_table` response!",
))?;

let config = response
.config
.into_iter()
.chain(self.user_config.props.clone())
.collect();
let config = table_file_io_config(&response, &self.user_config.props);

let file_io = self
.load_file_io(Some(metadata_location), Some(config))
Expand Down Expand Up @@ -825,6 +822,8 @@ impl Catalog for RestCatalog {
let request = context
.client
.request(Method::GET, context.config.table_endpoint(table_ident))
// Opt in to vended storage credentials.
.header("X-Iceberg-Access-Delegation", "vended-credentials")

@xanderbailey xanderbailey Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Java will read these headers off the catalog config, should we do the same?

header.X-Iceberg-Access-Delegation

Link to Java

.build()?;

let http_response = context.client.query_catalog(request).await?;
Expand All @@ -848,11 +847,7 @@ impl Catalog for RestCatalog {
}
};

let config = response
.config
.into_iter()
.chain(self.user_config.props.clone())
.collect();
let config = table_file_io_config(&response, &self.user_config.props);

let file_io = self
.load_file_io(response.metadata_location.as_deref(), Some(config))
Expand Down Expand Up @@ -1062,9 +1057,12 @@ impl Catalog for RestCatalog {
}
};

// Reload for a credentialed FileIO (commit response carries no credentials).
let file_io = self
.load_file_io(Some(&response.metadata_location), None)
.await?;
.load_table(commit.identifier())

@xanderbailey xanderbailey Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A full load table seems heavy to me. Can we not wire in the pre-credentialed FileIO into this method somehow?

.await?
.file_io()
.clone();

Table::builder()
.identifier(commit.identifier().clone())
Expand All @@ -1076,6 +1074,23 @@ impl Catalog for RestCatalog {
}
}

/// FileIO props: server `config`, then vended `storage_credentials` (longest prefix wins), then user props.
fn table_file_io_config(

@xanderbailey xanderbailey Jun 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also notice that Java constructs clientByPrefix Link which keeps credentials separate per prefix. So we can't support multi-location tables here I think? Also it would silently fail in those cases which I don't think is ideal? WDYT?

response: &LoadTableResult,
user_props: &HashMap<String, String>,
) -> HashMap<String, String> {
let mut config: HashMap<String, String> = response.config.clone();
if let Some(creds) = response.storage_credentials.as_ref() {
let mut sorted: Vec<&StorageCredential> = creds.iter().collect();
sorted.sort_by_key(|c| c.prefix.len());
for cred in sorted {
config.extend(cred.config.clone());
}
}
config.extend(user_props.clone());
config
}

#[cfg(test)]
mod tests {
use std::fs::File;
Expand Down Expand Up @@ -2602,6 +2617,7 @@ mod tests {

let config_mock = create_config_mock(&mut server).await;

// GET hit twice: commit refresh + post-commit reload.
let load_table_mock = server
.mock("GET", "/v1/namespaces/ns1/tables/test1")
.with_status(200)
Expand All @@ -2610,6 +2626,7 @@ mod tests {
env!("CARGO_MANIFEST_DIR"),
"load_table_response.json"
))
.expect(2)
.create_async()
.await;

Expand Down
Loading