AWS: restCredentialsProvider should return StsAssumeRoleCredentialsProvider when assume role configurations are set

[NathanCYee]

Feature Request / Improvement

Currently, RESTSigV4AuthSession calls the restCredentialsProvider function in AwsProperties to retrieve the credentials provider to sign the sigv4 request.

The restCredentialsProvider function currently follows the following decision chain:

1. If accessKeyId, secretAccessKey, and optionally sessionToken are set -> return a StaticCredentialsProvider
2. If clientCredentialsProvider is set, build the class using DynClasses and pass the property map clientCredentialsProviderProperties into the create function of the class
3. Otherwise return the default credential chain using DefaultCredentialsProvider.

However, this decision chain does not consider the credentials if client.factory=org.apache.iceberg.aws.AssumeRoleAwsClientFactory. The factory will define a role to assume that ISN'T used on REST catalog requests but IS used for S3FileIO, Glue Catalog, KMS, and DynamoDB operations causing divergent permissions for REST catalog operations.

Workarounds attempted:
AwsProperties allows the definition of alternative credential providers under client.credentials-provider. When this property is set, restCredentialsProvider will return the custom provider instead of the default chain.
However, this property is not compatible with software.amazon.awssdk.services.sts.auth.StsAssumeRoleCredentialsProvider since this property requires the credentials provider to have a create() or create(Map) function, while the assume role provider uses the builder pattern.

Potential Design:
Fastest path would be to update the credentialsProvider(String accessKeyId, String secretAccessKey, String sessionToken) function of AwsProperties to create and return a StsAssumeRoleCredentialsProvider if this.clientAssumeRoleArn is set. The AwsProperties class would be duplicating the createCredentialsProvider function of AssumeRoleAwsClientFactory. This could go between step 2 and 3 of the decision chain of restCredentialsProvider (before returning default credentials chain, check for clientAssumeRoleArn first and if it is set return the assume role credentials provider).

A more complete path would be to define another interface (possibly named CanCreateAwsCredentialsProvider?) that has a public function AwsCredentialsProvider createCredentialsProvider(). AssumeRoleAwsClientFactory can implement this class (and possibly other client factories such as LakeFormationAwsClientFactory). When CLIENT_FACTORY is set to a class that inherits this interface, instantiate the client factory and call the function between step 2 and 3 of the decision chain of restCredentialsProvider.

Third possible design: extract the four functions (sts, genSessionName, createCredentialsProvider, createAssumeRoleRequest) from AssumeRoleAwsClientFactory into a separate public utility class. Embed the utility class into both AwsProperties and AssumeRoleAwsClientFactory. Add a set of additional rest. properties for assume role as well.

Query engine

Spark

Willingness to contribute

• I can contribute this improvement/feature independently
• I would be willing to contribute this improvement/feature with guidance from the Iceberg community
• I cannot contribute this improvement/feature at this time

[xndai]

Thanks for reporting. Can you provide a bit more details - which REST catalog service you run into this? Is it S3 Tables REST catalog?

[NathanCYee]

Hi @xndai, thanks for the comment. I have observed this occurring in the s3 tables rest catalog so far but it should occur on the glue catalog as well since it is also SigV4 signed. I haven't tested on OSS Spark yet, will let you know reproduction steps shortly.

[GabrielBBaldez]

I dug into this and can confirm the root cause.

RESTSigV4AuthSession signs requests with AwsProperties#restCredentialsProvider(), which delegates to the internal credentialsProvider(...) decision chain. That chain only handles explicit static rest.* keys, a custom client.credentials-provider, and otherwise the DefaultCredentialsProvider — there is no branch for client.assume-role.*. So when the catalog is configured with AssumeRoleAwsClientFactory, the assumed role is applied to the S3/Glue/KMS/DynamoDB clients (via applyAssumeRoleConfigurations) but never to the SigV4 REST signing path. The REST calls silently fall through to the default credential chain — exactly the permission divergence reported here.

Opened #16794 implementing the "fastest path" from the description: an assume-role branch between the custom-provider and default-chain steps that returns an StsAssumeRoleCredentialsProvider built from the existing client.assume-role.* properties (arn, region, session name, timeout, external id, tags), mirroring AssumeRoleAwsClientFactory#createCredentialsProvider. Static rest.* keys and client.credentials-provider keep precedence, and it is covered with unit tests.

@NathanCYee apologies if you were already mid-work on this — happy to collaborate. @xndai I kept it to the minimal option-1; glad to extract a shared utility (option 3) instead if you'd prefer to avoid the small duplication with AssumeRoleAwsClientFactory.

[NathanCYee]

Hi @GabrielBBaldez,

Thanks for taking a look at this! I hope the details I gave were helpful to your implementation. Apologies for the lack of communication, this is my first issue on this project and I have been trying to get familiar with the contribution process 😅.

I can take a look at updating the documentation for the updated behavior. I was in the process of implementing option 3, happy to pull your fork and merge my in-progress work with yours.

[GabrielBBaldez]

No worries at all, and welcome — great first issue to dig into! 🙌 Your write-up was spot on and made the implementation straightforward.

Happy to join forces. To avoid duplicating effort: #16794 is up and green locally (option 1, the minimal path), so it's a working baseline. Since you're already on option 3 (the shared AssumeRole utility), that's likely the cleaner shape to fold everything into.

Either way, glad to combine and co-author — happy to add you on my fork (or you can PR your docs + option-3 work onto my branch) so it stays one clean PR crediting us both. Whatever's easiest for you!

[NathanCYee]

Thanks @GabrielBBaldez,

I set up a branch on NathanCYee/iceberg/aws-assume-role-rest-credentials that has forked yours. I will PR a docs update first, followed by the option-3 changes.