What is the problem the feature request solves?
Comet's scan currently only takes static credentials.
Iceberg-rust already supports a credential hook (AwsCredentialLoad / CustomAwsCredentialLoader in iceberg-storage-opendal). Comet is not wired to use it.
We should expose a pluggable CometCredentialProvider SPI and connect it to that existing hook.
Describe the potential solution
Thread the Java credential provider already in use with Spark into Comet, and make JNI calls when needed to get creds.
Additional context
No response
What is the problem the feature request solves?
Comet's scan currently only takes static credentials.
Iceberg-rust already supports a credential hook (AwsCredentialLoad / CustomAwsCredentialLoader in iceberg-storage-opendal). Comet is not wired to use it.
We should expose a pluggable CometCredentialProvider SPI and connect it to that existing hook.
Describe the potential solution
Thread the Java credential provider already in use with Spark into Comet, and make JNI calls when needed to get creds.
Additional context
No response