Skip to content

Merge upstream changes#19

Merged
robert3005 merged 198 commits into
mainfrom
rk/upstream
May 11, 2026
Merged

Merge upstream changes#19
robert3005 merged 198 commits into
mainfrom
rk/upstream

Conversation

@robert3005
Copy link
Copy Markdown
Member

@robert3005 robert3005 commented May 11, 2026

No description provided.

adutra and others added 30 commits March 31, 2026 14:39
* REST: Promote the S3 signing endpoint to the main spec

Dev ML discussion: https://lists.apache.org/thread/2kqdqb46j7jww36wwg4txv6pl2hqq9w7

This commit promotes the S3 remote signing endpoint from an AWS-specific implementation to a first-class REST catalog API endpoint.

This enables other storage providers (GCS, Azure, etc.) to eventually reuse the same signing endpoint pattern without duplicating the API definition.

Summary of changes:

- Added `/v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider}` endpoint to the main REST catalog OpenAPI spec.
- Defined `RemoteSignRequest`, `RemoteSignResult` and `RemoteSignResponse` schemas.
- Defined a new `provider` request body parameter in order to disambiguate requests from different storage providers.
- Deprecated the separate `s3-signer-open-api.yaml` spec from the AWS module (for removal).
- Updated the Python client.
…he#15049)

Introduces foundational types for V4 manifest support

These types follow the https://s.apache.org/iceberg-single-file-commit
and will be used by subsequent PRs for manifest reading/writing.

For now, we are adding these as package-private interfaces in core, and
eventually we will move them into api.
…5734)

Signal to the TableOperations that there is no retry needed for files which do not exist.
…5825)

* Spark: Control merge schema evolution by table property

Add a new table property write.spark.auto-schema-evolution (default true)
that controls whether the AUTOMATIC_SCHEMA_EVOLUTION capability is
reported to Spark. When set to false, Spark's MERGE WITH SCHEMA
EVOLUTION no longer evolves the target table schema.

Also add a guard in SparkWriteBuilder to reject mergeSchema write option
when the property is disabled.

* Remove unnecessary validation from SparkWriteBuilder

The capability removal in SparkTable is sufficient to control schema
evolution. The mergeSchema write option path already requires
accept-any-schema, making a second gate redundant.

* Address review comments

- Rename property to write.spark.auto-schema-evolution.enabled
- Rename caps to tableCapabilities in computeCapabilities
- Add explicit = in ALTER TABLE SET TBLPROPERTIES test SQL
This fixes Russell's feedback on apache#15049
to avoid version-specific language that will go stale.
Adds a build-time check that prevents accidental transitive dependency
leaks into shipped shadow JARs and distribution archives. A checked-in
runtime-deps.txt baseline lists every dependency resolved into each
bundled artifact. checkRuntimeDeps compares resolved deps against the
baseline and fails the build with a clear diff on mismatch, wired into
the check lifecycle so it runs in CI automatically.

This guards all 11 bundled modules: Spark runtime (3.4, 3.5, 4.0, 4.1),
Flink runtime (1.20, 2.0, 2.1), cloud bundles (AWS, Azure, GCP), and
Kafka Connect runtime.
Co-authored-by: manuzhang <owenzhang1990@gmail.com>
Bumps [mkdocs-redirects](https://github.com/ProperDocs/properdocs-redirects) from 1.2.2 to 1.2.3.
- [Release notes](https://github.com/ProperDocs/properdocs-redirects/releases)
- [Commits](ProperDocs/mkdocs-redirects@v1.2.2...v1.2.3)

---
updated-dependencies:
- dependency-name: mkdocs-redirects
  dependency-version: 1.2.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 8.0.0.
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](astral-sh/setup-uv@37802ad...cec2083)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 8.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…15886)

Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 7.20.0 to 7.21.0.
- [Release notes](https://github.com/OpenAPITools/openapi-generator/releases)
- [Changelog](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/release-summary.md)
- [Commits](OpenAPITools/openapi-generator@v7.20.0...v7.21.0)

---
updated-dependencies:
- dependency-name: org.openapitools:openapi-generator-gradle-plugin
  dependency-version: 7.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15889)

Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.78.0 to 26.79.0.
- [Release notes](https://github.com/googleapis/java-cloud-bom/releases)
- [Commits](googleapis/java-cloud-bom@v26.78.0...v26.79.0)

---
updated-dependencies:
- dependency-name: com.google.cloud:libraries-bom
  dependency-version: 26.79.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e#15890)

Bumps software.amazon.awssdk:bom from 2.42.18 to 2.42.23.

---
updated-dependencies:
- dependency-name: software.amazon.awssdk:bom
  dependency-version: 2.42.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `jetty` from 12.1.5 to 12.1.7.

Updates `org.eclipse.jetty:jetty-server` from 12.1.5 to 12.1.7

Updates `org.eclipse.jetty.ee10:jetty-ee10-servlet` from 12.1.5 to 12.1.7

---
updated-dependencies:
- dependency-name: org.eclipse.jetty:jetty-server
  dependency-version: 12.1.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: org.eclipse.jetty.ee10:jetty-ee10-servlet
  dependency-version: 12.1.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15891)

Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.2.10.Final to 4.2.12.Final.
- [Release notes](https://github.com/netty/netty/releases)
- [Commits](netty/netty@netty-4.2.10.Final...netty-4.2.12.Final)

---
updated-dependencies:
- dependency-name: io.netty:netty-buffer
  dependency-version: 4.2.12.Final
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* AWS: Add chunked encoding configuration for S3 requests

* add testMultipartUploadWithChunkedEncodingDisabled

* update open api define

* update

* update default value

* update case

* assert file contents in testMultipartUploadWithChunkedEncoding

* Remove s3.chunked-encoding-enabled config entry from REST catalog open API spec

* Use IOUtil.readFully for reliable reads in TestS3MultipartUpload

* ensure testIo is properly closed

* retrigger CI

* Change chunked encoding default to true to match AWS SDK behavior

* Fix test to verify explicit disable of chunked encoding instead of duplicating default
* Make MAX_WAIT_TIME_MS configurable for RESTTableScan

* fix style

* fix checkstyle: add hasMessage check to assertThatThrownBy

Co-authored-by: Isaac

* Address Amogh's comments

* address comments
…ion with Hive catalog (apache#15814)

* Docs: Add Hive Metastore schema validation warnings for DROP COLUMN and REORDER

When using a Hive catalog, ALTER TABLE DROP COLUMN (non-last column) and
ALTER COLUMN REORDER fail because the Hive Metastore validates schema
changes by comparing column types positionally. Dropping a middle column
shifts subsequent columns, causing HMS to reject the change as an
incompatible type change via MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange.

Add warning admonitions to spark-ddl.md (DROP COLUMN and REORDER sections)
and flink-ddl.md (Hive catalog section) documenting the limitation,
workaround (hive.metastore.disallow.incompatible.col.type.changes=false),
and trade-off (Hive engine can no longer read the table).

* Docs: Clarify HMS workaround for embedded vs remote deployment

* Docs: add more warning for spark-ddl.md

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com>
Fix zizmor ref-version-mismatch audit failure caused by the rolling
v7 tag moving to v7.0.1 while workflows pinned the v7.0.0 hash.

Regenerate Spark 4.1 runtime-deps.txt to reflect dependency changes
from recent dependabot bumps.


Made-with: Cursor

Co-authored-by: Neelesh Salian <n_salian@apple.com>
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 11, 2026

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 37 committers have signed the CLA.

❌ nastra
❌ rdblue
❌ manuzhang
❌ 0xMALVEE
❌ bharos
❌ Guosmilesmile
❌ hcrosse
❌ drexler-sky
❌ nssalian
❌ mxm
❌ RjLi13
❌ dramaticlly
❌ ebyhr
❌ talatuyarer
❌ davseitsev
❌ anoopj
❌ yadavay-amzn
❌ singhpk234
❌ swapna267
❌ kevinjqliu
❌ hemanthboyina
❌ adutra
❌ laserninja
❌ gaborkaszab
❌ sagib1
❌ soumyajit-sahu
❌ pvary
❌ milleniax
❌ stevenzwu
❌ oguzhanunlu
❌ Kurtiscwright
❌ aihuaxu
❌ thisismiller
❌ mukund-thakur
❌ amogh-jahagirdar
❌ huaxingao
❌ rexminnis
You have signed the CLA already but the status is still pending? Let us recheck it.

@robert3005 robert3005 force-pushed the rk/upstream branch 3 times, most recently from 2f27dea to 238ea6d Compare May 11, 2026 13:04
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 robert3005 merged commit 14d1907 into main May 11, 2026
21 of 46 checks passed
@robert3005
Copy link
Copy Markdown
Member Author

I have messed up merge and squashed, need to remake this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.