Merge upstream changes#19
Conversation
* REST: Promote the S3 signing endpoint to the main spec Dev ML discussion: https://lists.apache.org/thread/2kqdqb46j7jww36wwg4txv6pl2hqq9w7 This commit promotes the S3 remote signing endpoint from an AWS-specific implementation to a first-class REST catalog API endpoint. This enables other storage providers (GCS, Azure, etc.) to eventually reuse the same signing endpoint pattern without duplicating the API definition. Summary of changes: - Added `/v1/{prefix}/namespaces/{namespace}/tables/{table}/sign/{provider}` endpoint to the main REST catalog OpenAPI spec. - Defined `RemoteSignRequest`, `RemoteSignResult` and `RemoteSignResponse` schemas. - Defined a new `provider` request body parameter in order to disambiguate requests from different storage providers. - Deprecated the separate `s3-signer-open-api.yaml` spec from the AWS module (for removal). - Updated the Python client.
…he#15049) Introduces foundational types for V4 manifest support These types follow the https://s.apache.org/iceberg-single-file-commit and will be used by subsequent PRs for manifest reading/writing. For now, we are adding these as package-private interfaces in core, and eventually we will move them into api.
…5734) Signal to the TableOperations that there is no retry needed for files which do not exist.
…5825) * Spark: Control merge schema evolution by table property Add a new table property write.spark.auto-schema-evolution (default true) that controls whether the AUTOMATIC_SCHEMA_EVOLUTION capability is reported to Spark. When set to false, Spark's MERGE WITH SCHEMA EVOLUTION no longer evolves the target table schema. Also add a guard in SparkWriteBuilder to reject mergeSchema write option when the property is disabled. * Remove unnecessary validation from SparkWriteBuilder The capability removal in SparkTable is sufficient to control schema evolution. The mergeSchema write option path already requires accept-any-schema, making a second gate redundant. * Address review comments - Rename property to write.spark.auto-schema-evolution.enabled - Rename caps to tableCapabilities in computeCapabilities - Add explicit = in ALTER TABLE SET TBLPROPERTIES test SQL
This fixes Russell's feedback on apache#15049 to avoid version-specific language that will go stale.
Adds a build-time check that prevents accidental transitive dependency leaks into shipped shadow JARs and distribution archives. A checked-in runtime-deps.txt baseline lists every dependency resolved into each bundled artifact. checkRuntimeDeps compares resolved deps against the baseline and fails the build with a clear diff on mismatch, wired into the check lifecycle so it runs in CI automatically. This guards all 11 bundled modules: Spark runtime (3.4, 3.5, 4.0, 4.1), Flink runtime (1.20, 2.0, 2.1), cloud bundles (AWS, Azure, GCP), and Kafka Connect runtime.
Co-authored-by: manuzhang <owenzhang1990@gmail.com>
Bumps [mkdocs-redirects](https://github.com/ProperDocs/properdocs-redirects) from 1.2.2 to 1.2.3. - [Release notes](https://github.com/ProperDocs/properdocs-redirects/releases) - [Commits](ProperDocs/mkdocs-redirects@v1.2.2...v1.2.3) --- updated-dependencies: - dependency-name: mkdocs-redirects dependency-version: 1.2.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) from 7.6.0 to 8.0.0. - [Release notes](https://github.com/astral-sh/setup-uv/releases) - [Commits](astral-sh/setup-uv@37802ad...cec2083) --- updated-dependencies: - dependency-name: astral-sh/setup-uv dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…15886) Bumps [org.openapitools:openapi-generator-gradle-plugin](https://github.com/OpenAPITools/openapi-generator) from 7.20.0 to 7.21.0. - [Release notes](https://github.com/OpenAPITools/openapi-generator/releases) - [Changelog](https://github.com/OpenAPITools/openapi-generator/blob/master/docs/release-summary.md) - [Commits](OpenAPITools/openapi-generator@v7.20.0...v7.21.0) --- updated-dependencies: - dependency-name: org.openapitools:openapi-generator-gradle-plugin dependency-version: 7.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15889) Bumps [com.google.cloud:libraries-bom](https://github.com/googleapis/java-cloud-bom) from 26.78.0 to 26.79.0. - [Release notes](https://github.com/googleapis/java-cloud-bom/releases) - [Commits](googleapis/java-cloud-bom@v26.78.0...v26.79.0) --- updated-dependencies: - dependency-name: com.google.cloud:libraries-bom dependency-version: 26.79.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…e#15890) Bumps software.amazon.awssdk:bom from 2.42.18 to 2.42.23. --- updated-dependencies: - dependency-name: software.amazon.awssdk:bom dependency-version: 2.42.23 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps `jetty` from 12.1.5 to 12.1.7. Updates `org.eclipse.jetty:jetty-server` from 12.1.5 to 12.1.7 Updates `org.eclipse.jetty.ee10:jetty-ee10-servlet` from 12.1.5 to 12.1.7 --- updated-dependencies: - dependency-name: org.eclipse.jetty:jetty-server dependency-version: 12.1.7 dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.eclipse.jetty.ee10:jetty-ee10-servlet dependency-version: 12.1.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pache#15891) Bumps [io.netty:netty-buffer](https://github.com/netty/netty) from 4.2.10.Final to 4.2.12.Final. - [Release notes](https://github.com/netty/netty/releases) - [Commits](netty/netty@netty-4.2.10.Final...netty-4.2.12.Final) --- updated-dependencies: - dependency-name: io.netty:netty-buffer dependency-version: 4.2.12.Final dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* AWS: Add chunked encoding configuration for S3 requests * add testMultipartUploadWithChunkedEncodingDisabled * update open api define * update * update default value * update case * assert file contents in testMultipartUploadWithChunkedEncoding * Remove s3.chunked-encoding-enabled config entry from REST catalog open API spec * Use IOUtil.readFully for reliable reads in TestS3MultipartUpload * ensure testIo is properly closed * retrigger CI * Change chunked encoding default to true to match AWS SDK behavior * Fix test to verify explicit disable of chunked encoding instead of duplicating default
* Make MAX_WAIT_TIME_MS configurable for RESTTableScan * fix style * fix checkstyle: add hasMessage check to assertThatThrownBy Co-authored-by: Isaac * Address Amogh's comments * address comments
…ion with Hive catalog (apache#15814) * Docs: Add Hive Metastore schema validation warnings for DROP COLUMN and REORDER When using a Hive catalog, ALTER TABLE DROP COLUMN (non-last column) and ALTER COLUMN REORDER fail because the Hive Metastore validates schema changes by comparing column types positionally. Dropping a middle column shifts subsequent columns, causing HMS to reject the change as an incompatible type change via MetaStoreUtils#throwExceptionIfIncompatibleColTypeChange. Add warning admonitions to spark-ddl.md (DROP COLUMN and REORDER sections) and flink-ddl.md (Hive catalog section) documenting the limitation, workaround (hive.metastore.disallow.incompatible.col.type.changes=false), and trade-off (Hive engine can no longer read the table). * Docs: Clarify HMS workaround for embedded vs remote deployment * Docs: add more warning for spark-ddl.md * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Manu Zhang <OwenZhang1990@gmail.com>
Fix zizmor ref-version-mismatch audit failure caused by the rolling v7 tag moving to v7.0.1 while workflows pinned the v7.0.0 hash. Regenerate Spark 4.1 runtime-deps.txt to reflect dependency changes from recent dependabot bumps. Made-with: Cursor Co-authored-by: Neelesh Salian <n_salian@apple.com>
…ache#15941) This reverts commit 9a939d6.
…6277) Co-authored-by: Cursor <cursoragent@cursor.com>
…1.0 to 7.22.0 (apache#16278) Co-authored-by: Cursor <cursoragent@cursor.com>
|
|
2f27dea to
238ea6d
Compare
Signed-off-by: Robert Kruszewski <github@robertk.io>
|
I have messed up merge and squashed, need to remake this |
No description provided.