sync with open source how #118

lesterhaynes · 2024-03-14T08:15:08Z

Please add a meaningful description for your change here

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

* Fix inconsistent handling of Firestore Project and Database ID in routing header (resolves #36894) * Update CHANGED.md * Address reviewer comments from yixiaoshen * Fix unit tests * revert unnecessary change to read test * Revert unnecessary change to test helper

* fix retry scenario for query to table materialization * fix retry scenario for query to table materialization

* split hdfs into extra * CHANGES * tox * try/catch * test fixes * add to coverage tasks

* Test docker config to fix workflow. * enable buildx and load in python docker. * Another attempt by setting buildx project property. * Another attempt * Re-enable workflow and sickbay some failed tests for now. * Add release trigger

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.20.11 to 1.20.12. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/sns/v1.20.11...service/sns/v1.20.12) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.20.12 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.57.1 to 1.57.2. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@storage/v1.57.1...storage/v1.57.2) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-version: 1.57.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

The beam-linkage-check.sh script previously used 'git rev-parse --abbrev-ref HEAD' to save the current position before switching branches. When in detached HEAD state (common when verifying PRs), this returns the string 'HEAD' instead of a valid ref, causing the cleanup function to fail to restore the original position. This change: - Uses 'git rev-parse HEAD' to get the full commit SHA instead of branch name - Adds '-c advice.detachedHead=false' to suppress warnings in cleanup - Adds explanatory comments Fixes #20558

This reverts commit c72970d.

Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.44.0 to 0.45.0. - [Commits](golang/crypto@v0.44.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Upgrade bom. * Test fix. * License. * License.

* Remove dead code. * Lint.

Fixed documentation parameter names in WriteToKafka.__init__: - key_deserializer -> key_serializer - value_deserializer -> value_serializer Fixes #36931

…d HEAD state

* Update python-pipeline-dependencies.md * Update python-pipeline-dependencies.md

Bumps [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2) from 1.95.0 to 1.95.1. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.95.0...service/s3/v1.95.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-version: 1.95.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ks (#37292) Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.32.6 to 1.32.7. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@v1.32.6...v1.32.7) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-version: 1.32.7 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…#37155) * feat: Add support for custom prediction routes in Vertex AI inference using the `invoke_route` parameter and custom response parsing. * lint * lint 2 * fix: ensure invoke response is bytes and add type hint for request_body * test: mock `aiplatform.init` in VertexAI inference tests to prevent global state pollution. * lint * added the IT * added license * lint * updated endpoint * trigger postcommit * lint * lint * lint * fixed the response

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.20.18 to 1.20.19. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@service/emr/v1.20.18...feature/s3/manager/v1.20.19) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-version: 1.20.19 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update trivial inference for Python 3.14 * correct comment * Address review coments * avoid none case being incorrect * fix docstring

* Catch Import Error * import order

@ksobrenat32

* Update users.yml - fix bad role label @ksobrenat32 - is this correct? looks like only `beam_writer` failed. fyi @yalah5084 the previous change did not add the privileges so I'm trying to fix it here. * Update users.yml

…s. (#37294) Co-authored-by: Claude <cvandermerwe@google.com>

* Update neo4j resource manager * Fix a test log spam in it/common * Add back javadoc

Bumps [golang.org/x/sys](https://github.com/golang/sys) from 0.39.0 to 0.40.0. - [Commits](golang/sys@v0.39.0...v0.40.0) --- updated-dependencies: - dependency-name: golang.org/x/sys dependency-version: 0.40.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [golang.org/x/net](https://github.com/golang/net) from 0.48.0 to 0.49.0. - [Commits](golang/net@v0.48.0...v0.49.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-version: 0.49.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…type_hints() (#37299) * [Python] Fix AttributeError in ExternalTransform.expand by using get_type_hints() (Fixes #37289) * Fix lint: wrap long comment at line 808 * Fix yapf formatting and ignore local venv

* Remove standalone uses of the apitools HttpError class * Update sdks/python/apache_beam/testing/pipeline_verifiers_test.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * linting and formatting * more linting and formatting * even more linting --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…37324)

* Add LRU cache eviction to CachingStateProvider Fixes #37213 Implements LRU (Least Recently Used) cache eviction to prevent unbounded memory growth in long-running workers. Adds configurable maxCacheSize parameter (default: 1000 entries) and maintains LRU order using JavaScript Map's insertion order. - Add maxCacheSize constructor parameter with default value of 1000 - Implement evictIfNeeded() to remove oldest entry when cache is full - Implement touchCacheEntry() to move accessed items to end (LRU) - Add comprehensive test coverage in state_provider_test.ts This addresses the TODO comment in the code and improves reliability for production workloads. * Address review comments: size-based LRU eviction for CachingStateProvider - Fixed bug: removed incorrect evictIfNeeded() call in promise callback - Removed unnecessary this_ variable (arrow functions capture this) - Changed from count-based to size-based eviction (similar to Python statecache.py) - Added estimateSize() to calculate memory weight of cached values - Default cache weight: 100MB - Updated tests to work with weight-based eviction * Fix prettier formatting * Address review comments: circular references, eviction ordering, tests - Fixed sizeof function to handle circular references using visited Set - Fixed eviction ordering: add to cache first, then evict (fixes edge case) - Added test for oversized item that exceeds maxCacheWeight - Implemented custom sizeof instead of object-sizeof package (has Node.js compatibility issues) * Address Gemini comments: fix race condition, optimize evictIfNeeded - Fixed critical race condition in promise callback: only update cache if the entry is still the same promise we're resolving - Optimized evictIfNeeded: use entries() iterator and removed redundant checks

…37325) Bumps [keras](https://github.com/keras-team/keras) from 3.12.0 to 3.13.1. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](keras-team/keras@v3.12.0...v3.13.1) --- updated-dependencies: - dependency-name: keras dependency-version: 3.13.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add Terraform files to deploy Envoy RateLimiter * fix variables * Add nat creation command to readme * add hpa for memory * fix redability comments * fix comments * add license * Update examples/terraform/envoy-ratelimiter/README.md Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>

…d has a flaky test. (#37317)

github-actions bot added build java python examples go infra kotlin learning model labels Mar 14, 2024

pacoavila808 and others added 21 commits November 26, 2025 11:38

fix retry scenario for query to table materialization (#36912)

80ba916

* fix retry scenario for query to table materialization * fix retry scenario for query to table materialization

split hdfs into extra (#36773)

7323fb8

* split hdfs into extra * CHANGES * tox * try/catch * test fixes * add to coverage tasks

Update CHANGES to mention extras changes (#36875)

dde8937

propagate offset and record in output builder

81a838c

Revert "Fix CSP issues (#36902)" (#36929)

782a0e7

This reverts commit c72970d.

Fix runQuickstartJavaFlinkLocal (#36946)

43fdd50

Update BOM to 26.72.0 (#36919)

dbf4e5a

* Upgrade bom. * Test fix. * License. * License.

Remove python 3.9

d40da98

Remove dead code. (#36943)

e96c8dd

* Remove dead code. * Lint.

Fix typo in WriteToKafka docstring: deserializer -> serializer (#36934)

efed19a

Fixed documentation parameter names in WriteToKafka.__init__: - key_deserializer -> key_serializer - value_deserializer -> value_serializer Fixes #36931

Enforce version upper bound for scikit-learn. (#36948)

0fb3736

Merge pull request #36924: Fix linkage check script to handle detache…

c490d2d

…d HEAD state

Reflect that dill is no longer a default pickler. (#36903)

0592f4f

* Update python-pipeline-dependencies.md * Update python-pipeline-dependencies.md

Add nvidia license

d137814

Enforce hypothesis version upper bound (#36952)

bb8373f

dependabot bot and others added 30 commits January 12, 2026 10:58

update RateLimiter execution function name (#37287)

d022338

Add step to free disk space in workflow (#37246)

3783f58

Update trivial inference for Python 3.14 (#37248)

fdd25a4

* Update trivial inference for Python 3.14 * correct comment * Address review coments * avoid none case being incorrect * fix docstring

Bump Python dev container version (#37293)

1330346

Catch breaking import error (#37295)

752da38

* Catch Import Error * import order

Update users.yml - fix bad role label (#37192)

4dcc27d

* Update users.yml - fix bad role label @ksobrenat32 - is this correct? looks like only `beam_writer` failed. fyi @yalah5084 the previous change did not add the privileges so I'm trying to fix it here. * Update users.yml

Support picosecond tiemstamps when writing GenericRecord and Beam Row…

aad5864

…s. (#37294) Co-authored-by: Claude <cvandermerwe@google.com>

Add new features to changes.md (#37308)

331b08e

Bump golang.org/x/text from 0.32.0 to 0.33.0 in /sdks (#37301)

daa0f15

Update neo4j resource manager (#37310)

fc39ee6

* Update neo4j resource manager * Fix a test log spam in it/common * Add back javadoc

Revert spanner version (#37305)

097e90a

Fix java versions in release guide (#37314)

674763d

[Python] Fix AttributeError in ExternalTransform.expand by using get_…

68e0d66

…type_hints() (#37299) * [Python] Fix AttributeError in ExternalTransform.expand by using get_type_hints() (Fixes #37289) * Fix lint: wrap long comment at line 808 * Fix yapf formatting and ignore local venv

add back ticks to highlight command to be used (#37321)

3aaf839

Increase timeout for Python YAML RC validation job (#37322)

60f1981

[Do Not Merge] Test bumping httplib2 upper bound to modern version (#…

c489cb7

…37324)

fix: prevent NoSuchElementException in AppendFilesToTables (#37217)

17198be

Fix spanner io hang by closing connections (#37327)

cc53097

Pin spanner to old version in gcp bom (#37329)

82ebcb2

[Dataflow Streaming] Remove GetWorkBudgetRefresher which is unused an…

600bd61

…d has a flaky test. (#37317)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

sync with open source how #118

sync with open source how #118

Uh oh!

lesterhaynes commented Mar 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

sync with open source how #118

Are you sure you want to change the base?

sync with open source how #118

Uh oh!

Conversation

lesterhaynes commented Mar 14, 2024

GitHub Actions Tests Status (on master branch)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants