Add health check to watch.stream for silent connection drops

[Urvashi0109]

What type of PR is this?

/kind bug
/kind feature

What this PR does / why we need it:

When running a watch on Kubernetes objects (e.g., Jobs, Pods, Namespaces) and the Kubernetes control plane gets upgraded, the watch connection is silently dropped. The watcher hangs indefinitely - No exception is raised and no new events are received. This is because the TCP connection enters a half-open state where the client believes the connection is still alive, but the server side has been torn down during the upgrade.

This PR adds a _health_check_interval parameter to watch.stream() that detects silent connection drops and automatically reconnects:

• When _health_check_interval is set to a value > 0, a socket-level read timeout (_request_timeout) is configured on the HTTP connection
• If no data arrives within the specified interval, urllib3 raises a ReadTimeoutError
• The watch catches this exception and automatically reconnects using the last known resource_version, ensuring no events are missed
• The feature is disabled by default (_health_check_interval=0), preserving full backward compatibility
• When disabled, ReadTimeoutError propagates to the caller as before

Which issue(s) this PR fixes:

Fixes #2462

Special notes for your reviewer:

• This PR takes approach: leveraging urllib3's existing read timeout mechanism (_request_timeout) to break out of the blocking read, then catching the resulting ReadTimeoutError/ProtocolError exceptions
• The _ prefix on _health_check_interval follows the existing convention in this codebase (e.g., _preload_content, _request_timeout) for parameters that are consumed by the client library rather than passed to the API server
• 5 new unit tests added, all 24 tests (19 existing + 5 new) pass with zero regressions

Does this PR introduce a user-facing change?

Added `_health_check_interval` parameter to `watch.stream()` to detect and recover from silent connection drops during Kubernetes control plane upgrades. When set to a value > 0 (seconds), the watch will automatically reconnect if no data is received within the specified interval. Disabled by default for backward compatibility.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

NONE

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Urvashi0109
Once this PR has been reviewed and has the lgtm label, please assign yliaog for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• kubernetes/base/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Pull request overview

Adds an optional client-side “health check” mechanism to Watch.stream() to detect silent/half-open watch connections (e.g., during control plane upgrades) by configuring a read timeout and reconnecting automatically.

Changes:

• Add _health_check_interval parameter to Watch.stream() that sets _request_timeout (read timeout) when enabled.
• Catch ReadTimeoutError / ProtocolError during streaming to trigger a reconnect using the last known resource_version.
• Add unit tests covering reconnect behavior, default behavior, and _request_timeout handling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
kubernetes/base/watch/watch.py	Implements _health_check_interval handling, timeout configuration, and reconnect-on-timeout logic.
kubernetes/base/watch/watch_test.py	Adds unit tests for the new health-check/reconnect behavior and timeout parameter interactions.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[roycaihw]

I thought we already have multiple timeouts (supporting both client side and server side). Could you check if the existing timeouts address this problem? Or is it (HTTP connection timeout) supported in client-go only?

[roycaihw]

Here is an example of using timeout_seconds with watch:

python/examples/watch/pod_namespace_watch.py

Line 18 in bdfbaea

the `timeout_seconds` threshold and then move on to wait for another 10 events

cc @yliaog

[Urvashi0109]

Thanks for feedback @roycaihw , I will take a look 😊