[SPARK-55410][K8S] Improve SparkKubernetesDiagnosticsSetter to use patch instead of edit API

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to improve SparkKubernetesDiagnosticsSetter to use patch instead of edit API

Why are the changes needed?

Network Efficiency

• edit requires fetching the entire resource and sending the full updated resource back.
• patch only transmits the specific changes, making it much more network-efficient.

Concurrency & Conflict Resolution

• edit typically follows a Get -> Modify -> Update (PUT) pattern. Using this pattern creates a race condition where, if another client modifies the resource in between, a 409 Conflict error occurs due to a mismatched resourceVersion.
• patch sends only the changes (delta) to the server, where the merge is handled server-side. This significantly reduces the risk of conflicts, especially for simple operations like adding an annotation.

Does this PR introduce any user-facing change?

This will reduce the overhead of K8s control plane and the chance of 409 error.

How was this patch tested?

Pass the CIs with newly updated test case.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Opus 4.5 on Claude Code

[github-actions]

JIRA Issue Information

=== Sub-task SPARK-55410 ===
Summary: Improve SparkKubernetesDiagnosticsSetter to use patch instead of edit API
Assignee: None
Status: Open
Affected: ["4.2.0"]

---

This comment was automatically generated by GitHub Actions

[dongjoon-hyun]

GitHub Action is running here.

• https://github.com/dongjoon-hyun/spark/actions/runs/21772742974