[WIP] Upgrade spark-operator to v2.1.1 and Go to v1.24.1

[kevinliao852]

Tracking issue

Related #6686, #6348

Why are the changes needed?

• These changes are required to support newer Spark Operator features

What changes were proposed in this pull request?

How was this patch tested?

Labels

Please add one or more of the following labels to categorize your PR:

• added: For new features.
• changed: For changes in existing functionality.
• deprecated: For soon-to-be-removed features.
• removed: For features being removed.
• fixed: For any bug fixed.
• security: In case of vulnerabilities

This is important to improve the readability of release notes.

Setup process

Screenshots

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

Related PRs

Docs link

[welcome]

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

• Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
• Sign off your commits (Reference: DCO Guide).

[codecov]

Codecov Report

❌ Patch coverage is 95.45455% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.95%. Comparing base (d6df4be) to head (65b15a8).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
flyteplugins/go/tasks/plugins/k8s/spark/spark.go	95.45%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6865      +/-   ##
==========================================
+ Coverage   56.94%   56.95%   +0.01%     
==========================================
  Files         929      929              
  Lines       58150    58152       +2     
==========================================
+ Hits        33114    33122       +8     
+ Misses      21993    21987       -6     
  Partials     3043     3043              

Flag	Coverage Δ
unittests-datacatalog	53.51% <ø> (ø)
unittests-flyteadmin	53.14% <ø> (ø)
unittests-flytecopilot	43.06% <ø> (ø)
unittests-flytectl	64.00% <ø> (ø)
unittests-flyteidl	75.71% <ø> (ø)
unittests-flyteplugins	60.13% <95.45%> (ø)
unittests-flytepropeller	53.60% <ø> (+0.04%)	⬆️
unittests-flytestdlib	63.29% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.