Yunikorn Scheduler not enabled in "Spark Operator with YuniKorn" module

[punkwalker]

Description

Please provide a clear and concise description of the issue you are encountering, and a reproduction of your configuration.
I deployed the solution as per instructions from the Module Page
However, it does not install Yunikorn nor it configrued the batch scheduler on Spark Operator. I had to manually update enable_yunikorn variable for TF and add --enable-batch-scheduler=true & --kube-scheduler-names=yunikorn flags to the operator

• ✋ I have searched the open/closed issues and my issue is not listed.

⚠️ Note

Before you submit an issue, please perform the following for Terraform examples:

1. Remove the local .terraform directory (! ONLY if state is stored remotely, which hopefully you are following that best practice!): rm -rf .terraform/
2. Re-initialize the project root to pull down modules: terraform init
3. Re-attempt your terraform plan or apply and check if the issue still persists

Versions

• Module version [Required]:

• Terraform version:

terraform -version
Terraform v1.5.7
on darwin_arm64

• Provider version(s):

+ provider registry.terraform.io/alekc/kubectl v2.1.3
+ provider registry.terraform.io/hashicorp/aws v5.98.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.3.7
+ provider registry.terraform.io/hashicorp/helm v2.17.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.36.0
+ provider registry.terraform.io/hashicorp/null v3.2.4
+ provider registry.terraform.io/hashicorp/random v3.7.2
+ provider registry.terraform.io/hashicorp/time v0.13.1
+ provider registry.terraform.io/hashicorp/tls v4.1.0

Reproduction Code [Required]

Steps to reproduce the behavior:
Just deploy the solution as is per the instruction from here : https://awslabs.github.io/data-on-eks/docs/blueprints/data-analytics/spark-operator-yunikorn

Expected behavior

Should have installed Yunikorn and enabled it for Spark Operator

Actual behavior

N/A

Terminal Output Screenshot(s)

Additional context

[alanty]

Yunikorn was disabled because we were seeing odd issues with it scheduling pods (#690 was the best report I was able to gather at the time).

There's been a few releases since then so we just need to bump Yunikorn to the latest and test it out again. The locust benchmarking can generate a bunch of smaller jobs to test with.