feat(kubeflow): add setup_commands run once per pod before the job

[ko3n1g]

Background / motivation

• When running synced code via the Kubeflow workdir_pvc data-mover (no image rebuild), a container may be missing a dependency (e.g. a broken release-candidate image). There was no hook to run a command once per pod before the job.

What changed

• New KubeflowExecutor.setup_commands: list[str] — shell commands rendered into the generated launch.sh between the /nemo_run symlink and the training command.

Details

• launch.sh runs once per pod, before torchrun spawns the per-GPU ranks, so each command executes exactly once per node (not once per rank) and under set -e (a failure aborts the pod). Empty by default → no change to existing launch scripts.
• Typical use: setup_commands=["uv pip install nvidia-resiliency-ext==0.6.0"] to patch a missing dep into the container venv without rebuilding the image.

# rendered launch.sh (excerpt)
ln -sfn <code_dir> /nemo_run
echo "Running setup commands..."
uv pip install nvidia-resiliency-ext==0.6.0
echo "Starting training command..."
...

Tested

• Jinja2 render verified for both populated and empty setup_commands (valid bash either way). End-to-end validation pending via a Megatron-Bridge K8s job that sets it.