Add Kubernetes support via SkyPilot #2083

romilbhardwaj · 2025-12-08T06:11:41Z

Hi Monarch community 👋

This PR adds support for running Monarch workloads on Kubernetes and cloud VMs via SkyPilot.

Adds a new SkyPilotJob class that provisions Monarch workers on K8s (or any of the 20+ cloud providers supported by SkyPilot):

job = SkyPilotJob(
    meshes={"workers": 2},
    resources=sky.Resources(cloud=sky.Kubernetes(), accelerators="H200:1"), # SkyPilot resources object https://skypilot.readthedocs.io/en/latest/reference/yaml-spec.html#resources
    cluster_name="my-cluster",
    file_mounts={"/local/path": "/worker/path"}, # For any code/data sync
)

With this I was able to run the getting started example from Monarch docs on my multi-node Coreweave Kubernetes cluster with H200s. Full example in examples/skypilot_getting_started.py.

Notes:

Cold start: Currently builds Monarch from source on workers to keep client/worker versions in sync (~7-10 min). A pre-built container image would reduce this to ~30 seconds.
Networking: Assumes workers are in the same network as the client. If client-worker communication is not very chatty, SkyPilot's SSH tunneling could remove this requirement.

This is an early PR - I'm looking for feedback on the direction. I'm a maintainer of the SkyPilot project and was trying to run Monarch on my k8s cluster. Figured an integration with SkyPilot would be a good fit.

Happy to iterate on the implementation if this aligns with where you'd like Monarch to go.

Summary: This update introduces two new optional parameters, `workdir` and `file_mounts`, to the `SkyPilotJob` class. The `workdir` parameter allows users to specify a local directory to sync with the cluster, while `file_mounts` enables additional file mounts by mapping remote paths to local paths. These enhancements improve the flexibility and usability of job configurations in SkyPilot.

- Add SKY_README.md with comprehensive documentation: - Architecture overview - Implementation details - Usage examples - Troubleshooting guide - Networking considerations for Kubernetes - Add python/examples/skypilot_getting_started.py: - Example script demonstrating Monarch actors on SkyPilot - Supports multiple clouds (Kubernetes, AWS, GCP, Azure) - Configurable via command-line arguments - Update skypilot.py: - Add host mesh initialization wait - Improve logging for debugging - Fix worker command environment variable setup

meta-cla · 2025-12-08T06:11:47Z

Hi @romilbhardwaj!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

johnwhumphreys · 2025-12-09T02:32:09Z

@romilbhardwaj - I just saw this; sorry for missing it all day. I'll check it out in the morning and circle back :).

colin2328 · 2025-12-09T18:52:10Z

very cool! is this ready for review? (if so push it out of draft state?)

examples/skypilot/skypilot_job.py

colin2328 · 2025-12-09T19:02:00Z

python/monarch/_src/job/skypilot.py

+sudo apt-get update
+
+# Install system dependencies
+sudo apt-get install -y \


skypilot requires new system level dependencies?

hey @colin2328, these are dependencies for monarch. SkyPilot's dependencies are handled by the SkyPilot runtime during setup.

They're needed because this PR currently builds Monarch from source on the workers to keep client/worker versions in sync during development. Once this is merged and a Monarch wheel is released to PyPI, the setup simplifies to just pip install torchmonarch on workers.

The current approach is a temporary workaround for the chicken-and-egg problem: SkyPilotJob lives in Monarch source, but there's no released wheel with it yet.

Question on architecture: How should new schedulers extend Monarch's JobTrait?

Should they:

Live in Monarch source (python/monarch/_src/job/)

Pro: Clean import path, UX for users (from monarch.job import SkyPilotJob)

Con: Requires building from source during dev until wheel is released, may need version pinning after release?

Live in a separate package (user code)

Pro: Don't need to rebuild monarch

Con: Different import pattern (from <userpkg> import SkyPilotJob?)

Something else I'm missing.

Currently I have the SkyPilot implementation in python/monarch/_src/job/skypilot.py, but I can refactor it. Any guidance here would be appreciated 🙏

cc: @zdevito as this is an interesting question.

My intuition is that they should live in python/monarch/_src/job even though writing a job or changing the API will requires building from source for testing purposes. The testing cost is only paid by the developer, not the end user.

IIUC, keeping it in a separate package will still require installation on the workers while the code is in development. However it will not require building rust libraries which are probably the main source of the build latency.

It would be nice for JobTrait implementers to not require to build rust binaries while testing their code. I wonder if simply a cp of the new files into site-packages dir on the workers would work, though it seems like a hack.

Job implementations should be buildable on top of Monarch's public APIs. So, in general, I would favor the latter pattern, which also has good DevEx. I don't think we should accumulate many job implementations in core Monarch.

Note that this is distinct from the question of whether these are part of the Monarch "ecosystem" packages. In other words, we might maintain them in Monarch itself, but build and distribute them as separate packages.

We have previously talked about using pip flavors to bundle such dependencies for this purpose, e.g.,

$ pip install torchmonarch[skypilot]

etc.

romilbhardwaj · 2025-12-10T21:57:36Z

Thanks @colin2328 @johnwhumphreys - I'll polish the PR and get it ready for review. Have a question about the architecture here, happy to do whatever aligns with Monarch's recommended design.

shayne-fletcher · 2025-12-11T04:03:13Z

examples/skypilot/skypilot_ddp.py

+import os
+import sys
+
+# Set timeouts before importing monarch


am working on enabling

from monarch.config import configure configure( host_spawn_ready_timeout="300s", message_delivery_timeout="300s", mesh_proc_spawn_max_idle="300s" )

to replace the use of env-vars here.

Thanks @shayne-fletcher! That'll be super helpful for backends where cold starts can be > 30s.

romilbhardwaj · 2025-12-11T07:57:24Z

Monarch build was slowing down worker launches, so I've refactored along the lines of option 2 in 20d36e8. Let me know if this is not the right direction :)

Tested on Coreweave k8s cluster with H200s. PR is still a draft, will finish up the examples (torchtitan and ddp) + readme and mark ready for review soon!

colin2328 · 2025-12-11T18:00:46Z

@romilbhardwaj "the author does not have a valid CLA on file." can you acept the CLA? and publish?

meta-cla · 2025-12-11T19:24:28Z

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

ahmadsharif1 · 2025-12-11T22:06:53Z

examples/skypilot/skypilot_getting_started.py

+
+Usage:
+    # Run on Kubernetes:
+    python getting_started.py --cloud kubernetes --num-hosts 2


s/getting_started/skypilot_getting_started?

examples/skypilot/__init__.py

python/monarch/job/__init__.py

meta-codesync · 2025-12-11T22:42:12Z

@colin2328 has imported this pull request. If you are a Meta employee, you can view this in D88994552.

ahmadsharif1 · 2025-12-11T23:17:53Z

examples/skypilot/skypilot_ddp.py

+try:
+    import sky
+except ImportError:
+    print("ERROR: SkyPilot is not installed. Run: pip install skypilot[kubernetes]")


Maybe we should add this as an optional dependency in pyproject.toml if we move skypilot_job.py from examples to the job dir

romilbhardwaj · 2025-12-13T01:24:50Z

Cleaned up the readme, added an architecture diagram and included a DDP notebook inspired by the slurm example.

Tested on coreweave k8s with H200s.

This PR is ready for review now! cc @ahmadsharif1 @colin2328 @johnwhumphreys

romilbhardwaj · 2025-12-13T02:21:37Z

@romilbhardwaj How does skypilot do end to end testing in its CI? Maybe we should have a minikube/kind test setup in our CI to test this if we expose this to users. We don't want silent breakages if users are going to use this.

Great point. For our release testing, our CI spins up k8s clusters on EKS, GKE, CW and Nebius and runs our smoke test suite which includes model training examples from pytorch, ray, nemo and other popular frameworks.

For lightweight CPU-only tests, some parts of our CI uses kind (via sky local up) to create a local k8s cluster. If it is of interest, shouldn't be too hard to edit this example to run on CPUs only for the CI tests.

romilbhardwaj · 2025-12-13T02:26:21Z

Added some flags to allow the example to run on CPU-only k8s clusters

docs/source/examples/README.rst

ahmadsharif1

Looks good for the most part. You can wait for Colin to take a look as well.

I think he wants this to be part of the release, so you can address them and merge if you want

docs/source/index.md

examples/skypilot/__init__.py

examples/skypilot/monarch_quickstart.sky.yaml

examples/skypilot/skypilot_quickstart_driver.py

examples/skypilot/monarch_skypilot/skypilot_job.py

examples/skypilot/skypilot_job.py

examples/skypilot/monarch_skypilot/skypilot_job.py

romilbhardwaj · 2025-12-19T20:14:07Z

Thanks for the reviews @ahmadsharif1! I'll wait for a bit for @colin2328's feedback before working through the comments and doing a round of final testing.

Also I noticed the linter was failing - do I need to take any action? Couldn't find instructions to run the linter myself.

docs/source/index.md

colin2328 · 2025-12-19T20:42:28Z

@romilbhardwaj , thanks. I'll fix the lints and push for you. please address the comments first, rest is fine to me.

romilbhardwaj · 2025-12-22T19:39:54Z

Thanks @colin2328 and @ahmadsharif1 - I think I've addressed all comments. Please lmk if I missed something. This should be ready to go now.

Tested on 2x H200:8 on a Coreweave k8s cluster.

meta-codesync · 2025-12-22T21:34:06Z

@colin2328 merged this pull request in e409178.

Romil Bhardwaj and others added 10 commits December 5, 2025 06:49

Add SkyPilot integration for job launching

6ffd98c

fixes

b2344bf

Working example

a740ae1

fix

efa313f

updates

35b6e0e

cleanup

6a75678

cleanup

fd310d3

updates

8511e5a

romilbhardwaj mentioned this pull request Dec 8, 2025

TorchX Kubernetes Support #2039

Closed

romilbhardwaj marked this pull request as draft December 8, 2025 06:16

colin2328 reviewed Dec 9, 2025

View reviewed changes

johnwhumphreys added the kubernetes label Dec 9, 2025

romilbhardwaj added 3 commits December 11, 2025 01:31

Extract SkyPilotJob from monarch src

20d36e8

remove stale changes

e23bd3f

Add DDP and titan examples

40f3a6a

shayne-fletcher reviewed Dec 11, 2025

View reviewed changes

Update README.md

2132e3c

meta-cla bot added the CLA Signed This label is managed by the Meta Open Source bot. label Dec 11, 2025

ahmadsharif1 reviewed Dec 11, 2025

View reviewed changes

romilbhardwaj added 6 commits December 13, 2025 00:38

Docs updates

2d83527

updates

4d6ed27

updates

3f3e890

Add notes on how to set resources and num nodes

be7818e

fix ssh command

0a443b3

Update jupyter commands

5fcf775

romilbhardwaj marked this pull request as ready for review December 13, 2025 01:24

Add CPU-only support

56e7c8c

romilbhardwaj requested review from ahmadsharif1, colin2328 and shayne-fletcher December 16, 2025 16:35

ahmadsharif1 reviewed Dec 17, 2025

View reviewed changes

docs/source/examples/README.rst Outdated Show resolved Hide resolved

ahmadsharif1 approved these changes Dec 18, 2025

View reviewed changes

romilbhardwaj added 2 commits December 20, 2025 01:08

update docs

236a01a

review comments

8207b92

colin2328 reviewed Dec 19, 2025

View reviewed changes

docs/source/index.md Outdated Show resolved Hide resolved

romilbhardwaj added 4 commits December 22, 2025 23:27

Undo index.md changes

8edfa36

review comments, restructure the package

45a038d

renaming

845a38c

remove stale changes

bd140f4

meta-codesync bot closed this in e409178 Dec 22, 2025

facebook-github-bot added the Merged label Dec 22, 2025

Add Kubernetes support via SkyPilot #2083

Add Kubernetes support via SkyPilot #2083

Uh oh!

Conversation

romilbhardwaj commented Dec 8, 2025

Uh oh!

meta-cla bot commented Dec 8, 2025

Action Required

Process

Uh oh!

johnwhumphreys commented Dec 9, 2025

Uh oh!

colin2328 commented Dec 9, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj commented Dec 10, 2025

Uh oh!

shayne-fletcher Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj commented Dec 11, 2025

Uh oh!

colin2328 commented Dec 11, 2025

Uh oh!

meta-cla bot commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

meta-codesync bot commented Dec 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

romilbhardwaj commented Dec 13, 2025

Uh oh!

romilbhardwaj commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romilbhardwaj commented Dec 13, 2025

Uh oh!

Uh oh!

ahmadsharif1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

romilbhardwaj commented Dec 19, 2025

Uh oh!

Uh oh!

colin2328 commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

romilbhardwaj commented Dec 22, 2025

Uh oh!

romilbhardwaj Dec 10, 2025 •

edited

Loading

shayne-fletcher Dec 11, 2025 •

edited

Loading

romilbhardwaj commented Dec 13, 2025 •

edited

Loading

colin2328 commented Dec 19, 2025 •

edited

Loading