Update job runner spec with cross-account architecture design #4904
Conversation
How to use the Graphite Merge QueueAdd either label to this PR to merge it via the merge queue:
You must have a Graphite account in order to use the merge queue. Sign up using this link. An organization admin has enabled the Graphite Merge Queue in this repository. Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue. This stack of pull requests is managed by Graphite. Learn more about stacking. |
This comment has been minimized.
This comment has been minimized.
![graphite-app[bot]](https://avatars.githubusercontent.com/in/158384?s=60&v=4){kind=small}
nishu-builder
force-pushed
the
nishad/job-runner-spec-updates
branch
from
58c7118 to
8d52d84
Compare
This comment has been minimized.
This comment has been minimized.
rhysh
left a comment
rhysh
left a comment
There was a problem hiding this comment.
Some comments, non blocking
| from observatory. | ||
| - Run evaluation jobs in a dedicated AWS account separate from primary infrastructure | ||
| - Jobs don't submit their own results or pull inputs from Observatory | ||
| - Hot pool of pre-warmed nodes with per-job pod teardown (~5-10s startup target, one node per pod) |
There was a problem hiding this comment.
The hot pool is a strategy; the goals look like 1/ limit the impact of container escape by recycling nodes often, 2/ have fast startup
|
|
||
| **Dispatcher** (primary account, part of Observatory) | ||
|
|
||
| - Creates k8s jobs in eval cluster via cross-account kubeconfig |
There was a problem hiding this comment.
"... based on aws eks get-token"
|
|
||
| - Reads job spec from presigned GET URL (env var `JOB_SPEC_URI`) | ||
| - Downloads policies from presigned GET URLs | ||
| - Runs pure episode runner |
There was a problem hiding this comment.
"... in the same pod / container"
| - Downloads policies from presigned GET URLs | ||
| - Runs pure episode runner | ||
| - Writes results/replay to presigned PUT URLs | ||
| - No Observatory access, no AWS credentials, no network to primary account |
There was a problem hiding this comment.
Do we make any attempt to restrict the Internet access? (If not, then we'll also have to contend with "the policy is actually a DOS bot" type things.)
|
|
||
| 1. **Job Creation** (primary account) | ||
| - Observatory creates job row in Postgres (status=pending) | ||
| - Dispatcher generates presigned S3 URIs |
Merge activity
|
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
nishu-builder
force-pushed
the
nishad/job-runner-spec-updates
branch
from
652ed45 to
4d2744f
Compare
Co-Authored-By: Claude Opus 4.5 noreply@anthropic.com