docs(orchestrator): canonical cache overlay strategy and self-hosted lessons#567
Closed
frostebite wants to merge 2 commits into
Closed
docs(orchestrator): canonical cache overlay strategy and self-hosted lessons#567frostebite wants to merge 2 commits into
frostebite wants to merge 2 commits into
Conversation
…lessons
Adds two new sections to the orchestrator caching guide.
"Canonical Cache + Overlay (Advanced)" -- documents the new opt-in
`localCacheMode: canonical-overlay` value introduced in
game-ci/orchestrator. Covers:
* When to use it (multiple self-hosted runners on the same physical
host with multi-GB Library folders)
* Architecture (canonical store + per-runner hardlink overlay,
atomic-rename publish, SHA-versioned with `latest` pointer)
* Hardlink safety contract (write-temp-then-rename invariant; per-
subdirectory classifier table targeting Unity Library structure)
* Configuration example
* Sub-second hydration via `cacheMaterialize: prepared`
* Failure modes table (orphan staging cleanup, prepared-overlay
cancel-safety, missing-canonical recovery, sentinel mismatch
discard)
* Cross-platform behaviour table (NTFS/ReFS/ext4/xfs/btrfs/zfs/APFS/
Docker/cross-volume) with graceful fallback policy
* Six stated limitations (single-host topology, NTFS/ext4/xfs needed
for full benefit, junction safety on PowerShell, 1023-link NTFS
limit, modify-in-place propagation, multi-GB scale tested only
on NTFS)
* Performance characteristics table (eager 15-30 s, with junctions
2-8 s, prepared overlay < 1 s)
* Pointer to future strategies in the taxonomy (reflink-overlay,
bind-mount, nfs-passthrough, runner-affinity)
"Self-Hosted Operational Lessons" -- four lessons applicable to any
long-lived self-hosted runner that ephemeral runners never encounter:
* `PackageCache` subtree corruption from cross-runner restore --
detect-and-quarantine pattern for orphan-meta directories
* PowerShell forward-reference bug -- function calls before
definition silently fail in non-strict mode and downstream errors
surface as the failure
* GitHub Actions runner `_runner_file_commands` race -- concurrent
`core.setOutput` from fan-out steps races on the shared runner-temp
file; serialise output-emitting steps
* Path-filter on the workflow entrypoint -- `paths-ignore` for
`docs/**`, `*.md`, etc. to skip Unity CI on docs-only pushes
Inputs Reference table extended with the six new orchestrator-PR
inputs (`localCacheMode` enum extension, `canonicalCacheRoot`,
`canonicalCacheClassifier`, `canonicalCacheVersionRetention`,
`cacheMaterialize`, `cacheSentinelCanary`).
Pairs with `game-ci/orchestrator` PR (feat/canonical-cache-overlay
branch).
|
|
📝 WalkthroughWalkthroughThis documentation update introduces a new ChangesCanonical Cache + Overlay Feature Documentation
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Microsoft Presidio Analyzer (2.2.362)docs/03-github-orchestrator/07-advanced-topics/01-caching.mdxMicrosoft Presidio Analyzer failed to scan this file Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Visit the preview URL for this PR (updated for commit 8801768): https://game-ci-5559f--pr567-docs-canonical-cache-sgnlzgy7.web.app (expires Thu, 14 May 2026 16:05:12 GMT) 🔥 via Firebase Hosting GitHub Action 🌎 Sign: 1f0574f15f83e11bfc148eae8646486a6d0e078b |
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx (1)
530-538: ⚡ Quick winUse
**/*.mdand**/*.txtinpaths-ignoreto match markdown and text files across all directories.The patterns
'*.md'and'*.txt'match only files in the repository root; they do not match nested files likeblog/post.md. If the intent is to skip CI for docs-only commits across the entire repo, replace these with'**/*.md'and'**/*.txt'to catch all markdown and text files regardless of nesting depth.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx` around lines 530 - 538, The paths-ignore patterns only match root-level files; update the YAML in the on: push: block so that paths-ignore uses '**/*.md' and '**/*.txt' instead of '*.md' and '*.txt' to ensure markdown and text files in nested directories are ignored; modify the paths-ignore entry that currently lists 'docs/**', '*.md', '*.txt' to use '**/*.md' and '**/*.txt' alongside 'docs/**'.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx`:
- Line 557: The table entry for `canonicalCacheRoot` references `localCacheRoot`
which isn't defined here; add a corresponding `localCacheRoot` row (or a clear
inline link to its existing definition) so the fallback description is
self-contained—update the table to include a `localCacheRoot` row describing its
purpose and default path (e.g., `<localCacheRoot>/canonical` reference) and
ensure `canonicalCacheRoot` remains unchanged but now points to that
defined/linked `localCacheRoot`.
---
Nitpick comments:
In `@docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx`:
- Around line 530-538: The paths-ignore patterns only match root-level files;
update the YAML in the on: push: block so that paths-ignore uses '**/*.md' and
'**/*.txt' instead of '*.md' and '*.txt' to ensure markdown and text files in
nested directories are ignored; modify the paths-ignore entry that currently
lists 'docs/**', '*.md', '*.txt' to use '**/*.md' and '**/*.txt' alongside
'docs/**'.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 2d1d4998-eb45-4b9d-8ccb-25c2173c76ba
📒 Files selected for processing (1)
docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx
| | `skipCache` | Skip cache restore entirely | | ||
| | `useCompressionStrategy` | Use LZ4 compression for cache archives | | ||
| | `localCacheMode` | One of `tar`, `move-directory`, `copy-directory`, `canonical-overlay` | | ||
| | `canonicalCacheRoot` | Path for the canonical store (when `localCacheMode: canonical-overlay`); falls back to `<localCacheRoot>/canonical` | |
There was a problem hiding this comment.
canonicalCacheRoot description references localCacheRoot, but localCacheRoot is not defined in this table.
Please add a localCacheRoot row (or link to its definition) so the fallback path is self-contained and unambiguous here.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx` at line 557,
The table entry for `canonicalCacheRoot` references `localCacheRoot` which isn't
defined here; add a corresponding `localCacheRoot` row (or a clear inline link
to its existing definition) so the fallback description is self-contained—update
the table to include a `localCacheRoot` row describing its purpose and default
path (e.g., `<localCacheRoot>/canonical` reference) and ensure
`canonicalCacheRoot` remains unchanged but now points to that defined/linked
`localCacheRoot`.
4 tasks
Member
Author
|
Superseded by #569 — the combined branch merges this work in along with the other three docs branches from the May 2026 wave. Closing in favour of the combined PR. |
auto-merge was automatically disabled
May 7, 2026 16:37
Pull request was closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Documents the new opt-in
localCacheMode: 'canonical-overlay'value introduced in game-ci/orchestrator#22, plus four operational lessons that benefit any long-lived self-hosted runner.Adds two new sections to
docs/03-github-orchestrator/07-advanced-topics/01-caching.mdx:Canonical Cache + Overlay (Advanced)
latestpointercacheMaterialize: preparedreflink-overlay,bind-mount,nfs-passthrough,runner-affinitySelf-Hosted Operational Lessons
Four lessons that apply to any long-lived self-hosted runner. Symptoms ephemeral runners never see:
_runner_file_commandsrace -- concurrentcore.setOutputfrom fan-out steps races on the shared runner-temp file; serialise output-emitting stepspaths-ignorefordocs/**,*.md, etc. to skip Unity CI on docs-only pushesInputs Reference
Extended with the six new inputs from game-ci/orchestrator#22:
localCacheMode-- now acceptscanonical-overlayin addition to existing valuescanonicalCacheRoot-- path for the canonical storecanonicalCacheClassifier-- JSON describing per-subdirectory hardlink/junction/copy/skip strategycanonicalCacheVersionRetention-- how many SHA versions to keep per cache keycacheMaterialize--eager(live) orprepared(atomic-rename pre-built overlay)cacheSentinelCanary-- defense-in-depth corruption checkTest plan
action.ymlin feat(cache): canonical cache + hardlink overlay strategy (opt-in) orchestrator#22Pairs with
game-ci/orchestrator#22 -- the opt-in code that this documents. Both PRs are designed to land together.
Summary by CodeRabbit
Documentation
canonical-overlaycaching mode for multi-runner self-hosted environments, enabling efficient cache sharing through atomic content-addressed storage and per-runner overlay materialization