H-6664: Python monorepo infrastructure (uv workspace + turbo integration) — experimental spike#8994
Draft
claude[bot] wants to merge 2 commits into
Draft
H-6664: Python monorepo infrastructure (uv workspace + turbo integration) — experimental spike#8994claude[bot] wants to merge 2 commits into
claude[bot] wants to merge 2 commits into
Conversation
…o integration) - root pyproject.toml: uv workspace with shared ruff/basedpyright/pytest config and a single dev dependency group; committed root uv.lock - libs/@local/python-example: minimal example package (hatchling, typed, tested) whose package.json scripts wrap uv so turbo runs lint:ruff, lint:basedpyright and test:unit like any other workspace package - turbo.json: lint:ruff, lint:basedpyright and fix:ruff tasks - lint.yml: detect and run ruff/basedpyright per package, mirroring the existing ESLint/TSC steps; test.yml picks up test:unit automatically - prune-repository action: keep root pyproject.toml and uv.lock - check-license-in-workspaces: ignore .venv
github-actions
Bot
added
area/deps
![semgrep-code-hashintel[bot]](https://avatars.githubusercontent.com/u/22116112?s=60&v=4){kind=small}
Comment on lines
+16
to
+18
| [tool.uv] | ||
| package = false | ||
|
|
There was a problem hiding this comment.
This pyproject.toml configures uv but does not set a dependency cooldown. Newly published packages can be malicious or unstable. Add exclude-newer = "7 days" under [tool.uv] to wait 7 days before resolving newly published package versions. Added in: 0.9.17 Reference: https://docs.astral.sh/uv/concepts/resolution/#dependency-cooldowns
🍰 Fixed in commit d55400d 🍰
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8994 +/- ##
==========================================
- Coverage 59.68% 59.67% -0.01%
==========================================
Files 1371 1370 -1
Lines 134930 134915 -15
Branches 6067 6064 -3
==========================================
- Hits 80532 80514 -18
- Misses 53467 53471 +4
+ Partials 931 930 -1 Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Merging this PR will degrade performance by 15.38%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | bit_matrix/dense/iter_row[64] |
140.8 ns | 170 ns | -17.16% |
| ❌ | bit_matrix/dense/iter_row[200] |
185.8 ns | 215 ns | -13.57% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing claude/h-6664-python-monorepo-spike (d55400d) with main (6a22f5c)1
Footnotes
- requires-python >=3.14 across the workspace (matching internal-infra) and ruff target-version py314 - [tool.uv] exclude-newer = "7 days" per the uv-missing-dependency-cooldown Semgrep policy; re-locked (no resolved versions changed)
Contributor
Benchmark results
|
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2002 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 1002 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 3314 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 1527 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 2078 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 1033 | Flame Graph |
policy_resolution_medium
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 102 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 269 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 108 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 133 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 63 | Flame Graph |
policy_resolution_none
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 2 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 8 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 3 | Flame Graph |
policy_resolution_small
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| resolve_policies_for_actor | user: empty, selectivity: high, policies: 52 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: empty, selectivity: medium, policies: 26 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: high, policies: 94 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: seeded, selectivity: medium, policies: 27 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: high, policies: 66 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: low, policies: 1 | Flame Graph | |
| resolve_policies_for_actor | user: system, selectivity: medium, policies: 29 | Flame Graph |
read_scaling_complete
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id;one_depth | 1 entities | Flame Graph | |
| entity_by_id;one_depth | 10 entities | Flame Graph | |
| entity_by_id;one_depth | 25 entities | Flame Graph | |
| entity_by_id;one_depth | 5 entities | Flame Graph | |
| entity_by_id;one_depth | 50 entities | Flame Graph | |
| entity_by_id;two_depth | 1 entities | Flame Graph | |
| entity_by_id;two_depth | 10 entities | Flame Graph | |
| entity_by_id;two_depth | 25 entities | Flame Graph | |
| entity_by_id;two_depth | 5 entities | Flame Graph | |
| entity_by_id;two_depth | 50 entities | Flame Graph | |
| entity_by_id;zero_depth | 1 entities | Flame Graph | |
| entity_by_id;zero_depth | 10 entities | Flame Graph | |
| entity_by_id;zero_depth | 25 entities | Flame Graph | |
| entity_by_id;zero_depth | 5 entities | Flame Graph | |
| entity_by_id;zero_depth | 50 entities | Flame Graph |
read_scaling_linkless
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | 1 entities | Flame Graph | |
| entity_by_id | 10 entities | Flame Graph | |
| entity_by_id | 100 entities | Flame Graph | |
| entity_by_id | 1000 entities | Flame Graph | |
| entity_by_id | 10000 entities | Flame Graph |
representative_read_entity
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/block/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/book/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/building/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/organization/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/page/v/2
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/person/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/playlist/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/song/v/1
|
Flame Graph | |
| entity_by_id | entity type ID: https://blockprotocol.org/@alice/types/entity-type/uk-address/v/1
|
Flame Graph |
representative_read_entity_type
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| get_entity_type_by_id | Account ID: bf5a9ef5-dc3b-43cf-a291-6210c0321eba
|
Flame Graph |
representative_read_multiple_entities
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| entity_by_property | traversal_paths=0 | 0 | |
| entity_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| entity_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=0 | 0 | |
| link_by_source_by_property | traversal_paths=255 | 1,resolve_depths=inherit:1;values:255;properties:255;links:127;link_dests:126;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:0;link_dests:0;type:false | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:0;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:0;properties:2;links:1;link_dests:0;type:true | |
| link_by_source_by_property | traversal_paths=2 | 1,resolve_depths=inherit:0;values:2;properties:2;links:1;link_dests:0;type:true |
scenarios
| Function | Value | Mean | Flame graphs |
|---|---|---|---|
| full_test | query-limited | Flame Graph | |
| full_test | query-unlimited | Flame Graph | |
| linked_queries | query-limited | Flame Graph | |
| linked_queries | query-unlimited | Flame Graph |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Requested by Tim Diekmann · Slack thread
Warning
Experimental spike for H-6664 "Create Python monorepo infrastructure" (internal), opened so the approach can be picked apart before the real implementation. Totally fine to close this and start over.
🌟 What is the purpose of this PR?
Demonstrate proper Python monorepo infrastructure, borrowing internal-infra's uv-workspace pattern and wiring it into turborepo.
Before: Python packages are invisible to turbo and CI. There is no shared lint/type-check/test configuration and no lockfile policy — e.g. #8989 adds a uv package with no
package.json(so turbo never sees it), no CI, and a gitignoreduv.lock.After:
turbo run lint:ruff lint:basedpyright test:unit --filter '@local/python-example'runs ruff, basedpyright, and pytest for Python packages via uv — with shared config and one committeduv.lockat the repo root — and the existing Lint/Test CI pipelines pick Python packages up like any other workspace package.How it works
pyproject.tomlis a virtual workspace root ([tool.uv] package = false) listing Python packages as members, withrequires-python = ">=3.14". A singleuv.lockis committed at the repo root (do not gitignore it). Shared tooling lives in one rootdevdependency group:ruff,basedpyright,pytest,pytest-asyncio. Per org Semgrep policy (uv-missing-dependency-cooldown),[tool.uv] exclude-newer = "7 days"applies a dependency cooldown — releases younger than a week are ignored during resolution (recorded inuv.lockasexclude-newer-span = "P7D"; adding it changed no resolved versions).pyproject.toml:select = ["ALL"]with internal-infra's short, documented ignore list (COM812,D203,D213,FIX002,TD003) plus per-file ignores for tests;line-length = 100(matches the Rust width used here);target-version = "py314".typeCheckingMode = "all".--import-mode=importlib,asyncio_mode = "auto".package.jsonshim (the key part): each Python package carries apackage.jsonwhose scripts wrap uv, e.g."test:unit": "uv run --frozen --all-packages pytest". Turbo then treats the package like any other workspace member — task graph, caching,--filter, remote cache all work. Two subtleties:--all-packagesis required:uv runfrom a member directory only syncs that member's dependencies, so without it the rootdevgroup tools would be missing from the shared venv (verified with a fresh.venv).--project ../../..because pyright, unlike ruff and pytest, does not search parent directories for configuration.libs/@local/python-example: hatchling build backend, src layout,py.typed, fully typed module, passing pytest suite (incl. an async test to prove the pytest-asyncio wiring).CI (mostly the zero-new-workflow path)
test.yml: zero changes needed. It discovers packages viaturbo queryonTASK_NAME == "test:unit"; verified locally that@local/python-exampleshows up in exactly that query. Runners get uv 0.11.18 via mise (install-tools), anduv run --frozensyncs the venv on first use.lint.yml: small addition. It enumerates tools per package (lint:eslint,lint:tsc, clippy, …) rather than running a genericlinttask, so it genuinely can't pick Python up unaided. Addedlint:ruff/lint:basedpyrightdetection + run steps mirroring the existing ESLint/TSC pattern.prune-repositoryaction: now keeps the rootpyproject.toml+uv.lock(they aren't part ofturbo prune's output), souv run --frozenworks in the pruned per-package checkout.check-license-in-workspacesnow ignores.venv/(uv creates it at the repo root).turbo.jsonalready listedpyproject.tomland*.lockinglobalDependencies, so Python task caching invalidates correctly with no changes.🔗 Related links
libs/@hashintel/petrinaut-opt(unwired uv package, base branchcf-petrinaut-python)🔍 What does this change?
pyproject.toml(uv workspace + shared ruff/basedpyright/pytest config) and committed rootuv.locklibs/@local/python-example(pyproject.toml, package.json shim, LICENSE.md, typed module, tests)turbo.json: newlint:ruff,lint:basedpyright,fix:rufftaskspackage.json: matchinglint:ruff/lint:basedpyright/fix:ruffscripts, soyarn lintincludes Python.github/workflows/lint.yml: ruff/basedpyright detection + run steps.github/actions/prune-repository/action.yml: keeppyproject.toml+uv.locklibs/@local/repo-choreslicense check: ignore.venv/yarn.lock: registers the new workspace❓ Open questions for Bilal
Python version floor — resolved:
requires-python = ">=3.14"per Bilal, matching internal-infra (initially shipped as>=3.13). Note a workspace resolves on the intersection of all members, so FE-1162: Python optimisation script (Optuna) driving the Petrinaut CLI #8989's>=3.10package could only join by raising its own floor.Placement/naming:
libs/@local/python-examplewith dist namehash-python-example/ modulehash_python_example. Should Python packages live under a different convention (e.g. alongside@hashintel), and should the npm-scope name mirror the dist name?petrinaut-opt (FE-1162: Python optimisation script (Optuna) driving the Petrinaut CLI #8989): should it join the workspace (member entry + package.json shim + un-gitignore its lock into the root
uv.lock) whencf-petrinaut-pythonlands?CI pruning — reproduced and documented; fix deliberately not shipped. With a temporary second workspace member (
libs/@local/python-example-two, not committed) and the prune-repository action's steps run locally scoped to@local/python-example, the pruned tree lacks the sibling member and:uv sync --all-packages --lockedexits 1 —The lockfile at 'uv.lock' needs to be updated, but '--locked' was provided.(uv re-discovers the workspace frommembersand resolves 12 packages against a 13-package lock)uv run --frozen --all-packages pytestexits 2 —error: Failed to determine installation plan/Caused by: Distribution not found at: file:///…/out/libs/@local/python-example-twoi.e. the first additional Python package would break every pruned Python CI job. Per the Slack thread, the team is leaning toward removing
turbo prunefrom CI entirely rather than patchingprune.py, so no fix is included. Current prune usage, for that debate: theprune-repositoryaction (customprune.pywrappingturbo prune) is a CI-only checkout optimization —lint.yml(×1),test.yml(×2),bench.yml(×4),codspeed.yml(×1),deploy.yml(×1, sourcemaps job) — while the five docker image builds (hash-graph,hash-frontend,hash-api,hash-ai-worker-ts,hash-integration-worker) call turbo's nativeturbo prune --scope=… --dockerdirectly in their Dockerfiles and never useprune.py, so dropping the CI action would not leave the docker builds' prune path unexercised — they never shared code with it. Separately: no uv download cache in CI yet; worth adding?Line length: picked 100 (Rust width); repo conventions are split (JS 80, SQL 120, ruff default 88).
🚫 Blocked by
Nothing — but this is a spike; the real implementation is Bilal's roadmap item.
Pre-Merge Checklist 🚀
🚢 Has this modified a publishable library?
This PR:
📜 Does this require a change to the docs?
The changes in this PR:
🕸️ Does this require a change to the Turbo Graph?
The changes in this PR:
turbo.json's have been updated to reflect thisturbo prunefrom CI.uv run --frozenwithout--all-packagessilently falls back to aruff/pytestfound onPATHif the tool isn't in the venv — the scripts always pass--all-packagesto avoid this.🛡 What tests cover this?
libs/@local/python-example/tests/test_greeting.py(3 tests, incl. async) — run byturbo run test:unit.Verified locally (uv 0.11.18, turbo 2.6.3, yarn 4.16.0, CPython 3.14.6):
uv lock,uv sync --all-packages --locked,uv run --frozen --all-packages ruff check .,… ruff format --check .,… basedpyright --project ../../.. .(strict mode confirmed to really analyze files via an intentional error),… pytest(3 passed),turbo run lint:ruff lint:basedpyright test:unit --filter '@local/python-example'(3/3 successful, andFULL TURBOcache hits on re-run),yarn constraints,yarn lint:license-in-workspaces, taplo/oxfmt/markdownlint on the changed files.Not verified in this environment: the actual GitHub Actions runs (incl. mise-provisioned uv on runners) and the Rust
sort-package-jsoncheck (mise run lint:package-jsonneeds a cargo build) — watch the checks on this PR.❓ How to test this?
uv sync --all-packagesturbo run lint:ruff lint:basedpyright test:unit --filter '@local/python-example'