Skip to content

components: slim index + lazy-loaded bodies#939

Open
marcelveldt wants to merge 2 commits into
mainfrom
components-lazy-bodies
Open

components: slim index + lazy-loaded bodies#939
marcelveldt wants to merge 2 commits into
mainfrom
components-lazy-bodies

Conversation

@marcelveldt
Copy link
Copy Markdown
Contributor

What does this implement/fix?

PR 3 from #934 — split the component catalog so an idle dashboard stops carrying every component's config_entries tree in RAM. The slim index loads eagerly at startup; per-id bodies hydrate lazily on detail-view open through a bounded LRU.

Related issue:

Changes

  • definitions/components.index.json (slim, ~370KB) loads at startup; bodies live at definitions/components/<id>.json and hydrate via ComponentCatalog.get_body through a 64-entry LRU.
  • script/sync_components.py emits the split layout — bodies land in a sibling components.next/ dir first, then the whole dir + index swap into place so a Ctrl-C mid-write leaves the old catalog intact.
  • components/get_components (list) returns the slim ComponentCatalogIndexEntry shape; components/get_component (detail) keeps the full shape.
  • _FeaturedRecord stores underlying_id: str instead of the body; the body is fetched at materialise time. resolve_default_components / _apply_featured_presets thread the body through explicitly.
  • Scripts that need the full body (validate_definitions, check_catalog, sync_esphome_devices) join the index with the on-disk body files.

Types of changes

  • Bugfix (non-breaking change which fixes an issue) — bugfix
  • New feature (non-breaking change which adds functionality) — new-feature
  • Enhancement to an existing feature — enhancement
  • Breaking change (fix or feature that would cause existing functionality to not work as expected) — breaking-change
  • Refactor (no behaviour change) — refactor
  • Documentation only — docs
  • Maintenance / chore — maintenance
  • CI / workflow change — ci
  • Dependencies bump — dependencies

Frontend coordination

components/get_components list responses drop the config_entries field (the form renderer never used it on list views — it re-fetches via get_component when a card opens). Companion frontend PR needs to land in the same release window.

  • No frontend change needed
  • Companion frontend PR: esphome/device-builder-dashboard-frontend#<number — TBD>

Checklist

  • The code change is tested and works locally.
  • Pre-commit hooks pass (ruff, codespell, yaml/json/python checks).
  • Tests have been added or updated under tests/ where applicable.
  • components.index.json / definitions/components/*.json have not been hand-edited (regenerate via script/sync_components.py if a sync is needed).
  • Architecture-level changes are reflected in docs/ARCHITECTURE.md and/or docs/API.md.

Eager-loading the full ~22MB component catalog at startup carries
the per-field config_entries tree for every user session, even
though list / search / filter paths don't touch it. The slim index
loads at startup (~370KB); per-id body files hydrate on demand
through a bounded LRU when the user opens a detail view.

- definitions/components.index.json is the slim catalog; bodies
  live at definitions/components/<id>.json.
- ComponentCatalog.load() parses the index only.
  get_body(id) reads the body file through asyncio.to_thread
  and caches up to 64 entries in an LRU.
- components/get_components returns the slim
  ComponentCatalogIndexEntry shape; components/get_component
  keeps the full detail shape for the form renderer. Frontend
  coordination needed — companion PR drops list-view reads of
  config_entries.
- script/sync_components.py emits the split layout; build-time
  files swap atomically through a sibling components.next/ dir.
Copilot AI review requested due to automatic review settings May 22, 2026 07:52
@marcelveldt marcelveldt added the enhancement Improvement to an existing feature label May 22, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 22, 2026

Merging this PR will not alter performance

✅ 25 untouched benchmarks


Comparing components-lazy-bodies (5982704) with main (dd49cd3)

Open in CodSpeed

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 22, 2026

Codecov Report

❌ Patch coverage is 97.26027% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 99.23%. Comparing base (dd49cd3) to head (5982704).

Files with missing lines Patch % Lines
esphome_device_builder/controllers/components.py 96.77% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #939      +/-   ##
==========================================
- Coverage   99.24%   99.23%   -0.02%     
==========================================
  Files         191      191              
  Lines       14243    14296      +53     
==========================================
+ Hits        14136    14186      +50     
- Misses        107      110       +3     
Flag Coverage Δ
py3.12 99.18% <97.26%> (-0.02%) ⬇️
py3.14 99.23% <97.26%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...evice_builder/controllers/devices/add_component.py 100.00% <100.00%> (ø)
...e_device_builder/controllers/devices/controller.py 99.64% <100.00%> (ø)
...home_device_builder/controllers/devices/helpers.py 99.29% <100.00%> (ø)
...ce_builder/controllers/devices/mutations_create.py 95.38% <100.00%> (ø)
...vice_builder/controllers/devices/mutations_yaml.py 100.00% <100.00%> (ø)
esphome_device_builder/controllers/components.py 99.05% <96.77%> (-0.95%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@marcelveldt
Copy link
Copy Markdown
Contributor Author

codeql is flagging the _TIME_PERIOD_DEFAULT regex on script/sync_components.py:583 as a redos risk, but both alerts (#12, #13) already exist on main — they're being re-attributed to this PR because the 900+ added body files push the diff over the "code changes were too large" threshold codeql warns about in the summary. not introduced here. happy to look at that regex separately if it's worth tightening.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Improvement to an existing feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants