refactor(core): reduce layout/render hot-path allocations

[RtlZeroMemory]

Summary

This PR reduces allocation churn in the core layout/render hot paths without changing rendering behavior.

Changes in scope

• packages/core/src/layout/engine/layoutEngine.ts
  • Replaced string-based layout cache keys with nested numeric maps for cache lookup/store.
• packages/core/src/layout/engine/pool.ts
  • Increased layout scratch pool cap from 8 to 32.
  • Switched pool removal from splice to O(1) swap-pop.
• packages/core/src/layout/kinds/stack.ts
  • Pooled additional numeric scratch arrays in stack layout paths.
  • Added try/finally releases to avoid pool leaks.
• packages/core/src/renderer/renderToDrawlist/renderPackets.ts
  • RenderPacketRecorder.buildPacket() now transfers ownership of ops/resources instead of copying with .slice().
  • Removed conditional spread allocations in op construction.
  • Skip blobByResourceId allocation when packet has no resources.
• packages/core/src/renderer/renderToDrawlist/renderTree.ts
  • Guarded mergeThemeOverride calls when no theme override is provided.

Validation

• npx tsc -b packages/core packages/node packages/create-rezi
• node scripts/run-tests.mjs
• npx biome check packages/core/src/layout/engine/layoutEngine.ts packages/core/src/layout/engine/pool.ts packages/core/src/layout/kinds/stack.ts packages/core/src/renderer/renderToDrawlist/renderPackets.ts packages/core/src/renderer/renderToDrawlist/renderTree.ts

Manual PTY verification (starship)

• Ran starship in worker mode with deterministic viewport (300x68) and frame audit enabled.
• Exercised interactive keys: 1 2 3 4 5 6 t q.
• Audit check (node scripts/frame-audit-report.mjs ... --latest-pid) shows:
  • hash_mismatch_backend_vs_worker=0
  • route submissions/completions for bridge, engineering, crew, comms, cargo, settings.

Summary by CodeRabbit

• Performance Improvements

  • Redesigned layout caching for faster reuse and fewer recalculations.
  • Enlarged and improved pooling to reduce allocations during layout.
  • Reduced render packet payloads and made emission more efficient.

• Stability & Layout

  • Reworked stack layout into a two-pass flow for more accurate sizing and spacing.

• Bug Fixes

  • Fixed theme override handling to avoid unintended merges in themed containers.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 6e21c72 and e3f5d67.

📒 Files selected for processing (1)

• packages/core/src/layout/engine/layoutEngine.ts

---

📝 Walkthrough

Walkthrough

Replaces string-based layout cache with a nested map hierarchy, enlarges the array pool, rewrites stack layout into a two-phase sizing + cross-axis feedback flow with pooling, makes render-packet recorder internals mutable for reuse and optional-field emission tighter, and guards theme-override merging in render trees.

Changes

Cohort / File(s)	Summary
Layout Caching Infrastructure packages/core/src/layout/engine/layoutEngine.ts	Replaces string-composed cache keys with a nested multi-level Map structure (new internal cache types, NULL_FORCED_DIMENSION helper). Adds getLayoutCacheHit and setLayoutCacheValue to traverse/populate the hierarchical cache; replaces prior cacheKey reads/writes.
Pool Management packages/core/src/layout/engine/pool.ts	Increases MAX_POOL_SIZE from 8 to 32. acquireArray now removes reused entries by swapping/truncating instead of splice, and zeroes returned slices; release behavior unchanged.
Stack Layout Engine packages/core/src/layout/kinds/stack.ts	Introduces array pooling across measurement/layout (try/finally release), two-phase sizing (initial main pass + flex distribution), cross-axis feedback and rebalance, advanced flex handling (flexShrink/flexBasis/spacers), and separate constraint-pass vs non-constraint wrap planning.
Render Packet Emission packages/core/src/renderer/renderToDrawlist/renderPackets.ts	Makes RenderPacketRecorder.ops and .resources public/mutable for in-place reuse. buildPacket captures current state, clears internals, and returns frozen output. Emits ops with optional fields omitted when undefined and lazily accesses blobByResourceId, skipping ops when blobs are missing.
Theme Override Guard packages/core/src/renderer/renderToDrawlist/renderTree.ts	Adds guard to only call mergeThemeOverride when a theme override is defined, preserving currentTheme otherwise.

Sequence Diagram(s)

mermaid
sequenceDiagram
participant Client
participant LayoutEngine
participant LayoutCache
participant StackLayout
participant Pool
Client->>LayoutEngine: request layout(node, constraints)
LayoutEngine->>LayoutCache: getLayoutCacheHit(constraints, node coords)
alt cache hit
LayoutCache-->>LayoutEngine: cached LayoutResult
LayoutEngine-->>Client: return cached result
else cache miss
LayoutEngine->>StackLayout: measure/layout with constraints
StackLayout->>Pool: acquireArray(...) for temp arrays
Pool-->>StackLayout: array
StackLayout-->>Pool: releaseArray(...) (finally)
StackLayout-->>LayoutEngine: LayoutResult
LayoutEngine->>LayoutCache: setLayoutCacheValue(..., LayoutResult)
LayoutEngine-->>Client: return result
end

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• refactor(renderer): replace WeakMap theme propagation with stack #211: Overlaps modification area in renderTree.ts (theme propagation/override logic).
• fix(starship): restore overlay input routing and themed shell body #194: Overlaps layout/kinds/stack.ts changes around wrap/rebalance and flex/percent sizing.
• fix(core): stabilize core refactor and break internal cycles #182: Broadly related refactor touching layout engine, pooling, and stack layout flows.

Poem

🐇 Whiskers twitch for nested maps and queues,

Pools that grow and splice no more—hooray!
Two-pass hops through stretch and squeeze,
Packets lean and themes kept at bay.
I nibble bugs, then dance away. 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.65% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title 'refactor(core): reduce layout/render hot-path allocations' clearly and specifically summarizes the main objective: performance optimization through reduced allocation churn in core layout/render paths.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch perf/hotpath-cache-pooling-head

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

packages/core/src/layout/engine/layoutEngine.ts (1)

89-89: Add a fail-fast guard to prevent sentinel/key collisions.

If a negative forced dimension ever regresses in upstream callers, null and -1 currently collide in cache keying. A lightweight guard here prevents silent wrong-cache hits.

Suggested patch

 const NULL_FORCED_DIMENSION = -1;
 
 function forcedDimensionKey(value: number | null): number {
-  return value === null ? NULL_FORCED_DIMENSION : value;
+  if (value === null) return NULL_FORCED_DIMENSION;
+  if (value < 0) {
+    throw new Error("layout: forced dimensions must be >= 0");
+  }
+  return value;
 }

Also applies to: 113-115

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@packages/core/src/layout/engine/layoutEngine.ts` at line 89, Add a fail-fast
guard around any code paths that accept or key on a forcedDimension to prevent
negative values colliding with the sentinel NULL_FORCED_DIMENSION (constant
NULL_FORCED_DIMENSION = -1); specifically, at the start of the functions that
build cache keys or take a forcedDimension parameter, assert/throw if
forcedDimension < 0 and forcedDimension !== NULL_FORCED_DIMENSION (e.g., throw
new RangeError("invalid forcedDimension")); apply the same check in the other
places that use NULL_FORCED_DIMENSION (the other cache-keying/site where
forcedDimension is read—the region noted as also applying to 113-115) so any
regression to negative values fails fast instead of producing wrong cache hits.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/core/src/layout/engine/layoutEngine.ts`:
- Line 89: Add a fail-fast guard around any code paths that accept or key on a
forcedDimension to prevent negative values colliding with the sentinel
NULL_FORCED_DIMENSION (constant NULL_FORCED_DIMENSION = -1); specifically, at
the start of the functions that build cache keys or take a forcedDimension
parameter, assert/throw if forcedDimension < 0 and forcedDimension !==
NULL_FORCED_DIMENSION (e.g., throw new RangeError("invalid forcedDimension"));
apply the same check in the other places that use NULL_FORCED_DIMENSION (the
other cache-keying/site where forcedDimension is read—the region noted as also
applying to 113-115) so any regression to negative values fails fast instead of
producing wrong cache hits.

---

ℹ️ Review info

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 625c38c and 6e21c72.

📒 Files selected for processing (5)

• packages/core/src/layout/engine/layoutEngine.ts
• packages/core/src/layout/engine/pool.ts
• packages/core/src/layout/kinds/stack.ts
• packages/core/src/renderer/renderToDrawlist/renderPackets.ts
• packages/core/src/renderer/renderToDrawlist/renderTree.ts

[RtlZeroMemory]

Addressed the CodeRabbit nit: added a fail-fast guard in forcedDimensionKey to reject negative forced dimensions and prevent sentinel collisions (commit e3f5d67).