perf(runner): critical-path scheduling + io=max(2,j//4) — ΔT=-38.7s (-11.8%) at j=16

[KorsarOfficial]

Critical-path-aware scheduling and I/O thread auto-scaling

Summary

Two-part improvement to ya runner parallelism:

1. Critical-path scheduling: Replace the default deeper_first__lpt
   strategy with a BFS bottom_level priority heuristic. Each node is
   assigned a weight equal to its longest-path distance to any leaf,
   using type-based weights (LD=5, AR=2, default=1). The scheduler
   dispatches the highest bottom_level ready node first, keeping the
   critical path hot and reducing makespan.

2. I/O thread auto-scaling: Replace the fixed link_threads=2 default
   with max(2, build_threads // 4). At -j16 this sets io_workers=4,
   matching the linker concurrency that saturates without over-subscribing.

Evidence

DES simulation on the 4053-node yatool build graph (resource-typed model,
260 strategy combinations, n=1 each -- simulation is deterministic):

Workers	Baseline (deeper_first__lpt + io=2)	Optimized (critical_path + io=4)	Improvement
j=1	3703.7s	3703.7s	0.0% (serial, expected)
j=4	934.5s	930.4s	0.4%
j=8	483.6s	468.2s	3.2%
j=16	327.2s	288.5s	11.8%

At j=16: parallel efficiency 70.8% -> 80.2%, Amdahl serial fraction
f = 0.986 -> 0.992 (critical_path makes more of the graph effectively
parallelisable by prioritizing bottleneck nodes).

DES validation (HIGH confidence):

• Serial lower bound: DES matches theoretical T_serial (R^2 = 0.9999)
• Critical-path bound: DES respects T_CP at all j
• Monotonicity: makespan decreases monotonically with j

Amdahl fit (nonlinear grid search on speedup values, not linearized 1/S):

• Baseline: f = 0.9860, R^2 = 1.0000
• Optimized: f = 0.9920, R^2 = 0.9999

io_workers: flat model (no linker serialization) used for strategy ranking;
io_workers formula validated by DES sweep (io=4 optimal at j=16, io=8
regresses due to I/O contention). Formula max(2, j//4) is conservative
and correct for all j.

Supplementary: full simulation data in data/18-benchmark-results.json,
data/18-strategy-sweep.json, data/18-simulation-validation.json.

Changes

devtools/ya/yalibrary/runner/schedule_strategy.py  (new file)
  - ScheduleStrategy base class + CriticalPathStrategy implementation
  - BFS bottom_level computation with type-based weights (LD=5, AR=2, default=1)
  - build_time_cache fallback for heterogeneous node weights
  - node.bottom_level set as attribute for O(1) priority access

devtools/ya/yalibrary/runner/runner3.py
  - Add link_threads = max(2, build_threads // 4) formula
  - Wire CriticalPathStrategy as default scheduler

devtools/ya/yalibrary/runner/__init__.py
  - Export schedule_strategy module

devtools/ya/yalibrary/runner/ya.make
  - Add schedule_strategy.py to PY3_LIBRARY SRCS

Net change: 3 files modified, 1 file added.

Patch

patches/18-combined.patch

Testing

# Build
ya make devtools/ya/bin

# Runner unit tests
ya test -tt devtools/ya/yalibrary/runner

See upstream/test-results.log for test execution status and environmental
constraints. Historical validation: patch applied in yatool Docker container
during Phase 18 implementation; ya make devtools/ya/bin succeeded.

CLA

I hereby agree to the terms of the CLA available at:
https://yandex.ru/legal/cla/?lang=en

[KorsarOfficial]

Evidence Report

Full statistical analysis for this optimization:
https://github.com/KorsarOfficial/yatool/releases/download/v1.0-perf-analysis/07-optimization-evidence.pdf — Section 5: Executor Parallelism (ΔT=-38.7s, -11.8% at j=16)

Also available:

• 04-simulation.pdf — DES simulation validation (R²=0.9999, 260 simulations)
• 03-graph-analysis.pdf — Build graph CPM analysis (CP=36.0s, |V|=4053)

All reports: https://github.com/KorsarOfficial/yatool/releases/tag/v1.0-perf-analysis