Add local engine profiling workflow#44
Open
tony wants to merge 5 commits into
Open
Conversation
why: Fast-path planning work needs timings that separate engine discovery, planning, collection, and rendering costs. A dormant, sanitized profiler lets benchmarks capture those boundaries without exposing prompt text, command argv, or local paths. what: - Add typed engine profiling primitives with context-local spans and redacted subprocess samples. - Add a standalone engine profiling script plus benchmark entries whose names and descriptions disclose their limits. - Add tests for profiling phase output, subprocess redaction, default-path import avoidance, and benchmark coverage.
why: streamline-02 is scoped to observability, so bottleneck work needs a stable, privacy-safe way to profile command-shaped engine paths before changing planner or executor behavior. what: - Add profile_engine component routing for prompt search, conversation search, grep-shaped prompt and conversation search, and prompt-source find runs. - Add explicit capped benchmark selectors for the profile-engine components. - Cover component expansion, query redaction, result caps, and prompt/conversation scope selection with typed pytest cases.
why: streamline-02 should leave a clear, repeatable observability workflow for agents and maintainers without adding CI artifact upload or performance changes to this branch. what: - Document profile_engine components and the required pre-commit gate in AGENTS.md and developer docs. - Add repo-local $profile guidance and extend $benchmark with component selectors. - Test that repo-local agent skills keep the component argument and core profiler components visible.
why: The profile_engine all component accepts content search terms for search-like runs, but find-prompts enumerates prompt sources rather than prompt contents. Passing those terms into find-prompts made batch profiles undercount source enumeration versus the standalone profiler benchmark. what: - Make ProfileRunSpec explicitly opt legacy find into using terms as a source metadata pattern. - Report effective term_count per component so batch find-prompts shows no find pattern was applied. - Add regression coverage for all tmux find-prompts parity and legacy find term filtering.
why: Engine profiles need both machine-readable child-run streams and a readable top-spans view so bottleneck evidence can be inspected without exposing local paths or query text. what: - Add json, ndjson, and rich renderers to scripts/profile_engine.py. - Keep the existing JSON document shape as the default while flattening batch runs for NDJSON. - Cover renderer behavior and entry-point parsing with profile-engine script tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/profile_engine.pyinto component-oriented runs for prompt search, conversation search, grep-shaped prompt and conversation search, and prompt-source find.$profile/$benchmarkguidance in AGENTS.md, repo agent skills, and developer benchmark docs.Refs #42. CI artifact upload remains separate in #43.
Test Plan
rm -rf docs/_build; uv run ruff check . --fix --show-fixes; uv run ruff format .; uv run ty check; uv run py.test --reruns 0 -vvv; just build-docs;