Skip to content

docs: update incremental benchmarks (3.9.2) #901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
github-actions wants to merge 3 commits into
base: main
Choose a base branch
from
Open

docs: update incremental benchmarks (3.9.2) #901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
cc7f130
docs: update incremental benchmarks (3.9.2)
github-actions[bot] Apr 9, 2026
2f19349
Merge branch 'main' into benchmark/incremental-v3.9.2-20260409-054250
carlos-alm Apr 9, 2026
aa8b816
fix(test): add 3.9.2 native full-build to KNOWN_REGRESSIONS (#901)
carlos-alm Apr 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 63 additions & 11 deletions generated/benchmarks/INCREMENTAL-BENCHMARKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@ Import resolution: native batch vs JS fallback throughput.

| Version | Engine | Files | Full Build | No-op | 1-File | Resolve (native) | Resolve (JS) |
|---------|--------|------:|-----------:|------:|-------:|------------------:|-------------:|
| 3.9.2 | native | 667 | 9.4s ↑81% | 8ms ↓47% | 555ms ↓27% | 6ms ↓17% | 11ms ↓14% |
| 3.9.2 | wasm | 667 | 7.2s ↑4% | 17ms ↑21% | 598ms ~ | 6ms ↓17% | 11ms ↓14% |
Comment on lines +9 to +10
Copy link
Copy Markdown
Contributor

@greptile-apps greptile-apps bot Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Native engine slower than WASM — first time historically

The 3.9.2 native full build (9.4s) is now slower than WASM (7.2s), reversing every prior release (e.g., 3.9.1: native 5.2s vs WASM 6.9s). The +81% build time increase far outpaces the +17% file count growth. The phase breakdown in the 1-file profile shows the bottleneck is outside parsing — setupMs (37.8ms vs 1.8ms WASM), structureMs (127.9ms vs 34.6ms), cfgMs (53.6ms vs 0.3ms), and dataflowMs (31.3ms vs 0.5ms) are all dramatically worse in the native path. The code change driving this regression should be identified and addressed before 3.9.2 ships.

Copy link
Copy Markdown
Contributor

@carlos-alm carlos-alm Apr 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid concern. This PR is a docs-only recording of benchmark results — the regression itself requires changes to the native engine in crates/codegraph-core/, which is outside this PR's scope.

Created #903 to track the native engine full-build regression with the phase breakdown you identified (setup, structure, CFG, dataflow phases all significantly slower).

| 3.9.1 | native | 570 | 5.2s ↓20% | 15ms ↑150% | 757ms ↑44% | 7ms ↑37% | 12ms ↑13% |
| 3.9.1 | wasm | 570 | 6.9s ↑2% | 14ms ↑17% | 603ms ↑11% | 7ms ↑37% | 12ms ↑13% |
| 3.9.0 | native | 567 | 6.5s ~ | 6ms ↓25% | 527ms ↑1185% | 5ms ↓18% | 11ms ~ |
Expand Down Expand Up @@ -49,37 +51,87 @@ Import resolution: native batch vs JS fallback throughput.

### Latest results

**Version:** 3.9.1 | **Files:** 570 | **Date:** 2026-04-06
**Version:** 3.9.2 | **Files:** 667 | **Date:** 2026-04-09

#### Native (Rust)

| Metric | Value |
|--------|------:|
| Full build | 5.2s |
| No-op rebuild | 15ms |
| 1-file rebuild | 757ms |
| Full build | 9.4s |
| No-op rebuild | 8ms |
| 1-file rebuild | 555ms |

#### WASM

| Metric | Value |
|--------|------:|
| Full build | 6.9s |
| No-op rebuild | 14ms |
| 1-file rebuild | 603ms |
| Full build | 7.2s |
| No-op rebuild | 17ms |
| 1-file rebuild | 598ms |

#### Import Resolution

| Metric | Value |
|--------|------:|
| Import pairs | 951 |
| Native batch | 7ms |
| JS fallback | 12ms |
| Import pairs | 958 |
| Native batch | 6ms |
| JS fallback | 11ms |
| Per-import (native) | 0ms |
| Per-import (JS) | 0ms |
| Speedup ratio | 1.7x |
| Speedup ratio | 1.8x |

<!-- INCREMENTAL_BENCHMARK_DATA
[
{
"version": "3.9.2",
"date": "2026-04-09",
"files": 667,
"wasm": {
"fullBuildMs": 7216,
"noopRebuildMs": 17,
"oneFileRebuildMs": 598,
"oneFilePhases": {
"setupMs": 1.8,
"parseMs": 256.9,
"insertMs": 18.6,
"resolveMs": 1.8,
"edgesMs": 26.2,
"structureMs": 34.6,
"rolesMs": 80,
"astMs": 0.7,
"complexityMs": 0.5,
"cfgMs": 0.3,
"dataflowMs": 0.5,
"finalizeMs": 6.5
}
},
"native": {
"fullBuildMs": 9403,
"noopRebuildMs": 8,
"oneFileRebuildMs": 555,
"oneFilePhases": {
"setupMs": 37.8,
"parseMs": 50.8,
"insertMs": 25.5,
"resolveMs": 0.7,
"edgesMs": 28.6,
"structureMs": 127.9,
"rolesMs": 74,
"astMs": 25.6,
"complexityMs": 3,
"cfgMs": 53.6,
"dataflowMs": 31.3,
"finalizeMs": 0.7
}
},
"resolve": {
"imports": 958,
"nativeBatchMs": 5.8,
"jsFallbackMs": 10.5,
"perImportNativeMs": 0,
"perImportJsMs": 0
}
},
{
"version": "3.9.1",
"date": "2026-04-06",
Expand Down
4 changes: 4 additions & 0 deletions tests/benchmarks/regression-guard.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,17 @@ const SKIP_VERSIONS = new Set(['3.8.0']);
* - 3.9.1:1-file rebuild — continuation of the 3.9.0 regression; native
* incremental path still re-runs graph-wide phases on single-file rebuilds.
* Benchmark data shows 562 → 767ms (+36%). Same root cause as 3.9.0 entry.
*
* - 3.9.2:Full build — NativeDbProxy overhead causes native full build to
* regress from 5206ms to 9403ms (+81%). Fix tracked in PR #906.
*/
const KNOWN_REGRESSIONS = new Set([
'3.9.0:1-file rebuild',
'3.9.0:fnDeps depth 1',
'3.9.0:fnDeps depth 3',
'3.9.0:fnDeps depth 5',
'3.9.1:1-file rebuild',
'3.9.2:Full build',
]);

/**
Expand Down
Loading