hotfix(orb): fix production outage in /v1/public/stats caused by anti-join perf#5263
Merged
Conversation
The anti-join added in the previous commit used a correlated NOT EXISTS subquery with LOWER(ae.target_key) = ... -- wrapping the indexed target_key column in a function defeats the index, forcing a full scan of audit_events (100K+ rows) for every orb_pr_outcomes row. This took down /v1/public/stats in production within minutes of deploying (D1 CPU time limit exceeded, 503s). Replaces it with a LEFT JOIN ... WHERE ae.id IS NULL, comparing target_key directly with no function wrapping so D1 can use the index. Verified directly against production: ~60ms / ~31K rows read, versus a timeout before. Both sides of the comparison come from the same GitHub API casing, so no LOWER() is needed here.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5263 +/- ##
=======================================
Coverage 94.36% 94.36%
=======================================
Files 474 474
Lines 40096 40096
Branches 14620 14620
=======================================
Hits 37836 37836
Misses 1585 1585
Partials 675 675
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
NOT EXISTSsubquery withLOWER(ae.target_key) = .... Wrapping the indexedtarget_keycolumn inLOWER()defeats the index, forcing a full scan ofaudit_events(100K+ rows) for everyorb_pr_outcomesrow — a ~600M-comparison query. This took/v1/public/statsdown within minutes of deploying: D1 returned{"errors":[{"code":7429,"message":"D1 DB exceeded its CPU time limit and was reset."}]}, and the endpoint has been serving 503 (public_stats_unavailable) since.LEFT JOIN ... WHERE ae.id IS NULL, comparingtarget_keydirectly with no function wrapping, so D1 can actually use the index for the join.repo#pr_numbercomparison come from the same GitHub API casing (orb_pr_outcomes.repository_full_nameandaudit_events.target_keyboth derive from GitHub's realfull_name), so droppingLOWER()doesn't reintroduce a case-sensitivity bug.Validation
npm run typechecktest/integration/orb-outcomes.test.ts,test/unit/public-stats.test.ts,test/integration/public-stats-route*.test.ts— all pass unchanged (same expected numbers, confirming the rewrite is logically equivalent, not just faster).Safety