fix(api): paginate GET /task/{task_id}/result to prevent OOM on large scans#1697
Open
anshul23102 wants to merge 1 commit into
Open
fix(api): paginate GET /task/{task_id}/result to prevent OOM on large scans#1697anshul23102 wants to merge 1 commit into
anshul23102 wants to merge 1 commit into
Conversation
… scans
A wide-range scan (e.g. full-range nmap against a /16 subnet) can produce
tens of thousands of finding rows for a single task. GET /task/{task_id}/result
loaded every one of them into memory with no LIMIT before the response was
serialised, which can OOM-crash the backend process and freeze the frontend
rendering thousands of table rows at once.
Root cause: the query 'SELECT * FROM findings WHERE owner_id = ? AND
task_id = ? ORDER BY ...' had no bound, and severity_counts/finding_groups/
asset_summary were all computed by iterating the full in-memory list.
Fix:
- Add page/per_page query params (default 100, max 500 -- same convention
already used by GET /findings) and apply LIMIT/OFFSET to the findings
list actually returned to the client.
- Compute total_findings via a lightweight COUNT(*) query instead of
len(findings), so it reflects the whole scan even though only one page
of full finding objects is loaded.
- Aggregate views (severity_counts, finding_groups, asset_summary) must
reflect the entire scan, not just the current page, but loading every
row to get there would reintroduce the exact OOM this fixes. They're
computed from a bounded sample (capped at 5000 findings) for scans large
enough to need it, and from the full set for everything else -- so
aggregates stay accurate for the overwhelming majority of real scans
while memory use is capped for pathological ones.
- Cache key now includes page/per_page so different pages don't collide.
- Response includes total_findings, page, per_page, and has_more_findings
so the frontend can page through results instead of receiving a
silently truncated list.
Testing:
- testing/backend/integration/test_task_result_pagination.py: 7 new tests
covering default page size, page navigation, last-page has_more_findings,
severity_counts reflecting the whole scan regardless of page, the
per_page upper bound (422 above 500), small scans returning everything
on page 1, and the aggregation-sample cap on a 6000-finding scan.
- pytest testing/backend/unit -q -m "not benchmark" -- 2210 passed (6
pre-existing failures unrelated to this change, confirmed identical on
unmodified main -- sandbox/subprocess tests failing in this environment)
- pytest testing/backend/integration -q -m "not benchmark" -- 291 passed,
9 skipped
- ruff check backend testing/backend -- all checks passed
- scripts/check-artifacts.sh origin/main -- clean
Fixes utksh1#1621
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes #1621
A wide-range scan (e.g. full-range nmap against a /16 subnet) can produce tens of thousands of finding rows for a single task.
GET /task/{task_id}/resultloaded every one of them into memory with noLIMITbefore the response was serialised — this can OOM-crash the backend process and freeze the frontend rendering thousands of table rows at once.Root cause: the query
SELECT * FROM findings WHERE owner_id = ? AND task_id = ? ORDER BY ...had no bound, andseverity_counts/finding_groups/asset_summarywere all computed by iterating the full in-memory list.Approach
page/per_pagequery params (default 100, max 500 — matching the convention already used byGET /findings) and appliedLIMIT/OFFSETto the findings list actually returned to the client.total_findingsnow comes from a lightweightCOUNT(*)query instead oflen(findings), so it reflects the whole scan even though only one page of full finding objects is loaded.severity_counts,finding_groups,asset_summary) must reflect the entire scan, not just the current page — but loading every row to get there would reintroduce the exact OOM this fixes. They're computed from a bounded sample (capped at 5000 findings) only for scans large enough to need it, and from the full set otherwise, so aggregates stay exactly accurate for the overwhelming majority of real scans while memory use is capped for pathological ones.page/per_pageso different pages don't collide.total_findings,page,per_page, andhas_more_findingsso a future frontend change can page through results instead of receiving a silently truncated list.Related Issues
Closes #1621
Type of Change
How Has This Been Tested?
New test file
testing/backend/integration/test_task_result_pagination.py(7 tests):has_more_findingsseverity_countsreflects the whole scan regardless of which page is requestedper_pageupper bound rejects values above 500 with 422Full suite run locally:
Checklist
Additional Notes
I'm contributing this through GSSoC. While investigating this issue I found the same pattern (loading every finding row to build aggregate groups with no bound) in
GET /findingsandGET /finding-groups. I've flagged that separately rather than expanding this PR's scope, since this issue's reproduction steps specifically describe the per-task result view.