feat: rate-limiting & graceful degradation when an API key is missing (#61)#64
Merged
NovaCode37 merged 2 commits intoJun 3, 2026
Conversation
…NovaCode37#61) Introduce a standard module result status enum (ok | skipped | rate_limited | error) so key-dependent modules degrade gracefully instead of failing hard. - Add modules/module_status.py with classify()/reason_for()/annotate() helpers - Shodan, VirusTotal, AbuseIPDB, Censys, Leak-Lookup/HIBP and Telegram now report 'skipped' when an API key is absent and 'rate_limited' on HTTP 429 - Scan engine propagates the status over WebSocket, persists it for the results view, and only caches successful (ok) results - Frontend renders a per-module status badge on progress chips and result cards, with the reason for skipped/rate-limited modules - Add tests for the status enum and per-module skip behaviour; update existing tests to the new (non-error) contract Closes NovaCode37#61
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Introduces a standard per-module status vocabulary (ok/skipped/rate_limited/error) across backend modules and propagates it through WebSocket events so the dashboard can render clear badges/notices for key-dependent modules.
Changes:
- Added
modules/module_status.pyhelpers (annotate,classify,reason_for, CLI notice printing) and updated multiple modules to use them for missing keys / 429s. - Updated scan runner to emit standardized
module_doneevents (with reason) and only cache successful (ok) results. - Updated frontend result cards and progress chips to display
skipped/rate_limitedstates with reasons; adjusted tests accordingly.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| web/app.py | Standardizes module_done payloads, persists derived status on results, and caches only ok outcomes |
| modules/module_status.py | New shared status enum + helpers for backward-compatible classification and annotation |
| modules/shodan_lookup.py | Uses annotate for missing key / rate-limit / errors; prints degraded notices in CLI |
| modules/threat_intel.py | Adds skipped/rate_limited handling and CLI degraded-notice printing for VT/AbuseIPDB |
| modules/censys_lookup.py | Returns annotated statuses for missing key / 401 / 429 / 404 outcomes |
| modules/leak_lookup.py | Adds status fields, rate-limit/skip annotations, aggregates sub-source statuses |
| modules/telegram_lookup.py | Adds statuses and CLI degraded-notice printing; skips when bot token absent |
| frontend/src/lib/types.ts | Introduces ModuleStatus types and attaches them to module result interfaces |
| frontend/src/components/views/ScanResults.tsx | Adds KeyModuleCard and status badges/notices for key-dependent modules |
| frontend/src/components/views/ScanProgress.tsx | Extends live progress chips to show skipped/rate-limited with icons/styles |
| frontend/src/components/App.tsx | Formats module progress log lines per status; stores richer live statuses |
| tests/test_module_status.py | New tests for status classification, annotation, and key-missing skip behavior |
| tests/test_v2_1_modules.py | Updates module credential-missing expectations to skipped |
| tests/test_modules_extended.py | Updates multiple key-missing scenarios to skipped + status reason assertions |
| CHANGELOG.md | Documents new status enum, graceful degradation, caching change, and UI badges |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
320
to
325
| cache_target = args[0] if args else None | ||
| if name in _CACHED_MODULES and cache_target: | ||
| cached = _get_cached(name, str(cache_target)) | ||
| if cached is not None: | ||
| await _push(scan_id, {"type": "module_done", "module": name, "status": "ok", "cached": True}) | ||
| await _push(scan_id, _done_message(name, cached, cached=True)) | ||
| return cached |
Comment on lines
+311
to
+314
| if reason and status != OK: | ||
| msg["reason"] = reason | ||
| if status == "error": | ||
| msg["error"] = reason |
Comment on lines
59
to
67
| elif response.status_code == 404: | ||
| result["breached"] = False | ||
| result["status"] = OK | ||
| elif response.status_code == 401: | ||
| result["error"] = "HIBP API key required for breach lookup" | ||
| annotate(result, SKIPPED, "HIBP API key required for breach lookup") | ||
| elif response.status_code == 429: | ||
| annotate(result, RATE_LIMITED, "HIBP API rate limit reached") | ||
| else: | ||
| result["error"] = f"HIBP returned status {response.status_code}" |
| for (const msg of newMsgs) { | ||
| if (msg.type === 'module_start') next[msg.module] = 'running'; | ||
| else if (msg.type === 'module_done') next[msg.module] = msg.status === 'error' ? 'error' : 'ok'; | ||
| else if (msg.type === 'module_done') next[msg.module] = (msg.status as LiveModuleStatus) || 'ok'; |
Comment on lines
+241
to
+244
| type ModStatus = 'ok' | 'skipped' | 'rate_limited' | 'error'; | ||
|
|
||
| /** Derive the standard status from a module result (honours explicit status). */ | ||
| function modStatus(m?: { status?: string; error?: string | null } | null): ModStatus { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduce a standard module result status enum (ok | skipped | rate_limited | error) so key-dependent modules degrade gracefully instead of failing hard.
Closes #61
Summary
Add a standard
ok | skipped | rate_limited | errorstatus enum for module results so that key-dependent modules (Shodan, VirusTotal, AbuseIPDB, Censys, Leak-Lookup, Telegram) degrade gracefully when an API key is absent or rate-limited, instead of failing with a hard error.Changes
modules/module_status.py: status constants +classify(),reason_for(),annotate()helpersskippedon missing key,rate_limitedon HTTP 429web/app.py: scan engine derives and propagates status over WebSocket, only cachesokresultstests/test_module_status.py; updated existing tests to new contractType of change
Testing
Screenshots
N/A