PRD: Closed-loop attack-path validation and security research cockpit

[justsml]

This was generated by AI during triage.

Problem Statement

Agent Security has the right foundation for a chat-powered bug hunter, exploit finder, and patching partner: project-scoped memory, staged Mastra agents, target authorization, approval gates, lab runtime support, Render JSON surfaces, evidence ingestion foundations, and a patch-loop core. The product still feels closer to an expert chat scaffold than a durable autonomous security validation system.

From the user's perspective, the missing product shape is a closed loop: inventory scoped targets, map attack paths, validate exploitability with proof, prioritize what matters, patch or remediate, retest fixes, and preserve the evidence trail. Current gaps show up in evals and roadmap notes: findings can be produced without saved evidence artifacts, target IDs are not mandatory across all records, stage agents are still mostly prompt-driven instead of scheduler-driven, generated evidence is not indexed everywhere, approvals are not fully enforced end-to-end, and the UI does not yet provide a durable research cockpit.

Competitive products such as Horizon3.ai NodeZero, Pentera, CrowdStrike Falcon Exposure Management, Palo Alto Cortex/Xpanse, Picus, Cymulate, XBOW, and ProjectDiscovery Neo package this loop clearly. They emphasize proven attack paths, exploitability proof, risk prioritization, remediation guidance, fix verification, continuous monitoring, and stakeholder-ready reporting. Agent Security should move in that direction while preserving its differentiator: a transparent, approval-gated, project-organized agentic research workspace that can work across repos, local labs, web apps, APIs, packages, and patch workflows.

Solution

Build a closed-loop attack-path validation and security research cockpit.

The solution should make Attack Path the core product object that connects targets, tasks, evidence, approvals, findings, validation, remediation, retesting, reports, learned negative results, and technical telemetry. The chat remains the orchestration surface, but the durable project cockpit becomes the operator's source of truth. Every meaningful action should produce queryable records, evidence artifacts, blocker states, audit context, cost/token/runtime metrics, and tool-call telemetry. Every validated finding should have a reproducible validation plan and a retestable remediation path.

The first release should focus on making the current foundations deep and end-to-end rather than adding more standalone tools. The highest-leverage work is: mandatory target linkage, evidence ingestion everywhere, a real scheduler for staged agents, attack-path records, project cockpit views, reproducible validation plans, fix verification, risk prioritization, and preserved technical telemetry for developers and power users.

User Stories

1. As a security researcher, I want each project to show its targets, scope, tasks, evidence, findings, approvals, workspace state, costs, token usage, runtime stats, tool calls, and patches in one cockpit, so that I can understand the whole engagement without reconstructing it from chat history.
2. As a security researcher, I want every task, finding, artifact, approval, RAG chunk, and patch to link to a target ID, so that multi-target research stays precise and auditable.
3. As a security researcher, I want to define related targets such as repos, packages, web apps, APIs, auth domains, local labs, and deployment surfaces, so that the agent can reason about realistic attack chains.
4. As a security researcher, I want to see target relationships in a system map, so that I can understand how a flaw in one surface might affect another.
5. As a security researcher, I want the product to model attack paths as first-class records, so that validated risk is represented as a chain of evidence rather than a flat vulnerability list.
6. As a security researcher, I want an attack path to show prerequisites, entry points, boundaries crossed, observed evidence, candidate findings, validation status, and remediation status, so that I can reason about exploitability quickly.
7. As a security researcher, I want attack paths to link to saved artifacts and citations, so that every claim can be traced back to evidence.
8. As a security researcher, I want attack paths to show blocked steps and missing approvals, so that I know exactly what decision or capability is needed next.
9. As a security researcher, I want the agent to create evidence artifacts automatically from HTTP probes, command transcripts, browser captures, system maps, finding drafts, terminal notes, patch diffs, and test output, so that useful evidence is not lost.
10. As a security researcher, I want generated evidence to be redacted before storage, indexing, display, and telemetry, so that secrets are not accidentally leaked.
11. As a security researcher, I want evidence to be indexed into RAG with target, task, finding, artifact, and source metadata, so that future agent runs can retrieve the right context.
12. As a security researcher, I want a timeline of actions, approvals, tool calls, artifacts, findings, token usage, costs, and runtime stats, so that I can audit how a conclusion was reached and debug agent behavior.
13. As a security researcher, I want the staged agents to run through a scheduler with durable task ownership, retries, cancellation, locks, and handoffs, so that longer research workflows can survive interruptions.
14. As a security researcher, I want the chat orchestrator to remain the control surface while workers update durable project records, so that I can guide the work conversationally without losing state.
15. As a security researcher, I want passive tasks to run before active validation tasks, so that the system preserves safe security research defaults.
16. As a security researcher, I want active probes, browser mutation, exploit checks, downloads, file writes, credential tests, and shell commands to require explicit approval, so that authorized scope is enforced.
17. As a security researcher, I want approvals to bind to target, task, action, command intent, network profile, cwd, path prefixes, max runtime, output size, and single-use behavior where relevant, so that a broad approval cannot be replayed unexpectedly.
18. As a security researcher, I want network profiles to be enforced below labels through a real network manager, proxy, firewall namespace, or runner service, so that egress policy is technically enforced.
19. As a security researcher, I want blocker cards to show missing target authorization, missing approval, wrong target, wrong command, wrong network profile, workspace lock, tool failure, test failure, or expired decision, so that I can resolve halted work quickly.
20. As a security researcher, I want reproducible validation plans for candidate findings, so that a finding can be confirmed or rejected consistently.
21. As a security researcher, I want validation plans to include preconditions, safe payloads, expected evidence, observed evidence, stop conditions, cleanup notes, and replay-safe artifacts, so that exploit validation is non-destructive and auditable.
22. As a security researcher, I want the validation agent to challenge reachability, exploitability, severity, assumptions, and reproduction steps, so that speculative findings do not become reports.
23. As a security researcher, I want rejected findings and failed probes to become negative-results memory, so that the agent does not repeat unproductive paths.
24. As a security researcher, I want coverage records for routes, files, endpoints, symbols, controls, and hypotheses, so that I can see what was actually examined.
25. As a security researcher, I want gapfill tasks to be generated from coverage records and negative results, so that follow-up work improves confidence instead of wandering.
26. As a security researcher, I want finding dedupe to group symptoms by root cause, affected targets, variants, and evidence, so that reports avoid inflated counts.
27. As a security researcher, I want reachability tracing across target relationships, so that library flaws, API bugs, auth issues, and deployment weaknesses are evaluated in realistic context.
28. As a security researcher, I want a risk prioritization score based on exploitability, reachability, asset criticality, adversary interest, exposed surface, validation confidence, and compensating controls, so that the highest-risk work rises above noisy severity lists.
29. As a security researcher, I want the cockpit to show why a risk is prioritized, so that I can trust and challenge the score.
30. As a security researcher, I want crown-jewel objectives such as admin access, tenant compromise, sensitive data access, CI token exposure, package publish rights, or production write paths, so that attack paths are measured against meaningful impact.
31. As a security researcher, I want reports to separate evidence from inference, so that reviewers can tell what was proven and what remains hypothetical.
32. As a security researcher, I want operator reports with reproduction steps, evidence, remediation, validation status, retest instructions, and relevant technical telemetry, so that engineers can fix and debug issues efficiently.
33. As a security leader, I want executive reports with attack paths, business impact, risk trend, and remediation progress, so that I can communicate posture without losing access to deeper technical drilldowns.
34. As a developer, I want validated findings or explicit user requests to initiate patch loops, so that patching starts from a clear mandate.
35. As a developer, I want the patch loop to create an isolated workspace branch, record planned tests, save diffs, run relevant verification, and preserve outputs as evidence, so that remediation is minimal and reviewable.
36. As a developer, I want fix verification to rerun the relevant validation plan after remediation, so that I know the attack path is actually closed.
37. As a developer, I want retest results to update the finding, attack path, evidence timeline, technical telemetry, and report, so that status is consistent across the product.
38. As a developer, I want toolchain adapters for common ecosystems, so that discovery, build, test, lint, audit, and patch verification commands are safe and consistent.
39. As a developer, I want adapters to be introduced only where there are real repeated toolchains, so that the codebase avoids hypothetical seams.
40. As a red team operator, I want safe browser evidence capture for screenshots, DOM snapshots, network traces, storage state, and request/response artifacts, so that web findings can be proven visually and technically.
41. As a red team operator, I want browser mutation to remain approval-gated, so that the product does not bypass safety controls.
42. As a red team operator, I want local-lab workflows to allow approved non-destructive validation with lower friction, so that eval and training loops are productive.
43. As a red team operator, I want continuous scheduled checks for passive recon, exposed asset diffs, dependency drift, retests, and regression hunts, so that projects do not depend on one-time scans.
44. As a red team operator, I want active scheduled work to queue for approval, so that continuous mode remains safe.
45. As a detection engineer, I want optional security-control validation hooks for SIEM, EDR, WAF, and log sources, so that validated attack behavior can be checked against detection and prevention controls.
46. As a detection engineer, I want control validation results mapped to tactics, techniques, and evidence, so that security control gaps become actionable work.
47. As a product operator, I want model and tool quality metrics from evals to influence routing, so that cheap models handle routine tasks and stronger models handle validation and reachability.
48. As a product operator, I want to compare model cost, tokens, runtime, tool calls, evidence count, vulnerability count, and rubric checks across scenarios, so that agent quality improves through measurement.
49. As a product operator, I want eval failures such as missing evidence discipline or missing system model to generate product backlog signals, so that observed agent weaknesses become roadmap work.
50. As a technical user, I want costs, token counts, runtime stats, tool-call counts, and model usage details to remain visible and exportable, so that I can tune models, debug regressions, and manage spend.
51. As a technical user, I want high-level views to summarize telemetry without deleting drilldown access, so that both executive and developer workflows are supported.
52. As an implementation agent, I want deep modules with small stable interfaces for attack paths, evidence ingestion, scheduling, validation, risk scoring, blockers, telemetry, and fix verification, so that I can implement slices safely and test behavior in isolation.

Implementation Decisions

• Build or deepen an AttackPath module. Its interface should let callers create and update attack paths, attach steps, link targets, link tasks, link evidence, record validation state, record blockers, attach remediation, attach technical telemetry, and mark retest outcomes. The implementation should hide graph storage, ordering, provenance, and status transitions.
• Build or deepen a ProjectCockpit query module. Its interface should return durable project state for targets, scope, task board, attack paths, findings, evidence, approvals, workspaces, costs, token usage, runtime stats, tool calls, patches, blockers, and recent activity. The UI should consume this module rather than assembling parallel client-only state.
• Add or deepen a TechnicalTelemetry module. Its interface should record and query model usage, token counts, estimated costs, runtime, tool-call counts, eval scores, scenario metadata, and trace identifiers. Higher-level reports may summarize these values, but technical users must retain drilldown access.
• Deepen the TargetInventory module. Target IDs should become mandatory at the external seam for tasks, approvals, artifacts, findings, RAG chunks, patch records, and attack-path records once target selection UI exists.
• Deepen the EvidenceIngestion module. Its interface should accept generated text or binary-derived text with source metadata and return redaction status, artifact linkage, chunk linkage, and indexing results. Callers should not need to know about chunking, vector storage, redaction internals, or delete filters.
• Add a ResearchScheduler module. Its interface should enqueue stage tasks, lease work, record ownership, handle cancellation, retry failed work, preserve stage handoff summaries, and update task/blocker state. The chat orchestrator should call the scheduler instead of relying only on prompt-driven delegation for long-running flows.
• Add a ValidationPlan module. Its interface should create validation plans for candidate findings, record preconditions, safe payloads, expected evidence, observed evidence, stop conditions, cleanup notes, replay-safe artifacts, and validation decisions.
• Add a RiskPrioritization module. Its interface should score and explain risk using exploitability, reachability, asset criticality, exposed surface, adversary interest, validation confidence, compensating controls, and business objective impact.
• Add a Blocker module. Its interface should normalize blockers from approval checks, authorization checks, network policy, workspace locks, tool errors, missing evidence, and test failures into consistent records and Render JSON surfaces.
• Deepen the ApprovalDecision module. The interface should enforce durable decisions across target authorization, approval gates, lab commands, HTTP probes, and browser tool selection. Decisions should support expiry, replay prevention, target binding, task binding, action binding, command intent binding, network profile binding, and output constraints.
• Add or deepen a NetworkPolicy module. It should translate network profiles into enforceable runtime behavior through a concrete adapter such as proxy, firewall namespace, network manager, or runner service. Docker labels alone should remain metadata, not enforcement.
• Add a NegativeResultsMemory module. Its interface should record covered surfaces, rejected hypotheses, failed probes, false-positive reasons, validation reversals, and missed relationships as queryable project knowledge.
• Deepen the PatchLoop module into a FixVerification workflow. Its interface should start from a validated finding or explicit request, create an isolated branch/workspace, run planned toolchain tests, save diffs and outputs as evidence, rerun the validation plan, and update finding/attack-path status.
• Add ecosystem ToolchainAdapter modules only for repeated real toolchains. The first adapters should likely cover JavaScript/TypeScript, Python, Go, containerized web apps, OpenAPI/API targets, and package ecosystems. Each adapter should expose discovery, build, test, lint, audit, and patch verification behavior with safety defaults.
• Extend Render JSON catalog surfaces for attack paths, project cockpit panels, blocker cards, evidence timeline, validation plan, risk explanation, technical telemetry, fix verification, coverage summary, and executive/operator report summaries.
• Preserve Mastra-first orchestration. The security research agent remains the conversational orchestrator, stage agents remain specialists, and durable modules hold the product state that chat and UI both read.
• Preserve technical observability. Do not remove or hide stats, costs, token counts, runtime, tool-call counts, model usage, eval scores, or trace identifiers from technical users. Compact views may default to summaries, but the data should stay accessible for developer work.
• Preserve safety defaults. Passive review happens first, active validation is approval-gated, destructive behavior is refused, and all generated evidence is redacted before persistence, display, prompt injection, RAG indexing, or telemetry.
• The roadmap should be organized around the closed loop: Inventory, Map, Validate, Prioritize, Patch, Retest, Report, Learn.

Testing Decisions

• Good tests should exercise module interfaces and externally visible behavior, not implementation details. A test should assert that a caller can create an attack path, ingest evidence, schedule work, enforce approval, validate a finding, score risk, record telemetry, or verify a patch through the public interface without knowing internal storage layout.
• Test the AttackPath module with scenarios for creating paths, linking targets/tasks/evidence, recording step state, adding blockers, validating impact, attaching telemetry, and closing paths after retest.
• Test the ProjectCockpit query module with mixed project state to ensure it returns consistent targets, tasks, attack paths, evidence, findings, approvals, blockers, costs, token usage, runtime stats, tool calls, workspace state, and patches.
• Test the TechnicalTelemetry module with model usage records, token counts, estimated costs, runtime, tool-call counts, eval scores, and trace identifiers. Include queries for project-level, thread-level, run-level, model-level, and scenario-level aggregation.
• Test the EvidenceIngestion module with generated evidence from HTTP probes, command transcripts, system maps, finding drafts, terminal notes, patch diffs, and test output. Include redaction tests for .env values, tokens, cookies, auth headers, and common cloud credentials.
• Test the ResearchScheduler module with enqueue, lease, retry, cancellation, lock contention, blocked state, stage handoff, and recovery after process restart.
• Test the ValidationPlan module with candidate finding confirmation, rejection, missing evidence, replay-safe artifact recording, stop conditions, and cleanup notes.
• Test the RiskPrioritization module with explainable scoring cases that separate raw severity from validated exploitable risk.
• Test the Blocker module with missing authorization, missing approval, wrong target, wrong command, wrong network profile, expired approval, replayed approval, workspace busy, workspace archived, tool failure, and test failure.
• Test the ApprovalDecision module through tool-facing behavior rather than internal matching details. Existing approval, command policy, HTTP probing, and chat-run tests are prior art.
• Test the NetworkPolicy module with profile-to-adapter behavior and deny-by-default cases. Use fake adapters for unit tests and a narrow integration scenario for local-lab egress.
• Test NegativeResultsMemory with coverage records, failed probes, rejected hypotheses, and later retrieval during planning/gapfill.
• Test FixVerification with validated finding to branch/test/diff/retest flow and explicit user request to branch/test/diff flow. Existing patch-loop and artifact tests are prior art.
• Extend Render JSON validation tests when adding catalog surfaces, following existing UI block validation prior art.
• Extend live evals so scenarios score evidence artifact creation, system map updates, attack path creation, validation plan quality, safety gating, risk explanation, technical telemetry, and fix verification behavior.
• Add end-to-end local lab scenarios for authorize target -> recon -> task creation -> approved probe -> evidence/RAG indexing -> attack path -> candidate finding -> validation -> report.
• Add patch-loop scenarios for validated finding -> isolated branch/workspace -> patch -> tests -> diff artifact -> validation replay -> remediation summary.

Out of Scope

• Fully autonomous exploitation against real third-party targets without explicit authorization.
• Destructive testing, persistence, stealth, credential theft, exfiltration, denial-of-service, or bypassing approval gates.
• Replacing human expert review for final vulnerability disclosure or legal authorization decisions.
• Building every possible ecosystem adapter in the first release.
• Building a full SIEM, EDR, WAF, or SOAR product. Security-control validation should integrate with existing tools where available.
• Creating a vulnerability scanner clone that emits large CVE lists without exploitability validation.
• Replacing the existing Mastra-first architecture with a custom agent framework.
• Implementing arbitrary cloud remediation automation without scoped approvals and provider-specific safety review.
• Rewriting the whole UI. The first release should add durable cockpit views and Render JSON surfaces that reuse the existing design system.
• Removing or hiding stats, costs, token counts, runtime metrics, tool-call counts, model usage, eval scores, or trace identifiers from technical users.

Further Notes

• The strongest sequencing is: target linkage everywhere, evidence ingestion everywhere, preserved technical telemetry, real scheduler, attack-path records, project cockpit, validation plans, risk prioritization, fix verification.
• Recent attack-vector eval results show that the product can identify vulnerabilities but may fail evidence discipline and artifact creation. That should be treated as a core product gap, not just a prompt issue.
• Cost, token, runtime, tool-call, and eval telemetry are critical for technical users and developer work. They should be retained as first-class product data even when non-technical reports present a simpler summary.
• The product should avoid shallow pass-through modules. The deep modules listed above should concentrate behavior behind small interfaces so future agents and tests have stable seams.
• The competitive north star is not “an agent that chats about security.” It is “an evidence-backed, approval-gated, continuously improving security research cockpit that proves exploitability and helps close the loop while remaining inspectable by technical users.”