PRD: Reference-backed passive recon and auth-surface mapping loop

[justsml]

Problem Statement

Agent Security already has the core Mastra-backed foundation for authorized security research: project-scoped memory, RAG/Graph RAG, scoped target authorization, approval-gated lab commands, HTTP probe artifacts, task tracking, system maps, stage agents, and shared evidence ingestion. The current gap is not another foundation. The gap is turning those primitives into a reliable first vertical slice where a researcher can ask for passive reconnaissance and auth-surface mapping, watch the agent collect and normalize evidence, and get durable artifacts, system-map updates, RAG-indexed context, blocker records, and suggested next steps without creating a parallel execution or document pipeline.

From the user's perspective, the agent should behave like a careful bug-hunting partner: it should use curated references, preserve target scope and authorization, prefer passive analysis first, save evidence, update the project map, stop cleanly when blocked, and ask for approval before active or mutating work.

Solution

Build a reference-backed passive recon and auth-surface mapping loop on top of the existing Mastra-first architecture.

The first shippable slice should let a user provide an authorized domain or web app target and ask for passive auth research. The system should create or update research tasks, run only approved/passive collection paths by default, normalize discovery outputs from uploads, terminal notes, lab command transcripts, HTTP probe artifacts, and reference documents, persist raw and summarized artifacts, index useful generated evidence through the shared evidence ingestion pipeline, update the living system map, detect first-class blockers such as WAF/captcha/rate-limit/auth-required/approval-required, and render the result through existing Render JSON surfaces.

This should extend the current foundation: use Postgres/pgvector as the natural RAG backend, keep Mastra memory project-scoped while preserving thread artifact metadata, deepen existing approval-gated tools, and keep browser/tool execution behind durable approvals. LanceDB or a second vector/document stack is out of scope unless there is a proven need.

User Stories

1. As a security researcher, I want to ask for passive auth-surface mapping for an authorized target, so that I can quickly understand login, session, account, token, and admin-related routes.
2. As a security researcher, I want the agent to use project-scoped memory and RAG before touching a target, so that previous evidence and context are not lost between chat threads.
3. As a security researcher, I want uploaded recon files, pasted terminal output, and generated tool artifacts to flow through the same evidence pipeline, so that I can retrieve them consistently later.
4. As a security researcher, I want raw command output to be saved separately from normalized summaries, so that I can audit the agent's conclusions.
5. As a security researcher, I want cleaned URL inventories from tools such as gau, waybackurls, katana, httpx, and curl outputs, so that noisy discovery data becomes usable.
6. As a security researcher, I want auth endpoint candidates grouped by purpose and confidence, so that I can choose the next test deliberately.
7. As a security researcher, I want session, token, password reset, OAuth, SSO, API key, and admin-route signals highlighted, so that auth-specific work starts from evidence instead of guesswork.
8. As a security researcher, I want the system map updated with hosts, routes, auth boundaries, trust boundaries, data objects, relationships, and open questions, so that the project model reflects what was learned.
9. As a security researcher, I want each normalized summary to link back to raw artifact ids, so that I can trace every claim.
10. As a security researcher, I want generated evidence indexed into RAG with rich metadata, so that later searches can filter by project, thread, target, task, finding, source, phase, tags, and artifact kind.
11. As a security researcher, I want project-level retrieval by default and thread-level filtering when needed, so that I can balance broad context with focused evidence review.
12. As a security researcher, I want the agent to distinguish passive references from active probes, so that it does not cross approval boundaries accidentally.
13. As a security researcher, I want browser automation to remain approval-gated, so that live target mutation and bot-detection risk are explicit.
14. As a security researcher, I want the agent to stop when it sees bot blocking, captcha, WAF denial, rate limiting, or auth-required signals, so that the research stays scoped and low-risk.
15. As a security researcher, I want blocker records rendered with clear next actions, so that I know whether to provide authorization, adjust target scope, switch network profiles, or continue manually.
16. As a security researcher, I want reference guidance to be curated with license/source metadata, so that the project can use external knowledge responsibly.
17. As a security researcher, I want reference documents and notes to be searchable alongside evidence, so that the agent can ground recommendations in known techniques and hardening guidance.
18. As a security researcher, I want reference metadata such as domain tags, tool tags, technique tags, phase, active/passive classification, and approval relevance, so that retrieval returns the right guidance for the current task.
19. As a security researcher, I want the agent to create a concise auth-surface summary artifact, so that I can hand it to another researcher or future task.
20. As a security researcher, I want suggested next steps separated into passive review, approval-gated probes, and patch/report work, so that I can make informed choices.
21. As a security researcher, I want every active or mutating step to be represented as an approval-gated action, so that the application preserves consent and auditability.
22. As a security researcher, I want the agent to update the task board after planning and after evidence collection, so that workflow state matches reality.
23. As a security researcher, I want no-finding and weak-signal results to be captured, so that future work avoids repeating dead ends.
24. As a security researcher, I want generated artifacts to be redacted before persistence, indexing, display, and telemetry, so that secrets do not leak into project memory.
25. As a security researcher, I want outputs to mention confidence and evidence quality, so that hypotheses are not mistaken for validated findings.
26. As a security researcher, I want local lab commands to use the existing policy-gated lab command tool, so that there is one audited shell execution path.
27. As a security researcher, I want HTTP checks to use the existing scoped HTTP probe tool when active checks are approved, so that requests are rate-limited, saved, and tied to scope.
28. As a security researcher, I want the UX to show artifacts, evidence, plan, approvals, blockers, and suggested actions through the shared Render JSON catalog, so that the UI remains consistent.
29. As a security researcher, I want the first slice to work without a full scheduler, so that value ships before durable multi-worker orchestration.
30. As a security researcher, I want the later scheduler to reuse the same task, artifact, approval, and blocker records, so that the vertical slice can grow into durable staged workflows.
31. As an implementation agent, I want a small set of deep modules with testable interfaces, so that the feature can be implemented without spreading parsing and policy logic across tools.
32. As an implementation agent, I want normalizers to accept stored artifacts and text inputs rather than invoking tools directly, so that they can be tested with fixtures.
33. As an implementation agent, I want reference ingestion to reuse evidence ingestion, so that the app does not grow a parallel document pipeline.
34. As an implementation agent, I want RAG metadata enrichment to be centralized, so that every generated evidence source indexes consistently.
35. As an implementation agent, I want blocker classification to be centralized, so that tools and UI surfaces report stopped work consistently.
36. As a project maintainer, I want this implementation to preserve native Mastra and AI SDK integrations, so that the codebase avoids handmade framework glue.
37. As a project maintainer, I want Postgres/pgvector to remain the default RAG store, so that operational complexity stays low.
38. As a project maintainer, I want target ids to continue propagating through tasks, approvals, artifacts, findings, and RAG metadata, so that research remains attributable to scoped assets.
39. As a project maintainer, I want durable approval checks to be reused rather than bypassed, so that safety behavior remains consistent.
40. As a project maintainer, I want narrow tests around external behavior, so that future agents can refactor internals without breaking the feature contract.

Implementation Decisions

• Build a reference library feature as curated reference ingestion, not a random PDF dump. References should have metadata for title, source URL, license, domain tags, tool tags, technique tags, security phase, active/passive classification, approval relevance, and target ecosystem. PDFs should only be stored locally when licensing allows; otherwise store metadata and links.
• Extend the shared evidence source vocabulary to model reference material explicitly, such as reference document, reference note, and reference index sources.
• Reuse the existing evidence ingestion pipeline for references and generated evidence. Do not create a second document ingestion or vector-store pipeline.
• Enrich evidence/RAG metadata beyond the current core fields. The metadata contract should support target id, task id, finding id, source, domain tags, tool tags, technique tags, phase, signal score, artifact kind, approval-required status, and other structured values needed for filtered retrieval.
• Preserve project-scoped Mastra memory. Artifacts remain associated with chat threads, and RAG chunks carry both project and thread metadata so retrieval can be broad or narrow.
• Add a discovery artifact normalizer as the main deep module. It should accept messy discovery text/artifacts and produce structured URLs, endpoints, auth candidates, versioned routes, session routes, token observations, response families, interesting patterns, blocker signals, raw artifact ids, and a summary artifact id.
• Keep the normalizer pure enough to unit test with fixture inputs. It should not own command execution, approvals, object storage, or UI rendering.
• Add an auth-surface summarizer that turns normalized discovery into a durable artifact with grouped auth routes, confidence, evidence references, unknowns, and recommended next steps.
• Route lab command transcripts, HTTP probe artifacts, system-map updates, finding drafts, terminal notes, and patch diffs through the shared evidence ingestion path where they are not already wired.
• Add normalized blocker records for bot-block-detected, captcha-detected, waf-denied, rate-limited, auth-required, approval-required, target-authorization-required, workspace-locked, network-profile-blocked, and tool-unavailable.
• When bot/WAF/captcha/rate-limit blocking is detected, stop or downgrade intensity, save evidence, update the blocker record, and ask the human what to do next.
• Use existing security-task-tracker records to represent the passive recon/auth mapping plan and its follow-up tasks.
• Use existing security-system-map records to represent discovered URLs, hosts, auth flows, data objects, trust boundaries, finding signals, and open questions.
• Use existing security-agent-lab-command for approved shell execution. Do not add a separate shell/MCP execution path.
• Use existing security-http-probe for scoped active HTTP checks after approval. Passive mapping should not silently escalate to probes.
• Keep browser providers environment/config driven and approval-gated. Persist screenshots and network summaries as artifacts when browser use is approved later.
• Render user-facing workflow state through the shared Render JSON catalog: plans, approvals, evidence, artifacts, blockers, suggested actions, and findings.
• Keep the first milestone focused on one vertical slice: authorized target to passive URL/auth mapping to artifact creation to RAG indexing to anomaly summary to suggested next steps.
• Build the scheduler after the vertical slice. The scheduler should later add persisted task queues, stage ownership, concurrency controls, locks, cancellation, retries, blocked states, worker summaries, and artifact links per task.
• Do not change model configuration patterns. Continue preserving model refs as llm:// connection strings and using llm-strings at adapter boundaries.
• Do not flip RAG/memory to thread-local. Thread filters are retrieval controls, not the memory ownership model.

Testing Decisions

• Tests should assert external behavior and durable contracts, not implementation details. Good tests should verify the normalized outputs, metadata shape, artifact links, blocker classification, approval boundaries, and RAG indexing calls that downstream modules depend on.
• Add evidence ingestion tests for the new reference evidence sources and richer metadata. Prior art exists in the shared evidence ingestion tests that assert supported sources, redaction, chunking, and indexer metadata.
• Add discovery artifact normalizer tests with representative outputs from URL discovery, crawler, HTTP probe, and curl-style inputs. These should verify cleaned URL lists, auth candidates, token/session observations, response families, blocker signals, and raw artifact references.
• Add auth-surface summary tests that verify grouped routes, confidence, evidence references, unknowns, and next-step categories.
• Add blocker classifier tests for bot blocking, captcha, WAF denial, rate limiting, auth-required, approval-required, network-profile-blocked, workspace-locked, and tool-unavailable signals.
• Add task/system-map integration tests that verify passive auth mapping creates or updates tasks and system-map nodes/edges with target ids and evidence links.
• Add chat turn or agent-tool tests proving generated command/probe/system-map artifacts are routed through shared evidence ingestion and become retrievable metadata, following existing chat run and RAG-facing test patterns.
• Add Render JSON catalog validation tests if new blocker or reference surfaces are introduced. Prior art exists in the UI block tests for catalog validation.
• Add redaction tests for generated artifacts and reference notes before persistence/indexing, following the existing redaction coverage for tokens, auth headers, and cloud credentials.
• Do not require broad end-to-end scheduler tests for this PRD. Scheduler coverage belongs to a later PRD or follow-up issue.

Out of Scope

• Building a second RAG backend or introducing LanceDB as a production dependency.
• Replacing the Mastra-backed chat path, Mastra agent registry resolution, or project-scoped memory model.
• Building a parallel shell, browser, MCP, or document ingestion system.
• Running active scans, exploit validation, browser mutation, downloads, writes, or credential tests without explicit approval.
• Implementing the full durable scheduler in the first vertical slice.
• Implementing the full patching loop, branch operations, toolchain adapters, or PR handoff.
• Storing licensed PDFs locally when redistribution or storage rights are unclear.
• Treating hypotheses or weak signals as validated findings.
• Making target ids mandatory everywhere in this PRD; this work should prepare for that direction and propagate target ids where available.

Further Notes

The implementation should update the living TODO shape around "Build reference-backed passive recon and auth-surface mapping loop" once work begins. The best first milestone is: add reference evidence support, define the reference metadata format and a small curated seed list, build the passive discovery artifact normalizer, wire generated lab/probe/system-map evidence into shared ingestion where missing, create the auth-surface summary artifact, update agent/tool prompts so plain-language auth research goals follow this workflow, and add focused tests around metadata, artifact persistence, blocker detection, and RAG indexing.

The architectural through-line is: Mastra-first, pgvector-backed, approval-gated, artifact-centered, and incremental.