Agent Harness — Docs-as-code / Assurance Consistency Automation

[FScholPer]

Goal

Define and implement an automated harness for maintaining ISO 26262 / ASPICE assurance arguments consistent with Sphinx-needs artifacts under continuous change.

Why

Safety and process assurance arguments reference development artifacts (requirements, guidelines, tests, code links) as evidence. Every change to those artifacts can refute safety goals or invalidate process compliance. Currently, consistency checks between safety arguments and development artifacts are executed manually or not at all, blocking CI/CD in safety-critical contexts.

This issue establishes the OSS foundation for automated consistency checking in the docs-as-code domain, with an evolving harness that improves over time based on structured traces of prior runs.

Domain Framing

• Fixed component: the Lane A gate contract (traceability_gate.py, traceability_coverage.py, evidence schema)
• Variable component: the harness context, retrieval, and check sequencing presented before and around agent execution
• Base model: any — the harness is model-agnostic; agent tooling choice is Lane B
• Evaluation unit: one change scenario — a specific RST or needs.json change with a known expected gate outcome

Agent Roles

Role	Responsibility
Specifier	Defines one task as a spec.md (input, fixed context, success criteria, expected gate verdict)
Executor	Applies the change given the harness context
Validator	Runs traceability_gate.py and all consistency checks; emits structured trace artifacts
Distiller	Outer loop script: extracts structured fields from raw outputs and writes per-task trace files

The Distiller role is always deterministic Python — never an LLM. The outer-loop script owns distillation so the proposer never reads raw output.

Repository Entry Strategy

• Keep the top-level agent instruction file short and navigational (map, not encyclopedia)
• Put domain detail in indexed files close to the code and artifacts they describe
• Make the public rule catalog and task corpus queryable by filename, rule ID, and task ID
• Prefer machine-enforced invariants over long prose when a rule matters repeatedly

Artifact Metamodel

The Sphinx-needs metamodel in docs-as-code covers these artifact types that consistency rules may reference:

• requirement (tool_req, std_req, process_requirement)
• guideline (gd_guidl, gd_req)
• test_link (via tests field)
• code_link (via implements field)
• evidence (V&V results referenced in solutions)

Consistency Rule Catalog

A machine-readable catalog of rules linking argument elements to artifact types and change scenarios. Initial rules to specify:

• CR-001: if a complies link target is renamed or removed, all linking elements are directly impacted
• CR-002: if a requirement type changes, all guidelines claiming compliance are indirectly impacted
• CR-003: if a test reference is broken, the linked requirement loses its test coverage evidence
• CR-004: if a std_req changes content, all gd_guidl elements that comply with it require re-review
• CR-005: if coverage drops below configured threshold, the gate verdict changes from pass to fail

The catalog is a public OSS artifact. Internal OEM rule extensions remain private.

Change Impact Classification

Each trace artifact records impact class per affected element:

• direct_recheck — element is directly referenced by the changed artifact; must be rechecked immediately
• indirect_propagation — element is reachable via the argument graph from a directly impacted element
• revision_required — element's supporting evidence is invalidated and cannot be mechanically rechecked

Reusable Checkable Building Blocks

Reusable argument-and-check fragments that can be instantiated for common patterns:

• Checkable Goal referencing Requirements: goal element linked to requirement with consistency rules for content change and status change
• Checkable Solution referencing V&V Results: solution linked to test output with rules for test result regression
• Checkable Requirements Breakdown: parent requirement with child requirement links and coverage threshold rules

Each block ships with a corresponding automated check that can run in Lane A CI.

Public Task Corpus

A set of non-confidential change scenarios for evaluating harness candidates:

• Sourced from historical CI gate failures in the docs-as-code repository
• Seeded now with executable metrics.json fixtures derived from existing traceability_gate.py tests
• Initial runnable scenarios cover threshold failure, broken testcase references, and need-type-scoped pass behavior
• These seed scenarios already exist to make the outer loop and candidate validation runnable before full docs-snapshot tasks are added
• Each scenario: RST or needs.json snapshot + expected gate verdict + expected impacted element list
• Search set: 30-50 scenarios; held-out test set: 10-15 cleanly separated
• No field defects or confidential product data (those stay in the OEM internal file)

Trace Store Schema

Per-run trace artifacts written by the Distiller (outer loop), readable by proposer via grep/cat:

runs/
  <iteration>/
    <candidate>/
      meta.json                  # hypothesis, expected outcome, what changed
      score.json                 # gate pass/fail, coverage delta, provenance
      traces/
        <task_id>/
          gate_output.json       # structured gate result (not raw stdout)
          impacted_elements.json # impacted need IDs, impact class, rule ID
          score.json             # expected verdict vs observed, provenance metadata
          agent_diff.patch       # what the agent changed
evolution_summary.jsonl          # one line per candidate, all iterations

Provenance fields (per task score.json):

• execution_timestamp: ISO 8601 timestamp
• python_version: Python interpreter version
• environment_hash: stable hash of execution environment
• gate_script_version: traceability gate version
• responsible_role: who is accountable for this outcome (default: pr_creator)
• escalation_role: who to escalate gate failures to (default: harness_maintainer)
• waiver_authority: who can approve waivers (default: release_approver)

The proposer must start from evolution_summary.jsonl, then inspect only the
relevant candidate and task traces. The trace store is designed for selective
navigation, not monolithic prompt packing.

Harness Interface

Every candidate harness must satisfy this interface (Python):

class AssuranceHarness:
    def get_context(self, task_spec: dict) -> str:
        """Return context to present to the agent before it acts."""
        ...
    def post_process(self, agent_output: str, task_spec: dict) -> dict:
        """Optional: transform or validate agent output before gate runs."""
        ...

Candidates are single Python files. The outer loop loads and evaluates them without modification.

Safety restrictions (mandatory for all candidates):

• File scope: read only files in task_spec["input_path"] or referenced by consistency_rules
• No network access: no HTTP, DNS, or external services
• No side effects in get_context(): must be read-only
• Deterministic: same inputs → same context
• Tool safety: stdlib and repo-local modules only, no eval()/exec()

Lightweight Validation Before Full Evaluation

Before any expensive benchmark run, each candidate must pass a cheap validation step:

• import candidate harness module
• instantiate the harness class
• call get_context() on one tiny task spec
• verify trace artifact filenames and JSON serialization shape
• expose a small run-history query surface so agents can inspect failed tasks and candidate deltas without replaying whole histories

That lightweight layer should stay simple and deterministic: one validation entrypoint for candidate loading and one query helper over runs/ for top candidates, failed tasks, and candidate diffs.

This catches malformed candidates before they consume benchmark time.

Frameworks Used

• Spec Kit (specify): bootstrap task specs in spec.md format per change scenario
• Meta-Harness pattern: outer loop reads trace filesystem, proposes next candidate (Lane B)
• Sphinx-needs + traceability_gate.py: Lane A verdict (already exists)

Lane A Checks (mandatory)

1. Run traceability_coverage.py --json-output to extract metrics
2. Run traceability_gate.py to produce pass/fail verdict
3. Validate evidence artifact schema
4. Write structured trace artifacts to runs/
5. Run lightweight candidate validation before the full task set

Done When

• Consistency rule catalog (at least CR-001 through CR-005) is defined and machine-readable
• At least 20 public task scenarios exist in the search set
• Outer loop runs end-to-end: harness → gate → distilled trace → evolution_summary.jsonl
• Seed corpus includes executable repo-native scenarios sourced from existing gate tests
• One baseline harness candidate is evaluated and its trace is grep-able
• Agents can inspect prior runs through a small index-first query surface
• Lane A checks run in CI without LLM dependency
• Top-level agent guidance stays short and delegates to indexed domain docs

Parent: #2850

[FScholPer]

Security Architecture (Phase 2)

Docs-as-code agents face validation bypass attacks where malicious content in requirements, issues, or RST files contains hidden instructions that cause the agent to skip compliance checks or corrupt traceability links.

This architecture is inspired by SAFEHARNESS: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment (Lin et al., 2026), adapted for the docs-as-code domain.

Threat Model

T1: Indirect Injection via Requirements/Issues

• Adversary creates requirement or issue with embedded hidden instructions
• Agent reads malicious content as trusted context
• Agent bypasses validation steps per hidden instruction
• Non-compliant change merges → ISO 26262/ASPICE violation

T2: Validation Step Bypass

• Agent claims "all checks passed" without actually running them
• Gate sees fabricated evidence or incomplete validation
• Traceability breaks go undetected until audit

T3: Critical Artifact Corruption

• Agent deletes or corrupts needs.json
• Agent breaks critical complies, tests, or implements links
• Agent modifies requirement content inappropriately
• Manual recovery and re-review required

T4: Privilege Escalation

• Agent attempts to write outside docs/ directory
• Agent modifies files it should only read
• Agent accesses system directories (/etc, /root)

Defense Layers

Inspired by SAFEHARNESS's four-layer lifecycle integration model, adapted for docs-as-code security:

L1: Input Sanitization (INFORM)

Purpose: Filter all external content before it enters agent context

Mechanisms:

• Structural sanitization: strip HTML comments, hidden Unicode
• Pattern detection: flag "ignore previous instructions", "skip validation"
• Semantic filtering: LLM-aided extraction of factual content only
• Provenance tagging: mark source (requirement vs user input vs tool output)

Implementation: Issue #TBD-A

SAFEHARNESS reference: §3.3.1 Layer 1: INFORM

L2: Validation Verification (VERIFY)

Purpose: Verify agent actually ran required validation steps

Mechanisms:

• Rule-based compliance: check required links exist (complies, tests, implements)
• Contextual judge: LLM evaluates whether validation was complete
• Causal diagnostics: determine if action was caused by legitimate task or injection
• Three-tier cascade: escalate verification depth for suspicious actions

Implementation: Issue #TBD-B

SAFEHARNESS reference: §3.3.2 Layer 2: VERIFY

L3: Privilege Separation (CONSTRAIN)

Purpose: Enforce least-privilege access control

Mechanisms:

• Risk tiers: read_only → write_rst → edit_needs → destructive
• Capability tokens: time-limited, invocation-limited access per tool
• Path constraints: docs/ only, reject /etc, /root, parent directory traversal
• Tool integrity: HMAC verification of tool descriptions

Implementation: Issue #TBD-C

SAFEHARNESS reference: §3.3.3 Layer 3: CONSTRAIN

L4: Rollback & Adaptive Degradation (CORRECT)

Purpose: Last line of defense; safe recovery from failures

Mechanisms:

• State checkpointing: snapshot docs/ and needs.json before agent runs
• Attack-triggered rollback: restore on traceability_gate.py failure
• Adaptive degradation: progressively restrict privileges after failures
  • Level 0: normal (all tiers)
  • Level 1: disable destructive tools
  • Level 2: disable needs.json edits
  • Level 3: disable RST writes
  • Level 4: read-only mode
• Adaptive recovery: restore privileges after sustained safe behavior

Implementation: Issue #TBD-D

SAFEHARNESS reference: §3.3.4 Layer 4: CORRECT

Cross-Layer Mechanisms

Entropy Monitor:

• Track constraint violation rate in sliding window (default: 20 actions)
• Escalate L2 verification rigor when violations exceed threshold (default: 30%)
• Enables coordinated system-level response to composite attacks

Protected Memory with Provenance:

• All agent state stored with source type, timestamp, content hash
• Anomaly detection on mutation patterns
• Memory included in L4 checkpoints (prevents memory corruption from surviving rollback)

Sub-Issues

Phase 2 implementation (post-MVP):

• #TBD-A: Add content sanitization for requirements/issues/RST (L1)
• #TBD-B: Add validation step verification with causal diagnostics (L2)
• #TBD-C: Add risk tiers, capability tokens, and path constraints (L3)
• #TBD-D: Add state checkpointing, rollback, and adaptive degradation (L4)
• #TBD-E: Add entropy monitor and cross-layer coordination

Design Principles

• External verification: Validation results come from traceability_gate.py, not agent self-report
• Untrusted input: Requirements, issues, and RST files are treated as potentially adversarial
• Deterministic enforcement: Security checks are Lane A (OSS deterministic), not LLM-dependent
• Progressive degradation: System degrades gracefully under attack rather than failing open
• Lifecycle integration: Security embedded in harness execution phases, not external wrapper

Differences from SAFEHARNESS

The docs-as-code domain has distinct security properties compared to SAFEHARNESS's general agent setting:

Aspect	SAFEHARNESS (general)	Docs-as-Code (this work)
Primary threat	System compromise, data exfiltration	Compliance bypass, traceability corruption
Untrusted input	Web content, API responses	Requirements, issues, RST files
Tool risk	Shell commands, file system, network	Read RST, write RST, edit needs.json
Validation oracle	Multi-tier LLM judge	traceability_gate.py (deterministic)
Recovery goal	Safe rollback to checkpoint	Restore traceability consistency

References

• Lin, X., Liu, Y., Chen, Y., et al. (2026). SAFEHARNESS: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment. arXiv:2604.13630. https://arxiv.org/pdf/2604.13630
• OWASP (2023). OWASP Top 10 for Large Language Model Applications. https://owasp.org/www-project-top-10-for-large-language-model-applications/