diff --git a/CHANGELOG.md b/CHANGELOG.md index 3c2010a..eb3509c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,40 @@ All notable changes to this project are documented in this file. The project follows semantic versioning. +## [1.0.0] - 2026-06-06 + +### Added in 1.0.0 + +- Composable learning architecture documentation for routing lessons into + global, atom, project-local, skill-prevention, and detection targets. +- Routing metadata for captured and finalized learning notes, including lesson + family, scope, prevention targets, detection targets, template upstream + status, recurrence classification, and routing rationale. +- `--enforce-routing` validation for promoted lessons, including detection-only + routes that require a detection target. +- `summarize-runs` reporting for recurrence, missing routing, queue counts, and + duplicate-after-prevention metrics. +- `create-template-draft` for clean, draft-first handoffs into the agent + template repository. +- AI-agent runbook and prompt source for installing the full two-repository + auto-learning automation pipeline. +- Tests for routing enforcement, recurrence metrics, concurrent state updates, + transactional finalization, and template draft privacy handling. + +### Changed in 1.0.0 + +- Consolidation skills and automations now require routing-aware finalization + and distinguish prevention targets from detection targets. +- State persistence now uses locked atomic updates and validates state before + moving notes out of the queue. + +### Fixed in 1.0.0 + +- Prevented notes from being archived when state update or state loading fails. +- Rejected blank routing rationales when routing enforcement is enabled. +- Rejected blocked or unsanitized template draft creation. +- Restricted `summarize-runs --since` to `YYYY-MM-DD` date filtering. + ## [0.1.0] - 2026-05-19 ### Added diff --git a/README.md b/README.md index 2460206..7b26ed0 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,11 @@ reusable lessons into agent instructions and review skills. - [Purpose](#purpose) - [Repository Layout](#repository-layout) - [Install](#install) +- [Full Pipeline Setup](#full-pipeline-setup) - [Learning Store](#learning-store) +- [Instruction and Skill Context](#instruction-and-skill-context) +- [Conflict Detection Levels](#conflict-detection-levels) +- [Architecture](#architecture) - [Skills](#skills) - [Review Skill Hooks](#review-skill-hooks) - [Automations](#automations) @@ -37,7 +41,12 @@ The system keeps learning lightweight and auditable: ├── automations/ │ ├── midnight-consolidation.md │ ├── morning-review-email.md -│ └── noon-consolidation.md +│ ├── noon-consolidation.md +│ ├── template-weekly-upstream-apply.md +│ └── weekly-rule-audit.md +├── docs/ +│ ├── auto-learning-pipeline-automations.md +│ └── composable-learning-architecture.md ├── scripts/ │ └── agent_learning.py ├── skills/ @@ -62,7 +71,7 @@ The installer: - creates the learning store folders; - copies both skills into `~/.agents/skills`; - links `~/.codex/skills` to the installed `~/.agents/skills` copies; -- installs or updates the three Codex automations as active; +- installs or updates the daily Codex automations as active; - validates Markdown, shell, and Python files. Pass `--dir PATH` to choose any filesystem directory as the learning-store base. @@ -104,6 +113,25 @@ flowchart LR automations --> done["Finish install"] ``` +## Full Pipeline Setup + +For the complete two-repository auto-learning pipeline, install this repository +and the template repository, then ensure the weekly audit and template upstream +automations exist. + +Follow +[`docs/auto-learning-pipeline-automations.md`](docs/auto-learning-pipeline-automations.md) +when asking an AI agent to create or repair the Codex automations. The runbook +covers: + +| Component | Repo | +| --- | --- | +| Daily note consolidation | `agent-learning-system` | +| Morning review notification | `agent-learning-system` | +| Weekly duplicate and conflict audit | `agent-learning-system` | +| Approved template draft apply | `agents-file-templates-and-skills` | +| Generated-project refresh reports | `agents-file-templates-and-skills` | + ## Learning Store The canonical archive lives under `AGENT_LEARNING_DIR` plus @@ -132,7 +160,8 @@ This creates: Obsidian vault, Obsidian sees the generated Markdown files as normal notes. > Source: `scripts/agent_learning.py` commands `record`, `prepare-run`, -> `finalize-note`, and `write-report`. +> `finalize-note`, `write-report`, `summarize-runs`, and +> `create-template-draft`. ```mermaid flowchart LR @@ -168,20 +197,27 @@ The consolidator reads `inbox/`, `needs-review/`, and `state/processed.json`. It does not reread `processed/` history unless resolving a duplicate or conflict. -## Conflicts and precedence (visual) +## Instruction and Skill Context ```mermaid flowchart LR - accTitle: Rule precedence when two rules disagree - accDescr: Shows which rule wins when guidance conflicts. - g["Global: $HOME/AGENTS.md"] --> p["Project: /AGENTS.md"] - p --> s["Domain skill: $HOME/.agents/skills/*/SKILL.md"] - s --> d{"Same scope?"} - d -->|No| keep["Keep both; clarify scope"] - d -->|Yes| review["Send to needs-review"] + accTitle: Instruction and skill context + accDescr: Shows always-loaded instructions separately from invoked skills. + global["Global: ${HOME}/AGENTS.md"] --> project["Project: /AGENTS.md"] + project --> composed["Composed prevention context"] + skill["Invoked skill: ${HOME}/.agents/skills/*/SKILL.md"] --> workflow["Workflow or detection context"] + composed --> decision{"Same scope conflict?"} + workflow --> decision + decision -->|No| keep["Keep both; clarify scope"] + decision -->|Yes| review["Send to needs-review"] ``` -## Conflict detection levels +Project instructions and global instructions are prevention context when they +are loaded before the agent acts. Skills are workflow context: they can be +prevention only when the failing agent would normally invoke that skill before +acting. + +## Conflict Detection Levels ```mermaid flowchart LR @@ -197,6 +233,13 @@ flowchart LR The system currently supports a periodic audit via `audit-rules` (see below). +## Architecture + +The routing model is documented in +[`docs/composable-learning-architecture.md`](docs/composable-learning-architecture.md). +That document defines prevention targets, detection targets, recurrence +tracking, and draft-first upstreaming into reusable instruction templates. + ## Skills ### `record-agent-learning` @@ -215,6 +258,21 @@ Use from scheduled automation or manually when promoting learning notes. It classifies new notes, handles reviewed checkbox decisions, promotes strong lessons, moves notes, writes reports, and validates changed files. +Consolidations should follow the routing contract in +[`docs/composable-learning-architecture.md`](docs/composable-learning-architecture.md#required-routing-contract) +so reports distinguish prevention targets from detection targets. Pass +`--enforce-routing` when finalizing promoted lessons that should have a +prevention target. + +Create draft-first template handoffs with `create-template-draft`, then review +them in the template repository before curated atom edits. + +Measure recurrence and routing gaps with: + +```bash +python3 scripts/agent_learning.py summarize-runs --since "2026-06-01" +``` + ### `audit-rules` Scan promoted targets (default: `$HOME/AGENTS.md` and `$HOME/.agents/skills/**`) @@ -288,12 +346,16 @@ The active Codex automations should use the prompt files under `automations/`: - `agent-learning-midnight-consolidation` at `00:00` Europe/Rome; - `agent-learning-morning-review-email` at `08:30` Europe/Rome; -- `agent-learning-noon-consolidation` at `12:00` Europe/Rome. +- `agent-learning-noon-consolidation` at `12:00` Europe/Rome; +- `agent-learning-weekly-rule-audit` weekly before template apply; +- `agent-template-weekly-upstream-apply` weekly in the template repository. These automations are stored under `~/.codex/automations/` after Codex creates them. The shell installer also writes these records directly so a normal `./install.sh` run activates the daily loop without a separate manual Codex -step. +step. Use +[`docs/auto-learning-pipeline-automations.md`](docs/auto-learning-pipeline-automations.md) +for the full two-repository setup. This automation flow follows the prompt files in `automations/` and the notification behavior in `scripts/agent_learning.py notify`. @@ -305,7 +367,11 @@ flowchart LR midnight["00:00 consolidation"] --> consolidate["Process inbox and reviews"] noon["12:00 consolidation"] --> consolidate consolidate --> promote["Promote safe lessons"] + promote --> draft["Create approved-review template drafts when useful"] promote --> report["Write report and validate"] + weekly["Weekly rule audit"] --> audit["Check duplicates and conflicts"] + draft --> template["Weekly template upstream apply"] + template --> refresh["Write out-of-sync report"] morning["08:30 review email"] --> pending{"Pending reviews?"} pending -->|No| quiet["Do not send email"] pending -->|Yes| gmail{"Gmail connector available?"} @@ -321,7 +387,7 @@ uses the local `msmtp` fallback through `scripts/agent_learning.py notify`. Run: ```bash -markdownlint --config ~/.markdownlint.json AGENTS.md CHANGELOG.md README.md skills/**/*.md automations/*.md +markdownlint --config ~/.markdownlint.json AGENTS.md CHANGELOG.md README.md docs/*.md skills/**/*.md automations/*.md shellcheck --enable=all install.sh python3 -m py_compile scripts/agent_learning.py tests/test_agent_learning.py python3 -m unittest discover -s tests diff --git a/automations/midnight-consolidation.md b/automations/midnight-consolidation.md index 27c04b9..84e9849 100644 --- a/automations/midnight-consolidation.md +++ b/automations/midnight-consolidation.md @@ -9,13 +9,24 @@ Process new Obsidian learning notes from `inbox/` and reviewed notes from - `$HOME/AGENTS.md`; - the smallest relevant reusable skill; - the touched project's `AGENTS.md`; -- the existing agent-template mining workflow when useful. +- ignored template-upstream drafts when reusable atom changes are useful. Do not commit or push. Write a consolidation report and validate every changed -Markdown or shell file. +Markdown or shell file. Finalize promoted notes with routing metadata and +`--enforce-routing`. For detection-only lessons, use +`--scope skill-detection --detection-target ... --enforce-routing`. Omit +enforcement only for rejected, no-op, or pending-review outcomes. Use +`create-template-draft` for reusable template candidates before any curated +template edit. -After promotions, run a rule audit against installed promotion targets to catch -duplicates or potential conflicts: +After promotions, summarize recurrence and run a rule audit against installed +promotion targets to catch duplicates or potential conflicts: + +```bash +repo="/path/to/agent-learning-system" +python3 "${repo}/scripts/agent_learning.py" summarize-runs \ + --format markdown +``` ```bash repo="/path/to/agent-learning-system" diff --git a/automations/noon-consolidation.md b/automations/noon-consolidation.md index 28b7b94..3ef5ccb 100644 --- a/automations/noon-consolidation.md +++ b/automations/noon-consolidation.md @@ -9,10 +9,21 @@ ambiguous notes pending. Do not reread processed history except to resolve a specific duplicate or conflict. Do not commit or push. Write a consolidation report and validate every changed -Markdown or shell file. +Markdown or shell file. Finalize promoted notes with routing metadata and +`--enforce-routing`. For detection-only lessons, use +`--scope skill-detection --detection-target ... --enforce-routing`. Omit +enforcement only for rejected, no-op, or pending-review outcomes. Use +`create-template-draft` for reusable template candidates before any curated +template edit. -After promotions, run a rule audit against installed promotion targets to catch -duplicates or potential conflicts: +After promotions, summarize recurrence and run a rule audit against installed +promotion targets to catch duplicates or potential conflicts: + +```bash +repo="/path/to/agent-learning-system" +python3 "${repo}/scripts/agent_learning.py" summarize-runs \ + --format markdown +``` ```bash repo="/path/to/agent-learning-system" diff --git a/automations/template-weekly-upstream-apply.md b/automations/template-weekly-upstream-apply.md new file mode 100644 index 0000000..817a247 --- /dev/null +++ b/automations/template-weekly-upstream-apply.md @@ -0,0 +1,70 @@ +# Agent Template Weekly Upstream Apply + +Use the local agent-template repository in +`${HOME}/Development/agents-file-templates-and-skills`. + +Run the reviewed template upstream workflow: + +1. Inspect pending reusable-template handoffs: + + ```bash + python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --learning-upstream-summary + ``` + +2. Inspect `.work/learning-upstream/*.md`. + + Apply only drafts whose handoff explicitly says: + + - `Review status: approved` + - `Privacy verdict: clean` + +3. For each approved clean draft, run a dry run first: + + ```bash + python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --apply-learning-draft "" + ``` + +4. Apply only after the dry run is correct: + + ```bash + python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --apply-learning-draft "" \ + --apply + ``` + +5. Skip draft, needs-scrub, blocked, ambiguous, private-looking, + duplicate-looking, or unreviewed handoffs. Report each skipped draft with + the reason. + +6. Validate the template workflow: + + ```bash + markdownlint --config "${HOME}/.markdownlint.json" \ + README.md CHANGELOG.md TODO.md docs/*.md templates/**/*.md skills/**/*.md + python3 scripts/privacy_scan.py . + python3 -m py_compile \ + scripts/privacy_scan.py \ + skills/init-agents-file/scripts/init_agents_file.py \ + skills/update-agents-file-templates/scripts/update_agents_templates.py \ + tests/test_template_workflows.py + python3 -m unittest discover -s tests + ``` + +7. Run the out-of-sync scan: + + ```bash + python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --scan-root "${HOME}/Development" \ + --out-of-sync-report + ``` + +Final response must include approved drafts applied, skipped drafts and reasons, +validation results, out-of-sync summary, and any remaining manual action. + +Do not stage, commit, push, open pull requests, or edit unrelated files. diff --git a/automations/weekly-rule-audit.md b/automations/weekly-rule-audit.md index 64fae2a..88aa92f 100644 --- a/automations/weekly-rule-audit.md +++ b/automations/weekly-rule-audit.md @@ -7,6 +7,7 @@ push. Catch long-term drift: duplicate or potentially conflicting rules that have accumulated across `$HOME/AGENTS.md` and installed `$HOME/.agents/skills`. +Also report repeated lessons after a prevention target exists. ## Steps @@ -20,11 +21,22 @@ python3 "${repo}/scripts/agent_learning.py" audit-rules \ > /tmp/agent-learning-audit.json ``` -If `potential_conflicts` or `near_duplicates` is non-empty, write a short -Obsidian note under `AI Agent Learnings/reports/` summarizing: +Summarize recurrence: + +```bash +repo="/path/to/agent-learning-system" +python3 "${repo}/scripts/agent_learning.py" summarize-runs \ + --format markdown \ + > /tmp/agent-learning-recurrence.md +``` + +If `potential_conflicts`, `near_duplicates`, missing routing, or duplicate +after-prevention counts are non-empty, write a short Obsidian note under +`AI Agent Learnings/reports/` summarizing: - counts for `exact_duplicates`, `near_duplicates`, `potential_conflicts`; +- recurrence counts and missing routing count; - the top 3 highest-ratio pairs with file paths + line numbers; - the recommended action: merge/clarify scope or move to `needs-review`. -If all three lists are empty, do nothing. +If the audit and recurrence summary are clean, do nothing. diff --git a/docs/auto-learning-pipeline-automations.md b/docs/auto-learning-pipeline-automations.md new file mode 100644 index 0000000..3c5d172 --- /dev/null +++ b/docs/auto-learning-pipeline-automations.md @@ -0,0 +1,174 @@ +# Auto-Learning Pipeline Automations + +This runbook is for an AI agent installing or repairing the complete +auto-learning pipeline across `agent-learning-system` and +`agents-file-templates-and-skills`. + +## Table of Contents + +- [Scope](#scope) +- [Pipeline Map](#pipeline-map) +- [Required Automations](#required-automations) +- [Install Checklist](#install-checklist) +- [Validation](#validation) +- [Failure Controls](#failure-controls) + +## Scope + +The full pipeline needs both repositories: + +| Repository | Responsibility | +| --- | --- | +| `${HOME}/Development/agent-learning-system` | Record notes, consolidate reviews, route prevention targets, create template drafts, audit installed rules | +| `${HOME}/Development/agents-file-templates-and-skills` | Review approved drafts, update curated atoms, report projects that need refreshed generated instructions | + +Use `${HOME}` in prompts and docs. The Codex scheduler may store resolved local +workspace paths internally, but repository documentation must not hard-code a +personal home path. + +## Pipeline Map + +```mermaid +flowchart LR + review["Review or implementation session"] --> note["Learning note"] + note --> daily["Daily consolidation"] + daily --> route{"Reusable prevention target?"} + route -->|Local| live["Global, skill, or project AGENTS.md"] + route -->|Reusable atom| draft[".work/learning-upstream draft"] + live --> audit["Weekly rule audit"] + draft --> template["Weekly template upstream apply"] + template --> atoms["Curated template atoms"] + atoms --> refresh["Out-of-sync report"] + refresh --> projects["Project AGENTS.md refresh"] +``` + +## Required Automations + +| Automation ID | Cwd | Cadence | Prompt Source | Purpose | +| --- | --- | --- | --- | --- | +| `agent-learning-midnight-consolidation` | `agent-learning-system` | Daily at midnight | `automations/midnight-consolidation.md` | Process inbox and reviewed notes | +| `agent-learning-noon-consolidation` | `agent-learning-system` | Daily at noon | `automations/noon-consolidation.md` | Catch daytime notes and reviewed decisions | +| `agent-learning-morning-review-email` | `agent-learning-system` | Daily morning | `automations/morning-review-email.md` | Notify only when `needs-review/` has work | +| `agent-learning-weekly-rule-audit` | `agent-learning-system` | Weekly before template apply | `automations/weekly-rule-audit.md` | Detect duplicate, conflicting, or bloated installed rules | +| `agent-template-weekly-upstream-apply` | `agents-file-templates-and-skills` | Weekly after rule audit | `automations/template-weekly-upstream-apply.md` | Apply only approved clean template drafts and write refresh reports | + +The learning installer writes the daily learning automations. The weekly rule +audit and template upstream automation are Codex automations that an agent +should create or update explicitly if missing. + +## Install Checklist + +1. Verify both repositories exist: + + ```bash + test -d "${HOME}/Development/agent-learning-system" + test -d "${HOME}/Development/agents-file-templates-and-skills" + ``` + +2. Install or refresh the learning system: + + ```bash + cd "${HOME}/Development/agent-learning-system" + + set -a + . "${HOME}/.config/agent-learning-system/config.env" + set +a + + ./install.sh \ + --dir "${AGENT_LEARNING_DIR}" \ + --email "${AGENT_LEARNING_EMAIL}" \ + --email-provider "${AGENT_LEARNING_EMAIL_PROVIDER}" + ``` + + If the config file does not exist, run `./install.sh` with explicit + `--dir`, `--email`, and `--email-provider` values. Prefer direct filesystem + access to the learning store. + +3. Install or refresh the template skills: + + ```bash + cd "${HOME}/Development/agents-file-templates-and-skills" + make install AGENTS="openai" + ``` + + Add other agents only when the user asks for them. Use `make install-symlink` + when installed skills should point back to the checkout. + +4. Inspect existing Codex automations before creating anything: + + ```bash + find "${CODEX_HOME:-${HOME}/.codex}/automations" \ + -maxdepth 2 \ + -name automation.toml \ + -print + + rg -n "agent-learning|learning-upstream|template-upstream" \ + "${CODEX_HOME:-${HOME}/.codex}/automations" \ + -g automation.toml + ``` + +5. In Codex Desktop, use the automation tool to create or update missing + automations from [Required Automations](#required-automations). + + Use these rules: + + - prefer updating an existing matching automation over creating a duplicate; + - use `local` execution for these jobs; + - use the matching repository as the automation cwd; + - keep active jobs `ACTIVE`; + - preserve existing model and reasoning settings unless they are clearly + wrong; + - load prompt text from the prompt source files above; + - do not hand-edit scheduler records except through `install.sh` for the + daily learning automations. + +6. Ensure the template weekly job keeps curated edits gated: + + ```text + summary -> inspect drafts -> dry run approved clean drafts -> apply -> + validate -> out-of-sync report + ``` + + It must not apply draft, blocked, needs-scrub, duplicate-looking, private, or + unreviewed handoffs. + +## Validation + +After setup, verify the learning side: + +```bash +cd "${HOME}/Development/agent-learning-system" +python3 scripts/agent_learning.py init-store +python3 scripts/agent_learning.py prepare-run --json +markdownlint --config "${HOME}/.markdownlint.json" \ + AGENTS.md CHANGELOG.md README.md docs/*.md skills/**/*.md automations/*.md +shellcheck --enable=all install.sh +python3 -m py_compile scripts/agent_learning.py tests/test_agent_learning.py +python3 -m unittest discover -s tests +``` + +Verify the template side: + +```bash +cd "${HOME}/Development/agents-file-templates-and-skills" +python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --learning-upstream-summary +python3 skills/update-agents-file-templates/scripts/update_agents_templates.py \ + --template-repo . \ + --scan-root "${HOME}/Development" \ + --out-of-sync-report +make validate +``` + +## Failure Controls + +| Failure | Control | +| --- | --- | +| Duplicate automation | Search existing automation records before create | +| Learning notes never affect templates | Confirm `create-template-draft` runs during consolidation and the weekly template job exists | +| Template edits leak private data | Require `Privacy verdict: clean` and run `scripts/privacy_scan.py` | +| Templates change without review | Weekly template job applies only `approved` and `clean` drafts | +| Same mistake repeats after promotion | Check weekly rule audit plus `summarize-runs` recurrence output | +| Generated projects go stale | Review `.work/out-of-sync/` after template apply | +| Automation commits unexpectedly | Prompt every automation with no stage, commit, push, or PR side effects | diff --git a/docs/composable-learning-architecture.md b/docs/composable-learning-architecture.md new file mode 100644 index 0000000..ad5508a --- /dev/null +++ b/docs/composable-learning-architecture.md @@ -0,0 +1,236 @@ +# Composable Learning Architecture + +This document defines how the learning loop should improve live instructions, +project instructions, reusable skills, and template atoms without duplicating +rules. + +## Table of Contents + +- [Goal](#goal) +- [System Map](#system-map) +- [Routing Rule](#routing-rule) +- [Targets](#targets) +- [Required Routing Contract](#required-routing-contract) +- [Failure Modes](#failure-modes) +- [Template Upstreaming](#template-upstreaming) +- [Recurrence Measurement](#recurrence-measurement) +- [Automation Policy](#automation-policy) +- [Automation Install Runbook](#automation-install-runbook) + +## Goal + +The system must promote each real lesson to the place where it can prevent the +same class of mistake before the agent repeats it. + +```text +Wrong question: Where can this lesson be documented? +Right question: Where must this lesson live so the failing agent would read it +before acting? +``` + +Review skills are detection targets. They help catch issues later. Prevention +targets are the files or generated instruction layers an implementation agent +loads before acting. + +## System Map + +```mermaid +flowchart TD + session["Agent session"] --> note["Learning note"] + note --> consolidate["Consolidation"] + + consolidate --> route{"Route lesson"} + route --> global["${HOME}/AGENTS.md\nDirector policy"] + route --> atom["Template atom\nReusable project or task rule"] + route --> project["Project AGENTS.md\nProject-local rule"] + route --> skill["Reusable skill\nDetection or workflow rule"] + route --> review["needs-review\nUnclear, broad, or conflicting"] + + atom --> templates["agents-file-templates-and-skills"] + templates --> refresh["Refresh affected project AGENTS.md"] + global --> reference["Reference from project AGENTS.md"] + project --> local["Preserve project-local section"] + reference --> future + local --> future + refresh --> future["Future agent reads prevention rule"] + skill --> detect["Future review detects issue"] +``` + +## Routing Rule + +Every promoted lesson must answer this routing question: + +```text +Would the agent that made the mistake have read the promoted rule before acting? +``` + +| Answer | Required Action | +| --- | --- | +| Yes | Record the prevention target and rationale. | +| No | Add or propose a prevention target, not only a detection target. | +| Unclear | Move the note to `needs-review/`. | +| Too local | Archive with a project-local rationale or update only project rules. | + +## Targets + +| Target | Purpose | Use When | +| --- | --- | --- | +| Global director | Universal working style and safety policy | The rule applies across almost every project | +| Template atom | Reusable project-type or task rule | Similar projects should inherit the rule | +| Project `AGENTS.md` | Repo-specific facts and commands | The rule depends on one project | +| Reusable skill | Detection, review, remediation, or workflow behavior | The rule changes how a skill works | +| `needs-review/` | Human decision point | The rule is broad, risky, conflicting, or privacy-sensitive | + +Do not treat a review-skill update as prevention unless the failing agent would +normally load that skill before acting. + +## Required Routing Contract + +The helper commands write these fields and can enforce them with +`--enforce-routing` during finalization or report creation. + +Finalized note frontmatter should include: + +| Field | Meaning | +| --- | --- | +| `lesson_family` | Stable dedupe identity for recurrence tracking | +| `scope` | `global`, `atom`, `project-local`, `skill-prevention`, `skill-detection`, or `needs-review` | +| `prevention_targets` | JSON list of files or generated layers read before acting | +| `detection_targets` | JSON list of review or diagnostic skills that catch the issue later | +| `template_upstream_status` | `not-applicable`, `draft-created`, `promoted`, or `deferred` | +| `routing_rationale` | Why the target can prevent recurrence | + +Every consolidation report should include this section: + +```markdown +## Routing Decision + +- Lesson family: `` +- Scope: `` +- Prevention targets: + - `` +- Detection targets: + - `` +- Template upstream status: `` +- Routing rationale: `` +- Recurrence check: `` +``` + +If `prevention_targets` is empty and `scope` is not `skill-detection` or +`needs-review`, the promotion is incomplete. + +Use `skill-detection` when the lesson improves a review or diagnostic workflow +but no safe prevention target is known yet. Detection-only lessons must include +a recurrence follow-up so repeated findings can be re-routed to prevention. + +## Failure Modes + +| Risk | Failure Mode | Prevention | +| --- | --- | --- | +| Wrong target | Rule exists but the failing agent never reads it | Require `prevention_targets` | +| Review-only learning | Review skill catches the issue again | Separate detection from prevention | +| Too broad | Global file becomes noisy | Prefer atoms or project-local rules | +| Too narrow | Same rule is copied into many projects | Promote reusable rules into atoms | +| Conflicting atoms | Combined instructions disagree | Audit composed output before refresh | +| Instruction bloat | Project files become hard to scan | Keep atoms concise and split large atoms | +| Bad lesson | One-off or wrong rule is promoted | Use `needs-review/` and evidence checks | +| Privacy leak | Local paths or private values enter templates | Run privacy scan before template changes | +| False metrics | Duplicate count is misread | Track recurrence by lesson family and time | + +## Template Upstreaming + +Reusable prevention rules should flow into +`agents-file-templates-and-skills` as draft-first updates. + +```mermaid +flowchart LR + lesson["Reusable lesson"] --> draft["Write ignored .work draft"] + draft --> review["Review for scope, duplication, and privacy"] + review --> curate{"Safe and useful?"} + curate -->|Yes| template["Edit curated atom/template"] + curate -->|No| defer["Defer or needs-review"] + template --> report["Out-of-sync report"] + report --> refresh["Refresh affected projects"] +``` + +Draft-first means automation may create `.work/` candidate material, but curated +templates are edited only after review and privacy scrubbing. Draft text must +already be scrubbed; private values are rejected before the draft is written. + +The canonical handoff path and fields live in the template repository's +`docs/composable-agent-instructions.md#learning-handoff-format`. This repo owns +the routing decision that creates the draft; the template repository owns the +draft format, privacy review, curation, and project refresh. + +Create the ignored handoff draft with: + +```bash +python3 scripts/agent_learning.py create-template-draft \ + --template-repo "/path/to/agents-file-templates-and-skills" \ + --lesson-family "markdown-validation-contract" \ + --source-note "/path/to/note.md" \ + --proposed-template "docs" \ + --candidate-rule "Run markdownlint before completing Markdown documentation changes." \ + --prevention-target "atom:docs" \ + --routing-rationale "Documentation agents load the docs atom before editing Markdown." \ + --privacy-verdict clean \ + --enforce-routing +``` + +The template repository then owns: + +| Step | Command | +| --- | --- | +| Summarize drafts | `update_agents_templates.py --learning-upstream-summary` | +| Preview curated apply | `update_agents_templates.py --apply-learning-draft ` | +| Apply approved draft | `update_agents_templates.py --apply-learning-draft --apply` | +| Report stale projects | `update_agents_templates.py --out-of-sync-report` | + +## Recurrence Measurement + +The system should measure whether learning improved behavior: + +| Metric | Meaning | +| --- | --- | +| Time to process | How long inbox notes wait before consolidation | +| Duplicate before promotion | Normal backlog noise | +| Duplicate after prevention target | Possible routing or rule-quality failure | +| Detection-only recurrence | Review skill works, prevention is missing | +| Project refresh coverage | Which projects received updated atoms | + +A repeated lesson after a prevention target exists is not automatically a +failure, but it must be investigated. + +Summarize recurrence with: + +```bash +python3 scripts/agent_learning.py summarize-runs \ + --since "2026-06-01" \ + --format markdown +``` + +## Automation Policy + +| Automation | Safe Behavior | +| --- | --- | +| Noon and midnight consolidation | Promote bounded lessons and report routing metadata | +| Weekly rule audit | Audit duplicates, conflicts, bloat, and missing prevention targets | +| Template upstream draft | Create ignored `.work/learning-upstream/` drafts only | +| Template curation | Manual or explicitly requested apply step | + +No automation should commit, push, or silently edit curated templates without an +explicit apply request. + +## Automation Install Runbook + +The complete two-repository pipeline is documented in +[`auto-learning-pipeline-automations.md`](auto-learning-pipeline-automations.md). + +That runbook is the source of truth for AI agents installing or repairing: + +| Automation Class | Repository | +| --- | --- | +| Daily consolidation and review notification | `agent-learning-system` | +| Weekly rule audit | `agent-learning-system` | +| Weekly approved-draft template apply | `agents-file-templates-and-skills` | +| Out-of-sync reporting after template edits | `agents-file-templates-and-skills` | diff --git a/scripts/agent_learning.py b/scripts/agent_learning.py index 91d1651..6b612dd 100644 --- a/scripts/agent_learning.py +++ b/scripts/agent_learning.py @@ -4,6 +4,7 @@ from __future__ import annotations import argparse +from contextlib import contextmanager import difflib import json import os @@ -12,6 +13,7 @@ import subprocess import sys import textwrap +import time import uuid from datetime import datetime from pathlib import Path @@ -42,8 +44,12 @@ SECRET_ASSIGNMENT_PATTERN = ( "(?i)" + r"\b(?:token|secret|password|cookie|api[_-]?key)\b\s*[:=]" ) +LOCAL_PATH_PATTERN = ( + r"/(?:Users|home)/[A-Za-z0-9._-]+(?:/[^\s`'\"<>)]*)?" + r"|/(?:workspace|workspaces)/[A-Za-z0-9._-]+(?:/[^\s`'\"<>)]*)?" +) PRIVATE_PATTERNS = [ - re.compile("/" + r"Users/[A-Za-z0-9._-]+"), + re.compile(LOCAL_PATH_PATTERN), re.compile(EMAIL_PATTERN), re.compile(r"\b(?:10|192\.168|172\.(?:1[6-9]|2\d|3[0-1]))(?:\.\d{1,3}){2,3}\b"), re.compile(SECRET_ASSIGNMENT_PATTERN), @@ -98,6 +104,19 @@ DEFAULT_AUDIT_TARGETS = (Path.home() / "AGENTS.md",) DEFAULT_SIMILARITY_THRESHOLD = 0.92 DEFAULT_CONFLICT_THRESHOLD = 0.78 +ROUTING_SCOPES = ( + "global", + "atom", + "project-local", + "skill-prevention", + "skill-detection", + "needs-review", +) +TEMPLATE_UPSTREAM_STATUSES = ("not-applicable", "draft-created", "promoted", "deferred") +RECURRENCE_CHECKS = ("new", "duplicate-before-promotion", "duplicate-after-prevention") +STATE_LOCK_TIMEOUT_SECONDS = float(os.environ.get("AGENT_LEARNING_STATE_LOCK_TIMEOUT_SECONDS", "10.0")) +STATE_LOCK_OWNERLESS_GRACE_SECONDS = float(os.environ.get("AGENT_LEARNING_STATE_LOCK_OWNERLESS_GRACE_SECONDS", "5.0")) +STATE_LOCK_OWNER_FILE = "owner.json" class ConfigError(RuntimeError): @@ -150,7 +169,7 @@ def ensure_store(root: Path) -> None: (root / rel).mkdir(parents=True, exist_ok=True) state_path = root / "state" / "processed.json" if not state_path.exists(): - state_path.write_text(json.dumps({"processed": {}}, indent=2) + "\n", encoding="utf-8") + atomic_write_text(state_path, json.dumps({"processed": {}}, indent=2) + "\n") def slugify(value: str) -> str: @@ -229,6 +248,336 @@ def unique_path(path: Path) -> Path: raise RuntimeError(f"Could not create unique path for {path}") +def unique_values(values: list[str]) -> list[str]: + unique: list[str] = [] + for value in values: + cleaned = value.strip() + if cleaned and cleaned not in unique: + unique.append(cleaned) + return unique + + +def parse_json_list_value(value: str) -> list[str] | None: + if not value: + return [] + try: + payload = json.loads(value) + except json.JSONDecodeError: + try: + payload = json.loads(value.replace('\\"', '"')) + except json.JSONDecodeError: + return None + if not isinstance(payload, list): + return None + return [str(item) for item in payload if str(item).strip()] + + +def parse_json_list(value: str) -> list[str]: + return parse_json_list_value(value) or [] + + +def coerce_list(value: Any) -> list[str]: + if value is None: + return [] + if isinstance(value, list): + return [str(item) for item in value if str(item).strip()] + if isinstance(value, str): + parsed = parse_json_list_value(value) + if parsed is not None: + return parsed + stripped = value.strip() + return [stripped] if stripped else [] + return [str(value)] + + +def json_list(values: list[str]) -> str: + return json.dumps(unique_values(values)) + + +def note_title(body: str, fallback: Path) -> str: + for raw in body.splitlines(): + stripped = raw.strip() + if stripped.startswith("#"): + return stripped.lstrip("#").strip() or fallback.stem + return fallback.stem + + +def routing_args_present(args: argparse.Namespace) -> bool: + fields = ( + "lesson_family", + "scope", + "prevention_target", + "detection_target", + "template_upstream_status", + "routing_rationale", + "recurrence_check", + ) + return any(bool(getattr(args, field, None)) for field in fields) + + +def routing_from_args( + args: argparse.Namespace, + fm: dict[str, str] | None = None, + body: str = "", + source: Path | None = None, +) -> dict[str, Any]: + current = fm or {} + fallback = source or Path("learning.md") + family = ( + getattr(args, "lesson_family", None) + or current.get("lesson_family") + or slugify(note_title(body, fallback)) + ) + scope = getattr(args, "scope", None) or current.get("scope", "") + prevention_targets = unique_values( + coerce_list(current.get("prevention_targets")) + coerce_list(getattr(args, "prevention_target", None)) + ) + detection_targets = unique_values( + coerce_list(current.get("detection_targets")) + coerce_list(getattr(args, "detection_target", None)) + ) + template_status = ( + getattr(args, "template_upstream_status", None) + or current.get("template_upstream_status") + or "not-applicable" + ) + rationale = getattr(args, "routing_rationale", None) or current.get("routing_rationale", "") + recurrence = getattr(args, "recurrence_check", None) or current.get("recurrence_check") or "new" + + metadata = { + "lesson_family": str(family), + "scope": str(scope), + "prevention_targets": prevention_targets, + "detection_targets": detection_targets, + "template_upstream_status": str(template_status), + "routing_rationale": str(rationale), + "recurrence_check": str(recurrence), + } + validate_routing_values(metadata) + return metadata + + +def validate_routing_values(metadata: dict[str, Any]) -> None: + scope = str(metadata.get("scope", "")) + template_status = str(metadata.get("template_upstream_status", "")) + recurrence = str(metadata.get("recurrence_check", "")) + if scope and scope not in ROUTING_SCOPES: + raise ConfigError(f"Unsupported routing scope: {scope}") + if template_status and template_status not in TEMPLATE_UPSTREAM_STATUSES: + raise ConfigError(f"Unsupported template upstream status: {template_status}") + if recurrence and recurrence not in RECURRENCE_CHECKS: + raise ConfigError(f"Unsupported recurrence check: {recurrence}") + + +def enforce_routing_contract(metadata: dict[str, Any]) -> None: + family = str(metadata.get("lesson_family", "")).strip() + scope = str(metadata.get("scope", "")).strip() + rationale = str(metadata.get("routing_rationale", "")).strip() + prevention_targets = metadata.get("prevention_targets") or [] + detection_targets = metadata.get("detection_targets") or [] + if not family: + raise ConfigError("Routing metadata requires --lesson-family.") + if not scope: + raise ConfigError("Routing metadata requires --scope.") + if scope != "needs-review" and not rationale: + raise ConfigError("Routing metadata requires --routing-rationale.") + if scope not in {"skill-detection", "needs-review"} and not prevention_targets: + raise ConfigError("Routing metadata requires --prevention-target for prevention-capable scopes.") + if scope == "skill-detection" and not detection_targets: + raise ConfigError("Routing metadata requires --detection-target for skill-detection scope.") + + +def apply_routing_frontmatter(fm: dict[str, str], metadata: dict[str, Any]) -> None: + fm["lesson_family"] = str(metadata["lesson_family"]) + fm["scope"] = str(metadata["scope"]) + fm["prevention_targets"] = json_list(metadata["prevention_targets"]) + fm["detection_targets"] = json_list(metadata["detection_targets"]) + fm["template_upstream_status"] = str(metadata["template_upstream_status"]) + fm["routing_rationale"] = str(metadata["routing_rationale"]) + fm["recurrence_check"] = str(metadata["recurrence_check"]) + + +def routing_state_payload(metadata: dict[str, Any]) -> dict[str, Any]: + return { + "lesson_family": metadata["lesson_family"], + "scope": metadata["scope"], + "prevention_targets": metadata["prevention_targets"], + "detection_targets": metadata["detection_targets"], + "template_upstream_status": metadata["template_upstream_status"], + "routing_rationale": metadata["routing_rationale"], + "recurrence_check": metadata["recurrence_check"], + } + + +def routing_markdown(metadata: dict[str, Any]) -> str: + prevention = metadata.get("prevention_targets") or [] + detection = metadata.get("detection_targets") or [] + lines = [ + "## Routing Decision", + "", + f"- Lesson family: `{metadata.get('lesson_family') or 'not-recorded'}`", + f"- Scope: `{metadata.get('scope') or 'not-recorded'}`", + "- Prevention targets:", + ] + lines.extend(f" - `{item}`" for item in prevention) + if not prevention: + lines.append(" - `none-recorded`") + lines.append("- Detection targets:") + lines.extend(f" - `{item}`" for item in detection) + if not detection: + lines.append(" - `none-recorded`") + lines.extend( + [ + f"- Template upstream status: `{metadata.get('template_upstream_status')}`", + f"- Routing rationale: {metadata.get('routing_rationale') or 'Not recorded.'}", + f"- Recurrence check: `{metadata.get('recurrence_check')}`", + ] + ) + return "\n".join(lines) + "\n" + + +def atomic_write_text(path: Path, text: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(f".{path.name}.{os.getpid()}.{uuid.uuid4().hex}.tmp") + try: + tmp.write_text(text, encoding="utf-8") + os.replace(tmp, path) + finally: + if tmp.exists(): + tmp.unlink() + + +def process_is_running(pid: int) -> bool: + if pid <= 0: + return False + try: + os.kill(pid, 0) + except ProcessLookupError: + return False + except PermissionError: + return True + return True + + +def reclaim_dead_state_lock(lock_dir: Path) -> bool: + owner_path = lock_dir / STATE_LOCK_OWNER_FILE if lock_dir.is_dir() else lock_dir + try: + owner = json.loads(owner_path.read_text(encoding="utf-8")) + pid = int(owner.get("pid")) + except (FileNotFoundError, json.JSONDecodeError, OSError, TypeError, ValueError): + return False + if process_is_running(pid): + return False + try: + if lock_dir.is_dir(): + owner_path.unlink() + lock_dir.rmdir() + else: + lock_dir.unlink() + except OSError: + return False + return True + + +def lock_age_seconds(lock_dir: Path) -> float: + try: + return time.time() - lock_dir.stat().st_mtime + except OSError: + return 0.0 + + +def reclaim_ownerless_state_lock(lock_dir: Path) -> bool: + if not lock_dir.exists() or lock_age_seconds(lock_dir) < STATE_LOCK_OWNERLESS_GRACE_SECONDS: + return False + owner_path = lock_dir / STATE_LOCK_OWNER_FILE if lock_dir.is_dir() else lock_dir + try: + owner = json.loads(owner_path.read_text(encoding="utf-8")) + int(owner.get("pid")) + return False + except (FileNotFoundError, json.JSONDecodeError, OSError, TypeError, ValueError): + pass + try: + if lock_dir.is_dir(): + children = list(lock_dir.iterdir()) + if children and not all( + child.name.startswith(f".{STATE_LOCK_OWNER_FILE}.") and child.name.endswith(".tmp") + for child in children + ): + return False + shutil.rmtree(lock_dir) + else: + lock_dir.unlink() + except OSError: + return False + return True + + +def acquire_state_lock_without_hardlink(lock_dir: Path, owner_payload: str) -> bool: + fd: int | None = None + try: + fd = os.open(lock_dir, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600) + with os.fdopen(fd, "w", encoding="utf-8") as handle: + fd = None + handle.write(owner_payload) + except FileExistsError: + return False + except Exception: + if fd is not None: + os.close(fd) + try: + lock_dir.unlink() + except FileNotFoundError: + pass + raise + return True + + +def acquire_state_lock(lock_dir: Path) -> bool: + pending_path = lock_dir.with_name(f".{lock_dir.name}.{os.getpid()}.{uuid.uuid4().hex}.tmp") + owner_payload = json.dumps({"pid": os.getpid()}) + "\n" + try: + pending_path.write_text(owner_payload, encoding="utf-8") + os.link(pending_path, lock_dir) + except FileExistsError: + return False + except OSError: + return acquire_state_lock_without_hardlink(lock_dir, owner_payload) + finally: + try: + pending_path.unlink() + except FileNotFoundError: + pass + return True + + +@contextmanager +def state_update_lock(root: Path): + lock_dir = root / "state" / ".processed.lock" + lock_dir.parent.mkdir(parents=True, exist_ok=True) + deadline = time.monotonic() + STATE_LOCK_TIMEOUT_SECONDS + acquired = False + while not acquired: + if acquire_state_lock(lock_dir): + acquired = True + continue + if reclaim_dead_state_lock(lock_dir) or reclaim_ownerless_state_lock(lock_dir): + continue + if time.monotonic() >= deadline: + raise ConfigError(f"Timed out waiting for state lock: {lock_dir}") + time.sleep(0.05) + try: + yield + finally: + if acquired: + try: + lock_dir.unlink() + except FileNotFoundError: + pass + + +def privacy_findings(text: str) -> list[str]: + return [pattern.pattern for pattern in PRIVATE_PATTERNS if pattern.search(text)] + + def read_json_input(source: str) -> dict[str, Any]: if source == "-": return json.load(sys.stdin) @@ -271,6 +620,23 @@ def command_record(args: argparse.Namespace) -> int: "Promotion Decision": "Pending consolidation.", } body = f"# {title}\n\n" + "\n".join(section(name, value) for name, value in fields.items()) + routing = routing_from_args( + argparse.Namespace( + lesson_family=data.get("lesson_family") or getattr(args, "lesson_family", None), + scope=data.get("scope") or getattr(args, "scope", None), + prevention_target=coerce_list(data.get("prevention_targets")) + + coerce_list(getattr(args, "prevention_target", None)), + detection_target=coerce_list(data.get("detection_targets")) + + coerce_list(getattr(args, "detection_target", None)), + template_upstream_status=data.get("template_upstream_status") + or getattr(args, "template_upstream_status", None), + routing_rationale=data.get("routing_rationale") or getattr(args, "routing_rationale", None), + recurrence_check=data.get("recurrence_check") or getattr(args, "recurrence_check", None), + ), + {}, + body, + Path(f"{slugify(title)}.md"), + ) fm = { "id": note_id, "created_at": now_iso(), @@ -281,6 +647,7 @@ def command_record(args: argparse.Namespace) -> int: "genericity": genericity, "promoted_targets": "[]", } + apply_routing_frontmatter(fm, routing) filename = f"{datetime.now().strftime('%Y%m%d-%H%M%S')}-{slugify(title)}-{note_id[:8]}.md" path = unique_path(root / "inbox" / filename) write_note(path, fm, body) @@ -353,13 +720,23 @@ def command_prepare_run(args: argparse.Namespace) -> int: def load_state(root: Path) -> dict[str, Any]: state_path = root / "state" / "processed.json" if state_path.exists(): - return json.loads(state_path.read_text(encoding="utf-8")) + try: + return json.loads(state_path.read_text(encoding="utf-8")) + except json.JSONDecodeError as exc: + raise ConfigError(f"State file is malformed: {state_path}") from exc return {"processed": {}} def save_state(root: Path, state: dict[str, Any]) -> None: state_path = root / "state" / "processed.json" - state_path.write_text(json.dumps(state, indent=2, sort_keys=True) + "\n", encoding="utf-8") + atomic_write_text(state_path, json.dumps(state, indent=2, sort_keys=True) + "\n") + + +def update_processed_state(root: Path, note_id: str, entry: dict[str, Any]) -> None: + with state_update_lock(root): + state = load_state(root) + state.setdefault("processed", {})[note_id] = entry + save_state(root, state) def command_finalize_note(args: argparse.Namespace) -> int: @@ -367,9 +744,14 @@ def command_finalize_note(args: argparse.Namespace) -> int: root = learning_root(config) ensure_store(root) source = validate_note_source(root, Path(args.file)) - fm, body = read_note(source) + original_text = source.read_text(encoding="utf-8") + fm, body = parse_frontmatter(original_text) note_id = fm.get("id", source.stem) status = args.status + routing = routing_from_args(args, fm, body, source) + if args.enforce_routing: + enforce_routing_contract(routing) + apply_routing_frontmatter(fm, routing) fm["status"] = status fm["processed_at"] = now_iso() fm["processing_rationale"] = args.rationale @@ -386,23 +768,34 @@ def command_finalize_note(args: argparse.Namespace) -> int: else: raise SystemExit(f"Unsupported status: {status}") - dest.parent.mkdir(parents=True, exist_ok=True) - write_note(source, fm, body) - if source == dest.resolve(): - final = dest - else: - final = unique_path(dest) - if source != final.resolve(): - shutil.move(str(source), str(final)) - - state = load_state(root) - state.setdefault("processed", {})[note_id] = { - "path": str(final), - "status": status, - "processed_at": fm["processed_at"], - "rationale": args.rationale, - } - save_state(root, state) + moved = False + final = dest + with state_update_lock(root): + state = load_state(root) + dest.parent.mkdir(parents=True, exist_ok=True) + write_note(source, fm, body) + if source == dest.resolve(): + final = dest + else: + final = unique_path(dest) + try: + if source != final.resolve(): + shutil.move(str(source), str(final)) + moved = True + state.setdefault("processed", {})[note_id] = { + "path": str(final), + "status": status, + "processed_at": fm["processed_at"], + "rationale": args.rationale, + **routing_state_payload(routing), + } + save_state(root, state) + except Exception: + if moved and final.exists() and not source.exists(): + shutil.move(str(final), str(source)) + if source.exists(): + source.write_text(original_text, encoding="utf-8") + raise print(final) return 0 @@ -413,6 +806,12 @@ def command_write_report(args: argparse.Namespace) -> int: ensure_store(root) run_id = args.run_id or datetime.now().strftime("%Y%m%d-%H%M%S") path = unique_path(root / "reports" / f"{run_id}.md") + routing_section = "" + if routing_args_present(args): + routing = routing_from_args(args, {}, "", Path(f"{run_id}.md")) + if args.enforce_routing: + enforce_routing_contract(routing) + routing_section = "\n" + routing_markdown(routing) body = textwrap.dedent( f"""\ # Agent Learning Consolidation {run_id} @@ -422,6 +821,7 @@ def command_write_report(args: argparse.Namespace) -> int: ## Summary {args.summary.strip()} + {routing_section} """ ) path.write_text(body, encoding="utf-8") @@ -429,6 +829,243 @@ def command_write_report(args: argparse.Namespace) -> int: return 0 +def frontmatter_from_state_entry(entry: dict[str, Any]) -> dict[str, Any]: + path_value = str(entry.get("path") or "") + path = Path(path_value) if path_value else Path() + if path_value and path.exists(): + fm, _body = read_note(path) + return { + "lesson_family": fm.get("lesson_family", entry.get("lesson_family", "")), + "scope": fm.get("scope", entry.get("scope", "")), + "prevention_targets": parse_json_list(fm.get("prevention_targets", "")), + "detection_targets": parse_json_list(fm.get("detection_targets", "")), + "template_upstream_status": fm.get( + "template_upstream_status", + entry.get("template_upstream_status", ""), + ), + "routing_rationale": fm.get("routing_rationale", entry.get("routing_rationale", "")), + "recurrence_check": fm.get("recurrence_check", entry.get("recurrence_check", "")), + } + return { + "lesson_family": entry.get("lesson_family", ""), + "scope": entry.get("scope", ""), + "prevention_targets": coerce_list(entry.get("prevention_targets")), + "detection_targets": coerce_list(entry.get("detection_targets")), + "template_upstream_status": entry.get("template_upstream_status", ""), + "routing_rationale": entry.get("routing_rationale", ""), + "recurrence_check": entry.get("recurrence_check", ""), + } + + +def processed_entry_in_window(entry: dict[str, Any], since: str) -> bool: + if not since: + return True + processed_at = str(entry.get("processed_at") or "") + return bool(processed_at) and processed_at[:10] >= since + + +def validate_since_date(value: str) -> None: + if not value: + return + if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value): + raise ConfigError("--since must be a YYYY-MM-DD date.") + try: + datetime.strptime(value, "%Y-%m-%d") + except ValueError as exc: + raise ConfigError("--since must be a YYYY-MM-DD date.") from exc + + +def command_summarize_runs(args: argparse.Namespace) -> int: + config = load_config(args.config) + root = learning_root(config) + ensure_store(root) + validate_since_date(args.since) + state = load_state(root) + processed = state.get("processed", {}) + by_scope: dict[str, int] = {} + by_recurrence: dict[str, int] = {} + by_family: dict[str, int] = {} + duplicate_after_prevention = 0 + missing_routing = 0 + processed_count = 0 + + for entry in processed.values(): + if not isinstance(entry, dict) or not processed_entry_in_window(entry, args.since): + continue + processed_count += 1 + routing = frontmatter_from_state_entry(entry) + scope = str(routing.get("scope") or "not-recorded") + recurrence = str(routing.get("recurrence_check") or "not-recorded") + family = str(routing.get("lesson_family") or "not-recorded") + prevention_targets = routing.get("prevention_targets") or [] + by_scope[scope] = by_scope.get(scope, 0) + 1 + by_recurrence[recurrence] = by_recurrence.get(recurrence, 0) + 1 + by_family[family] = by_family.get(family, 0) + 1 + if recurrence == "duplicate-after-prevention": + duplicate_after_prevention += 1 + if ( + entry.get("status") == "processed" + and scope not in {"skill-detection", "needs-review"} + and not prevention_targets + ): + missing_routing += 1 + + payload = { + "learning_root": str(root), + "since": args.since or "", + "processed_count": processed_count, + "queue": { + "inbox": len(list((root / "inbox").glob("*.md"))), + "needs_review": len(list((root / "needs-review").glob("*.md"))), + }, + "by_scope": dict(sorted(by_scope.items())), + "by_recurrence": dict(sorted(by_recurrence.items())), + "lesson_families": dict(sorted(by_family.items())), + "missing_routing_count": missing_routing, + "duplicate_after_prevention_count": duplicate_after_prevention, + } + if args.format == "markdown": + print(summary_markdown(payload)) + else: + print(json.dumps(payload, indent=2)) + return 0 + + +def summary_markdown(payload: dict[str, Any]) -> str: + lines = [ + "# Agent Learning Run Summary", + "", + f"- Since: {payload['since'] or 'all-time'}", + f"- Processed: {payload['processed_count']}", + f"- Inbox: {payload['queue']['inbox']}", + f"- Needs review: {payload['queue']['needs_review']}", + f"- Missing routing: {payload['missing_routing_count']}", + f"- Duplicate after prevention: {payload['duplicate_after_prevention_count']}", + "", + "## Recurrence", + "", + ] + for key, count in payload["by_recurrence"].items(): + lines.append(f"- `{key}`: {count}") + lines.extend(["", "## Scope", ""]) + for key, count in payload["by_scope"].items(): + lines.append(f"- `{key}`: {count}") + lines.extend(["", "## Lesson Families", ""]) + for key, count in payload["lesson_families"].items(): + lines.append(f"- `{key}`: {count}") + return "\n".join(lines) + + +def command_create_template_draft(args: argparse.Namespace) -> int: + template_repo = Path(args.template_repo).expanduser().resolve() + if not (template_repo / "templates.yml").exists(): + raise ConfigError(f"Template repository is missing templates.yml: {template_repo}") + lesson_family = args.lesson_family.strip() + if not lesson_family: + raise ConfigError("--lesson-family is required.") + candidate_rule = args.candidate_rule.strip() + if not candidate_rule: + raise ConfigError("--candidate-rule is required.") + if privacy_findings(candidate_rule): + raise ConfigError("Candidate rule must be scrubbed before writing a template draft.") + if args.privacy_verdict != "clean": + raise ConfigError("Template draft creation requires --privacy-verdict clean.") + + metadata = { + "lesson_family": lesson_family, + "scope": "atom", + "prevention_targets": unique_values(args.prevention_target or []), + "detection_targets": unique_values(args.detection_target or []), + "template_upstream_status": "draft-created", + "routing_rationale": args.routing_rationale.strip(), + "recurrence_check": args.recurrence_check, + } + validate_routing_values(metadata) + if args.enforce_routing: + enforce_routing_contract(metadata) + + source_note = Path(args.source_note).name + refresh_triggers = unique_values(args.refresh_trigger or []) + body = learning_upstream_draft_markdown( + lesson_family=lesson_family, + source_note=source_note, + proposed_template=args.proposed_template, + candidate_rule=candidate_rule, + metadata=metadata, + privacy_verdict=args.privacy_verdict, + review_status=args.review_status, + refresh_triggers=refresh_triggers, + ) + if privacy_findings(body): + raise ConfigError("Draft body must be scrubbed before writing a template draft.") + draft_dir = template_repo / ".work" / "learning-upstream" + path = unique_path(draft_dir / f"{slugify(lesson_family)}.md") + atomic_write_text(path, body) + print(path) + return 0 + + +def learning_upstream_draft_markdown( + *, + lesson_family: str, + source_note: str, + proposed_template: str, + candidate_rule: str, + metadata: dict[str, Any], + privacy_verdict: str, + review_status: str, + refresh_triggers: list[str], +) -> str: + prevention_targets = metadata.get("prevention_targets") or [] + detection_targets = metadata.get("detection_targets") or [] + lines = [ + f"# Learning Upstream Draft: {lesson_family}", + "", + "## Handoff", + "", + f"- Lesson family: `{lesson_family}`", + f"- Source note: `{source_note}`", + f"- Proposed template: `{proposed_template}`", + f"- Review status: `{review_status}`", + f"- Privacy verdict: `{privacy_verdict}`", + f"- Recurrence check: `{metadata.get('recurrence_check')}`", + "", + "## Candidate Rule", + "", + candidate_rule, + "", + "## Prevention Targets", + "", + ] + lines.extend(f"- `{item}`" for item in prevention_targets) + if not prevention_targets: + lines.append("- `none-recorded`") + lines.extend(["", "## Detection Targets", ""]) + lines.extend(f"- `{item}`" for item in detection_targets) + if not detection_targets: + lines.append("- `none-recorded`") + lines.extend(["", "## Refresh Triggers", ""]) + lines.extend(f"- `{item}`" for item in refresh_triggers) + if not refresh_triggers: + lines.append("- `none-recorded`") + lines.extend( + [ + "", + "## Routing Rationale", + "", + metadata.get("routing_rationale") or "Not recorded.", + "", + "## Review Decision", + "", + "- [ ] Promote to curated atom/template", + "- [ ] Defer for more evidence", + "- [ ] Reject as not reusable", + "", + ] + ) + return "\n".join(lines) + + def pending_review_notes(root: Path) -> list[dict[str, str]]: notes: list[dict[str, str]] = [] for path in sorted((root / "needs-review").glob("*.md")): @@ -778,6 +1415,18 @@ def command_hook_review_skills(args: argparse.Namespace) -> int: return 0 +def add_routing_arguments(parser: argparse.ArgumentParser, *, include_enforce: bool = False) -> None: + parser.add_argument("--lesson-family") + parser.add_argument("--scope", choices=ROUTING_SCOPES) + parser.add_argument("--prevention-target", action="append") + parser.add_argument("--detection-target", action="append") + parser.add_argument("--template-upstream-status", choices=TEMPLATE_UPSTREAM_STATUSES) + parser.add_argument("--routing-rationale") + parser.add_argument("--recurrence-check", choices=RECURRENCE_CHECKS) + if include_enforce: + parser.add_argument("--enforce-routing", action="store_true") + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser(description=__doc__) parser.add_argument("--config", type=Path, default=DEFAULT_CONFIG) @@ -799,6 +1448,7 @@ def build_parser() -> argparse.ArgumentParser: record.add_argument("--verification") record.add_argument("--future-detection") record.add_argument("--future-prevention") + add_routing_arguments(record) record.set_defaults(func=command_record) prepare = sub.add_parser("prepare-run") @@ -809,13 +1459,36 @@ def build_parser() -> argparse.ArgumentParser: finalize.add_argument("--status", choices=["processed", "needs-review", "rejected"], required=True) finalize.add_argument("--rationale", required=True) finalize.add_argument("--run-id", default="") + add_routing_arguments(finalize, include_enforce=True) finalize.set_defaults(func=command_finalize_note) report = sub.add_parser("write-report") report.add_argument("--run-id", default="") report.add_argument("--summary", required=True) + add_routing_arguments(report, include_enforce=True) report.set_defaults(func=command_write_report) + summary = sub.add_parser("summarize-runs") + summary.add_argument("--since", default="") + summary.add_argument("--format", choices=["json", "markdown"], default="json") + summary.set_defaults(func=command_summarize_runs) + + draft = sub.add_parser("create-template-draft") + draft.add_argument("--template-repo", required=True) + draft.add_argument("--lesson-family", required=True) + draft.add_argument("--source-note", required=True) + draft.add_argument("--proposed-template", required=True) + draft.add_argument("--candidate-rule", required=True) + draft.add_argument("--prevention-target", action="append") + draft.add_argument("--detection-target", action="append") + draft.add_argument("--routing-rationale", default="") + draft.add_argument("--recurrence-check", choices=RECURRENCE_CHECKS, default="new") + draft.add_argument("--privacy-verdict", choices=["clean", "needs-scrub", "blocked"], required=True) + draft.add_argument("--review-status", default="draft") + draft.add_argument("--refresh-trigger", action="append") + draft.add_argument("--enforce-routing", action="store_true") + draft.set_defaults(func=command_create_template_draft) + notify = sub.add_parser("notify") notify.add_argument("--send-msmtp", action="store_true") notify.set_defaults(func=command_notify) diff --git a/skills/consolidate-agent-learnings/SKILL.md b/skills/consolidate-agent-learnings/SKILL.md index 88052e0..6dc9082 100644 --- a/skills/consolidate-agent-learnings/SKILL.md +++ b/skills/consolidate-agent-learnings/SKILL.md @@ -28,47 +28,78 @@ the Obsidian agent-learning inbox and reviewed `needs-review` notes. python3 "${repo}/scripts/agent_learning.py" init-store ``` -2. Prepare the run and inspect the returned JSON: +1. Prepare the run and inspect the returned JSON: ```bash repo="/path/to/agent-learning-system" python3 "${repo}/scripts/agent_learning.py" prepare-run ``` -3. Process `inbox` notes. +1. Process `inbox` notes. - Promote clear, safe, reusable lessons automatically. - Move uncertain notes to `needs-review/` with the review checkbox block. - Move too-local or already-covered notes to `processed/YYYY/MM/` with a no-op rationale. -4. Process `needs-review` notes only when one checkbox is selected. +1. Process `needs-review` notes only when one checkbox is selected. - `Approve`: promote the proposed rule as written. - `Retry`: promote after considering the user's edited text. - `Reject`: archive as rejected without promotion. - No checkbox: leave pending. - Multiple checkboxes: leave pending and report ambiguity. -5. Promotion targets are bounded: +1. Promotion targets are bounded: - `$HOME/AGENTS.md` for truly general behavior. - The smallest relevant reusable review or remediation skill. - The touched project's `AGENTS.md` when the lesson is project-local. - The existing `update-agents-file-templates` workflow after central or project instruction changes when template promotion is useful. -6. Finalize each note with the helper: +1. Finalize each note with the helper: ```bash repo="/path/to/agent-learning-system" python3 "${repo}/scripts/agent_learning.py" finalize-note \ --file "/path/to/note.md" \ --status processed \ - --rationale "Promoted to global AGENTS.md and pre-pr-review-gate." + --rationale "Promoted to docs atom draft and docs release review skill." \ + --lesson-family "markdown-validation-contract" \ + --scope atom \ + --prevention-target "atom:docs" \ + --detection-target "skill:pre-pr-review-docs-release" \ + --template-upstream-status draft-created \ + --routing-rationale "Documentation agents load the docs atom before editing Markdown." \ + --recurrence-check new \ + --enforce-routing ``` Use `--status needs-review` for pending review and `--status rejected` for a - reviewed rejection. + reviewed rejection. For detection-only lessons, use + `--scope skill-detection --detection-target ... --enforce-routing`. Omit + `--enforce-routing` only for no-op, rejected, or pending-review outcomes. -7. Write a report: +1. When a reusable prevention rule belongs in the template repository, create + an ignored draft handoff. Do not edit curated templates from this skill + unless the user explicitly asks for template apply: + + ```bash + repo="/path/to/agent-learning-system" + python3 "${repo}/scripts/agent_learning.py" create-template-draft \ + --template-repo "/path/to/agents-file-templates-and-skills" \ + --lesson-family "markdown-validation-contract" \ + --source-note "/path/to/note.md" \ + --proposed-template "docs" \ + --candidate-rule "Run markdownlint before completing Markdown documentation changes." \ + --prevention-target "atom:docs" \ + --routing-rationale "Documentation agents load the docs atom before editing Markdown." \ + --privacy-verdict clean \ + --enforce-routing + ``` + + The template repository owns reviewed apply and generated-project refresh + reports through its `update-agents-file-templates` workflow. + +1. Write a report: ```bash repo="/path/to/agent-learning-system" @@ -77,7 +108,16 @@ the Obsidian agent-learning inbox and reviewed `needs-review` notes. --summary "Processed 3 notes; promoted 1; moved 2 to review." ``` -8. Validate every changed Markdown file with: +1. Summarize recurrence and routing gaps: + + ```bash + repo="/path/to/agent-learning-system" + python3 "${repo}/scripts/agent_learning.py" summarize-runs \ + --since "YYYY-MM-DD" \ + --format markdown + ``` + +1. Validate every changed Markdown file with: ```bash markdownlint --config ~/.markdownlint.json @@ -85,7 +125,7 @@ the Obsidian agent-learning inbox and reviewed `needs-review` notes. Also run `shellcheck --enable=all` for changed shell files. -9. Run a conflict/duplication audit on the promotion targets: +1. Run a conflict/duplication audit on the promotion targets: ```bash repo="/path/to/agent-learning-system" diff --git a/tests/test_agent_learning.py b/tests/test_agent_learning.py index e230508..02a8683 100644 --- a/tests/test_agent_learning.py +++ b/tests/test_agent_learning.py @@ -1,9 +1,12 @@ from __future__ import annotations +import importlib.util import json +import os import subprocess import sys import tempfile +import time import unittest from pathlib import Path @@ -12,19 +15,32 @@ SCRIPT = ROOT / "scripts" / "agent_learning.py" +def load_agent_learning_module(): + spec = importlib.util.spec_from_file_location("agent_learning_under_test", SCRIPT) + if spec is None or spec.loader is None: + raise RuntimeError("Unable to load agent_learning module for tests.") + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + class AgentLearningTests(unittest.TestCase): def run_helper( self, config: Path, *args: str, check: bool = True, + env: dict[str, str] | None = None, ) -> subprocess.CompletedProcess[str]: + clean_env = {key: value for key, value in os.environ.items() if not key.startswith("AGENT_LEARNING_")} + clean_env.update(env or {}) return subprocess.run( [sys.executable, str(SCRIPT), "--config", str(config), *args], text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, check=check, + env=clean_env, ) def write_config(self, directory: Path) -> Path: @@ -93,6 +109,9 @@ def test_record_creates_inbox_note(self) -> None: text = note.read_text(encoding="utf-8") self.assertIn("status: \"inbox\"", text) self.assertIn("## Future Prevention", text) + self.assertIn("lesson_family: \"config-drift-broke-validation\"", text) + self.assertIn("prevention_targets: \"[]\"", text) + self.assertIn("template_upstream_status: \"not-applicable\"", text) def test_new_config_uses_agent_learning_dir_as_base_directory(self) -> None: with tempfile.TemporaryDirectory() as raw: @@ -165,6 +184,424 @@ def test_finalize_note_is_idempotent_in_state(self) -> None: self.assertEqual(state["processed"]["abc"]["status"], "processed") self.assertIn("/processed/", state["processed"]["abc"]["path"]) + def test_finalize_note_records_routing_metadata(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "done.md" + note.write_text( + """--- +id: "abc" +status: "inbox" +--- +# Done +""", + encoding="utf-8", + ) + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Promoted to reusable template atom.", + "--lesson-family", + "markdown-validation-contract", + "--scope", + "atom", + "--prevention-target", + "atom:documentation-validation", + "--detection-target", + "skill:pre-pr-review-docs-release", + "--template-upstream-status", + "draft-created", + "--routing-rationale", + "Documentation agents load the atom before editing Markdown.", + "--recurrence-check", + "new", + "--enforce-routing", + ) + moved = Path(result.stdout.strip()) + text = moved.read_text(encoding="utf-8") + self.assertIn("lesson_family: \"markdown-validation-contract\"", text) + self.assertIn("prevention_targets: \"[\\\"atom:documentation-validation\\\"]\"", text) + state = json.loads((root / "state" / "processed.json").read_text(encoding="utf-8")) + self.assertEqual(state["processed"]["abc"]["scope"], "atom") + self.assertEqual(state["processed"]["abc"]["template_upstream_status"], "draft-created") + + def test_finalize_note_enforces_prevention_targets_when_requested(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "done.md" + original = """--- +id: "abc" +status: "inbox" +--- +# Done +""" + note.write_text(original, encoding="utf-8") + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Missing prevention target.", + "--lesson-family", + "missing-target", + "--scope", + "global", + "--routing-rationale", + "Global agents load this before acting.", + "--enforce-routing", + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("requires --prevention-target", result.stderr) + self.assertEqual(note.read_text(encoding="utf-8"), original) + + def test_finalize_note_treats_empty_json_prevention_targets_as_missing(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "empty-targets.md" + original = """--- +id: "empty-targets" +status: "inbox" +lesson_family: "empty-targets" +scope: "atom" +prevention_targets: "[]" +detection_targets: "[]" +routing_rationale: "Atom-scope lessons require a concrete prevention target." +--- +# Empty Targets +""" + note.write_text(original, encoding="utf-8") + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Empty JSON list is not a target.", + "--enforce-routing", + check=False, + ) + + self.assertEqual(result.returncode, 2) + self.assertIn("requires --prevention-target", result.stderr) + self.assertEqual(note.read_text(encoding="utf-8"), original) + + def test_finalize_note_enforces_routing_rationale_when_requested(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "done.md" + original = """--- +id: "abc" +status: "inbox" +--- +# Done +""" + note.write_text(original, encoding="utf-8") + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Missing routing rationale.", + "--lesson-family", + "missing-rationale", + "--scope", + "atom", + "--prevention-target", + "atom:docs", + "--enforce-routing", + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("requires --routing-rationale", result.stderr) + self.assertEqual(note.read_text(encoding="utf-8"), original) + + def test_finalize_note_enforces_detection_target_for_detection_scope(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "detect.md" + note.write_text( + """--- +id: "detect" +status: "inbox" +--- +# Detect +""", + encoding="utf-8", + ) + missing = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Detection-only route.", + "--lesson-family", + "detection-only", + "--scope", + "skill-detection", + "--routing-rationale", + "Review skill catches the issue before final response.", + "--enforce-routing", + check=False, + ) + self.assertEqual(missing.returncode, 2) + self.assertIn("requires --detection-target", missing.stderr) + + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Detection-only route.", + "--lesson-family", + "detection-only", + "--scope", + "skill-detection", + "--detection-target", + "skill:pre-pr-review-docs-release", + "--routing-rationale", + "Review skill catches the issue before final response.", + "--enforce-routing", + ) + moved = Path(result.stdout.strip()) + self.assertTrue(moved.exists()) + + def test_concurrent_finalize_note_preserves_state_entries(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + notes = [] + for note_id in ["one", "two"]: + note = root / "inbox" / f"{note_id}.md" + note.write_text( + f"""--- +id: "{note_id}" +status: "inbox" +--- +# {note_id} +""", + encoding="utf-8", + ) + notes.append(note) + + clean_env = {key: value for key, value in os.environ.items() if not key.startswith("AGENT_LEARNING_")} + processes = [ + subprocess.Popen( + [ + sys.executable, + str(SCRIPT), + "--config", + str(config), + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + f"Processed {note.stem}.", + ], + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + env=clean_env, + ) + for note in notes + ] + for process in processes: + stdout, stderr = process.communicate(timeout=10) + self.assertEqual(process.returncode, 0, stderr or stdout) + + state = json.loads((root / "state" / "processed.json").read_text(encoding="utf-8")) + self.assertIn("one", state["processed"]) + self.assertIn("two", state["processed"]) + + def test_finalize_note_lock_failure_keeps_note_in_queue(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "locked.md" + original = """--- +id: "locked" +status: "inbox" +--- +# Locked +""" + note.write_text(original, encoding="utf-8") + (root / "state" / ".processed.lock").mkdir() + + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "State lock is held.", + check=False, + env={"AGENT_LEARNING_STATE_LOCK_TIMEOUT_SECONDS": "0.05"}, + ) + + self.assertEqual(result.returncode, 2) + self.assertIn("Timed out waiting for state lock", result.stderr) + self.assertEqual(note.read_text(encoding="utf-8"), original) + self.assertFalse(list((root / "processed").glob("**/*.md"))) + + def test_finalize_note_does_not_reclaim_owned_state_lock_by_age(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "old-lock.md" + original = """--- +id: "old-lock" +status: "inbox" +--- +# Old Lock +""" + note.write_text(original, encoding="utf-8") + lock_dir = root / "state" / ".processed.lock" + lock_dir.write_text(json.dumps({"pid": os.getpid()}) + "\n", encoding="utf-8") + old_time = time.time() - 3600 + os.utime(lock_dir, (old_time, old_time)) + + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "State lock is held.", + check=False, + env={"AGENT_LEARNING_STATE_LOCK_TIMEOUT_SECONDS": "0.05"}, + ) + + self.assertEqual(result.returncode, 2) + self.assertIn("Timed out waiting for state lock", result.stderr) + self.assertTrue(lock_dir.exists()) + self.assertEqual(note.read_text(encoding="utf-8"), original) + self.assertFalse(list((root / "processed").glob("**/*.md"))) + + def test_finalize_note_reclaims_old_ownerless_state_lock(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "ownerless.md" + note.write_text( + """--- +id: "ownerless" +status: "inbox" +--- +# Ownerless +""", + encoding="utf-8", + ) + lock_dir = root / "state" / ".processed.lock" + lock_dir.mkdir() + old_time = time.time() - 3600 + os.utime(lock_dir, (old_time, old_time)) + + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Ownerless lock recovered.", + "--lesson-family", + "lock-recovery", + "--scope", + "project-local", + "--prevention-target", + "project-local:agent-learning-system", + "--routing-rationale", + "Project automation preserves queue state.", + "--enforce-routing", + env={"AGENT_LEARNING_STATE_LOCK_OWNERLESS_GRACE_SECONDS": "0.01"}, + ) + + moved = Path(result.stdout.strip()) + self.assertTrue(moved.exists()) + self.assertFalse(lock_dir.exists()) + + def test_state_lock_falls_back_when_hard_links_are_unavailable(self) -> None: + agent_learning = load_agent_learning_module() + with tempfile.TemporaryDirectory() as raw: + root = Path(raw) + original_link = agent_learning.os.link + + def fail_link(_source: Path, _dest: Path) -> None: + raise OSError("hard links unavailable") + + agent_learning.os.link = fail_link + try: + with agent_learning.state_update_lock(root): + lock_path = root / "state" / ".processed.lock" + self.assertTrue(lock_path.exists()) + self.assertEqual(json.loads(lock_path.read_text(encoding="utf-8"))["pid"], os.getpid()) + self.assertFalse((root / "state" / ".processed.lock").exists()) + finally: + agent_learning.os.link = original_link + + def test_finalize_note_malformed_state_keeps_note_in_queue(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "bad-state.md" + original = """--- +id: "bad-state" +status: "inbox" +--- +# Bad State +""" + note.write_text(original, encoding="utf-8") + state_path = root / "state" / "processed.json" + state_path.write_text("{", encoding="utf-8") + + result = self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Malformed state blocks processing.", + check=False, + ) + + self.assertEqual(result.returncode, 2) + self.assertIn("State file is malformed", result.stderr) + self.assertEqual(note.read_text(encoding="utf-8"), original) + self.assertFalse(list((root / "processed").glob("**/*.md"))) + def test_finalize_note_rejects_file_outside_learning_store(self) -> None: with tempfile.TemporaryDirectory() as raw: directory = Path(raw) @@ -223,6 +660,240 @@ def test_finalize_note_can_keep_review_note_in_place(self) -> None: self.assertTrue(note.exists()) self.assertFalse((root / "needs-review" / "review-2.md").exists()) + def test_write_report_can_include_routing_decision(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + result = self.run_helper( + config, + "write-report", + "--run-id", + "run-1", + "--summary", + "Processed one reusable lesson.", + "--lesson-family", + "markdown-validation-contract", + "--scope", + "atom", + "--prevention-target", + "atom:documentation-validation", + "--template-upstream-status", + "draft-created", + "--routing-rationale", + "Documentation agents load the atom before editing Markdown.", + "--recurrence-check", + "new", + "--enforce-routing", + ) + report = Path(result.stdout.strip()) + self.assertEqual(report.parent, root / "reports") + text = report.read_text(encoding="utf-8") + self.assertIn("## Routing Decision", text) + self.assertIn("Lesson family: `markdown-validation-contract`", text) + + def test_summarize_runs_counts_recurrence(self) -> None: + with tempfile.TemporaryDirectory() as raw: + config = self.write_config(Path(raw)) + root = Path(self.run_helper(config, "init-store").stdout.strip()) + note = root / "inbox" / "done.md" + note.write_text( + """--- +id: "abc" +status: "inbox" +--- +# Done +""", + encoding="utf-8", + ) + self.run_helper( + config, + "finalize-note", + "--file", + str(note), + "--status", + "processed", + "--rationale", + "Repeated after prevention.", + "--lesson-family", + "markdown-validation-contract", + "--scope", + "atom", + "--prevention-target", + "atom:documentation-validation", + "--recurrence-check", + "duplicate-after-prevention", + ) + rejected = root / "inbox" / "rejected.md" + rejected.write_text( + """--- +id: "rejected" +status: "inbox" +--- +# Rejected +""", + encoding="utf-8", + ) + self.run_helper( + config, + "finalize-note", + "--file", + str(rejected), + "--status", + "rejected", + "--rationale", + "Not reusable.", + ) + result = self.run_helper(config, "summarize-runs", "--format", "json") + payload = json.loads(result.stdout) + self.assertEqual(payload["processed_count"], 2) + self.assertEqual(payload["duplicate_after_prevention_count"], 1) + self.assertEqual(payload["missing_routing_count"], 0) + self.assertEqual(payload["lesson_families"]["markdown-validation-contract"], 1) + + datetime_since = self.run_helper( + config, + "summarize-runs", + "--since", + "2026-06-01T12:00:00", + check=False, + ) + self.assertEqual(datetime_since.returncode, 2) + self.assertIn("YYYY-MM-DD", datetime_since.stderr) + + def test_create_template_draft_uses_source_note_filename_only(self) -> None: + with tempfile.TemporaryDirectory() as raw: + directory = Path(raw) + config = self.write_config(directory) + template_repo = directory / "templates" + template_repo.mkdir() + (template_repo / "templates.yml").write_text("templates: []\n", encoding="utf-8") + source_note = directory / "private" / "note.md" + result = self.run_helper( + config, + "create-template-draft", + "--template-repo", + str(template_repo), + "--lesson-family", + "markdown-validation-contract", + "--source-note", + str(source_note), + "--proposed-template", + "documentation-validation", + "--candidate-rule", + "Run markdownlint before completing Markdown documentation changes.", + "--prevention-target", + "atom:documentation-validation", + "--routing-rationale", + "Documentation agents load the atom before editing Markdown.", + "--privacy-verdict", + "clean", + "--enforce-routing", + ) + draft = Path(result.stdout.strip()) + text = draft.read_text(encoding="utf-8") + self.assertIn("Source note: `note.md`", text) + self.assertNotIn(str(source_note), text) + + def test_create_template_draft_requires_clean_privacy_verdict(self) -> None: + with tempfile.TemporaryDirectory() as raw: + directory = Path(raw) + config = self.write_config(directory) + template_repo = directory / "templates" + template_repo.mkdir() + (template_repo / "templates.yml").write_text("templates: []\n", encoding="utf-8") + result = self.run_helper( + config, + "create-template-draft", + "--template-repo", + str(template_repo), + "--lesson-family", + "blocked", + "--source-note", + "note.md", + "--proposed-template", + "docs", + "--candidate-rule", + "Run markdownlint before completing Markdown documentation changes.", + "--prevention-target", + "atom:docs", + "--privacy-verdict", + "blocked", + "--routing-rationale", + "Docs agents load this atom before editing documentation.", + "--enforce-routing", + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("requires --privacy-verdict clean", result.stderr) + self.assertFalse((template_repo / ".work").exists()) + + def test_create_template_draft_rejects_private_candidate_rule(self) -> None: + with tempfile.TemporaryDirectory() as raw: + directory = Path(raw) + config = self.write_config(directory) + template_repo = directory / "templates" + template_repo.mkdir() + (template_repo / "templates.yml").write_text("templates: []\n", encoding="utf-8") + result = self.run_helper( + config, + "create-template-draft", + "--template-repo", + str(template_repo), + "--lesson-family", + "private-path", + "--source-note", + "note.md", + "--proposed-template", + "docs", + "--candidate-rule", + "Do not hard-code 10.0.0.1 in docs.", + "--prevention-target", + "atom:docs", + "--privacy-verdict", + "needs-scrub", + "--enforce-routing", + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("must be scrubbed", result.stderr) + + def test_create_template_draft_rejects_linux_local_candidate_rule(self) -> None: + candidate_rules = [ + "Do not write /home/alice/vault into shared handoffs.", + "Do not write /workspace/private-repo into shared handoffs.", + ] + for candidate_rule in candidate_rules: + with self.subTest(candidate_rule=candidate_rule): + with tempfile.TemporaryDirectory() as raw: + directory = Path(raw) + config = self.write_config(directory) + template_repo = directory / "templates" + template_repo.mkdir() + (template_repo / "templates.yml").write_text("templates: []\n", encoding="utf-8") + result = self.run_helper( + config, + "create-template-draft", + "--template-repo", + str(template_repo), + "--lesson-family", + "private-path", + "--source-note", + "note.md", + "--proposed-template", + "docs", + "--candidate-rule", + candidate_rule, + "--prevention-target", + "atom:docs", + "--privacy-verdict", + "clean", + "--enforce-routing", + check=False, + ) + self.assertEqual(result.returncode, 2) + self.assertIn("must be scrubbed", result.stderr) + self.assertFalse((template_repo / ".work").exists()) + def test_notify_refuses_placeholder_email_for_msmtp(self) -> None: with tempfile.TemporaryDirectory() as raw: config = self.write_config(Path(raw))