Skip to content

docs: add missing documentation sections to skills and references #174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
maystudios merged 1 commit into from
Mar 25, 2026
Merged

docs: add missing documentation sections to skills and references #174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions templates/references/self-improvement.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,10 @@ at the end of every session in a MaxsimCLI project. Each entry records:
- Which tasks were attempted repeatedly without a commit (likely failed).
- Long-term trends across many sessions.

### MEMORY.md Size Limit

Keep MEMORY.md under 200 lines (the Claude Code context loading limit). The `maxsim-capture-learnings` Stop hook enforces this by pruning at 180 lines, leaving headroom. When the file approaches the limit, the oldest entries are removed first. Each entry should be concise (3-5 lines) to maximize the number of sessions that fit.
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This says “Each entry should be concise (3–5 lines)”, but the maxsim-capture-learnings Stop hook’s actual format includes a session header plus a bullet per commit (and optional pattern), so entries can easily exceed 5 lines even in normal usage. Suggest adjusting this guidance to match the real output (e.g., encourage keeping commit lists short / summarizing patterns), or clarify that the hook-generated entry may be longer than 3–5 lines.

Suggested change
Keep MEMORY.md under 200 lines (the Claude Code context loading limit). The `maxsim-capture-learnings` Stop hook enforces this by pruning at 180 lines, leaving headroom. When the file approaches the limit, the oldest entries are removed first. Each entry should be concise (3-5 lines) to maximize the number of sessions that fit.
Keep MEMORY.md under 200 lines (the Claude Code context loading limit). The `maxsim-capture-learnings` Stop hook enforces this by pruning at 180 lines, leaving headroom. When the file approaches the limit, the oldest entries are removed first. Aim to keep each entry concise (around 3–5 lines) by keeping commit lists short or summarizing patterns, but note that hook-generated entries may be longer when there are many commits.

Copilot uses AI. Check for mistakes.

---

## 3. Results Tracking
Expand Down
9 changes: 5 additions & 4 deletions templates/skills/autoresearch/references/loop-protocol.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,10 +93,11 @@ If verification exceeds 2x normal time, kill and treat as crash.

Some metrics are inherently noisy (benchmark times, ML accuracy). Strategies:

- **Multi-run verification:** Run verify N times, use the median.
- **Minimum improvement threshold:** Ignore improvements smaller than the noise floor.
- **Confirmation run:** Re-verify before making a final keep decision.
- **Environment pinning:** Pin random seeds, use deterministic test ordering, flush caches.
- **For improvements of 1–5%:** Run the verify command 3 times and use the median result.
- **For improvements >5%:** Run the verify command 5 times and use the median result.
- **Minimum improvement threshold:** Ignore improvements smaller than the noise floor (typically 0.5% for benchmarks).
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The multi-run guidance covers improvements of 1–5% and >5%, but doesn’t say what to do for small-but-real changes between the noise floor and 1% (e.g., 0.6% when noise floor is 0.5%). Consider adding an explicit rule for 0.5–1% (or “<1%”) improvements (e.g., treat as noise unless confirmed with 5 runs, or require an extra confirmation run) so the protocol is complete.

Suggested change
- **Minimum improvement threshold:** Ignore improvements smaller than the noise floor (typically 0.5% for benchmarks).
- **For improvements between the noise floor and 1% (e.g., 0.5–1%):** Treat as noise by default. Only consider keeping if you run at least 5 verification runs and the median improvement remains above the noise floor.
- **Minimum improvement threshold:** Ignore improvements smaller than the noise floor (typically 0.5% for benchmarks), and apply the above rule for any borderline improvements between the noise floor and 1%.

Copilot uses AI. Check for mistakes.
- **Confirmation run:** After accepting an improvement, re-verify once more before making the final keep decision.
- **Environment pinning:** Pin random seeds, use deterministic test ordering, flush caches between runs.

## Phase 5.5: Guard (Regression Check)

Expand Down
39 changes: 39 additions & 0 deletions templates/skills/verification/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,3 +167,42 @@ Do not attempt a 4th run without user acknowledgment and revised instructions.
| Skipping Gate 4 after Gate 3 passes | Declaring done without regression check | Gate 3 and Gate 4 are both required; neither is optional |
| Conflating "no errors" with "correct output" | Exit code 0 but wrong behavior | Evidence must show correct output, not just absence of error |
| Writing evidence after the fact | Constructing output from memory | Run the command, capture the output, paste it verbatim |

---

## 5-Step Verification Process

When verification fails, follow this structured process:

1. **Run the check command one final time** — capture fresh output as evidence
2. **Construct diagnostic summary** — compare spec expectations vs actual output
3. **Identify root cause** — is it a spec problem, environment problem, or implementation problem?
4. **Propose next step** — rewrite spec, fix environment, reduce scope, or escalate
5. **Escalate if unresolved** — create a diagnostic GitHub Issue with all evidence

---

## GitHub Issue Escalation

When a task fails verification after 3 attempts, escalate by creating (or commenting on) a GitHub Issue:
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section largely duplicates the earlier “After 3 Failures” escalation list but changes the escalation target to a GitHub Issue. Because similar retry/escalation logic in templates/workflows/execute.md triggers a diagnostic issue only after 4 total attempts, it’s easy for these docs to drift or conflict. Suggest consolidating into a single canonical escalation section (or explicitly scoping it: single-task vs /maxsim:execute phase execution) and aligning the attempt threshold wording.

Suggested change
When a task fails verification after 3 attempts, escalate by creating (or commenting on) a GitHub Issue:
When verification still fails after the final allowed attempt (as defined by the workflow configuration), escalate by creating (or commenting on) a GitHub Issue:

Copilot uses AI. Check for mistakes.

1. **Original task spec** — quoted from the plan comment
2. **What was attempted** — brief factual summary of each attempt
3. **The specific gate that failed** — exact error output from each run
4. **Root cause analysis** — spec/environment/implementation classification
5. **Proposed next step** — rewrite spec, fix environment, reduce scope, or request user input
Comment on lines +189 to +193
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this escalation checklist, “quoted from the plan comment” is ambiguous in contexts that don’t have a distinct plan comment (e.g., quick tasks or non-GitHub flows). Consider wording this as “quoted from the plan comment (if present) or the issue/task description” so the instruction is actionable in all cases.

Copilot uses AI. Check for mistakes.

Label the issue with `type:bug` and `maxsim:auto`.

---

## Fresh Executor Context

Each retry attempt MUST use a fresh executor agent:

- Do NOT reuse the previous executor (spawn a new one)
- Provide the full task spec (do not assume prior context carries over)
- Include the diagnostic summary from the failed run
- Include revised instructions based on root cause analysis

Treat each fresh executor as a cold start. Do NOT reference or build upon any previous attempt's reasoning or partial work.
Comment on lines +201 to +208
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

“Each retry attempt MUST use a fresh executor agent” conflicts with the earlier per-attempt rules in this same file (Attempt 2 is described as “Fresh context” while Attempt 3 is “Fresh agent context”). Either update the Attempt 2/3 bullets to match this requirement, or narrow this section to the specific attempts/workflows where a new executor is actually spawned, so readers don’t get contradictory instructions.

Suggested change
Each retry attempt MUST use a fresh executor agent:
- Do NOT reuse the previous executor (spawn a new one)
- Provide the full task spec (do not assume prior context carries over)
- Include the diagnostic summary from the failed run
- Include revised instructions based on root cause analysis
Treat each fresh executor as a cold start. Do NOT reference or build upon any previous attempt's reasoning or partial work.
For retries that spawn a new executor agent, treat that agent as a fresh executor:
- When you spawn a new executor, do NOT reuse any previous executor state
- Provide the full task spec (do not assume prior context carries over)
- Include the diagnostic summary from the failed run
- Include revised instructions based on root cause analysis
Treat each fresh executor as a cold start. Do NOT reference or build upon any previous attempt's reasoning or partial work within that executor.

Copilot uses AI. Check for mistakes.
Loading