diff --git a/README.md b/README.md index c5dd4c4..a1c60df 100644 --- a/README.md +++ b/README.md @@ -43,6 +43,44 @@ pnpm test:e2e # Playwright end-to-end tests pnpm benchmark:solve ``` +## Solver Analysis CLI + +`pnpm benchmark:solve` is the unified batch solver-analysis command. It scans +`dataset/public` and local `dataset/private` manifests by default, runs each +puzzle once, and writes timestamped text and TSV analysis artifacts under +`benchmark-results///`. + +Target specific datasets or puzzle IDs while developing: + +```bash +pnpm benchmark:solve --dataset dataset/public/masyu.json --ids masyu-20x20-8268975,masyu-25x25-988309 +``` + +Useful options: + +```text +--dataset Repeatable manifest path +--ids Repeatable puzzle-ID filter +--max-steps Default: 2000 +--timeout-ms Per-puzzle timeout, default: 60000 +--telemetry off | summary, default: summary +--format text | tsv | both | json, default: both +--out-dir Default: benchmark-results +``` + +The default `both` mode prints and saves `summary.txt`, then writes +`puzzles.tsv`, `rule-attempts.tsv`, and `strong-inference.tsv`. These long-form +tables are intended for spreadsheets, pivot tables, and scripts. With +`--telemetry off`, only `puzzles.tsv` is generated. JSON is reserved for +complete debugging output and is written only when explicitly requested with +`--format json`. + +Summary telemetry reports rule attempts, misses, durations, and supported +strong-inference work. Reports also list strong-inference rules whose detailed +telemetry interface has not been implemented, so empty Strong data is not +mistaken for a rule that never ran. Generated artifacts have no compatibility +guarantee. + ## App Surfaces - **Solver** (`/`): import a puzzle, run one deduction at a time, jump through the replay timeline, inspect explanations and strong-inference branches, view stats, and export supported states. diff --git a/dataset/public/masyu.json b/dataset/public/masyu.json index 4a17353..33164f8 100644 --- a/dataset/public/masyu.json +++ b/dataset/public/masyu.json @@ -498,6 +498,15 @@ "height": 15, "tags": ["easy", "Medium"], "source": "example.txt" + }, + { + "id": "masyu-25x25-1202688", + "puzzleType": "masyu", + "sourceUrl": "https://puzz.link/p?mashu/25/25/00k900o0j0la9a1120000000312la7b1i9036g6ii900110l303600i42102c953163101f179344169f01309119i300aa909221502f080c3f3079m20163l39f10a03p00300400163fi593j031039kj1i01600000c713b0e09190e23632f6099b040c0f999c020909p00", + "width": 25, + "height": 25, + "tags": ["hard", "Large"], + "source": "example.txt" } ] } diff --git a/docs/PROJECT_GUIDE_EN.md b/docs/PROJECT_GUIDE_EN.md index f66ebff..377694a 100644 --- a/docs/PROJECT_GUIDE_EN.md +++ b/docs/PROJECT_GUIDE_EN.md @@ -142,15 +142,45 @@ Run: pnpm benchmark:solve ``` -The benchmark runner scans public/private manifests, runs each puzzle with the default plugin rule order, and writes reports to `benchmark-results/.report.json`. +The unified solver-analysis runner scans public/private manifests, runs each +puzzle once with the default plugin rule order, and writes timestamped analysis +artifacts under `benchmark-results///`. Default benchmark settings: - `maxSteps = 2000` - `timeoutMs = 60000` - `ruleProfile = "default"` +- `telemetry = "summary"` -Reports include per-puzzle status, step count, duration, terminal completion report, `ruleUsage`, and compact `ruleSteps`. Full `steps` are intentionally omitted to keep reports small. +Targeted development runs can select explicit manifests and puzzle IDs: + +```bash +pnpm benchmark:solve --dataset dataset/public/masyu.json --ids masyu-20x20-8268975,masyu-25x25-988309 +``` + +CLI options: + +- `--dataset ` is repeatable; without it, public/private manifests are scanned. +- `--ids ` is repeatable and filters selected manifests. +- `--max-steps ` and `--timeout-ms ` set per-puzzle limits. +- `--telemetry off | summary` controls observer collectors. +- `--format text | tsv | both | json` controls generated artifacts; default + `both` means text plus TSV and never includes JSON. +- `--out-dir ` changes the artifact output directory. + +The default run prints and saves `summary.txt`, plus three long-form tables: +`puzzles.tsv` for one-row-per-puzzle outcomes, `rule-attempts.tsv` for per-rule +cost and hit behavior, and `strong-inference.tsv` for detailed Strong work. +When telemetry is off, only the puzzle table is generated. JSON retains the +complete schema-v2 report for urgent debugging and is written only by explicit +`--format json`. + +Rule attempt telemetry includes misses and the final no-hit scan. Strong +inference telemetry includes explicit plugin-declared coverage, so unsupported +collection interfaces are distinguishable from a run that produced no Strong +events. Generated artifacts are development outputs with no compatibility +guarantee. Full replay steps and rule-step locations are intentionally omitted. ## 8. AI Agent Quick Start @@ -168,6 +198,7 @@ For targeted work: - Slitherlink rules: start at `src/domain/rules/slither/rules.ts`. - Masyu rules: start at `docs/MASYU_AGENT_BRIEF.md`, then inspect `src/domain/rules/masyu/rules.ts`. +- Rule documentation: follow `docs/RULE_DOCUMENTATION_AGENT_WORKFLOW.md`. - Editor/UI work: inspect the relevant `src/features/*` component plus page tests. - Benchmark work: read `src/domain/benchmark/runner.ts` and `scripts/benchmark-solve.ts`. - Historical Masyu plans: check `docs/legacy/` only when old design context is useful. diff --git a/docs/RULE_DOCUMENTATION_AGENT_WORKFLOW.md b/docs/RULE_DOCUMENTATION_AGENT_WORKFLOW.md new file mode 100644 index 0000000..06b0626 --- /dev/null +++ b/docs/RULE_DOCUMENTATION_AGENT_WORKFLOW.md @@ -0,0 +1,363 @@ +# Rule Documentation Agent Workflow + +Use this workflow when documenting registered PuzzleKit solver rules. It is +designed for AI agents that must read the production implementation, explain the +deduction accurately, and add an in-app rule page without changing solver +behavior. + +## Goal + +For each selected rule, produce documentation that lets a player answer three +questions: + +1. **What it does:** What decisions can this rule add to the board? +2. **When it triggers:** What exact board state allows the deduction? +3. **Why it works:** Why would every alternative violate a puzzle rule? + +The production rule implementation is the source of truth. Existing names, +messages, tests, technique notes, and external terminology are supporting +evidence only. + +## Required Reading + +Read these files before documenting a rule: + +1. The rule factory and every helper that materially affects its trigger or + conclusion. +2. The rule registration file to understand its order and puzzle family. +3. Existing focused tests to see supported orientations, boundary cases, and + multi-match behavior. +4. `src/features/docs/ruleDocRegistry.tsx` +5. `src/features/docs/ruleExamples.ts` +6. `src/features/docs/RuleExample.tsx` +7. The `.rule-example*` styles in `src/app/workspace.css` +8. Existing MDX documents under `docs/content//rules/` + +Do not infer behavior from the rule name alone. + +## Workflow + +### 1. Build a Rule Fact Sheet + +Before editing, write a private fact sheet from the implementation: + +```text +Rule ID: +Current display name: +Factory function: +Puzzle family: +Trigger: +Conclusions: +Orientations or mirrored forms: +Existing-state requirements: +Boundary behavior: +Multiple matches in one application: +Reason the deduction is valid: +Related implementation helpers: +``` + +Resolve discrepancies by trusting current production code. If the code appears +logically incorrect or broader than the intended technique, stop documenting +and report the suspected rule issue separately. + +### 2. Review the User-Facing Name + +Prefer a short, recognizable technique name that describes the pattern or +reasoning concept. + +- Change the display name when the current name is overly implementation-driven, + verbose, or ambiguous. +- Preserve the rule ID and factory function by default. They may be referenced + by deep links, traces, tests, benchmarks, or external notes. +- Update the solver-step message when the existing wording no longer matches the + improved name or explanation. +- Keep the message concise and state the trigger plus conclusion. Do not attempt + to reproduce the full MDX explanation in the solver trace. + +### 3. Write the MDX Document + +Write in clear English. Every rule document must contain exactly these three +sections: + +```md +## What it does + +State the visible deductions the rule makes. Include distinct conclusion types +in this section. + +## When it triggers + +State the minimum board conditions required before the rule can act. Explicitly +say when no prior decisions are required. + +## Why it works + +Give the shortest complete logical argument. Explain which puzzle constraint +would be violated by the alternatives. +``` + +Writing rules: + +- Use player-facing puzzle terms, not implementation variables. +- Be precise about `line`, `crossed out`, `inside`, `outside`, clue values, + pearls, vertices, and cells. +- Describe rotations and mirrored forms only when they materially clarify the + trigger. +- Mention multiple subcases only when they share one compact explanation. +- Avoid restating the same sentence in more than one section. +- Do not add `Conclusions`, `Limits and related techniques`, or generic + background sections. +- Do not claim a deduction is unconditional when it relies on the single-loop + rule, a non-empty remainder, uniqueness, or another global assumption. + +### 4. Decide Whether to Add a Canvas Example + +Classify the rule before creating example data. A rule does not need to fit into +one Canvas to qualify. Prefer a small multi-Canvas group over omitting a useful +deterministic explanation merely because the rule has several separable cases. + +The classifications are: + +- **Simple deterministic:** use one Canvas case. +- **Separable deterministic:** use multiple Canvas cases. +- **Strong inference:** never add Canvas cases. +- **Too complex or too coupled:** defer the Canvas and leave an explicit + reminder for the author. + +#### A. Simple deterministic rule: add an example + +Add a Before/After Canvas example when all of these are true: + +- The trigger fits on one small board. +- One static Before state makes the reason recognizable. +- The complete conclusion is understandable in one After state. +- The example does not need branch history, hidden candidates, or several + sequential deductions. + +Example requirements: + +- Use the smallest board that leaves comfortable visual padding. +- Center the relevant pattern when practical. +- Put prerequisite decisions in `before`. +- Put only this rule's new decisions in `after`. +- Do not highlight clue cells or add colored cell backgrounds. +- Do not use inference overlays or dashed conclusion marks. +- In the After view, highlight only newly decided edges or lines through the + standard board decision rendering. New lines remain solid; new crosses remain + visually distinct. +- The caption should state the deduction, not repeat the full proof. + +#### B. Separable deterministic rule: add multiple Canvas cases + +A somewhat complex rule should still receive Canvas documentation when its +important triggers or conclusions can be divided into independent, +direct Before/After cases. Do not defer merely because one board cannot express +the whole rule clearly. + +Use one case per distinct point that a reader needs to compare, such as: + +- Two equally important completion branches. +- Distinct deterministic trigger shapes with the same underlying rule. +- Different direct conclusion types that would be confusing on one board. + +Case-selection rules: + +- Prefer an even number of cases, normally `2` or `4`, because the desktop + layout uses two columns. +- Do not invent a redundant or weak case solely to make the count even. A clear + three-case group is better than four cases with filler. +- Normally stop at four representative cases. More than four is a strong signal + that the rule should be deferred or redesigned with author input. +- Give every case a stable, descriptive `id`. +- Give every case in a multi-case group a short, parallel title. +- Use similarly sized puzzles where practical. Prefer matching dimensions; small + differences are acceptable when the Canvas cards still feel balanced. +- Avoid grouping boards when the largest is roughly more than twice the width + or height of the smallest. +- Keep every case independent. A reader must not need to understand one case as + an intermediate step or hidden prerequisite of another. + +Each case must still satisfy all visual and diff requirements from category A. + +#### C. Strong-inference rule: never add a Canvas example + +Do not create a Before/After Canvas example for strong inference, assumption +inference, contradiction probing, or branch-comparison rules. A static pair +hides the reasoning chain and can misrepresent why the conclusion is valid. + +This restriction overrides every other Canvas guideline. Even if a +strong-inference rule appears easy to illustrate or can be divided into several +small boards, leave its Canvas example empty. Document the rule in MDX and +report: + +```text +Canvas example deferred: this is a strong-inference rule whose explanation +requires branch or contradiction history. +``` + +#### D. Complex rule: defer when unclear + +Defer the Canvas example when cases require intermediate history, branch +reasoning, hidden candidate state, more than four representative cases, or +cannot be understood as independent Before/After states. Also defer when the +cases are too coupled, the necessary split is unclear, or implementing the +group confidently would be disproportionately difficult. + +Do not invent an oversimplified example. Leave the rule's example registration +empty and make the missing Canvas obvious: + +- Add a concise `TODO(rule-doc-canvas)` comment beside the relevant location in + `src/features/docs/ruleExamples.ts`. Include the puzzle ID, rule ID, and exact + blocker so a future author can find and resume the work. +- Always state the deferral and exact blocker in the delivery report under + `Needs author input`. + +Use this wording: + +```text +Canvas example deferred: needs , +but