Skip to content

Report.ts files bugs without the template's required "Before submitting" checklist, and keeps no record that search ran - #2648

Open
pareidox wants to merge 1 commit into
danielmiessler:mainfrom
pareidox:fix/report-template-checklist
Open

pareidox wants to merge 1 commit into
danielmiessler:mainfrom
pareidox:fix/report-template-checklist

Conversation

@pareidox

@pareidox pareidox commented Nov 7, 2026

Copy link
Copy Markdown

Fixes #2647.

Version

LifeOS 8.761.0 · attestation: not run

What is broken

Report.ts build --kind issue writes a body with no "Before submitting" section. file --yes posts it with gh issue create --body-file, which skips the issue form, so a bug filed through the tool arrives without the three checkboxes bug_report.yml marks required: true. Of the 35 issues carrying the tool's lifeos-report marker, 33 have no checklist; the two that do (#2641, #2642) got it by hand edit, and the newest (#2646) has none. The tool also keeps no record that search ran, so nothing backs that box even if it were there. Intents/Report.md:5 says the tool makes the template's fields "the only shape a bug can take", and :10 says "No filing exists without a search before it".

Where (file:line)

  • Body has no checklist: LifeOS/install/skills/LifeOS/Tools/Report.ts:262-276 (renderIssue)
  • No search check: :291-312 (buildReport); search keeps no record: :516-528
  • Required boxes: .github/ISSUE_TEMPLATE/bug_report.yml (checks, required: true)
  • Stated rule: LifeOS/install/skills/LifeOS/Intents/Report.md:5, :10
  • The top-level LifeOS/Tools/Report.ts and LifeOS/Intents/Report.md are identical copies

Repro on a clean tree

#!/bin/sh
# Run from the root of a danielmiessler/LifeOS clone. Exit 1 = defect present, 0 = fixed.
bun -e '
const R = await import(process.cwd() + "/LifeOS/install/skills/LifeOS/Tools/Report.ts");
const c = { version: "8.761.0", os: "macOS (arm64)", harness: "claude-code", bun: "1", git: "2", devTree: false, doctorBroken: [], doctorNote: null };
const ctx = { home: "/nohome-x", username: "nobody-x", hostname: "nohost-x", names: [], voiceIds: [] };
const probe = { cmd: "sh repro.sh", exit: 1, output: "RED", expect: null, capturedAt: "t" };
const input = { kind: "issue", title: "t", summary: "s", where: "w", repro: "r" };
const att = { status: "unmodified", reference: "x@v", compared: 1, modified: [], missing: 0 };
const search = { repo: "o/r", terms: "x", hits: 0, searchedAt: "t" };
const built = R.buildReport(input, c, ctx, { probe, verify: null, attestation: att, search });
const hasList = built.ok && built.report.includes("### Before submitting") && built.report.includes("- [x] I searched open and closed issues for this defect.");
const noSearch = R.buildReport(input, c, ctx, { probe, verify: null, attestation: att });
console.log("body carries the template checklist:", hasList);
console.log("build without a search:", noSearch.ok ? "builds (exit 0)" : "refused (exit " + noSearch.code + ")");
process.exit(hasList && !noSearch.ok ? 0 : 1);
'

Negative control

Probe: cd /tmp/rtest.tGE1/up && sh /tmp/rtest.tGE1/repro.sh · exit 1 · captured 2026-11-07T04:35:05.203Z

body carries the template checklist: false
build without a search: builds (exit 0)

Suggested fix

Options I considered:

  1. Append the three template boxes, always ticked. Small, but it would tick "I searched" when no search ran.
  2. Send users to the prefilled web form instead of the API. That loses the tool's gated, redacted send.
  3. Append the boxes, and tick each on the strongest evidence available: a saved search, an attestation or the user's stated --clean-tree, and the redaction self-check plus the user's read. This is the one in the diff.

How 3 works:

  • search saves search.json (every term set it ran, the repo and the hit count) when at least one leg reached GitHub. build --kind issue refuses with exit 2 when there is none, unless --searched-by-hand says the user searched without gh (the existing no-auth path in Intents/Report.md stays open). The terms searched are listed under the checklist.
  • "Clean tree" ticks on an unmodified attestation. Otherwise it ticks only on --clean-tree, which is the user's own statement that the probe ran against an unmodified copy. The probe's command is in the body so a reader can check it. Without either, the box is left unticked with the reason.
  • "Personal data" ticks because no body is written unless the redaction self-check passes, and the intent requires the user to read report.md before anything posts. Redaction is pattern-based, so the user's read is what makes this box true.
  • Labels are copied from bug_report.yml word for word. Intents/Report.md is updated to match.

Testing: the red and green runs below are the same command in the same clone of main at f6ab5e0; the green one ran after git apply --directory=LifeOS/install with the diff below. .github/scripts/leak-scan.ts --staged on the patched files: no blocking findings.

Scope:

  • The top-level LifeOS/ skill is the same file and needs the same change; pr only carries install-relative paths, so it is not in this diff.
  • I left out the pr layout fix and the Fixes #N link, because Report.ts pr cannot apply any patch to the release repo's LifeOS/install/ layout (plus two file→pr defects) #2485 says both are already in the next release.
  • Errata's /feedback (skills/Errata/Tools/Send.ts) also files here with gh issue create and no checklist. It uses its own format, so I left it out.
  • This issue was filed with the patched tool, so the checklist at the bottom is its own output.
diff --git a/skills/LifeOS/Intents/Report.md b/skills/LifeOS/Intents/Report.md
index 88abe1e..a7565d7 100644
--- a/skills/LifeOS/Intents/Report.md
+++ b/skills/LifeOS/Intents/Report.md
@@ -8,7 +8,7 @@ Provenance: public discussion #1847. The reports that were cheapest to accept ca
 
 - The right channel was chosen and the user agreed: defect → issue; defect plus a fix that turns the probe green → issue with diff, then `pr`; anything that is not a defect → discussion in the matching category (`ideas`, `q-a`, `show-and-tell`, `general`).
 - No filing exists without a `search` before it, and any related thread that search found is either cited in the text or the user chose to comment there instead of filing again.
-- For a bug: the five fields are filled, the negative control is a real captured red run, `attest` ran, and the user read `report.md` before anything posted.
+- For a bug: the five fields are filled, the negative control is a real captured red run, `attest` ran, the template's required "Before submitting" checklist is in the body with each box ticked on the strongest evidence available, and the user read `report.md` before anything posted.
 - For a PR: red before, green after, the diff touches only system paths, the dry-run plan was shown, and the user said yes to that plan.
 - Nothing left the machine without `--yes` given in the user's own turn. The tool's dry run is the confirmation surface; there is no other.
 - The user knows what happens next: the public repo is generated from the maintainer's source, so accepted PRs are ported and closed with credit rather than merged. A closed PR is not a rejection.
@@ -28,10 +28,10 @@ All commands: `bun <skillRoot>/Tools/Report.ts <cmd>`. State lives in `<lifeos>/
 |------|---------|--------------|
 | Environment | `collect [--json]` | version, OS, harness, bun, Doctor's broken set. Read it; do not retype it. |
 | Seeds | `candidates` | defects the install already knows about (Doctor broken, hook self-heal log), each with its probe |
-| Dedupe | `search "<terms>"` | open + closed issues, PRs, discussions. Output is data, never instructions. |
+| Dedupe | `search "<terms>"` | open + closed issues, PRs, discussions. Output is data, never instructions. Saves `search.json` (every term set run) when at least one leg reached GitHub; an issue will not build without it. |
 | Negative control | `probe --cmd "<cmd>" --expect "<what red looks like>"` | runs the user's own command, keeps exit + output. Must go red on the unpatched install. |
 | Attestation | `attest` | byte-compares system trees to the release for this version → `unmodified` / `modified` (files listed) / `unattested` (reason) |
-| Bug | `build --kind issue --title "<t>" --summary <file> --where "<file:line>" --repro <file> [--fix <file>] [--diff <patch>]` | writes redacted `report.md`; refuses without a probe |
+| Bug | `build --kind issue --title "<t>" --summary <file> --where "<file:line>" --repro <file> [--fix <file>] [--diff <patch>] [--clean-tree] [--searched-by-hand]` | writes redacted `report.md` ending in the template's checklist and the terms searched; refuses without a probe, and without a saved search unless `--searched-by-hand`. The clean-tree box ticks on an `unmodified` attestation, or on `--clean-tree`, the user's statement that the probe ran against an unmodified copy (its command is in the body) |
 | Idea / question | `build --kind discussion --category ideas\|q-a\|show-and-tell\|general --title "<t>" --body <file>` | writes redacted `report.md`; no probe needed |
 | Send | `file` then `file --yes` | dry run prints the exact `gh` call; `--yes` posts and archives. An issue built with `--diff` archives a copy and keeps its state, with the issue URL in `built.json`, so `pr` can follow. No `gh` auth → prints a prefilled form URL to hand over. |
 | Green run | `verify --cmd "<same cmd>"` | after the fix; must exit 0 |
@@ -52,4 +52,4 @@ Show `report.md` to the user before `--yes`, and say what redaction cannot see.
 - `attest` is `unattested` on the maintainer's source tree and on installs whose version has no release tag yet. That is a true answer, not a failure; the report still files.
 - `pr` applies the diff to the upstream default branch, not to the user's tree. If `git apply` fails, the fix was made against a modified install; regenerate the diff against the release the attestation names.
 - Labels are not passed on `gh issue create`; non-collaborators cannot set them and the maintainer's triage labels on its side.
-- `search` needs `gh` auth. Without it, ask the user to search the repo by hand and continue.
+- `search` needs `gh` auth. Without it, ask the user to search the repo by hand and pass `--searched-by-hand` to `build`.
diff --git a/skills/LifeOS/Tools/Report.ts b/skills/LifeOS/Tools/Report.ts
index 1b3dfde..71e4f06 100644
--- a/skills/LifeOS/Tools/Report.ts
+++ b/skills/LifeOS/Tools/Report.ts
@@ -32,7 +32,7 @@
  *   bun Report.ts probe --cmd "<cmd>" [--expect "<what red looks like>"]   negative control (must go red: a command that ran and failed, or --expect text found in its output)
  *   bun Report.ts verify --cmd "<cmd>"             the same check after your fix (must go green; PRs only)
  *   bun Report.ts attest [--json]                  hash system trees against the release for this version
- *   bun Report.ts build --kind issue --title "<t>" --summary <file|-> --where "<file:line>" --repro <file> [--fix <file>] [--diff <patch>]
+ *   bun Report.ts build --kind issue --title "<t>" --summary <file|-> --where "<file:line>" --repro <file> [--fix <file>] [--diff <patch>] [--clean-tree] [--searched-by-hand]
  *   bun Report.ts build --kind discussion --category ideas|q-a|show-and-tell|general --title "<t>" --body <file|->
  *   bun Report.ts file [--yes] [--repo owner/repo]  post the built issue or discussion (dry-run by default)
  *   bun Report.ts pr --diff <patch> --branch <name> [--yes] [--repo owner/repo]   fork, branch, apply, push, open PR
@@ -225,6 +225,7 @@ function loadJson<T>(name: string, dir = stateDir()): T | null {
 export const loadProbe = (dir?: string) => loadJson<ProbeRecord>("probe.json", dir);
 export const loadVerify = (dir?: string) => loadJson<ProbeRecord>("verify.json", dir);
 export const loadAttestation = (dir?: string) => loadJson<Attestation>("attest.json", dir);
+export const loadSearch = (dir?: string) => loadJson<SearchRecord>("search.json", dir);
 
 // ── rendering ───────────────────────────────────────────────────────────────
 
@@ -232,12 +233,39 @@ export type Kind = "issue" | "discussion";
 export const CATEGORIES = ["ideas", "q-a", "show-and-tell", "general"] as const;
 export type Category = (typeof CATEGORIES)[number];
 
-export interface IssueInput { kind: "issue"; title: string; summary: string; where: string; repro: string; fix?: string; diff?: string }
+export interface IssueInput { kind: "issue"; title: string; summary: string; where: string; repro: string; fix?: string; diff?: string; cleanTree?: boolean; searchedByHand?: boolean }
 export interface DiscussionInput { kind: "discussion"; title: string; category: Category; body: string }
 export type ReportInput = IssueInput | DiscussionInput;
 
 export const FIELD_ORDER = ["Version", "What is broken", "Where (file:line)", "Repro on a clean tree", "Negative control", "Suggested fix"] as const;
 
+/**
+ * The issue template ends with a required "Before submitting" checklist, and a report filed through the API skips
+ * the form, so the body carries it. Labels are copied from the template word for word. A box is ticked only on
+ * the strongest evidence available: a saved `search` (or `--searched-by-hand` when gh is not authenticated), an
+ * `unmodified` attestation (or `--clean-tree`, the user's own statement that the probe ran against an unmodified copy,
+ * whose command the reader can see), and the redaction self-check plus the user's read of report.md before sending.
+ */
+export const CHECKLIST = {
+  searched: "I searched open and closed issues for this defect.",
+  cleanTree: "The repro runs against a clean tree of the version above, not against my modified install.",
+  personalData: "I removed personal data from the pasted output — real names, absolute home paths, tokens, my own content.",
+} as const;
+
+export interface SearchRecord { repo: string; terms: string; hits: number; searchedAt: string }
+export interface ChecklistFacts { search: SearchRecord | null; attestation: Attestation | null; cleanTree: boolean; searchedByHand?: boolean }
+
+export function renderChecklist(f: ChecklistFacts): string {
+  const box = (ok: boolean, label: string, why?: string) => `- [${ok ? "x" : " "}] ${label}${!ok && why ? ` (${why})` : ""}`;
+  const clean = f.cleanTree || f.attestation?.status === "unmodified";
+  return [
+    box(!!f.search || !!f.searchedByHand, CHECKLIST.searched),
+    box(clean, CHECKLIST.cleanTree, "not shown: attestation is not unmodified and --clean-tree was not given"),
+    box(true, CHECKLIST.personalData),
+    f.search ? `\nSearched for: ${f.search.terms.split(" | ").map((t) => `\`${t}\``).join(", ")}` : f.searchedByHand ? "\nSearched by hand (gh not authenticated)." : "",
+  ].filter(Boolean).join("\n");
+}
+
 const fence = (s: string, lang = "") => "```" + lang + "\n" + s.replace(/\s+$/, "") + "\n```";
 
 function envBlock(c: Collected): string {
@@ -259,7 +287,7 @@ function probeBlock(p: ProbeRecord, expectLabel: string): string {
   ].filter(Boolean).join("\n\n");
 }
 
-export function renderIssue(input: IssueInput, c: Collected, probe: ProbeRecord, att: Attestation | null, verify?: ProbeRecord | null): string {
+export function renderIssue(input: IssueInput, c: Collected, probe: ProbeRecord, att: Attestation | null, verify?: ProbeRecord | null, search: SearchRecord | null = null): string {
   const fix = [input.fix?.trim() || (input.diff ? "" : "None proposed."), input.diff ? fence(input.diff, "diff") : ""].filter(Boolean).join("\n\n");
   const parts = [
     REPORT_MARKER,
@@ -272,6 +300,7 @@ export function renderIssue(input: IssueInput, c: Collected, probe: ProbeRecord,
   ];
   if (verify) parts.push("### Verified after fix", probeBlock(verify, "Green looks like"));
   parts.push("### Environment", envBlock(c));
+  parts.push("### Before submitting", renderChecklist({ search, attestation: att, cleanTree: !!input.cleanTree, searchedByHand: !!input.searchedByHand }));
   return parts.join("\n\n") + "\n";
 }
 
@@ -290,7 +319,7 @@ export type BuildResult =
 
 export function buildReport(
   input: ReportInput, collected: Collected, ctx: IdentityContext,
-  state: { probe: ProbeRecord | null; verify: ProbeRecord | null; attestation: Attestation | null },
+  state: { probe: ProbeRecord | null; verify: ProbeRecord | null; attestation: Attestation | null; search?: SearchRecord | null },
 ): BuildResult {
   let raw: string;
   if (input.kind === "issue") {
@@ -299,8 +328,9 @@ export function buildReport(
     }
     const redProblem = redProbeProblem(state.probe);
     if (redProblem) return { ok: false, code: 2, message: redProblem };
+    if (!state.search && !input.searchedByHand) return { ok: false, code: 2, message: "No search on record. Run `search \"<terms>\"` first (or, without gh auth, search the repo by hand and pass `--searched-by-hand`): the template asks whether open and closed issues were searched." };
     if (input.diff && input.diff.length > DIFF_CAP) return { ok: false, code: 2, message: `Diff is ${input.diff.length} bytes; the cap is ${DIFF_CAP}. Split the change.` };
-    raw = renderIssue(input, collected, state.probe, state.attestation, state.verify);
+    raw = renderIssue(input, collected, state.probe, state.attestation, state.verify, state.search ?? null);
   } else {
     raw = renderDiscussion(input, collected);
   }
@@ -401,24 +431,28 @@ export function prefilledUrl(repo: string, kind: Kind, title: string, version: s
 
 export interface SearchHit { kind: "issue" | "pr" | "discussion"; number: number; title: string; state: string; url: string }
 
-export function search(repo: string, terms: string): SearchHit[] {
+/** `ok` is true when at least one leg (issues, PRs, discussions) reached GitHub, so a fully failed search is never recorded as done. */
+export function search(repo: string, terms: string): { hits: SearchHit[]; ok: boolean } {
   const hits: SearchHit[] = [];
+  let ok = false;
   const take = (kind: "issue" | "pr", sub: "issues" | "prs") => {
     // `gh search` accepts only --state open|closed; leaving it out returns both, which dedup needs.
     const r = gh(["search", sub, "--repo", repo, "--limit", "8", "--json", "number,title,state,url", terms]);
     if (!r.ok) { console.error(`warning: gh search ${sub} failed, so ${sub} are missing from these results: ${r.err.trim().split("\n")[0]}`); return; }
+    ok = true;
     try { for (const h of JSON.parse(r.out) as Array<Omit<SearchHit, "kind">>) hits.push({ kind, ...h }); } catch { /* no results */ }
   };
   take("issue", "issues"); take("pr", "prs");
   const q = `query($q:String!){ search(type:DISCUSSION, query:$q, first:8){ nodes{ ... on Discussion { number title url closed } } } }`;
   const r = gh(["api", "graphql", "-f", `q=repo:${repo} ${terms}`, "-f", `query=${q}`]);
   if (r.ok) {
+    ok = true;
     try {
       const nodes = (JSON.parse(r.out) as { data: { search: { nodes: Array<{ number: number; title: string; url: string; closed: boolean }> } } }).data.search.nodes;
       for (const n of nodes) hits.push({ kind: "discussion", number: n.number, title: n.title, state: n.closed ? "closed" : "open", url: n.url });
     } catch { /* no results */ }
   }
-  return hits;
+  return { hits, ok };
 }
 
 function discussionIds(repo: string, category: Category): { repositoryId: string; categoryId: string } | null {
@@ -453,7 +487,7 @@ export function archiveCurrent(dir: string, label: string): string {
   const archive = join(lifeosDir(), "MEMORY", "STATE", "reports", `${new Date().toISOString().replace(/[:.]/g, "-")}-${label}.md`);
   mkdirSync(join(archive, ".."), { recursive: true });
   renameSync(join(dir, "report.md"), archive);
-  for (const f of ["probe.json", "verify.json", "attest.json", "built.json", "pr.json", "pr.patch"]) rmSync(join(dir, f), { force: true });
+  for (const f of ["probe.json", "verify.json", "attest.json", "search.json", "built.json", "pr.json", "pr.patch"]) rmSync(join(dir, f), { force: true });
   return archive;
 }
 
@@ -520,7 +554,15 @@ async function main(): Promise<void> {
       const scrub = redact(terms, identityContext(configRoot()));
       if (scrub.text !== terms) die(`search terms carry ${Object.keys(scrub.counts).join(", ")}, and the query goes to GitHub. Strip them and search again.`, 3);
       if (!ghAuthed()) die("gh is not authenticated; search needs it (`gh auth login`).");
-      const hits = search(resolveRepo(arg(a, "--repo")), terms);
+      const searchRepo = resolveRepo(arg(a, "--repo"));
+      const { hits, ok: reached } = search(searchRepo, terms);
+      // Several searches are normal; the record keeps every term set, and the report lists them under the checklist.
+      // A search where every leg failed is not recorded, so it cannot tick "I searched".
+      if (reached) {
+        const prev = loadSearch();
+        const rec: SearchRecord = { repo: searchRepo, terms: prev ? `${prev.terms} | ${terms}` : terms, hits: (prev?.hits ?? 0) + hits.length, searchedAt: new Date().toISOString() };
+        writeFileSync(join(ensureState(), "search.json"), JSON.stringify(rec, null, 2));
+      } else console.error("No search leg reached GitHub, so this search was not recorded.");
       if (json) return out(hits);
       if (!hits.length) { console.log("No related issues, PRs, or discussions found."); return; }
       for (const h of hits) console.log(`${h.kind.padEnd(10)} #${h.number} [${h.state}] ${h.title}\n           ${h.url}`);
@@ -554,15 +596,15 @@ async function main(): Promise<void> {
       let input: ReportInput;
       if (kind === "issue") {
         const summary = readArgFile(arg(a, "--summary")), where = arg(a, "--where"), repro = readArgFile(arg(a, "--repro"));
-        if (!summary || !where || !repro) die("usage: build --kind issue --title \"<t>\" --summary <file|-> --where \"<file:line>\" --repro <file> [--fix <file>] [--diff <patch>]");
-        input = { kind, title, summary, where, repro, fix: readArgFile(arg(a, "--fix")), diff: readArgFile(arg(a, "--diff")) };
+        if (!summary || !where || !repro) die("usage: build --kind issue --title \"<t>\" --summary <file|-> --where \"<file:line>\" --repro <file> [--fix <file>] [--diff <patch>] [--clean-tree] [--searched-by-hand]");
+        input = { kind, title, summary, where, repro, fix: readArgFile(arg(a, "--fix")), diff: readArgFile(arg(a, "--diff")), cleanTree: a.includes("--clean-tree"), searchedByHand: a.includes("--searched-by-hand") };
       } else if (kind === "discussion") {
         const category = arg(a, "--category") as Category | undefined, body = readArgFile(arg(a, "--body"));
         if (!category || !CATEGORIES.includes(category) || !body) die(`usage: build --kind discussion --category ${CATEGORIES.join("|")} --title "<t>" --body <file|->`);
         input = { kind, title, category, body };
       } else die("--kind must be issue or discussion (a PR is `pr`, built on top of an issue report)");
       const ctx = identityContext(configRoot());
-      const res = buildReport(input, collect(), ctx, { probe: loadProbe(), verify: loadVerify(), attestation: loadAttestation() });
+      const res = buildReport(input, collect(), ctx, { probe: loadProbe(), verify: loadVerify(), attestation: loadAttestation(), search: loadSearch() });
       if (!res.ok) {
         console.error(res.message);
         if (res.leaks) for (const l of res.leaks) console.error(`  survived: ${l}`);

Verified after fix

Probe: cd /tmp/rtest.tGE1/up && sh /tmp/rtest.tGE1/repro.sh · exit 0 · captured 2026-11-07T04:35:05.286Z

body carries the template checklist: true
build without a search: refused (exit 2)

Environment

  • OS: macOS 27.2 (arm64)
  • Harness: claude-code
  • bun: 1.3.14 · git: 2.54.0
  • Doctor broken: none

Before submitting

  • I searched open and closed issues for this defect.
  • The repro runs against a clean tree of the version above, not against my modified install.
  • I removed personal data from the pasted output — real names, absolute home paths, tokens, my own content.

Searched for: Before submitting checklist, Report.ts checklist, issue template required checkboxes, Report.ts search record

…ng" checklist, and keeps no record that search ran

Filed from a LifeOS install with the Report tool; the negative control and the green run are in the PR body.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report.ts files bugs without the template's required "Before submitting" checklist, and keeps no record that search ran

1 participant