Skip to content

fix: ignore bootstrapped eval result bundles #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aryeko merged 1 commit into from
Jul 4, 2026
Merged

fix: ignore bootstrapped eval result bundles #15

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion src/bootstrap.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,19 @@ const initFilesForSuite = (suite) => {
content: [
"# Eval Results",
"",
"Generated evaluation result bundles are written here. Keep durable summaries, but do not commit bulky transient outputs unless your repo policy requires them.",
"Generated evaluation result bundles are written here. Keep durable summaries, but do not commit raw transient outputs unless your repo policy requires them.",
"",
"The sibling `.gitignore` keeps run bundles local by default while preserving this README and curated files a human intentionally adds.",
"",
].join("\n"),
},
{
relativePath: "evals/results/.gitignore",
content: [
"# Keep raw eval result bundles local unless a human curates a summary.",
"*",

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 4, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Allow curated result summaries to be added normally

When a consumer creates a curated summary under evals/results/, this * rule ignores every new file except the two explicit exceptions, so git status hides something like evals/results/summary.md and plain git add evals/results/summary.md is rejected unless the user knows to force-add or edit the ignore file. I verified this with git check-ignore -v, and Git's gitignore docs describe ! as only re-including matching exception paths; this contradicts the generated README/docs policy that humans can commit curated summaries while only raw run bundles stay local.

Useful? React with 👍 / 👎.

"!README.md",
"!.gitignore",
"",
].join("\n"),
},
Expand Down
18 changes: 18 additions & 0 deletions tests/bootstrap-cli.test.mjs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ describe("bootstrap CLI", () => {

expect(result.status).toBe(0);
expect(result.stdout).toContain("Would write evals/eval-kit.config.json");
expect(result.stdout).toContain("Would write evals/results/.gitignore");
expect(fs.existsSync(path.join(root, "evals"))).toBe(false);
});

Expand All @@ -58,6 +59,23 @@ describe("bootstrap CLI", () => {
root: "cases",
},
});
expect(
fs.readFileSync(
path.join(root, "evals", "results", ".gitignore"),
"utf8",
),
).toBe(
[
"# Keep raw eval result bundles local unless a human curates a summary.",
"*",
"!README.md",
"!.gitignore",
"",
].join("\n"),
);
expect(
fs.readFileSync(path.join(root, "evals", "results", "README.md"), "utf8"),
).toContain("keeps run bundles local by default");

const doctor = runCli(root, ["doctor"]);
expect(doctor.status).toBe(0);
Expand Down