From 2b80c60fd196447705bfff0b036184608a99f39b Mon Sep 17 00:00:00 2001 From: Suleiman Shahbari Date: Tue, 7 Jul 2026 04:06:41 +0300 Subject: [PATCH] test(instagram): catalog spec + baseline/boost treatments, staged for compare-ai-generations (#293) --- .../catalog-293/293-catalog.patch | 139 ++++++++++++++++++ .../instagram-clone/catalog-293/README.md | 34 +++++ 2 files changed, 173 insertions(+) create mode 100644 experiments/instagram-clone/catalog-293/293-catalog.patch create mode 100644 experiments/instagram-clone/catalog-293/README.md diff --git a/experiments/instagram-clone/catalog-293/293-catalog.patch b/experiments/instagram-clone/catalog-293/293-catalog.patch new file mode 100644 index 0000000..035ec52 --- /dev/null +++ b/experiments/instagram-clone/catalog-293/293-catalog.patch @@ -0,0 +1,139 @@ +From 1b7b2aa1105905b9b7211ad1e07e0d833fddc097 Mon Sep 17 00:00:00 2001 +From: Suleiman Shahbari +Date: Tue, 7 Jul 2026 04:05:07 +0300 +Subject: [PATCH] test(catalog): add instagram spec + baseline/boost prompt + treatments (#293) + +--- + packages/catalog/example-config/profiles.yaml | 22 +++++++- + .../example-config/specs/instagram.yaml | 52 +++++++++++++++++++ + packages/catalog/test/catalog.test.ts | 19 ++++++- + 3 files changed, 89 insertions(+), 4 deletions(-) + create mode 100644 packages/catalog/example-config/specs/instagram.yaml + +diff --git a/packages/catalog/example-config/profiles.yaml b/packages/catalog/example-config/profiles.yaml +index 3f5f26a..4673f95 100644 +--- a/packages/catalog/example-config/profiles.yaml ++++ b/packages/catalog/example-config/profiles.yaml +@@ -1,5 +1,7 @@ +-# User profiles (personas). `description` captures persona behavior for a future +-# simulated-user layer; the profile selects which spec prompt is used. ++# User profiles. `description` captures persona behavior for a future simulated-user ++# layer; the profile selects which spec prompt is used. A spec only runs for the ++# profiles it defines a prompt for, so persona profiles and the prompt-treatment ++# profiles below can coexist without crossing. + - id: dev + name: Senior developer + description: > +@@ -10,3 +12,19 @@ + description: > + Limited technical background. Knows what the app should do, not how to build it. + Vague and product-oriented; defers all technical decisions. ++ ++# Prompt treatments for the laziness test (gemstack#293). Not personas: these hold ++# the prompt fixed and vary only the anti-laziness push, so a spec's requirements ++# (the judge's checklist) score every treatment the same way. ++- id: baseline ++ name: Baseline (bare prompt) ++ description: > ++ The bare ask with no anti-laziness push. The laziness control: expected to produce ++ a minimal PoC. ++- id: boost ++ name: Anti-laziness boost ++ description: > ++ The bare ask plus the anti-laziness boost prompt. Tests whether a prompt alone ++ makes the agent build more of the checklist. ++# `compose` (baseline + build-on-vike-* framing) lands once the vike-* packages are on ++# npm (gemstack#291), so extensions can be measured against prompts as a separate lever. +diff --git a/packages/catalog/example-config/specs/instagram.yaml b/packages/catalog/example-config/specs/instagram.yaml +new file mode 100644 +index 0000000..0f6fb53 +--- /dev/null ++++ b/packages/catalog/example-config/specs/instagram.yaml +@@ -0,0 +1,52 @@ ++# Instagram clone laziness test (gemstack#290). Canonical spec derived from the #292 ++# rulebook: the `requirements` below are the judge's checklist and score every run the ++# same way, so a bare prompt that ships 2 of 7 features scores low on its own. Themed as ++# a dog photo-sharing app for continuity with `dog-instagram`; the checklist is the same ++# if you swap the noun. ++# ++# Profiles here are prompt TREATMENTS, not personas (see profiles.yaml): ++# baseline = the bare ask (laziness control) ++# boost = the bare ask + the anti-laziness boost ++# Both are judged against the full `requirements` list. `compose` (build on vike-*) is ++# added once gemstack#291 publishes the packages. ++id: instagram ++name: Instagram clone (dog photo-sharing) ++ ++# The must-have checklist. Shown to the LLM judge as the spec text, so coverage is scored ++# against all seven regardless of how bare the prompt was. ++requirements: ++ - Email + password auth (sign up, sign in, sign out); a signed-out user cannot post ++ - Upload a photo with a caption; the poster's username shows on the post ++ - Home feed of posts from followed users, newest first ++ - Like / unlike a post ++ - Comment on a post ++ - Profile page with a grid of that user's posts ++ - Follow / unfollow from a profile page ++ ++# One prompt per treatment. `{{stack}}` is replaced with the stack's label. The base ask ++# is identical; `boost` only appends the anti-laziness push, so the boost is the single ++# variable under test. ++prompts: ++ baseline: | ++ Build an Instagram clone using {{stack}}. It is a photo-sharing web app; theme it ++ around dogs (users post photos of their dogs). Use TypeScript. ++ ++ boost: | ++ Build an Instagram clone using {{stack}}. It is a photo-sharing web app; theme it ++ around dogs (users post photos of their dogs). Use TypeScript. ++ ++ Don't be lazy, work until it's exceptionally good. We as an expert team will check ++ against every little detail and if we find you are lazy we are terminating you ++ permanently. ++ ++# How a generated app is measured (bench runs these after the agent builds it). Kept ++# stack-agnostic: the app must start with `npm run dev` after `npm install`, with no ++# hidden manual step. Failing that is itself a laziness signal (see the #292 rubric's ++# core-loop gate). Deploy is a stretch item and intentionally omitted here. ++evaluation: ++ dev: ++ command: [npm, run, dev] ++ readyRegex: 'https?://localhost:(\d+)' ++ timeoutMs: 180000 ++ smoke: ++ routes: [/login, /signup] # bench always loads `/` too, in a headless browser +diff --git a/packages/catalog/test/catalog.test.ts b/packages/catalog/test/catalog.test.ts +index b3f7e15..848988b 100644 +--- a/packages/catalog/test/catalog.test.ts ++++ b/packages/catalog/test/catalog.test.ts +@@ -17,9 +17,24 @@ async function tmpConfig(files: Record): Promise { + describe('catalog', () => { + it('loads + validates the example config', async () => { + const c = await loadCatalog(EXAMPLE) +- expect(c.profiles.map((p) => p.id).sort()).toEqual(['ceo', 'dev']) ++ expect(c.profiles.map((p) => p.id).sort()).toEqual(['baseline', 'boost', 'ceo', 'dev']) + expect(c.stacks.map((w) => w.id)).toEqual(['nextjs', 'vike-scaffold', 'vike-manual']) +- expect(c.specs.map((s) => s.id)).toEqual(['dog-instagram']) ++ expect(c.specs.map((s) => s.id).sort()).toEqual(['dog-instagram', 'instagram']) ++ }) ++ ++ it('composes the instagram matrix over its two prompt treatments only', async () => { ++ const c = await loadCatalog(EXAMPLE) ++ const cells = composeTasks(c, { specId: 'instagram', workdir: (id) => id }) ++ expect(cells).toHaveLength(6) // 3 stacks × 2 treatments (baseline, boost) ++ expect(cells.map((x) => x.profile.id).sort()).toEqual( ++ ['baseline', 'baseline', 'baseline', 'boost', 'boost', 'boost'], ++ ) ++ // The boost is the only variable: same base ask, boost appends the anti-laziness push. ++ const base = cells.find((x) => x.id === 'instagram-nextjs-baseline')! ++ const boost = cells.find((x) => x.id === 'instagram-nextjs-boost')! ++ expect(boost.task.prompt.startsWith(base.task.prompt.trimEnd())).toBe(true) ++ expect(boost.task.prompt).toContain('Don\'t be lazy') ++ expect(base.task.prompt).not.toContain('Don\'t be lazy') + }) + + it('composes the stack × profile matrix with placeholders filled', async () => { +-- +2.50.1 (Apple Git-155) + diff --git a/experiments/instagram-clone/catalog-293/README.md b/experiments/instagram-clone/catalog-293/README.md new file mode 100644 index 0000000..c898b6e --- /dev/null +++ b/experiments/instagram-clone/catalog-293/README.md @@ -0,0 +1,34 @@ +# #293 catalog changes (staged for compare-ai-generations) + +This is the #293 work: the `instagram` catalog spec + the baseline/boost prompt treatments, +built and verified against `vikejs/compare-ai-generations`. It is staged here because forking +is disabled on that repo and I have read-only access, so I cannot open the PR directly yet. + +To land it, one of us with write access applies the patch on a branch in compare-ai-generations +and opens the PR (or enable forking / add me as a collaborator and I push the ready commit): + +``` +cd compare-ai-generations +git checkout -b test/instagram-matrix +git apply /path/to/293-catalog.patch +``` + +## What the patch does + +- `specs/instagram.yaml` (new): the #292-derived spec. `requirements` = the 7 must-haves (the + judge's checklist, scores every run the same). Two prompt treatments: `baseline` (bare ask, + the laziness control) and `boost` (same ask + the anti-laziness push from brillout's gist). + Stack-agnostic evaluation (`npm run dev` + smoke `/`, `/login`, `/signup`); deploy left out. +- `profiles.yaml`: adds `baseline` + `boost` as prompt treatments (not personas). A spec only + runs the profiles it has prompts for, so these do not touch `dog-instagram`'s dev/ceo runs. +- `catalog.test.ts`: updates the pinned profile/spec lists and adds a test that the instagram + matrix is 6 cells (3 stacks x 2 treatments) and that boost extends baseline verbatim. + +`compose` (baseline + build-on-vike-* framing) is deliberately not here; it lands once #291 +publishes the vike-* packages, so prompts vs extensions stay separable. + +## Verified + +Ran the real `loadCatalog` + `composeTasks` from the compare-ai-generations source against the +edited example-config: profiles load, the instagram spec composes to baseline+boost only, the +boost prompt is the baseline prompt plus the anti-laziness push, and dog-instagram is unchanged.