Data Collection Workflow Update

.env.example

@@ -9,11 +9,19 @@ CLERK_SECRET_KEY=sk_test_...
 # Generate at https://openrouter.ai/settings/keys
 OPENROUTER_API_KEY=sk-or-...
 
+# TinyFish — required by populate agent web search/fetch.
+# Generate at https://agent.tinyfish.ai/api-keys
+TINYFISH_API_KEY=
+
 # Generate once after the first `make dev` with:
 #   docker compose exec convex ./generate_admin_key.sh
 # Used by the backend container to call internal Convex functions.
 CONVEX_SELF_HOSTED_ADMIN_KEY=
 
+# Durable store for self-healing populate recipe manifests.
+# Docker dev overrides this to /app/.bigset/populate-recipes on a named volume.
+POPULATE_RECIPE_STORE_DIR=.bigset/populate-recipes
+
 # PostHog (optional — leave blank to disable analytics entirely in local dev).
 # Get from https://us.posthog.com/project/settings/general.
 NEXT_PUBLIC_POSTHOG_KEY=

.gitignore

@@ -1,5 +1,6 @@
 .DS_Store
 node_modules/
+backend/node_modules
 .env
 .env.local
 Project_BigSet_brief.md
@@ -14,16 +15,22 @@ Project_BigSet_brief.md
 *.log
 npm-debug.log*
 yarn-debug.log*
+/benchmark-results/
 
 # Local-only files
 *.bak
 tmp/
 temp/
 
 .mastra
+.bigset/
+
+# Accidental root-level npm install (use backend/ and frontend/ package managers).
+/package.json
+/package-lock.json
 
 # Local tarballs
 *.tgz
 
 # Internal docs
-BigSet Technical Specs & Goals.md
+BigSet Technical Specs & Goals.md

CLAUDE.md

@@ -29,7 +29,7 @@ Backend is Fastify + Mastra. Fastify serves the HTTP API (Clerk JWT auth on prot
 
 The schema inference pipeline: frontend calls `POST /infer-schema` → Fastify verifies the Clerk JWT → calls `inferSchema()` in `backend/src/pipeline/schema-inference.ts` → Claude Sonnet 4.6 via OpenRouter → returns a Zod-validated `DatasetSchema` → frontend maps it to editable columns in the wizard.
 
-The populate pipeline: frontend calls `POST /populate` with `{ datasetId, datasetName, description, columns }` → Fastify verifies the Clerk JWT → triggers `populateWorkflow` which: (1) clears existing rows, (2) builds a prompt from the schema, (3) runs the populate agent (Claude Sonnet 4.6) which searches the web via TinyFish APIs, then inserts rows into Convex one by one. Rows appear in realtime on the frontend via Convex reactive queries.
+The populate pipeline: frontend calls `POST /populate` with `{ datasetId, datasetName, description, columns }` → Fastify verifies the Clerk JWT → runs the self-healing populate service. The service builds or reuses a recipe, runs the Mastra populate runtime against TinyFish search/fetch, validates source-backed rows, repairs bad recipes, promotes the passing recipe, then atomically replaces the dataset rows in Convex. Rows appear in realtime on the frontend via Convex reactive queries.
 
 Convex functions use `ctx.auth.getUserIdentity()` to get the authenticated user. The `ownerId` field on datasets stores `identity.subject` (Clerk user ID). Do not pass `ownerId` from the client.
 
@@ -49,4 +49,10 @@ Convex is self-hosted — it does NOT hot-reload when you edit files in `fronten
 
 In CI/prod, run `npx convex deploy` with `CONVEX_SELF_HOSTED_URL` and `CONVEX_SELF_HOSTED_ADMIN_KEY` set as env vars.
 
+## Self-Healing Verification
+
+Run `make verify-self-healing` before handing the stack to another agent. It runs backend tests, backend build, adapter syntax checks, and a no-key benchmark smoke that should block cleanly without spending API credits.
+
+Use `bash scripts/verify-self-healing-stack.sh --real-benchmark` for the 2-prompt real Mastra benchmark, and `bash scripts/verify-self-healing-stack.sh --convex-push --dataset-id <dataset-id>` for a live app dataset dry-run. Export the required env vars before live modes; the verifier does not parse secret files itself. Add `--commit` only when you intentionally want to replace rows.
+
 This is an open-source (AGPL) project. Do not commit secrets, API keys, or internal docs.

backend/.env.example

@@ -1,6 +1,7 @@
 CLIENT_ORIGIN=http://localhost:3500
 CONVEX_URL=http://localhost:3210
 PORT=3501
+POPULATE_RECIPE_STORE_DIR=.bigset/populate-recipes
 
 # Required once the backend starts writing rows via internal Convex mutations.
 # Generate with: docker compose exec convex ./generate_admin_key.sh
@@ -11,10 +12,43 @@ CONVEX_SELF_HOSTED_ADMIN_KEY=
 CLERK_SECRET_KEY=
 CLERK_PUBLISHABLE_KEY=
 
-# OpenRouter API key — required by schema inference.
+# OpenRouter API key — required by schema inference and populate.
 # Generate at https://openrouter.ai/settings/keys
 OPENROUTER_API_KEY=sk-or-...
 
 # TinyFish API key — used by the populate agent for web search and fetch.
 # Generate at https://agent.tinyfish.ai/api-keys
 TINYFISH_API_KEY=
+
+# Optional model overrides (see backend/src/openrouter-models.ts).
+# Schema inference defaults to anthropic/claude-sonnet-4-6.
+# Populate and other non-inference tasks default to google/gemini-3.1-flash-lite.
+# OPENROUTER_MODEL=google/gemini-3.1-flash-lite
+# OPENROUTER_POPULATE_MODEL=google/gemini-3.1-flash-lite
+
+# Populate runtime limits (see src/pipeline/populate-runtime-limits.ts).
+# POPULATE_MAX_FETCH_CALLS caps prioritized source URLs for the populate agent (default 50).
+# POPULATE_MAX_ROWS=100
+# POPULATE_MAX_SEARCH_CALLS=25
+# POPULATE_MAX_FETCH_CALLS=50
+
+# Parallel populate workers (triage + extract per URL shard).
+# POPULATE_URLS_PER_WORKER=5
+# POPULATE_MAX_TINYFISH_AGENT_RUNS=5
+# POPULATE_ENABLE_TINYFISH_AGENT=true
+# POPULATE_TINYFISH_AGENT_POLL_TIMEOUT_MS=480000
+
+# Central collection memory (repair_loop placeholder + agent_visited_urls).
+# Defaults to a sibling of POPULATE_RECIPE_STORE_DIR (e.g. .bigset/collection-memory).
+# POPULATE_ENABLE_COLLECTION_MEMORY=true
+# POPULATE_COLLECTION_MEMORY_DIR=.bigset/collection-memory
+# POPULATE_MAX_REPAIR_LOOPS=3
+
+# Playwright agent dock (default off; replays URLs with saved Tinyfish emitted_process).
+# POPULATE_ENABLE_PLAYWRIGHT_AGENT=false
+# POPULATE_MAX_PLAYWRIGHT_AGENT_RUNS=5
+# POPULATE_PLAYWRIGHT_AGENT_POLL_TIMEOUT_MS=480000
+
+# When true, Mastra populate benchmark runs write intermediate artifacts under
+# BIGSET_BENCHMARK_ARTIFACT_DIR/debug/ (JSON + CSV). No effect when unset/false.
+# POPULATE_BENCHMARK_DEBUG=false

backend/.gitignore

@@ -1,5 +1,16 @@
 node_modules/
 dist/
 .env
+.env.local
 *.tsbuildinfo
 drizzle/
+
+# Local populate / collection runtime (see POPULATE_RECIPE_STORE_DIR, POPULATE_COLLECTION_MEMORY_DIR).
+# Root .gitignore also ignores .bigset/ when those dirs live there.
+
+# BigSet_Data_Collection_Agent CLI output (default: runs/{run_id}/ under cwd).
+runs/
+memory/
+
+# Mastra local dev output
+.mastra/