Problem
Two related issues that should be solved together:
1. Populate blocks the HTTP request
backend/src/index.ts:72-73 — await run.start(...) runs the entire populate workflow synchronously within the Fastify request handler. The agent typically makes ~80 tool calls (web searches + row inserts) over 2-5 minutes. This:
- Ties up a Fastify worker thread for the full duration
- Will hit proxy/load-balancer timeouts in production
- Makes the frontend hang on the fetch call with no progress feedback
The endpoint should return immediately with a run/job ID and execute the workflow asynchronously.
2. Dataset status badges are cosmetic — never updated
The StatusBadge component renders three states (building, live, paused) and the schema supports them, but:
datasets.create always sets status: "building" and nothing ever transitions it to live or paused- The
updateStatus mutation exists (convex/datasets.ts:117) but is never called — not by the backend, not by the frontend, not by the workflow
- The populate workflow doesn't set status to
building before starting or live on completion
- There are no UI controls to pause/resume a dataset
The status field is fully wired in the DB and UI display layer but has no actual lifecycle driving it.
Proposed Solution
-
Make /populate async — return { runId } immediately. Run the workflow in the background (fire-and-forget with error logging, or a proper job queue).
-
Drive status from the workflow — the populate workflow should:
- Set status →
building when it starts
- Set status →
live on successful completion
- Set status →
paused (or keep building) on failure, with an error field
-
Frontend polls or subscribes to status — since Convex queries are reactive, the StatusBadge will update in realtime once the backend writes the transitions. The frontend just needs to stop blocking on the populate call and instead watch the dataset's status field.
-
Consider adding a lastError field to datasets so failed populates surface a reason in the UI rather than silently staying in building forever.
Files Involved
backend/src/index.ts — populate route (make async)backend/src/mastra/workflows/populate.ts — add status transitionsfrontend/convex/datasets.ts — updateStatus mutation (already exists, needs callers)frontend/app/dataset/[id]/page.tsx — stop awaiting populate, react to status changesfrontend/components/dataset/StatusBadge.tsx — already handles all three states
🤖 Generated with Claude Code
Problem
Two related issues that should be solved together:
1. Populate blocks the HTTP request
backend/src/index.ts:72-73—await run.start(...)runs the entire populate workflow synchronously within the Fastify request handler. The agent typically makes ~80 tool calls (web searches + row inserts) over 2-5 minutes. This:The endpoint should return immediately with a run/job ID and execute the workflow asynchronously.
2. Dataset status badges are cosmetic — never updated
The
StatusBadgecomponent renders three states (building,live,paused) and the schema supports them, but:datasets.createalways setsstatus: "building"and nothing ever transitions it toliveorpausedupdateStatusmutation exists (convex/datasets.ts:117) but is never called — not by the backend, not by the frontend, not by the workflowbuildingbefore starting orliveon completionThe status field is fully wired in the DB and UI display layer but has no actual lifecycle driving it.
Proposed Solution
Make
/populateasync — return{ runId }immediately. Run the workflow in the background (fire-and-forget with error logging, or a proper job queue).Drive status from the workflow — the populate workflow should:
buildingwhen it startsliveon successful completionpaused(or keepbuilding) on failure, with an error fieldFrontend polls or subscribes to status — since Convex queries are reactive, the
StatusBadgewill update in realtime once the backend writes the transitions. The frontend just needs to stop blocking on the populate call and instead watch the dataset's status field.Consider adding a
lastErrorfield to datasets so failed populates surface a reason in the UI rather than silently staying inbuildingforever.Files Involved
backend/src/index.ts— populate route (make async)backend/src/mastra/workflows/populate.ts— add status transitionsfrontend/convex/datasets.ts—updateStatusmutation (already exists, needs callers)frontend/app/dataset/[id]/page.tsx— stop awaiting populate, react to status changesfrontend/components/dataset/StatusBadge.tsx— already handles all three states🤖 Generated with Claude Code