improvement(mothership): user_table speed parity — limit bounds, background import/delete/update jobs (#5012)

* improvement(mothership): user_table speed parity — limit bounds, async import/delete/update jobs

- query_rows / filter ops clamp limit to the contract maxes; query_rows
  skips execution metadata.
- import_file / create_from_file (large CSV/TSV) and delete_rows_by_filter
  (>1000 unbounded matches) dispatch background table jobs, claiming the
  per-table job slot; inline paths claim the slot too.
- update_rows_by_filter now escalates the same way: >1000 unbounded matches
  run as a background table job (new 'update' job type + runTableUpdate worker
  + tableUpdateTask), so a broad update on a huge table no longer loads every
  row into the request. Best-effort/non-atomic and skips workflow recompute
  (documented); unique-column patches stay inline. Pagination is limit/offset.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(mothership): trim user_table catalog copy to the essentials

Drop the verbose doomedCount/affectedCount, delete-mask, workflow-recompute,
and unique-column asides from the bulk-op descriptions. The model only needs:
large ops return { jobId }, limit maxes at 1000, one job per table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* improvement(mothership): make user_table limit cap internal, not model-facing

The model can now pass any limit — no "cannot exceed 1000" rejection. 1000
becomes an internal threshold: query_rows clamps the page to MAX_QUERY_LIMIT
(totalCount signals truncation; the model pages with offset), and bulk filter
ops above the cap run as background jobs.

update_rows_by_filter loads full row data inline, so an explicit limit above
the cap escalates to the background worker with a new maxRows budget (the worker
stops after maxRows; update has no read mask so the cap is exact). delete only
loads ids inline, so an explicit limit (any size) stays inline — only unbounded
deletes use the masked background path, which would over-hide a bounded delete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* improvement(mothership): bounded delete above the cap runs async, not inline

An explicit delete limit now mirrors update: ≤1000 runs inline, above the cap it
escalates to the background worker honoring the limit via maxRows — instead of
always staying inline. The worker stops after maxRows (per-page fetch capped to
the remaining budget).

Bounded background deletes skip pendingDeleteMask: the filter-based mask hides
every match, which would over-hide the rows beyond the cap the job never deletes.
Unmasked, a bounded delete is eventually consistent like a bounded update (rows
disappear as deleted), and doomedCount is omitted from the payload so the count
isn't double-subtracted.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(mothership): tidy user_table limit/offset param copy

Drop "Any value is allowed" from the limit description and restore the original
offset description.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* fix(tables): skip pendingDeleteMask for bounded background deletes

The bounded-delete commit (f1ee3e9) persisted maxRows and omitted doomedCount
but the pendingDeleteMask guard that makes it work was left uncommitted, so the
shipped mask still hid every filter+cutoff match — over-hiding the rows beyond
maxRows that the job never deletes (they vanished from reads until the job ended,
then reappeared). Return no mask when maxRows is set: a bounded delete is
eventually consistent (rows disappear as deleted), like a bounded update.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* docs(mothership): drop redundant background note from limit arg

The op descriptions already cover background escalation; the limit arg only
needs to say what the param does.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: TheodoreSpeaks

apps/sim/background/table-update.ts

@@ -0,0 +1,38 @@
+import { task } from '@trigger.dev/sdk'
+import {
+  markTableUpdateFailed,
+  runTableUpdate,
+  type TableUpdatePayload,
+} from '@/lib/table/update-runner'
+
+/**
+ * `TableUpdatePayload` with the cutoff as an ISO string — task payloads cross a JSON boundary, so
+ * the Date is rehydrated in `run` rather than trusting payload serialization.
+ */
+export interface TableUpdateTaskPayload extends Omit<TableUpdatePayload, 'cutoff'> {
+  cutoff: string
+}
+
+/**
+ * Trigger.dev wrapper around `runTableUpdate`. Errors propagate out of `run` so the retry policy
+ * fires; the job is marked failed only in `onFailure`, after the final attempt. Retry-safe: the
+ * worker keysets by id with a `created_at <= cutoff` floor and the JSONB-merge patch is idempotent
+ * (re-applying the same patch to an already-patched row is a no-op), so a retried attempt re-walks
+ * and re-applies whatever remains. The `table_jobs` ownership gate stops a retried run that lost
+ * the job within one page.
+ */
+export const tableUpdateTask = task({
+  id: 'table-update',
+  machine: 'small-1x',
+  retry: { maxAttempts: 3 },
+  queue: {
+    name: 'table-update',
+    concurrencyLimit: 10,
+  },
+  run: async (payload: TableUpdateTaskPayload) => {
+    await runTableUpdate({ ...payload, cutoff: new Date(payload.cutoff) })
+  },
+  onFailure: async ({ payload, error }) => {
+    await markTableUpdateFailed(payload.tableId, payload.jobId, error)
+  },
+})

apps/sim/lib/copilot/generated/tool-catalog-v1.ts

@@ -3958,7 +3958,8 @@ export const UserTable: ToolCatalogEntry = {
           },
           limit: {
             type: 'number',
-            description: 'Maximum rows to return or affect (optional, default 100)',
+            description:
+              'Maximum rows to return or affect (optional, default 100). Omit on update_rows_by_filter / delete_rows_by_filter to act on every match.',
           },
           mapping: {
             type: 'object',

apps/sim/lib/copilot/generated/tool-schemas-v1.ts

@@ -3686,7 +3686,8 @@ export const TOOL_RUNTIME_SCHEMAS: Record<string, ToolRuntimeSchemaEntry> = {
             },
             limit: {
               type: 'number',
-              description: 'Maximum rows to return or affect (optional, default 100)',
+              description:
+                'Maximum rows to return or affect (optional, default 100). Omit on update_rows_by_filter / delete_rows_by_filter to act on every match.',
             },
             mapping: {
               type: 'object',