Skip to content

mem migrate silently drops rows with nested idField (e.g. Attio companies id.record_id) #132

@moekatib

Description

@moekatib

Summary

`mem migrate` treated `profile.idField` as a flat property name (`row[idField]`) instead of a dotted path. On Attio companies (`idField: "id.record_id"`) where the legacy SQLite stores `id` as a JSON-stringified `{workspace_id, object_id, record_id}`, every row failed to resolve and was silently counted as "skipped" — including on an empty memory store, where skipped is never a legitimate outcome.

Repro

rm -rf ~/.one/mem.pglite
one --agent mem migrate -y
→ {\"platform\":\"attio\",\"model\":\"attioCompanies\",\"rowsSeen\":2024,\"inserted\":0,\"updated\":0,\"skipped\":2024}

Root cause

  1. `migrate.ts` used `row[idField]` (flat lookup) instead of `getByDotPath(row, idField)` (same mechanism sync uses).
  2. Legacy SQLite JSON-stringifies nested objects before INSERT, so `row.id` was a string like `'{"record_id":"..."}'` — a dotted path wouldn't walk through it anyway without hydration.
  3. The report exposed a single `skipped` counter, so a profile misconfiguration and a real key conflict looked identical.

Fix

Shipped in `feat/unified-memory` (next push):

  • `reviveStringifiedJson()` rehydrates top-level JSON-stringified columns before id resolution, matching what sync sees live.
  • Migrate now calls `getByDotPath(row, idField)` with the rehydrated row.
  • Hard reject when the resolved id is a nested object (same guard sync test/mem-writer apply) — prevents silent `[object Object]` collapse.
  • Report splits `skipped` into `skippedUnresolvedId` + `skippedError`.
  • Human output prints a warning when every row is skipped ("profile missing or idField doesn't resolve").

Verified

Unit tests for rehydration + dotted-path resolution on the exact shape that broke (Attio `id.record_id`).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions