Skip to content

Bug: memory embeddings silently dropped on failure; no backfill path #29

@raphasouthall

Description

@raphasouthall

Summary

56 of 599 memories (9%) have `embedding=NULL` and are therefore invisible to semantic reranking and dedup. All were written between 2026-03-24 and 2026-04-02 (no new ones since), consistent with a historical Ollama outage. There is no retry, no flag, and no backfill command — so these 56 are permanently semantic-search-blind.

Root cause

`save_memory()` at `src/neurostack/memories.py:170-179` catches embedding failure as DEBUG and commits the row with a NULL embedding:

```python
try:
...
embedding_blob = embedding_to_blob(emb)
except Exception as exc:
log.debug("Could not embed memory (non-fatal): %s", exc)
```

Same silent swallow in `update_memory` (line 294) and `merge_memories` (line 547).

`cli/index.py:36` (`cmd_backfill`) only backfills summaries and triples for notes — never memory embeddings.

Consequence: `_hybrid_memory_search` (`memories.py:706-717`, filters `if r.get("embedding")`) and `find_similar_memories` (line 455) silently exclude these rows. They only surface via FTS5 keyword match.

Proposed fix

  1. Add an `embed_pending` boolean column on `memories` in `schema.py`.
  2. In `save_memory` / `update_memory` / `merge_memories`: set `embed_pending=1` on failure, log at WARNING (not DEBUG).
  3. Extend `cmd_backfill` in `cli/index.py` with a memory-embedding pass (iterate `WHERE embedding IS NULL`, call `get_embedding`, clear flag).
  4. Wire it into `watcher.py` so new failures self-heal on next index cycle.

Expected effect

Coverage 91% → 100%. Future embedding-service outages self-heal instead of permanently corrupting the memory store.

Key files

  • `src/neurostack/memories.py:170-179, 253-295, 538-548, 706-717`
  • `src/neurostack/cli/index.py:36-58`
  • `src/neurostack/schema.py` (new column)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions