Bug: extractMinMessages=2 + autoCaptureSeenTextCount 累积逻辑失效 → 所有单轮对话都掉入 regex fallback，污染全库为脏数据

[lintianyuan666]

Plugin Version

1.1.0

OpenClaw Version

2026.3.28

Bug Description

开启 smartExtraction: true 后，正常配置 extractMinMessages: 2，但 auto-capture 几乎所有对话都落入 regex fallback 写入 raw text，导致库内记忆全部是脏数据（l0_abstract == text，无 LLM 蒸馏）。
代码有两条累积路径试图凑够 extractMinMessages，两条都失效：

───

路径A：autoCaptureSeenTextCount diffing（失效）

// index.ts line 2169-2176
const previousSeenCount = autoCaptureSeenTextCount.get(sessionKey) ?? 0;
let newTexts = eligibleTexts; // ← 每次 agent_end 的 eligibleTexts 是"当前事件的消息数"，不是"历史累积量"
if (pendingIngressTexts.length > 0) {
newTexts = pendingIngressTexts;
} else if (previousSeenCount > 0 && eligibleTexts.length > previousSeenCount) {
newTexts = eligibleTexts.slice(previousSeenCount); // ← 永远不会触发，因为 eligibleTexts.length === previousSeenCount === 1
}
autoCaptureSeenTextCount.set(sessionKey, eligibleTexts.length); // ← 每次覆盖成"1"，diffing 失效

在单轮 DM 场景：

• 事件1：eligibleTexts=1, previousSeenCount=0 → newTexts=1 → smart extraction 跳过（需要≥2）
• 事件2：eligibleTexts=1, previousSeenCount=1 → 1 > 1 为 false → newTexts=1 → 同样跳过

日志佐证：

08:44:28 smart-extractor: extracted 3 candidates ← 历史累积生效过一次（跨会话或特定模式）
08:46:41 regex fallback found 1 capturable text(s) ← 后续全走 regex

───

路径B：pendingIngressTexts 跨消息累积（冷启动失效）

// message_received hook — 累积入口
const conversationKey = buildAutoCaptureConversationKeyFromIngress(channelId, conversationId);
queue.push(normalized); // ← 来自用户发送的 ingress 消息

// agent_end hook — 消费出口
const conversationKey = buildAutoCaptureConversationKeyFromSessionKey(sessionKey); // ← 格式: "agent:::"
const pendingIngressTexts = autoCapturePendingIngressTexts.get(conversationKey) ?? [];

问题：pendingIngressTexts.length > 0 时会用 pending 队列替代当前 texts，但这段代码只在 previousSeenCount > 0 时才可能有意义（否则 pending 队列里的内容永远是那1条刚进门的 ingress 消息）。

且 pending 队列只在 previousSeenCount > 0 && eligibleTexts.length > previousSeenCount 时才被"考虑"——第一次对话永远没有 previousSeenCount，永远用 eligibleTexts，永远凑不到2。

───

结果

对话模式	eligibleTexts	smartExtraction	regex fallback	结果
单轮 DM（1条 user msg）	1	❌ 跳过（<2）	✅ 触发	⚠️ 脏数据
多轮历史累积成功	≥2	✅ 触发	❌ 不触发	✅ 正常
LLM extraction 失败	≥2	❌ 失败	✅ 触发	⚠️ 脏数据

───

日志：
memory-pro: smart-extractor: extracted 3 candidate(s) ← smart extraction 成功
memory-pro: smart-extractor: created [cases] Memory-lanceDB-pro dirty data issue
memory-pro: smart-extractor: created [preferences] Model preference: Yunwu GPT-4o
memory-pro: smart-extracted 2 created, 0 merged, 1 skipped ← 正常
regex fallback found 1 capturable text(s) ← 单轮 DM 落入 fallback
memory-lancedb-pro: auto-captured 1 memories for agent main in scope agent:main ← 脏数据

Expected Behavior

改 extractMinMessages 语义
将 extractMinMessages 从"每轮 eligible texts 数量"改为"smart extraction 触发前需要累积的最小 conversation rounds"，并在 session 级别真正做累积计数，而不是依赖 per-event 的 diffing hack。

Steps to Reproduce

以上

Error Logs / Screenshots

Embedding Provider

None

OS / Platform

No response

[lintianyuan666]

完全依靠Noise Filter，只要过了，就必然写进去。如果过了的是没意义的东西，就会fallback写入脏数据

[AliceLJY]

Thanks for the detailed analysis and the code walkthrough! You've correctly identified the issue — eligibleTexts is per-event (always 1 for single-turn DM), so previousSeenCount never advances ahead of it, and the eligibleTexts.length > previousSeenCount condition never fires. Smart extraction is permanently skipped and every single-turn conversation falls through to regex fallback.

The fix should change extractMinMessages semantics to count conversation rounds at the session level rather than relying on per-event eligibleTexts diffing. Tracking for beta.11.

In the meantime, setting extractMinMessages: 1 is the workaround to unblock. Appreciate the log evidence!

[qaqcvc]

哈哈哈，我也是类似情况。auto-recall全部超时。一看最近入库的信息，全是垃圾session id，写入速度还贼快。库里400多条信息，大概300条可能都是垃圾，还有把整个session放进去的，结果送去jina rerank，直接几百k的token花掉了

[jlin53882]

Update: Proposed Fix in PR #518

We've implemented a fix for this issue based on AliceLJY's confirmation of the correct direction.

Changes (6 Fixes)

1. Fix /lesson命令是自己添加吗 没成功 #1: buildAutoCaptureConversationKeyFromIngress - DM fallback to channelId (fixes pendingIngressTexts never written for Discord DM)
2. Fix 关于rerank #2: autoCaptureSeenTextCount accumulates instead of overwriting per-event
3. Fix rerankEndpoint属性没有加到openclaw.plugin.json里面 #3: Smart extraction threshold uses cumulative turn count
4. Fix 插件报错LLM request rejected: invalid params, tool result's tool id(call_function_ypi0ldybep4v_1) not found (2013) #5: isExplicitRememberCommand guard preserved
5. Fix 使用 embedding-3 模型时，即使配置了 dimensions: 1024，API 调用后返回的向量维度与配置不匹配 #6: extractMinMessages capped at 100
6. Fix memory-pro reembed fails: clampInt is not defined #7: MAX_MESSAGE_LENGTH=5000 guard

Open Questions (for maintainer review)

1. Semantic change: extractMinMessages changed from per-event message count to cumulative turns. AliceLJY confirmed this is correct. Is this acceptable as a bug fix?
2. Window size: pendingIngressTexts rolling window is -6. Sufficient for extractMinMessages up to ~6, but insufficient if extractMinMessages >> 6. Should there be dynamic adjustment?
3. isExplicitRememberCommand: After Fix rerankEndpoint属性没有加到openclaw.plugin.json里面 #3, multi-turn conversations rarely satisfy texts.length===1. Is this the intended behavior?

Full PR: #518

[jlin53882]

PR Review Updates (after maintainer feedback)

All three open questions have been addressed.

Q1: delete() removal - is this intentional? ✅

Yes, it is intentional. The original design called for deleting pendingIngressTexts after each �gent_end to prevent stale data. However, with cumulative counting (Fix #2), pendingIngressTexts is event-scoped (only contains messages since the last �gent_end), so deletion is no longer necessary. The rolling window (queue.slice(-6)) ensures bounded memory growth. Fix #3 uses REPLACE (not APPEND) when pendingIngressTexts.length > 0, so history accumulation is correctly handled by currentCumulativeCount alone.

Q2: Added test ✅

New test
unCumulativeTurnCountingScenario in est/smart-extractor-branches.mjs verifies:

• Turn 1: cumulative=1 < 2 → skipped ✅
• Turn 2: cumulative=2 >= 2 → triggers smart extraction ✅

Q3: Changelog updated ✅

• Added test file reference ( est/smart-extractor-branches.mjs)
• Improved breaking change label with ⚠️ Breaking change prefix + specific adjustment guidance
• Added specific behavior notes for extractMinMessages cap and MAX_MESSAGE_LENGTH guard

---

OpenCode review confirmed: All 3 fixes verified correct. Changes are complete and consistent.

[jlin53882]

Phase 2: Design Questions for Maintainer Input

Phase 1 cleanup (PR #534) is ready — fixes createMockApi interface and removes dead code.

Before Phase 2 (semantic fixes), we need maintainer input on these 5 questions:

---

Q1: Smart Extraction Re-trigger Policy

Issue: currentCumulativeCount is monotonic (no reset). Once it crosses extractMinMessages=N, every subsequent turn triggers smart extraction.

Current behavior (with Phase 1):

• Turn 1: cumulative=1, skip
• Turn 2: cumulative=2, trigger ✓
• Turn 3: cumulative=3, trigger again ⚠️
• Turn 4: cumulative=4, trigger again ⚠️

Is this the intended behavior? Should there be a session-level once-only flag?

---

Q2: REPLACE vs APPEND Strategy

PR #518 chose REPLACE (newTexts = pendingIngressTexts) to avoid deduplication issues when the same text appears in both pendingIngressTexts and eligibleTexts.

Alternative: Keep APPEND but deduplicate before joining.

Which strategy is preferred?

---

Q3: Stale pendingIngressTexts Prevention

Issue: autoCapturePendingIngressTexts.delete(conversationKey) was removed in PR #518, relying on REPLACE + slice(-6) to prevent stale data.

Concern: If agent_end fires without a preceding message_received (e.g., direct trigger, retry/replay), stale texts could be consumed.

Is there a mechanism to ensure message_received always precedes agent_end for a given conversationKey?

---

Q4: extractMinMessages Semantic Change

PR #518 changes semantics from "per-event message count" to "cumulative conversation turns".

This is a breaking change for users who configured extractMinMessages based on the old semantics.

Is this semantic change acceptable as a bug fix, or should it be a separate config option?

---

Q5: isExplicitRememberCommand Guard (Dead Code)

Phase 1 removes this guard as unreachable under REPLACE, but I want to confirm this is correct:

Original intent: Handle single "請記住..." messages without enough context, by appending priorRecentTexts.

Why unreachable now: Under REPLACE, texts = pendingIngressTexts (rolling window of up to 6 messages), so texts.length === 1 can never be true in multi-turn scenarios.

Should this behavior be preserved? If so, where should the context-enrichment logic live?

---

PR #534 (Phase 1): #534
PR #518 (main PR): #518

[jlin53882]

Phase 2 對抗分析結果（作者 + OpenCode 對抗產出）

經過 OpenCode 對抗式分析，以下是 Q1~Q5 的最終推薦方案：

---

Q1: Smart Extraction Re-trigger Policy

OpenCode 反對：binary flag 會破壞多輪對話核心設計，建議冷卻期（cooldown）。
作者反擊：冷卻期會阻擋合理的新 extract 需求。

最終共識：雙方都不完美 → 採用「連續新訊息計數」機制。

// 初始 extract 後，messagesSinceLastExtract 歸零
// 每收到新訊息：messagesSinceLastExtract++
// 達到 reExtractAfterNNewMessages 才再次 extract
if (currentCumulativeCount >= minMessages && !alreadyTriggered) {
  // 執行 smart extraction
  smartExtractionTriggeredForSession.set(sessionKey, true);
  messagesSinceLastExtract.set(sessionKey, 0);
} else if (alreadyTriggered && messagesSinceLastExtract.get(sessionKey) >= config.reExtractAfterNNewMessages) {
  // 再次觸發
  messagesSinceLastExtract.set(sessionKey, 0);
  // 執行 smart extraction
}

優點：比 binary flag 更精準（不lock out），比每輪都觸發更節省資源。

---

Q2: REPLACE vs APPEND Strategy

OpenCode 反對：REPLACE 會「丟資料」（turn N 的內容在 turn N+1 才抵達會被覆蓋）。
作者反擊：這是對滾動窗口設計的誤解——pendingIngressTexts 是同一個 conversationKey 的訊息佇列，每次 REPLACE 取窗口內全部歷史。

最終共識：維持 REPLACE + 加入 deduplication（用 sequence_id 防重）。

理由：REPLACE 配合 slice(-6) 滾動窗口是正確的，不丟資料。

---

Q3: Stale pending texts 防重機制

OpenCode 反對：sequence counter 邏輯有缺陷（decrement 到 0 時的 timing 問題），建議 timestamp。
作者反擊：timestamp 有 clock skew 問題，sequence counter 更確定性。

最終共識：兩者結合。

• sequence counter：確認「有沒有對應的 message_received 才處理」
• timestamp（輔助）：作為時間邊界，超過 5 分鐘視為 stale

---

Q4: extractMinMessages 語義改變

OpenCode 反對：用「bug fix」包裝 semantic change 危險，建議新增獨立參數。
作者反擊：舊語義在 DM 場景根本無法正常運作，新增參數讓使用者困惑。

最終共識：接受語義改變，在 CHANGELOG + README 明確說明新語義。

extractMinMessages=N = 累計 N 筆訊息後觸發 smart extraction

---

Q5: isExplicitRememberCommand Guard

OpenCode 反對：缺乏量化證據，不應直接移除。
作者反擊：在 REPLACE 模式下，數學上不可能同時滿足 guard 條件。

最終共識：Phase 1 已移除是正確的。補充 comment 說明原因。

---

額外發現（OpenCode 提出）

問題	嚴重性	備註
smartExtractor.filterNoiseByEmbedding 或 extractAndPersist 拋異常時 hook 行為未定義	中	建議加 try/catch
smart extraction 成功時 return，失敗時 continue——但 regex fallback 的行為假設可能不一致	低	需驗證

---

PR #534（Phase 1）已合併 ✅

Phase 1 清理已完成（createMockApi 接受第 5 參數 + 移除 dead guard）。

#534

---

Phase 2 待實作項目（按優先順序）

1. re-trigger 機制：連續新訊息計數（Q1）
2. stale 防重：sequence counter + timestamp 雙重（Q3）
3. REPLACE deduplication：sequence_id 去重（Q2）
4. 文件化：CHANGELOG + README 說明新語義（Q4）
5. Error handling：try/catch 包裝 smart extractor call

[jlin53882]

額外問題驗證結果

@AliceLJY 麻煩你看一下 這兩個問題確認看看需不需要修。
麻煩你了🙏

已驗證 OpenCode 提出的兩個額外問題，結論如下：

---

Issue A: Exception handling（✅ 確認存在）

Code 位置：index.ts ~line 2831-2842

const cleanTexts = await smartExtractor.filterNoiseByEmbedding(texts);
// 沒有 try/catch ↑

const stats = await smartExtractor.extractAndPersist(
  conversationText, sessionKey,
  { scope: defaultScope, scopeFilter: accessibleScopes },
);
// 沒有 try/catch ↑

結論：兩個 async call 都沒有 try/catch 包裝。如果任何一個 throw，整個 agent_end hook 會往上拋，可能導致上層（例如 OpenClaw plugin framework）捕獲並中斷後續流程。

建議：

let cleanTexts: string[] = [];
try {{
  cleanTexts = await smartExtractor.filterNoiseByEmbedding(texts);
}} catch (err) {{
  api.logger.error(`memory-lancedb-pro: filterNoiseByEmbedding failed: ${{err}}`);
  return;
}}

let stats = null;
try {{
  stats = await smartExtractor.extractAndPersist(...);
}} catch (err) {{
  api.logger.error(`memory-lancedb-pro: extractAndPersist failed: ${{err}}`);
  return;
}}

---

Issue B: Regex fallback 使用 texts 而非 cleanTexts（⚠️ 可能是設計意圖）

Code 位置：index.ts ~line 2869

api.logger.info(`...falling back to regex capture`);
// regex fallback：
const toCapture = texts.filter((text) => text && shouldCapture(text) && !isNoise(text));
//                     ↑ 用的是原始 texts，不是 cleanTexts

觀察：smart extraction 使用 cleanTexts（經過 embedding-based noise filter），regex fallback 使用 texts（原始未過濾）。這兩者不一致。

可能不是 bug：regex fallback 有自己的 !isNoise(text) 判斷（word-level noise filter），所以可能是有意讓 regex fallback 自己處理 noise。

建議：在 code comment 中說明這個設計意圖，避免未來 maintainer 誤解。

---

問題	驗證結論	嚴重性
Exception 未處理	✅ 確認存在，需加 try/catch	中
Regex fallback 用 texts 而非 cleanTexts	⚠️ 可能是設計意圖，需 comment 說明	低