Skip to content

Commit bdd5f04

Browse files
whorne89claude
andauthored
fix: extraction model + prompt tuning + seed format (#169)
* fix: switch extraction to gate model and tighten prompt to reduce junk memories Extraction was using gpt-5.2 via generateText() instead of gpt-4.1-nano. The smart model was extracting memories from casual one-off questions (e.g. "what happened to chuck norris"). Now uses GATE_MODEL directly like selection does. Also strengthened the extraction prompt — NONE is the default, asking a question is not energy, added concrete NONE examples. Seed SQL reformatted to match runtime extraction style. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: remove seed SQL from repo Steve will run it separately — doesn't belong in the codebase. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: add optional model param to generateText instead of accessing openai client directly Per Steve's review — extraction and selection now use generateText() with an optional model override instead of reaching into openAiService.openai. Web search tools are skipped when using a custom model (gate model doesn't need them). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 03c2ec4 commit bdd5f04

File tree

4 files changed

+20
-280
lines changed

4 files changed

+20
-280
lines changed

packages/backend/src/ai/ai.constants.ts

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,8 @@ about the people in this conversation that would be worth remembering for future
106106
The participant named "Moonbeam" (or "muzzle3") is the bot. You can see its messages for context (to understand
107107
what humans were reacting to), but extract observations about the HUMANS only.
108108
109+
YOUR DEFAULT ANSWER IS NONE. Only extract something if you are confident it meets the criteria below.
110+
109111
WHAT TO EXTRACT:
110112
- Specific statements or positions someone argued with conviction
111113
- How someone interacts with Moonbeam and others (telling it off, asking it to settle arguments, testing it,
@@ -115,6 +117,7 @@ WHAT TO EXTRACT:
115117
116118
WHAT TO SKIP:
117119
- Idle chatter, one-liners, greetings, link shares without commentary
120+
- Someone asking a question — asking about a topic is NOT the same as caring about it
118121
- Names of partners, kids, or family members (e.g. "his wife Katie", "her son Jake")
119122
- Addresses, workplaces, or job titles (e.g. "works at Capital One", "lives in Cranford")
120123
- Medical info (e.g. "diagnosed with ADD", "had hernia surgery")
@@ -127,12 +130,22 @@ HOW TO DECIDE:
127130
Look for energy. Did someone care enough to write more than a sentence? Did they argue back and forth? Did they
128131
directly engage with Moonbeam or another person? If the conversation is just casual banter, the answer is NONE.
129132
133+
A single question to Moonbeam is NOT energy. Someone asking "what happened to chuck norris" is idle curiosity, not
134+
a memorable observation. You need to see sustained engagement — multiple messages, a debate, a strong reaction,
135+
someone going off about something they care about.
136+
137+
EXAMPLES OF NONE (do not extract from conversations like these):
138+
- Someone asks Moonbeam a factual question and gets an answer
139+
- Someone shares a link with no commentary
140+
- A few people exchange short one-liners or greetings
141+
- Someone makes a single joke or observation and moves on
142+
130143
EXISTING MEMORIES (for context — do not duplicate these):
131144
{existing_memories}
132145
133146
For each observation, classify:
134147
- NEW: not captured in existing memories
135-
- REINFORCE: an existing memory came up again
148+
- REINFORCE: an existing memory came up again — only if the conversation shows genuine sustained engagement with the topic, not just a passing mention
136149
- EVOLVE: contradicts or meaningfully updates an existing memory
137150
138151
Return a JSON array, or the string NONE if nothing is worth extracting. Most of the time, NONE is the right answer.

packages/backend/src/ai/ai.service.ts

Lines changed: 3 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -25,10 +25,6 @@ import {
2525
import { MemoryPersistenceService } from './memory/memory.persistence.service';
2626
import { MemoryWithSlackId } from '../shared/db/models/Memory';
2727
import { logger } from '../shared/logger/logger';
28-
import {
29-
ResponseOutputMessage,
30-
ResponseOutputText,
31-
} from 'openai/resources/responses/responses';
3228
import { SlackService } from '../shared/services/slack/slack.service';
3329
import { MuzzlePersistenceService } from '../muzzle/muzzle.persistence.service';
3430
import { OpenAIService } from './openai/openai.service';
@@ -348,18 +344,7 @@ export class AIService {
348344
.replace('{all_memories_grouped_by_user}', formattedMemories);
349345

350346
try {
351-
const response = await this.openAiService.openai.responses.create({
352-
model: GATE_MODEL,
353-
input: prompt,
354-
});
355-
356-
const textBlock = response.output.find(
357-
(block): block is ResponseOutputMessage => block.type === 'message',
358-
);
359-
const outputText = textBlock?.content?.find(
360-
(block): block is ResponseOutputText => block.type === 'output_text',
361-
);
362-
const raw = outputText?.text?.trim();
347+
const raw = await this.openAiService.generateText(prompt, 'selection', undefined, GATE_MODEL);
363348

364349
if (!raw) return [];
365350

@@ -467,10 +452,10 @@ export class AIService {
467452
const extractionInput = `${conversationHistory}\n\nMoonbeam: ${moonbeamResponse}`;
468453
const prompt = MEMORY_EXTRACTION_PROMPT.replace('{existing_memories}', existingMemoriesText);
469454

470-
const result = await this.openAiService.generateText(extractionInput, 'extraction', prompt);
455+
const result = await this.openAiService.generateText(extractionInput, 'extraction', prompt, GATE_MODEL);
471456

472457
if (!result) {
473-
this.aiServiceLogger.warn('Extraction returned no result from generateText');
458+
this.aiServiceLogger.warn('Extraction returned no result');
474459
return;
475460
}
476461

packages/backend/src/ai/openai/openai.service.ts

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,11 @@ export class OpenAIService {
1212
apiKey: process.env.OPENAI_API_KEY,
1313
});
1414

15-
generateText = (text: string, userId: string, instructions?: string) => {
15+
generateText = (text: string, userId: string, instructions?: string, model?: string) => {
1616
return this.openai.responses
1717
.create({
18-
model: GPT_MODEL,
19-
tools: [{ type: 'web_search_preview' }],
18+
model: model || GPT_MODEL,
19+
...(model ? {} : { tools: [{ type: 'web_search_preview' }] }),
2020
instructions: instructions,
2121
input: text,
2222
user: `${userId}-DaBros2016`,

0 commit comments

Comments
 (0)