fix: Improve memory storage to prevent incorrect facts and redundancies

This commit addresses several issues with memory extraction and storage:

1. **Distinguish user facts from assistant statements**: Updated the extraction
   prompt to only extract facts explicitly stated by the user, not speculative
   statements or assumptions made by the assistant. This prevents the system
   from storing assistant guesses as if they were user-provided facts.

2. **Enhanced graph cleanup for redundancy removal**: Improved the cleanup
   prompt with detailed instructions for identifying and removing:
   - Redundant/duplicate nodes
   - Speculative or incorrect information
   - Outdated information
   - Contradictions (keeping most recent/specific information)

3. **Atlas-based contradiction detection**: Modified the cleanup process to
   use the User Atlas as the source of truth. When user corrections are made
   (reflected in the atlas), the nightly cleanup now detects and removes
   contradicting graph nodes.

4. **Fixed missing import**: Added missing `env` import in routes/dream.ts

These changes ensure that:
- Only user-stated facts are extracted and stored
- The nightly cleanup actively removes redundant and incorrect information
- User corrections (via atlas updates) propagate to the graph during cleanup
- The memory system maintains higher accuracy and data quality over time

Author: claude

src/lib/extract-graph.ts

@@ -122,17 +122,28 @@ Extract the graph from the following ${sourceType}:
 ${content}
 </${sourceType}>
 
+CRITICAL EXTRACTION RULES - READ CAREFULLY:
+
+When extracting from conversations:
+- ONLY extract facts that the USER explicitly stated, confirmed, or provided
+- DO NOT extract speculative statements, suggestions, or assumptions made by the assistant
+- DO NOT treat assistant's questions as facts (e.g., "Are you working on X?" is NOT a fact that the user is working on X)
+- DO NOT extract assistant's interpretations unless the user confirmed them
+- If the assistant makes a statement and the user corrects it, ONLY extract the user's correction as fact
+- Prioritize user's own statements about themselves, their experiences, preferences, and circumstances
+- Be especially cautious with information only mentioned by the assistant - verify if the user confirmed it
+
 Extract, for example, the following elements:
-1. People mentioned (real or fictional)
-2. Locations discussed
-3. Events that occurred or were mentioned
-4. Objects or items of significance
-5. Emotions expressed or discussed
-6. Concepts or ideas explored
-7. Media mentioned (books, movies, articles, etc.)
-8. Temporal references (dates, times, periods)
-9. The assistant's emotions and feelings
-10. The assistant's internal insights or discoveries about the user
+1. People mentioned by the user (real or fictional)
+2. Locations the user discussed or mentioned
+3. Events that the user stated occurred or experienced
+4. Objects or items the user mentioned as significant
+5. Emotions the user expressed or discussed
+6. Concepts or ideas the user explored or mentioned
+7. Media the user mentioned (books, movies, articles, etc.)
+8. Temporal references the user provided (dates, times, periods)
+9. The user's preferences, goals, projects, and plans
+10. Facts the user stated about other people or entities
 
 For each element, create a node with:
 - A unique temporary ID (format: "temp_[type]_[number]", e.g., "temp_person_1") if it's a NEW node.
@@ -142,6 +153,7 @@ For each element, create a node with:
 
 Then, link these nodes with edges.
 - Edges are mainly used to represent "facts" about nodes. For example, if you have a Person node and an Event node, you can create an edge from the Person node to the Event node to represent the fact that the person participated in the event.
+- ONLY create edges for facts explicitly stated by the user, not assistant assumptions
 - Edges are unique by source node, target node, and edge type
 - In the edge description for facts, give a succinct description of the fact. Add some minimal context to aid retrieval, but keep it concise.
 - Ideally, edges link to already-existing nodes. If the node isn't existing, create it.
@@ -153,11 +165,11 @@ Rules of the graph:
 - Omit unnecessary details in node names, eg. "John Doe" instead of "John Doe (person)"
 - Nodes are independent of context and represent a *single* thing. Bad example: "John - the person taking a walk". Good example: "John" (Person node, no description) linked to [PARTICIPATED_IN] "John's walk on 2025-05-18" (Event node), linked to [OCCURRED_ON] "2025-05-18" (Temporal node).
 - Don't create nodes for things that should be represented by edges.
-
+- Avoid redundant or duplicate information - if a fact is already represented, don't create another node or edge for it
 
 Then create edges between these nodes to represent their relationships using the appropriate edge types.
 
-Focus on extracting the most significant and meaningful information. Quality is more important than quantity.`;
+Focus on extracting the most significant and meaningful information that the USER provided. Quality and accuracy are more important than quantity.`;
 
   const completion = await client.beta.chat.completions.parse({
     messages: [{ role: "user", content: prompt }],

src/lib/jobs/cleanup-graph.ts

@@ -369,6 +369,12 @@ async function proposeGraphCleanup(
   modelId: string,
 ): Promise<CleanupProposal> {
   const client = await createCompletionClient(userId);
+
+  // Fetch user atlas for context about user corrections and current state
+  const { getAtlas } = await import("../atlas");
+  const db = await useDatabase();
+  const { description: userAtlas } = await getAtlas(db, userId);
+
   const nodesList = temp.nodes
     .map(
       (n) =>
@@ -381,9 +387,59 @@ async function proposeGraphCleanup(
         `<edge source="${e.sourceTemp}" target="${e.targetTemp}" type="${e.type}">${e.description}</edge>`,
     )
     .join("\n");
-  const prompt = `You are a graph cleaning assistant. Given this subgraph, propose merges (pairs of temp IDs to merge), deletes (temp IDs to remove), additions (new edges, each with a concise description of its meaning), and any new nodes. Delete any unclear or non-useful nodes entirely and ensure all edges linked to a deleted node are removed.
+  const prompt = `You are a graph cleaning assistant. Your task is to analyze this subgraph and propose improvements to ensure accuracy, remove redundancies, and maintain data quality.
+${
+  userAtlas
+    ? `
+**User Atlas Context:**
+The following is the current User Atlas, which represents the most up-to-date factual information about the user. Use this to identify any nodes in the graph that contradict or are outdated compared to the atlas.
+
+<user_atlas>
+${userAtlas}
+</user_atlas>
+`
+    : ""
+}
+
+**Critical Cleaning Rules:**
+
+1. **Check Against User Atlas**: If a User Atlas is provided above, use it as the source of truth:
+   - Delete any nodes that contradict information in the atlas
+   - Remove nodes that represent outdated information superseded by the atlas
+   - The atlas reflects user corrections and the most current factual information
+
+2. **Remove Redundant Nodes**: Identify and merge nodes that represent the same entity or concept. Look for:
+   - Duplicate entities with slightly different labels (e.g., "John Smith" and "John")
+   - Multiple nodes describing the same event or concept
+   - Nodes that could be consolidated without losing information
+
+3. **Delete Incorrect or Speculative Information**: Remove nodes that represent:
+   - Speculative or assumed information (not explicitly stated facts)
+   - Outdated information that has been superseded or contradicted
+   - Unclear or non-useful nodes with vague descriptions
+   - Nodes that don't represent factual information about the user
+
+4. **Identify Contradictions**: When you find contradicting information:
+   - Keep the most recent or most specific information
+   - Delete older or vaguer contradicting nodes
+   - Prefer user-stated facts over inferred information
+   - Always prefer atlas information over graph nodes when they conflict
+
+5. **Improve Connections**: Add missing edges between related nodes that should be connected
+
+6. **Remove Redundant Edges**: Don't create edges that duplicate existing relationships
+
+**Your Response Should Include:**
+- **merges**: Pairs of temp IDs where nodes are duplicates (remove will be merged into keep)
+- **deletes**: Temp IDs of nodes to completely remove (speculative, outdated, or unclear)
+- **additions**: New edges to add (with concise, factual descriptions)
+- **newNodes**: Any new nodes needed (only if genuinely missing and factual)
+
+**Important**: Be aggressive about removing redundant and speculative information. Quality and accuracy are more important than quantity.
+
 Nodes:
 ${nodesList}
+
 Edges:
 ${edgesList}
 `;

src/routes/dream.ts

@@ -3,6 +3,7 @@ import { defineEventHandler, readBody } from "h3";
 import { CleanupGraphParams } from "~/lib/jobs/cleanup-graph";
 import { batchQueue, DreamJobData, flowProducer } from "~/lib/queues";
 import { dreamRequestSchema, dreamResponseSchema } from "~/lib/schemas/dream";
+import { env } from "~/utils/env";
 
 export default defineEventHandler(async (event) => {
   const { userId, assistantId, assistantDescription } =