feat:Add Policy-Based Memory Storage, Retrieval, and Chat Integration

[esafwan]

Overview

This PR represents a comprehensive integration that brings both the advanced backend Scoped Memory Architecture (merged from feature/scoped-memory-implementation) and a completely overhauled Dynamic Two-Level Navigation & Sidebar System together with high-fidelity frontend UI dashboards and live chat integrations.

It provides agents with transparent, policy-governed memory recall/saving mechanisms, adds a full-featured Memory Management Dashboard, and introduces responsive, Linear-style accordion and flyout popover sidebar rails for high-end aesthetic value and intuitive user navigation.

---

Key Components

1. Scoped Memory Backend & DB Framework

• Doctypes Integrated: Added Memory Record and Memory Policy doctypes.
• Preset Policies: Created database seeds for four policy models (Conservative, Conversational, Research, Operational) that specify records capacity, token budgets, visibility checks, auto-promotions, and approval steps.
• Search & Storage APIs: Exposed robust whitelisted server methods:
  • save_memory_record: Performs validation against active agent policy constraints (allowed record types, confidence, importance scores).
  • search_memory_records: Implements hybrid searching across scopes (User, Agent, Conversation, Site, Global) ordered by importance/recency.
  • get_memory_record and archive_memory_record: Governs secure retrieval and archiving.
  • promote_memory_to_knowledge: Seamlessly projects structured memory chunks to semantic sqlite-fts or Chroma vector spaces.
• Background Jobs: Integrated background extraction worker run_background_memory_extraction scheduled automatically at the end of each message stream to ingest new facts and preferences asynchronously.

2. High-Fidelity Memory UI & Chat Integration

• Memory Dashboard (/memory): Built a fully responsive page under the Knowledge submenu that displays all active memories using unified custom cards.
• Memory Card (MemoryCard.tsx): Integrates visual badges and unique icons tailored to the specific memory category (Fact, Preference, Instruction, Context, Reflection), shows importance scores, confidence gauges, and supports immediate archive operations with smooth micro-animations.
• Chat Context Badges (MemoryContextBadge.tsx): Injected context badges within ChatMessage.tsx displaying next to response timestamps. Clicking the badge opens a lazy-loaded popover detailing the exact facts or preferences the AI utilized to construct its personal reply.

3. Dynamic Two-Level Navigation Sidebar

• Unified Sidebar Structure (app-sidebar.tsx): Restructured navigation hierarchy by grouping the table builder (Tables), file ingestion (Documents), and memory into a single, cohesive "Knowledge" section.
• Accordion Toggle Behavior (nav-main.tsx): Implemented a smooth accordion control where expanding one section (e.g., Settings) automatically collapses any other active group, avoiding cluttered navigation rails.
• Linear-Style Flyout Popovers: Added a highly premium popover drawer for collapsed sidebar states. When the rail is minimized, hovering over or clicking a parent icon triggers a side-aligned flyout menu showing children without jarring layouts.

---

Verification & QA

• TypeScript & Compilation: Executed yarn build on host with zero errors. All type definitions, schemas, popover handlers, and API states are 100% correct.
• Frappe Asset Linking: Executed bench build cleanly linking frontend distribution bundles to Frappe's public assets.
• App Integration Unit Tests: Executed the bench run-tests --app huf suite in the devcontainer, returning 100% success on all database integration, backend knowledge tool, and schema validations.
• Lifecycle API Testing: Validated the complete data flow using direct database insertions and queries inside the devcontainer:
  • Executed save_memory_record to create a Preference user memory, which successfully persisted to MariaDB under user scope.
  • Executed search_memory_records matching the custom query and confirming importance-sorted returns.
  • Executed archive_memory_record setting the record state to Archived safely.