Name: Cortex - PetalFlow Memory & Knowledge Service Type: Go service (MCP server) Created: 2026-03-04 Status: Development Phase - Phase 1 Complete
Cortex provides persistent context, vector-backed knowledge retrieval, and conversation memory for PetalFlow agents. It implements four memory primitives:
- Conversation Memory - Agent dialogue history
- Knowledge Store - Vector-indexed documents (RAG)
- Workflow Context - Shared state across tasks/runs
- Entity Memory - Auto-extracted knowledge graph
- Language: Go
- Storage: SQLite + vec0 (default), PostgreSQL + pgvector (production)
- Protocol: MCP (Model Context Protocol)
- Embeddings: Via Iris service
.project/petalflow-cortex-frd.md
- Read FRD document (2182 lines)
- Created detailed implementation plan (88 tasks across 4 phases)
- Created task checklist for tracking progress
- Project initialized
Milestone 1.1 Completed:
- Go module initialized
- Directory structure created
- Core dependencies added (cobra, viper, zap, uuid, sqlite3, pgx)
- Configuration system implemented with tests
- Shared type definitions created for all 4 memory primitives
Milestone 1.2 In Progress:
- Storage backend interface defined
- SQLite backend scaffold created
- Schema migrations implemented (all 15 tables)
- Conversation storage implemented with full test coverage
- AppendMessage, GetMessages, ListThreads, GetThread
- UpdateThread, DeleteThread, StoreMessageEmbedding
- MarkMessagesSummarized
- Knowledge storage implemented with full test coverage
- CreateCollection, GetCollection, ListCollections, DeleteCollection
- InsertDocument, GetDocument, DeleteDocument
- InsertChunks (batch with embeddings), GetAdjacentChunks
- CollectionStats, SearchChunks (placeholder for vec0)
- Context storage implemented with full test coverage
- GetContext, SetContext (with optimistic concurrency)
- ListContextKeys (with prefix filtering), DeleteContext
- GetContextHistory (version audit trail)
- CleanupExpiredContext, CleanupRunContext
- Entity storage implemented with full test coverage
- UpsertEntity, GetEntityByID, GetEntityByName, ResolveAlias
- ListEntities (with type filter and sorting), DeleteEntity
- InsertMention, GetMentions, MergeEntities
- UpsertRelationship, GetRelationships, RegisterAlias
- StoreEntityEmbedding, SearchEntities
- EnqueueExtraction, DequeueExtraction, CompleteExtraction, GetExtractionQueueStats
Milestone 1.2 Completed:
- sqlite-vec extension integrated for vector similarity search
- Added sqlite-vec-go-bindings CGO dependency
- Implemented binary encoding for embeddings (little-endian float32)
- SearchMessages: semantic search across conversation messages
- SearchChunks: semantic search across knowledge chunks with metadata filters
- SearchEntities: semantic search across entity summaries
- All search methods use vec_distance_cosine() for brute-force KNN
- Cosine distance converted to similarity score (0-1 range)
- Support for TopK, MinScore filtering, and type-specific filters
- Full test coverage added for vector search functionality
Milestone 1.3 Completed:
- Embedding provider interface defined
- Provider interface with Embed() and EmbedBatch() methods
- EmbeddingRequest/Response types for Iris API contract
- Iris embedding client implemented
- HTTP client for Iris /embeddings endpoint
- Batch support with configurable size limits
- Handles empty inputs and error responses
- LRU embedding cache implemented
- CachedProvider wraps any Provider with caching
- SHA-256 text hashing for cache keys
- Cache immutability to prevent mutation issues
- Stats and clear operations for monitoring
- Full test coverage with mock server tests
Milestone 1.4 Completed:
- Conversation engine implemented with all core operations
- Append: message creation with auto thread creation, content truncation, embedding generation
- History: retrieve messages with cursor pagination, optional summary
- Search: semantic search across messages using embeddings
- Clear: delete thread and all messages
- ListThreads: paginated thread listing
- GetThread/UpdateThread: thread metadata management
- MarkSummarized: track summarized messages
- Role validation (user, assistant, system, tool)
- Config-driven semantic search (can be disabled)
- Full test coverage (12 tests)
Milestone 1.5 Completed:
- Knowledge store engine implemented with all core operations
- Ingest: document ingestion with chunking and embedding generation
- Search: semantic search across chunks with context window expansion
- Get/Delete: document retrieval and removal
- Collections: create, list, get, delete with stats
- Text chunking strategies implemented
- Fixed: word-based chunking with configurable overlap
- Sentence: splits on sentence boundaries, cascades to fixed for long sentences
- Paragraph: splits on double newlines, cascades to sentence for long paragraphs
- Batch embedding generation during ingest
- Graceful degradation when embeddings fail
- Full test coverage (44 tests across chunker and engine)
Milestone 1.6 Completed:
- Workflow context engine implemented with all core operations
- Get/Set: key-value storage with version tracking
- Optimistic concurrency control via expected version
- TTL support with automatic expiration
- Run-scoped vs persistent context separation
- Key listing with prefix filtering and pagination
- Version history retrieval
- Merge operations with multiple strategies
- replace: simple value replacement
- append: array concatenation
- max/min: numeric comparison
- sum: numeric accumulation
- deep_merge: recursive map merging with array strategy options
- Full test coverage (31 tests)
Milestone 1.7 Completed:
- Entity memory engine implemented with all core operations
- Create/Get/Update/Delete: full CRUD with validation
- GetByName: lookup by canonical name
- Resolve: name-or-alias resolution (tries canonical name first, then aliases)
- List: paginated listing with optional type filter
- Search: semantic search across entity summaries using embeddings
- Alias management implemented
- AddAlias: register alternative names for entities
- Resolution prioritizes canonical name over aliases
- Relationship management implemented
- AddRelationship: create typed relationships between entities
- GetRelationships: retrieve relationships with direction filtering
- Supports bidirectional and unidirectional relationships
- Mention tracking implemented
- RecordMention: track entity mentions with source references
- GetMentions: retrieve mentions with pagination
- Automatic mention count tracking on entities
- Entity merging implemented
- Merge: combine two entities, preserving aliases and relationships
- Automatic alias creation from source entity name
- Attribute/metadata merging from both entities
- Extraction queue implemented
- EnqueueExtraction: add messages for entity extraction processing
- DequeueExtraction: retrieve next pending item for processing
- CompleteExtraction: mark items as processed (completed/failed)
- Stats: queue metrics for monitoring
- Query operation for combined entity retrieval
- Returns entity with relationships and recent mentions in single call
- Full test coverage (28 tests)
Milestone 1.8 Completed:
- MCP Server implemented with stdio transport
- Uses mark3labs/mcp-go v0.44.1 library
- All 16 tool definitions from FRD section 4.1 implemented
- Functional options pattern for tool parameters
- Conversation tools implemented
- conversation_append: add messages to threads
- conversation_history: retrieve messages with pagination
- conversation_search: semantic search across messages
- Knowledge tools implemented
- knowledge_ingest: document ingestion with chunking
- knowledge_search: semantic search with context window
- knowledge_collections: create/list/delete collections
- Context tools implemented
- context_get: retrieve values by key
- context_set: store values with TTL and versioning
- context_merge: merge values with strategy
- context_list: list keys with prefix filtering
- Entity tools implemented
- entity_query: lookup by name/alias with mentions
- entity_search: semantic search across entities
- entity_relationships: get entity relationships
- entity_update: modify entity attributes/aliases
- entity_merge: combine duplicate entities
- entity_list: paginated listing with filters
- Namespace enforcement via allowedNamespace config
- Validates namespace on every tool call
- Supports open mode (all namespaces) or restricted mode
- JSON result serialization for all tool responses
- Full test coverage (16 tests)
Files Created:
.project/IMPLEMENTATION_PLAN.md- Detailed plan with task descriptions.project/TASK_CHECKLIST.md- Quick-reference checklistgo.mod,go.sum- Go modulecmd/cortex/main.go- CLI entrypointinternal/cmd/root.go- Cobra root commandinternal/config/config.go- Configuration systeminternal/config/config_test.go- Config testsinternal/storage/backend.go- Storage interfaceinternal/storage/sqlite/sqlite.go- SQLite backendinternal/storage/sqlite/migrations.go- Schema migrationsinternal/storage/sqlite/conversation.go- Conversation storage opsinternal/storage/sqlite/conversation_test.go- Conversation testsinternal/storage/sqlite/knowledge.go- Knowledge storage opsinternal/storage/sqlite/knowledge_test.go- Knowledge testsinternal/storage/sqlite/context.go- Context storage opsinternal/storage/sqlite/context_test.go- Context testsinternal/storage/sqlite/entity.go- Entity storage opsinternal/storage/sqlite/entity_test.go- Entity testsinternal/storage/sqlite/vector_search_test.go- Vector search testsinternal/embedding/provider.go- Embedding provider interfaceinternal/embedding/iris.go- Iris HTTP client implementationinternal/embedding/cache.go- LRU embedding cacheinternal/embedding/provider_test.go- Embedding testsinternal/conversation/engine.go- Conversation memory engineinternal/conversation/engine_test.go- Conversation engine testsinternal/knowledge/chunker.go- Text chunking strategiesinternal/knowledge/chunker_test.go- Chunker testsinternal/knowledge/engine.go- Knowledge store engineinternal/knowledge/engine_test.go- Knowledge engine testsinternal/context/engine.go- Workflow context engineinternal/context/engine_test.go- Context engine testsinternal/entity/engine.go- Entity memory engineinternal/entity/engine_test.go- Entity engine testsinternal/server/mcp.go- MCP server implementationinternal/server/mcp_test.go- MCP server testspkg/types/*.go- Shared type definitions
Milestone 1.9 Completed:
- CLI serve command implemented
- Wires config, storage, embedding, and all engines
- Supports --namespace flag for restricted mode
- Supports --mcp flag (default true)
internal/cmd/serve.go- serve command implementation
Milestone 2.1 Completed:
- Summarization client implemented
- HTTP client for Iris /v1/completions endpoint
- SummarizeMessages convenience method
- System prompt for conversation summarization
- Conversation engine Summarize operation
- Splits messages into "to summarize" and "to keep"
- Stores summary in thread record
- Marks summarized messages
- Rolling summary incorporates previous summaries
- Auto-summarization trigger in History method
- Checks message count vs threshold
- Triggers summarization automatically when exceeded
- SkipAutoSummarize option for internal use
- MCP handler for conversation_summarize
- Accepts namespace, thread_id, keep_recent parameters
- Returns summary and message counts
- Serve command wires up summarizer
- Creates summarization client when Iris is configured
- Sets summarizer on conversation engine
- Full test coverage (new tests: 10+)
New Files:
internal/summarization/client.go- LLM completion clientinternal/summarization/client_test.go- Client tests
Milestone 2.2 Completed:
- Semantic chunking implemented with embedding-based topic detection
- SemanticChunker uses sliding windows of sentences
- Embeds each window and computes cosine similarity between adjacent windows
- Detects breakpoints where similarity drops below threshold
- Configurable via WithSimilarityThreshold() and WithWindowSize()
- Falls back to sentence chunking for short content or embedding failures
- Integrated with knowledge engine
- Engine dispatches to SemanticChunker when strategy is "semantic"
- Graceful fallback to sentence chunking on errors
- Semantic chunker only initialized when embedding provider is available
- Full test coverage (15+ new tests)
New Files:
internal/knowledge/semantic_chunker.go- Semantic chunker implementationinternal/knowledge/semantic_chunker_test.go- Semantic chunker tests
Milestone 2.3 Completed:
- Entity extraction client implemented
- LLM-powered entity extraction via Iris completion API
- JSON response parsing with validation
- Entity normalization and filtering
- Co-mentioned entity relationship extraction
- Name resolver implemented
- Blocking strategy (first 3 chars, initialism matching)
- Fuzzy matching using Levenshtein distance
- Multi-phase resolution: exact → alias → fuzzy → new
- Queue processor implemented
- Background worker with configurable polling interval
- Batch processing with retry and exponential backoff
- Support for extraction modes: off, sampled, whitelist, full
- ProcessSingle for on-demand extraction
- ExtractionEnqueuerAdapter for bridging engine interfaces
- Extraction hooks added to engines
- Conversation engine queues messages for extraction on append
- Knowledge engine queues chunks for extraction on ingest
- Fire-and-forget pattern - extraction failures don't block operations
- Serve command wiring
- Creates extractor, resolver, queue processor when Iris configured
- Sets extraction enqueuer on conversation and knowledge engines
- Starts queue processor in background
- Full test coverage (35+ new tests)
New Files:
internal/entity/extractor.go- Entity extraction clientinternal/entity/extractor_test.go- Extractor testsinternal/entity/resolver.go- Name resolution with blocking/fuzzyinternal/entity/resolver_test.go- Resolver testsinternal/entity/queue.go- Queue processor with enqueuer adapterinternal/entity/queue_test.go- Queue processor tests
Milestone 2.4 Completed:
- Context version history MCP handler implemented
context_historytool returns version history for a key- Supports run_id scoping, cursor pagination, and limit
- Version conflict detection already in place
expected_versionparameter on context_set and context_merge- Returns version conflict error on mismatch
- Full test coverage (2 new tests)
Modified Files:
internal/server/mcp.go- Added context_history tool and handlerinternal/server/mcp_test.go- Added context history tests
Milestone 2.5 Completed:
- CLI Management Commands implemented for all memory primitives
- Knowledge CLI: ingest, ingest-dir, search, stats, collections, create-collection
- Conversation CLI: history, append, search, list, clear, summarize
- Context CLI: get, set, delete, list, history, cleanup
- Entity CLI: create, get, delete, list, search, add-alias, add-relationship, merge, queue-stats
- Namespace CLI: stats, delete (with confirmation)
- All commands support --namespace flag for isolation
- JSON output support where applicable
New Files:
internal/cmd/knowledge.go- Knowledge store CLI commandsinternal/cmd/conversation.go- Conversation memory CLI commandsinternal/cmd/context.go- Workflow context CLI commandsinternal/cmd/entity.go- Entity memory CLI commandsinternal/cmd/namespace.go- Namespace management CLI commands
Milestone 3.1 Completed:
- PostgreSQL + pgvector backend implemented as production alternative to SQLite
- Full feature parity with SQLite backend for all storage operations
- Backend selection via config:
storage.backend: "pgvector"withstorage.database_url
Key Implementation Details:
- pgvector.go: Core Backend struct with pgxpool connection management
- migrations.go: PostgreSQL schema with pgvector extension, HNSW indexes
- conversation.go: Thread/message operations with semantic search
- knowledge.go: Collection/document/chunk operations with vector search
- context.go: Key-value context with version history and TTL
- entity.go: Entity extraction queue, relationships, alias resolution
- storage.go: Backend factory function for runtime selection
PostgreSQL-specific Features:
- HNSW vector indexes for efficient approximate nearest neighbor search
- TIMESTAMPTZ for proper timezone handling
- JSONB for metadata storage with native JSON operations
ON CONFLICT DO UPDATEfor upsertsFOR UPDATE SKIP LOCKEDfor atomic queue processing- pgx/v5 with connection pooling (25 max, 5 min connections)
CLI Backend Selection:
- All CLI commands updated to use shared
createStorage()factory - Supports "sqlite", "pgvector", "postgres", "postgresql" backend values
- Validates database_url is present when pgvector selected
New Files:
internal/storage/pgvector/pgvector.go- Backend struct and connection managementinternal/storage/pgvector/migrations.go- PostgreSQL schema (16 tables)internal/storage/pgvector/conversation.go- Conversation storage operationsinternal/storage/pgvector/knowledge.go- Knowledge storage operationsinternal/storage/pgvector/context.go- Context storage operationsinternal/storage/pgvector/entity.go- Entity storage operationsinternal/cmd/storage.go- Shared backend factory function
Milestone 3.2 Completed (SSE Transport):
- Server-Sent Events transport implemented for web-based MCP clients
- Supports both stdio (default) and SSE transport modes
- CLI flags:
--transport stdio|sseand--port 9810
Implementation Details:
ServeSSE(addr string)method added to MCP server- Uses mcp-go's NewSSEServer with keep-alive enabled
- SSE endpoint at
/ssefor event streaming - Message endpoint at
/messagefor client-to-server communication - Session ID generated per connection for stateful communication
Usage Examples:
# Default stdio transport
cortex serve
# SSE transport on port 9810
cortex serve --transport sse --port 9810
# SSE with namespace restriction
cortex serve --transport sse --port 9810 --namespace my-workflowModified Files:
internal/server/mcp.go- Added ServeSSE and SSEServer methodsinternal/cmd/serve.go- Added --transport and --port flags
Milestone 3.3 Completed (Observability):
- Prometheus metrics with all core metrics from FRD section 11.1
- Structured JSON logging with zap
- Metrics HTTP server with health endpoint
Prometheus Metrics Implemented:
cortex_operations_total- Counter by primitive, action, namespace, statuscortex_operation_duration_seconds- Histogram for operation latencycortex_search_latency_seconds- Histogram for vector search latencycortex_search_result_count- Histogram for result countscortex_embedding_latency_seconds- Histogram for embedding API latencycortex_embedding_requests_total- Counter for embedding API callscortex_extraction_queue_size- Gauge for pending queue itemscortex_extraction_dead_letter_count- Gauge for failed itemscortex_extraction_processed_total- Counter by statuscortex_storage_operation_duration_seconds- Histogram for storage ops
Structured Logging Features:
- JSON format when
structured_logging: true(default) - Context fields: request_id, namespace, thread_id, run_id, entity_id
- Duration tracking in milliseconds
- Error context propagation
- Configurable log level
Endpoints:
/metrics- Prometheus metrics (port 9811 default)/health- Health check returning "ok"
New Files:
internal/observability/metrics.go- Prometheus metrics definitionsinternal/observability/logging.go- Structured logging with zapinternal/observability/server.go- Metrics HTTP server
Modified Files:
internal/cmd/serve.go- Initialize logging and metrics on startup
Milestone 3.4 Completed (Retention & Garbage Collection):
- Background garbage collector with configurable intervals
- TTL cleanup runs on
ttl_cleanup_interval(default 60s) - Full GC runs on
gc_interval(default 24h)
GC Operations Implemented:
CleanupExpiredContext- Remove entries past TTL expirationDeleteOldConversations- Remove threads older than retention periodPruneStaleEntities- Remove entities with low mention counts and old last_seen_atDeleteOrphanedChunks- Remove chunks whose documents were deletedCleanupContextHistory- Remove old version history entriesCleanupOldRunContext- Remove old run-scoped context
CLI GC Command:
# Run all GC tasks
cortex gc --all
# Run specific tasks
cortex gc --expired-ttl --orphaned-chunks
# Dry run to see what would be deleted
cortex gc --dry-runNew Files:
internal/storage/backend.go- Added GarbageCollection interfaceinternal/storage/sqlite/gc.go- SQLite GC implementationinternal/storage/pgvector/gc.go- pgvector GC implementationinternal/gc/collector.go- Background GC workerinternal/cmd/gc.go- CLI gc command
Modified Files:
internal/cmd/serve.go- Start GC collector on server startup
Milestone 3.5 Completed (Backup & Export):
- SQLite online backup using SQLite3 backup API (hot backup, no downtime)
- JSON export for namespace data migration
Backup Features:
cortex backup --output backup.db- Create consistent SQLite backup- Progress reporting during backup
- PostgreSQL guidance (use pg_dump)
Export Features:
cortex export --namespace X --output export.json- Export namespace data- Options: --no-embeddings, --conversations-only, --knowledge-only, etc.
- Exports threads, messages, collections, documents, chunks, context, entities
New Files:
internal/storage/sqlite/backup.go- SQLite backup implementationinternal/storage/export.go- Export data structuresinternal/storage/sqlite/export.go- SQLite export implementationinternal/cmd/backup.go- CLI backup commandinternal/cmd/export.go- CLI export command
- Project Structure & Go Module
- Storage Interface & SQLite Backend
- Embedding Provider (Iris Integration)
- Conversation Memory Engine
- Knowledge Store Engine
- Workflow Context Engine
- Entity Memory Engine
- MCP Server
- CLI Serve Command
- Conversation Summarization
- Semantic Chunking
- Entity Extraction Pipeline
- Context Version History
- CLI Commands
- Embedding Cache (deferred - already have basic LRU cache)
- pgvector Backend
- SSE Transport
- Prometheus Metrics & Structured Logging
- Retention & Garbage Collection
- Backup & Export
- Hybrid Search (FTS5 + sqlite-vec, tsvector + pgvector)
- Bulk Ingest (concurrent workers, progress callbacks)
- Web Dashboard (Cortex-specific UI)
- Terminal UI (TUI with Bubble Tea)
None yet.
- Project follows the Cortex FRD specification exactly
- Each phase has verification tests before moving forward
- Phase 3 Production Hardening is complete
Milestone 4.1 Completed (Hybrid Search):
- FTS5 full-text search integrated with SQLite
- fts5 virtual tables for messages, chunks, and entities
- Triggers for automatic FTS index maintenance
- BM25 scoring combined with cosine similarity
- Three search modes: vector, fts, hybrid (RRF fusion)
- tsvector full-text search integrated with pgvector
- tsvector columns with GIN indexes
- ts_rank scoring combined with cosine similarity
- Same three search modes for feature parity
- MCP tools updated with search_mode parameter
New/Modified Files:
internal/storage/sqlite/fts.go- FTS5 schema and migrationsinternal/storage/sqlite/fts_search.go- Hybrid search implementationinternal/storage/pgvector/fts.go- tsvector schema and migrationsinternal/storage/pgvector/fts_search.go- Hybrid search implementationinternal/server/mcp.go- Updated search handlers for mode parameter
Milestone 4.2 Completed (Bulk Ingest):
- BulkIngest method added to knowledge engine
- Concurrent worker pool with configurable concurrency
- Progress callbacks for UI integration
- Batch processing with error collection
- MCP tool
knowledge_bulk_ingestfor batch document ingestion - Fixed SQLite in-memory concurrency issue (SetMaxOpenConns(1))
Milestone 4.3 Completed (Web Dashboard):
- Cortex-specific web dashboard (not PetalFlow integration)
- Go embedded templates and static files
- Dark theme with clean, minimal design
- HTMX-powered real-time search
Dashboard Features:
- Home: Overview with stats (collections, documents, conversations, entities)
- Knowledge: Collections list, collection detail, document viewer, search
- Conversations: Thread list, conversation detail with messages
- Context: Inspector placeholder (no list method available)
- Entities: Entity list, entity detail with relationships and mentions, search
New Files:
internal/dashboard/dashboard.go- Server, embedded files, template helpersinternal/dashboard/handlers.go- HTTP handlers for all pagesinternal/dashboard/static/style.css- Dark theme CSSinternal/dashboard/templates/*.html- 10 HTML templates
Dashboard Stack:
- Go html/template with embedded files (//go:embed)
- HTMX 1.9.10 for interactive search
- CSS custom properties for theming
- No JavaScript framework dependencies
Milestone 4.4 Completed (Terminal UI):
- Terminal UI implemented using Bubble Tea framework
- Dark theme matching the web dashboard colors
- Vim-style navigation (j/k for up/down)
- Number keys (1-5) for quick section switching
TUI Features:
- Dashboard: Overview with stats for all memory primitives
- Knowledge: Collection list with document counts, detail view
- Conversations: Thread list with messages viewer
- Context: Info page (no list API available)
- Entities: Entity list with relationships and mentions detail
New Files:
internal/tui/tui.go- Main app model, navigation, view routinginternal/tui/styles.go- Lipgloss styles matching web dashboard themeinternal/tui/helpers.go- Helper functions (timeAgo, truncate)internal/tui/dashboard.go- Dashboard view with stats loadinginternal/tui/knowledge.go- Knowledge browser with collection detailinternal/tui/conversation.go- Conversation viewer with messagesinternal/tui/entity.go- Entity explorer with relationships/mentionsinternal/cmd/tui.go- CLI command to launch TUI
TUI Stack:
- charmbracelet/bubbletea v1.3.10 - Elm architecture TUI framework
- charmbracelet/lipgloss v1.1.0 - Declarative terminal styling
- charmbracelet/bubbles v1.0.0 - Pre-built TUI components