Skip to content

Latest commit

 

History

History
244 lines (216 loc) · 10.9 KB

File metadata and controls

244 lines (216 loc) · 10.9 KB

Source Code Structure

Overview

MementoGraph is a Cargo workspace with five Rust crates and a React + TypeScript frontend embedded in the Tauri application crate. Each crate maps to one architectural layer, with memento-core providing shared types across all layers.

Top-Level Layout

MementoGraph/
├── Cargo.toml                    # Workspace manifest
├── Cargo.lock
├── .gitignore
├── README.md
├── docs/                         # Developer documentation
│   ├── source-code-structure.md  # This file
│   ├── cognitive-object-extraction-prompt-guide.md
│   └── ...
├── crates/
│   ├── memento-core/             # Shared types, traits, errors
│   ├── memento-ingestion/        # Layer 1: Ingestion Engine
│   ├── memento-extraction/       # Layer 2: Semantic Extraction
│   ├── memento-graph-engine/     # Layer 3: Graph Engine
│   └── memento-app/              # Layer 4: Tauri binary + frontend
└── .kiro/
    └── specs/                    # Feature specifications

Crate Details

crates/memento-core/

Shared domain types, traits, and error definitions. Every other crate depends on this. Contains no business logic — only data structures and interfaces.

memento-core/
├── Cargo.toml
└── src/
    ├── lib.rs                    # Re-exports all modules
    ├── types/
    │   ├── mod.rs
    │   ├── content.rs            # ContentSource, Priority, IngestionRequest, IngestionAck
    │   ├── chunk.rs              # Chunk, ChunkId, ContentHash, SourceReference, SourceLocation
    │   ├── cognitive_object.rs   # CognitiveObject, Entity, Concept, Claim, Relationship, RelationshipLabel
    │   ├── graph.rs              # GraphNode, NodeId, NodeType, GraphEdge, EdgeId, ClusterId
    │   └── config.rs             # Application configuration types
    ├── traits/
    │   ├── mod.rs
    │   ├── parser.rs             # ContentParser trait
    │   ├── chunker.rs            # Chunker trait
    │   ├── inference.rs          # InferenceBackend trait
    │   └── graph_store.rs        # GraphStore trait
    ├── errors/
    │   ├── mod.rs
    │   ├── ingestion.rs          # IngestionError
    │   ├── extraction.rs         # ExtractionError
    │   └── graph.rs              # GraphError
    └── schema/
        └── cognitive_object.json # JSON schema for constrained decoding

crates/memento-ingestion/

Layer 1: Content parsing, semantic chunking, embedding generation, caching, and the async job queue.

memento-ingestion/
├── Cargo.toml
└── src/
    ├── lib.rs
    ├── queue/
    │   ├── mod.rs
    │   └── job_queue.rs          # Bounded async job queue (capacity 64)
    ├── parsers/
    │   ├── mod.rs
    │   ├── registry.rs           # ParserRegistry dispatch
    │   ├── pdf.rs                # PDF parser
    │   ├── youtube.rs            # YouTube transcript fetcher
    │   ├── epub.rs               # EPUB parser
    │   ├── web.rs                # Web page article extractor
    │   ├── rss.rs                # RSS feed poller
    │   ├── newsletter.rs         # Newsletter/email parser
    │   ├── markdown.rs           # Markdown file reader
    │   └── github.rs             # GitHub repo cloner/ingester
    ├── chunker/
    │   ├── mod.rs
    │   ├── semantic.rs           # Semantic boundary detection chunker
    │   └── tokenizer.rs          # Token counting utilities
    ├── embedding/
    │   ├── mod.rs
    │   ├── generator.rs          # Local embedding model wrapper (batched)
    │   └── cache.rs              # Content-hash → embedding file-backed cache
    └── pipeline.rs               # Async pipeline orchestration (parser → chunker → embedder → channel)

crates/memento-extraction/

Layer 2: LLM inference backends, constrained decoding, extraction orchestration, and model hot-swapping.

memento-extraction/
├── Cargo.toml
└── src/
    ├── lib.rs
    ├── backends/
    │   ├── mod.rs
    │   ├── llamacpp.rs           # llama.cpp backend adapter
    │   ├── ollama.rs             # Ollama backend adapter
    │   └── chatgpt.rs            # ChatGPT-compatible API adapter
    ├── decoding/
    │   ├── mod.rs
    │   └── constrained.rs        # Grammar/schema enforcement for structured output
    ├── orchestrator.rs           # Extraction loop: receive chunks, call backend, emit CognitiveObjects
    ├── model_swap.rs             # Hot-swap logic with graceful transition
    ├── prompt.rs                 # Prompt templates and few-shot examples
    └── novelty.rs                # Novelty signal detection (graph comparison)

crates/memento-graph-engine/

Layer 3: SurrealDB operations, node/edge CRUD, relationship inference, community detection, PageRank, and temporal weighting.

memento-graph-engine/
├── Cargo.toml
└── src/
    ├── lib.rs
    ├── store/
    │   ├── mod.rs
    │   ├── surrealdb.rs          # GraphStore implementation backed by SurrealDB
    │   ├── nodes.rs              # Node upsert, merge, normalization
    │   ├── edges.rs              # Edge creation, reinforcement, dormancy
    │   └── queries.rs            # Search, contradictions, cluster queries, timeline
    ├── inference/
    │   ├── mod.rs
    │   ├── similarity.rs         # Cosine similarity computation via USearch
    │   ├── community.rs          # Louvain community detection
    │   └── pagerank.rs           # PageRank variant for relevance propagation
    ├── temporal/
    │   ├── mod.rs
    │   ├── weighting.rs          # Combined weight calculation (recency + frequency + similarity)
    │   └── decay.rs              # Exponential decay and dormancy management
    ├── processor.rs              # CognitiveObject consumer: channel → node/edge writes
    └── schema.rs                 # SurrealDB schema initialization and migrations

crates/memento-app/

Layer 4: Tauri application binary, command handlers, event emission, and the React frontend.

memento-app/
├── Cargo.toml
├── tauri.conf.json               # Tauri configuration
├── build.rs                      # Tauri build script
├── src/
│   ├── main.rs                   # Tauri app entry point, runtime initialization
│   ├── commands/
│   │   ├── mod.rs
│   │   ├── ingestion.rs          # submit_content, get_job_status
│   │   ├── graph.rs              # get_graph_snapshot, get_node_detail, search_nodes, etc.
│   │   └── config.rs             # configure_backend, privacy settings
│   ├── events.rs                 # Tauri event emission (node-added, clustering-complete, etc.)
│   ├── state.rs                  # Application state management (DB handles, channels, config)
│   └── privacy.rs                # Network request filtering and logging
└── frontend/
    ├── package.json
    ├── tsconfig.json
    ├── vite.config.ts
    ├── index.html
    └── src/
        ├── main.tsx              # React entry point
        ├── App.tsx               # Root component, routing
        ├── api/
        │   ├── commands.ts       # Typed Tauri command wrappers
        │   └── events.ts         # Tauri event listeners and types
        ├── components/
        │   ├── graph/
        │   │   ├── GraphCanvas.tsx       # Sigma.js/Cosmograph WebGL renderer
        │   │   ├── NodeDetail.tsx        # Node click detail panel
        │   │   ├── ClusterLabel.tsx      # Hover label component
        │   │   └── TimeSlider.tsx        # Graph evolution replay control
        │   ├── search/
        │   │   └── SemanticSearch.tsx    # Search input and results
        │   ├── explorer/
        │   │   ├── ContradictionExplorer.tsx
        │   │   ├── InterestDrift.tsx
        │   │   └── Digest.tsx
        │   ├── ingestion/
        │   │   ├── SubmitContent.tsx     # File picker, URL input, paste area
        │   │   └── JobStatus.tsx         # Queue status and notifications
        │   └── settings/
        │       └── BackendConfig.tsx     # Model/backend configuration panel
        ├── hooks/
        │   ├── useGraph.ts              # Graph data fetching and state
        │   ├── useIngestion.ts          # Ingestion submission and status
        │   └── useEvents.ts             # Tauri event subscription hooks
        ├── types/
        │   └── index.ts                 # TypeScript type definitions mirroring Rust types
        └── styles/
            └── ...                      # CSS/Tailwind styles

Conventions

  • One module per concern — Each file handles one responsibility. If a file exceeds ~300 lines, split it.
  • Traits in memento-core — All cross-crate interfaces are defined as traits in core. Implementations live in their respective crates.
  • Errors are typed — Each crate has its own error enum. Use thiserror for derivation. Propagate with ? and From impls.
  • Tests live next to code — Unit tests go in #[cfg(test)] mod tests at the bottom of each file. Integration tests go in crates/<name>/tests/.
  • Frontend mirrors backend structure — TypeScript types in frontend/src/types/ should stay in sync with Rust types in memento-core.

Runtime Data Directory

Separate from source code, the application stores user data at:

~/.memento-graph/
├── data/
│   └── surreal/              # SurrealDB embedded data files
├── vectors/
│   ├── nodes.usearch         # Node embedding HNSW index
│   └── chunks.usearch        # Chunk embedding HNSW index
├── cache/
│   └── embeddings/           # Content-hash → embedding (bincode files)
├── models/
│   ├── extraction/           # LLM model files (.gguf)
│   └── embedding/            # Embedding model files
└── config.toml               # User configuration (backend, thresholds, intervals)

Dependency Flow

memento-app
├── memento-ingestion
│   └── memento-core
├── memento-extraction
│   └── memento-core
├── memento-graph-engine
│   └── memento-core
└── memento-core

No circular dependencies. Each layer crate depends only on memento-core. The app crate depends on all layer crates and wires them together.