[INIT] Complete road map

architecture/section_06_knowledge_hub_architecture.md

@@ -0,0 +1,221 @@
+# Section 6: Knowledge Hub Architecture
+
+> **ProjectOS** — AI-powered project management and knowledge platform
+> This section details the continuous ingestion pipeline that turns a connected repository, its tickets, its CI config, and project meetings into the queryable brain behind the Onboarding Assistant (FR-7.1–FR-7.9, FR-5.7).
+
+---
+
+## 6.1 Scope and Boundary
+
+The Knowledge Hub is the **write path** into `proj_{project_id}_knowledge` (the ChromaDB collection dedicated to code/docs/tickets/CI/meetings, per ADR-2 in §3.6). It owns:
+
+- Repository connection and webhook registration (FR-7.1, FR-7.4).
+- Initial full index and incremental re-index (FR-7.2, FR-7.4).
+- Multi-source ingestion: code, docs, issues/PRs, CI/CD definitions, Azure DevOps work items (FR-7.5–FR-7.7).
+- Chunk metadata persistence in Postgres `knowledge_chunks` (§5.3.6) and vector persistence in ChromaDB.
+- Index health reporting (FR-7.9).
+
+It does **not** own retrieval or answer generation — that's the RAG pipeline and Onboarding Assistant, covered in Section 7 and Section 8. The Knowledge Hub's only contract with those layers is: "every chunk it writes is embedded, tagged with accurate metadata, and filterable by `project_id` and `chunk_type`."
+
+---
+
+## 6.2 Source Connectors
+
+| Source | Connector | Auth | Trigger | FR |
+|---|---|---|---|---|
+| GitHub repo files | `PyGithub` (`Repository.get_contents`, `get_git_tree`) | GitHub App installation token or PAT (encrypted, `integrations.encrypted_credentials`) | Initial connect + push webhook | FR-7.1, FR-7.2 |
+| GitHub Issues & PRs | `PyGithub` (`get_issues`, `get_pulls`, comment threads) | Same as above | Initial connect + `issues`/`pull_request` webhook | FR-7.5 |
+| CI/CD definitions | `PyGithub` file read on `.github/workflows/*.yml` | Same as above | Same as code files (push webhook covers workflow file changes) | FR-7.6 |
+| Azure DevOps repos & work items | `azure-devops` Python SDK (`GitClient`, `WorkItemTrackingClient`) | Azure DevOps PAT (encrypted) | Initial connect + Azure DevOps service hook | FR-7.7 |
+| Meeting summaries | Internal — `MeetingService` calls `IngestionService.ingest_meeting()` directly after summarization completes | N/A (internal call, not external API) | On meeting summarization completion | FR-5.7 |
+
+Each connector implements a common interface so the ingestion pipeline doesn't special-case providers past the fetch step:
+
+```python
+# app/services/knowledge_hub/connectors/base.py
+class SourceConnector(Protocol):
+    async def list_files(self, repo: Repository) -> list[FileRef]: ...
+    async def fetch_content(self, repo: Repository, file_ref: FileRef) -> bytes: ...
+    async def list_changed_files(self, repo: Repository, before_sha: str, after_sha: str) -> list[FileRef]: ...
+```
+
+`GitHubConnector` and `AzureDevOpsConnector` both implement this; `list_changed_files` is what makes FR-7.4's incremental re-index possible without a full tree walk on every push.
+
+---
+
+## 6.3 Ingestion Pipeline Flow
+
+```mermaid
+flowchart TD
+    A[Trigger] -->|initial connect| B[Full Repo Walk]
+    A -->|GitHub push webhook| C[Diff Changed Files\nlist_changed_files before_sha→after_sha]
+    A -->|issues/pull_request webhook| D[Fetch Issue/PR Body + Comments]
+    A -->|Azure DevOps service hook| E[Fetch Work Item / Repo Diff]
+    A -->|meeting summarized| F[MeetingService.ingest_meeting]
+
+    B --> G{File Type Router}
+    C --> G
+    D --> H[Ticket Chunker\nsingle chunk: title + body + comments]
+    E --> H
+    F --> I[Meeting Chunker\nfixed-size, 500-token windows\nwith 50-token overlap]
+
+    G -->|*.py .js .ts .go .java etc| J[tree-sitter Chunker\nfunction/class boundaries]
+    G -->|*.md README wiki| K[Markdown Chunker\nsplit by heading, max 800 tokens]
+    G -->|.github/workflows/*.yml\nazure-pipelines.yml| L[YAML Chunker\nwhole-file if under 2000 tokens,\notherwise split by top-level job/stage key]
+    G -->|binary / >5MB / .lock .min.js| M[Skip — record in\ningestion_jobs.error_log as 'skipped_unsupported']
+
+    J --> N[Chunk Batch\ngroups of 20]
+    K --> N
+    L --> N
+    H --> N
+    I --> N
+
+    N --> O[Voyage AI Embed\nvoyage-code-2 for code chunks,\ntext-embedding-3-small for doc/ticket/meeting chunks]
+    O --> P{Embed Success?}
+    P -->|yes| Q[Upsert to ChromaDB\ncollection proj_id_knowledge]
+    P -->|no, retryable| R[Exponential Backoff Retry\n5s → 300s max, 5 attempts]
+    R --> O
+    P -->|no, exhausted retries| S[Record in ingestion_jobs.error_log\nstatus=failed, reason=embedding_error]
+
+    Q --> T[Write knowledge_chunks row\nchroma_id, file_path, commit_sha, metadata]
+    T --> U[Update ingestion_jobs\nfiles_processed++, last_indexed_at=now]
+    S --> U
+```
+
+**Why the router branches by file type, not by source**: a `.md` file is chunked the same way whether it came from a GitHub repo or an Azure DevOps repo — the chunker cares about *content shape*, not *provider*. This keeps `IngestionService` from growing a combinatorial number of code paths as more providers get added.
+
+---
+
+## 6.4 Chunking Strategy
+
+### 6.4.1 Code — tree-sitter (FR-7.2)
+
+Naive line-based chunking (e.g., every 50 lines) routinely splits a function in half, which destroys retrieval quality — the embedding for "half a function" doesn't match a query about what that function does. `tree-sitter` (via `tree-sitter-languages`) parses each file into a syntax tree and chunks at **function and class boundaries**:
+
+```python
+# app/services/knowledge_hub/chunkers/code_chunker.py
+def chunk_code_file(content: str, language: str) -> list[Chunk]:
+    parser = get_parser(language)  # tree-sitter-languages registry
+    tree = parser.parse(content.encode())
+    chunks = []
+    for node in walk_top_level_definitions(tree.root_node):  # function_definition, class_definition
+        if node.type in ("function_definition", "class_definition", "method_definition"):
+            chunks.append(Chunk(
+                text=content[node.start_byte:node.end_byte],
+                start_line=node.start_point[0] + 1,
+                end_line=node.end_point[0] + 1,
+                chunk_type="code",
+            ))
+    if not chunks:  # file has no top-level functions/classes (e.g. a constants file)
+        chunks.append(Chunk(text=content, start_line=1, end_line=content.count("\n") + 1, chunk_type="code"))
+    return chunks
+```
+
+A function/class exceeding **1500 tokens** is split further at statement-block boundaries (still tree-sitter-aware, never mid-line) to stay within the embedding model's effective context window.
+
+### 6.4.2 Docs — heading-based markdown split (FR-7.2)
+
+Markdown is split at heading boundaries (`##`, `###`), capped at 800 tokens per chunk, using `mistune`'s AST to find heading nodes rather than regex — front-matter and code blocks inside docs must not be split mid-block.
+
+### 6.4.3 CI/CD YAML (FR-7.6)
+
+Most workflow files are small enough to embed whole (better retrieval context than fragments — "what does the deploy job do" needs the full job definition). Files over 2000 tokens split by top-level YAML key (`jobs.<name>`, `stages.<name>`).
+
+### 6.4.4 Tickets (FR-7.5)
+
+One chunk per issue/PR: title + body + all comments concatenated, capped at 1500 tokens (oldest comments truncated first if over cap — the issue title and body are never truncated, since that's the highest-signal part).
+
+### 6.4.5 Meetings (FR-5.7)
+
+Fixed-size 500-token windows with 50-token overlap over the structured summary text (decisions + action items + open questions concatenated), not the raw transcript — the raw transcript is noisy and rarely what an onboarding query needs; the *summary* is what should be retrievable ("what did we decide about the reporting widget").
+
+---
+
+## 6.5 Incremental Indexing (FR-7.4)
+
+On a `push` webhook:
+
+1. `list_changed_files(before_sha, after_sha)` — a single Git diff API call, not a tree walk. Returns added/modified/deleted file paths only.
+2. For **deleted** files: `DELETE FROM knowledge_chunks WHERE repository_id = ? AND file_path = ?`, then `chroma_collection.delete(ids=[...chroma_ids])` — stale chunks for a removed file are never left retrievable.
+3. For **added/modified** files: delete any existing chunk rows for that `file_path` (a file may have had 3 chunks before and now has 5 after a refactor — no attempt to diff at the chunk level, the whole file's chunk set is replaced), then re-chunk, re-embed, re-insert.
+4. `repositories.last_indexed_at` and the triggering `ingestion_jobs` row are updated only after all changed files in the push are processed — a webhook that touches 40 files is one `ingestion_jobs` row, not 40, so FR-7.9's health endpoint reports coherent per-push status.
+
+This is why `knowledge_chunks` (§5.3.6) has the index `(project_id, repository_id, file_path)` — it's the exact filter used in step 3's delete-then-reinsert, on every incremental update.
+
+---
+
+## 6.6 Vector Store Namespacing (FR-7.8)
+
+One ChromaDB collection per project, `proj_{project_id}_knowledge`, distinct from `proj_{project_id}_planning` (ADR-2). Within a project's knowledge collection, chunks from multiple connected repositories coexist, disambiguated by `metadata.repository_id` — a project with two repos does **not** get two collections, since cross-repo retrieval ("does the frontend or the backend own auth?") is a legitimate onboarding query that a per-repo collection split would make impossible without a fan-out query.
+
+Chunk metadata written to Chroma (mirrors the Postgres `knowledge_chunks` row so retrieval never needs a Postgres join on the hot path):
+
+```python
+chroma_collection.upsert(
+    ids=[chunk.chroma_id],
+    embeddings=[vector],
+    documents=[chunk.text],
+    metadatas=[{
+        "project_id": str(project_id),
+        "repository_id": str(repository_id),
+        "chunk_type": "code",
+        "file_path": "app/services/auth_service.py",
+        "language": "python",
+        "start_line": 42,
+        "end_line": 78,
+        "commit_sha": "a1b2c3d",
+        "last_updated": "2026-07-01T12:00:00Z",
+    }],
+)
+```
+
+---
+
+## 6.7 Failure Handling & Retry (NFR-AVAIL-4)
+
+| Failure | Handling |
+|---|---|
+| GitHub/Azure API rate-limited (`403`/`429`) | Respect `Retry-After` header; if absent, exponential backoff starting at 5 s, capped at 300 s, 5 attempts, then mark file `skipped_rate_limited` in `ingestion_jobs.error_log` |
+| File fetch 404 (deleted between webhook and fetch) | Treated as a delete — chunk rows removed, not retried |
+| tree-sitter parse error (malformed syntax, unsupported language) | Fall back to whole-file chunking (§6.4.1's no-definitions branch); logged as `parse_fallback`, not a failure |
+| Voyage AI embedding API error | Exponential backoff (same schedule as above); after exhaustion, `EmbeddingError` is raised, caught at the service boundary, recorded per-file in `error_log`, and the rest of the batch continues — one bad file never blocks the other 19 in its batch (§4.7.5's `asyncio.gather` batches) |
+| ChromaDB unreachable | `IngestionJob.status = 'failed'`; `NFR-AVAIL-3`'s degradation applies — reads against the knowledge collection return `503`, ingestion retries the whole job on next webhook or manual re-index trigger |
+| Process restart mid-job | `ingestion_jobs.status = 'running'` rows found stuck on startup are requeued by the same recovery routine as §4.7.3 (ADR-6), scoped additionally to `trigger != 'initial'` re-running only the outstanding file diff, not the whole repo |
+
+---
+
+## 6.8 Index Health Endpoint (FR-7.9)
+
+```
+GET /api/v1/projects/{project_id}/knowledge/health
+```
+
+```json
+{
+  "repositories": [
+    {
+      "repository_id": "uuid",
+      "repo_url": "https://github.com/acme/backend",
+      "index_status": "indexed",
+      "last_indexed_at": "2026-07-02T09:14:00Z",
+      "chunks_by_type": { "code": 4213, "doc": 89, "ticket": 512, "ci": 6 },
+      "failed_files": [
+        { "file_path": "vendor/legacy.min.js", "reason": "skipped_unsupported" }
+      ]
+    }
+  ],
+  "meetings_indexed": 34
+}
+```
+
+Computed as one `GROUP BY repository_id, chunk_type` aggregation over `knowledge_chunks` joined with the latest `ingestion_jobs.error_log` per repository — no ChromaDB round-trip needed, since chunk counts and error state live entirely in Postgres metadata (§5.1, principle 4).
+
+---
+
+## 6.9 Key Decisions
+
+**ADR-K1: Chunk by content shape, not by source provider.** A single router (§6.3) dispatches on file extension/MIME, so GitHub and Azure DevOps content reuse the same four chunkers. Adding a third provider (e.g., GitLab) means writing a `SourceConnector`, not a new chunking path.
+
+**ADR-K2: Delete-then-reinsert per file on incremental update, not chunk-level diffing.** Diffing chunk-by-chunk within a modified file would require stable chunk identity across refactors (a renamed function shifts every chunk's line range) — not worth the complexity at ProjectOS's scale (NFR-SCALE-1: 500 files/min initial throughput target, 60 s incremental SLA already met by whole-file replacement).
+
+**ADR-K3: Meetings are ingested as their structured summary, not raw transcript.** The Knowledge Hub retrieves what's useful to a query ("what did we decide"), not everything that was said. Raw transcripts stay in `meetings.raw_transcript` (Postgres) for audit but are never embedded.