Skip to content

NOTES.md

Latest commit

 

History

History
225 lines (190 loc) · 10.9 KB

NOTES.md

File metadata and controls

225 lines (190 loc) · 10.9 KB

Session State - 2026-01-16

Current Task

BATCH 2 COMPLETE - TerminalLoop, ShadowValidator, @workflow decorators, BackgroundTaskManager

Latest Session (2026-01-16) - Batch 2 Features

  • Implemented TerminalLoop with pause/resume/cancel, state machine, retry with exponential backoff
  • Enhanced ShadowValidator with AST comparison, mutation detection, import tracking
  • Added @workflow/@step decorators with dependency resolution, parallel execution, parameter injection
  • Added BackgroundTaskManager with priority queue, worker pool, retry logic, dead letter queue
  • Fixed all 66 batch 2 TDD tests to align with implementation APIs
  • 680 tests passing (1 skipped)
  • Extracted 6 learnings to knowledge graph
  • Cleaned up ~100 generated doc/temp files

Batch 2 Commits

  • 59b6b5e feat(batch2): implement TerminalLoop, ShadowValidator, @workflow decorators, BackgroundTaskManager
  • 1dee7eb chore: clean up generated docs and update package.json

Batch 2 Key Files

  • src/agents/terminal_loop.py - TerminalLoop + LoopState + LoopIteration
  • src/agents/shadow_validator.py - ShadowValidator + AST comparison
  • src/agents/workflows.py - @workflow/@step decorators + WorkflowStepDef
  • src/agents/background_tasks.py - BackgroundTaskManager + priority queue
  • tests/test_yoink_batch2.py - 66 TDD spec tests (all passing)

Batch 2 Key Decisions

  • asyncio.Semaphore for worker pool concurrency limiting
  • asyncio.Event for CPU-efficient pause/resume in TerminalLoop
  • @step decorator attaches metadata to methods, @workflow collects via dir(cls)
  • is_valid = False when mutations OR security violations detected
  • Task handlers accept optional data param: async def handler(data=None)
  • Compare enum values not strings: status == TaskManagerStatus.COMPLETED

Batch 2 Learnings Extracted (group: cross-project-learnings)

  • learning-task-orchestrator-async-patterns-20260116
  • learning-task-orchestrator-tdd-workflow-20260116
  • learning-task-orchestrator-decorator-workflow-20260116
  • learning-task-orchestrator-self-healing-20260116
  • learning-task-orchestrator-ast-validation-20260116
  • learning-task-orchestrator-background-tasks-20260116

Previous Progress (Phases 1-10)

  • Phase 1: Foundation (Trial, Graders, Export, Integration)
  • Phase 2: MCP Server Integration (semantic failure tracking, eval in handlers)
  • Phase 3: Evaluation Suites (unit.py, resilience.py)
  • Phase 4: Training Data Export (JSONL export to D:\Research\training-data)
  • Phase 5: Graphiti Immune System (failure_store, pattern_matcher, guardrails, core)
  • Phase 6: Full Integration (immune hooks in spawn_agent/parallel, model graders, MCP tools)
  • Phase 7: Production Ready (specialized graders, Graphiti persistence, dashboard, CI/CD)
  • Phase 8: Advanced Features (Langfuse deep integration, alerting, federation, ML prediction)
  • Phase 9: Cross-Project Federation (registry, decay, federation MCP tools)
  • Phase 10: Live Graphiti Sync (sync protocol, subscriber, engine, conflict resolver, hooks, monitor)

Archive: Session 2026-01-12

Learnings Extracted to Graphiti

Ran /learn extract and stored 5 key patterns:

  1. Multi-Agent Swarm Orchestration - QUEEN-WORKER/MESH patterns, 5-6 parallel Gemini Pro agents
  2. Self-Healing Immune System - Risk scoring, hash dedup, 72h decay, ML prediction
  3. Cross-Project Federation - WebSocket sync, version vectors, conflict resolution, hooks
  4. MCP Server Design - 29 tools across 7 categories, lazy init, structured JSON
  5. AI Evaluation Pipeline - Grader chains, Gemini Flash caching, JSONL export

All stored under group_id: project_task_orchestrator for cross-project recall.

Phase 10: Live Graphiti Sync

Used /flow to spawn 6 parallel Gemini Pro agents (QUEEN-WORKER pattern) for Task 5:

  1. Agent 1: Sync Protocol - WebSocket-based protocol with heartbeats, exponential backoff
  2. Agent 2: Pattern Subscriber - Real-time subscription manager with event buffering
  3. Agent 3: Sync Engine - Bidirectional push/pull engine with batch processing
  4. Agent 4: Conflict Resolver - Version vector conflict detection with LWW/merge strategies
  5. Agent 5: Sync Hooks - Middleware-style hook system for sync lifecycle events
  6. Agent 6: Sync Monitor - Health tracking with latency, stall detection, alerts

Files Created:

  • src/evaluation/immune_system/live_sync/__init__.py - Module exports
  • src/evaluation/immune_system/live_sync/sync_protocol.py - Protocol + client
  • src/evaluation/immune_system/live_sync/pattern_subscriber.py - Subscriber
  • src/evaluation/immune_system/live_sync/sync_engine.py - Engine
  • src/evaluation/immune_system/live_sync/conflict_resolver.py - Conflict resolution
  • src/evaluation/immune_system/live_sync/sync_hooks.py - Hooks middleware
  • src/evaluation/immune_system/live_sync/sync_monitor.py - Health monitor

New MCP Tools: sync_status, sync_trigger, sync_alerts Tests: 43 new tests for live sync (213 total tests passing)

Previous Work (Phase 9 - Federation)

  1. Used /flow to spawn 5 parallel Gemini Pro agents (MESH pattern)
  2. Each agent designed a component:
    • Agent 1: Portfolio Registry (namespaces.json, PortfolioProject dataclass)
    • Agent 2: MCP Tools (4 federation tools with schemas)
    • Agent 3: Sync Protocol (bidirectional sync, conflict resolution)
    • Agent 4: Pattern Decay (exponential decay with reinforcement)
    • Agent 5: Integration Hooks (pre-spawn, post-failure, periodic sync)
  3. Created registry.py, decay.py, 4 federation MCP tools, 39 tests

Commits (All Pushed)

  • 76dbfa1 feat(evaluation): add agent evaluation system for quality gates (Phase 1)
  • 2e8fe33 feat(evaluation): complete Phase 2 - semantic failures, eval suites
  • 0a79dc9 feat(evaluation): add Graphiti Immune System (Phase 5)
  • ec1fdd6 feat(evaluation): complete Phase 6 - full integration
  • b7d5cca feat(evaluation): complete Phase 7+8 - production ready with advanced features
  • 653c59b feat(mcp): add alert_list, alert_clear, predict_risk MCP tools
  • 98e01a2 feat(federation): implement cross-project pattern federation (Phase 9)
  • 784a90f feat(live-sync): implement real-time Graphiti federation sync (Phase 10)

MCP Tools (29 Total)

Task Management:      tasks_list, tasks_add, tasks_sync_email, tasks_schedule,
                      tasks_complete, tasks_analyze, tasks_briefing
Cost & Health:        cost_summary, cost_set_budget, healing_status
Agent Execution:      spawn_agent, spawn_parallel_agents
Immune System:        immune_status, immune_check, immune_failures,
                      immune_dashboard, immune_sync
Alerting:             alert_list, alert_clear
Prediction:           predict_risk
Federation:           federation_status, federation_subscribe,
                      federation_search, federation_decay
Live Sync:            sync_status, sync_trigger, sync_alerts

Test Status

  • 213 tests passing
  • Run with: JWT_SECRET_KEY=test123 python -m pytest tests/ -v

Key Files

Core Evaluation

  • src/evaluation/__init__.py - All exports (70+ symbols)
  • src/evaluation/trial.py - Trial schema
  • src/evaluation/graders/ - Code + Model graders

Immune System

  • src/evaluation/immune_system/core.py - ImmuneSystem singleton
  • src/evaluation/immune_system/federation.py - Cross-project sharing
  • src/evaluation/immune_system/registry.py - Portfolio project registry
  • src/evaluation/immune_system/decay.py - Pattern relevance decay

Alerting & Prediction

  • src/evaluation/alerting/manager.py - AlertManager
  • src/evaluation/prediction/classifier.py - FailurePredictor

Live Sync

  • src/evaluation/immune_system/live_sync/__init__.py - Module exports
  • src/evaluation/immune_system/live_sync/sync_protocol.py - WebSocket protocol
  • src/evaluation/immune_system/live_sync/pattern_subscriber.py - Event subscriber
  • src/evaluation/immune_system/live_sync/sync_engine.py - Bidirectional sync
  • src/evaluation/immune_system/live_sync/conflict_resolver.py - Version vectors
  • src/evaluation/immune_system/live_sync/sync_hooks.py - Middleware hooks
  • src/evaluation/immune_system/live_sync/sync_monitor.py - Health monitor

MCP Server

  • src/mcp/server.py - 29 MCP tools with handlers

Architecture Overview

Evaluation System:
  Trial -> GraderPipeline -> [NonEmpty, Length, JSON, Regex, Model] -> GraderResult

Immune System:
  pre_spawn_check(prompt) -> ImmuneResponse (risk_score, guardrails)
  record_failure() -> FailurePattern -> PatternMatcher -> Graphiti

Federation:
  RegistryManager -> [task-orchestrator, construction-connect, ...]
  PatternFederation -> subscribe -> search_global_patterns -> import_pattern
  PatternDecaySystem -> S(t) = S_last * 2^(-Δt/h) + W_outcome

Alerting:
  AlertManager -> [HighRiskThreshold, FrequencySpike, NewPatternDetected]
  Notifiers: Console, Webhook, Slack

Prediction:
  FailurePredictor -> FeatureExtractor (TF-IDF + meta) -> RandomForest

Live Sync:
  PatternSyncClient -> WebSocket -> SyncMessage (heartbeat, pattern_created/updated/deleted)
  PatternSubscriber -> event queue -> callbacks -> PatternEvent
  SyncEngine -> push_batch/pull_batch -> PeerSyncState tracking
  ConflictResolver -> VersionVector -> LWW/Merge/Manual strategies
  SyncHooks -> before_push/after_push/before_pull/after_pull/on_error
  SyncHealthMonitor -> latency/failures/stalls -> SyncAlert

MCP Integration:
  spawn_agent/parallel -> immune pre-check -> evaluate -> record failures
  Federation tools: status, subscribe, search, decay
  Sync tools: sync_status, sync_trigger, sync_alerts

Key Decisions

  • Non-blocking evaluation: failures logged but don't block responses
  • Lazy singleton initialization for MCP handlers (hasattr pattern)
  • Hash-based failure deduplication: sha256(operation:type:input[:100])[:16]
  • Model graders use Gemini Flash with MD5 caching
  • Pattern decay: 72-hour half-life, 14-day staleness threshold
  • Hybrid registry: static namespaces.json + dynamic Graphiti discovery

Graphiti Learnings Stored

Query with /recall using these group_ids:

  • project_task_orchestrator - All patterns from this project
  • Pattern names:
    • learning-task-orchestrator-multi-agent-swarm-20260112
    • learning-task-orchestrator-immune-system-20260112
    • learning-task-orchestrator-federation-sync-20260112
    • learning-task-orchestrator-mcp-server-design-20260112
    • learning-task-orchestrator-evaluation-pipeline-20260112

Next Steps (Optional)

  1. Train ML predictor with production JSONL data
  2. Fine-tune model graders based on collected evaluations
  3. Create admin web dashboard for monitoring
  4. Add more alert notifiers (email, PagerDuty)
  5. Add pattern import/export between projects
  6. Apply learnings to other portfolio projects

Context to Preserve

  • GitHub repo: https://github.com/TC407-api/task-orchestrator
  • All 10 phases complete + learnings extracted
  • 213 tests passing, verification passed
  • 29 MCP tools available
  • Commit 784a90f pushed to origin
  • Ready for production deployment