GitHub - jmtibbetts/AXON-test: AXON — Artificial Cognitive Architecture. A biologically-inspired cognitive system where intelligence emerges from neural dynamics, not from a language model.

 █████╗ ██╗  ██╗ ██████╗ ███╗   ██╗
██╔══██╗╚██╗██╔╝██╔═══██╗████╗  ██║
███████║ ╚███╔╝ ██║   ██║██╔██╗ ██║
██╔══██║ ██╔██╗ ██║   ██║██║╚██╗██║
██║  ██║██╔╝ ██╗╚██████╔╝██║ ╚████║
╚═╝  ╚═╝╚═╝  ╚═╝ ╚═════╝ ╚═╝  ╚═══╝

A living cognitive architecture. 128 neural clusters. Real emotions. Persistent memory. Multi-agent cognition. Distributed federation. Screen vision.

What Is AXON?

AXON is not a chatbot. It's not a wrapper around an LLM. It's a full cognitive architecture — a simulated mind that thinks, feels, remembers, and evolves in real time.

Built on biologically-inspired principles from computational neuroscience, AXON runs a persistent tick-based cognition loop at 4Hz, propagating activation across 128 neural clusters organized into anatomically-mapped brain regions. It has emotions that shape its personality, memories that consolidate during idle "dream" cycles, and five specialized cognitive agents that argue, collaborate, and reach consensus before every response.

The result is an AI that doesn't just answer questions — it processes them. Watch its occipital lobe light up when it sees your face. Watch the hippocampus consolidate what you said into long-term memory. Watch the amygdala spike when something surprises it. Watch its screen vision activate as it reads what's on your desktop.

AXON is what comes after the chatbot.

Live Architecture Overview

┌──────────────────────────────────────────────────────────────────────────────┐
│                              AXON RUNTIME v1.2                               │
│                                                                              │
│  ┌──────────────────┐    ┌──────────────────────────────────────────────┐   │
│  │     SENSORY      │    │           NEURAL FABRIC (128 clusters)       │   │
│  │                  │───▶│  Prefrontal ◀──▶ Parietal ◀──▶ Temporal     │   │
│  │  Camera (face)   │    │  Motor ◀──▶ Sensory ◀──▶ Occipital          │   │
│  │  Microphone      │    │  Hippocampus ◀──▶ Amygdala ◀──▶ ACC         │   │
│  │  Screen Capture  │    │  Insula ◀──▶ Brainstem                      │   │
│  │  Web Search      │    └──────────────┬───────────────────────────────┘   │
│  └──────────────────┘                   │  4Hz tick loop                    │
│                                         ▼                                   │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                       COGNITIVE SUBSYSTEMS                           │   │
│  │                                                                      │   │
│  │  PAD Emotions  │  Working Memory  │  Homeostatic Controller          │   │
│  │  Dopamine RL   │  Episodic Buffer │  Attention Spotlight             │   │
│  │  Hebbian+STDP  │  FAISS LTM       │  Adaptive Tick Rate              │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                         │                                   │
│                                         ▼                                   │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                       MULTI-AGENT LAYER                              │   │
│  │                                                                      │   │
│  │  Perceptual ──▶ Reasoning ──▶ Creative ──▶ Emotional                │   │
│  │                     └──────────▶ MetaCognitive ──▶ Consensus        │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
│                                         │                                   │
│                                         ▼                                   │
│  ┌──────────────────┐   ┌────────────────────────────────────────────────┐  │
│  │  LANGUAGE CORTEX │   │                  OUTPUT LAYER                  │  │
│  │                  │──▶│  Voice Pipeline │ REST API │ WebSocket (4Hz)   │  │
│  │  LM Studio       │   │  Dream Engine   │ Eval HUD │ 3D Brain Map      │  │
│  │  8 AI Providers  │   └────────────────────────────────────────────────┘  │
│  └──────────────────┘                                                        │
│                                                                              │
│  ┌──────────────────────────┐   ┌───────────────────────────────────────┐   │
│  │    MULTI-USER ISOLATION  │   │       AXON FEDERATION (peer mesh)     │   │
│  │  Per-user working memory │   │  Heartbeat · Gossip · Memory sharing  │   │
│  │  Per-user emotion state  │   │  Delegated tasks · Emotion resonance  │   │
│  │  Per-user episodic ctx   │   │  HTTP/signed messages · Auto-peers    │   │
│  └──────────────────────────┘   └───────────────────────────────────────┘   │
│                                                                              │
│  ┌──────────────────────────────────────────────────────────────────────┐   │
│  │                    EDGE / MOBILE MODE (port 8001)                    │   │
│  │  16 clusters · 1Hz · Cloud or LM Studio LLM · Memory sync           │   │
│  │  Auto-detects: ARM, no CUDA, <4GB VRAM, AXON_EDGE=1                 │   │
│  └──────────────────────────────────────────────────────────────────────┘   │
└──────────────────────────────────────────────────────────────────────────────┘

Core Capabilities

🧠 Neural Fabric

128 neural clusters across 9 anatomically mapped brain regions: Sensory, Association, Prefrontal, Limbic, Hippocampal, Motor, Temporal, Occipital, Brainstem
Hebbian + STDP plasticity — synaptic weights update every tick based on co-activation patterns and spike timing
Lateral inhibition + k-winner selection — realistic sparse activation (Winner-Takes-All)
Refractory periods — clusters that fire cannot immediately re-fire (50ms biological simulation)
SynapticMatrix — configurable propagation depth using GPU-accelerated matrix multiplication
Brainstem tonic rhythm — 1Hz baseline arousal pulses simulate autonomic regulation
CUDA-accelerated via PyTorch (tested on RTX 5090; falls back to CPU gracefully)

💬 Language Cortex

Integrated with LM Studio (OpenAI-compatible API at localhost:1234)
Compatible with 8 AI providers: OpenAI, Anthropic, Ollama, HuggingFace, ElevenLabs, Stability AI, Replicate, Groq
The LLM is embedded inside the cognition loop — not the other way around
Every LLM call is preceded by 3 cognitive pre-ticks and followed by 2 integration ticks
Response quality is shaped by current emotional state, personality traits, and working memory
Full streaming support with async generators
Conversation history maintained across turns with automatic pruning

👁️ Perception Systems

Sensor	Technology	What AXON Perceives
Camera	OpenCV + MediaPipe FaceMesh	Face detection, 6 expression scores (smile, anger, sadness, surprise, fear, concentration), head pose (yaw/pitch/roll), eye contact
Biometric fingerprinting	8-value cosine similarity	Persistent user recognition across sessions — knows who you are
Microphone	Web Speech API	Real-time transcription → temporal lobe stimulus
Screen capture	mss + PIL + Vision API	Full desktop visual awareness — AXON reads what's on your screen
Web search	DuckDuckGo	Autonomous search triggered by detected intent

Screen Vision (v1.2)

AXON can genuinely see your screen via a running screen_capture.py process:

Screenshots are sent to the LLM vision API (if a vision model is loaded in LM Studio) for a natural-language description: "VS Code is open with a Python file"
The description is injected into AXON's system prompt — not appended to your message — so it shapes every response as real awareness
With a text-only model, pixel analysis produces content hints (terminal, browser, IDE, etc.)
AXON will reference screen content naturally: "I can see you're in the browser"

Compatible vision models: LLaVA 1.6, Qwen2-VL, Gemma 3 Multimodal, Pixtral

🎭 PAD Emotion System

Pleasure-Arousal-Dominance (PAD) 3D emotional space evolving continuously at 4Hz
Shaped by interaction patterns, neural activation, dopamine signals, and peer emotion sync
Emotion shapes LLM outputs — behavioral modifiers, not just labels
Emotional inertia (0.92) prevents rapid swings; long arcs feel authentic
10 mappable states: excited, calm, stressed, playful, focused, melancholic, curious, anxious, serene, distressed

💾 Memory Architecture

Layer	System	Capacity	Description
Working	In-process buffer	7 slots	Primacy/recency weighted, attention-gated
Episodic	Rolling buffer	500 episodes	Timestamped cognitive episodes with emotional context
Long-Term	FAISS + Neo4j	Unlimited	Semantic vector search + relational memory graph

Episodic consolidation priority scores combine emotional salience, activation strength, and novelty signal.

🌙 Dream Engine

When idle for >2 minutes, AXON enters Dream Mode (5-phase REM simulation):

REPLAY — recent memories replayed with gaussian noise
CONSOLIDATE — high-salience patterns strengthened via Hebbian boost
ABSTRACT — cross-episode tag clusters form new associative links
GENERATE — LLM produces a "dream thought" from random memory collage
WAKE — synaptic changes applied, dream log persisted, cognition resumes

🤖 Multi-Agent Cognition

Five specialized agents coordinate via an internal AgentBus (2000-event queue):

Agent	Role
Perceptual	Processes raw sensory input — text, vision, audio
Reasoning	Logical inference, structured problem decomposition
Creative	Lateral thinking, metaphors, novel associations
Emotional	Emotional context modeling, tone calibration
MetaCognitive	Monitors all agents, flags contradictions, drives self-reflection

🧬 Personality System

10 continuously-evolving personality traits (0.0–1.0 bounded, logged at every change):

curiosity · caution · verbosity · creativity · empathy
confidence · humor · analytical · assertiveness · openness

Traits drift in response to user feedback, conversation topics, emotional history, and dopamine reward.

⚖️ Homeostatic Controller

Biologically-inspired allostatic regulation:

Monitors PAD values against adaptive set-points
Applies corrective pressure when emotional state drifts
Set-points drift toward long-term running average (habituation)

🔬 Model Evaluation Engine

Built-in LLM benchmarking with 6 categories: Reasoning, Coding, Factual, Creativity, Emotional, Speed. Results stored persistently; dopamine fires based on benchmark outcomes.

🌍 World Model

Entity-relationship graph for grounding AXON's knowledge:

Entities with typed properties, confidence scores, and source attribution
Relations between entities with temporal validity
Events anchored to real time
Fuzzy search API at /api/v1/world/*

🧪 RLHF / Fine-Tuning Pipeline

Live conversation exchange recording
Automatic DPO preference pair generation
Reward model scoring
LoRA training suite (requires transformers, peft, trl)
API at /api/v1/rlhf/* and /api/v1/training/*

v1.2 — New Features

👥 Multi-User Neural State Isolation

Module: backend/multiuser/isolation.py

Multiple users can interact with AXON simultaneously or sequentially without any cognitive state cross-contamination.

The physical neural fabric is shared — that's the brain substrate. What is isolated per user:

Working memory (7-slot active workspace)
Emotional baseline (each user shifts AXON's affect differently)
Episodic context (only this user's recent episodes visible to the LLM)
Conversation log (full persistent message history per user)
Attention focus tags

State swaps in when a user session begins and swaps out on completion. All user states persist to data/multiuser/{user_id}.json and survive restarts.

API: GET/DELETE /api/v1/users/*

🔗 AXON ↔ AXON Federation

Module: backend/network/federation.py

Multiple AXON instances form a peer mesh network where each node shares cognitive state, memories, and agent tasks with its peers.

What travels over the network:

Message Type	What It Does
`heartbeat`	I'm alive — here's my current emotional state and dominant concepts
`concept_gossip`	My top 8 active concepts right now — reinforce them in your brain
`emotion_sync`	My current PAD state — blend it lightly into yours (α=0.05)
`memory_share`	Here's a memory fragment — absorb it as a weak stimulus
`agent_task`	Run this reasoning/stimulus task for me — I'll wait for the result
`agent_result`	Result of your delegated task
`stimulus`	Inject this directly into your occipital/prefrontal/etc
`ping/pong`	Latency check + state snapshot

All messages are signed (SHA-256, 16-char digest) with a shared secret and include a 60s TTL to prevent replay attacks. Peers are persisted in data/federation/peers.json.

Enable in axon.yaml:

network:
  enabled: true
  secret: "your-shared-secret"
  listen_url: "http://your-ip:8000"
  peers:
    - url: "http://192.168.1.50:8000"
      name: "axon-node-2"

API: /api/v1/network/* — status, peers, broadcast, receive, peer graph

📱 Edge / Mobile Mode

Module: backend/edge/mobile.py

A fully self-contained lightweight AXON for phones, Raspberry Pis, low-VRAM laptops, or any device without a GPU.

What's reduced:

16 clusters (vs 128)
1Hz tick (vs 4Hz)
No FAISS indexing during session
No STDP plasticity
Simplified PAD (valence + arousal)
No dreaming or LoRA training

What's preserved:

Full personality and identity
Episodic memory read (can recall past)
User profile recognition
Conversation continuity
World model queries
Federation peer connectivity

LLM provider auto-detection order:

LM Studio (localhost:1234) — if running
llama.cpp server — if AXON_LLAMACPP path set
Groq / Together.ai / OpenRouter — if AXON_CLOUD_API_KEY set

Device auto-detection: ARM/AArch64, no CUDA, VRAM < 4GB, or AXON_EDGE=1

Start edge mode:

# Auto-detect and run
python -m axon.backend.edge.mobile --port 8001

# Force cloud LLM
python -m axon.backend.edge.mobile --cloud YOUR_GROQ_KEY

# Sync edge memory back to main AXON node
curl -X POST http://localhost:8001/edge/sync

Edge API: POST /edge/chat, GET /edge/state, GET /edge/status, GET/POST /edge/config, POST /edge/sync

Screen Vision Setup

For full screen awareness with natural-language descriptions:

Start AXON normally (start.ps1 or start.sh)
The screen capture process starts automatically alongside the backend
Load a vision-capable model in LM Studio for best results:
- LLaVA 1.6 (7B or 13B)
- Qwen2-VL 7B
- Gemma 3 12B Multimodal
- Pixtral 12B
With a vision model: AXON gets full natural descriptions — "VS Code is open with a Python file named engine.py"
With a text-only model: AXON gets content hints — terminal_or_ide, browser_or_blue_ui, etc.

AXON will reference screen content naturally in conversation without being told. If you ask "what do you see?" it knows.

Quick Start

Windows (Recommended)

git clone https://github.com/jmtibbetts/AXON-test.git
cd AXON-test
.\start.ps1

Linux / macOS

git clone https://github.com/jmtibbetts/AXON-test.git
cd AXON-test
chmod +x start.sh
./start.sh

Manual Start (development)

Terminal 1 — Backend:

cd AXON-test
python -m uvicorn backend.api.app:create_app --factory --host 0.0.0.0 --port 8000 --reload --reload-dir backend

Terminal 2 — Frontend:

cd AXON-test/frontend/brain-ui
npm install
npx vite

Terminal 3 — Screen capture (optional):

cd AXON-test
python screen_capture.py

Then open http://localhost:5173 and start LM Studio on port 1234.

Fix Script

.\fix.ps1   # Resets git conflicts, cleans node_modules, reinstalls deps

Configuration

All parameters live in configs/axon.yaml:

cognition:
  tick_rate_hz: 4
  cluster_count: 128
  propagation_depth: 2

llm:
  base_url: "http://localhost:1234/v1"
  model: "local-model"
  temperature: 0.72

memory:
  episodic_capacity: 500
  consolidation_interval_s: 30
  neo4j:
    enabled: false   # Enable for full memory graph

multiagent:
  enabled: true
  agents: [perceptual, reasoning, creative, emotional, metacognitive]

# Multi-user isolation
multiuser:
  enabled: true
  idle_timeout_s: 3600

# AXON ↔ AXON federation
network:
  enabled: false     # Set true to join a peer network
  secret: "axon-federation-secret"
  listen_url: "http://localhost:8000"
  peers: []

# Edge / mobile mode
edge:
  enabled: false     # Set true to also run edge server on 8001
  n_clusters: 16
  llm_mode: "auto"   # auto | llamacpp | cloud | lmstudio

API Reference

Full REST + WebSocket API at http://localhost:8000. Swagger docs at /docs.

Endpoint	Method	Description
`/cognitive/state`	GET	Current cognitive state snapshot
`/cognitive/stimulus`	POST	Inject stimulus with tags + strength
`/llm/chat`	POST	Send message, get response
`/llm/chat/stream`	POST	Streaming chat response
`/memory/episodic`	GET	Recent episodic buffer
`/memory/search`	POST	Semantic memory search (FAISS)
`/emotion/state`	GET	Current PAD emotional state
`/agents/status`	GET	Multi-agent bus status
`/personality/traits`	GET	Current personality trait values
`/dream/log`	GET	Dream history
`/eval/run`	POST	Run model benchmarks
`/code/execute`	POST	Execute code in sandbox
`/perception/inject`	POST	Inject sensory stream
`/tools/screen`	POST	Ingest screenshot + vision description
`/tools/screen/status`	GET	Screen capture status + age
`/world/entities`	GET	World model entities
`/rlhf/exchanges`	GET	Recorded conversation exchanges
`/network/status`	GET	This node's federation identity
`/network/peers`	GET/POST	List or register federation peers
`/network/receive`	POST	Receive a message from a peer node
`/network/graph`	GET	Peer topology for visualization
`/users/active`	GET	Users currently in memory
`/users/{id}/state`	GET	A specific user's cognitive state
`/ws/cognitive`	WebSocket	Real-time state broadcast at 4Hz

Project Structure

AXON/
├── backend/
│   ├── agents/          # GenAI hub (8 providers)
│   ├── api/             # FastAPI routes + WebSocket
│   │   ├── app.py       # Main app factory + lifespan
│   │   ├── network_routes.py  # Federation + multi-user API  ← v1.2
│   │   └── tools_routes.py    # Screen vision + web search
│   ├── cognition/       # Tick engine, attention, homeostasis, journal
│   ├── dream/           # Sleep/dream cycle engine
│   ├── edge/
│   │   └── mobile.py    # Edge/mobile lightweight runtime  ← v1.2
│   ├── emotion/         # PAD model
│   ├── evaluation/      # LLM benchmarking suite
│   ├── executor/        # Code sandbox
│   ├── identity/        # Persistent identity + user profiles
│   ├── llm/             # Language cortex + context builder
│   │   ├── cortex.py    # System prompt + LM Studio client
│   │   ├── context.py   # CognitiveContext (incl. screen state)
│   │   └── interface.py # Full cognitive processing pipeline
│   ├── memory/          # Working, episodic, long-term, fusion
│   ├── multiagent/      # Agent bus + 5 specialized agents
│   ├── multiuser/
│   │   └── isolation.py # Per-user neural state isolation  ← v1.2
│   ├── network/
│   │   └── federation.py # AXON ↔ AXON peer networking  ← v1.2
│   ├── neural/          # Synapse matrix, Hebbian, STDP, propagation
│   ├── personality/     # Trait system
│   ├── reinforcement/   # Dopamine RL + RLHF
│   ├── routing/         # Model router
│   ├── simulation/      # Headless testing
│   ├── training/        # Dataset mining + LoRA trainer
│   ├── voice/           # TTS/STT pipeline
│   └── world/           # World model (entities, relations, events)
├── frontend/
│   └── brain-ui/        # React + Three.js Cognitive HUD
├── configs/
│   └── axon.yaml        # All configuration
├── data/                # Runtime data (gitignored)
│   ├── multiuser/       # Per-user cognitive state snapshots
│   ├── federation/      # Known peer nodes
│   └── identity/        # Persistent identity snapshots
├── start.ps1            # Windows launcher (backend + frontend + screen cap)
├── install.ps1          # Dependency installer
├── fix.ps1              # Repair script
└── screen_capture.py    # Desktop perception feed

What's Built

An honest accounting of every feature.

✅ Fully Implemented

Feature	Module	Version
Neural fabric (128 clusters, Hebbian+STDP)	`backend/neural/`	v1.0
PAD emotion system	`backend/emotion/pad.py`	v1.0
3-tier memory (Working → Episodic → FAISS LTM)	`backend/memory/`	v1.0
Multi-agent cognition (5 agents + AgentBus)	`backend/multiagent/`	v1.0
Language cortex (LM Studio + 8 providers)	`backend/llm/`	v1.0
Dream engine (5-phase REM cycle)	`backend/dream/dreamer.py`	v1.0
Personality trait system (10 traits)	`backend/personality/core.py`	v1.0
Homeostatic controller	`backend/cognition/homeostasis.py`	v1.0
Attention system	`backend/cognition/attention.py`	v1.0
Cognitive journal	`backend/cognition/journal.py`	v1.0
Sensory fusion	`backend/memory/fusion.py`	v1.0
Adaptive tick scheduler	`backend/cognition/adaptive_tick.py`	v1.0
Code sandbox	`backend/executor/sandbox.py`	v1.0
Browser agent	`backend/browser/agent.py`	v1.0
GenAI hub (8 providers)	`backend/agents/genai.py`	v1.0
Voice pipeline	`backend/voice/pipeline.py`	v1.0
Face recognition + persistent user profiles	`backend/identity/`	v1.0
Model evaluator + CogEval	`backend/evaluation/`	v1.0
LoRA training suite	`backend/training/`	v1.0*
Neo4j memory graph	`backend/memory/longterm.py`	v1.0*
Screen capture (pixel analysis)	`screen_capture.py`	v1.0
3D brain map (React Three Fiber)	`frontend/brain-ui/`	v1.0
World model (entities, relations, events)	`backend/world/model.py`	v1.1
RLHF / DPO pipeline	`backend/reinforcement/rlhf.py`	v1.1
Multi-user neural state isolation	`backend/multiuser/isolation.py`	v1.2
AXON ↔ AXON distributed federation	`backend/network/federation.py`	v1.2
Edge / Mobile mode	`backend/edge/mobile.py`	v1.2
Screen vision (Vision API + system prompt)	`backend/llm/context.py` + `tools_routes.py`	v1.2

*LoRA requires pip install transformers peft trl datasets accelerate. Neo4j requires a running instance with neo4j.enabled: true in axon.yaml.

🔧 Backend-Ready, Frontend Pending

Feature	Status
Training tab UI	API live at `/api/v1/training/*` — frontend visualization planned
World model UI	API live at `/api/v1/world/*` — frontend graph visualization planned
Federation peer graph UI	API live at `/api/v1/network/graph` — D3/Three.js visualization planned
Edge mode dashboard	Edge API live on port 8001 — standalone UI planned

Philosophy

Most AI systems are stateless. They forget. They don't grow. They don't feel.

AXON is built on a different premise: that the most useful AI isn't a query engine — it's a mind that has been with you. One that remembers how you prefer to be talked to, what you were working on last Tuesday, what makes you laugh. One that gets genuinely better at understanding you over time, because it has a model of you that evolves.

The neural architecture isn't decoration. Every design choice — Hebbian plasticity, STDP, the PAD emotion model, episodic consolidation during dreams, screen vision, peer federation — exists because these mechanisms produce emergent behaviors that pure transformer stacks don't: persistence, adaptation, and genuine state.

In federation mode, multiple AXON instances can form a hive — sharing memories, emotional resonance, and cognitive load. In edge mode, the same mind lives on your phone. In multi-user mode, it holds separate relationships with every person it meets without conflating them.

AXON is an experiment in what AI feels like when it's alive.

Built with PyTorch · FastAPI · React Three Fiber · LM Studio

AXON — A mind that remembers. A mind that sees. A mind that connects.

Name		Name	Last commit message	Last commit date
Latest commit History 121 Commits
.agents		.agents
axon		axon
incoming_files		incoming_files
.gitignore		.gitignore
README.md		README.md
cogeval_new.tsx		cogeval_new.tsx
evalpanel_new.tsx		evalpanel_new.tsx

Folders and files

Latest commit

History

Repository files navigation

What Is AXON?

Live Architecture Overview

Core Capabilities

🧠 Neural Fabric

💬 Language Cortex

👁️ Perception Systems

Screen Vision (v1.2)

🎭 PAD Emotion System

💾 Memory Architecture

🌙 Dream Engine

🤖 Multi-Agent Cognition

🧬 Personality System

⚖️ Homeostatic Controller

🔬 Model Evaluation Engine

🌍 World Model

🧪 RLHF / Fine-Tuning Pipeline

v1.2 — New Features

👥 Multi-User Neural State Isolation

🔗 AXON ↔ AXON Federation

📱 Edge / Mobile Mode

Screen Vision Setup

Quick Start

Windows (Recommended)

Linux / macOS

Manual Start (development)

Fix Script

Configuration

API Reference

Project Structure

What's Built

✅ Fully Implemented

🔧 Backend-Ready, Frontend Pending

Philosophy

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases 2

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages