Cognitive-Physical Decoupling — A Session-Centered Runtime for Embodied Intelligence

_{English · 中文}

---

📢 Changelog

Version	Date	Update
	2026-06-11	Cleaned protocol files and docs; game scenario separated to general-game-agent branch; main branch now focused on sim & real
	2026-06-5	Optimize the user-friendly onboarding process; Communication Protocol Specification; More reasonable coding standards; Game Agent & Benchmarking ready
	2026-05-25	Strict separation of PolicySkillRuntime / BuiltinSkillRuntime; Game Agent & Benchmarking ready
	2026-05-20	Perception plugin system: SensorConfig / PerceptionConfig YAML + EnvironmentWriter auditable writeback
	2026-05-18	Session-Centered Runtime MVP: DummySimTarget + DummyAdapter + DummyClient serial pipeline
	2026-04-29	Hackathon baseline: plugin-based HAL, ReKep / SAM3 real-robot grasping & VLN full pipeline

---

🤔 Why PhyAgentOS?

Traditional "LLM-direct-to-hardware" approaches tightly couple reasoning to execution — switching robots means rewriting the entire pipeline. PhyAgentOS changes this through Cognitive-Physical Decoupling + Session-Centered Runtime:

🔌	One Codebase, Any Hardware — Adding a new robot means implementing one Target Adapter (~100 lines); zero changes to the scheduling layer.
🛡️	Three Safety Layers — Critic validation → Strict Preflight → Target-side SafetyGuard; mandatory for real-robot deployment.
📋	Fully Auditable — State, actions, and perception results are written to Markdown + YAML files; every step is traceable and reproducible.
🔄	Zero-Friction Migration — The same Session protocol runs identically across sim and real targets.

_{▲ Session-Centered Runtime Architecture Overview}

---

✨ Core Features

🔄	Session-Centered Runtime	WatchdogSupervisor → SessionRunner → SkillRuntime → TargetSessionHandle execution pipeline, replacing the legacy Driver-Center architecture
🎯	Target-Configured	Three target kinds — debug / simulation / real_robot — registered in TARGETS.md, adapters attached on demand
🧩	Adapter + Bridge	TargetAdapter + PolicyAdapter + ActionBridge three-way decoupling with explicit observation/action contracts; AdapterPlan auto-composed, eliminating target×skill combinatorial explosion
⚡	Dual Skill Runtimes	PolicySkillRuntime maintains policy closed-loop + BuiltinSkillRuntime manages agent interactive loop
🛡️	Strict Preflight	Runtime validation checks (target / sensor / perception / adapter contract / action contract / tool); failures are rejected before execution starts
📝	File Protocol Matrix	TARGETS.md · SKILLRUNTIME.md · SESSIONS.md · ENVIRONMENT.md · LESSONS.md + external YAML configs
🔐	Multi-Layer Safety	Critic validation → Preflight contract checks → Target-side SafetyGuard → Operator Override
🌐	Fleet Mode	Multi-robot coordination with shared + per-robot workspaces, priority-based serial scheduling

---

🚀 5-Minute Quick Start

1	Install git clone https://github.com/PhyAgentOS/PhyAgentOS.git && cd PhyAgentOS pip install -e . # Python ≥ 3.11 pip install -e ".[dev]" # Dev dependencies
2	Initialize Workspace paos onboard
3	Start Agent paos agent
4	Optional: Connect Runtime Services # LIBERO benchmark TargetWS machine MUJOCO_GL=egl PYTHONWARNINGS=ignore \ conda run -n liberopi python PhyAgentOS/runtime/targets/remote/libero/server.py \ --host 0.0.0.0 --port 9002 # pi0.5 policy machine conda run -n lerobot-pi python -m PhyAgentOS.runtime.policy.openpi.lerobot_pi0_server \ --model-dir /path/to/pi05/checkpoint --host 0.0.0.0 --port 8000

paos agent and paos gateway create the runtime workspace and start the session watchdog automatically when runtime is enabled in config. Runtime targets are declared in TARGETS.md, executable runtimes in SKILLRUNTIME.md, and the Agent queues work by appending sessions to SESSIONS.md.

paos agent -m "run the configured LIBERO benchmark task"

---

🗂️ Protocol Files

Context Loading	File	Owner	Purpose
Always loaded into the agent system prompt	AGENTS.md	Agent workspace	Project-level operating rules for the agent
Always loaded into the agent system prompt	SOUL.md	Agent workspace	Identity, high-level behavior, and assistant style
Always loaded into the agent system prompt	USER.md	Agent workspace	User preferences and durable profile notes
Always loaded into the agent system prompt	TOOLS.md	Agent workspace	Tool usage policy and available tool guidance
Always loaded into the agent system prompt	SKILLS.md	Agent workspace	Agent-facing skill discovery and loading rules
Loaded when present; filtered by enabled runtime targets where applicable	EMBODIED.md	Agent workspace	Human-readable target capability descriptions
Loaded when present as state, not bootstrap policy	ENVIRONMENT.md	Agent/runtime workspace	Current target and scene/environment state
Loaded when present as memory/state	LESSONS.md	Agent workspace	Operational lessons and failure notes
Loaded when present as task state	TASK.md	Agent workspace	Multi-step task decomposition and progress
Runtime protocol; read before scheduling sessions	RUNTIME.md	Runtime workspace	Instructions for writing valid runtime sessions
Runtime protocol; read before scheduling sessions	TARGETS.md	Runtime workspace	Enabled targets, endpoint/adapter/config references, supported skill runtimes
Runtime protocol; read before scheduling sessions	SKILLRUNTIME.md	Runtime workspace	Policy/builtin skill runtime registry and execution contracts
Runtime queue/state; written by Agent and watchdog	SESSIONS.md	Runtime workspace	Pending/running/completed execution sessions and results

SKILLS.md is for agent capabilities and skill discovery. SKILLRUNTIME.md is for runtime execution contracts; it is paired with TARGETS.md and SESSIONS.md.

---

📦 Project Structure

PhyAgentOS/
│
├── PhyAgentOS/agent/          # Track A  ─  Planner / Critic / Memory
│
├── PhyAgentOS/runtime/        # Track B  ─  Execution Plane
│   ├── watchdog/              #   WatchdogSupervisor
│   ├── sessions/              #   SessionRunner / TargetSessionHandle
│   ├── targets/               #   RolloutTarget (debug·sim·real)
│   │   └── remote/libero/     #   LIBERO benchmark TargetWS server + proxy
│   ├── skillruntime/          #   PolicySkillRuntime / BuiltinSkillRuntime
│   ├── adapters/              #   TargetAdapter / PolicyAdapter / Bridge
│   │   ├── libero/            #   LIBERO target adapter
│   │   └── openpi/            #   OpenPI policy adapters
│   ├── policy/openpi/         #   OpenPI client + LeRobot pi0-family server
│   ├── perception/            #   Perception Runtime / EnvironmentWriter
│   ├── preflight/             #   RuntimeCompatibilityPreflight
│   └── schemas/               #   Pydantic Schema
│
├── configs/runtime/           # Sensor / Perception / Contract YAML
├── scripts/                   # Utility scripts
├── workspace/                 # Agent workspace; runtime files may share it by config
├── docs/                      # Documentation
└── tests/                     # Tests

---

🏷️ Supported Targets

	Kind	Location	Examples
🐛	debug	Local	echo / mock / dry-run — zero-hardware protocol pipeline validation
🧪	simulation	Remote	RoboCasa, LIBERO — benchmark evaluation & batch experience mining
🤖	real_robot	Remote	Franka, Go2, XLeRobot, AgileX PIPER — real-world deployment

All targets are registered in TARGETS.md, identified by target_adapter:// URI. More examples & demos → Project Website

---

📖 Documentation

Document	Audience	Description
🌐 Website	Everyone	Full docs, architecture details, demos
📘 User Manual	Users	Installation, deployment, and operation guide
📙 Dev Guide	Developers	Secondary development, hardware integration, plugin authoring

---

🤝 Contributing

PRs and Issues are welcome! Check our development roadmap here → Dev Plan.

---

Built on nanobot

Jointly developed by Sun Yat-sen University HCP Lab & Peng Cheng Laboratory

       
_{MIT License · Copyright © 2025-2026 PhyAgentOS}