Skip to content

Kvadratni/Aquarius

BranchesTags

Repository files navigation

Aquarius 🌊

A native macOS AI agent that lives on your screen.

Aquarius is an embodied AI assistant that manifests as a floating, animated "ghost" — a living Metal shader that breathes, pulses, and reacts to your voice. It can see your screen, control your interface, and handle complex tasks while you focus on what matters.

Aquarius Logo

✨ Features

🗣️ Natural Voice Interaction

  • Push-to-Talk: Hold Control + G to speak — the ghost awakens and listens
  • Real-time voice conversations powered by Gemini Live
  • Bidirectional audio streaming with low latency

🧠 Dual-Brain Architecture

Brain Model Purpose
Fast Brain Gemini 2.5 Flash (Native Audio) Instant conversational responses
Deep Brain Gemini 3 Pro Complex reasoning & analysis

👁️ Vision & Awareness

  • Screen Sharing: Say "share my screen" and the agent can see what you see
  • Visual Halo: A subtle glow around your screen indicates when sharing is active
  • Real-time screen analysis for context-aware assistance

🛠️ Extensive Tool System

Aquarius uses a discoverable tool architecture — just describe what you need in natural language:

  • "Run npm install" → finds and uses shell tool
  • "Open Safari" → finds and uses open_application tool
  • "Edit the config file" → finds and uses text_editor tool
  • "Click on the submit button" → finds and uses click_mouse tool

Available Tools:

Developer Tools

  • shell — Execute terminal commands
  • text_editor — View, create, and edit files with precise replacements
  • analyze_code — Extract symbols, functions, classes from codebases
  • search_files — Grep through directories
  • list_directory / find_files — Navigate filesystems

Automation Tools

  • click_mouse / type_text — Control mouse and keyboard
  • run_applescript — Native macOS automation
  • open_application / close_application / focus_application — App management
  • find_element_coordinates — UI element discovery

Memory System

  • save_memory — Persist information long-term
  • search_memory — Recall stored knowledge
  • update_core_profile — Store user preferences & facts

Background Tasks & Task Queue

  • Long-running research and analysis tasks
  • Async execution with completion notifications
  • Configurable tool access levels
  • Idle-triggered execution — queued tasks run when you're inactive
  • Scheduled tasks — "remind me in 10 minutes", "check this at 3pm"
  • Repeatable tasks — daily, weekly, or custom intervals
  • Beautiful task overlay — ask "what's in your task list?" for an ethereal UI

⏰ Task Queue & Scheduling

  • "Remind me to check the build in 10 minutes" → schedules a reminder
  • "Research competitor pricing when I'm not busy" → queues for idle time
  • "Check my email every morning at 9am" → creates repeatable task
  • "What's in your task list?" → shows ethereal overlay with all tasks

Task Priorities:

Priority Behavior
Immediate Runs now in background
When Idle Waits for user inactivity
Scheduled Runs at specific time
Reminder Notifies user at specific time
Repeatable Runs at intervals (daily/weekly/custom)

Task Management:

  • "Cancel that reminder" → uses cancel_scheduled tool
  • "Stop the research task" → uses cancel_task tool
  • "What tasks are pending?" → uses get_tasks to list all tasks with IDs
  • "Show me my tasks" → displays ethereal overlay via show_task_list

🧠 Self-Learning Agent

Aquarius learns from your conversations, behavior, and connected services to become more helpful over time:

  • Pattern Detection: Extracts topics from conversations using NLP
  • Time-Based Learning: Notices when you ask about certain things
  • Recurring Interests: Tracks topics you frequently discuss
  • Proactive Suggestions: Offers help based on learned patterns
  • Automation Suggestions: Proposes tasks that could be automated
  • MCP Exploration: Proactively explores connected services (email, calendar, files) to discover patterns
  • Semantic Deduplication: Uses AI to merge similar patterns, keeping your learning store clean

Learning Sources:

Source What's Learned
Conversations Topics, preferences, workflows from your chats
MCP Tools Patterns from email, calendar, files (e.g., meeting habits, frequent contacts)
Ambient Audio Optionally learns from meetings via on-device transcription

How Learnings Are Used:

Method What's Included When
Auto-injected in context High-confidence patterns only (≥75%) Every conversation
get_patterns tool All patterns (any confidence) On request
Proactive suggestions Contextually relevant patterns During idle time

Learning Examples:

  • "What have you learned about me?" → shows all learned patterns via get_patterns
  • "Forget everything you've learned" → clears all patterns
  • User frequently asks about weather at 8am → Aquarius may proactively mention weather
  • MCP exploration finds unread emails pile up on weekends → suggests automation

Privacy-First:

  • All learning happens locally on your Mac
  • No data leaves your device
  • Toggle learning on/off in Settings
  • Toggle MCP exploration separately in Settings
  • View and delete learned patterns anytime
  • Configure idle threshold for learning in Settings
  • Block patterns you don't want re-learned

💡 Proactive Suggestions

Aquarius can proactively offer helpful suggestions based on learned patterns:

Suggestion Types:

Type Example
Automation "You manually check email every morning — want me to summarize it for you?"
Prediction "You have a meeting in 30 minutes — want me to prep?"
Reminder "You usually review PRs around this time"
Contextual "I see you're in Xcode — need help with Swift?"

Smart Delivery:

  • Only suggests when you're not actively engaged
  • Rate-limited (configurable, default 2/hour max)
  • Never repeats similar suggestions within 24 hours
  • Tracks delivered messages to prevent duplicates across restarts
  • Minimum confidence threshold (configurable)

Natural Integration:

  • Suggestions are delivered as natural conversation, not intrusive alerts
  • Aquarius stays silent if a suggestion isn't relevant or was recently mentioned
  • You can configure suggestion types in Settings

Settings:

  • Toggle proactive suggestions on/off
  • Adjust suggestions per hour (1-5)
  • Set confidence threshold (50%-90%)
  • Enable/disable automation vs prediction suggestions

🧹 Intelligent Data Management

Aquarius automatically manages its data storage to prevent disk bloat:

Memory Cleanup:

  • Memories with time-sensitive content ("tomorrow", "next week") auto-expire
  • Unused memories gradually decay in relevance
  • Old, low-relevance memories are summarized using AI
  • Duplicate memories are automatically merged

Conversation Compaction:

  • Recent conversations (7 days) kept in full
  • Older conversations summarized into insights
  • Summaries preserved as learning patterns

Log Rotation:

  • Daily log files with 7-day retention
  • 5MB size limit per file with automatic rotation

🔌 MCP (Model Context Protocol) Support

Extend Aquarius with external tools via MCP servers:

  • Standardized Integration: Connect any MCP-compatible server
  • Tool Discovery: MCP tools are automatically discoverable
  • Visual Output: Servers with UI can show WebView popups
  • Proactive Exploration: Aquarius explores connected MCPs to learn patterns

Example MCP Servers:

  • Gmail/Email servers for inbox insights
  • Google Calendar for scheduling patterns
  • Filesystem server for file operations
  • Slack/Communication tools for collaboration patterns
  • Database connectors
  • Custom tool servers

MCP Exploration (Proactive Learning):

When idle, Aquarius can automatically explore your connected MCP servers to discover patterns and automation opportunities:

Phase What Happens
Phase 1: Individual Exploration Each MCP gets its own agent with only its tools (parallel)
Phase 2: Cross-MCP Analysis All reports analyzed together for patterns

This enables insights like:

  • "You have 47 unread emails, mostly from automated systems"
  • "Meeting invites tend to arrive on Tuesdays"
  • "Files in ~/Projects haven't been accessed in 30 days"

Toggle MCP exploration in Settings → Learning → MCP Exploration.

Transport Types:

  • stdio — Local process (npx, python, node)
  • http — Remote HTTP server with SSE streaming
  • sse — Server-Sent Events

Configuration: Add servers via Settings → MCP tab, or edit ~/Library/Application Support/Aquarius/mcp_servers.json:

{
  "servers": {
    "filesystem": {
      "transport": "stdio",
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "~/Documents"],
      "enabled": true
    },
    "remote-api": {
      "transport": "http",
      "url": "https://mcp.example.com/api",
      "headers": {"Authorization": "Bearer token"},
      "enabled": true
    }
  }
}

🎨 The Ghost

A beautiful, breathing visualization powered by Metal shaders:

  • Red — Listening to your voice
  • Purple — Speaking a response
  • Orange — Working on a task
  • Idle — Small dot, always present

📋 Requirements

🚀 Getting Started

Option A: Download Pre-built Release

  1. Download the latest Aquarius-unsigned.zip from Releases
  2. Unzip the archive
  3. Remove quarantine (required for unsigned apps): 
    xattr -cr ~/Downloads/Aquarius.app
  4. Open the app normally

⚠️ The app is unsigned, so macOS Gatekeeper will block it without the xattr command above.

Option B: Build from Source

1. Clone the Repository

git clone git@github.com:Kvadratni/Aquarius.git
cd Aquarius

2. Generate the Xcode Project (Optional)

If you have xcodegen installed:

brew install xcodegen
xcodegen generate

3. Build & Run

Option A: Xcode GUI

open Aquarius.xcodeproj

Then build and run with Cmd + R in Xcode.

Option B: Command Line

# Clean build (removes previous build artifacts)
rm -rf build && xcodebuild -scheme Aquarius -destination 'platform=macOS' -derivedDataPath build build

# Open the built app
open build/Build/Products/Debug/Aquarius.app

4. First Launch Setup

  1. Enter your Gemini API Key in the setup window
  2. Grant Permissions when prompted:
    • Microphone — Required for voice input
    • Screen Recording — Required for screen sharing feature
    • Accessibility — Only needed for UI automation tools (mouse/keyboard control)

⌨️ Keyboard Shortcuts

Shortcut Action
Control + G Toggle listening mode

📋 Logging

Logs are written to a rotating log system:

~/Library/Application Support/Aquarius/Logs/aquarius-YYYY-MM-DD.log

Features:

  • Daily rotation with date-based filenames
  • Auto-rotates when file exceeds 5MB
  • Automatically deletes logs older than 7 days

To tail today's logs in real-time:

tail -f ~/Library/Application\ Support/Aquarius/Logs/aquarius-$(date +%Y-%m-%d).log

🔐 Permissions

Aquarius requires these permissions to function:

Permission Why It's Needed
Microphone Voice input for conversations
Screen Recording Screen sharing with the AI
Accessibility UI automation (mouse/keyboard control) — only needed if using automation tools

💡 The global hotkey (Control + G) works without any permissions using macOS's Carbon API.

⚠️ The app runs outside the sandbox to enable global hotkey detection and system-wide automation.

🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Aquarius App                            │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐     │
│  │   Ghost     │    │   Audio     │    │   Vision    │     │
│  │  (Metal)    │    │  Manager    │    │  Service    │     │
│  └──────┬──────┘    └──────┬──────┘    └──────┬──────┘     │
│         │                  │                  │             │
│         └──────────────────┼──────────────────┘             │
│                            │                                │
│                   ┌────────▼────────┐                       │
│                   │   AgentState    │                       │
│                   │  (ViewModel)    │                       │
│                   └────────┬────────┘                       │
│                            │                                │
│         ┌──────────────────┼──────────────────┐             │
│         │                  │                  │             │
│  ┌──────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐     │
│  │   Gemini    │    │  Reasoning  │    │    Tool     │     │
│  │    Live     │    │   Service   │    │  Registry   │     │
│  │ (Fast Brain)│    │(Deep Brain) │    │             │     │
│  └─────────────┘    └─────────────┘    └─────────────┘     │
└─────────────────────────────────────────────────────────────┘

📁 Project Structure

Aquarius/
├── AquariusApp.swift       # App entry point & window management
├── AgentState.swift        # Central state management & idle detection
├── ContentView.swift       # Ghost visualization
├── FloatingPanel.swift     # Transparent floating window
├── ScreenHaloView.swift    # Screen sharing indicator
├── TaskListOverlayView.swift # Ethereal task list UI
├── TaskListPanel.swift     # Floating panel for task overlay
├── MCPUIPanel.swift        # WebView panel for MCP UI output
├── MCPSettingsTab.swift    # MCP server management UI
├── Services/
│   ├── GeminiLiveService.swift    # WebSocket voice connection
│   ├── ReasoningService.swift     # Deep thinking integration
│   ├── ToolRegistry.swift         # Discoverable tools system
│   ├── TaskManager.swift          # Task queue, scheduling & execution
│   ├── LearningStore.swift        # Persistent pattern storage
│   ├── ObservationService.swift   # Behavior learning & analysis
│   ├── MemoryCleanupService.swift # Intelligent memory cleanup & summarization
│   ├── ScreenContextService.swift # Screen OCR & window tracking
│   ├── ConceptAnalysisService.swift # LLM concept extraction
│   ├── MCPClient.swift            # JSON-RPC client for MCP servers
│   ├── MCPManager.swift           # MCP server coordination
│   ├── MCPTypes.swift             # MCP data models
│   ├── MCPExplorationService.swift # Proactive MCP exploration for learning
│   ├── DeveloperService.swift     # Code & file operations
│   ├── ActionService.swift        # Mouse/keyboard automation
│   ├── AccessibilityService.swift # UI element discovery
│   ├── VisionService.swift        # Screen capture
│   ├── MemoryService.swift        # Long-term memory with decay
│   ├── LogManager.swift           # Rotating log system
│   ├── AudioManager.swift         # Mic & speaker handling
│   └── ...
└── Resources/
    ├── Shaders.metal       # Ghost visual effects
    └── logo.png

🤝 Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

📄 License

MIT License — see LICENSE for details.

Built with 💜 for macOS

About

Realtime Voice agent for your Mac powered by Gemini.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

2 tags

Packages

 
 
 

Contributors

Languages