Skip to content

telemetry_analysis.md

Latest commit

 

History

History
123 lines (95 loc) · 3.9 KB

telemetry_analysis.md

File metadata and controls

123 lines (95 loc) · 3.9 KB

Telemetry Analysis: Why PostHog Events Aren't Being Sent by Default

Executive Summary

The telemetry system is implemented but not actively integrated into the PraisonAI Agents codebase. While PostHog is properly configured and functional, telemetry data is never sent because:

  1. No automatic integration: Agent and PraisonAIAgents classes don't use telemetry
  2. No automatic flush: Events are collected but never sent to PostHog
  3. No lifecycle hooks: No atexit handler or periodic flush mechanism

Current State

What's Working ✓

  • PostHog client is properly initialized with API key and host
  • Telemetry can be enabled/disabled via environment variables
  • Privacy-first design with anonymous tracking
  • Manual telemetry tracking and flushing works correctly

What's Not Working ✗

  • Agent class doesn't integrate telemetry
  • PraisonAIAgents class doesn't integrate telemetry
  • No automatic flush() calls anywhere in the codebase
  • Integration module (integration.py) is not used

Technical Analysis

1. PostHog Configuration (telemetry.py)

# Lines 84-93: PostHog is initialized correctly
if POSTHOG_AVAILABLE:
    try:
        self._posthog = Posthog(
            project_api_key='phc_skZpl3eFLQJ4iYjsERNMbCO6jfeSJi2vyZlPahKgxZ7',
            host='https://eu.i.posthog.com'
        )
    except:
        self._posthog = None

2. Flush Implementation (telemetry.py)

# Lines 208-224: flush() sends events but is never called automatically
def flush(self):
    if hasattr(self, '_posthog') and self._posthog:
        try:
            self._posthog.capture(
                distinct_id='anonymous',
                event='sdk_used',
                properties={...}
            )
            self._posthog.capture('test-id', 'test-event')
        except:
            pass

3. Missing Integration Points

Agent Class (agent/agent.py)

  • No telemetry imports
  • No telemetry initialization in init
  • No telemetry tracking in start(), run(), or execute_tool()

PraisonAIAgents Class (agents/agents.py)

  • No telemetry imports
  • No telemetry initialization
  • No telemetry tracking in start() or workflow execution

Root Causes

  1. Incomplete Implementation: The telemetry system was built but never integrated into the main classes
  2. No Automatic Lifecycle Management: No atexit handler or periodic flush
  3. Integration Module Not Used: integration.py has the code but it's never imported/called

Recommendations

Immediate Fixes

  1. Add automatic integration to Agent class:
# In agent/agent.py __init__
from ..telemetry import get_telemetry
self._telemetry = get_telemetry()

# In start() method
if self._telemetry and self._telemetry.enabled:
    self._telemetry.track_agent_execution(self.name, success=True)
  1. Add atexit handler in telemetry.py:
import atexit

def _flush_on_exit():
    if _telemetry_instance:
        _telemetry_instance.flush()

atexit.register(_flush_on_exit)
  1. Add periodic flush:
# Flush after N events or T seconds
if self._metrics["total_events"] >= 100:
    self.flush()

Long-term Improvements

  1. Make telemetry opt-in by default with clear documentation
  2. Add telemetry configuration options (flush interval, batch size)
  3. Implement proper error handling for network failures
  4. Add telemetry dashboard for users to see their usage

Testing

Created test files demonstrate:

  • test_posthog.py: PostHog is working correctly
  • test_posthog_detailed.py: Events are sent when flush() is called
  • test_telemetry_integration.py: Integration is missing

Conclusion

The telemetry infrastructure is well-designed but incomplete. The main issue is that telemetry code exists in isolation and is never called by the core Agent/PraisonAIAgents classes. Additionally, even if it were integrated, the lack of automatic flush mechanisms means data would never reach PostHog.