Skip to content

WHATSAPP_INTEGRATION.md

Latest commit

 

History

History
353 lines (266 loc) · 13.6 KB

WHATSAPP_INTEGRATION.md

File metadata and controls

353 lines (266 loc) · 13.6 KB

WhatsApp Integration Guide

Related docs: Tech overview · Architecture · Configuration guide

Overview

Astromesh supports WhatsApp as a messaging channel through the Meta Cloud API. Incoming WhatsApp messages are received via a webhook, routed to a configured agent for processing, and the agent's response is sent back to the user through the WhatsApp Business API.

The integration consists of three components:

  • WhatsApp Client (astromesh/channels/whatsapp.py) — Handles webhook verification, signature validation, payload parsing, and outbound message delivery via the Meta Graph API.
  • Webhook Route (astromesh/api/routes/whatsapp.py) — FastAPI endpoints that receive Meta webhook events and dispatch messages to the agent runtime in the background.
  • Channel Configuration (config/channels.yaml) — Declares credentials, the default agent, and rate limiting settings.
WhatsApp User
    |
    v
Meta Cloud API
    |
    v  (POST /v1/channels/whatsapp/webhook)
Astromesh Webhook Route
    |
    v
Agent Runtime (whatsapp-assistant)
    |
    v  (Graph API)
Meta Cloud API
    |
    v
WhatsApp User

Prerequisites

  • A Meta Business Account (free at business.facebook.com)
  • A Meta Developer App with the WhatsApp product added (developers.facebook.com)
  • A phone number registered with WhatsApp Business
  • Astromesh platform running (local or deployed)
  • Python 3.12+ and uv

Quick Start

  1. Install Astromesh (if not already done):

    uv sync

  2. Set environment variables with your Meta app credentials:

    export WHATSAPP_VERIFY_TOKEN="my-secret-verify-token"
export WHATSAPP_ACCESS_TOKEN="EAAx..."
export WHATSAPP_PHONE_NUMBER_ID="106..."
export WHATSAPP_APP_SECRET="abc123..."   # optional but recommended

  3. Start the Astromesh server:

    uv run uvicorn astromesh.api.main:app --host 0.0.0.0 --port 8000

  4. Expose your local server using ngrok (for development):

    ngrok http 8000

  5. Configure the webhook in Meta Developer Dashboard:

    • Go to your app > WhatsApp > Configuration
    • Set the Callback URL to https://<your-ngrok-url>/v1/channels/whatsapp/webhook
    • Set the Verify Token to the same value as WHATSAPP_VERIFY_TOKEN
    • Subscribe to the messages webhook field

  6. Send a test message from your WhatsApp phone to the registered business number. The whatsapp-assistant agent will process it and reply.

Environment Variables

Variable Required Description
WHATSAPP_VERIFY_TOKEN Yes A secret token you create for Meta webhook verification. Must match the value entered in the Meta Developer Dashboard.
WHATSAPP_ACCESS_TOKEN Yes Access token from Meta for calling the WhatsApp Cloud API. Found under your app's WhatsApp > API Setup page.
WHATSAPP_PHONE_NUMBER_ID Yes The phone number ID assigned to your WhatsApp Business number. Found under WhatsApp > API Setup.
WHATSAPP_APP_SECRET No Your Meta app's secret key. Used to validate X-Hub-Signature-256 headers on incoming webhooks. Strongly recommended for production.

Agent Configuration

WhatsApp messages are routed to the agent specified in config/channels.yaml (default: whatsapp-assistant). The sample agent definition lives at config/agents/whatsapp-assistant.agent.yaml:

apiVersion: astromesh/v1
kind: Agent
metadata:
  name: whatsapp-assistant
  version: "1.0.0"
  namespace: messaging
  labels:
    channel: whatsapp
    tier: production

spec:
  identity:
    display_name: "WhatsApp Assistant"
    description: "Conversational assistant optimized for WhatsApp messaging"

  model:
    primary:
      provider: ollama
      model: "llava:7b"
      endpoint: "http://ollama:11434"
      parameters:
        temperature: 0.4
        max_tokens: 1024
    fallback:
      provider: ollama
      model: "llama3.1:8b"
      endpoint: "http://ollama:11434"
      parameters:
        temperature: 0.4
        max_tokens: 1024
    routing:
      strategy: capability_match

  prompts:
    system: |
      You are a friendly and helpful WhatsApp assistant. Keep your responses concise
      and conversational — max 1-2 short paragraphs. Use a warm, approachable tone.

      You can receive images and will describe or answer questions about them.
      When you receive non-image media (audio, video, documents), acknowledge
      the attachment and let the user know what you can help with.

      Guidelines:
      - Be direct and get to the point quickly
      - Use simple language; avoid jargon
      - If you don't know something, say so and offer alternatives
      - Never share sensitive information in chat

  orchestration:
    pattern: react
    max_iterations: 5
    timeout_seconds: 30

  memory:
    conversational:
      backend: sqlite
      strategy: sliding_window
      max_turns: 30
      ttl: 86400

  guardrails:
    input:
      - type: pii_detection
        action: redact
      - type: max_length
        max_chars: 2000
    output:
      - type: pii_detection
        action: redact

Key considerations for WhatsApp agents:

  • Keep max_tokens low (1024 or less). WhatsApp messages display best when short.
  • Set timeout_seconds. WhatsApp users expect fast replies; 30 seconds is a reasonable upper bound.
  • Enable PII guardrails. WhatsApp messages often contain personal data. The pii_detection guardrail with redact action prevents the agent from echoing sensitive information.

To use a different agent, update the default_agent field in config/channels.yaml.

Webhook Setup

Development (ngrok)

  1. Install ngrok and authenticate:

    ngrok config add-authtoken <your-ngrok-token>

  2. Start Astromesh on port 8000, then start ngrok:

    ngrok http 8000

  3. Copy the https:// forwarding URL from ngrok output (e.g., https://a1b2c3d4.ngrok-free.app).

  4. In the Meta Developer Dashboard under WhatsApp > Configuration:

    • Callback URL: https://a1b2c3d4.ngrok-free.app/v1/channels/whatsapp/webhook
    • Verify Token: the value of your WHATSAPP_VERIFY_TOKEN environment variable
    • Click Verify and Save

  5. Under Webhook Fields, subscribe to messages.

Note: ngrok URLs change each time you restart. Update the callback URL in the Meta dashboard after each restart, or use a paid ngrok plan with a stable subdomain.

Production

  1. Deploy Astromesh behind a reverse proxy (nginx, Caddy, etc.) with a valid TLS certificate. Meta requires HTTPS for webhooks.

  2. Set the Callback URL in the Meta Developer Dashboard to your production endpoint:

    https://api.yourdomain.com/v1/channels/whatsapp/webhook

  3. Set the Verify Token to match your WHATSAPP_VERIFY_TOKEN environment variable and click Verify and Save.

  4. Subscribe to the messages webhook field.

  5. Set WHATSAPP_APP_SECRET to enable signature validation on incoming webhook requests (see Security Considerations).

Message Flow

1. User sends a WhatsApp message
2. Meta Cloud API delivers a POST to /v1/channels/whatsapp/webhook
3. Webhook route validates X-Hub-Signature-256 (if app secret is configured)
4. Webhook route parses the payload and extracts text messages
5. Each message is dispatched to the agent runtime as a background task
6. Agent runtime runs the configured agent (whatsapp-assistant)
   - Loads conversation memory (keyed by wa_<phone_number>)
   - Renders the system prompt
   - Executes the orchestration pattern (ReAct)
   - Applies guardrails on input and output
7. Agent response is sent back via Meta Graph API POST to /<phone_number_id>/messages
8. User receives the reply in WhatsApp

Channel Configuration

Channel settings are defined in config/channels.yaml:

channels:
  whatsapp:
    # WhatsApp Business Cloud API settings
    verify_token: "${WHATSAPP_VERIFY_TOKEN}"
    access_token: "${WHATSAPP_ACCESS_TOKEN}"
    phone_number_id: "${WHATSAPP_PHONE_NUMBER_ID}"
    app_secret: "${WHATSAPP_APP_SECRET}"

    # Which agent handles WhatsApp conversations
    default_agent: "whatsapp-assistant"

    # Rate limiting for outgoing messages
    rate_limit:
      window_seconds: 60
      max_messages: 30
Field Description
verify_token Token for Meta webhook verification handshake
access_token Bearer token for outbound Graph API calls
phone_number_id Phone number ID for sending messages
app_secret App secret for webhook signature validation
default_agent Name of the agent YAML (must exist in config/agents/)
rate_limit.window_seconds Rolling window duration for rate limiting
rate_limit.max_messages Maximum outbound messages per window

Credential values use the ${VAR_NAME} syntax and are resolved from environment variables at startup.

Multimedia Messages

The WhatsApp integration supports receiving multimedia messages including images, audio, video, and documents. Media is downloaded from Meta's servers, processed in-memory, and discarded after the agent responds — no media is stored on disk.

Supported Media Types

Type Behavior LLM Format
Image Downloaded and base64-encoded Sent as image_url content part to vision-capable models
Audio Downloaded Text description passed to agent (e.g., [Attached audio: audio/ogg, 15234 bytes])
Video Downloaded Text description passed to agent
Document Downloaded Text description with filename passed to agent

How It Works

  1. User sends a photo (or other media) via WhatsApp
  2. Meta webhook delivers the message with a media ID
  3. Astromesh downloads the media bytes via the Graph API (GET /{media_id} → download URL → bytes)
  4. The build_multimodal_query() utility converts the message:
    • Images → base64 data URL in OpenAI image_url format
    • Other media → text description for the agent
  5. The ModelRouter auto-detects image content and routes to a vision-capable model (e.g., llava:7b)
  6. Agent processes the query and responds as text via WhatsApp

Vision Model Configuration

To handle images, configure a vision-capable model as primary in the agent YAML:

spec:
  model:
    primary:
      provider: ollama
      model: "llava:7b"
    fallback:
      provider: ollama
      model: "llama3.1:8b"
    routing:
      strategy: capability_match

The capability_match routing strategy ensures image queries go to vision models, while text-only queries can use any available model.

Media Captions

WhatsApp image messages can include a caption. When present, the caption is included as the text part of the multimodal query alongside the image, so the agent can answer questions about the image directly.

Troubleshooting

Webhook verification fails

  • Confirm that WHATSAPP_VERIFY_TOKEN matches the value entered in the Meta Developer Dashboard.
  • Verify the callback URL ends with /v1/channels/whatsapp/webhook and is reachable over HTTPS.
  • Check Astromesh server logs for the incoming verification request.

Messages not arriving

  • Confirm the messages webhook field is subscribed in the Meta dashboard.
  • Check that your ngrok tunnel (or production endpoint) is running and accessible.
  • Look at Meta's webhook test tool in the dashboard to see delivery status.
  • Verify the phone number is registered and the WhatsApp Business account is active.

Agent not responding

  • Check that the agent specified in config/channels.yaml (default_agent) has a matching YAML file in config/agents/.
  • Verify the LLM provider endpoint is reachable (e.g., Ollama is running).
  • Review Astromesh logs for errors in background task processing.
  • Confirm WHATSAPP_ACCESS_TOKEN is valid and has not expired.

Rate limiting

  • Meta enforces rate limits on the WhatsApp Cloud API. If you receive 429 responses, reduce message throughput.
  • The rate_limit section in config/channels.yaml controls Astromesh's outbound rate. Adjust max_messages and window_seconds as needed.

Signature validation errors

  • Ensure WHATSAPP_APP_SECRET matches your app's secret in the Meta Developer Dashboard (under App Settings > Basic).
  • If you do not need signature validation during development, leave WHATSAPP_APP_SECRET unset. The client will skip validation when no secret is configured.

Security Considerations

  • Always use HTTPS in production. Meta requires a valid TLS certificate for webhook URLs. Use a reverse proxy with automatic certificate management (e.g., Caddy, Certbot with nginx).
  • Enable signature validation. Set WHATSAPP_APP_SECRET to your Meta app secret. The webhook route validates the X-Hub-Signature-256 header on every incoming request, rejecting payloads with an invalid or missing signature.
  • Enable PII guardrails. WhatsApp messages frequently contain phone numbers, names, and other personal data. Configure pii_detection guardrails on both input and output to redact sensitive information before it reaches the LLM or is stored in memory.
  • Apply rate limiting. Use the rate_limit configuration in config/channels.yaml to cap outbound messages per time window. This protects against runaway loops and excessive API usage.
  • Rotate access tokens. Meta access tokens can expire. Use System User tokens for long-lived production deployments and rotate them periodically.
  • Restrict network access. In production, restrict inbound traffic to Meta's webhook IP ranges where possible.