🏦 Multi-Agent AI KYC System

A sophisticated Multi-Agent AI System for automated Know Your Customer (KYC) verification, built with LangGraph state machines, Google Gemini LLM, and a tiered memory architecture. Designed for insurance and financial services applications.

📋 Table of Contents

Overview
Features
Architecture
Tech Stack
Project Structure
Prerequisites
Installation
Configuration
Running the Application
API Reference
Usage Examples
Memory System
Troubleshooting
Contributing
License

🎯 Overview

The Multi-Agent AI KYC System automates the customer identity verification process for TATA AIA Life Insurance. It uses a hierarchical multi-agent architecture where:

A Main Orchestrator routes user intents to appropriate workflows
Specialist Agents handle specific document verifications (Aadhaar, PAN, Passport, DL, Form 60)
LangGraph State Machines manage complex, interruptible verification workflows
Tiered Memory System maintains conversation context across sessions

The system supports both Web Interface (Flask + WebSocket) and CLI modes.

✨ Features

🤖 Multi-Agent Architecture

Main Orchestrator: Intent recognition and routing using structured LLM outputs
KYC Manager Agent: Delegates tasks to specialist agents
Specialist Agents: Aadhaar, PAN, Passport, DL, and Form 60 verification
General Query Agent: Handles insurance-related questions
PAN Check Agent: Determines if user needs PAN or Form 60

📄 Document Verification

Aadhaar Verification: eKYC via OTP or document upload with OCR
PAN Card Verification: NSDL database validation + OCR support
Passport Verification: Document extraction and validation
Driving License: OCR-based verification
Form 60: For users without PAN card

🧠 Advanced Memory System

L1 Working Memory: Redis-based short-term conversation buffer
L2 Episodic Memory: LLM-generated summaries stored in Redis
L3 Semantic Memory: Mem0-powered long-term user preferences

🎨 Modern Web Interface

Real-time WebSocket communication
Interactive FAQ sidebar
Live process status tracking
Responsive design for mobile/desktop
Typing indicators and smooth animations

🔧 Developer Features

LangSmith integration for tracing and debugging
Structured outputs with Pydantic models
Comprehensive error handling with retry mechanisms
Modular, extensible architecture

🏗️ Architecture

┌─────────────────────────────────────────────────────────────────┐
│                        User Interface                           │
│                  (Web App / CLI / API Client)                   │
└─────────────────────────────┬───────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      Main Orchestrator                          │
│  • Intent Recognition (LLM + Pydantic Structured Output)        │
│  • Workflow Routing                                             │
│  • Memory Context Integration                                   │
└─────────────────────────────┬───────────────────────────────────┘
                              │
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│  KYC Manager    │ │ General Query   │ │  PAN Check      │
│     Agent       │ │    Agent        │ │    Agent        │
└────────┬────────┘ └─────────────────┘ └─────────────────┘
         │
         ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Specialist Agents                            │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐│
│  │ Aadhaar  │ │   PAN    │ │ Passport │ │    DL    │ │Form 60 ││
│  │  Agent   │ │  Agent   │ │  Agent   │ │  Agent   │ │ Agent  ││
│  │(LangGraph│ │(LangGraph│ │(LangGraph│ │(LangGraph│ │        ││
│  │  FSM)    │ │  FSM)    │ │  FSM)    │ │  FSM)    │ │        ││
│  └──────────┘ └──────────┘ └──────────┘ └──────────┘ └────────┘│
└─────────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────────┐
│                      External Services                          │
│  ┌──────────────┐ ┌──────────────┐ ┌──────────────┐            │
│  │ UIDAI DB     │ │ NSDL DB      │ │ Document     │            │
│  │ (Aadhaar)    │ │ (PAN)        │ │ Intelligence │            │
│  └──────────────┘ └──────────────┘ └──────────────┘            │
└─────────────────────────────────────────────────────────────────┘

🛠️ Tech Stack

Component	Technology
LLM	Google Gemini 2.5 Flash/Pro
Agent Framework	LangGraph (State Machines)
Observability	LangSmith
Memory - L1	Redis
Memory - L3	Mem0
Web Framework	Flask + Flask-SocketIO
API Framework	FastAPI
OCR	Azure Document Intelligence
Data Validation	Pydantic
Language	Python 3.10+

📁 Project Structure

Multi-Agent-AI-KYC-System/
│
├── agent/                      # Agent implementations
│   ├── __init__.py
│   ├── base_agent.py          # Base class for all agents
│   ├── aadhar_agent.py        # Aadhaar verification (LangGraph FSM)
│   ├── pan_agent.py           # PAN verification (LangGraph FSM)
│   ├── passport_agent.py      # Passport verification
│   ├── dl_agent.py            # Driving License verification
│   ├── form60_agent.py        # Form 60 processing
│   ├── kyc_agent.py           # KYC Manager Agent
│   ├── genral_query_agent.py  # General insurance queries
│   └── pan_check_agent.py     # PAN existence checker
│
├── api/                        # External API integrations
│   ├── __init__.py
│   └── ocr_api.py             # Azure Document Intelligence
│
├── app/                        # FastAPI application
│   ├── __init__.py
│   ├── main.py                # FastAPI app definition
│   ├── models.py              # API request/response models
│   ├── dependencies.py        # Dependency injection
│   └── routers/
│       └── chat.py            # Chat API endpoints
│
├── config/                     # Configuration management
│   ├── __init__.py
│   ├── config.py              # Pydantic settings
│   └── settings.py            # Environment settings
│
├── data/                       # Mock databases
│   ├── database_nsdl.csv      # PAN verification data
│   └── database_uidai.csv     # Aadhaar verification data
│
├── memory/                     # Memory management
│   ├── __init__.py
│   └── memory.py              # Tiered memory system
│
├── models/                     # Pydantic models
│   ├── __init__.py
│   └── intent.py              # Intent classification models
│
├── orchestrator/               # Main orchestration logic
│   ├── __init__.py
│   └── router.py              # Main Orchestrator
│
├── prompts/                    # LLM prompt templates
│   ├── __init__.py
│   ├── aadhar_prompts.py
│   ├── pan_prompts.py
│   ├── form60_prompts.py
│   ├── greeting_prompts.py
│   ├── orchestrate.py
│   ├── prompts.py
│   └── prompts_short.py
│
├── static/                     # Web static files
│   ├── css/
│   │   └── style.css
│   └── js/
│       ├── script.js
│       └── sw.js
│
├── templates/                  # Jinja2 templates
│   └── index.html             # Main web interface
│
├── tools/                      # Verification tools
│   ├── __init__.py
│   ├── aadhar_tools.py        # Aadhaar validation tools
│   ├── pan_tools.py           # PAN validation tools
│   ├── ocr_tool.py            # OCR utilities
│   ├── ocr_pan_tool.py        # PAN OCR processor
│   └── tools_definition.py    # Tool definitions
│
├── .env                        # Environment variables (create this)
├── app.py                      # Flask web application
├── llm.py                      # LLM client factory
├── main_cli.py                 # CLI interface
├── requirements.txt            # Python dependencies
├── run_api.py                  # FastAPI startup script
├── run_web.py                  # Flask startup script
├── state.py                    # State definitions (TypedDict)
├── client_example.py           # API client example
├── webhook_example.py          # Webhook integration example
└── README.md                   # This file

📋 Prerequisites

Before running the application, ensure you have:

Python 3.10 or higher
pip (Python package manager)
Git (for cloning the repository)

Required API Keys/Services:

Google Gemini API Key (for LLM)
Mem0 API Key (for semantic memory)
Redis Cloud instance (for working memory)
Azure Document Intelligence (optional, for OCR)
LangSmith API Key (optional, for tracing)

🚀 Installation

1. Clone the Repository

git clone https://github.com/ishaanparikh14/Multi-Agent-AI-KYC-System.git
cd Multi-Agent-AI-KYC-System

2. Create a Virtual Environment

Windows:

python -m venv .venv
.venv\Scripts\activate

macOS/Linux:

python3 -m venv .venv
source .venv/bin/activate

3. Install Dependencies

pip install -r requirements.txt

Or install packages individually:

pip install openai langgraph langsmith mem0ai fastapi uvicorn flask flask-socketio python-multipart pydantic pydantic-settings httpx aiohttp requests aiofiles pandas Pillow redis python-dotenv typing-extensions Jinja2

⚙️ Configuration

1. Create Environment File

Create a .env file in the project root:

# LLM Configuration (Google Gemini)
GEMINI_API_KEY=your_gemini_api_key_here
GEMINI_BASE_URL=https://generativelanguage.googleapis.com/v1beta/openai/

# Memory - Mem0 (Long-term semantic memory)
MEM0_API_KEY=your_mem0_api_key_here

# Memory - Redis (Short-term working memory)
REDIS_HOST=your_redis_host
REDIS_DB_NAME=your_redis_db_name
REDIS_PASSWORD=your_redis_password

# OCR - Azure Document Intelligence (Optional)
DOCUMENT_INTELLIGENCE_API_KEY=your_azure_di_key
DOCUMENT_INTELLIGENCE_ENDPOINT=https://your-resource.cognitiveservices.azure.com

# LangSmith - Observability (Optional)
LANGSMITH_TRACING=false
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
LANGSMITH_API_KEY=your_langsmith_api_key
LANGSMITH_PROJECT=your_project_name

# ChromaDB (Optional - for vector storage)
CHROMA_TOKEN=your_chroma_token
CHROMA_TENANT=your_tenant
CHROMA_DATABASE=your_database

# Cohere (Optional - for embeddings)
COHERE_API_KEY=your_cohere_api_key

2. Get API Keys

Service	Sign Up URL	Purpose
Google Gemini	Google AI Studio	LLM for conversations
Mem0	Mem0.ai	Long-term memory
Redis Cloud	Redis Cloud	Short-term memory
Azure DI	Azure Portal	Document OCR
LangSmith	LangSmith	Tracing & debugging

🏃 Running the Application

Option 1: Web Interface (Recommended)

python run_web.py

Then open your browser to: http://localhost:5000

Option 2: Command Line Interface

python main_cli.py

Option 3: FastAPI Server

python run_api.py

Or with uvicorn directly:

uvicorn app.main:app --host 0.0.0.0 --port 8000 --reload

API documentation available at: http://localhost:8000/docs

📡 API Reference

Chat Endpoint

POST /api/chat

Send a message to the KYC assistant.

Request Body:

{
    "message": "I want to start KYC verification",
    "session_id": "optional-session-id"
}

Response:

{
    "response": "Welcome! I'm Siddhi from TATA AIA...",
    "session_id": "abc123",
    "active_workflow": "aadhaar",
    "kyc_step": "awaiting_verification_method_selection",
    "completed_workflows": []
}

Session Status

GET /api/session/{session_id}/status

Get current session status and progress.

Response:

{
    "active_workflow": "aadhaar",
    "kyc_step": "awaiting_otp",
    "completed_workflows": ["pan"],
    "aadhar_details": {},
    "pan_details": {}
}

💡 Usage Examples

Starting KYC Verification

User: Hi, I want to complete my KYC
Bot: Welcome! I'm Siddhi from TATA AIA Life Insurance. 
     Let's start with Aadhaar verification. Would you prefer:
     1. eKYC (OTP-based verification)
     2. DigiLocker

Aadhaar Verification Flow

User: I'll go with eKYC
Bot: Great! Please enter your 12-digit Aadhaar number.

User: 123456789012
Bot: I've sent an OTP to your registered mobile. Please enter the 6-digit OTP.

User: 123456
Bot: ✅ Aadhaar verified successfully!
     Name: John Doe
     DOB: 01/01/1990
     Is this information correct?

PAN Verification Flow

User: Now I want to verify my PAN
Bot: I see your name from Aadhaar is "John Doe" and DOB is "01/01/1990".
     Please enter your PAN card number.

User: ABCDE1234F
Bot: ✅ PAN verified successfully against NSDL database!

🧠 Memory System

The system implements a 3-tier memory architecture:

L1 - Working Memory (Redis)

Stores last 6 conversation turns
Fast access for immediate context
Automatic sliding window

L2 - Episodic Memory (Redis)

LLM-generated summaries of conversations
Triggered after threshold of turns
Preserves important facts and decisions

L3 - Semantic Memory (Mem0)

Long-term user preferences
Cross-session memory
Agent-specific memories (Aadhaar agent, PAN agent, etc.)

🔧 Troubleshooting

Common Issues

1. Port Already in Use

# Windows
netstat -ano | findstr :5000
taskkill /PID <PID> /F

# Linux/macOS
lsof -i :5000
kill -9 <PID>

2. Redis Connection Failed

Verify Redis host, port (default: 10908), and password in .env
Check if Redis Cloud firewall allows your IP

3. LLM API Rate Limits

The system includes automatic retry with exponential backoff
Consider upgrading your Gemini API quota

4. Import Errors

Ensure you're running from the project root directory
Activate the virtual environment before running

5. OCR Not Working

Verify Azure Document Intelligence credentials
Check endpoint URL format

Debug Mode

Enable debug logging:

# In config/config.py
debug: bool = True

Enable LangSmith tracing:

LANGSMITH_TRACING=true

🤝 Contributing

Contributions are welcome! Please follow these steps:

Fork the repository
Create a feature branch (git checkout -b feature/AmazingFeature)
Commit your changes (git commit -m 'Add AmazingFeature')
Push to the branch (git push origin feature/AmazingFeature)
Open a Pull Request

Code Style

Follow PEP 8 guidelines
Use type hints
Add docstrings to functions and classes

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👤 Author

Ishaan Parikh

GitHub: @ishaanparikh14

🙏 Acknowledgments

LangChain for the LangGraph framework
Google for Gemini API
Mem0 for the memory platform
TATA AIA for the use case inspiration

Made with ❤️ for the future of AI-powered KYC

Name		Name	Last commit message	Last commit date
Latest commit History 12 Commits
__pycache__		__pycache__
agent		agent
api		api
app		app
config		config
data		data
memory		memory
models		models
orchestrator		orchestrator
prompts		prompts
static		static
templates		templates
tools		tools
.env		.env
README.md		README.md
WEB_README.md		WEB_README.md
app.py		app.py
client_example.py		client_example.py
llm.py		llm.py
main_cli.py		main_cli.py
requirements.txt		requirements.txt
run_api.py		run_api.py
run_web.py		run_web.py
state.py		state.py
test_pan.jpeg		test_pan.jpeg
test_web.py		test_web.py
webhook_example.py		webhook_example.py

Folders and files

Latest commit

History

Repository files navigation

🏦 Multi-Agent AI KYC System

📋 Table of Contents

🎯 Overview

✨ Features

🤖 Multi-Agent Architecture

📄 Document Verification

🧠 Advanced Memory System

🎨 Modern Web Interface

🔧 Developer Features

🏗️ Architecture

🛠️ Tech Stack

📁 Project Structure

📋 Prerequisites

Required API Keys/Services:

🚀 Installation

1. Clone the Repository

2. Create a Virtual Environment

3. Install Dependencies

⚙️ Configuration

1. Create Environment File

2. Get API Keys

🏃 Running the Application

Option 1: Web Interface (Recommended)

Option 2: Command Line Interface

Option 3: FastAPI Server

📡 API Reference

Chat Endpoint

Session Status

💡 Usage Examples

Starting KYC Verification

Aadhaar Verification Flow

PAN Verification Flow

🧠 Memory System

L1 - Working Memory (Redis)

L2 - Episodic Memory (Redis)

L3 - Semantic Memory (Mem0)

🔧 Troubleshooting

Common Issues

Debug Mode

🤝 Contributing

Code Style

📄 License

👤 Author

🙏 Acknowledgments

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages