A complete, up-to-date database of AI search engine crawler user-agents and robots.txt configuration examples.
Check your site against all these crawlers instantly — no installation needed:
- AI Crawler Access Checker — Verify which AI search engines can access your site
- AI Robots.txt Generator — Generate an optimized robots.txt for AI crawlers
- Full GEO Scan — Comprehensive 11-signal AI search readiness scan
| User-Agent | Company | Purpose |
|---|---|---|
GPTBot |
OpenAI | Training data |
OAI-SearchBot |
OpenAI | ChatGPT Search |
ChatGPT-User |
OpenAI | Browsing mode |
PerplexityBot |
Perplexity AI | Search indexing |
Google-Extended |
Gemini/AI features | |
ClaudeBot |
Anthropic | Claude training |
Applebot-Extended |
Apple | Apple Intelligence |
Meta-ExternalAgent |
Meta | Meta AI |
cohere-ai |
Cohere | Model training |
Bytespider |
ByteDance | TikTok/AI |
- GEOScore — Check your site's AI search readiness
- AI Robots.txt Generator — Generate optimized robots.txt
- Robots.txt Checker — Validate AI crawler access
- GEOScore AI Scanner — Check your website's AI search visibility across 11 signals
- AI Robots.txt Generator — Generate optimized robots.txt for AI crawlers
- AI Crawler Access Checker — Verify which AI bots can access your site
- awesome-geo — Curated list of GEO resources, tools, and guides
- geo-scoring-methodology — Open methodology for scoring AI search readiness
- ai-robots-txt-generator — Generate optimized robots.txt for AI crawlers
- geo-checklist — Interactive pre-launch GEO readiness checklist
- geo-badge-generator — Generate badges showing your GEO readiness score
- llms-txt-examples — Real-world llms.txt implementation examples by industry
- geo-config-examples — Ready-to-use AI search optimization configs for popular frameworks
- geo-case-studies — Real-world GEO optimization case studies with before/after data
- ai-search-readiness-framework — 11-signal AI search readiness evaluation framework