AI Crawler Access: GPTBot, ClaudeBot, PerplexityBot Explained
Every major AI engine has its own web crawler that accesses and indexes content from the web. These crawlers work similarly to Googlebot but serve a different purpose: instead of building a search index, they gather content that AI engines use for real-time retrieval and model training. Understanding which crawlers exist, how they behave, and how to configure your site to allow them is the first technical step in AEO.
GPTBot (OpenAI)
GPTBot is OpenAI's web crawler, used for both training data collection and real-time web browsing in ChatGPT. Its user agent string is 'GPTBot' and OpenAI publishes its IP ranges for verification. GPTBot respects robots.txt and crawls at a rate that should not significantly impact site performance. To allow GPTBot full access, add: User-agent: GPTBot / Allow: / to your robots.txt. OpenAI also provides a separate user agent 'ChatGPT-User' for the real-time browsing tool. Allow both for maximum ChatGPT citation coverage.
ClaudeBot (Anthropic)
ClaudeBot is Anthropic's web crawler, used for Claude AI's real-time web access and content indexing. Its user agent string is 'ClaudeBot'. Anthropic respects robots.txt instructions and publishes documentation on ClaudeBot behavior. To allow ClaudeBot, add: User-agent: ClaudeBot / Allow: / to your robots.txt. Claude AI uses web retrieval to supplement its knowledge base when users ask questions about specific brands or current information, making ClaudeBot access important for brands targeting Claude citation.
PerplexityBot (Perplexity AI)
PerplexityBot powers Perplexity AI's real-time web search and is one of the most important AI crawlers for brand citation because Perplexity prominently displays cited sources in every response. Its user agent string is 'PerplexityBot'. Perplexity AI uses citations as a core part of its user experience, meaning a site that is accessible to PerplexityBot and has well-structured content has a significant advantage in Perplexity citation frequency. Allow PerplexityBot with: User-agent: PerplexityBot / Allow: / in robots.txt.
Google-Extended (Google AI)
Google-Extended is a separate crawler used for Google's AI features including AI Overviews (formerly Search Generative Experience) and Bard/Gemini. It is distinct from Googlebot, which crawls for traditional search. Sites can block Google-Extended without affecting their traditional Google rankings, but doing so removes them from AI Overviews consideration. If you want to appear in Google AI Overviews, which are shown for a growing percentage of search queries, Google-Extended must be allowed in your robots.txt.
Verifying crawler access
After updating your robots.txt, verify access by checking your web server logs for requests from each crawler's user agent. You can also use Google Search Console's URL Inspection tool for Googlebot and Google-Extended. For third-party AI crawlers, use your site's access logs to confirm they are arriving and not receiving 403 or 404 responses. Run an AEO audit at /aeo/scores after making robots.txt changes to confirm that crawler access is no longer a failing signal in your score.
Ready to improve your AI visibility?
Run a free audit and get your score across 6 AEO categories.
Verify your AI crawler access