AEO Glossary
An AI crawler (also called an AI web scraper or LLM training crawler) is a web bot operated by an AI company to crawl public websites and collect text for training data or real-time retrieval. Major AI crawlers include OpenAI's GPTBot, Anthropic's ClaudeBot, Perplexity's PerplexityBot, and Google's Google-Extended.
If AI crawlers are blocked by your robots.txt - either intentionally or by a blanket disallow - AI models cannot crawl your site, cannot learn from your content, and are less likely to cite you accurately. Many sites inadvertently block AI crawlers with legacy robots.txt rules written before these crawlers existed.
Check your robots.txt at yourdomain.com/robots.txt and look for rules that might block GPTBot, ClaudeBot, PerplexityBot, or * (all bots). Add explicit Allow: / rules for each AI crawler. For Cloudflare-protected sites, check that Cloudflare's bot fight mode isn't blocking known AI crawlers.
See how your site scores on AI Crawler
Run a free AEO audit and get a 6-category breakdown including this metric.
Run Free AEO Audit →