AEO Glossary
robots.txt is a text file placed at the root of a website (yourdomain.com/robots.txt) that specifies crawling rules for web robots (crawlers and spiders). It uses User-agent directives to target specific bots and Allow/Disallow directives to control which paths they can access.
Many websites accidentally block AI crawlers with legacy robots.txt configurations - a blanket Disallow: / for all non-Google bots, or an old rule that predates GPTBot and ClaudeBot. If your robots.txt blocks AI crawlers, no amount of structured data or content optimization will help because the AI models simply cannot read your site.
Check yourdomain.com/robots.txt today. Look for any rules that would block GPTBot, ClaudeBot, or PerplexityBot - either directly or via a wildcard user-agent rule. Add explicit Allow: / rules for each AI crawler. If you need to restrict certain paths (admin areas, private content), use path-level Disallow rather than blocking the entire site.
See how your site scores on robots.txt
Run a free AEO audit and get a 6-category breakdown including this metric.
Run Free AEO Audit →