You probably don't think about robots.txt anymore. It's 2025, and most SEO teams stopped caring about that file sometime around 2015. Then ChatGPT happened. Now OpenAI's crawler (GPTBot) is spidering the web constantly, looking for content to feed into Claude, Gemini, and every other AI model that wants to be useful. And your robots.txt might be telling it to go away.
Here's the problem: if GPTBot can't crawl your site, your content can't show up in AI answers. That's AEO (AI Engine Optimization) 101. ChatGPT won't cite you. Gemini won't surface your data. Claude won't find your expertise. You become invisible to the tools that are now the primary way people search.
The mistake happens silently. Most robots.txt files use a blanket rule like `User-agent: *` followed by `Disallow: /` or `Disallow: /admin`. That catches all bots. It caught Googlebot fine for years. But when OpenAI's crawler shows up, the same rule applies - and your content gets blocked.
Blocking AI crawlers isn't always accidental. Some brands think they can prevent their content from being used in AI responses by locking out GPTBot. That strategy doesn't work. Your competitors will get crawled. Your content will still appear in AI answers (competitors will cite you anyway). And you'll miss the chance to shape how your brand appears in AI citations.
Check your robots.txt file right now. Go to `yoursite.com/robots.txt` and search for GPTBot. If you see a `Disallow` rule that applies to it, you need to fix this today. The fix is simple: allow GPTBot to crawl your entire site, just like Googlebot.
The standard robots.txt should look like this: allow the User-agent for `*` (all bots), then add specific rules if needed - but don't block OpenAI. Better yet, explicitly allow GPTBot by adding a separate User-agent rule that says `Allow: /`. You want full visibility into how your content gets used in AI responses.
Some sites block GPTBot because they think it's too aggressive or because they read outdated guidance about protecting content. The real issue: you can't protect content by hiding it. Once your stuff is public, it's discoverable. The only question is whether you'll be in the room when it gets cited.
This matters because AEO isn't just about getting crawled. It's about getting cited correctly. When GPTBot finds your content, it can use proper attribution in AI answers. Your brand name appears. Your URL appears. A customer using ChatGPT sees that you're the source. That's traffic. That's authority.
GEO (Generative Engine Optimization) is becoming as important as SEO. The shift is already happening. But it only works if crawlers can actually reach your content. Blocking GPTBot is like unpublishing half your site - except nobody realizes it.
Beyond robots.txt, you need to understand your full AI visibility picture. Crawlers are reaching you, but are you optimized for how AI models use content? Are your key claims clear and discoverable? Is your structured data set up so that AI can understand what you do? Most brands haven't even checked.
Start by fixing robots.txt. Then run a full audit of your AI visibility. Check if GPTBot can crawl your site, whether your content is actually being picked up by AI models, and how you're being cited in responses. It takes 10 minutes. The difference in your AI presence over the next year will be enormous.
Head to engagemii.com/aeo and check your free AEO score. You'll get a real assessment of whether crawlers can reach you and how visible you actually are in AI responses. Fix the robots.txt issue first. Then see what else is holding you back from AI citations.
Ready to find out if AI can cite your brand?
Get Your Free AEO Score