GPTBot Crawls Steady. ClaudeBot Crawls in Bursts. What 2.85 Million AI Bot Visits Taught Us.

I run a website called engagemii.com that scores other websites for AI visibility. Our domain holds about 1.7 million scored business pages plus a couple thousand pages of editorial content, methodology, and product. That makes us a useful sample for watching how AI training bots actually behave in the wild. They visit us a lot.

AI training bots vastly outpace SEO bots in raw crawl volume. Source: Engagemii bot traffic dashboard, 2026.

What is more interesting than the totals is the way the two leaders behave.

Two completely different design choices

GPTBot is a metronome. It hits the site somewhere between 1 and 4 times per hour, almost every hour of the day, on weekdays and weekends, holidays included. There is no week where it goes quiet. There is no day where it shows up in a sudden flood. It is the most boring possible bot to graph. The line is essentially flat, and the lifetime total of 1.5 million is the product of that steady metronomic pace running for many months.

When you stand back and look at the trend lines, you can see two completely different design choices. One company has built a bot that polls the open web continuously, presumably distributing its fetches across thousands of small parallel jobs to minimize the chance of overwhelming any one host. The other company has built a bot that batches its work, presumably to take advantage of caching, parallelism, and dataset-style fetch jobs. Both approaches reach the same end result, which is that the AI engine ingests roughly the same percentage of the open web. They just get there by different roads.

What this means for site owners

This matters more than it might seem.

If you are a site owner trying to make sense of why your traffic looks one way and your traffic from one of these bots looks completely different, the pattern you see depends on which bot is doing the crawling. If ClaudeBot decides to sweep your site, the load can spike to thousands of requests per minute for a few minutes, and then return to nothing. That can look like an attack if you are not expecting it. The user agent string is, of course, identifiable, and Anthropic publishes their bot's behavior publicly. We have not seen anything from ClaudeBot that violates standard crawler etiquette. They respect robots.txt. They obey crawl-delay directives. They identify themselves correctly. They just do their crawling in bursts instead of drips.

GPTBot, on the other hand, will essentially never show up as a noticeable spike on your server logs. The traffic is too distributed in time. You can confirm GPTBot is reading your site only if you look at the cumulative count over the course of a week or month. From any single hour of logs, GPTBot looks like a couple of small isolated requests at most.

The cumulative reality is that both bots are reading your content seriously. The ratio between them on our site is roughly 1.7 to 1 in favor of GPTBot, but ClaudeBot has been catching up over the last two months. We did not run an experiment on this. We just watched our logs.

Why both companies are doing this

Both companies are continuously updating the world models that power their chat products. The simplest way to update a model is to keep reading the web. The model is only as good as the data it has, and the data it has needs to be current to be useful. If you ask ChatGPT or Claude about a small business or a niche product or a research paper from last week, the only way the model can give you a useful answer is if the relevant bot read that content recently and folded it into the training set or the retrieval system.

What you should change

If you are operating a website today, this should change a few of your assumptions.

First, the user agents your site is built to respond to are no longer just human browsers, Googlebot, and Bingbot. The two most important user agents for your discoverability over the next 24 months are GPTBot and ClaudeBot. If your robots.txt blocks them, you are not in the answer. We have written about this elsewhere. Most small business sites have this problem and do not know they do.

Second, your scoring on AI visibility (we call this AEO, Answer Engine Optimization) determines how often these bots even bother to read you. We ran an analysis on around 395,000 brands that have at least one AI bot crawl on record and found that the median site at AEO score 4 gets 1.3 bot crawls per month, while the median site at AEO score 8 gets 6.3 crawls per month. There is a steep cliff around score 8. The structural signals that get you over that cliff are inexpensive to install and most sites have not installed them.

Third, the burst pattern from ClaudeBot is going to make some people nervous about their server costs. It probably should not. Even in our heaviest observed burst, the crawler respected our infrastructure and finished its sweep without breaking anything. If your site is on standard hosting, you should be able to absorb an Anthropic-sized burst the way you would absorb a moderate news traffic spike. If your site cannot absorb that, you probably have other infrastructure problems you should be thinking about anyway.

About this analysis

We are going to keep watching this. Our site is in an unusual position to monitor it because we host a large number of brand pages that AI engines treat as a useful directory of small businesses. They show up to read them and they come back. We have built a small dashboard for this on the admin side and we plan to make it public soon. If you want to see your own AI bot traffic, the easiest path right now is the free AEO score at engagemii.com/aeo. If your site is set up correctly to be read by these bots, the score will tell you. If your site is not, the score will tell you that too, and the report will give you the specific things to fix.

Free AEO score: engagemii.com/aeo.

If you want to cite this article, the URL is engagemii.com/blog/gptbot-steady-claudebot-bursts-2-6-million-ai-crawls.