# AI crawler traffic statistics: AI bots vs Googlebot on a sports and politics site

MissedTakes grades NFL and US election pundits' predictions. Every request from a crawler that names itself in its user agent is logged as it reaches the site; this page adds them up. It is one sports and politics data site's logs, not the web.

Requests logged on MissedTakes from 2026-09-06 to 2026-10-05 (UTC), 30 days.

## How many requests do AI crawlers make compared to Googlebot?

AI crawlers made 33 times as many requests as Googlebot: 119,259 against 3,664 in the 30 days to 2026-10-05. OAI-SearchBot alone made 17 times as many as Googlebot.

## Which AI crawler crawls the most?

OAI-SearchBot, OpenAI's crawler, made the most requests of any AI crawler: 60,522 in the 30 days to 2026-10-05, 51% of all AI crawler requests. Next came ClaudeBot (42,542) and GPTBot (15,989).

By company, OpenAI's crawlers made 64% of AI crawler requests, Anthropic's 36% and Perplexity's under 1%.

| Crawler | Company | Type | Requests | Refused (403) | Share of bot requests |
| --- | --- | --- | ---: | ---: | ---: |
| OAI-SearchBot | OpenAI | AI | 60,522 | — | 29% |
| Unrecognized bots | Unknown | Other | 56,521 | — | 27% |
| ClaudeBot | Anthropic | AI | 42,542 | — | 20% |
| YandexBot | Yandex | Search engine | 21,955 | — | 10% |
| GPTBot | OpenAI | AI | 15,989 | — | 8% |
| Baiduspider | Baidu | Search engine | 7,831 | — | 4% |
| Googlebot | Google | Search engine | 3,664 | — | 2% |
| Bingbot | Microsoft | Search engine | 1,846 | — | 1% |
| ChatGPT-User | OpenAI | AI | 180 | — | under 1% |
| Applebot | Apple | Search engine | 63 | — | under 1% |
| PerplexityBot | Perplexity | AI | 20 | — | under 1% |
| LivelapBot | Livelap | Other | 8 | 6 | under 1% |
| Claude-User | Anthropic | AI | 6 | — | under 1% |
| DuckDuckBot | DuckDuckGo | Search engine | 3 | — | under 1% |

## What share of crawler traffic is AI?

AI crawlers made 56% of the 211,150 requests we logged from bots in the 30 days to 2026-10-05; search engine crawlers made 17%, and 27% came from scrapers and bots that named no crawler we recognize.

## Which crawlers does MissedTakes block?

LivelapBot (6) is banned on this site and answered with a 403 everywhere but robots.txt, so its requests in the 30 days to 2026-10-05 are attempts, not pages read.

## Are AI crawlers collecting training data or answering users?

Crawlers their operators say collect training data made 49% of AI crawler requests; AI search indexers made 51%, and fetches made when a user asked an assistant under 1%.

| Stated purpose | Crawlers | Requests | Share of AI requests |
| --- | --- | ---: | ---: |
| Collects training data | ClaudeBot, GPTBot | 58,531 | 49% |
| Builds an AI search index | OAI-SearchBot, PerplexityBot | 60,542 | 51% |
| Fetches a page when a user asks | ChatGPT-User, Claude-User | 186 | under 1% |

## Which pages do AI crawlers read most?

Graded takes drew the most AI crawler requests, 29% of AI crawler requests, followed by NFL player pages (15%) and about and policy pages (14%). Googlebot's most-requested family was graded takes, at 39% of its requests.

Served requests only; a refused request read nothing.

| Pages | AI crawlers | Share | Googlebot | Share |
| --- | ---: | ---: | ---: | ---: |
| Graded takes | 34,382 | 29% | 1,431 | 39% |
| NFL player pages | 17,373 | 15% | 191 | 5% |
| About and policy pages | 16,445 | 14% | 14 | under 1% |
| Pundit records | 16,047 | 13% | 365 | 10% |
| NFL team pages | 7,302 | 6% | 39 | 1% |
| NFL hub and leaderboard pages | 6,833 | 6% | 73 | 2% |
| Publisher pages | 5,490 | 5% | 115 | 3% |
| Prediction-type pages | 4,195 | 4% | 30 | 1% |
| Posts | 3,309 | 3% | 74 | 2% |
| The home page | 1,979 | 2% | 10 | under 1% |
| Robots.txt, sitemaps and llms.txt | 1,759 | 1% | 1,097 | 30% |
| Other URLs | 1,486 | 1% | 17 | under 1% |
| NFL game pages | 1,353 | 1% | 83 | 2% |
| Election race pages | 654 | 1% | 70 | 2% |
| Other election pages | 650 | 1% | 55 | 2% |
| The methodology | 2 | under 1% | 0 | 0% |

## Do AI crawlers read llms.txt?

No AI crawler requested /llms.txt or /llms-full.txt in the 30 days to 2026-10-05, against 710 AI crawler requests for robots.txt.

19,951 AI crawler requests were for the markdown version of a page (the same URL plus /md), 17% of all AI crawler requests.

## Is AI crawling growing?

AI crawler requests rose 273% week over week: 63,137 in the week starting 2026-09-28, against 16,939 the week before. Across the last 8 complete weeks the weekly count ranged from 1,136 to 108,280.

The logs this page reads begin on 2026-08-03, 64 days of history; 8 complete weeks are compared here.

| Week starting | AI crawlers | Googlebot | All bots |
| --- | ---: | ---: | ---: |
| 2026-08-10 | 108,280 | 671 | 111,053 |
| 2026-08-17 | 1,136 | 1,414 | 3,801 |
| 2026-08-24 | 3,078 | 109 | 6,678 |
| 2026-08-31 | 2,734 | 383 | 7,956 |
| 2026-09-07 | 5,287 | 95 | 11,410 |
| 2026-09-14 | 31,140 | 129 | 44,601 |
| 2026-09-21 | 16,939 | 125 | 32,043 |
| 2026-09-28 | 63,137 | 2,798 | 116,587 |

## How this is counted

- Counted when the user agent names a crawler we recognize, before the page is served and whatever we answer. Built scripts and styles are not counted; pages, share-card images, feeds, sitemaps, robots.txt and llms.txt are. Ordinary browser visits are never recorded.
- Banned crawlers are logged, then refused with a 403. jscrawler and LivelapBot are also denied by a firewall rule ahead of the site, so most of their attempts never reach this log.
- User agents are self-reported. 3,655 of Googlebot's 3,664 requests (100%) came from Google's own 66.249.x.x crawler addresses; the rest used the Googlebot name from other addresses.
- A markdown fetch is a request for a page's /md twin by its own address; an agent that negotiates markdown at the page's URL is logged as a request for the page.
- One small, specialized site, not the web. Aggregates only: no addresses, no URLs, no single visits. Whole UTC days, ending yesterday; re-read every few hours.

## Cite these numbers

Free to reuse under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) with credit and a link. Please give the dates: the window moves every day.

MissedTakes, "AI crawler traffic statistics", missedtakes.co/ai-crawlers, accessed 2026-10-06. Requests from 2026-09-06 to 2026-10-05 (UTC).

CSV: https://missedtakes.co/ai-crawlers.csv