AI crawler traffic statistics: AI bots vs Googlebot on a sports and politics site
MissedTakes grades NFL and US election pundits’ predictions. Every request from a crawler that names itself in its user agent is logged as it reaches the site, and this page adds them up: AI crawlers next to Googlebot, by company, by purpose and by kind of page. It is one sports and politics data site’s logs, not the web.
Requests logged on MissedTakes from 2026-09-06 to 2026-10-05 (UTC), 30 days.
- AI crawler requests
- 119,259
- Googlebot requests
- 3,664
- AI vs Googlebot
- 33 times
- Top AI crawler
- OAI-SearchBot
30 days to 2026-10-05
same days
as many requests
60,522 requests (OpenAI)
How many requests do AI crawlers make compared to Googlebot?
AI crawlers made 33 times as many requests as Googlebot: 119,259 against 3,664 in the 30 days to 2026-10-05. OAI-SearchBot alone made 17 times as many as Googlebot.
Which AI crawler crawls the most?
Download CSVOAI-SearchBot, OpenAI's crawler, made the most requests of any AI crawler: 60,522 in the 30 days to 2026-10-05, 51% of all AI crawler requests. Next came ClaudeBot (42,542) and GPTBot (15,989).
By company, OpenAI's crawlers made 64% of AI crawler requests, Anthropic's 36% and Perplexity's under 1%.
| Crawler | Company | Type | Requests | Refused (403) | Share of bot requests |
|---|---|---|---|---|---|
| OAI-SearchBot | OpenAI | AI | 60,522 | — | 29% |
| Unrecognized bots | Unknown | Other | 56,521 | — | 27% |
| ClaudeBot | Anthropic | AI | 42,542 | — | 20% |
| YandexBot | Yandex | Search engine | 21,955 | — | 10% |
| GPTBot | OpenAI | AI | 15,989 | — | 8% |
| Baiduspider | Baidu | Search engine | 7,831 | — | 4% |
| Googlebot | Search engine | 3,664 | — | 2% | |
| Bingbot | Microsoft | Search engine | 1,846 | — | 1% |
| ChatGPT-User | OpenAI | AI | 180 | — | under 1% |
| Applebot | Apple | Search engine | 63 | — | under 1% |
| PerplexityBot | Perplexity | AI | 20 | — | under 1% |
| LivelapBot | Livelap | Other | 8 | 6 | under 1% |
| Claude-User | Anthropic | AI | 6 | — | under 1% |
| DuckDuckBot | DuckDuckGo | Search engine | 3 | — | under 1% |
What share of crawler traffic is AI?
AI crawlers made 56% of the 211,150 requests we logged from bots in the 30 days to 2026-10-05; search engine crawlers made 17%, and 27% came from scrapers and bots that named no crawler we recognize.
Which crawlers does MissedTakes block?
LivelapBot (6) is banned on this site and answered with a 403 everywhere but robots.txt, so its requests in the 30 days to 2026-10-05 are attempts, not pages read.
Are AI crawlers collecting training data or answering users?
Crawlers their operators say collect training data made 49% of AI crawler requests; AI search indexers made 51%, and fetches made when a user asked an assistant under 1%.
| Stated purpose | Crawlers | Requests | Share of AI requests |
|---|---|---|---|
| Collects training data | ClaudeBot, GPTBot | 58,531 | 49% |
| Builds an AI search index | OAI-SearchBot, PerplexityBot | 60,542 | 51% |
| Fetches a page when a user asks | ChatGPT-User, Claude-User | 186 | under 1% |
Which pages do AI crawlers read most?
Graded takes drew the most AI crawler requests, 29% of AI crawler requests, followed by NFL player pages (15%) and about and policy pages (14%). Googlebot's most-requested family was graded takes, at 39% of its requests.
| Pages | AI crawlers | Share | Googlebot | Share |
|---|---|---|---|---|
| Graded takes | 34,382 | 29% | 1,431 | 39% |
| NFL player pages | 17,373 | 15% | 191 | 5% |
| About and policy pages | 16,445 | 14% | 14 | under 1% |
| Pundit records | 16,047 | 13% | 365 | 10% |
| NFL team pages | 7,302 | 6% | 39 | 1% |
| NFL hub and leaderboard pages | 6,833 | 6% | 73 | 2% |
| Publisher pages | 5,490 | 5% | 115 | 3% |
| Prediction-type pages | 4,195 | 4% | 30 | 1% |
| Posts | 3,309 | 3% | 74 | 2% |
| The home page | 1,979 | 2% | 10 | under 1% |
| Robots.txt, sitemaps and llms.txt | 1,759 | 1% | 1,097 | 30% |
| Other URLs | 1,486 | 1% | 17 | under 1% |
| NFL game pages | 1,353 | 1% | 83 | 2% |
| Election race pages | 654 | 1% | 70 | 2% |
| Other election pages | 650 | 1% | 55 | 2% |
| The methodology | 2 | under 1% | 0 | 0% |
Do AI crawlers read llms.txt?
No AI crawler requested /llms.txt or /llms-full.txt in the 30 days to 2026-10-05, against 710 AI crawler requests for robots.txt.
19,951 AI crawler requests were for the markdown version of a page (the same URL plus /md), 17% of all AI crawler requests.
Is AI crawling growing?
AI crawler requests rose 273% week over week: 63,137 in the week starting 2026-09-28, against 16,939 the week before. Across the last 8 complete weeks the weekly count ranged from 1,136 to 108,280.
The logs this page reads begin on 2026-08-03, 64 days of history; 8 complete weeks are compared here.
| Week starting | AI crawlers | Googlebot | All bots |
|---|---|---|---|
| 2026-08-10 | 108,280 | 671 | 111,053 |
| 2026-08-17 | 1,136 | 1,414 | 3,801 |
| 2026-08-24 | 3,078 | 109 | 6,678 |
| 2026-08-31 | 2,734 | 383 | 7,956 |
| 2026-09-07 | 5,287 | 95 | 11,410 |
| 2026-09-14 | 31,140 | 129 | 44,601 |
| 2026-09-21 | 16,939 | 125 | 32,043 |
| 2026-09-28 | 63,137 | 2,798 | 116,587 |
How this is counted
- A request is counted when its user agent names a crawler we recognize, before the page is served and whatever we answer. The site’s built scripts and styles are not counted; pages, share-card images, feeds, sitemaps, robots.txt and llms.txt are. Ordinary browser visits are never recorded.
- Banned crawlers are counted too: their requests are logged, then refused with a 403. jscrawler and LivelapBot are also denied by a firewall rule ahead of the site, so most of their attempts never reach this log at all.
- User agents are self-reported, and anyone can claim to be a crawler. AI crawlers are counted as they name themselves. 3,655 of Googlebot's 3,664 requests (100%) came from Google's own 66.249.x.x crawler addresses; the rest used the Googlebot name from other addresses.
- Google-Extended and Applebot-Extended are robots.txt switches, not crawlers, so they do not appear in logs. Google’s AI features read pages through Googlebot itself, so some Googlebot requests serve AI answers too; a log cannot tell which.
- A markdown fetch here is a request for a page’s /md twin by its own address. An agent that asks for markdown at the page’s URL (with an Accept header) gets the twin too, but is logged as a request for the page.
- MissedTakes is a small, specialized site: pundit records, graded takes, and NFL and election pages. A larger site, or one that blocks different crawlers, will see different numbers. This is one site, not the web.
- “Unrecognized” is traffic that looks automated but names no crawler we know. Only totals by crawler and by kind of page are published here: no addresses, no URLs, no single visits. How the log is kept is in the site notice.
- Whole UTC days, ending yesterday; re-read every few hours.
Cite these numbers
Free to reuse under CC BY 4.0 with credit and a link to this page. Please give the dates: the window moves every day. Figures for 2026-09-06 to 2026-10-05, read 2026-10-06.
MissedTakes, "AI crawler traffic statistics", missedtakes.co/ai-crawlers, accessed 2026-10-06. Requests from 2026-09-06 to 2026-10-05 (UTC).