Skip to main content
Live data · Updated 17 Nov 2026

AI Crawler IndexWhich AI bots crawl a real website, and how often

AI crawlers made 56% of our bot traffic in the last 30 days. We log every crawler that visits mojodojo.io and sort it into AI training bots, AI search bots, assistants reading a page for a user, and search engines like Google and Bing. Each visit is checked against the IP addresses the bot's owner publishes, so anything pretending to be Googlebot or GPTBot is left out. We started checking IPs on 7 Oct 2026, so earlier visits are sorted by the name the bot gives itself.

AI crawlers
55.8%

71,520 of 128,216 counted bot hits.

Search engines
24.0%

AI bots made 2.32 hits for every search engine hit.

Live AI fetches
2,598

Pages an assistant like ChatGPT read on the spot because someone asked it a question.

Fake bots
—

IP checks have just started. This fills in as verified traffic arrives.

Trend

Crawler hits per day, fakes removed

19 Oct 2026: 3,707 hits20 Oct 2026: 3,743 hits21 Oct 2026: 3,489 hits22 Oct 2026: 5,008 hits23 Oct 2026: 4,387 hits24 Oct 2026: 4,049 hits25 Oct 2026: 4,430 hits26 Oct 2026: 3,604 hits27 Oct 2026: 3,446 hits28 Oct 2026: 4,907 hits29 Oct 2026: 5,834 hits30 Oct 2026: 4,096 hits31 Oct 2026: 3,545 hits1 Nov 2026: 3,535 hits2 Nov 2026: 3,463 hits3 Nov 2026: 3,986 hits4 Nov 2026: 4,193 hits5 Nov 2026: 7,117 hits6 Nov 2026: 4,491 hits7 Nov 2026: 4,187 hits8 Nov 2026: 4,332 hits9 Nov 2026: 3,717 hits10 Nov 2026: 4,016 hits11 Nov 2026: 3,740 hits12 Nov 2026: 5,389 hits13 Nov 2026: 4,848 hits14 Nov 2026: 4,510 hits15 Nov 2026: 5,141 hits16 Nov 2026: 4,613 hits17 Nov 2026: 2,693 hits
19 Oct 2026Peak day: 7,117 hits17 Nov 2026
  • AI training
  • AI search
  • AI user fetch
  • Search engines
  • SEO tools
  • Social previews
  • Archive
  • Other bots
Split

Who is crawling

CategoryHitsShare
AI training60,21547.0%
AI search8,7076.8%
AI user fetch2,5982.0%
Search engines30,81524.0%
SEO tools22,76417.8%
Social previews2,5602.0%
Archive5570.4%
Other bots00.0%

Spoofed hits are excluded from every share on this page. 0 were removed in this period.

Leaderboard

Every bot, ranked

BotTypeHitsPer dayPagesHits / pageVerifiedFakes blocked
Meta-ExternalAgentMetaAI training52,5841,7532,58420.3——
PetalBotHuaweiSearch engines16,3795463,7844.3——
AhrefsBotAhrefsSEO tools16,1805393,4174.7——
PerplexityBotPerplexityAI search7,4572491,2795.8——
GooglebotGoogleSearch engines6,2742091,6603.8——
BingbotMicrosoftSearch engines4,8761632,7111.8——
AmazonbotAmazonAI training3,4221141,1493.0——
ClaudeBotAnthropicAI training2,660891,4571.8——
SemrushBotSemrushSEO tools2,280761,3701.7——
facebookexternalhitMetaSocial previews1,860628422.1——
MJ12botMajesticSEO tools1,619549141.8——
ApplebotAppleSearch engines1,529516842.2——
DotBotMozSEO tools1,479491,0141.5——
Screaming FrogScreaming FrogSEO tools1,206401,1611.0——
OAI-SearchBotOpenAIAI search1,060354632.3——
Claude-UserAnthropicAI user fetch983333213.1——
ChatGPT-UserOpenAIAI user fetch978333442.8——
GPTBotOpenAIAI training938317541.2——
SogouSogouSearch engines644223511.8——
LinkedInBotLinkedInSocial previews575191733.8——
Internet ArchiveInternet ArchiveArchive557193121.8——
GoogleOtherGoogleSearch engines502173151.6——
DuckAssistBotDuckDuckGoAI user fetch27491891.4——
BaiduspiderBaiduSearch engines22881521.5——
CCBotCommon CrawlAI training22271611.4——
YandexBotYandexSearch engines20471411.4——
BytespiderByteDanceAI training19471401.4——
cohere-aiCohereAI training19471531.3——
Claude-SearchBotAnthropicAI search19061451.3——
Meta-ExternalFetcherMetaAI user fetch18561421.3——
DuckDuckBotDuckDuckGoSearch engines1796603.0——
Perplexity-UserPerplexityAI user fetch17861371.3——
SlackbotSlackSocial previews632154.2——
TwitterbotXSocial previews231102.3——
DiscordbotDiscordSocial previews14172.0——
PinterestbotPinterestSocial previews14134.7——
WhatsAppMetaSocial previews7032.3——
TelegramBotTelegramSocial previews4031.3——
DiffbotDiffbotAI training1011.0——
Bot awards

The standouts of the last 30 days

Most aggressive
Meta-ExternalAgent
1,753 hits a day · Meta · AI training

Most hits per day

Most polite
Screaming Frog
1.0 hits per page · Screaming Frog · SEO tools
Top AI bot
GPTBot (OpenAI) · 1.2 hits per page

Fewest hits per page: visits once and moves on

Most obsessive
LinkedInBot
33.8 hits per page · LinkedIn · Social previews
Top AI bot
Meta-ExternalAgent (Meta) · 20.3 hits per page

Most hits per page: keeps fetching the same few

Most thorough
PetalBot
3,784 pages · Huawei · Search engines
Top AI bot
Meta-ExternalAgent (Meta) · 2,584 pages

Most different pages visited

Rarest visitor
Screaming Frog
4 days, 1,206 hits · Screaming Frog · SEO tools
Top AI bot
Bytespider (ByteDance) · 14 days, 194 hits

Fewest days seen (at least 10 hits)

Most impersonated
Collecting

Most fake hits caught by the IP check

A bot needs at least 100 hits in the period to win (10 for rarest visitor). Where an AI bot would win its own category, it is listed underneath.

Methodology

How the numbers are made

Source. Every page request to mojodojo.io whose user agent looks like a crawler, logged at the web server with the visitor's IP. One site, so read the numbers as a case study rather than a census. Tracking since 6 June 2026. IPs were recorded from 7 Oct 2026; hits before that are classified by user agent only and counted as unverified.

Categories. AI training bots collect pages for model training (GPTBot, ClaudeBot, CCBot). AI search bots build the index behind AI answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). AI user fetches are live reads on behalf of a person (ChatGPT-User, Claude-User, Perplexity-User).

Verification. A hit counts as verified when its IP sits inside the ranges the operator publishes (OpenAI, Anthropic, Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs), or passes forward-confirmed reverse DNS. Ranges refresh daily. A hit that fails the check is marked spoofed and left out. Bots whose operator publishes no method are counted as unverifiable.

Cite this. Mojo Dojo, “AI Crawler Index”, mojodojo.io/ai-crawler-index, accessed 17 Nov 2026. Free to reuse under CC BY 4.0 with a link back.

BotTypeCheck
ChatGPT-UserAI user fetchPublished IP ranges
Claude-UserAI user fetchPublished IP ranges
Perplexity-UserAI user fetchPublished IP ranges
MistralAI-UserAI user fetchNot verifiable
DuckAssistBotAI user fetchNot verifiable
Meta-ExternalFetcherAI user fetchNot verifiable
OAI-SearchBotAI searchPublished IP ranges
Claude-SearchBotAI searchPublished IP ranges
PerplexityBotAI searchPublished IP ranges
GPTBotAI trainingPublished IP ranges
ClaudeBotAI trainingPublished IP ranges
CCBotAI trainingPublished IP ranges
Meta-ExternalAgentAI trainingNot verifiable
FacebookBotAI trainingNot verifiable
BytespiderAI trainingNot verifiable
AmazonbotAI trainingReverse DNS
cohere-aiAI trainingNot verifiable
DiffbotAI trainingNot verifiable
GooglebotSearch enginesIP ranges + reverse DNS
GoogleOtherSearch enginesIP ranges + reverse DNS
BingbotSearch enginesIP ranges + reverse DNS
DuckDuckBotSearch enginesPublished IP ranges
ApplebotSearch enginesIP ranges + reverse DNS
YandexBotSearch enginesReverse DNS
BaiduspiderSearch enginesReverse DNS
PetalBotSearch enginesReverse DNS
SogouSearch enginesNot verifiable
AhrefsBotSEO toolsPublished IP ranges
SemrushBotSEO toolsReverse DNS
MJ12botSEO toolsNot verifiable
DotBotSEO toolsNot verifiable
Screaming FrogSEO toolsNot verifiable
facebookexternalhitSocial previewsNot verifiable
TwitterbotSocial previewsNot verifiable
LinkedInBotSocial previewsNot verifiable
PinterestbotSocial previewsReverse DNS
SlackbotSocial previewsNot verifiable
DiscordbotSocial previewsNot verifiable
TelegramBotSocial previewsNot verifiable
WhatsAppSocial previewsNot verifiable
Internet ArchiveArchiveNot verifiable