Web crawler lookup
Browse observed bot user-agents and identify crawler families, source signals, and request patterns.
Meta External Agent
Meta · AI or training crawler
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
SleepBot
Unknown · Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36
FacebookExternalHit
Meta · Generic crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
YandexBot
Yandex · Search/index
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Bingbot
Microsoft · Search/index
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
Baiduspider
Baidu · Search/index
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Googlebot
Google · Search/index
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
AhrefsBot
Ahrefs · Generic crawler
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Qwantbot
Unknown · Generic crawler
Mozilla/5.0 (compatible; Qwantbot/1.0_2601038; +https://help.qwant.com/bot/)
Bytespider
ByteDance · AI or training crawler
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)
TikTokSpider
ByteDance · Generic crawler
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)
StartmeBot
Unknown · Generic crawler
StartmeBot/1.0 (+https://start.me/bot)
SERankingBacklinksBot
Unknown · Generic crawler
Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)
MJ12bot
Majestic · Generic crawler
Mozilla/5.0 (compatible; MJ12bot/v2.0.5; http://mj12bot.com/)
Other
Unknown · Generic crawler
WhatsApp/2.2616.100 W
Other
Unknown · Generic crawler
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Applebot
Apple · Search/index
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
keys-so-bot
Unknown · Generic crawler
Mozilla/5.0 (keys-so-bot)
Amazonbot
Amazon · Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
ClaudeBot
Anthropic · AI or training crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
360Spider
Unknown · Generic crawler
User-Agent:Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36; 360Spider
DotBot
Moz · Generic crawler
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
ChatGPT-User
OpenAI · User-triggered fetch
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
PetalBot
Unknown · Generic crawler
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)