Web crawler lookup
Browse known bot user-agents and identify crawler families, source signals, behavior flags, and request patterns.
500 crawler profiles
Checking crawler access for your own site? Run the AI Agent Readiness Scanner to test robots.txt, llms.txt, structured data, and AI crawler readiness.
060Spider
Generic crawler
Mozilla/5.0 (Windows NT 5.1; WOW64) AppleWebKit/537.06 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 060Spider
2ip bot
Developer tool
2ip bot/1.1 (+http://2ip.io)
360Spider
Search crawler
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/037.36; 360Spider
AbbySEOBot
Generic crawler
AbbySEOBot/1.0 (+https://abbyseo.com/bot)
ABEvalBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ABEvalBot/0.1) Version/11.1.2 Safari/605.1.15
AcademicBotRTU
Generic crawler
AcademicBotRTU (https://academicbot.rtu.lv; mailto:[email protected])
adbeat_bot
Data collection crawler
adbeat_bot
AdsBot-Google
Search crawler
AdsBot-Google (+http://www.google.com/adsbot.html)
AdsBot-Google-Mobile
Generic crawler
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)
AffsignalCrawler
Generic crawler
Mozilla/5.0 (compatible; AffsignalCrawler/1.0; +https://affsignal.com/bot)
AgentDataBot
Generic crawler
AgentDataBot/1.0 (+https://agentdata.run/bot)
AhrefsBot
Ahrefs · SEO audit crawler
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
AhrefsSiteAudit
SEO audit crawler
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
AI2Bot
Data collection crawler
AI2 Bot
AIFengXiangTa-ToolIndexBot
Generic crawler
AIFengXiangTa-ToolIndexBot/0.1 (+personal non-commercial AI tool index; single-threaded, respects robots.txt)
AIFinderBot
Generic crawler
AIFinderBot/0.1 (+https://github.com/kisookang/AIFinder; korean-fit-check)
aiHitBot
Data collection crawler
Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)
AionBot
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 (compatible; AionBot/1.0)
aitoolnavio-crawler
Generic crawler
Mozilla/5.0 (compatible; aitoolnavio-crawler/1.0; +https://aitoolnavio.com)
AkoniLeadHuntBot
Generic crawler
AkoniLeadHuntBot/0.1 (+https://www.recruiting-systems.com)
AlexandriaOrgBot
Search crawler
Mozilla/5.0 (Linux) (compatible; AlexandriaOrgBot/1.0; +https://www.alexandria.org/bot.html)
AliyunSecBot
Generic crawler
AliyunSecBot/Aliyun ([email protected])
Amazonbot
Amazon · Search crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
AndersPinkBot
Data collection crawler
Mozilla/5.0 (compatible; AndersPinkBot/1.1; +http://anderspink.com/bot.html)
AonsightMetaBot
Generic crawler
Mozilla/5.0 (compatible; AonsightMetaBot/1.0; +https://aonsight.com/bot)
ApcosysBot
Generic crawler
ApcosysBot/1.0 (+https://apcosys.net)
Applebot
Apple · Search crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Applebot
Apple · Data collection crawler
Applebot-Extended
AppWatchBot
Generic crawler
AppWatchBot/0.1 (+https://appwatch.ai/bot; freshness check only)
Aranet-SearchBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36
archive.org_bot
Generic crawler
Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/7d76126 warc/v0.8.102
Arquivo-web-crawler
Archive crawler
Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling)
AtlasBot
Generic crawler
AtlasBot/1.0 (+https://atlas.harmoniamix.com/bot)
Atom Feed Robot
URL fetcher
RSSMicro.com RSS/Atom Feed Robot
AudienceExplorerBot
Generic crawler
Mozilla/5.0 (compatible; AudienceExplorerBot/0.2; site classification, one request per site)
AutotouchCompanyCrawler
Generic crawler
AutotouchCompanyCrawler/1.0 (+https://autotouch.ai)
AuvraStarshotBot
Generic crawler
Mozilla/5.0 (compatible; AuvraStarshotBot/1.0; +https://auvralabs.com)
AvanooCatalogBot
Generic crawler
Mozilla/5.0 (compatible; AvanooCatalogBot/1.0)
AwarioBot
Data collection crawler
Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)
AwarioSmartBot
Generic crawler
AwarioSmartBot/1.0 (+https://awario.com/bots.html; [email protected])
AzureAI-SearchBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;
BacklinksExtendedBot
Generic crawler
Mozilla/5.0 (compatible; BacklinksExtendedBot)
Baiduspider
Baidu · Search crawler
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
Barkrowler bot
SEO audit crawler
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
BarrigazBot
Generic crawler
BarrigazBot/2.0 (+https://barrigaz.com/bot; [email protected])
BazQux bot
URL fetcher
Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)
BeansLLM-CorpusBot
Generic crawler
BeansLLM-CorpusBot/1.1 (+https://scrape.beansllm.ai/bot; respects robots.txt)
Bingbot
Microsoft · Search crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36
BingPreview
URL fetcher
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
bitlybot
URL fetcher
bitlybot/3.0 (+http://bit.ly/)
BitSightBot
Data collection crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.86 Safari/537.36 BitSightBot/1.0
BixelBot
Generic crawler
BixelBot/1.0 (+https://bixel.com/docs/crawler)
BlueBot-scout-url
Generic crawler
Mozilla/5.0 (compatible; BlueBot-scout-url/0.1)
Brand360Bot
Generic crawler
Mozilla/5.0 (compatible; Brand360Bot/1.0; +https://be1st.ai)
Bravebot
Generic crawler
Mozilla/5.0 (compatible; Bravebot/1.0; +https://brave.com/search/)
Brightbot
Generic crawler
Brightbot 1.0
BugBuzzerBot
Generic crawler
BugBuzzerBot/0.1
Bytespider
ByteDance · Search crawler
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])
CCBot
Common Crawl · Data collection crawler
CCBot/2.0 (https://commoncrawl.org/faq/)
ChatGLM-Spider
Generic crawler
Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://zhipuai.cn/)
ChatGPT-User
OpenAI · AI assistant
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Claude-SearchBot
Anthropic · AI search crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +mailto:[email protected]
Claude-User
Anthropic · AI assistant
Claude-User (claude-code/2.1.321; +https://support.anthropic.com/)
ClaudeBot
Anthropic · Data collection crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; ClaudeBot/1.0; [email protected])
CloudflareBrowserRenderingCrawler
Generic crawler
CloudflareBrowserRenderingCrawler/1.0
ClueWeb-Crawler
Generic crawler
ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:[email protected])
CodeMMPretrainCrawler
Generic crawler
CodeMMPretrainCrawler/1.0
cohere-training-data-crawlera
Data collection crawler
cohere-training-data-crawler ([email protected])
CountryBot
Generic crawler
Mozilla/5.0 (compatible; CountryBot/0.1; +https://example.org/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36
CreditOneBot
Generic crawler
Mozilla/5.0 (compatible; CreditOneBot/1.0; +https://www.credit-one.it)
CricketBot
Generic crawler
CricketBot/1.0 (+https://engine.cricket/bot; [email protected])
CrossoverLeadsBot
Generic crawler
Mozilla/5.0 (compatible; CrossoverLeadsBot/1.0; +https://crossoverleads.com/bot)
DataBeezBot
Generic crawler
DataBeezBot/1.0 (+https://www.databeez.it)
DataForSeoBot
Generic crawler
Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
Daum bot
Search crawler
Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
dcrawl bot
URL fetcher
dcrawl/1.0
deadlinkchecker bot
SEO audit crawler
www.deadlinkchecker.com Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
DeccanCorpusBot
Generic crawler
Mozilla/5.0 (compatible; DeccanCorpusBot/1.0; +https://deccan.ai/bot) AppleWebKit/537.36
DeepSeekBot
Generic crawler
Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/)
DesearchBot
Generic crawler
Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler)
Diffbot
Data collection crawler
Mozilla/5.0 (compatible; Diffbot/1.0; +https://diffbot.com)
DigitaluBot
Generic crawler
DigitaluBot/1.0 (+https://digitalu.live/bot; platform detection; one request per site)
Discordbot
URL fetcher
Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)
DomainStatsBot
Data collection crawler
DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)
DotBot
Moz · SEO audit crawler
Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])
DuckAssistBot
AI assistant
DuckAssistBot/1.1 (https://duckduckgo.com/duckassistbot)
DuckDuckBot
Search crawler
DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)
eBoolBot
Generic crawler
eBoolBot/0.1 (+https://www.ebool.com/about)
ElevenlabsCrawler
Generic crawler
Mozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bots)
EmailEnrichmentBot
Generic crawler
Mozilla/5.0 (compatible; EmailEnrichmentBot/1.0; +contact-verification)
Embedly
URL fetcher
Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)
EnforceGateBot
Generic crawler
Mozilla/5.0 (compatible; EnforceGateBot/0.1; +https://www.enforcegate.com/bot)
EtaoSpider
URL fetcher
EtaoSpider
ev-crawler
Data collection crawler
Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/web-crawler)
EveMeCrawler
Generic crawler
EveMeCrawler/1.0 (+https://eveme.app)
ExaSearchBot
Generic crawler
Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)
EzoicBot
Ezoic · Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 X-Middleton/1 (compatible; EzoicBot-IntegrationCheck; +https://www.ezoic.com/bot/)
FacebookExternalHit
Meta · URL fetcher
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
FacturDirectoryBot
Generic crawler
FacturDirectoryBot/1.0 (+https://data.facturmfg.com/)
FAST-WebCrawler
Generic crawler
FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)
FeedBot
Generic crawler
FeedBot/1.0
Feedfetcher-Google
URL fetcher
Feedfetcher-Google
Feedly bot
URL fetcher
Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)
FirmableBot
Generic crawler
FirmableBot/1.0 (domain-check)
FlipboardProxy
URL fetcher
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)
FokalsBot
Generic crawler
Mozilla/5.0 (compatible; FokalsBot/1.0; +https://www.fokals.com/bot)
FreshRSS bot
URL fetcher
FreshRSS/1.23.1 (Linux; https://freshrss.org)
Friendica bot
URL fetcher
Friendica/x.x.x
FudgeResearchBot
Generic crawler
FudgeResearchBot/1.0 (+https://design.withfudge.com; [email protected])
FullStackGTM-bot
Generic crawler
Mozilla/5.0 (compatible; FullStackGTM-bot/1.0; +https://fullstackgtm.com/bot)
FundigBot
Generic crawler
FundigBot/1.0 (+https://console.fundig.app/bot)
fyndbot
Generic crawler
fyndbot (page-validation; https://fynd.bot)
GeneralCrawlBot
Generic crawler
GeneralCrawlBot/0.1
GeniusDataBot
Generic crawler
Mozilla/5.0 (compatible; GeniusDataBot/1.0; +https://geniusdata.com)
GenomeBot
Generic crawler
Mozilla/5.0 (compatible; GenomeBot/1.0; +https://51ultron.com)
GlobRadarBot
Generic crawler
Mozilla/5.0 (compatible; GlobRadarBot/1.0; +https://globradar.com/bot)
GolgulBot
Generic crawler
GolgulBot/1.0 (+https://golgul.com; news date backfill)
Google Web Preview
URL fetcher
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51
Google-InspectionTool
Developer tool
Mozilla/5.0 (compatible; Google-InspectionTool/1.0)
Google-Read-Aloud
URL fetcher
Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)
Google-Safety bot
Security scanner
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)
Googlebot
Google · Data collection crawler
Uses standard Googlebot user agent strings
Googlebot
Google · Search crawler
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Googlebot
Google · Search crawler
Googlebot-Image/1.0
Googlebot
Google · Search crawler
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)
Googlebot
Google · Search crawler
Mozilla/5.0 (compatible; Googlebot-News/2.1; +http://www.google.com/bot.html)
Googlebot
Google · Search crawler
Googlebot-Video/1.0
GoogleOther
Google · URL fetcher
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; GoogleOther)
GPTBot
OpenAI · Data collection crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)
GrokBot
Generic crawler
Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/)
HaloBot
Generic crawler
Mozilla/5.0 (compatible; HaloBot/1.0)
HaoSouSpider
Search crawler
Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)
Hatena bot
Archive crawler
HatenaBlog-bot/0.02 (+https://help.hatenablog.com/entry/about-hatenablogbot)
HeadlessChrome
Developer tool
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/125.0.6422.142 Safari/537.36
heritrix
Archive crawler
Mozilla/5.0 (compatible; heritrix/3.1.1 +http://archive.org)
HiWolfsBot
Generic crawler
HiWolfsBot/1.0 (+https://hiwolfs.com; careers-page indexer)
HTTrack bot
Archive crawler
Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)
HuaweiCrawler
Generic crawler
Mozilla/5.0 (Windows NT 10.0; WOW64; iPhone) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.4188.69 Safari/537.36 HuaweiCrawler
HunterLogosRobot
Generic crawler
HunterLogosRobot/1.0 (+https://logos.hunter.io/robot)
hypestat bot
SEO audit crawler
Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/bot)
ia_archiver
Archive crawler
ia_archiver
iaskspider
Generic crawler
Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)
IbouBot
Generic crawler
Mozilla/5.0 (compatible; IbouBot/1.0; [email protected]; +https://ibou.io/iboubot.html)
ICP-Research-Bot
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36 ICP-Research-Bot
Iframely bot
URL fetcher
Iframely/1.3.1 (+https://iframely.com/docs/about)
ImagesiftBot
URL fetcher
Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)
ImprintsCollectionBot
Generic crawler
Mozilla/5.0 (compatible; ImprintsCollectionBot/1.0; +https://www.fim.uni-passau.de/data-science)
IndexaroLeadBot
Generic crawler
IndexaroLeadBot/0.1 (+https://indexaro.com; polite startup discovery)
INETDEX-BOT
Generic crawler
Mozilla/5.0 (compatible; INETDEX-BOT/2.1; +https://inetdex.com/bot.php)
inoreader bot
URL fetcher
Inoreader/1.0 (http://www.inoreader.com)
IntercomDetectBot
Generic crawler
IntercomDetectBot/1.0 (+https://supportman.io/bot)
IonCrawl
Data collection crawler
IonCrawl
IoRobot-Python
Generic crawler
IoRobot-Python/1.0
IsonSearchBot
Generic crawler
IsonSearchBot/1.0 (+https://search.isonai.com/bot)
jobs-crawler
Generic crawler
jobs-crawler/0.2
JobUnlockedBot
Generic crawler
Mozilla/5.0 (compatible; JobUnlockedBot/1.0; +https://jobunlocked.com)
Jugendschutzprogramm-Crawler
Data collection crawler
Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de
KakaoBot
Generic crawler
Mozilla/5.0 (compatible; KakaoBot/1.0; +https://cs.kakao.com/helps_html/1073213012)
KambriumSourcingBot
Generic crawler
KambriumSourcingBot/0.1 (+https://kambrium.ai; contact: [email protected])
Kangaroo Bot
Data collection crawler
Kangaroo Bot
KeenableBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot)
keys-so-bot
Generic crawler
Mozilla/5.0 (keys-so-bot)
Kimi-SearchBot
Generic crawler
Mozilla/5.0 (compatible; Kimi-SearchBot/1.0; +https://kimi.ai/)
KimiBot
Generic crawler
Mozilla/5.0 (compatible; KimiBot/1.0; +https://kimi.ai/)
Konturbot
Generic crawler
Mozilla/5.0 (compatible; Konturbot/1.2; +http://kontur.ru; [email protected])
KosoBot
Generic crawler
KosoBot/1.0 (+https://koso.ai/bot)
KStandBot
Data collection crawler
Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
lainarchive-crawler
Generic crawler
lainarchive-crawler/1.0 (+https://lainarchive.com; [email protected])
LanternBot
Generic crawler
LanternBot/1.0 (+https://selfhost-search.rork.app/about; independent search index)
LeadFinderBot
Generic crawler
LeadFinderBot/1.0 public business contact discovery
LeadGennyBot
Generic crawler
Mozilla/5.0 (compatible; LeadGennyBot/1.0)
LeadGenResearchBot
Generic crawler
Mozilla/5.0 (compatible; LeadGenResearchBot/1.0; India PG checkout research; polite)
LeadGrowBot
Generic crawler
Mozilla/5.0 (compatible; LeadGrowBot/1.0)
LeadHQBot
Generic crawler
Mozilla/5.0 (compatible; LeadHQBot/1.0; +https://lead.verifyhq.io)
LeadOceanBot
Generic crawler
LeadOceanBot/1.0 (+https://leadocean.io/bot)
LightspeedSystemsCrawler
Security scanner
LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)
linkdexbot
SEO audit crawler
Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/)
LinkedInBot
SEO audit crawler
LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)
linkfluence bot
Data collection crawler
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
LinkScoutBot
Generic crawler
Mozilla/5.0 (compatible; LinkScoutBot/1.0)
LinkupBot
Generic crawler
LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; [email protected])
ListerBot
Generic crawler
ListerBot/0.1 (+https://www.lister.ai/contact; directory research)
LoadedBot
Generic crawler
Mozilla/5.0 (compatible; LoadedBot/1.0; +https://loaded.ai/bot)
LocalDomainCrawler
Generic crawler
LocalDomainCrawler/5.0
LohiSoftBot
Generic crawler
LohiSoftBot/1.0 (+https://lohisoft.com/bot)
LyonlBot
Generic crawler
Mozilla/5.0 (compatible; LyonlBot/1.0; +https://lyonl.com/bot.html)
magpie-crawler
Data collection crawler
magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)
Mail.RU_Bot
Search crawler
Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
MapTheNetBot
Generic crawler
MapTheNetBot/1.0 (+https://www.mapthenet.org/bot)
MarinaCrawler
Generic crawler
Mozilla/5.0 (compatible; MarinaCrawler/1.0; +https://paradisemedia.com)
Mastodon bot
URL fetcher
Mastodon/{version} (+https://{instance-domain}/)
mcbot
Generic crawler
Mozilla/5.0 (compatible; mcbot/0.1)
MeasureboardResearchBot
Generic crawler
Mozilla/5.0 (compatible; MeasureboardResearchBot/1.0; +https://measureboard.com/bot)
Mediapartners-Google
URL fetcher
Mediapartners-Google
Mediatoolkitbot
Data collection crawler
Mediatoolkitbot
MeridianBot
Generic crawler
Mozilla/5.0 (compatible; MeridianBot/1.0)
Meta External Agent
Meta · AI training crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Meta External Agent
Meta · Data collection crawler
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Meta-ExternalFetcher
AI search crawler
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
MetaCrawler
Generic crawler
MetaCrawler/1.0 (+https://metacrawler.com/bot)
MetaInspector bot
URL fetcher
MetaInspector/[version] (+https://github.com/jaimeiniesta/metainspector)
MeteoriaBot
Generic crawler
Mozilla/5.0 (compatible; MeteoriaBot/1.0; +https://meteoria.ai; [email protected])
MillionRoundsBot
Generic crawler
MillionRoundsBot/0.1 (+https://millionrounds.com/methodology)
MinakiBot
Generic crawler
Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0 Safari/537.36 (compatible; MinakiBot/0.1; +mailto:[email protected])
Miniflux bot
URL fetcher
Mozilla/5.0 (compatible; Miniflux/2.x.x; +https://miniflux.app)
Miruu-RecruiterBot
Generic crawler
Miruu-RecruiterBot/1.0 (+https://miruu.app)
MistralAI-User
Mistral AI · AI assistant
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)
MJ12bot
Majestic · SEO audit crawler
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
MojeekBot
Mojeek · Search crawler
Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html)
MsnBot-Media
Generic crawler
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b
MuckRack bot
Data collection crawler
Mozilla/5.0 (compatible; MuckRack/1.0; +https://muckrack.com)
mypagesbot
Generic crawler
Mozilla/5.0 (compatible; mypagesbot/1.0; +https://mypages.com/bot)
NagetBot
Generic crawler
Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)
NavCrawl
Generic crawler
NavCrawl/0.4 (+https://example.com/bot)
NetcraftSurveyAgent
Data collection crawler
Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])
NetpeakCheckerBot
Generic crawler
Mozilla/5.0 (compatible; NetpeakCheckerBot/3.7; +https://netpeaksoftware.com/checker)
Netvibes bot
URL fetcher
Netvibes (http://www.netvibes.com)
news-please bot
Data collection crawler
news-please
NewsBlur bot
URL fetcher
NewsBlur Feed Fetcher - {subscriber_count} subscribers - {site_url} (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)
NewSiteLeadBot
Generic crawler
NewSiteLeadBot/1.0 (+https://newsitelead.com/bot)
NewsletterFormResearchBot
Generic crawler
NewsletterFormResearchBot/1.3.1 (+read-only public-web research; no form submissions)
NewsNow bot
URL fetcher
NewsNow
newspaper bot
URL fetcher
newspaper/0.2.8
Nextcloud bot
Developer tool
Mozilla/5.0 (Nextcloud)
Nicecrawler
Data collection crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36
NoryoLogoDiscoveryBot
Generic crawler
NoryoLogoDiscoveryBot/1.0 (+https://noryo.app/bot)
NovaAct bot
URL fetcher
NovaAct
Nutch bot
Search crawler
Nutch/NutchOrg/NutchCVS
OAI-AdsBot
OpenAI · Ads safety crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot
OAI-SearchBot
OpenAI · AI search crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Offline Explorer bot
Archive crawler
Offline Explorer/2.5
omgili bot
Data collection crawler
omgili/0.5 +http://omgili.com
online-webceo-bot
SEO audit crawler
Mozilla/5.0 (compatible; online-webceo-bot/1.0; http://online.webceo.com)
OpenIBot
Generic crawler
Mozilla/5.0 (compatible; OpenIBot/1.0; +https://openi.ai)
OpenindexSpider
SEO audit crawler
Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)
openmagpie-jobcrawler
Generic crawler
Mozilla/5.0 (compatible; openmagpie-jobcrawler/0.1)
opr-crawler
Generic crawler
Mozilla/5.0 (compatible; opr-crawler/0.1.0; +mailto:[email protected])
Orbbot
Generic crawler
Mozilla/5.0 (compatible; Orbbot/1.1;)
osentix-crawler
Generic crawler
Mozilla/5.0 (compatible; osentix-crawler/1.0; +https://osentix.com/bot)
Other
Generic crawler
2ip bot/1.1 (+https://2ip.io/bot/)
Other
Generic crawler
ADOPS.online Domain Intelligence Bot (B2B Validation)
Other
Generic crawler
AITopToolsListingPreview/1.0 (+https://aitoptools.com)
Other
Generic crawler
AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)
Other
Generic crawler
Barrigaz harvester/2.0 (crawler; contact: [email protected])
Other
Generic crawler
BilateralNavigator-PageFetch/1.0 (+https://bilateralnavigator.com/bot)
Other
Generic crawler
CMSSurvey/1.0; +https://addedlovely.com/crawler
Other
Generic crawler
company-db-worker/0.1 (+https://businessmcp.com/bot)
Other
Generic crawler
CompleteGrowLinkHunter/2.0 respectful broken-link research crawler (+https://completegrow.com.au/)
Other
Generic crawler
Crawl360/1.0 (+account research; contact: [email protected])
Other
Generic crawler
CrawlConsole/1.0 (+https://crawlconsole.com)
Other
Generic crawler
CrawlConsole-MarkdownGenerator/1.0
Other
Generic crawler
CRM Hunter/1.0 (+https://example.com/bot)
Other
Generic crawler
ct-sender-poc/1.0 (stage 3 fetch)
Other
Generic crawler
DataEdge/crawler (+https://github.com/millstonehq/data-edge/issues)
Other
Generic crawler
EcommerceDetector/1.0 (+https://example.com/bot)
Other
Generic crawler
evergreen-site-survey/1.0 (+https://evergreendb.ai/crawler; one homepage request per domain)
Other
Generic crawler
favi.sh/1.0 (+https://favi.sh/legal/abuse; logo fetcher; contact [email protected])
Other
Generic crawler
file-preview-api-site-info/1.0
Other
Generic crawler
FirmaAPI/1.0 (+https://firmaapi.com/bot; [email protected]) SiteTara/1.0
Other
Generic crawler
first-sight-ct-monitor/3 (certificate transparency monitor; reads one public page per new certificate; +https://github.com/HaliSol)
Other
Generic crawler
FortLineAgentShieldDiscovery/1.0 (+https://fortline.ai/bot; AI app directory research; contact: [email protected])
Other
Generic crawler
go_infrastructure_fetch_cache/0.2.2 (+https://github.com/baditaflorin/go_infrastructure_fetch_cache)
Other
Generic crawler
Google-Adwords-Instant (+http://www.google.com/adsbot.html)
Other
Generic crawler
HeartbeatMonitor/1.0
Other
Generic crawler
meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Other
Generic crawler
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Other
Generic crawler
monitor-telegram-clone-realtime/1.0
Other
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; HubSpot Crawler; +https://www.hubspot.com) Chrome/131.0.0.0 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Other
Generic crawler
Mozilla/5.0 (compatible; AITopHubLiveCheck/1.0; +https://aitophub.example/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (compatible; ant-html/1.0; +https://kuaishou.com/bot)
Other
Generic crawler
Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/)
Other
Generic crawler
Mozilla/5.0 (compatible; crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; CT-WP-Scanner/1.0; +https://example.com/bot)
Other
Generic crawler
Mozilla/5.0 (compatible; dead-link-checker/0.1; +https://git.woa.com/web_search/spider/fetcher/databus/agent-browser)
Other
Generic crawler
Mozilla/5.0 (compatible; deadweb-prober/1.0; +https://github.com/Crawlora-org/dead-web-commoncrawl)
Other
Generic crawler
Mozilla/5.0 (compatible; DmarcerSpotter/1.0; +https://dmarcer.net/bot)
Other
Generic crawler
Mozilla/5.0 (compatible; domains-distillery/1.0; +https://crawler.validus.ai)
Other
Generic crawler
Mozilla/5.0 (compatible; ExaUser/1.0; +https://crawler.exa.ai/exauser)
Other
Generic crawler
Mozilla/5.0 (compatible; fresh-domain-watch/1.0; +mailto:[email protected]; opt-out: email UNSUB CRAWLER yourdomain.com)
Other
Generic crawler
Mozilla/5.0 (compatible; GlassDollarWWWIndexer/1.0; +https://www.angeldata.eu/bot)
Other
Generic crawler
Mozilla/5.0 (compatible; HubSpot Crawler; HubSpot Domain check; +https://www.hubspot.com)
Other
Generic crawler
Mozilla/5.0 (compatible; ibou-search/2.0; +https://ibou.com/crawlers.html)
Other
Generic crawler
Mozilla/5.0 (compatible; MerchantDiscovery/1.0; market research crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; Meta-ExternalFetcher/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; Meta-WebIndexer/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; Niche logo fetch)
Other
Generic crawler
Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
Other
Generic crawler
Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)
Other
Generic crawler
Mozilla/5.0 (compatible; urlsuma/2.0; +https://urlsuma.de/bot.html)
Other
Generic crawler
Mozilla/5.0 (compatible; WebCrawler/1.0)
Other
Generic crawler
Mozilla/5.0 (compatible; WebFetchHub/1.0)
Other
Generic crawler
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html��)
Other
Generic crawler
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Other
Generic crawler
Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])
Other
Generic crawler
Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
Other
Generic crawler
Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots)
Other
Generic crawler
Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b
Other
Generic crawler
Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)
Other
Generic crawler
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)
Other
Generic crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)
Other
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b
Other
Generic crawler
Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 [email protected]
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36 site-checker/0.1 (+https://payme.io/bot)
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 (compatible; Epochal-Search/1.0; +https://crawler.epochal.com)
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.32
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/152.0.7977.64 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/931.0.6778.764 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.76 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36
Other
Generic crawler
Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))
Other
Generic crawler
MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 4.4.2; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; Sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm#07)
Other
Generic crawler
NetAPI/1.1 (+https://netapi.com/bot.html)
Other
Generic crawler
[NVIDIA]-[[email protected]]-[NSPECT-I05S-FIEU]-[NVIDIABOT]
Other
Generic crawler
PageReceipt/1.0 (+local research; fetch-tls-track; curl-plain)
Other
Generic crawler
Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/)
Other
Generic crawler
Pandalytics/2.0 (https://domainsbot.com/pandalytics/)
Other
Generic crawler
PulseGate-Indexer/1.0 (+https://www.pulsegate.ai/bot)
Other
Generic crawler
RecordedFuture Global Inventory Crawler
Other
Generic crawler
Screaming Frog SEO Spider/19.2
Other
Generic crawler
Screaming Frog SEO Spider/20.2
Other
Generic crawler
Screaming Frog SEO Spider/23.0
Other
Generic crawler
Screaming Frog SEO Spider/23.1
Other
Generic crawler
Screaming Frog SEO Spider/24.0
Other
Generic crawler
Screaming Frog SEO Spider/24.3
Other
Generic crawler
SearchAtlas Bot (https://www.searchatlas.com)
Other
Generic crawler
Sogou Push Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07��)
Other
Generic crawler
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Other
Generic crawler
SpeshyWatcher/1.0 (+https://speshy.com.au/bot; price & stock watch on behalf of a subscriber)
Other
Generic crawler
SSI-Nutch/1.23 (SSI broad web crawler; https://ssi.inc/; [email protected])
Other
Generic crawler
WarmBadge-Trawler/1.0 (trust-layer research crawler; https://warmbadge.io/crawler)
Other
Generic crawler
WhatsApp/2.23.20.0
Other
Generic crawler
WhatsApp/2.2616.100 W
Other
Generic crawler
WhatsApp/2.2620.102 W
Other
Generic crawler
WhatsApp/2
PageSourceSearchBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PageSourceSearchBot/1.0; +https://pagesourcesearch.com/bot) Chrome/128.0.0.0 Safari/537.36
PaigeHeraldBot
Generic crawler
PaigeHeraldBot/1.0 (B2B directory research; +mailto:[email protected])
Pandalytics bot
Data collection crawler
Pandalytics/2.0 (https://domainsbot.com/pandalytics/)
PanguBot
Data collection crawler
PanguBot
panscient.com bot
Data collection crawler
panscient.com
PayMeMerchantBot
Generic crawler
PayMeMerchantBot/1.0 (+https://bot.payme.io)
PBN.BOT
Generic crawler
Mozilla/5.0 (compatible; PBN.BOT Monitoring/1.0)
peer39_crawler
Data collection crawler
peer39_crawler
PegiiBot
Generic crawler
Mozilla/5.0 (compatible; PegiiBot/1.0)
Perplexity-User
Perplexity · AI assistant
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user
PerplexityBot
Perplexity · Data collection crawler
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)
PetalBot
Search crawler
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Pingdom.com bot
URL fetcher
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)
Pinterestbot
URL fetcher
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)
PipericBot
Generic crawler
PipericBot/1.0 (+https://piperic.com/bot)
PocketParser
URL fetcher
PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)
PodcastPartnerResearchBot
Generic crawler
PodcastPartnerResearchBot/1.0 (+contact: owner; one-off research crawl)
Pomothy-Bot
Generic crawler
Mozilla/5.0 (compatible; Pomothy-Bot/1.0)
PopulusBot
Generic crawler
PopulusBot/1.0 (business contact pages; contact: [email protected])
PREMSGOBOT
Generic crawler
Mozilla/5.0 (compatible; PREMSGOBOT/4.0; +https://premsgo.fr/bot/; crawl global juillet/aout 2026)
ProductLookoutBot
Generic crawler
Mozilla/5.0 (compatible; ProductLookoutBot/1.0)
ProsAccessBot
Generic crawler
ProsAccessBot/1.0 (+https://prosaccess.com/privacy/professionals)
proximic
Data collection crawler
Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)
PublicEmailResearchBot
Generic crawler
Mozilla/5.0 (compatible; PublicEmailResearchBot/1.0; +https://example.com/bot-info)
PublicWWWBot
Generic crawler
Mozilla/5.0 (compatible; PublicWWWBot/1.0; +https://publicwww.com/bot.html)
PurchaseDetectorBot
Generic crawler
PurchaseDetectorBot/1.0 (+https://example.com/bot; [email protected])
QlyzeBot
Generic crawler
Mozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot)
Querit-SearchBot
Generic crawler
Mozilla/5.0 (compatible; Querit-SearchBot/1.0; +https://www.querit.ai)
QueritBot
Generic crawler
Mozilla/5.0 (compatible; QueritBot/1.0; +https://www.querit.ai/bot)
Quora-Bot
Search crawler
Quora-Bot/1.0
Qwairy-Bot
Generic crawler
Qwairy-Bot/1.0 (+https://qwairy.co)
Qwantbot
Search crawler
Mozilla/5.0 (compatible; Qwantbot/1.0; +https://help.qwant.com/bot/)
RankingsAtlasBot
Generic crawler
RankingsAtlasBot/0.1 (research crawler; one homepage per site; honours robots.txt)
RateOnTime-Bot
Generic crawler
RateOnTime-Bot/1.0 (+https://rateontime.com)
ReconBot
Generic crawler
Mozilla/5.0 (compatible; ReconBot/1.0; +https://scanner.insecurest.xyz/bot; [email protected])
redditbot
URL fetcher
redditbot
rednote-websearch-bot
Generic crawler
rednote-websearch-bot
RefgrowResearchBot
Generic crawler
RefgrowResearchBot/1.0 (+https://refgrow.com; contact [email protected])
Reflectionbot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Reflectionbot/1.0; +https://reflection.ai/bot) Chrome/151.0.0.0 Safari/537.36
RepetivBot
Generic crawler
Mozilla/5.0 (compatible; RepetivBot/1.0; +mailto:[email protected])
RevontPipelineBot
Generic crawler
Mozilla/5.0 (compatible; RevontPipelineBot/1.0; +https://revont.io)
RmagedonBot
Generic crawler
RmagedonBot/1.0 (+https://rmagedon.com/bot; [email protected])
robots.txt
Generic crawler
ccrawl/1.0 (company cache; respects robots.txt; contact: [email protected])
rogerbot
SEO audit crawler
rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, [email protected])
rolltheweb-crawler
Generic crawler
rolltheweb-crawler/2.2 (+https://rolltheweb.com/; homepage-only; obeys robots.txt)
S33DBot
Generic crawler
S33DBot/4.0 (+https://s33d.fr/bot)
SaaSBrowserBot
Generic crawler
Mozilla/5.0 (compatible; SaaSBrowserBot/1.0; +https://saasbrowser.com/bot)
SaaSRankingBot
Generic crawler
SaaSRankingBot/1.0 (+https://saas-ranking.com/bot)
SaluteBot
Generic crawler
Mozilla/5.0 (compatible; SaluteBot/1.0)
ScalinthBot
Generic crawler
Mozilla/5.0 (compatible; ScalinthBot/1.0; +https://scalinth.com/bot)
scoop.it bot
Archive crawler
Mozilla/5.0 (compatible; scoopit-crawler/3; +https://www.scoop.it/bot.html)
ScoutnerBot
Generic crawler
Mozilla/5.0 (compatible; ScoutnerBot/1.0; +https://scoutner.com/bot)
Scrapy bot
Developer tool
Scrapy/2.8.0 (+https://scrapy.org)
Screaming Frog SEO Spider
SEO audit crawler
Screaming Frog SEO Spider/[version number]
ScroResearchBot
Generic crawler
Mozilla/5.0 (compatible; ScroResearchBot/1.0; +https://scro.io; research draft)
SearchataCrawler
Generic crawler
SearchataCrawler/1.0 (+https://searchata.com/benchmarks/methodology)
SearchBot
Generic crawler
SearchBot/1.0 (compatible; +https://search.theradicalparty.com/bot)
SEBot-WA
Generic crawler
SEBot-WA
SeekportBot
Search crawler
Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)
SemanticScholarBot
URL fetcher
Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)
SemanticVisionsBot
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (Compatible; SemanticVisionsBot/1.0; +http://semantic-visions.com)
SemrushBot
Semrush · SEO audit crawler
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
SemrushBot
Semrush · SEO audit crawler
Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)
SemrushBot
Semrush · SEO audit crawler
SemrushBot-CT
SemrushBot
Semrush · SEO audit crawler
Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)
SemrushBot
Semrush · SEO audit crawler
SemrushBot-SWA
SenutoBot
SEO audit crawler
SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)
SeobilityBot
SEO audit crawler
SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)
SEOJuice-SearchBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/bot
SEOkicks-Robot
SEO audit crawler
Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robot.html)
SEOlizer bot
SEO audit crawler
SEOlizer/1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)
SERankingBacklinksBot
Generic crawler
Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)
SerendeputyBot
URL fetcher
SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)
SerpCrawler
Generic crawler
SerpCrawler/0.1 (+local research crawler)
Serpstatbot
Serpstat · SEO audit crawler
serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; [email protected])
SeznamBot
Search crawler
Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)
ShapBot
Parallel · AI search crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0
ShopContactCrawler
Generic crawler
ShopContactCrawler/0.1
SimplePie bot
URL fetcher
SimplePie/[VERSION] (Feed Parser; http://simplepie.org; Allow like Gecko) Build/[BUILD_NUMBER]
SiteAnalyzerbot
Generic crawler
SiteAnalyzerbot/4.0; +https://site-analyzer.pro/
SiteAuditBot
SEO audit crawler
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
SiteBot
Generic crawler
Mozilla/5.0 (compatible; SiteBot/1.0)
SiteCheckerBotCrawler
SEO audit crawler
SiteCheckerBotCrawler
SitemapCrawler
Generic crawler
SitemapCrawler/1.0 (semantic-visions.com crawler)
SkyclazBot
Generic crawler
Mozilla/5.0 (compatible; SkyclazBot/1.0; +https://skyclaz.com/bot)
Slack-ImgProxy
URL fetcher
Slack-ImgProxy 0.19 (+https://api.slack.com/robots)
Slackbot
URL fetcher
Slackbot 1.0 (+https://api.slack.com/robots)
Slackbot-LinkExpanding
URL fetcher
Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
SleepBot
Generic crawler
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36
SMTBot
Data collection crawler
Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)
Snap URL Preview
Snap · URL fetcher
Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots
Snap URL Preview
Snap · URL fetcher
Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots
SofyaBot
Generic crawler
SofyaBot/1.0 (+https://sofya.co/bot)
Sogou web spider
Search crawler
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
SparkleBot
Generic crawler
SparkleBot/1.0 (+https://sparkle.io/bot)
SpiderLing
SpiderLing · Generic crawler
Mozilla/5.0 (compatible; SpiderLing; +https://www.sketchengine.eu/crawler/)
SQLBot
Generic crawler
Mozilla/5.0 (compatible; SQLBot/2.0;)
SQLSpider
Generic crawler
Mozilla/5.0 (compatible; SQLSpider/1.7.0)
srchs-research-bot
Generic crawler
Mozilla/5.0 (compatible; srchs-research-bot/0.1; public small-business domain research; contact: [email protected])
StartmeBot
URL fetcher
StartmeBot/1.0 (+https://start.me/bot)
Storebot-Google
Generic crawler
Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/321.0.0.0 Safari/533.36
StormIntelCrawler
Generic crawler
StormIntelCrawler/1.0 (StormIntel Crawler; http://stormintel.com; [email protected])
Stripebot
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 (Stripebot/1.0; +https://docs.stripe.com/stripebot-crawler)
StyneBot
Generic crawler
Mozilla/5.0 (compatible; StyneBot/1.0; +https://styne.ai/bot)
SumbleBot
Generic crawler
python-httpx/0.28.1 (SumbleBot/1.0)
Superfeedr bot
URL fetcher
Superfeedr Bot (feed-id=12345)
SurdotlyBot
Security scanner
Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Synapse bot
URL fetcher
Mozilla/4.0 (compatible; Synapse)
T2Bot
Generic crawler
Mozilla/5.0 (compatible; T2Bot/1.0; +https://t2ind.com/bot)
t3versionsBot
Developer tool
Mozilla/5.0 (compatible; t3versionsBot/1.2; +https://www.t3versions.com/bot)
TalarionSearchBot
Generic crawler
Mozilla/5.0 (compatible; TalarionSearchBot/1.0; research; contact: [email protected])
TechLeadsBot
Generic crawler
Mozilla/5.0 (compatible; TechLeadsBot/1.0; +https://techleads.fyi/bot)
TechScanBot
Generic crawler
Mozilla/5.0 (compatible; TechScanBot/1.0)
TelegramBot
URL fetcher
TelegramBot
TenMillionDomainsBot
Generic crawler
Mozilla/5.0 (compatible; TenMillionDomainsBot/2.0; +https://github.com/Crawlora-org/ten-million-domains)
teoma bot
Search crawler
Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
TernCrawler
Generic crawler
Mozilla/5.0 (compatible; TernCrawler/1.0)
TestSearchSpider
Generic crawler
TestSearchSpider/0.1
TflowSitemapBot
Generic crawler
Mozilla/5.0 (compatible; TflowSitemapBot/1.0)
TheMapBot
Generic crawler
TheMapBot/0.1 (SaaS product enumeration research; +mailto:[email protected])
TikTokSpider
ByteDance · Generic crawler
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; [email protected])
Timpibot
Data collection crawler
Timpibot/1.0 (+http://timpi.io/crawler)
TinEye-bot
URL fetcher
Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)
Tiny Tiny RSS bot
URL fetcher
Tiny Tiny RSS/[version] (https://tt-rss.org/)
TLDWPBot
Generic crawler
TLDWPBot/1.0 (+https://tldwp.com/about.php)
TokenOpsBot
Generic crawler
Mozilla/5.0 (compatible; TokenOpsBot/1.0; +https://tokenops.io)
ToolissBot
Generic crawler
Mozilla/5.0 (compatible; ToolissBot/1.0)
tranco-crawler
Generic crawler
tranco-crawler/1.0
TrendBowlBot
Generic crawler
Mozilla/5.0 (compatible; TrendBowlBot/1.0; [email protected])
trendictionbot
Data collection crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0
trendictionbot0.5.0
Generic crawler
Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0
TrendsmapResolver
Data collection crawler
Trendsmap Resolver/0.0.1
TriumviryResearchBot
Generic crawler
Mozilla/5.0 (compatible; TriumviryResearchBot/0.1; +mailto:[email protected])
TurnitinBot
Data collection crawler
TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)
TweetmemeBot
URL fetcher
Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
Twingly Recon
Data collection crawler
Mozilla/5.0 (compatible; Twingly Recon; twingly.com), Twingly Recon-Klondike/1.0 (+https://developer.twingly.com)
Twitterbot
X · URL fetcher
TelegramBot (like TwitterBot)
um-LN
Data collection crawler
Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected]; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1
ValtrBot
Generic crawler
Mozilla/5.0 (compatible; ValtrBot/1.0; +https://valtr.xyz/bot)
VantageBot
Generic crawler
VantageBot/1.0 (+a16z company identity verification; contact [email protected])
VelenPublicWebCrawler
Data collection crawler
Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
VerticalSaaSResearchBot
Generic crawler
Mozilla/5.0 (compatible; VerticalSaaSResearchBot/1.0; homepage metadata research)
Viber bot
AI assistant
Viber
viewsrc-bot
Generic crawler
viewsrc-bot/0.1 (+https://viewsrc.io/bot)
virustotal bot
Security scanner
VirusTotal
vkShare
URL fetcher
Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)
VStatMetadataBot
Generic crawler
VStatMetadataBot/1.0 (+https://vstat.info/)
W3C_CSS_Validator
Developer tool
Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See http://validator.w3.org/services)
W3C_Validator
Developer tool
W3C_Validator/<version> http://validator.w3.org/services
WanscannerBot
Generic crawler
Mozilla/5.0 (X11; Linux x86_64; WanscannerBot/1.2; +https://abuse.pend.re) Gecko/20100101 Firefox/10.0
WappalyzerBot
Generic crawler
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/154.0.0.0 Safari/537.36 WappalyzerBot/1.0 (+https://www.wappalyzer.com/)
