Web crawler lookup

Browse known bot user-agents and identify crawler families, source signals, behavior flags, and request patterns.

500 crawler profiles

Checking crawler access for your own site? Run the AI Agent Readiness Scanner to test robots.txt, llms.txt, structured data, and AI crawler readiness.

060Spider

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 5.1; WOW64) AppleWebKit/537.06 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 060Spider

2ip bot

Developer tool

Developer toolURL fetcherNot AI trainingRobots control limitedUser-triggeredv1.1

2ip bot/1.1 (+http://2ip.io)

360Spider

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/037.36; 360Spider

AbbySEOBot

Generic crawler

Generic crawlerv1.0

AbbySEOBot/1.0 (+https://abbyseo.com/bot)

ABEvalBot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ABEvalBot/0.1) Version/11.1.2 Safari/605.1.15

AcademicBotRTU

Generic crawler

Generic crawler

AcademicBotRTU (https://academicbot.rtu.lv; mailto:[email protected])

adbeat_bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

adbeat_bot

AdsBot-Google

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggered

AdsBot-Google (+http://www.google.com/adsbot.html)

AdsBot-Google-Mobile

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

AffsignalCrawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; AffsignalCrawler/1.0; +https://affsignal.com/bot)

AgentDataBot

Generic crawler

Generic crawlerv1.0

AgentDataBot/1.0 (+https://agentdata.run/bot)

AhrefsBot

Ahrefs · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv7.0

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

AhrefsSiteAudit

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv6.1

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

AI2Bot

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

AI2 Bot

AIFengXiangTa-ToolIndexBot

Generic crawler

Generic crawlerv0.1

AIFengXiangTa-ToolIndexBot/0.1 (+personal non-commercial AI tool index; single-threaded, respects robots.txt)

AIFinderBot

Generic crawler

Generic crawlerv0.1

AIFinderBot/0.1 (+https://github.com/kisookang/AIFinder; korean-fit-check)

aiHitBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.9

Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)

AionBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 (compatible; AionBot/1.0)

aitoolnavio-crawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; aitoolnavio-crawler/1.0; +https://aitoolnavio.com)

AkoniLeadHuntBot

Generic crawler

Generic crawlerv0.1

AkoniLeadHuntBot/0.1 (+https://www.recruiting-systems.com)

AlexandriaOrgBot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (Linux) (compatible; AlexandriaOrgBot/1.0; +https://www.alexandria.org/bot.html)

AliyunSecBot

Generic crawler

Generic crawlervAliyun

AliyunSecBot/Aliyun ([email protected])

Amazonbot

Amazon · Search crawler

Search crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UAv0.1

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

AndersPinkBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.1

Mozilla/5.0 (compatible; AndersPinkBot/1.1; +http://anderspink.com/bot.html)

AonsightMetaBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; AonsightMetaBot/1.0; +https://aonsight.com/bot)

ApcosysBot

Generic crawler

Generic crawlerv1.0

ApcosysBot/1.0 (+https://apcosys.net)

Applebot

Apple · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv0.1

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Applebot

Apple · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

Applebot-Extended

AppWatchBot

Generic crawler

Generic crawlerv0.1

AppWatchBot/0.1 (+https://appwatch.ai/bot; freshness check only)

Aranet-SearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36

archive.org_bot

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/7d76126 warc/v0.8.102

Arquivo-web-crawler

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtSystem-triggered

Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling)

AtlasBot

Generic crawler

Generic crawlerv1.0

AtlasBot/1.0 (+https://atlas.harmoniamix.com/bot)

Atom Feed Robot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggered

RSSMicro.com RSS/Atom Feed Robot

AudienceExplorerBot

Generic crawler

Generic crawlerBrowser-like UAv0.2

Mozilla/5.0 (compatible; AudienceExplorerBot/0.2; site classification, one request per site)

AutotouchCompanyCrawler

Generic crawler

Generic crawlerv1.0

AutotouchCompanyCrawler/1.0 (+https://autotouch.ai)

AuvraStarshotBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; AuvraStarshotBot/1.0; +https://auvralabs.com)

AvanooCatalogBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; AvanooCatalogBot/1.0)

AwarioBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)

AwarioSmartBot

Generic crawler

Generic crawlerv1.0

AwarioSmartBot/1.0 (+https://awario.com/bots.html; [email protected])

AzureAI-SearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;

BacklinksExtendedBot

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; BacklinksExtendedBot)

Baiduspider

Baidu · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.0

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Barkrowler bot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv0.9

Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

BarrigazBot

Generic crawler

Generic crawlerv2.0

BarrigazBot/2.0 (+https://barrigaz.com/bot; [email protected])

BazQux bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv2.4

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)

BeansLLM-CorpusBot

Generic crawler

Generic crawlerv1.1

BeansLLM-CorpusBot/1.1 (+https://scrape.beansllm.ai/bot; respects robots.txt)

Bingbot

Microsoft · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

BingPreview

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0b

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

bitlybot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv3.0

bitlybot/3.0 (+http://bit.ly/)

BitSightBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.86 Safari/537.36 BitSightBot/1.0

BixelBot

Generic crawler

Generic crawlerv1.0

BixelBot/1.0 (+https://bixel.com/docs/crawler)

BlueBot-scout-url

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; BlueBot-scout-url/0.1)

Brand360Bot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; Brand360Bot/1.0; +https://be1st.ai)

Bravebot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; Bravebot/1.0; +https://brave.com/search/)

Brightbot

Generic crawler

Generic crawler

Brightbot 1.0

BugBuzzerBot

Generic crawler

Generic crawlerv0.1

BugBuzzerBot/0.1

Bytespider

ByteDance · Search crawler

Search crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; [email protected])

CCBot

Common Crawl · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredv2.0

CCBot/2.0 (https://commoncrawl.org/faq/)

ChatGLM-Spider

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://zhipuai.cn/)

ChatGPT-User

OpenAI · AI assistant

AI assistantURL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Claude-SearchBot

Anthropic · AI search crawler

AI search crawlerSearch crawlerNot AI trainingObeys robots.txtUser-triggeredv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +mailto:[email protected]

Claude-User

Anthropic · AI assistant

AI assistantURL fetcherNot AI trainingObeys robots.txtUser-triggered

Claude-User (claude-code/2.1.321; +https://support.anthropic.com/)

ClaudeBot

Anthropic · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; ClaudeBot/1.0; [email protected])

CloudflareBrowserRenderingCrawler

Generic crawler

Generic crawlerv1.0

CloudflareBrowserRenderingCrawler/1.0

ClueWeb-Crawler

Generic crawler

Generic crawlerv1.0

ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:[email protected])

CodeMMPretrainCrawler

Generic crawler

Generic crawlerv1.0

CodeMMPretrainCrawler/1.0

cohere-training-data-crawlera

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

cohere-training-data-crawler ([email protected])

CountryBot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; CountryBot/0.1; +https://example.org/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36

CreditOneBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; CreditOneBot/1.0; +https://www.credit-one.it)

CricketBot

Generic crawler

Generic crawlerv1.0

CricketBot/1.0 (+https://engine.cricket/bot; [email protected])

CrossoverLeadsBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; CrossoverLeadsBot/1.0; +https://crossoverleads.com/bot)

DataBeezBot

Generic crawler

Generic crawlerv1.0

DataBeezBot/1.0 (+https://www.databeez.it)

DataForSeoBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

Daum bot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv4.1

Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)

dcrawl bot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggeredv1.0

dcrawl/1.0

deadlinkchecker bot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

www.deadlinkchecker.com Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

DeccanCorpusBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; DeccanCorpusBot/1.0; +https://deccan.ai/bot) AppleWebKit/537.36

DeepSeekBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/)

DesearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; DesearchBot/1.0; +https://www.desearch.ai/crawler)

Diffbot

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; Diffbot/1.0; +https://diffbot.com)

DigitaluBot

Generic crawler

Generic crawlerv1.0

DigitaluBot/1.0 (+https://digitalu.live/bot; platform detection; one request per site)

Discordbot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv2.0

Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)

DomainStatsBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.0

DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)

DotBot

Moz · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.2

Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected])

DuckAssistBot

AI assistant

AI assistantNot AI trainingObeys robots.txtUser-triggeredv1.1

DuckAssistBot/1.1 (https://duckduckgo.com/duckassistbot)

DuckDuckBot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.1

DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

eBoolBot

Generic crawler

Generic crawlerv0.1

eBoolBot/0.1 (+https://www.ebool.com/about)

ElevenlabsCrawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ElevenlabsCrawler/1.0; +https://elevenlabs.io/docs/bots)

EmailEnrichmentBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; EmailEnrichmentBot/1.0; +contact-verification)

Embedly

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv0.2

Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)

EnforceGateBot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; EnforceGateBot/0.1; +https://www.enforcegate.com/bot)

EtaoSpider

URL fetcher

URL fetcherRobots control limited

EtaoSpider

ev-crawler

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/web-crawler)

EveMeCrawler

Generic crawler

Generic crawlerv1.0

EveMeCrawler/1.0 (+https://eveme.app)

ExaSearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)

EzoicBot

Ezoic · Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 X-Middleton/1 (compatible; EzoicBot-IntegrationCheck; +https://www.ezoic.com/bot/)

FacebookExternalHit

Meta · URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv1.1

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

FacturDirectoryBot

Generic crawler

Generic crawlerv1.0

FacturDirectoryBot/1.0 (+https://data.facturmfg.com/)

FAST-WebCrawler

Generic crawler

Generic crawlerv3.8

FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)

FeedBot

Generic crawler

Generic crawlerv1.0

FeedBot/1.0

Feedfetcher-Google

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggered

Feedfetcher-Google

Feedly bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.0

Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

FirmableBot

Generic crawler

Generic crawlerv1.0

FirmableBot/1.0 (domain-check)

FlipboardProxy

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv1.2

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)

FokalsBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; FokalsBot/1.0; +https://www.fokals.com/bot)

FreshRSS bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.23.1

FreshRSS/1.23.1 (Linux; https://freshrss.org)

Friendica bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredvx.x.x

Friendica/x.x.x

FudgeResearchBot

Generic crawler

Generic crawlerv1.0

FudgeResearchBot/1.0 (+https://design.withfudge.com; [email protected])

FullStackGTM-bot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; FullStackGTM-bot/1.0; +https://fullstackgtm.com/bot)

FundigBot

Generic crawler

Generic crawlerv1.0

FundigBot/1.0 (+https://console.fundig.app/bot)

fyndbot

Generic crawler

Generic crawler

fyndbot (page-validation; https://fynd.bot)

GeneralCrawlBot

Generic crawler

Generic crawlerv0.1

GeneralCrawlBot/0.1

GeniusDataBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; GeniusDataBot/1.0; +https://geniusdata.com)

GenomeBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; GenomeBot/1.0; +https://51ultron.com)

GlobRadarBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; GlobRadarBot/1.0; +https://globradar.com/bot)

GolgulBot

Generic crawler

Generic crawlerv1.0

GolgulBot/1.0 (+https://golgul.com; news date backfill)

Google Web Preview

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51

Google-InspectionTool

Developer tool

Developer toolURL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.0

Mozilla/5.0 (compatible; Google-InspectionTool/1.0)

Google-Read-Aloud

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggeredBrowser-like UA

Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)

Google-Safety bot

Security scanner

Security scannerNot AI trainingRobots control limitedSystem-triggeredBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Googlebot

Google · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

Uses standard Googlebot user agent strings

Googlebot

Google · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.1

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot

Google · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.0

Googlebot-Image/1.0

Googlebot

Google · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.1

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

Googlebot

Google · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.1

Mozilla/5.0 (compatible; Googlebot-News/2.1; +http://www.google.com/bot.html)

Googlebot

Google · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.0

Googlebot-Video/1.0

GoogleOther

Google · URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; GoogleOther)

GPTBot

OpenAI · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.4

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)

GrokBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/)

HaloBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; HaloBot/1.0)

HaoSouSpider

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

Hatena bot

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtUser-triggered

HatenaBlog-bot/0.02 (+https://help.hatenablog.com/entry/about-hatenablogbot)

HeadlessChrome

Developer tool

Developer toolURL fetcherNot AI trainingRobots control limitedUser-triggeredv125.0.6422.142

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/125.0.6422.142 Safari/537.36

heritrix

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv3.1.1

Mozilla/5.0 (compatible; heritrix/3.1.1 +http://archive.org)

HiWolfsBot

Generic crawler

Generic crawlerv1.0

HiWolfsBot/1.0 (+https://hiwolfs.com; careers-page indexer)

HTTrack bot

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtUser-triggered

Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

HuaweiCrawler

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; WOW64; iPhone) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.4188.69 Safari/537.36 HuaweiCrawler

HunterLogosRobot

Generic crawler

Generic crawlerv1.0

HunterLogosRobot/1.0 (+https://logos.hunter.io/robot)

hypestat bot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/bot)

ia_archiver

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtSystem-triggered

ia_archiver

iaskspider

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; iaskspider/1.0; MSIE 6.0)

IbouBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; IbouBot/1.0; [email protected]; +https://ibou.io/iboubot.html)

ICP-Research-Bot

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36 ICP-Research-Bot

Iframely bot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggeredv1.3.1

Iframely/1.3.1 (+https://iframely.com/docs/about)

ImagesiftBot

URL fetcher

URL fetcherAI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)

ImprintsCollectionBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ImprintsCollectionBot/1.0; +https://www.fim.uni-passau.de/data-science)

IndexaroLeadBot

Generic crawler

Generic crawlerv0.1

IndexaroLeadBot/0.1 (+https://indexaro.com; polite startup discovery)

INETDEX-BOT

Generic crawler

Generic crawlerBrowser-like UAv2.1

Mozilla/5.0 (compatible; INETDEX-BOT/2.1; +https://inetdex.com/bot.php)

inoreader bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.0

Inoreader/1.0 (http://www.inoreader.com)

IntercomDetectBot

Generic crawler

Generic crawlerv1.0

IntercomDetectBot/1.0 (+https://supportman.io/bot)

IonCrawl

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

IonCrawl

IoRobot-Python

Generic crawler

Generic crawlerv1.0

IoRobot-Python/1.0

IsonSearchBot

Generic crawler

Generic crawlerv1.0

IsonSearchBot/1.0 (+https://search.isonai.com/bot)

jobs-crawler

Generic crawler

Generic crawlerv0.2

jobs-crawler/0.2

JobUnlockedBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; JobUnlockedBot/1.0; +https://jobunlocked.com)

Jugendschutzprogramm-Crawler

Data collection crawler

Data collection crawlerNot AI trainingRobots control limitedSystem-triggered

Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de

KakaoBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; KakaoBot/1.0; +https://cs.kakao.com/helps_html/1073213012)

KambriumSourcingBot

Generic crawler

Generic crawlerv0.1

KambriumSourcingBot/0.1 (+https://kambrium.ai; contact: [email protected])

Kangaroo Bot

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

Kangaroo Bot

KeenableBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/bot)

keys-so-bot

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (keys-so-bot)

Kimi-SearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; Kimi-SearchBot/1.0; +https://kimi.ai/)

KimiBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; KimiBot/1.0; +https://kimi.ai/)

Konturbot

Generic crawler

Generic crawlerBrowser-like UAv1.2

Mozilla/5.0 (compatible; Konturbot/1.2; +http://kontur.ru; [email protected])

KosoBot

Generic crawler

Generic crawlerv1.0

KosoBot/1.0 (+https://koso.ai/bot)

KStandBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

lainarchive-crawler

Generic crawler

Generic crawlerv1.0

lainarchive-crawler/1.0 (+https://lainarchive.com; [email protected])

LanternBot

Generic crawler

Generic crawlerv1.0

LanternBot/1.0 (+https://selfhost-search.rork.app/about; independent search index)

LeadFinderBot

Generic crawler

Generic crawlerv1.0

LeadFinderBot/1.0 public business contact discovery

LeadGennyBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LeadGennyBot/1.0)

LeadGenResearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LeadGenResearchBot/1.0; India PG checkout research; polite)

LeadGrowBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LeadGrowBot/1.0)

LeadHQBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LeadHQBot/1.0; +https://lead.verifyhq.io)

LeadOceanBot

Generic crawler

Generic crawlerv1.0

LeadOceanBot/1.0 (+https://leadocean.io/bot)

LightspeedSystemsCrawler

Security scanner

Security scannerNot AI trainingRobots control limitedSystem-triggeredBrowser-like UA

LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)

linkdexbot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv2.1

Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/)

LinkedInBot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)

linkfluence bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])

LinkScoutBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LinkScoutBot/1.0)

LinkupBot

Generic crawler

Generic crawlerv1.0

LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; [email protected])

ListerBot

Generic crawler

Generic crawlerv0.1

ListerBot/0.1 (+https://www.lister.ai/contact; directory research)

LoadedBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LoadedBot/1.0; +https://loaded.ai/bot)

LocalDomainCrawler

Generic crawler

Generic crawlerv5.0

LocalDomainCrawler/5.0

LohiSoftBot

Generic crawler

Generic crawlerv1.0

LohiSoftBot/1.0 (+https://lohisoft.com/bot)

LyonlBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; LyonlBot/1.0; +https://lyonl.com/bot.html)

magpie-crawler

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.1

magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

Mail.RU_Bot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv2.0

Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

MapTheNetBot

Generic crawler

Generic crawlerv1.0

MapTheNetBot/1.0 (+https://www.mapthenet.org/bot)

MarinaCrawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; MarinaCrawler/1.0; +https://paradisemedia.com)

Mastodon bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Mastodon/{version} (+https://{instance-domain}/)

mcbot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; mcbot/0.1)

MeasureboardResearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; MeasureboardResearchBot/1.0; +https://measureboard.com/bot)

Mediapartners-Google

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggered

Mediapartners-Google

Mediatoolkitbot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

Mediatoolkitbot

MeridianBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; MeridianBot/1.0)

Meta External Agent

Meta · AI training crawler

AI training crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta External Agent

Meta · Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredv1.1

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Meta-ExternalFetcher

AI search crawler

AI search crawlerSearch crawlerNot AI trainingRobots control limitedUser-triggeredv1.1

meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

MetaCrawler

Generic crawler

Generic crawlerv1.0

MetaCrawler/1.0 (+https://metacrawler.com/bot)

MetaInspector bot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggered

MetaInspector/[version] (+https://github.com/jaimeiniesta/metainspector)

MeteoriaBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; MeteoriaBot/1.0; +https://meteoria.ai; [email protected])

MillionRoundsBot

Generic crawler

Generic crawlerv0.1

MillionRoundsBot/0.1 (+https://millionrounds.com/methodology)

MinakiBot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0 Safari/537.36 (compatible; MinakiBot/0.1; +mailto:[email protected])

Miniflux bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv2.x.x

Mozilla/5.0 (compatible; Miniflux/2.x.x; +https://miniflux.app)

Miruu-RecruiterBot

Generic crawler

Generic crawlerv1.0

Miruu-RecruiterBot/1.0 (+https://miruu.app)

MistralAI-User

Mistral AI · AI assistant

AI assistantURL fetcherUser-triggeredBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)

MJ12bot

Majestic · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAvv1.4.8

Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)

MojeekBot

Mojeek · Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv0.11

Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html)

MsnBot-Media

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b

MuckRack bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; MuckRack/1.0; +https://muckrack.com)

mypagesbot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; mypagesbot/1.0; +https://mypages.com/bot)

NagetBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; NagetBot/1.0; +https://naget.ai/bot)

NavCrawl

Generic crawler

Generic crawlerv0.4

NavCrawl/0.4 (+https://example.com/bot)

NetcraftSurveyAgent

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; [email protected])

NetpeakCheckerBot

Generic crawler

Generic crawlerBrowser-like UAv3.7

Mozilla/5.0 (compatible; NetpeakCheckerBot/3.7; +https://netpeaksoftware.com/checker)

Netvibes bot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggered

Netvibes (http://www.netvibes.com)

news-please bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtUser-triggered

news-please

NewsBlur bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

NewsBlur Feed Fetcher - {subscriber_count} subscribers - {site_url} (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

NewSiteLeadBot

Generic crawler

Generic crawlerv1.0

NewSiteLeadBot/1.0 (+https://newsitelead.com/bot)

NewsletterFormResearchBot

Generic crawler

Generic crawlerv1.3.1

NewsletterFormResearchBot/1.3.1 (+read-only public-web research; no form submissions)

NewsNow bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggered

NewsNow

newspaper bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv0.2.8

newspaper/0.2.8

Nextcloud bot

Developer tool

Developer toolURL fetcherNot AI trainingObeys robots.txtUser-triggered

Mozilla/5.0 (Nextcloud)

Nicecrawler

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.1

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36

NoryoLogoDiscoveryBot

Generic crawler

Generic crawlerv1.0

NoryoLogoDiscoveryBot/1.0 (+https://noryo.app/bot)

NovaAct bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

NovaAct

Nutch bot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggered

Nutch/NutchOrg/NutchCVS

OAI-AdsBot

OpenAI · Ads safety crawler

Ads safety crawlerNot AI trainingSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

OAI-SearchBot

OpenAI · AI search crawler

AI search crawlerSearch crawlerNot AI trainingObeys robots.txtUser-triggeredv1.4

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Offline Explorer bot

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtUser-triggered

Offline Explorer/2.5

omgili bot

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredv0.5

omgili/0.5 +http://omgili.com

online-webceo-bot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; online-webceo-bot/1.0; http://online.webceo.com)

OpenIBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; OpenIBot/1.0; +https://openi.ai)

OpenindexSpider

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

openmagpie-jobcrawler

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; openmagpie-jobcrawler/0.1)

opr-crawler

Generic crawler

Generic crawlerBrowser-like UAv0.1.0

Mozilla/5.0 (compatible; opr-crawler/0.1.0; +mailto:[email protected])

Orbbot

Generic crawler

Generic crawlerBrowser-like UAv1.1

Mozilla/5.0 (compatible; Orbbot/1.1;)

osentix-crawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; osentix-crawler/1.0; +https://osentix.com/bot)

Other

Generic crawler

Generic crawler

2ip bot/1.1 (+https://2ip.io/bot/)

Other

Generic crawler

Generic crawler

ADOPS.online Domain Intelligence Bot (B2B Validation)

Other

Generic crawler

Generic crawler

AITopToolsListingPreview/1.0 (+https://aitoptools.com)

Other

Generic crawler

Generic crawler

AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)

Other

Generic crawler

Generic crawler

Barrigaz harvester/2.0 (crawler; contact: [email protected])

Other

Generic crawler

Generic crawler

BilateralNavigator-PageFetch/1.0 (+https://bilateralnavigator.com/bot)

Other

Generic crawler

Generic crawler

CMSSurvey/1.0; +https://addedlovely.com/crawler

Other

Generic crawler

Generic crawler

company-db-worker/0.1 (+https://businessmcp.com/bot)

Other

Generic crawler

Generic crawler

CompleteGrowLinkHunter/2.0 respectful broken-link research crawler (+https://completegrow.com.au/)

Other

Generic crawler

Generic crawler

Crawl360/1.0 (+account research; contact: [email protected])

Other

Generic crawler

Generic crawler

CrawlConsole/1.0 (+https://crawlconsole.com)

Other

Generic crawler

Generic crawler

CrawlConsole-MarkdownGenerator/1.0

Other

Generic crawler

Generic crawler

CRM Hunter/1.0 (+https://example.com/bot)

Other

Generic crawler

Generic crawler

ct-sender-poc/1.0 (stage 3 fetch)

Other

Generic crawler

Generic crawler

DataEdge/crawler (+https://github.com/millstonehq/data-edge/issues)

Other

Generic crawler

Generic crawler

EcommerceDetector/1.0 (+https://example.com/bot)

Other

Generic crawler

Generic crawler

evergreen-site-survey/1.0 (+https://evergreendb.ai/crawler; one homepage request per domain)

Other

Generic crawler

Generic crawler

favi.sh/1.0 (+https://favi.sh/legal/abuse; logo fetcher; contact [email protected])

Other

Generic crawler

Generic crawler

file-preview-api-site-info/1.0

Other

Generic crawler

Generic crawler

FirmaAPI/1.0 (+https://firmaapi.com/bot; [email protected]) SiteTara/1.0

Other

Generic crawler

Generic crawler

first-sight-ct-monitor/3 (certificate transparency monitor; reads one public page per new certificate; +https://github.com/HaliSol)

Other

Generic crawler

Generic crawler

FortLineAgentShieldDiscovery/1.0 (+https://fortline.ai/bot; AI app directory research; contact: [email protected])

Other

Generic crawler

Generic crawler

go_infrastructure_fetch_cache/0.2.2 (+https://github.com/baditaflorin/go_infrastructure_fetch_cache)

Other

Generic crawler

Generic crawler

Google-Adwords-Instant (+http://www.google.com/adsbot.html)

Other

Generic crawler

Generic crawler

HeartbeatMonitor/1.0

Other

Generic crawler

Generic crawler

meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Other

Generic crawler

Generic crawler

meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Other

Generic crawler

Generic crawler

monitor-telegram-clone-realtime/1.0

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; HubSpot Crawler; +https://www.hubspot.com) Chrome/131.0.0.0 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; AITopHubLiveCheck/1.0; +https://aitophub.example/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; ant-html/1.0; +https://kuaishou.com/bot)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; CT-WP-Scanner/1.0; +https://example.com/bot)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; dead-link-checker/0.1; +https://git.woa.com/web_search/spider/fetcher/databus/agent-browser)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; deadweb-prober/1.0; +https://github.com/Crawlora-org/dead-web-commoncrawl)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; DmarcerSpotter/1.0; +https://dmarcer.net/bot)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; domains-distillery/1.0; +https://crawler.validus.ai)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; ExaUser/1.0; +https://crawler.exa.ai/exauser)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; fresh-domain-watch/1.0; +mailto:[email protected]; opt-out: email UNSUB CRAWLER yourdomain.com)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; GlassDollarWWWIndexer/1.0; +https://www.angeldata.eu/bot)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; HubSpot Crawler; HubSpot Domain check; +https://www.hubspot.com)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; ibou-search/2.0; +https://ibou.com/crawlers.html)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; MerchantDiscovery/1.0; market research crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Meta-ExternalFetcher/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Meta-WebIndexer/1.0; +https://developers.facebook.com/docs/sharing/webmasters/crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Niche logo fetch)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; urlsuma/2.0; +https://urlsuma.de/bot.html)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; WebCrawler/1.0)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; WebFetchHub/1.0)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html��)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; [email protected])

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Linux; Android 13) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36 Edg/143.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Edg/146.0.0.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 [email protected]

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36 site-checker/0.1 (+https://payme.io/bot)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 (compatible; Epochal-Search/1.0; +https://crawler.epochal.com)

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.32

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/152.0.7977.64 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/931.0.6778.764 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.76 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36

Other

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64; rv:147.0) Gecko/20100101 Firefox/147.0 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Other

Generic crawler

Generic crawlerBrowser-like UA

MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 4.4.2; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; Sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm#07)

Other

Generic crawler

Generic crawler

NetAPI/1.1 (+https://netapi.com/bot.html)

Other

Generic crawler

Generic crawler

[NVIDIA]-[[email protected]]-[NSPECT-I05S-FIEU]-[NVIDIABOT]

Other

Generic crawler

Generic crawler

PageReceipt/1.0 (+local research; fetch-tls-track; curl-plain)

Other

Generic crawler

Generic crawler

Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/)

Other

Generic crawler

Generic crawler

Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

Other

Generic crawler

Generic crawler

PulseGate-Indexer/1.0 (+https://www.pulsegate.ai/bot)

Other

Generic crawler

Generic crawler

RecordedFuture Global Inventory Crawler

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/19.2

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/20.2

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/23.0

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/23.1

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/24.0

Other

Generic crawler

Generic crawler

Screaming Frog SEO Spider/24.3

Other

Generic crawler

Generic crawler

SearchAtlas Bot (https://www.searchatlas.com)

Other

Generic crawler

Generic crawler

Sogou Push Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07��)

Other

Generic crawler

Generic crawler

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Other

Generic crawler

Generic crawler

SpeshyWatcher/1.0 (+https://speshy.com.au/bot; price & stock watch on behalf of a subscriber)

Other

Generic crawler

Generic crawler

SSI-Nutch/1.23 (SSI broad web crawler; https://ssi.inc/; [email protected])

Other

Generic crawler

Generic crawler

WarmBadge-Trawler/1.0 (trust-layer research crawler; https://warmbadge.io/crawler)

Other

Generic crawler

Generic crawler

WhatsApp/2.23.20.0

Other

Generic crawler

Generic crawler

WhatsApp/2.2616.100 W

Other

Generic crawler

Generic crawler

WhatsApp/2.2620.102 W

Other

Generic crawler

Generic crawler

WhatsApp/2

PageSourceSearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PageSourceSearchBot/1.0; +https://pagesourcesearch.com/bot) Chrome/128.0.0.0 Safari/537.36

PaigeHeraldBot

Generic crawler

Generic crawlerv1.0

PaigeHeraldBot/1.0 (B2B directory research; +mailto:[email protected])

Pandalytics bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredv2.0

Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

PanguBot

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggered

PanguBot

panscient.com bot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

panscient.com

PayMeMerchantBot

Generic crawler

Generic crawlerv1.0

PayMeMerchantBot/1.0 (+https://bot.payme.io)

PBN.BOT

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; PBN.BOT Monitoring/1.0)

peer39_crawler

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

peer39_crawler

PegiiBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; PegiiBot/1.0)

Perplexity-User

Perplexity · AI assistant

AI assistantURL fetcherNot AI trainingRobots control limitedUser-triggeredv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

PerplexityBot

Perplexity · Data collection crawler

Data collection crawlerSearch crawlerAI trainingObeys robots.txtSystem-triggeredv1.0

Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

PetalBot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

Pingdom.com bot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggeredBrowser-like UA

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)

Pinterestbot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)

PipericBot

Generic crawler

Generic crawlerv1.0

PipericBot/1.0 (+https://piperic.com/bot)

PocketParser

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggeredv2.0

PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)

PodcastPartnerResearchBot

Generic crawler

Generic crawlerv1.0

PodcastPartnerResearchBot/1.0 (+contact: owner; one-off research crawl)

Pomothy-Bot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; Pomothy-Bot/1.0)

PopulusBot

Generic crawler

Generic crawlerv1.0

PopulusBot/1.0 (business contact pages; contact: [email protected])

PREMSGOBOT

Generic crawler

Generic crawlerBrowser-like UAv4.0

Mozilla/5.0 (compatible; PREMSGOBOT/4.0; +https://premsgo.fr/bot/; crawl global juillet/aout 2026)

ProductLookoutBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ProductLookoutBot/1.0)

ProsAccessBot

Generic crawler

Generic crawlerv1.0

ProsAccessBot/1.0 (+https://prosaccess.com/privacy/professionals)

proximic

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

PublicEmailResearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; PublicEmailResearchBot/1.0; +https://example.com/bot-info)

PublicWWWBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; PublicWWWBot/1.0; +https://publicwww.com/bot.html)

PurchaseDetectorBot

Generic crawler

Generic crawlerv1.0

PurchaseDetectorBot/1.0 (+https://example.com/bot; [email protected])

QlyzeBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot)

Querit-SearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; Querit-SearchBot/1.0; +https://www.querit.ai)

QueritBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; QueritBot/1.0; +https://www.querit.ai/bot)

Quora-Bot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.0

Quora-Bot/1.0

Qwairy-Bot

Generic crawler

Generic crawlerv1.0

Qwairy-Bot/1.0 (+https://qwairy.co)

Qwantbot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; Qwantbot/1.0; +https://help.qwant.com/bot/)

RankingsAtlasBot

Generic crawler

Generic crawlerv0.1

RankingsAtlasBot/0.1 (research crawler; one homepage per site; honours robots.txt)

RateOnTime-Bot

Generic crawler

Generic crawlerv1.0

RateOnTime-Bot/1.0 (+https://rateontime.com)

ReconBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ReconBot/1.0; +https://scanner.insecurest.xyz/bot; [email protected])

redditbot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

redditbot

rednote-websearch-bot

Generic crawler

Generic crawler

rednote-websearch-bot

RefgrowResearchBot

Generic crawler

Generic crawlerv1.0

RefgrowResearchBot/1.0 (+https://refgrow.com; contact [email protected])

Reflectionbot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Reflectionbot/1.0; +https://reflection.ai/bot) Chrome/151.0.0.0 Safari/537.36

RepetivBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; RepetivBot/1.0; +mailto:[email protected])

RevontPipelineBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; RevontPipelineBot/1.0; +https://revont.io)

RmagedonBot

Generic crawler

Generic crawlerv1.0

RmagedonBot/1.0 (+https://rmagedon.com/bot; [email protected])

robots.txt

Generic crawler

Generic crawler

ccrawl/1.0 (company cache; respects robots.txt; contact: [email protected])

rogerbot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.2

rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, [email protected])

rolltheweb-crawler

Generic crawler

Generic crawlerv2.2

rolltheweb-crawler/2.2 (+https://rolltheweb.com/; homepage-only; obeys robots.txt)

S33DBot

Generic crawler

Generic crawlerv4.0

S33DBot/4.0 (+https://s33d.fr/bot)

SaaSBrowserBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; SaaSBrowserBot/1.0; +https://saasbrowser.com/bot)

SaaSRankingBot

Generic crawler

Generic crawlerv1.0

SaaSRankingBot/1.0 (+https://saas-ranking.com/bot)

SaluteBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; SaluteBot/1.0)

ScalinthBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ScalinthBot/1.0; +https://scalinth.com/bot)

scoop.it bot

Archive crawler

Archive crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

Mozilla/5.0 (compatible; scoopit-crawler/3; +https://www.scoop.it/bot.html)

ScoutnerBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ScoutnerBot/1.0; +https://scoutner.com/bot)

Scrapy bot

Developer tool

Developer toolURL fetcherNot AI trainingObeys robots.txtUser-triggeredv2.8.0

Scrapy/2.8.0 (+https://scrapy.org)

Screaming Frog SEO Spider

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggered

Screaming Frog SEO Spider/[version number]

ScroResearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ScroResearchBot/1.0; +https://scro.io; research draft)

SearchataCrawler

Generic crawler

Generic crawlerv1.0

SearchataCrawler/1.0 (+https://searchata.com/benchmarks/methodology)

SearchBot

Generic crawler

Generic crawlerv1.0

SearchBot/1.0 (compatible; +https://search.theradicalparty.com/bot)

SEBot-WA

Generic crawler

Generic crawler

SEBot-WA

SeekportBot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)

SemanticScholarBot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)

SemanticVisionsBot

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (Compatible; SemanticVisionsBot/1.0; +http://semantic-visions.com)

SemrushBot

Semrush · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

SemrushBot

Semrush · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)

SemrushBot

Semrush · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggered

SemrushBot-CT

SemrushBot

Semrush · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv0.97

Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

SemrushBot

Semrush · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggered

SemrushBot-SWA

SenutoBot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredv1.0

SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)

SeobilityBot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggered

SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)

SEOJuice-SearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/bot

SEOkicks-Robot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robot.html)

SEOlizer bot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.2

SEOlizer/1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)

SERankingBacklinksBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)

SerendeputyBot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv0.8.6

SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)

SerpCrawler

Generic crawler

Generic crawlerv0.1

SerpCrawler/0.1 (+local research crawler)

Serpstatbot

Serpstat · SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtSystem-triggeredv2.1

serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; [email protected])

SeznamBot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv4.0

Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)

ShapBot

Parallel · AI search crawler

AI search crawlerSearch crawlerNot AI trainingObeys robots.txtSystem-triggeredv0.1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0

ShopContactCrawler

Generic crawler

Generic crawlerv0.1

ShopContactCrawler/0.1

SimplePie bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

SimplePie/[VERSION] (Feed Parser; http://simplepie.org; Allow like Gecko) Build/[BUILD_NUMBER]

SiteAnalyzerbot

Generic crawler

Generic crawlerv4.0

SiteAnalyzerbot/4.0; +https://site-analyzer.pro/

SiteAuditBot

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggeredBrowser-like UAv0.97

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)

SiteBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; SiteBot/1.0)

SiteCheckerBotCrawler

SEO audit crawler

SEO audit crawlerNot AI trainingObeys robots.txtUser-triggered

SiteCheckerBotCrawler

SitemapCrawler

Generic crawler

Generic crawlerv1.0

SitemapCrawler/1.0 (semantic-visions.com crawler)

SkyclazBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; SkyclazBot/1.0; +https://skyclaz.com/bot)

Slack-ImgProxy

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggered

Slack-ImgProxy 0.19 (+https://api.slack.com/robots)

Slackbot

URL fetcher

URL fetcherNot AI trainingRobots control limitedUser-triggered

Slackbot 1.0 (+https://api.slack.com/robots)

Slackbot-LinkExpanding

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

SleepBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36

SMTBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

Snap URL Preview

Snap · URL fetcher

URL fetcherUser-triggered

Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots

Snap URL Preview

Snap · URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots

SofyaBot

Generic crawler

Generic crawlerv1.0

SofyaBot/1.0 (+https://sofya.co/bot)

Sogou web spider

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggered

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

SparkleBot

Generic crawler

Generic crawlerv1.0

SparkleBot/1.0 (+https://sparkle.io/bot)

SpiderLing

SpiderLing · Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (compatible; SpiderLing; +https://www.sketchengine.eu/crawler/)

SQLBot

Generic crawler

Generic crawlerBrowser-like UAv2.0

Mozilla/5.0 (compatible; SQLBot/2.0;)

SQLSpider

Generic crawler

Generic crawlerBrowser-like UAv1.7.0

Mozilla/5.0 (compatible; SQLSpider/1.7.0)

srchs-research-bot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; srchs-research-bot/0.1; public small-business domain research; contact: [email protected])

StartmeBot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredv1.0

StartmeBot/1.0 (+https://start.me/bot)

Storebot-Google

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/321.0.0.0 Safari/533.36

StormIntelCrawler

Generic crawler

Generic crawlerv1.0

StormIntelCrawler/1.0 (StormIntel Crawler; http://stormintel.com; [email protected])

Stripebot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/156.0.0.0 Safari/537.36 (Stripebot/1.0; +https://docs.stripe.com/stripebot-crawler)

StyneBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; StyneBot/1.0; +https://styne.ai/bot)

SumbleBot

Generic crawler

Generic crawlerv1.0

python-httpx/0.28.1 (SumbleBot/1.0)

Superfeedr bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Superfeedr Bot (feed-id=12345)

SurdotlyBot

Security scanner

Security scannerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)

Synapse bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Mozilla/4.0 (compatible; Synapse)

T2Bot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; T2Bot/1.0; +https://t2ind.com/bot)

t3versionsBot

Developer tool

Developer toolNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.2

Mozilla/5.0 (compatible; t3versionsBot/1.2; +https://www.t3versions.com/bot)

TalarionSearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TalarionSearchBot/1.0; research; contact: [email protected])

TechLeadsBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TechLeadsBot/1.0; +https://techleads.fyi/bot)

TechScanBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TechScanBot/1.0)

TelegramBot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

TelegramBot

TenMillionDomainsBot

Generic crawler

Generic crawlerBrowser-like UAv2.0

Mozilla/5.0 (compatible; TenMillionDomainsBot/2.0; +https://github.com/Crawlora-org/ten-million-domains)

teoma bot

Search crawler

Search crawlerNot AI trainingObeys robots.txtSystem-triggered

Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

TernCrawler

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TernCrawler/1.0)

TestSearchSpider

Generic crawler

Generic crawlerv0.1

TestSearchSpider/0.1

TflowSitemapBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TflowSitemapBot/1.0)

TheMapBot

Generic crawler

Generic crawlerv0.1

TheMapBot/0.1 (SaaS product enumeration research; +mailto:[email protected])

TikTokSpider

ByteDance · Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; [email protected])

Timpibot

Data collection crawler

Data collection crawlerAI trainingRobots control limitedSystem-triggeredv1.0

Timpibot/1.0 (+http://timpi.io/crawler)

TinEye-bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.31

Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)

Tiny Tiny RSS bot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

Tiny Tiny RSS/[version] (https://tt-rss.org/)

TLDWPBot

Generic crawler

Generic crawlerv1.0

TLDWPBot/1.0 (+https://tldwp.com/about.php)

TokenOpsBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TokenOpsBot/1.0; +https://tokenops.io)

ToolissBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ToolissBot/1.0)

tranco-crawler

Generic crawler

Generic crawlerv1.0

tranco-crawler/1.0

TrendBowlBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; TrendBowlBot/1.0; [email protected])

trendictionbot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0

trendictionbot0.5.0

Generic crawler

Generic crawlerBrowser-like UA

Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0

TrendsmapResolver

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggered

Trendsmap Resolver/0.0.1

TriumviryResearchBot

Generic crawler

Generic crawlerBrowser-like UAv0.1

Mozilla/5.0 (compatible; TriumviryResearchBot/0.1; +mailto:[email protected])

TurnitinBot

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredv2.1

TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)

TweetmemeBot

URL fetcher

URL fetcherNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv3.0

Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

Twingly Recon

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UA

Mozilla/5.0 (compatible; Twingly Recon; twingly.com), Twingly Recon-Klondike/1.0 (+https://developer.twingly.com)

Twitterbot

X · URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggered

TelegramBot (like TwitterBot)

um-LN

Data collection crawler

Data collection crawlerNot AI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; um-LN/1.0; mailto: [email protected]; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1

ValtrBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; ValtrBot/1.0; +https://valtr.xyz/bot)

VantageBot

Generic crawler

Generic crawlerv1.0

VantageBot/1.0 (+a16z company identity verification; contact [email protected])

VelenPublicWebCrawler

Data collection crawler

Data collection crawlerAI trainingObeys robots.txtSystem-triggeredBrowser-like UAv1.0

Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)

VerticalSaaSResearchBot

Generic crawler

Generic crawlerBrowser-like UAv1.0

Mozilla/5.0 (compatible; VerticalSaaSResearchBot/1.0; homepage metadata research)

Viber bot

AI assistant

AI assistantURL fetcherNot AI trainingObeys robots.txtUser-triggered

Viber

viewsrc-bot

Generic crawler

Generic crawlerv0.1

viewsrc-bot/0.1 (+https://viewsrc.io/bot)

virustotal bot

Security scanner

Security scannerNot AI trainingObeys robots.txtUser-triggered

VirusTotal

vkShare

URL fetcher

URL fetcherNot AI trainingObeys robots.txtUser-triggeredBrowser-like UA

Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)

VStatMetadataBot

Generic crawler

Generic crawlerv1.0

VStatMetadataBot/1.0 (+https://vstat.info/)

W3C_CSS_Validator

Developer tool

Developer toolURL fetcherNot AI trainingObeys robots.txtUser-triggeredv2.0

Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See http://validator.w3.org/services)

W3C_Validator

Developer tool

Developer toolURL fetcherNot AI trainingObeys robots.txtUser-triggered

W3C_Validator/<version> http://validator.w3.org/services

WanscannerBot

Generic crawler

Generic crawlerBrowser-like UAv1.2

Mozilla/5.0 (X11; Linux x86_64; WanscannerBot/1.2; +https://abuse.pend.re) Gecko/20100101 Firefox/10.0

WappalyzerBot

Generic crawler

Generic crawlerBrowser-like UAHeadless browserv1.0

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/154.0.0.0 Safari/537.36 WappalyzerBot/1.0 (+https://www.wappalyzer.com/)