Web crawler lookup

2ip bot/1.1 (+http://2ip.io)

User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0; 360Spider

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ABEvalBot/0.1) Version/11.1.2 Safari/605.1.15

AcademicBotRTU (https://academicbot.rtu.lv; mailto:caps@rtu.lv)

adbeat_bot

AdsBot-Google (+http://www.google.com/adsbot.html)

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.137 Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)

Mozilla/5.0 (compatible; AffsignalCrawler/1.0; +https://affsignal.com/bot)

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

AI2 Bot

Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)

Mozilla/5.0 (compatible; aitoolnavio-crawler/1.0; +https://aitoolnavio.com)

AkoniLeadHuntBot/0.1 (+https://www.recruiting-systems.com)

Mozilla/5.0 (Linux) (compatible; AlexandriaOrgBot/1.0; +https://www.alexandria.org/bot.html)

AliyunSecBot/Aliyun (AliyunSecBot@service.alibaba.com)

Amazonbot

Amazon · Search crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

Mozilla/5.0 (compatible; AndersPinkBot/1.1; +http://anderspink.com/bot.html)

Mozilla/5.0 (compatible; AonsightMetaBot/1.0; +https://aonsight.com/bot)

Applebot

Apple · Search crawler

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Applebot

Apple · Data collection crawler

Applebot-Extended

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36

Arquivo-web-crawler (compatible; heritrix/3.4.0-20200304 +https://arquivo.pt/faq-crawling)

RSSMicro.com RSS/Atom Feed Robot

AutotouchCompanyCrawler/1.0 (+https://autotouch.ai)

Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)

AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;

Mozilla/5.0 (compatible; BacklinksExtendedBot)

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 2 subscribers)

Bingbot

Microsoft · Search crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

BingPreview

URL fetcher

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

bitlybot/3.0 (+http://bit.ly/)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.86 Safari/537.36 BitSightBot/1.0

BixelBot/1.0 (+https://bixel.com/docs/crawler)

Mozilla/5.0 (compatible; BlueBot-scout-url/0.1)

Mozilla/5.0 (compatible; Brand360Bot/1.0; +https://be1st.ai)

Brightbot 1.0

BugBuzzerBot/0.1

Bytespider

ByteDance · Search crawler

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; https://bytedance.sg.larkoffice.com/docx/K5bxdypulop3IIxrJb0lOjLVgFe)

CCBot

Common Crawl · Data collection crawler

CCBot/2.0 (https://commoncrawl.org/faq/)

ChatGPT-User

OpenAI · AI assistant

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Claude-SearchBot

Anthropic · AI search crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-SearchBot/1.0; +mailto:support@anthropic.com

Claude-User

Anthropic · AI assistant

Claude-User (claude-code/2.1.205; +https://support.anthropic.com/)

ClaudeBot

Anthropic · Data collection crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:cmu-clueweb-crawler@andrew.cmu.edu)

cohere-training-data-crawler (+crawler@cohere.ai)

Mozilla/5.0 (compatible; CountryBot/0.1; +https://example.org/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36

Mozilla/5.0 (compatible; CrossoverLeadsBot/1.0; +https://crossoverleads.com/bot)

Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)

Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)

dcrawl/1.0

www.deadlinkchecker.com Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36

Mozilla/5.0 (compatible; Diffbot/1.0; +https://diffbot.com)

Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)

DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)

Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)

DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)

Mozilla/5.0 (compatible; EnforceGateBot/0.1; +https://www.enforcegate.com/bot)

EtaoSpider

Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36 X-Middleton/1 (compatible; EzoicBot-IntegrationCheck; +https://www.ezoic.com/bot/)

FacebookExternalHit

Meta · URL fetcher

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0

FeedBot/1.0

Feedfetcher-Google

Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)

FreshRSS/1.23.1 (Linux; https://freshrss.org)

Friendica/x.x.x

fyndbot (crawler; https://fynd.bot)

Mozilla/5.0 (compatible; GenomeBot/1.0; +https://51ultron.com)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/534.51 (KHTML, like Gecko; Google Web Preview) Chrome/12.0.742 Safari/534.51

Mozilla/5.0 (compatible; Google-InspectionTool/1.0)

Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Googlebot

Google · Data collection crawler

Uses standard Googlebot user agent strings

Googlebot

Google · Search crawler

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot

Google · Search crawler

Googlebot-Image/1.0

Googlebot

Google · Search crawler

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +https://www.google.com/bot.html)

Googlebot

Google · Search crawler

Mozilla/5.0 (compatible; Googlebot-News/2.1; +http://www.google.com/bot.html)

Googlebot

Google · Search crawler

Googlebot-Video/1.0

GoogleOther

Google · URL fetcher

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.137 Mobile Safari/537.36 (compatible; GoogleOther)

GPTBot

OpenAI · Data collection crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot)

Mozilla/5.0 (compatible; HaloBot/1.0)

Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)

HatenaBlog-bot/0.02 (+https://help.hatenablog.com/entry/about-hatenablogbot)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/125.0.6422.142 Safari/537.36

Mozilla/5.0 (compatible; heritrix/3.1.1 +http://archive.org)

Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5; iPhone) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4033.37 Safari/537.36 HuaweiCrawler

Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/bot)

ia_archiver

Mozilla/5.0 (compatible; IbouBot/1.0; +bot@ibou.io; +https://ibou.io/iboubot.html)

Iframely/1.3.1 (+https://iframely.com/docs/about)

Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)

Mozilla/5.0 (compatible; INETDEX-BOT/2.1; +https://inetdex.com/bot.php)

Inoreader/1.0 (http://www.inoreader.com)

IntercomDetectBot/1.0 (+https://supportman.io/bot)

IonCrawl

IoRobot-Python/1.0

Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de

KambriumSourcingBot/0.1 (+https://kambrium.ai; contact: hello@kambrium.ai)

Kangaroo Bot

Mozilla/5.0 (keys-so-bot)

Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

Mozilla/5.0 (compatible; LeadGrowBot/1.0)

LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)

Mozilla/5.0 (compatible; linkdexbot/2.1; +https://www.linkdex.com/en-us/about/bots/)

LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)

Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)

Mozilla/5.0 (compatible; LinkScoutBot/1.0)

LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so)

LocalDomainCrawler/5.0

Mozilla/5.0 (compatible; LyonlBot/1.0; +https://lyonl.com/bot.html)

magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)

Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)

Mozilla/5.0 (compatible; MarinaCrawler/1.0; +https://paradisemedia.com)

Mastodon/{version} (+https://{instance-domain}/)

Mozilla/5.0 (compatible; MeasureboardResearchBot/1.0; +https://measureboard.com/bot)

Mediapartners-Google

Mediatoolkitbot

Mozilla/5.0 (compatible; MeridianBot/1.0)

Meta External Agent

Meta · AI training crawler

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta External Agent

Meta · Data collection crawler

meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

MetaCrawler/1.0 (+https://metacrawler.com/bot)

MetaInspector/[version] (+https://github.com/jaimeiniesta/metainspector)

Mozilla/5.0 (compatible; Miniflux/2.x.x; +https://miniflux.app)

Miruu-RecruiterBot/1.0 (+https://miruu.app)

Mozilla/5.0 (compatible; MJ12bot/v2.0.5; http://mj12bot.com/)

Mozilla/5.0 (compatible; MojeekBot/0.11; +https://www.mojeek.com/bot.html)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) MsnBot-Media /1.0b

Mozilla/5.0 (compatible; MuckRack/1.0; +https://muckrack.com)

Mozilla/5.0 (compatible; mypagesbot/1.0; +https://mypages.com/bot)

NavCrawl/0.4 (+https://example.com/bot)

Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)

Netvibes (http://www.netvibes.com)

news-please

NewsBlur Feed Fetcher - {subscriber_count} subscribers - {site_url} (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)

NewsNow

newspaper/0.2.8

Mozilla/5.0 (Nextcloud)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36

NovaAct

Nutch/NutchOrg/NutchCVS

OAI-AdsBot

OpenAI · Ads safety crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

OAI-SearchBot

OpenAI · AI search crawler

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

Offline Explorer/2.5

omgili/0.5 +http://omgili.com

Mozilla/5.0 (compatible; online-webceo-bot/1.0; http://online.webceo.com)

Mozilla/5.0 (compatible; OpenindexSpider; +http://www.openindex.io/en/webmasters/spider.html)

Mozilla/5.0 (compatible; openmagpie-jobcrawler/0.1)

Mozilla/5.0 (compatible; osentix-crawler/1.0; +https://osentix.com/bot)

2ip bot/1.1 (+https://2ip.io/bot/)

ADOPS.online Domain Intelligence Bot (B2B Validation)

AIWebIndex/2.0 (+https://lyrenth.com/bot; AI-readable web index)

BilateralNavigator-PageFetch/1.0 (+https://bilateralnavigator.com/bot)

CMSSurvey/1.0; +https://addedlovely.com/crawler

CompleteGrowLinkHunter/2.0 respectful broken-link research crawler (+https://completegrow.com.au/)

CrawlConsole/1.0 (+https://crawlconsole.com)

CrawlConsole-MarkdownGenerator/1.0

go_infrastructure_fetch_cache/0.2.2 (+https://github.com/baditaflorin/go_infrastructure_fetch_cache)

HeartbeatMonitor/1.0

meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

monitor-telegram-clone-realtime/1.0

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; HubSpot Crawler; +https://www.hubspot.com) Chrome/131.0.0.0 Safari/537.36

Mozilla/5.0 (compatible; ant-html/1.0; +https://kuaishou.com/bot)

Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)

Mozilla/5.0 (compatible; crawler)

Mozilla/5.0 (compatible; CT-WP-Scanner/1.0; +https://example.com/bot)

Mozilla/5.0 (compatible; dead-link-checker/0.1; +https://git.woa.com/web_search/spider/fetcher/databus/agent-browser)

Mozilla/5.0 (compatible; deadweb-prober/1.0; +https://github.com/Crawlora-org/dead-web-commoncrawl)

Mozilla/5.0 (compatible; HubSpot Crawler; HubSpot Domain check; +https://www.hubspot.com)

Mozilla/5.0 (compatible; MerchantDiscovery/1.0; market research crawler)

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

Mozilla/5.0 (compatible; urlsuma/2.0; +https://urlsuma.de/bot.html)

Mozilla/5.0 (compatible; WebCrawler/1.0)

Mozilla/5.0 (compatible; WebFetchHub/1.0)

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html��)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)

Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots)

Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b

Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 skype-url-preview@microsoft.com

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.76 (KHTML, like Gecko; Google Web Preview) Chrome/131.0.6778.264 Safari/537.36

MQQBrowser/26 Mozilla/5.0 (Linux; U; Android 4.4.2; zh-cn; MB200 Build/GRJ22; CyanogenMod-7) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1 (compatible; Sogou web spider/4.0; +http://www.sogou.com/docs/help/webmasters.htm#07)

Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/)

Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

Screaming Frog SEO Spider/19.2

Screaming Frog SEO Spider/23.1

Screaming Frog SEO Spider/24.0

Screaming Frog SEO Spider/24.3

SearchAtlas Bot (https://www.searchatlas.com)

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

WarmBadge-Trawler/1.0 (trust-layer research crawler; https://warmbadge.io/crawler)

WhatsApp/2.23.20.0

WhatsApp/2.2616.100 W

WhatsApp/2.2620.102 W

WhatsApp/2

Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

PanguBot

panscient.com

Mozilla/5.0 (compatible; PBN.BOT Monitoring/1.0)

peer39_crawler

Mozilla/5.0 (compatible; PegiiBot/1.0)

Perplexity-User

Perplexity · AI assistant

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

PerplexityBot

Perplexity · Data collection crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)

Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)

PipericBot/1.0 (+https://piperic.com/bot)

PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)

Mozilla/5.0 (compatible; ProductLookoutBot/1.0)

Mozilla/5.0 (compatible; proximic; +https://www.comscore.com/Web-Crawler)

Mozilla/5.0 (compatible; PublicEmailResearchBot/1.0; +https://example.com/bot-info)

Mozilla/5.0 (compatible; Querit-SearchBot/1.0; +https://www.querit.ai)

Mozilla/5.0 (compatible; QueritBot/1.0; +https://www.querit.ai/bot)

Quora-Bot/1.0

Qwairy-Bot/1.0 (+https://qwairy.co)

Mozilla/5.0 (compatible; Qwantbot/1.0_2600400; +https://help.qwant.com/bot/)

redditbot

rednote-websearch-bot

Mozilla/5.0 (compatible; Reflectionbot/1.0; +https://reflection.ai/bot)

rogerbot/1.2 (https://moz.com/help/guides/moz-procedures/what-is-rogerbot, rogerbot-crawler+aardwolf-crawler-XX@moz.com)

S33DBot/4.0 (+https://s33d.fr/bot)

Mozilla/5.0 (compatible; SaaSBrowserBot/1.0; +https://saasbrowser.com/bot)

Mozilla/5.0 (compatible; ScalinthBot/1.0; +https://scalinth.com/bot)

Mozilla/5.0 (compatible; scoopit-crawler/3; +https://www.scoop.it/bot.html)

Scrapy/2.8.0 (+https://scrapy.org)

Screaming Frog SEO Spider/[version number]

SearchBot/1.0 (compatible; +https://search.theradicalparty.com/bot)

Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)

Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (Compatible; SemanticVisionsBot/1.0; +http://semantic-visions.com)

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)

SemrushBot-CT

Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)

SemrushBot-SWA

SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)

SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/bot

Mozilla/5.0 (compatible; SEOkicks-Robot; +https://www.seokicks.de/robot.html)

SEOlizer/1.2 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)

Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)

SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)

SerpCrawler/0.1 (+local research crawler)

serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)

Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0

SimplePie/[VERSION] (Feed Parser; http://simplepie.org; Allow like Gecko) Build/[BUILD_NUMBER]

Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; SiteBot/1.0)

SiteCheckerBotCrawler

SitemapCrawler/1.0 (semantic-visions.com crawler)

Slack-ImgProxy 0.19 (+https://api.slack.com/robots)

Slackbot 1.0 (+https://api.slack.com/robots)

Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36

Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)

Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots

Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots

Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

Mozilla/5.0 (compatible; SpiderLing; +https://nlp.fi.muni.cz/projects/biwec/)

Mozilla/5.0 (compatible; SQLBot/2.0;)

StartmeBot/1.0 (+https://start.me/bot)

Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/587.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36

StormIntelCrawler/1.0 (StormIntel Crawler; http://stormintel.com; info@stormintel.com)

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (Stripebot/1.0; +https://docs.stripe.com/stripebot-crawler)

python-httpx/0.28.1 (SumbleBot/1.0)

Superfeedr Bot (feed-id=12345)

Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)

Mozilla/4.0 (compatible; Synapse)

Mozilla/5.0 (compatible; t3versionsBot/1.0; +https://www.t3versions.com/bot)

Mozilla/5.0 (compatible; TechScanBot/1.0)

TelegramBot

Mozilla/5.0 (compatible; TenMillionDomainsBot/2.0; +https://github.com/Crawlora-org/ten-million-domains)

Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

TikTokSpider

ByteDance · Generic crawler

Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)

Timpibot/1.0 (+http://timpi.io/crawler)

Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)

Tiny Tiny RSS/[version] (https://tt-rss.org/)

Mozilla/5.0 (compatible; TokenOpsBot/1.0; +https://tokenops.io)

Mozilla/5.0 (compatible; ToolissBot/1.0)

tranco-crawler/1.0

Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0

Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0

Trendsmap Resolver/0.0.1

TurnitinBot/2.1 (http://www.turnitin.com/robot/crawlerinfo.html)

Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)

Mozilla/5.0 (compatible; Twingly Recon; twingly.com), Twingly Recon-Klondike/1.0 (+https://developer.twingly.com)

TelegramBot (like TwitterBot)

Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com; Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1

Mozilla/5.0 (compatible; ValtrBot/1.0; +https://valtr.xyz/bot)

Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)

Mozilla/5.0 (compatible; VerticalSaaSResearchBot/1.0; homepage metadata research)

Viber

VirusTotal

Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)

Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0 (See http://validator.w3.org/services)

W3C_Validator/<version> http://validator.w3.org/services

WarmerlyBot/1.0 (+https://warmerly.com/bot)

Mozilla/5.0 (compatible; WebatlaBot/1.0; +https://bot.webatla.com; abuse@webatla.com)

Mozilla/5.0 (compatible; webinfo-bot/1.0; +https://realwebdata.com/bot)

Mozilla/5.0 (compatible; WebTrackrCrawler/1.0; +https://affsignal.com/bot)

Webzio

Mozilla/5.0 (compatible; WellKnownBot/0.1; +https://well-known.dev/about/#bot)

WhatsApp/[version number]

WhiteCircleTrustBot/1.0 (+https://whitecircle.ai/bots/trust-center-scraper)

Mozilla/5.0 (compatible; wpbot/1.4; +https://forms.gle/ajBaxygz9jSR8p8G9)

Mozilla/5.0 (compatible; WSIP-Bot/1.0; +https://example.com/bot)

yacybot (/global; amd64 Linux 6.1.0-18-amd64; java 17.0.10; Europe/en) http://yacy.net/bot.html

Mozilla/5.0 (compatible; YaDirectFetcher/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0

Mozilla/5.0 (compatible; YandexAdditional/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexAdditionalBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexAdNet/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexComBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexDirect/2.0; Dyatel)

Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexForDomain/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)

YandexMarket/1.0

Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexMetrika/4.0; +http://yandex.com/bots)

Mozilla/5.0 (iPhone; CPU iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.2 Mobile/15E148 Safari/604.1 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexMobileScreenShotBot/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexOntoDB/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexOntoDBAPI/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexRCA/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexSearchShop/3.0; +https://yandex.com/support/webmaster/robot-workings/check-yandex-robots.html)

YandexSitelinksBot

Mozilla/5.0 (compatible; YandexSpravBot/1.0; +http://yandex.com/bots)

YandexTracker-API/<version>

Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVerticals/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots)

Mozilla/5.0 (compatible; YandexWebmaster/3.0; +http://yandex.com/bots)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Yeti/1.1; +https://naver.me/spd) Chrome/106.0.5249.0 Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.2 Mobile/15E148 YisouSpider/5.0 Safari/604.1

YouBot

You.com · AI search crawler

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/142.0.0.0 Safari/537.36

ZoominfoBot (zoominfobot at zoominfo dot com)

Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)