Notes on building agentic experiences, from visibility, crawlability, and usability. Start building for the next trillion users of the internet.
Learn how to verify AI bot traffic before you allow, block, or throttle crawlers. Check user agents, status codes, behavior, and crawler profiles with a practical workflow.
Learn the difference between Claude-User and ClaudeBot, what each request can prove, and how to monitor delegated AI agent access without blocking useful traffic.
Google-Extended has no separate HTTP user agent. Learn what appears in server logs, what crawler analytics can prove, and how to avoid false attribution.
Learn how to check whether PerplexityBot crawled your website, what status code it received, and how crawler access connects to Perplexity AI search visibility.
Learn why Claude may not be able to access your website, how ClaudeBot and Claude-User differ, and what to check in robots.txt, CDN, WAF, redirects, and crawler logs.
Learn how to tell whether ChatGPT is using your website, from weak signals like referrals and citations to stronger evidence from OpenAI crawler activity.
Should you block AI crawlers in robots.txt? Learn how to decide which bots to allow, which to restrict, and how to avoid losing AI search visibility.
Google Analytics is built around human traffic, not AI crawler visibility. Learn which AI bots may visit your website, why GA can miss them, and how to monitor them with CrawlConsole.
ChatGPT ads introduce a new crawler visibility problem for advertisers. Learn what OAI-AdsBot is, why OpenAI uses it, and how to check whether it can crawl your landing pages.
