AI crawler analytics

AI crawler analytics shows which verified crawlers request your site, which pages they choose, and where those requests fail. Use it to prioritize content that agents and search systems already seek.

Core metrics

The performance view includes:

• Requests: verified crawler requests during the selected period.
• Unique pages: distinct normalized paths requested by crawlers.
• Top pages: destination paths ranked by request count.
• Top crawlers: verified crawler names and operators.
• Status codes: captured response outcomes when available.
• Recent requests: a time-ordered sample for investigation.

Read page demand

High crawler demand is a reason to keep a page accurate, crawlable, and available as structured content. Compare top pages with product priorities and documentation coverage.

For example, CrawlConsole's own property currently shows strong crawler interest in web crawler identities, WebMCP, MCP discovery, backlink analysis, and agent-readable Markdown. That evidence informed the first sections of this documentation.

Investigate HTTP errors

The HTTP errors view focuses on verified crawler requests that received 4xx or 5xx responses. Review the requested path, crawler, last-seen time, and frequency before deciding whether to restore content or add a redirect.

Do not redirect every crawler-requested path. Confirm whether the route was moved, misspelled, intentionally private, or generated by an agent before creating a permanent redirect.

Review repeated missing links

The hallucinated links view identifies repeated crawler-requested 404 paths. These are candidates, not automatic proof that an AI system invented the URL.

Use this workflow:

1. Check internal links, sitemaps, feeds, and historical URLs.
2. Search external sources for references to the missing path.
3. Restore the page when the intent is valid and useful.
4. Add a narrow redirect when a clear replacement exists.
5. Leave the path missing when it has no valid intent.

Query analytics with an agent

Use list_properties first, then pass the canonical property id to get_crawler_analytics, get_http_errors, or get_hallucinated_links.

List my CrawlConsole properties. For crawlconsole.com, summarize crawler page demand over the last 30 days and identify any repeated missing paths that need review.