# CrawlConsole documentation

> Complete published documentation for crawler analytics, MCP agents, WebMCP, backlinks, agentic commerce, and CrawlConsole APIs.

---
title: "CrawlConsole documentation"
description: "Install CrawlConsole, measure AI crawler activity, connect agents, and work with backlink and agentic commerce data."
section: "Start here"
version: "latest"
locale: "en"
updated: "2026-07-18T21:55:51.000Z"
---

# CrawlConsole documentation

Install CrawlConsole, measure AI crawler activity, connect agents, and work with backlink and agentic commerce data.

CrawlConsole gives technical and growth teams one place to understand how search engines and AI agents interact with their sites. Use these docs to install telemetry, connect an AI agent through MCP, and turn crawl and backlink data into actions.

## Choose a starting point

- [Get started](/docs/quickstart) if this is your first CrawlConsole property.
- [Install crawler analytics](/docs/installation/crawler-analytics) to begin collecting verified crawler requests.
- [Connect an AI agent](/docs/agents/mcp) to query CrawlConsole through MCP.
- [Set up WebMCP](/docs/webmcp/setup) to expose safe, structured browser tools on your site.
- [Understand backlinks](/docs/backlinks/overview) to work with Domain Score, referring domains, and link evidence.

## Built for people and agents

Every published page is available as HTML and Markdown. Add `.md` to a documentation URL for the Markdown version, or use [llms.txt](/docs/llms.txt) and [llms-full.txt](/docs/llms-full.txt) to discover or ingest the full documentation set.

The docs also expose a searchable MCP tool through the CrawlConsole MCP server. This keeps product answers grounded in the same published pages that people can read here.

## Documentation lifecycle

Documentation can be edited in Git as Markdown or managed in the CrawlConsole admin CMS. A publish action updates navigation, search, Markdown exports, and the [documentation sitemap](/sitemap-docs.xml) from one source.


---

---
title: "Quickstart"
description: "Create a property, install telemetry, verify the first crawler request, and connect your first agent."
section: "Start here"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Quickstart

Create a property, install telemetry, verify the first crawler request, and connect your first agent.

This quickstart takes you from an empty workspace to a property that can collect crawler telemetry and answer agent queries.

## Create a property

1. Sign in to CrawlConsole.
2. Open **Properties** and choose **Add property**.
3. Enter the canonical public URL for the site.
4. Keep the generated property key available for the installation step.

Use the exact production host. If both `www.example.com` and `example.com` resolve, choose the host used by your canonical URLs.

## Install crawler analytics

Open the property, then go to **Settings** and **Property Settings**. Copy the installation snippet and add it to the shared layout or template that renders every public page.

```html
<script
  async
  src="https://analytics.crawlconsole.com/tracker.js"
  data-project-key="YOUR_PROJECT_KEY"
></script>
```

Use the snippet shown in your property settings as the source of truth. It includes the correct project key and any current installation options.

## Verify telemetry

Return to **Property Settings** and confirm that crawler telemetry changes from **Pending** to **Connected**. The first verified request can take a few minutes to appear after installation.

Then open **Performance** to review:

- total verified crawler requests;
- crawler operators and user agents;
- destination pages;
- recent requests;
- HTTP errors and repeated missing paths.

## Connect an AI agent

Create an Agent API key from **Settings** and **API Keys**, then add the CrawlConsole MCP endpoint to your agent client.

```json
{
  "mcpServers": {
    "crawlconsole": {
      "url": "https://crawlconsole.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_AGENT_API_KEY"
      }
    }
  }
}
```

Keep the key in a secret manager or local environment configuration. Do not commit it to a repository.

## Run a first query

Ask your agent to list properties, select the property id for your site, and retrieve crawler analytics for the last 30 days. Property-specific tools require the canonical id returned by `list_properties`.

Next, read [AI crawler analytics](/docs/analytics/ai-crawler-analytics) or [MCP and agents](/docs/agents/mcp).


---

---
title: "Install crawler analytics"
description: "Add CrawlConsole telemetry to a site and verify that crawler, visitor, and WebMCP signals are connected."
section: "Installation"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Install crawler analytics

Add CrawlConsole telemetry to a site and verify that crawler, visitor, and WebMCP signals are connected.

CrawlConsole crawler analytics identifies verified crawler requests that ordinary browser analytics often miss. Install the tracker once in the shared site layout so it loads on every public route.

## Before you begin

You need:

- a claimed CrawlConsole property;
- permission to edit the site's shared layout or tag manager;
- the property key shown in CrawlConsole;
- a production deployment where crawler requests can reach the site.

## Add the tracker

Place the property-specific script near the end of the document `<head>` or in the site's standard analytics integration point.

```html
<script
  async
  src="https://analytics.crawlconsole.com/tracker.js"
  data-project-key="YOUR_PROJECT_KEY"
></script>
```

The script is asynchronous and should not block page rendering. Use one property key per canonical site.

## Framework placement

| Platform | Recommended placement |
| --- | --- |
| Next.js | Root layout or the shared analytics component |
| WordPress | CrawlConsole plugin or the global theme header |
| Shopify | Global theme layout before `</head>` |
| Webflow | Project-level custom code in the head |
| Static HTML | Shared head partial on every public page |

## Verify the connection

Open **Property Settings** and check the independent status for crawler, visitor, and WebMCP telemetry. Each signal has its own connected time and last-seen time.

If the status remains pending:

1. Open the deployed page source and search for the project key.
2. Confirm the request to `analytics.crawlconsole.com` is not blocked by Content Security Policy.
3. Confirm a consent manager is not suppressing the script for all traffic.
4. Check that the production host matches the claimed property.
5. Request the page from a clean browser session, then check the last-seen time again.

## Content Security Policy

Allow the tracker host in the relevant directives:

```text
script-src https://analytics.crawlconsole.com
connect-src https://analytics.crawlconsole.com
```

Merge these sources into the site's existing policy. Do not replace a complete Content Security Policy with this small example.

## What happens next

New verified crawler requests appear in the property performance view. CrawlConsole groups them by page, crawler, operator, and time, then makes the same property data available to approved AI agents.


---

---
title: "AI crawler analytics"
description: "Read verified AI crawler traffic, page demand, crawler operators, HTTP errors, and repeated missing links."
section: "Analytics"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# AI crawler analytics

Read verified AI crawler traffic, page demand, crawler operators, HTTP errors, and repeated missing links.

AI crawler analytics shows which verified crawlers request your site, which pages they choose, and where those requests fail. Use it to prioritize content that agents and search systems already seek.

## Core metrics

The performance view includes:

- **Requests**: verified crawler requests during the selected period.
- **Unique pages**: distinct normalized paths requested by crawlers.
- **Top pages**: destination paths ranked by request count.
- **Top crawlers**: verified crawler names and operators.
- **Status codes**: captured response outcomes when available.
- **Recent requests**: a time-ordered sample for investigation.

## Read page demand

High crawler demand is a reason to keep a page accurate, crawlable, and available as structured content. Compare top pages with product priorities and documentation coverage.

For example, CrawlConsole's own property currently shows strong crawler interest in web crawler identities, WebMCP, MCP discovery, backlink analysis, and agent-readable Markdown. That evidence informed the first sections of this documentation.

## Investigate HTTP errors

The HTTP errors view focuses on verified crawler requests that received `4xx` or `5xx` responses. Review the requested path, crawler, last-seen time, and frequency before deciding whether to restore content or add a redirect.

Do not redirect every crawler-requested path. Confirm whether the route was moved, misspelled, intentionally private, or generated by an agent before creating a permanent redirect.

## Review repeated missing links

The hallucinated links view identifies repeated crawler-requested `404` paths. These are candidates, not automatic proof that an AI system invented the URL.

Use this workflow:

1. Check internal links, sitemaps, feeds, and historical URLs.
2. Search external sources for references to the missing path.
3. Restore the page when the intent is valid and useful.
4. Add a narrow redirect when a clear replacement exists.
5. Leave the path missing when it has no valid intent.

## Query analytics with an agent

Use `list_properties` first, then pass the canonical property id to `get_crawler_analytics`, `get_http_errors`, or `get_hallucinated_links`.

```text
List my CrawlConsole properties. For crawlconsole.com, summarize crawler page demand over the last 30 days and identify any repeated missing paths that need review.
```


---

---
title: "Set up WebMCP"
description: "Register safe browser tools, define path rules, and monitor WebMCP calls and failures for a property."
section: "WebMCP"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Set up WebMCP

Register safe browser tools, define path rules, and monitor WebMCP calls and failures for a property.

WebMCP lets a website expose structured, permissioned tools to compatible browser agents. CrawlConsole helps you configure those tools and measure registrations, calls, failures, and active sessions.

## Check readiness first

Use the [AI Agent Readiness Scanner](/webmcp/checker) to inspect a public site before installing WebMCP. The scan checks whether the site exposes the expected signals and returns a public result page that can be shared with a developer.

## Open the WebMCP workspace

1. Open a CrawlConsole property.
2. Expand **WebMCP** in the property navigation.
3. Create or select a configuration.
4. Add tools with clear names, descriptions, and input contracts.
5. Add path rules that limit where each tool can be available.

Tool descriptions should explain when an agent should call the tool, what it returns, and any preconditions. Avoid broad descriptions such as "manage the website."

## Design safe tools

Start with read-only actions that have a narrow effect. Good first tools include:

- retrieve a product or article by canonical id;
- search public catalog content;
- check inventory or availability;
- inspect a page-specific structured record;
- submit a low-risk support or lead form with explicit confirmation.

Require authentication and a clear user confirmation before tools create orders, change account data, send messages, or perform other consequential actions.

## Apply path rules

Path rules determine which pages can register a tool. Use the smallest pattern that covers the intended experience.

```text
/products/*
/collections/*
/support/contact
```

Do not register every tool on every page by default. Page-specific registration gives agents better context and reduces accidental tool selection.

## Monitor the installation

WebMCP analytics separates:

- tag loads;
- tool registrations;
- tool calls;
- successful and failed calls;
- active sessions;
- recent failure messages.

Use `get_webmcp_analytics` through MCP when an agent needs the same property-level view.


---

---
title: "MCP and AI agents"
description: "Connect an agent to CrawlConsole, choose scopes, discover tools, and query owned property telemetry safely."
section: "Agents"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# MCP and AI agents

Connect an agent to CrawlConsole, choose scopes, discover tools, and query owned property telemetry safely.

CrawlConsole exposes a remote Model Context Protocol server at `https://crawlconsole.com/mcp`. Agent API keys control access to backlink intelligence, owned properties, telemetry, and WebMCP analytics.

## Create an Agent API key

Open **Settings** and **API Keys**, then create a key for the agent or integration. Give each client its own key so access can be reviewed and revoked independently.

Copy the raw secret when it is created. CrawlConsole stores only the protected key record and cannot show the raw secret again.

## Configure a client

Use a remote HTTP MCP configuration with a bearer token.

```json
{
  "mcpServers": {
    "crawlconsole": {
      "url": "https://crawlconsole.com/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_AGENT_API_KEY"
      }
    }
  }
}
```

Client configuration formats vary. Use the client's secret storage when it provides one.

## Discover available tools

Call `dataset_catalog` to list the tools and active limits available to the key. Common workflows include:

- `domain_authority` for Domain Score and graph authority;
- `referring_domains` for domain-level backlink sources;
- `competitor_link_gap` for domains that link to competitors but not the target;
- `list_properties` for canonical owned property ids;
- `get_crawler_analytics` for verified crawler page demand;
- `get_webmcp_analytics` for WebMCP registrations and calls;
- `search_docs` for grounded CrawlConsole product instructions.

## Work with property tools

Property tools require the canonical id returned by `list_properties`. Do not pass a domain name where a `propertyId` is required.

```text
Call list_properties. Find the property whose domain is example.com, then call get_property_telemetry_status and get_crawler_analytics for the last 30 days.
```

## Protect keys and scopes

- Create one key per agent or integration.
- Grant only the scopes the client needs.
- Keep keys out of browser code and public repositories.
- Revoke keys that are no longer in use.
- Review Agent usage for unexpected tools or request volume.

OAuth discovery is also available from CrawlConsole's well-known authorization metadata for clients that support the remote authorization flow.


---

---
title: "Backlink intelligence"
description: "Understand Domain Score, referring domains, link gaps, historical metrics, and page-level evidence workflows."
section: "Backlinks"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Backlink intelligence

Understand Domain Score, referring domains, link gaps, historical metrics, and page-level evidence workflows.

CrawlConsole turns Common Crawl graph data into domain-level backlink intelligence that people and agents can query. The main concepts are Domain Score, referring domains, link gaps, and evidence.

## Domain Score

Domain Score is CrawlConsole's authority metric derived from Common Crawl graph signals. A domain profile can include:

- Domain Score;
- backlink edges;
- referring domains;
- host count;
- harmonic centrality rank;
- source release information.

Use the score as a comparative signal, not as a guarantee of rankings or traffic.

## Referring domains

`referring_domains` returns domains that link to the target domain, ordered with link and authority context. This dataset is domain-level. It does not claim to return every exact source page URL.

Use cursors when additional pages of referring domains are available. Keep the target domain exact and normalized.

## Competitor link gaps

`competitor_link_gap` finds domains that link to selected competitors but not to the target. Use it to create a qualified research list, then investigate whether each source has a relevant page, directory, comparison, or editorial context.

## Historical intelligence

Historical backlink and authority data helps separate one-time crawl changes from sustained movement. Compare releases and time periods before concluding that a domain gained or lost meaningful authority.

## Page-level evidence

Page-level research is a separate workflow because Common Crawl domain edges do not identify a fresh, verified source URL on their own. Continue with [page-level backlink evidence](/docs/backlinks/page-level-evidence) when the question asks which exact pages link.


---

---
title: "Page-level backlink evidence"
description: "Research exact source pages, verify live links, and save evidence without overstating domain-level backlink data."
section: "Backlinks"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Page-level backlink evidence

Research exact source pages, verify live links, and save evidence without overstating domain-level backlink data.

Domain-level referring data tells you which domains are connected in the crawl graph. It does not automatically identify the current source page that contains a link. Use the evidence workflow when a user asks for exact URLs or asks which pages link.

## Start with referring domains

Call `referring_domains` for the target. Select the returned `linkingDomain` values that need source-page research.

## Create research tasks

Pass up to 50 source domains to `get_backlink_research_tasks`. The result gives the agent a structured next step for fresh web research.

```text
Use referring_domains for example.com. Pass the first 25 linkingDomain values to get_backlink_research_tasks, then find and verify the exact source pages.
```

## Verify the live page

For each source domain:

1. Search for likely source pages.
2. Open the candidate page.
3. Confirm that the page contains an `href` to the target domain.
4. Capture the actual source title, linked target URL, visible anchor text, and nearby text.
5. Record confidence based on the strength of the verification.

Do not treat an empty cache lookup as proof that no backlink exists. It only means evidence has not been saved for that target and source pair.

## Save confirmed evidence

Call `save_backlink_evidence` after verification. Saving confirmed evidence is unmetered and remains available even when metered lookup quota is exhausted.

Use `list_backlink_evidence` to retrieve evidence that was previously saved. It is a cache lookup, not a fresh web search.

## Report uncertainty clearly

Separate these statements in any result:

- the source domain appears in Common Crawl domain-level data;
- a live source page was found;
- the page contains a confirmed link;
- the target URL, anchor, and surrounding text were captured.

This prevents a domain-level edge from being presented as a verified live backlink.


---

---
title: "Agentic commerce"
description: "Evaluate commerce readiness for agents with product discovery, UCP checks, structured content, and WebMCP tools."
section: "Agentic commerce"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Agentic commerce

Evaluate commerce readiness for agents with product discovery, UCP checks, structured content, and WebMCP tools.

Agentic commerce is the set of product, checkout, and site capabilities that let an AI agent discover an item, understand its attributes, and complete a user-approved task safely.

## Evaluate the current site

Start with two public checks:

- [Product Search](/agentic-commerce/product-search) tests how a product can be found for a natural-language shopping intent.
- [UCP Checker](/agentic-commerce/ucp-checker) inspects readiness for Universal Commerce Protocol signals.

Use the results as implementation guidance, then verify the site again after changes are deployed.

## Make products understandable

Product pages should expose stable identifiers, descriptive names, availability, pricing, variants, canonical URLs, and structured data. Agents need consistent facts more than marketing-only copy.

Keep the same values aligned across:

- visible product content;
- structured data;
- feeds and catalog APIs;
- WebMCP tool responses;
- checkout and availability systems.

## Add controlled actions

WebMCP can expose product search, availability, and other structured actions to compatible agents. Start with read-only tools and add consequential actions only with authentication, validation, and user confirmation.

## Measure agent access

Crawler analytics shows whether AI and search crawlers reach commerce pages. WebMCP analytics shows whether compatible agents register and call tools. Use both signals to distinguish content discovery from tool execution.


---

---
title: "API reference"
description: "Use CrawlConsole's public HTTP endpoints and interactive request examples."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# API reference

Use CrawlConsole's public HTTP endpoints and interactive request examples.

The API reference is generated from the OpenAPI document published at [openapi.json](/docs/openapi.json). Each operation page includes its method, path, inputs, response contract, code examples, and an interactive request builder.

## Base URL

```text
https://crawlconsole.com
```

The first documented endpoints are public, narrowly scoped checkers. Authenticated product and Agent APIs use their own session or bearer-token requirements.

## Errors

Endpoints return JSON error bodies with an appropriate HTTP status when input is invalid or the operation fails.

```json
{
  "error": "A valid domain is required."
}
```

## OpenAPI source

The docs navigation and operation pages update from the same OpenAPI file. Add an operation to `content/docs/openapi.json` to publish a new API page and include it in the documentation sitemap.


---

---
title: "Get Domain Score"
description: "Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-07-19T07:40:40.360Z"
---

# Get Domain Score

Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain.

Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain.


---

---
title: "Check WebMCP readiness"
description: "Scan a public domain for AI agent and WebMCP readiness signals."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-07-19T07:40:40.360Z"
---

# Check WebMCP readiness

Scan a public domain for AI agent and WebMCP readiness signals.

Scan a public domain for AI agent and WebMCP readiness signals.


---

---
title: "Check UCP readiness"
description: "Inspect a commerce domain for Universal Commerce Protocol readiness."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-07-19T07:40:40.360Z"
---

# Check UCP readiness

Inspect a commerce domain for Universal Commerce Protocol readiness.

Inspect a commerce domain for Universal Commerce Protocol readiness.


---

---
title: "Publish agent-readable Markdown"
description: "Create concise Markdown pages for products and agents, then publish and monitor them from a CrawlConsole property."
section: "Guides"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---

# Publish agent-readable Markdown

Create concise Markdown pages for products and agents, then publish and monitor them from a CrawlConsole property.

Agent-readable Markdown gives language models a clean version of important page content without navigation, scripts, or presentation markup. CrawlConsole properties can generate, edit, publish, and disable Markdown pages.

## Choose useful source pages

Prioritize pages that answer durable product or implementation questions:

- product and feature overviews;
- installation instructions;
- API and integration guides;
- pricing or plan definitions;
- policies that agents may need to cite;
- high-demand pages identified by crawler analytics.

## Generate a draft

Open a property's **Markdown Pages** section and choose a source URL. CrawlConsole retrieves the page, extracts its main content, and creates an editable Markdown draft.

Review headings, links, code, and claims before publishing. Generated content is a draft, not an automatic source of truth.

## Publish and update

Publishing makes the Markdown page available at its assigned path. When the source content changes, regenerate or edit the draft and publish a new version.

Disable a page when its source is removed or the content should no longer be available to agents.

## Connect discovery files

Link important Markdown pages from `llms.txt` or another agent discovery index. Keep URLs stable, include useful descriptions, and avoid listing private or user-specific content.

This documentation platform follows the same approach: every page has a `.md` representation, plus complete `llms.txt` and `llms-full.txt` indexes.


---

---
title: "Author and publish documentation"
description: "Write docs in Git or the admin CMS, preview changes, publish revisions, and update search and sitemap outputs."
section: "Documentation platform"
version: "latest"
locale: "en"
updated: "2026-07-18T21:55:51.000Z"
---

# Author and publish documentation

Write docs in Git or the admin CMS, preview changes, publish revisions, and update search and sitemap outputs.

CrawlConsole documentation supports two authoring paths: repository Markdown for Git workflows and a web CMS for editorial workflows. Both feed the same public content index.

## Author in Git

Add Markdown files under `content/docs`. Frontmatter controls navigation, search, access, SEO, and sitemap behavior.

```markdown
---
title: "Page title"
description: "A concise page summary."
section: "Guides"
sectionOrder: 70
navigationOrder: 20
keywords: [keyword one, keyword two]
priority: 0.7
---

Write the page in Markdown.
```

Run the app locally and open `/docs` to preview repository content.

## Author in the CMS

Platform admins can open **Admin** and **Documentation** to:

- create a page;
- edit Markdown with a live preview;
- set the slug, section, locale, version, visibility, SEO, and sitemap fields;
- save drafts without changing the published page;
- publish a revision;
- restore an earlier revision;
- archive or delete CMS pages;
- review docs traffic, searches, and feedback.

A published CMS page overrides a repository page with the same slug, locale, and version. Draft edits remain private until the next publish action.

## Publish outputs

One publish action updates:

- the HTML page;
- sidebar and top-level documentation navigation;
- search results;
- the `search_docs` MCP tool;
- `.md`, `llms.txt`, and `llms-full.txt` exports;
- the dedicated docs sitemap;
- the changelog RSS feed when the page is in the Changelog section.

## Access and personalization

CMS pages can be public, authenticated, or admin-only. Public sitemap and agent exports include only public, indexable pages. Locale and version fields allow alternate published variants without duplicating the renderer.


---

---
title: "Documentation changelog"
description: "Product and documentation platform changes that affect CrawlConsole integrations and workflows."
section: "Changelog"
version: "latest"
locale: "en"
updated: "2026-07-18T21:55:51.000Z"
---

# Documentation changelog

Product and documentation platform changes that affect CrawlConsole integrations and workflows.

Subscribe to the [documentation RSS feed](/docs/rss.xml) to follow published changes.

## July 2026

- Added the first-party CrawlConsole documentation platform at `/docs`.
- Added Git Markdown and admin CMS authoring with drafts, publishing, and revisions.
- Added generated search and the `search_docs` MCP tool.
- Added per-page Markdown, `llms.txt`, `llms-full.txt`, `skill.md`, OpenAPI, RSS, and a dedicated sitemap.
- Added initial guides based on crawler demand for crawler analytics, WebMCP, MCP agents, backlinks, and agent-readable Markdown.


---