# CrawlConsole documentation
> Complete published documentation for crawler analytics, MCP agents, WebMCP, backlinks, agentic commerce, and CrawlConsole APIs.
---
title: "CrawlConsole documentation"
description: "Install CrawlConsole, measure AI crawler activity, connect agents, and work with backlink and agentic commerce data."
section: "Start here"
version: "latest"
locale: "en"
updated: "2026-07-19T14:52:23.000Z"
---
# CrawlConsole documentation
Install CrawlConsole, measure AI crawler activity, connect agents, and work with backlink and agentic commerce data.
CrawlConsole gives technical and growth teams one place to understand how search engines and AI agents interact with their sites. Use these docs to install telemetry, connect an AI agent through MCP, and turn crawl and backlink data into actions.
## Choose a starting point
- [Get started](/docs/quickstart) if this is your first CrawlConsole property.
- [Install crawler analytics](/docs/installation/crawler-analytics) to begin collecting verified crawler requests.
- [Connect an AI agent](/docs/agents/mcp) to query CrawlConsole through MCP.
- [Connect Google Search Console](/docs/integrations/google-search-console) to query search performance and index status.
- [Set up WebMCP](/docs/webmcp/setup) to expose safe, structured browser tools on your site.
- [Understand backlinks](/docs/backlinks/overview) to work with Domain Score, referring domains, and link evidence.
## Built for people and agents
Every published page is available as HTML and Markdown. Add `.md` to a documentation URL for the Markdown version, or use [llms.txt](/docs/llms.txt) and [llms-full.txt](/docs/llms-full.txt) to discover or ingest the full documentation set.
The docs also expose a searchable MCP tool through the CrawlConsole MCP server. This keeps product answers grounded in the same published pages that people can read here.
## Documentation lifecycle
Documentation can be edited in Git as Markdown or managed in the CrawlConsole admin CMS. A publish action updates navigation, search, Markdown exports, and the [documentation sitemap](/sitemap-docs.xml) from one source.
---
---
title: "Quickstart"
description: "Create a property, install telemetry, verify the first crawler request, and connect your first agent."
section: "Start here"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Quickstart
Create a property, install telemetry, verify the first crawler request, and connect your first agent.
This quickstart takes you from an empty workspace to a property that can collect crawler telemetry and answer agent queries.
## Create a property
1. Sign in to CrawlConsole.
2. Open **Properties** and choose **Add property**.
3. Enter the canonical public URL for the site.
4. Keep the generated property key available for the installation step.
Use the exact production host. If both `www.example.com` and `example.com` resolve, choose the host used by your canonical URLs.
## Install crawler analytics
Open the property, then go to **Settings** and **Property Settings**. Copy the installation snippet and add it to the shared layout or template that renders every public page.
```html
```
Use the snippet shown in your property settings as the source of truth. It includes the correct project key and any current installation options.
## Verify telemetry
Return to **Property Settings** and confirm that crawler telemetry changes from **Pending** to **Connected**. The first verified request can take a few minutes to appear after installation.
Then open **Performance** to review:
- total verified crawler requests;
- crawler operators and user agents;
- destination pages;
- recent requests;
- HTTP errors and repeated missing paths.
## Connect an AI agent
Create an Agent API key from **Settings** and **API Keys**, then add the CrawlConsole MCP endpoint to your agent client.
```json
{
"mcpServers": {
"crawlconsole": {
"url": "https://crawlconsole.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_AGENT_API_KEY"
}
}
}
}
```
Keep the key in a secret manager or local environment configuration. Do not commit it to a repository.
## Run a first query
Ask your agent to list properties, select the property id for your site, and retrieve crawler analytics for the last 30 days. Property-specific tools require the canonical id returned by `list_properties`.
Next, read [AI crawler analytics](/docs/analytics/ai-crawler-analytics) or [MCP and agents](/docs/agents/mcp).
---
---
title: "Install crawler analytics"
description: "Add CrawlConsole telemetry to a site and verify that crawler, visitor, and WebMCP signals are connected."
section: "Installation"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Install crawler analytics
Add CrawlConsole telemetry to a site and verify that crawler, visitor, and WebMCP signals are connected.
CrawlConsole crawler analytics identifies verified crawler requests that ordinary browser analytics often miss. Install the tracker once in the shared site layout so it loads on every public route.
## Before you begin
You need:
- a claimed CrawlConsole property;
- permission to edit the site's shared layout or tag manager;
- the property key shown in CrawlConsole;
- a production deployment where crawler requests can reach the site.
## Add the tracker
Place the property-specific script near the end of the document `` or in the site's standard analytics integration point.
```html
```
The script is asynchronous and should not block page rendering. Use one property key per canonical site.
## Framework placement
| Platform | Recommended placement |
| --- | --- |
| Next.js | Root layout or the shared analytics component |
| WordPress | CrawlConsole plugin or the global theme header |
| Shopify | Global theme layout before `` |
| Webflow | Project-level custom code in the head |
| Static HTML | Shared head partial on every public page |
## Verify the connection
Open **Property Settings** and check the independent status for crawler, visitor, and WebMCP telemetry. Each signal has its own connected time and last-seen time.
If the status remains pending:
1. Open the deployed page source and search for the project key.
2. Confirm the request to `analytics.crawlconsole.com` is not blocked by Content Security Policy.
3. Confirm a consent manager is not suppressing the script for all traffic.
4. Check that the production host matches the claimed property.
5. Request the page from a clean browser session, then check the last-seen time again.
## Content Security Policy
Allow the tracker host in the relevant directives:
```text
script-src https://analytics.crawlconsole.com
connect-src https://analytics.crawlconsole.com
```
Merge these sources into the site's existing policy. Do not replace a complete Content Security Policy with this small example.
## What happens next
New verified crawler requests appear in the property performance view. CrawlConsole groups them by page, crawler, operator, and time, then makes the same property data available to approved AI agents.
---
---
title: "AI crawler analytics"
description: "Read verified AI crawler traffic, page demand, crawler operators, HTTP errors, and repeated missing links."
section: "Analytics"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# AI crawler analytics
Read verified AI crawler traffic, page demand, crawler operators, HTTP errors, and repeated missing links.
AI crawler analytics shows which verified crawlers request your site, which pages they choose, and where those requests fail. Use it to prioritize content that agents and search systems already seek.
## Core metrics
The performance view includes:
- **Requests**: verified crawler requests during the selected period.
- **Unique pages**: distinct normalized paths requested by crawlers.
- **Top pages**: destination paths ranked by request count.
- **Top crawlers**: verified crawler names and operators.
- **Status codes**: captured response outcomes when available.
- **Recent requests**: a time-ordered sample for investigation.
## Read page demand
High crawler demand is a reason to keep a page accurate, crawlable, and available as structured content. Compare top pages with product priorities and documentation coverage.
For example, CrawlConsole's own property currently shows strong crawler interest in web crawler identities, WebMCP, MCP discovery, backlink analysis, and agent-readable Markdown. That evidence informed the first sections of this documentation.
## Investigate HTTP errors
The HTTP errors view focuses on verified crawler requests that received `4xx` or `5xx` responses. Review the requested path, crawler, last-seen time, and frequency before deciding whether to restore content or add a redirect.
Do not redirect every crawler-requested path. Confirm whether the route was moved, misspelled, intentionally private, or generated by an agent before creating a permanent redirect.
## Review repeated missing links
The hallucinated links view identifies repeated crawler-requested `404` paths. These are candidates, not automatic proof that an AI system invented the URL.
Use this workflow:
1. Check internal links, sitemaps, feeds, and historical URLs.
2. Search external sources for references to the missing path.
3. Restore the page when the intent is valid and useful.
4. Add a narrow redirect when a clear replacement exists.
5. Leave the path missing when it has no valid intent.
## Query analytics with an agent
Use `list_properties` first, then pass the canonical property id to `get_crawler_analytics`, `get_http_errors`, or `get_hallucinated_links`.
```text
List my CrawlConsole properties. For crawlconsole.com, summarize crawler page demand over the last 30 days and identify any repeated missing paths that need review.
```
---
---
title: "Set up WebMCP"
description: "Register safe browser tools, define path rules, and monitor WebMCP calls and failures for a property."
section: "WebMCP"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Set up WebMCP
Register safe browser tools, define path rules, and monitor WebMCP calls and failures for a property.
WebMCP lets a website expose structured, permissioned tools to compatible browser agents. CrawlConsole helps you configure those tools and measure registrations, calls, failures, and active sessions.
## Check readiness first
Use the [AI Agent Readiness Scanner](/webmcp/checker) to inspect a public site before installing WebMCP. The scan checks whether the site exposes the expected signals and returns a public result page that can be shared with a developer.
## Open the WebMCP workspace
1. Open a CrawlConsole property.
2. Expand **WebMCP** in the property navigation.
3. Create or select a configuration.
4. Add tools with clear names, descriptions, and input contracts.
5. Add path rules that limit where each tool can be available.
Tool descriptions should explain when an agent should call the tool, what it returns, and any preconditions. Avoid broad descriptions such as "manage the website."
## Design safe tools
Start with read-only actions that have a narrow effect. Good first tools include:
- retrieve a product or article by canonical id;
- search public catalog content;
- check inventory or availability;
- inspect a page-specific structured record;
- submit a low-risk support or lead form with explicit confirmation.
Require authentication and a clear user confirmation before tools create orders, change account data, send messages, or perform other consequential actions.
## Apply path rules
Path rules determine which pages can register a tool. Use the smallest pattern that covers the intended experience.
```text
/products/*
/collections/*
/support/contact
```
Do not register every tool on every page by default. Page-specific registration gives agents better context and reduces accidental tool selection.
## Monitor the installation
WebMCP analytics separates:
- tag loads;
- tool registrations;
- tool calls;
- successful and failed calls;
- active sessions;
- recent failure messages.
Use `get_webmcp_analytics` through MCP when an agent needs the same property-level view.
---
---
title: "Google Search Console"
description: "Connect a Search Console site to a CrawlConsole property and query performance comparisons, opportunities, URL inspections, and sitemap status through MCP."
section: "Integrations"
version: "latest"
locale: "en"
updated: "2026-07-21T12:32:35.000Z"
---
# Google Search Console
Connect a Search Console site to a CrawlConsole property and query performance comparisons, opportunities, URL inspections, and sitemap status through MCP.
Connect Google Search Console to bring first-party organic search data into a CrawlConsole property. The CrawlConsole MCP server can then query and compare performance, identify opportunities, inspect URLs individually or in batches, and read submitted sitemap status without exposing Google credentials to the agent.
## Connect a Search Console site
1. Open the CrawlConsole property you want to configure.
2. Open **Settings**, then **Data Connectors**.
3. Select **Connect Google Search Console** and approve the requested Google access.
4. Choose the Search Console domain property or URL-prefix property that represents the CrawlConsole property.
5. Confirm that Search Console appears as connected.
You need permission to edit data sources in the CrawlConsole workspace and access to the selected site in Google Search Console. A Search Console connection belongs to the CrawlConsole property and its workspace. Agent keys use their own scopes and optional domain restrictions to determine which workspace properties they can query; access is not inherited from the individual Google account that originally completed OAuth.
## Give an agent access
Create an Agent API key under **Settings** and **API Keys**. Grant `properties:read` so the agent can call `list_properties`, and grant `telemetry:read` for the Search Console tools.
Always begin with `list_properties` and pass its canonical `propertyId` to the other tools. The GSC tools do not accept a Search Console site URL, OAuth connection id, or access token. CrawlConsole resolves the connector from the authorized workspace and property on every call.
```text
Call list_properties, find example.com, and use its propertyId to return
Google Search Console opportunities for the last 28 complete days.
```
## Available MCP tools
### `get_gsc_performance`
Query clicks, impressions, CTR, and average position. Group results by any supported combination of date, hour, query, page, country, device, and search appearance. You can also apply AND filters, select a Google search surface, include fresh data, compare with the immediately preceding period, and paginate with `nextCursor`.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"days": 28,
"dimensions": ["query", "page"],
"searchType": "web",
"comparePreviousPeriod": true,
"limit": 100
}
```
The tool defaults to the last 28 complete UTC days and 100 rows. A response can contain up to 1,000 rows; pass the returned opaque cursor back with the same query to continue. Explicit `startDate` and `endDate` values use inclusive `YYYY-MM-DD` dates.
### `compare_gsc_performance`
Compare dimension-level performance across two arbitrary date ranges. Rows that appear in only one period are retained, and the response includes current and baseline metrics, absolute and percentage changes, and ranking improvement. Sorting is applied after Google returns its top source rows.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"currentStartDate": "2026-07-01",
"currentEndDate": "2026-07-20",
"baselineStartDate": "2026-06-11",
"baselineEndDate": "2026-06-30",
"dimensions": ["query", "page"],
"sortBy": "absoluteClicksChange",
"limit": 100
}
```
The tool fetches up to 1,000 source rows per period by default and allows up to 5,000. Check the returned truncation fields before treating a comparison as comprehensive.
### `get_gsc_opportunities`
Return a prioritized layer derived from current and previous Search Console performance. It can identify:
- striking-distance queries within a selected position range;
- high-impression results with low CTR;
- query and page pairs whose clicks declined, including pairs that disappeared from the current period;
- possible cannibalization where multiple pages receive impressions for the same query.
The thresholds are inputs, so an agent can tune them to the site instead of relying on hidden scoring rules. The tool defaults to a 28-day comparison and returns up to 200 opportunities.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"days": 28,
"types": ["striking_distance", "declining", "cannibalization"],
"minImpressions": 100,
"minPosition": 4,
"maxPosition": 20,
"limit": 50
}
```
### `inspect_gsc_url`
Inspect one absolute HTTP or HTTPS URL inside the connected Search Console property. The response includes the Search Console inspection link and available index verdict, coverage, robots, fetch, canonical, last-crawl, AMP, mobile-usability, and rich-result fields.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"url": "https://example.com/docs/getting-started"
}
```
For a URL-prefix Search Console site, the URL must share its origin and path prefix. For a domain property, the root domain and its subdomains are allowed. CrawlConsole rejects out-of-property URLs before asking Google to inspect them.
### `inspect_gsc_urls`
Inspect up to 10 URLs in one call. CrawlConsole validates every URL before contacting Google, uses bounded concurrency, and returns the complete inspection for each URL plus categorized lists for indexing, canonical, robots, fetch, structured-data, and AMP issues.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"urls": [
"https://example.com/docs/getting-started",
"https://example.com/docs/installation"
]
}
```
### `get_gsc_sitemaps`
List submitted sitemap paths, last submission and download times, pending state, warnings, errors, and submitted/indexed counts by content type. Pass `sitemapIndex` to list child sitemaps from one sitemap index. This tool is read-only: it does not submit or delete sitemaps.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"includeContents": true,
"limit": 100
}
```
### `get_gsc_sitemap`
Fetch the status of one exact sitemap, including its pending/index state, submission and download times, errors, warnings, and submitted/indexed counts by content type.
```json
{
"propertyId": "PROPERTY_ID_FROM_LIST_PROPERTIES",
"sitemapUrl": "https://example.com/sitemap.xml",
"includeContents": true
}
```
## Access, usage, and freshness
Search Console calls are owned-property queries. They do not consume CrawlConsole's metered backlink or external-intelligence Agent allowance, including when that allowance is exhausted. A generous per-key concurrency safety limit still applies, and Google may temporarily limit a busy Search Console property. If a call is limited, retry with fewer parallel requests.
CrawlConsole briefly caches identical GSC reads to reduce latency and unnecessary Google API calls. Search Console performance itself can be delayed or incomplete; use `dataState: "all"` when fresh data matters, and check the returned incomplete-date metadata before treating the newest rows as final.
Search Analytics returns Google's top rows for a query rather than a guaranteed exhaustive export. Use focused dimensions and filters, and paginate when you need deeper result sets.
## Troubleshooting
- **Property not found:** call `list_properties` again and use the canonical id available to the current Agent key.
- **Search Console is not connected:** connect it from that property's **Data Connectors** settings.
- **Connection does not match:** reconnect and select the Search Console site for the same domain as the CrawlConsole property.
- **Authorization unavailable:** reconnect Google Search Console; CrawlConsole may no longer have permission to refresh access.
- **Inspection URL is outside the property:** use a URL covered by the selected domain or URL-prefix property.
- **Temporary limit:** reduce parallel calls and retry shortly.
The page is part of the dedicated [documentation sitemap](/sitemap-docs.xml). Publishing a CMS override or updating this Markdown source refreshes the page, navigation, agent exports, search index, and sitemap entry together.
---
---
title: "MCP and AI agents"
description: "Connect an agent to CrawlConsole, choose scopes, discover tools, and query owned property telemetry safely."
section: "Agents"
version: "latest"
locale: "en"
updated: "2026-07-21T12:32:28.000Z"
---
# MCP and AI agents
Connect an agent to CrawlConsole, choose scopes, discover tools, and query owned property telemetry safely.
CrawlConsole exposes a remote Model Context Protocol server at `https://crawlconsole.com/mcp`. Agent API keys control access to backlink intelligence, owned properties, telemetry, connected Google Search Console data, and WebMCP analytics.
## Create an Agent API key
Open **Settings** and **API Keys**, then create a key for the agent or integration. Give each client its own key so access can be reviewed and revoked independently.
Copy the raw secret when it is created. CrawlConsole stores only the protected key record and cannot show the raw secret again.
## Configure a client
Use a remote HTTP MCP configuration with a bearer token.
```json
{
"mcpServers": {
"crawlconsole": {
"url": "https://crawlconsole.com/mcp",
"headers": {
"Authorization": "Bearer YOUR_AGENT_API_KEY"
}
}
}
}
```
Client configuration formats vary. Use the client's secret storage when it provides one.
## Discover available tools
Call `dataset_catalog` to list the tools and active limits available to the key. Common workflows include:
- `domain_authority` for Domain Score and graph authority;
- `referring_domains` for domain-level backlink sources;
- `competitor_link_gap` for domains that link to competitors but not the target;
- `list_properties` for canonical owned property ids;
- `get_crawler_analytics` for verified crawler page demand;
- `get_gsc_performance` for connected Search Console clicks, impressions, CTR, and position;
- `compare_gsc_performance` for dimension-level changes between arbitrary periods;
- `get_gsc_opportunities` for prioritized organic-search opportunities;
- `inspect_gsc_url` for Google index and canonical status;
- `inspect_gsc_urls` for batch inspection and categorized indexing issues;
- `get_gsc_sitemaps` for submitted sitemap status;
- `get_gsc_sitemap` for one exact sitemap's status and content counts;
- `get_webmcp_analytics` for WebMCP registrations and calls;
- `search_docs` for grounded CrawlConsole product instructions.
## Work with property tools
Property tools require the canonical id returned by `list_properties`. Do not pass a domain name where a `propertyId` is required.
```text
Call list_properties. Find the property whose domain is example.com, then call get_property_telemetry_status and get_crawler_analytics for the last 30 days.
```
Read [Google Search Console](/docs/integrations/google-search-console) for connector setup, tool inputs, access boundaries, and freshness guidance.
## Protect keys and scopes
- Create one key per agent or integration.
- Grant only the scopes the client needs.
- Keep keys out of browser code and public repositories.
- Revoke keys that are no longer in use.
- Review Agent usage for unexpected tools or request volume.
OAuth discovery is also available from CrawlConsole's well-known authorization metadata for clients that support the remote authorization flow.
---
---
title: "Backlink intelligence"
description: "Understand Domain Score, referring domains, link gaps, historical metrics, and page-level evidence workflows."
section: "Backlinks"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Backlink intelligence
Understand Domain Score, referring domains, link gaps, historical metrics, and page-level evidence workflows.
CrawlConsole turns Common Crawl graph data into domain-level backlink intelligence that people and agents can query. The main concepts are Domain Score, referring domains, link gaps, and evidence.
## Domain Score
Domain Score is CrawlConsole's authority metric derived from Common Crawl graph signals. A domain profile can include:
- Domain Score;
- backlink edges;
- referring domains;
- host count;
- harmonic centrality rank;
- source release information.
Use the score as a comparative signal, not as a guarantee of rankings or traffic.
## Referring domains
`referring_domains` returns domains that link to the target domain, ordered with link and authority context. This dataset is domain-level. It does not claim to return every exact source page URL.
Use cursors when additional pages of referring domains are available. Keep the target domain exact and normalized.
## Competitor link gaps
`competitor_link_gap` finds domains that link to selected competitors but not to the target. Use it to create a qualified research list, then investigate whether each source has a relevant page, directory, comparison, or editorial context.
## Historical intelligence
Historical backlink and authority data helps separate one-time crawl changes from sustained movement. Compare releases and time periods before concluding that a domain gained or lost meaningful authority.
## Page-level evidence
Page-level research is a separate workflow because Common Crawl domain edges do not identify a fresh, verified source URL on their own. Continue with [page-level backlink evidence](/docs/backlinks/page-level-evidence) when the question asks which exact pages link.
---
---
title: "Page-level backlink evidence"
description: "Research exact source pages, verify live links, and save evidence without overstating domain-level backlink data."
section: "Backlinks"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Page-level backlink evidence
Research exact source pages, verify live links, and save evidence without overstating domain-level backlink data.
Domain-level referring data tells you which domains are connected in the crawl graph. It does not automatically identify the current source page that contains a link. Use the evidence workflow when a user asks for exact URLs or asks which pages link.
## Start with referring domains
Call `referring_domains` for the target. Select the returned `linkingDomain` values that need source-page research.
## Create research tasks
Pass up to 50 source domains to `get_backlink_research_tasks`. The result gives the agent a structured next step for fresh web research.
```text
Use referring_domains for example.com. Pass the first 25 linkingDomain values to get_backlink_research_tasks, then find and verify the exact source pages.
```
## Verify the live page
For each source domain:
1. Search for likely source pages.
2. Open the candidate page.
3. Confirm that the page contains an `href` to the target domain.
4. Capture the actual source title, linked target URL, visible anchor text, and nearby text.
5. Record confidence based on the strength of the verification.
Do not treat an empty cache lookup as proof that no backlink exists. It only means evidence has not been saved for that target and source pair.
## Save confirmed evidence
Call `save_backlink_evidence` after verification. Saving confirmed evidence is unmetered and remains available even when metered lookup quota is exhausted.
Use `list_backlink_evidence` to retrieve evidence that was previously saved. It is a cache lookup, not a fresh web search.
## Report uncertainty clearly
Separate these statements in any result:
- the source domain appears in Common Crawl domain-level data;
- a live source page was found;
- the page contains a confirmed link;
- the target URL, anchor, and surrounding text were captured.
This prevents a domain-level edge from being presented as a verified live backlink.
---
---
title: "Agentic commerce"
description: "Evaluate commerce readiness for agents with product discovery, UCP checks, structured content, and WebMCP tools."
section: "Agentic commerce"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Agentic commerce
Evaluate commerce readiness for agents with product discovery, UCP checks, structured content, and WebMCP tools.
Agentic commerce is the set of product, checkout, and site capabilities that let an AI agent discover an item, understand its attributes, and complete a user-approved task safely.
## Evaluate the current site
Start with two public checks:
- [Product Search](/agentic-commerce/product-search) tests how a product can be found for a natural-language shopping intent.
- [UCP Checker](/agentic-commerce/ucp-checker) inspects readiness for Universal Commerce Protocol signals.
Use the results as implementation guidance, then verify the site again after changes are deployed.
## Make products understandable
Product pages should expose stable identifiers, descriptive names, availability, pricing, variants, canonical URLs, and structured data. Agents need consistent facts more than marketing-only copy.
Keep the same values aligned across:
- visible product content;
- structured data;
- feeds and catalog APIs;
- WebMCP tool responses;
- checkout and availability systems.
## Add controlled actions
WebMCP can expose product search, availability, and other structured actions to compatible agents. Start with read-only tools and add consequential actions only with authentication, validation, and user confirmation.
## Measure agent access
Crawler analytics shows whether AI and search crawlers reach commerce pages. WebMCP analytics shows whether compatible agents register and call tools. Use both signals to distinguish content discovery from tool execution.
---
---
title: "API reference"
description: "Use CrawlConsole's public HTTP endpoints and interactive request examples."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# API reference
Use CrawlConsole's public HTTP endpoints and interactive request examples.
The API reference is generated from the OpenAPI document published at [openapi.json](/docs/openapi.json). Each operation page includes its method, path, inputs, response contract, code examples, and an interactive request builder.
## Base URL
```text
https://crawlconsole.com
```
The first documented endpoints are public, narrowly scoped checkers. Authenticated product and Agent APIs use their own session or bearer-token requirements.
## Errors
Endpoints return JSON error bodies with an appropriate HTTP status when input is invalid or the operation fails.
```json
{
"error": "A valid domain is required."
}
```
## OpenAPI source
The docs navigation and operation pages update from the same OpenAPI file. Add an operation to `content/docs/openapi.json` to publish a new API page and include it in the documentation sitemap.
---
---
title: "Get Domain Score"
description: "Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-08-12T09:13:44.023Z"
---
# Get Domain Score
Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain.
Return CrawlConsole Domain Score and Common Crawl graph metrics for an exact domain.
---
---
title: "Check WebMCP readiness"
description: "Scan a public domain for AI agent and WebMCP readiness signals."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-08-12T09:13:44.023Z"
---
# Check WebMCP readiness
Scan a public domain for AI agent and WebMCP readiness signals.
Scan a public domain for AI agent and WebMCP readiness signals.
---
---
title: "Check UCP readiness"
description: "Inspect a commerce domain for Universal Commerce Protocol readiness."
section: "API reference"
version: "latest"
locale: "en"
updated: "2026-08-12T09:13:44.023Z"
---
# Check UCP readiness
Inspect a commerce domain for Universal Commerce Protocol readiness.
Inspect a commerce domain for Universal Commerce Protocol readiness.
---
---
title: "Publish agent-readable Markdown"
description: "Create concise Markdown pages for products and agents, then publish and monitor them from a CrawlConsole property."
section: "Guides"
version: "latest"
locale: "en"
updated: "2026-07-18T20:52:38.000Z"
---
# Publish agent-readable Markdown
Create concise Markdown pages for products and agents, then publish and monitor them from a CrawlConsole property.
Agent-readable Markdown gives language models a clean version of important page content without navigation, scripts, or presentation markup. CrawlConsole properties can generate, edit, publish, and disable Markdown pages.
## Choose useful source pages
Prioritize pages that answer durable product or implementation questions:
- product and feature overviews;
- installation instructions;
- API and integration guides;
- pricing or plan definitions;
- policies that agents may need to cite;
- high-demand pages identified by crawler analytics.
## Generate a draft
Open a property's **Markdown Pages** section and choose a source URL. CrawlConsole retrieves the page, extracts its main content, and creates an editable Markdown draft.
Review headings, links, code, and claims before publishing. Generated content is a draft, not an automatic source of truth.
## Publish and update
Publishing makes the Markdown page available at its assigned path. When the source content changes, regenerate or edit the draft and publish a new version.
Disable a page when its source is removed or the content should no longer be available to agents.
## Connect discovery files
Link important Markdown pages from `llms.txt` or another agent discovery index. Keep URLs stable, include useful descriptions, and avoid listing private or user-specific content.
This documentation platform follows the same approach: every page has a `.md` representation, plus complete `llms.txt` and `llms-full.txt` indexes.
---
---
title: "Author and publish documentation"
description: "Write docs in Git or the admin CMS, preview changes, publish revisions, and update search and sitemap outputs."
section: "Documentation platform"
version: "latest"
locale: "en"
updated: "2026-07-18T21:55:51.000Z"
---
# Author and publish documentation
Write docs in Git or the admin CMS, preview changes, publish revisions, and update search and sitemap outputs.
CrawlConsole documentation supports two authoring paths: repository Markdown for Git workflows and a web CMS for editorial workflows. Both feed the same public content index.
## Author in Git
Add Markdown files under `content/docs`. Frontmatter controls navigation, search, access, SEO, and sitemap behavior.
```markdown
---
title: "Page title"
description: "A concise page summary."
section: "Guides"
sectionOrder: 70
navigationOrder: 20
keywords: [keyword one, keyword two]
priority: 0.7
---
Write the page in Markdown.
```
Run the app locally and open `/docs` to preview repository content.
## Author in the CMS
Platform admins can open **Admin** and **Documentation** to:
- create a page;
- edit Markdown with a live preview;
- set the slug, section, locale, version, visibility, SEO, and sitemap fields;
- save drafts without changing the published page;
- publish a revision;
- restore an earlier revision;
- archive or delete CMS pages;
- review docs traffic, searches, and feedback.
A published CMS page overrides a repository page with the same slug, locale, and version. Draft edits remain private until the next publish action.
## Publish outputs
One publish action updates:
- the HTML page;
- sidebar and top-level documentation navigation;
- search results;
- the `search_docs` MCP tool;
- `.md`, `llms.txt`, and `llms-full.txt` exports;
- the dedicated docs sitemap;
- the changelog RSS feed when the page is in the Changelog section.
## Access and personalization
CMS pages can be public, authenticated, or admin-only. Public sitemap and agent exports include only public, indexable pages. Locale and version fields allow alternate published variants without duplicating the renderer.
---
---
title: "Documentation changelog"
description: "Product and documentation platform changes that affect CrawlConsole integrations and workflows."
section: "Changelog"
version: "latest"
locale: "en"
updated: "2026-07-18T21:55:51.000Z"
---
# Documentation changelog
Product and documentation platform changes that affect CrawlConsole integrations and workflows.
Subscribe to the [documentation RSS feed](/docs/rss.xml) to follow published changes.
## July 2026
- Added the first-party CrawlConsole documentation platform at `/docs`.
- Added Git Markdown and admin CMS authoring with drafts, publishing, and revisions.
- Added generated search and the `search_docs` MCP tool.
- Added per-page Markdown, `llms.txt`, `llms-full.txt`, `skill.md`, OpenAPI, RSS, and a dedicated sitemap.
- Added initial guides based on crawler demand for crawler analytics, WebMCP, MCP agents, backlinks, and agent-readable Markdown.
---