Publish agent-readable Markdown

Agent-readable Markdown gives language models a clean version of important page content without navigation, scripts, or presentation markup. CrawlConsole properties can generate, edit, publish, and disable Markdown pages.

Choose useful source pages

Prioritize pages that answer durable product or implementation questions:

• product and feature overviews;
• installation instructions;
• API and integration guides;
• pricing or plan definitions;
• policies that agents may need to cite;
• high-demand pages identified by crawler analytics.

Generate a draft

Open a property's Markdown Pages section and choose a source URL. CrawlConsole retrieves the page, extracts its main content, and creates an editable Markdown draft.

Review headings, links, code, and claims before publishing. Generated content is a draft, not an automatic source of truth.

Publish and update

Publishing makes the Markdown page available at its assigned path. When the source content changes, regenerate or edit the draft and publish a new version.

Disable a page when its source is removed or the content should no longer be available to agents.

Connect discovery files

Link important Markdown pages from llms.txt or another agent discovery index. Keep URLs stable, include useful descriptions, and avoid listing private or user-specific content.

This documentation platform follows the same approach: every page has a .md representation, plus complete llms.txt and llms-full.txt indexes.