CrawlConsole
Back to blog

MCP Server Architecture: How to Make Website Tools Easier for AI Agents to Discover

Learn how MCP server architecture affects AI agent discovery, WebMCP actions, tool descriptions, observability, and what website teams should test before publishing.

Brittany JiaoWebMCP

MCP is quickly becoming the interface layer between AI agents and tools.

That creates a new website problem.

It is not enough to publish a tool and hope agents figure it out.

If a website exposes actions, data, product search, documentation, or workflows to agents, the structure of that tool layer affects whether an AI agent can:

  • discover the right tool
  • understand what it does
  • choose it over similar options
  • call it with the right inputs
  • recover from errors
  • avoid unsafe actions
  • create an observable trail

That is where MCP server architecture starts to overlap with Agent Experience.

For website teams, MCP architecture is not only a backend design topic. It is part of discoverability, usability, and monitoring.

Why MCP architecture matters for websites

Most websites were built for human navigation.

Humans can scan a nav bar, ignore irrelevant links, infer intent from layout, and recover when a page is unclear.

Agents need a more explicit interface.

They need to know:

  • what tool exists
  • what action it performs
  • what inputs are required
  • what the result means
  • what should not be done
  • whether the action is safe
  • where errors should route
  • whether human confirmation is needed

The June 2026 MCP architecture paper describes recurring server patterns and notes concerns around authentication, versioning, and observability. It also highlights a practical issue: exposing too many tools in one context can make tool selection harder.

That is exactly the website problem.

If your agent-facing layer exposes every page, endpoint, product, filter, and form with weak descriptions, agents may choose the wrong path or avoid the tool entirely.

The five patterns website teams should understand

The paper identifies five recurring MCP server architecture patterns. Website teams do not need to memorize the taxonomy, but the patterns are useful for deciding how agent-facing tools should be published.

1. Resource Gateway

A Resource Gateway mainly gives agents access to content or data.

For a website, that might include:

  • documentation
  • product catalogs
  • pricing pages
  • crawler profiles
  • help center content
  • inventory data
  • structured page metadata

The agent's main job is to retrieve or read.

For CrawlConsole-style sites, this maps to pages like the Web Crawlers directory, where agents may need to understand crawler identities and user-agent behavior.

What to monitor:

  • which resources are requested
  • whether important pages are reachable
  • whether crawlers or agents revisit updated pages
  • whether source pages appear in prompt tests
  • whether the agent gets stale or incomplete information

2. Tool Orchestrator

A Tool Orchestrator coordinates multiple actions.

For a website, this might look like:

  • search products
  • filter by use case
  • compare options
  • open a product page
  • check policy details
  • start checkout or request pricing

This is relevant for Agentic Commerce and Agentic Commerce Product Search, where the agent's task may involve multiple steps.

The risk is that the agent chooses the wrong tool at the wrong time.

What to monitor:

  • tool sequence
  • failed actions
  • repeated searches
  • abandoned flows
  • unexpected next steps
  • actions that should require human confirmation

3. Stateful Session Server

A Stateful Session Server remembers context across steps.

That matters when an agent is not just reading one page.

Examples:

  • compare three products over multiple turns
  • build a quote
  • continue a support request
  • save a workflow state
  • refine a product search
  • carry user constraints from one action to another

For website teams, state introduces a measurement problem.

If you only track pageviews, you cannot see whether the agent preserved the right context.

What to monitor:

  • session continuity
  • user or agent identity
  • step-by-step state changes
  • dropped context
  • repeated clarification attempts
  • whether the final action matches the original intent

4. Proxy Aggregator

A Proxy Aggregator hides several downstream systems behind one agent-facing layer.

For a website, that might combine:

  • product database
  • docs
  • inventory
  • pricing
  • reviews
  • support policies
  • checkout system
  • CRM or lead forms

The advantage is simplicity for the agent.

The risk is opacity for the website team.

If everything flows through one proxy, you still need observability into which underlying system was used.

What to monitor:

  • downstream endpoint
  • source system
  • latency
  • error source
  • permissions
  • whether the agent received complete data
  • which layer caused a failed action

5. Domain-Specific Adapter

A Domain-Specific Adapter exposes a focused workflow for one domain.

Examples:

  • product search for ecommerce
  • crawler lookup for technical SEO
  • domain inspection for analytics
  • MCP server discovery for developers
  • compliance document retrieval for procurement

This is often the best pattern for website-facing agent actions because the tool has a clear job.

For CrawlConsole, MCP Finder and MCP Finder for GitHub are closer to this idea than a generic "search everything" interface.

What to monitor:

  • whether the agent understands the domain
  • whether the tool description matches the real job
  • which queries lead to use
  • which outputs satisfy the task
  • whether related pages support discovery

Do not expose every action at once

One practical lesson from MCP architecture work is that tool choice can degrade when too many tools are exposed in the same context.

For website teams, this means:

  • do not publish every internal endpoint as an agent tool
  • do not describe every CTA as equally important
  • do not mix read-only actions with high-risk actions without clear boundaries
  • do not make agents choose from dozens of vague tools

Instead, start with a smaller set of high-value actions.

Good first actions:

  • search products
  • check crawler identity
  • retrieve documentation
  • compare plan features
  • inspect a domain
  • find an MCP server
  • verify WebMCP setup

Bad first actions:

  • submit any form without context
  • modify user data
  • purchase without confirmation
  • delete or publish content
  • trigger workflows with unclear side effects

Use WebMCP Checker to test whether an action is clear enough before treating it as agent-ready.

What makes a tool discoverable to agents?

An agent-discoverable tool needs more than a clever name.

It needs:

  • a specific action name
  • a plain-language description
  • clear required inputs
  • expected output
  • constraints
  • safety boundary
  • related source pages
  • error behavior
  • example use cases
  • crawlable support content

For example, this is weak:

Search

This is stronger:

Search CrawlConsole web crawler profiles by crawler name, user agent, or bot purpose.
Returns matching crawler profile pages with source URLs and suggested monitoring fields.

The stronger version tells the agent what the tool does, what inputs are appropriate, and what kind of output to expect.

Connect MCP tools to website pages

MCP tools should not live in isolation.

Agents still need supporting pages that explain:

  • what the tool is for
  • who should use it
  • what terms mean
  • what the workflow is
  • what related actions exist
  • what evidence should be monitored

This is why internal linking matters.

An article about MCP tool discovery should link to:

The goal is to create a connected entity graph.

That helps humans understand the workflow, and it gives crawlers and agents a clearer map of what each page or tool is supposed to do.

Test discovery before publishing broadly

Before promoting an MCP or WebMCP tool, run prompt tests.

Use the Prompt Library to save prompts like:

Find a tool that can check whether a website has WebMCP actions.

Find an MCP discovery tool for GitHub-related servers.

What tool should I use to identify AI crawler user agents on a website?

Which website action should an AI agent use to search products by intent?

Track whether the answer:

  • finds the right page
  • chooses the right tool
  • describes the action accurately
  • links to the right source
  • avoids unsafe assumptions
  • misses a related CrawlConsole page

Prompt tests will not prove every agent will behave the same way.

They give you a repeatable signal before real traffic becomes large enough to measure.

Monitor after publishing

After publishing a tool or action, watch the evidence layer.

Crawler evidence:

  • which crawlers reached the page
  • which crawler profiles are being requested
  • whether important source pages get revisited
  • whether unknown agents appear

Agent evidence:

  • which tool/action was selected
  • what input was used
  • whether the call succeeded
  • what error occurred
  • whether human confirmation was needed

Business evidence:

  • whether the action led to a useful visit
  • whether a product, demo, or signup flow started
  • whether the user returned
  • whether the result matched the intent

For delegated-agent traffic, profiles like Claude-User help distinguish a normal crawler from an agent that may be acting on behalf of a real user.

MCP architecture checklist for website teams

Use this before publishing or promoting an MCP/WebMCP tool.

1. Pick the pattern: resource, orchestrator, stateful session, proxy, or domain adapter.
2. Keep the first tool set small and specific.
3. Give every action a clear name, input, output, and constraint.
4. Separate read-only actions from actions that modify, submit, or buy.
5. Link every tool to supporting website pages.
6. Check whether support pages are crawlable.
7. Validate action descriptions with WebMCP Checker.
8. Run repeatable prompt tests before launch.
9. Monitor crawler visits, tool calls, errors, and outcomes after publishing.
10. Update descriptions when agents choose the wrong tool.

The bottom line

MCP server architecture is becoming part of website discoverability.

The question is no longer only:

Can an agent access our tool?

The better question is:

Can an agent discover the right tool, understand when to use it, call it safely, and leave enough evidence for us to improve the flow?

For website teams, that means MCP and WebMCP should be treated as part of Agent Experience.

Use MCP Finder to understand discovery demand. Use WebMCP and WebMCP Checker to make actions clearer. Use the Prompt Library to test whether agents can find and describe the right tool.

Then monitor what actually happens after agents arrive.

That is how MCP architecture becomes measurable.