CrawlConsole
Back to blog

How to Check If PerplexityBot Crawled Your Website

Learn how to check whether PerplexityBot crawled your website, what status code it received, and how crawler access connects to Perplexity AI search visibility.

Brittany JiaoCrawler Guides

Perplexity is one of the AI search engines site owners care about most because it often shows sources directly inside answers.

That creates a practical question:

Did PerplexityBot actually crawl my website?

The answer is not the same as "does Perplexity mention my brand?" or "did I get referral traffic from perplexity.ai?"

Those are useful signals, but they happen later in the visibility chain. Before your page can be surfaced, cited, or summarized, Perplexity's crawler needs some path to discover and access your content.

This guide walks through how to check whether PerplexityBot crawled your website, what response it received, and what to fix if your important pages are not being reached.

Why PerplexityBot Matters

PerplexityBot is Perplexity's documented web crawler.

Perplexity's crawler documentation says site owners should allow PerplexityBot in robots.txt and permit requests from its published IP ranges if they want their site to appear in search results.

Perplexity's help center also says its crawler will not index the full or partial text content of sites that disallow it through robots.txt.

That makes PerplexityBot a crawler-access question, not just a content-quality question.

If PerplexityBot cannot reach your page, your page may have a harder time becoming part of Perplexity's search and answer workflow.

The Important Distinction: Citation, Referral, Or Crawl?

Do not mix these signals together.

| Signal | What it means | What it does not prove | |---|---|---| | Perplexity citation | Your page appears as a source in an answer | When PerplexityBot crawled it | | Perplexity referral | A human clicked from Perplexity to your site | Whether other pages were crawled | | PerplexityBot request | The crawler requested a URL | Whether the page was cited later | | Status code | The response the crawler received | Whether the content was useful | | Prompt test | Perplexity mentions or misses you | Whether the crawler was blocked |

A complete workflow needs more than one signal.

For CrawlConsole, the crawler layer is the part that deserves separate tracking:

  • user agent
  • requested URL
  • timestamp
  • status code
  • redirect behavior
  • blocked requests
  • revisit patterns

That is the evidence you need before deciding whether a Perplexity visibility issue is a content problem, indexing problem, crawl-access problem, or measurement problem.

Step 1: Pick The Pages Perplexity Should Understand

Start with a short list.

Do not audit the entire site first.

Pick pages that match real search or recommendation intent:

  • homepage
  • product pages
  • pricing pages
  • comparison pages
  • documentation
  • glossary pages
  • category pages
  • high-intent blog posts
  • original research or data pages
  • use-case pages

For each page, write down why Perplexity should care.

Example:

| URL | Why it matters | |---|---| | /pricing | Buyers ask AI tools to compare vendor pricing | | /docs | Developers ask agents how to integrate a product | | /blog/perplexitybot-guide | Search teams ask how Perplexity crawler access works | | /agentic-commerce/product-search | Agents may need a product discovery path |

This keeps the audit practical.

You are not trying to prove "Perplexity likes our site." You are checking whether the right pages are reachable and understandable.

Step 2: Check Robots.txt For PerplexityBot Rules

Open:

https://example.com/robots.txt

Look for rules that mention:

User-agent: PerplexityBot

Also check broader rules:

User-agent: *

A simple allow pattern might look like this:

User-agent: PerplexityBot
Allow: /

A selective policy might allow useful public pages and block low-value paths:

User-agent: PerplexityBot
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /search
Disallow: /*?sort=
Disallow: /*?filter=
Allow: /

A full block looks like:

User-agent: PerplexityBot
Disallow: /

Be careful with platform-managed robots.txt rules.

Some CDN or bot-management features can modify robots.txt behavior or add crawler restrictions. If your app serves one robots file but the edge layer serves another, the crawler sees the edge version.

The question is not:

What do we think our robots.txt says?

The question is:

What does PerplexityBot receive when it fetches the live robots.txt file?

Step 3: Check CDN, WAF, And Bot Protection Rules

Robots.txt is only one layer.

PerplexityBot can be allowed in robots.txt and still fail because of:

  • Cloudflare bot rules
  • WAF challenges
  • IP reputation blocks
  • country blocks
  • rate limits
  • JavaScript challenges
  • security middleware
  • custom firewall rules
  • edge redirects

This is common with AI crawlers.

A human loads the page normally. Googlebot gets a clean response. But PerplexityBot receives a 403, 429, redirect loop, or challenge page.

Check whether your infrastructure treats PerplexityBot differently from normal browser traffic.

Important fields:

  • user agent
  • IP or ASN
  • URL
  • status code
  • firewall action
  • bot score
  • cache status
  • redirect target

If you use Cloudflare, Fastly, Akamai, Vercel, or another edge provider, review the security logs, not just your application logs.

The crawler may never reach your app.

Step 4: Search Logs For PerplexityBot

Now check actual requests.

Search for:

PerplexityBot

The documented user-agent string includes:

PerplexityBot/1.0

Useful log fields:

| Field | Why it matters | |---|---| | timestamp | shows when the crawler visited | | user agent | identifies PerplexityBot | | URL | shows which page was requested | | status code | shows whether the page loaded | | referrer | often empty, but useful if present | | IP/ASN | helps validate crawler source | | cache status | shows edge behavior | | WAF action | shows blocks or challenges |

Do not stop at "we saw PerplexityBot once."

You need page-level detail.

Ask:

  • Did it crawl only the homepage?
  • Did it reach product or docs pages?
  • Did it visit the new blog post?
  • Did it crawl comparison pages?
  • Did it receive a 200?
  • Did it get blocked with 403?
  • Did it hit 404 pages?
  • Did it get rate-limited with 429?
  • Did it revisit after updates?

That is the difference between crawler presence and crawler usefulness.

Step 5: Validate The Crawler Identity

User agents can be spoofed.

If the traffic volume is small and harmless, matching the user-agent token may be enough for a first pass.

If the traffic is high-volume, expensive, or security-sensitive, validate more carefully:

  • compare the IP against Perplexity's published IP ranges
  • check reverse DNS where available
  • review ASN and network ownership
  • compare behavior against expected crawler paths
  • check whether the request respects robots.txt
  • review edge security labels

Use CrawlConsole's PerplexityBot page as a crawler identity reference, then combine it with your own logs.

The crawler profile helps identify the bot. Your logs show what happened on your site.

Step 6: Check Whether PerplexityBot Reaches Fresh Content

For AI search visibility, new content matters.

After publishing a new page, check:

This is a useful post-publish workflow:

publish page -> submit sitemap -> add internal links -> monitor crawler visits -> check status codes -> update internal links if ignored

If PerplexityBot only visits old pages and never reaches fresh content, look at your discovery paths:

  • Is the new page in the sitemap?
  • Is it internally linked from relevant pages?
  • Is it buried behind JavaScript navigation?
  • Does it have a clean canonical?
  • Is it blocked by robots.txt?
  • Is the page returning a clean 200?

Post-publish monitoring is where crawler analytics become practical.

Step 7: Run Perplexity Prompt Tests Separately

After checking crawler access, run prompt tests.

Prompt tests answer a different question:

Does Perplexity currently surface, cite, or understand this topic?

Use repeatable prompts:

  • "What tools help monitor AI crawler traffic?"
  • "How can I tell if PerplexityBot crawled my website?"
  • "What is the difference between PerplexityBot and GPTBot?"
  • "How should websites handle AI crawlers in robots.txt?"
  • "Which products help ecommerce sites prepare for AI shopping agents?"

Track:

  • whether your brand appears
  • whether your page is cited
  • whether competitors appear instead
  • whether the answer uses old information
  • whether the same prompt changes after publishing or updating content

Use the Prompt Library to make these tests repeatable.

But keep the layers separate:

  • prompt tests show answer behavior
  • crawler logs show access behavior

You need both.

Step 8: Fix The Most Common PerplexityBot Problems

If PerplexityBot is not crawling useful pages, start with these fixes.

Problem: PerplexityBot is blocked in robots.txt

Review whether the block is intentional.

If you want Perplexity visibility, allow important public content and block only low-value paths.

Problem: PerplexityBot receives 403

Check CDN, WAF, and bot-management rules.

The issue may be at the edge, not in your app.

Problem: PerplexityBot only crawls the homepage

Improve internal links from the homepage and major hub pages to the pages you want crawled.

Add relevant links from older high-traffic content.

Problem: PerplexityBot reaches pages but not fresh content

Check sitemap updates, canonical tags, publish timing, and internal links.

Make sure the new page is not orphaned.

Problem: Perplexity cites competitors instead

This may not be a crawler issue.

Improve page specificity, comparison content, original examples, author credibility, and external mentions.

Crawler access gets you into the game. It does not guarantee citations.

PerplexityBot Crawl Checklist

Use this checklist for every important page:

  • Robots.txt: PerplexityBot is allowed on useful public pages.
  • Status code: the crawler receives 200, not 403, 404, 429, or a challenge page.
  • Edge rules: CDN, WAF, and bot protection are not silently blocking the crawler.
  • Crawler identity: user agent and IP source are reviewed when traffic matters.
  • Fresh content: new pages are in the sitemap and internally linked.
  • Page-level logs: PerplexityBot visits are tied to specific URLs, not just domain-level traffic.
  • Prompt tests: Perplexity answer behavior is monitored separately from crawler access.
  • Revisits: important pages are checked after updates, not only after first publish.

Where CrawlConsole Fits

CrawlConsole is useful because it separates crawler visibility from normal web analytics.

Google Analytics can show human sessions. Google Search Console can show search impressions. Perplexity referrals can show some downstream traffic. But none of those alone tell you the full crawler story.

For PerplexityBot, you want to know:

  • which URLs it requested
  • when it requested them
  • what status code it received
  • whether it reached the pages you care about
  • whether it came back after changes
  • whether it behaved differently from GPTBot, OAI-SearchBot, or ClaudeBot

Start with PerplexityBot in the Web Crawlers directory, then monitor page-level activity after publishing or updating content.

That is how Perplexity visibility becomes measurable instead of anecdotal.

Related CrawlConsole Resources

If you are building a broader AI crawler monitoring workflow, pair this PerplexityBot checklist with these CrawlConsole resources:

The Bottom Line

If you care about Perplexity search visibility, do not only ask whether Perplexity mentions your brand.

Ask whether PerplexityBot can actually crawl the pages you want Perplexity to understand.

The practical workflow is:

  1. Pick the pages that matter.
  2. Check robots.txt.
  3. Check CDN and WAF rules.
  4. Search logs for PerplexityBot.
  5. Validate crawler identity when needed.
  6. Monitor fresh content after publishing.
  7. Run prompt tests separately.
  8. Use crawler data to decide what to fix next.

Perplexity visibility starts with useful content, but it also depends on crawler access.

If the crawler cannot reach the page, the answer engine may never get the chance to cite it.