Cloudflare

To connect your website to CrawlConsole using Cloudflare Workers, follow these steps. This guide assumes you're already using Cloudflare as a Proxy and will guide you through the process of creating a Worker that connects to the CrawlConsole API.

Create a Worker

After you've connected your website and are using Cloudflare as a proxy, the next step is to create a Worker.

On your account page, click on "Compute (Workers)"

After clicking, you'll get to the Workers & Pages page, where there's a "Create" button. Click the "Create" button

Next, choose Start with Hello World! We'll deploy then edit the Worker code later.

Name the worker whatever you'd like. In this screenshot, I named the worker "crawlconsole-example". Click Deploy.

After that, you'll get a success page. Next, we need to setup the Environment Variables, so click on Continue to project

On the Projects page, there's to Settings > Variables and Secrets

A slider will show up on the right to create an environment variable

First, create the CRAWLCONSOLE_PROJECT_KEY as Type Text and add the value for your project. You can view your projects on CrawlConsole with this link: Projects

Next, click on "+ Add Variable" and add in the CRAWLCONSOLE_API_KEY as Type Secret which you can get from your settings page here: Account Settings

When you're done, click on "Deploy" and it will deploy your worker with these keys accessible. Note: Since we haven't added in the code yet, they won't be accessed just yet, but that's the next step.

After you've added the environment variables, click on the "Edit Code" button at the top right.

And you'll be taken to this screen where you can enter the Worker code (javascript)

Copy and paste this code into the editor:

Then click on Deploy

At this point, you should have:

• Environment Variables set for your API key and Project Key
• A Deployed Worker

The final step is to add this Worker as part of the "Routes", so that when your domain is pinged, the Worker will run async and map the LLM crawler data to your account.

On the Settings page for your Worker, go to Domains & Routes and click "+ Add"

On the menu that opens on the right, Click on "Route" then select your domain and enter your domain yourdomain.com/*

Then click on "Add Route"

Once you're done that, you should be all set!

Go to ChatGPT or Perplexity and prompt it with:

Tell me everything you know about youdomain.com and it should crawl it and ping your domain which you can see on your Analytics Page!