Skip to main content
Anyone can send a request that claims to be an AI crawler: the user agent is a plain header that is easy to fake. So the Agent Edge Network only treats a crawler as genuine when the request also comes from an IP address that the AI vendor publishes for that crawler. Only verified crawlers can receive the optimized version of your page.

​
Verification levels

Each AI crawler visit carries one of two levels. Agent Analytics shows them as Verified and Unverified in Recent activity.

​
IP_VERIFIED

Shown as Verified. The request identifies itself as a known AI crawler and comes from one of that vendor’s published IP ranges. Result: eligible for the optimized version, depending on your serve mode and the crawler’s purpose.

​
UA_ONLY

Shown as Unverified. The request identifies itself as a known AI crawler, but its IP address is not in that vendor’s published ranges. Result: always gets your normal page. The visit is still logged, so you can see it in Agent Analytics.
Requests that don’t identify as a known AI crawler, including all your human visitors, are ignored: they get your normal page and are not reported.

​
The crawler list

Bluemoon maintains the list of known AI crawlers and their published IP ranges. You don’t manage it yourself, and your install picks up updates without a redeploy.

​
Purpose classes

Every known crawler has a purpose class describing why it fetches your page. Agent Analytics counts visits by purpose under Why they came: The drop-ins enable the two answer-engine purposes by default:
A crawler gets the optimized version only if it is verified and its purpose is enabled. Training crawlers and unknown agents never get it, even when their IP address checks out.

​
Search crawlers are never served

Googlebot, Bingbot and Applebot always get your normal page, whatever your settings. Their visits are not logged in Agent Analytics. See Cloaking firewall for why.

​
Content parity

A verified crawler gets the same information as your page, in a format that is easier for answer engines to read. The optimized version never adds claims your page doesn’t make. See Optimized pages.

​
Troubleshooting unverified hits

It may be crawling from IP addresses that are not yet on Bluemoon’s list. Contact us with the crawler name and a few example visits, and we can look into it.
Your install checks the visitor’s IP address. If your site sits behind a proxy or load balancer, make sure it passes the original client IP to your app (for example in the X-Forwarded-For header). Otherwise every crawler looks like it comes from your proxy and can’t be verified.
The Next.js middleware, the Cloudflare Worker, the WordPress plugin, and the Node.js and PHP snippets verify IPv4 addresses only. A crawler that reaches your site over IPv6 is logged as Unverified (UA_ONLY) and gets your normal page.

Serve modes

How to go from logging verified crawlers to serving them.

Quickstart

Connect a domain, get your edge token, and install the Agent Edge Network.