Which AI Engines Read Your Site? Now You Can Prove It

Every GEO tool will tell you how visible you are in AI answers. Very few can tell you which AI engines actually read your site, and none of them can tell you whether the crawler that read it was real.

That gap matters more than it sounds. So we built the other half: what happens on your side of the conversation, proven rather than claimed.

The problem with measuring AI visibility today

Most GEO products measure the answer side. They run prompts through ChatGPT, Perplexity and Gemini and record whether your brand shows up. It is useful, and it is noisy. When SparkToro ran about 3,000 prompts, fewer than 1 in 100 runs returned the same list of brands, and fewer than 1 in 1,000 returned them in the same order.

The tools that look at your traffic instead count crawler hits by User-Agent, sometimes checked against published IP ranges. A User-Agent is a string anyone can type. Watch any public site for a week and you will find requests that claim to be a well-known AI crawler and fail the proof. The forged “ChatGPT-User” requests we see are not reading blog posts. They ask for files like /.env, /api/.env and /.well-known/jwks.json.

A dashboard that counts those as AI engines reading your content is not measuring AI visibility. It is measuring who is willing to lie about being ChatGPT.

1. Crawled pages: verified reads, page by page

Crawled pages now answers one question: which AI engines read this page, and did they prove it?

Every page shows its reads in three columns:

• Verified: the crawler proved its identity with a Web Bot Auth signature, published IP ranges, forward-confirmed reverse DNS, or Cloudflare’s verified-bot category.
• Unproven: the request used a crawler’s name and offered no proof.
• Forged: the request used a crawler’s name and failed the proof.

Pages are ranked by verified reads, not raw hits, and only AI crawlers are counted. Googlebot, Bingbot and Applebot are included because the same fetch feeds both search and AI answers; link previews, SEO tools and HTTP clients are not.

History is kept as a daily rollup of bot traffic only, so the trend survives after detailed logs age out. Look back 7, 14, 30 or 90 days.

2. Fetched on behalf of a user

Most AI crawling is bulk: training crawlers and search indexers working through your sitemap. A smaller kind is worth more. When someone asks ChatGPT, Claude or Perplexity about something and the assistant fetches your page live to answer, the request comes from ChatGPT-User, Claude-User or Perplexity-User.

WebDecoy now logs every one of those fetches, with the page, the assistant and whether its identity checked out. It is the closest honest signal that an AI answer used your page.

Look at the forged row. Something claiming to be ChatGPT asked for an .env file. Counted by User-Agent alone, that would be one more “AI read” on your dashboard.

We are careful about what we call it. It is a fetch, not a citation. Your server can see that an assistant read the page because a person asked; it cannot see what the answer said. Anyone who tells you otherwise from server logs is guessing.

3. AI search access: are you turning away the crawlers that send visitors?

This is the one that costs sites traffic without anyone noticing.

Security layers are getting aggressive about AI crawlers, often for good reasons. Cloudflare now blocks AI agent and training crawlers by default on new domains’ ad-monetized pages. WordPress security plugins block unfamiliar bots. A robots.txt written to keep out training crawlers can easily catch OAI-SearchBot or PerplexityBot too, and those are the crawlers that put you in AI search results.

The AI search access check looks at three things for every crawler whose operator sends visitors back (OpenAI, Anthropic, Perplexity, Google, Microsoft and others):

• Your robots.txt, read by the same rules WebDecoy’s own crawler obeys, crawler by crawler.
• Your crawler policy, flagging any protected path set to refuse AI search crawlers or assistants.
• Your edge, reporting verified search crawlers and assistants that were actually turned away, and where. A finding reads like “in the last 7 days your edge refused 41 fetches by verified search crawlers on /products/*”, which is something you can act on.

Refusing training crawlers is never flagged. That is a choice, and a reasonable one. Refusing the crawlers that send you visitors is usually an accident, and the fix is one policy change away.

4. AI referrals, counted honestly

The other side of the exchange is what AI products send back. The Crawl-to-Referral Ratio compares, per AI company, how much it crawled with how many visitors it sent.

That ratio used to undercount referrals, and we would rather say so than hide it. It counted only AI visitors who also looked automated, so a site with real ChatGPT visitors could see “sent nothing back”. That is fixed. Every visit an AI product sends is now counted, by the page tag, the Cloudflare or Netlify edge sensor, the WordPress plugin or the Node SDK. When a site runs more than one of them, each day’s figure comes from the most complete one, so no visit is counted twice.

What is stored is a count: the AI product, the landing page’s path and a number. No IP address, user agent, cookie, session, query string or full referrer. There is no row per visitor.

Put next to Crawled pages, you get the whole loop per page: which AI engines read it, which assistants fetched it for a person, and how many visitors each AI product sent to it.

What we will not claim

We do not track prompts, and we do not promise citations or rankings. No study has shown that llms.txt, Content Signals or any robots.txt directive gets you cited more often. Ahrefs found that 97% of published llms.txt files got no requests at all, and most of the few requests came from SEO tools rather than AI engines.

What we do promise is evidence: which AI engines really read you, which ones you are accidentally turning away, and what they sent back. Visibility you can prove is a better starting point than visibility you have to estimate.

Get it

Everything above is live on the AI Traffic and Crawled pages screens for every WebDecoy site.

• Page tag: AI referrals counted automatically.
• Cloudflare edge sensor: crawler reads, edge refusals and AI referrals. Click Update available under Integrations → Cloudflare to get the latest build.
• Netlify extension: the same, on Netlify.
• WordPress: update to plugin 2.10 and connect to WebDecoy Cloud.
• Node.js: npm install @webdecoy/node@0.17. AI referral counting is on by default with an API key.

Start free and see which AI engines are reading your site, and which ones are only pretending to.