Free · no account

Is your site blocking AI crawlers?

Paste a URL and see which AI crawlers can actually reach your site. We read your robots.txt and request your homepage as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more. If an answer engine is being turned away, no page is good enough to fix it.

What the checker does

robots.txt, parsed

We read your robots.txt and show, per crawler, whether it is allowed, partially blocked, or blocked outright.

The edge, probed

For crawlers with a public user agent, we request your homepage as that crawler and catch a CDN or firewall turning it away.

Sitemap and llms.txt

We also flag whether you declare a sitemap and whether you have an llms.txt, the two other files an engine looks for.

Questions about AI crawlers

Usually by accident. A robots.txt rule, or a CDN, firewall, or bot-protection layer like Cloudflare, can turn away crawlers such as GPTBot, ClaudeBot, or PerplexityBot while your site looks fine in a normal browser. If a crawler can't reach you, that engine can't read or cite you at all.

Crawlers in? Now be worth quoting.

Letting crawlers in is step one. Score how citable your page is with the free AEO checker, then list your product for a permanent, structured page built to be cited.

Run the AEO checkerList your product free