Skip to main content
Crescade

Can ChatGPT, Claude and Perplexity read your site?

We read your robots.txt, show which AI crawlers it lets in, quote the line that decides it, and give you a corrected file that keeps your other rules.

Free · a few seconds · no signup · we only read public pages

Secure verification will run before submission.

What it checks

  • 13 AI crawlers from OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta and ByteDance
  • The exact rule that decides access to your homepage, quoted with its line number
  • Search and assistant crawlers kept apart from training crawlers, so you can allow one and block the other
  • A corrected robots.txt that keeps your existing private paths private

We read public pages only, the same way a crawler does, and we do not store your result.

Example output · Blocked · robots.txt line 4

Disallow: / applies to OAI-SearchBot, so ChatGPT search cannot show your pages.

Try: give AI search crawlers the same rules as Googlebot.

Questions

Should I block AI crawlers?

It depends on the crawler. Training crawlers such as GPTBot and CCBot collect pages for model training; search and assistant crawlers such as OAI-SearchBot and PerplexityBot fetch pages so answers can cite you. They use separate names, so you can block training and still be cited.

Does blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether Gemini models may use your pages. Google Search crawls with Googlebot, which this setting does not touch.

My robots.txt allows everything. Why do AI tools still not see my site?

robots.txt is one gate. A firewall or CDN bot setting, such as Cloudflare's AI bot blocking, can refuse these crawlers before they ever read the file. Check those settings too.

How fast does a change take effect?

The robots.txt standard (RFC 9309) asks crawlers not to cache the file for more than 24 hours. Being picked up in AI answers takes longer, because the pages still have to be crawled and indexed.

More free checks