Test Your Robots.txt

A robots.txt file lives at the root of your domain and tells web crawlers which parts of your site they may or may not request. It uses User-agent groups with Allow and Disallow rules, plus optional Sitemap and Crawl-delay directives. It is a voluntary standard, so well-behaved crawlers respect it, but it does not enforce access control.

AI crawlers such as OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Common Crawl's CCBot, and Google-Extended read robots.txt to decide what content they may fetch for training or answer generation. To allow or block them you add a User-agent group naming the specific bot. Blocking GPTBot, for example, keeps your pages out of that crawler's data collection.

Google and most modern crawlers use the longest-match-wins rule: the rule whose path matches the most characters of the requested URL takes precedence. If an Allow rule and a Disallow rule match with the exact same path length, the Allow rule wins. This lets you block a folder with Disallow while re-opening a specific file inside it with a more specific Allow.

No. Disallow only stops compliant crawlers from fetching the page content; it does not remove an already indexed URL, and a blocked URL can still appear in results from external links. To keep a page out of the index, allow crawling and use a noindex meta tag or X-Robots-Tag header, or remove the page entirely.