Control AI Crawlers with Robots.txt

A robots.txt file is a text file placed at the root of your website that tells web crawlers and bots which pages or sections they are allowed or not allowed to access. It follows the Robots Exclusion Protocol standard.

It depends on your goals. Blocking AI crawlers prevents them from training on your content, which protects your intellectual property. However, it may reduce your visibility in AI-powered search results and chatbot responses. Consider your priorities before deciding.

GPTBot is OpenAI's web crawler that collects data used for training AI models. It is separate from ChatGPT-User, which is the crawler used when ChatGPT browses the web in real-time to answer user queries. You can block one without blocking the other.

No, Google uses Googlebot for search rankings, not AI-specific crawlers. However, blocking Google-Extended may prevent your content from appearing in Google's AI Overviews (formerly SGE). Your traditional search rankings will remain unaffected.

Upload the generated robots.txt file to your website's root directory so it is accessible at example.com/robots.txt. Most web hosts allow you to upload files via FTP, file manager, or your CMS settings. Make sure the file is named exactly 'robots.txt' and is in the root, not a subdirectory.