--- title: "OAI-SearchBot vs GPTBot vs ChatGPT-User: Search Visibility and Crawl Controls" canonical: https://snezzi.com/blog/oai-searchbot-vs-gptbot-vs-chatgpt-user/ source: https://snezzi.com/blog/oai-searchbot-vs-gptbot-vs-chatgpt-user/ published: 2026-09-10 modified: 2026-10-16 author: "Nikunj Thakkar" category: "AI Visibility" --- > Canonical page: https://snezzi.com/blog/oai-searchbot-vs-gptbot-vs-chatgpt-user/ # OAI-SearchBot vs GPTBot vs ChatGPT-User: Search Visibility and Crawl Controls OAI-SearchBot controls whether a site can appear in ChatGPT search answers. GPTBot controls whether content may be used for model training. ChatGPT-User handles user-initiated page visits. Search and training preferences use independent robots.txt directives. User-initiated visits are different: robots.txt may not apply to ChatGPT-User requests. SEO and growth teams often need to separate search visibility from training preferences. The three agents serve distinct roles, so a single robots.txt rule rarely fits every goal. Official documentation from OpenAI clarifies the differences and the practical steps required to apply them. This guide walks through each agent, a side-by-side comparison, an example robots.txt pattern, a verification procedure, and a measurement approach that keeps retrieval, citations, brand mentions, and conversions separate. ## The three agents serve different purposes OpenAI operates three distinct user agents that interact with websites. Each one addresses a separate function in how ChatGPT discovers, learns from, or displays content. Treating them as one traffic source is the most common mistake teams make when they first open robots.txt. OAI-SearchBot manages search discovery. It determines whether pages become eligible for inclusion in ChatGPT search results. GPTBot manages training data selection. It decides whether content may enter future model training runs. ChatGPT-User manages on-demand page fetches triggered by individual users inside the product experience. Search discovery and training use have independent robots.txt settings; user-initiated retrieval has separate access considerations. [OpenAI's bot documentation](https://developers.openai.com/api/docs/bots) describes each agent with its own purpose and control surface, which is why a publisher can allow search discovery while blocking training use, or block search while still permitting user-triggered visits. The choice depends on the specific outcome the site owner wants to achieve, not on a default “allow all” or “block all” posture. In practice, that means three separate decisions rather than one. You decide whether you want eligibility for ChatGPT search answers. You decide whether new crawls of your public pages may feed future training. You decide how you feel about user-initiated fetches when someone asks ChatGPT to open or summarize a specific URL. Mixing those decisions into a single User-agent block creates side effects you did not intend, and it makes later debugging harder because you cannot tell which preference caused a change in behavior. Write the policy down before you edit the file. Note the business reason for each allow or disallow line. That short record becomes the baseline when someone later asks why search visibility moved, why training preference changed, or why a user-triggered fetch still succeeded after a crawl rule was tightened. ## Agent comparison table Use the table below as a decision aid before you edit robots.txt. Read across each row for purpose, control, blocking consequence, and the limitation that still applies after you act. The rows mirror how OpenAI's bot documentation separates search discovery, training preference, and user-initiated fetches into distinct agents. | Agent | Purpose | Applicable control | Consequence of blocking | Limitation | |----------------|----------------------------------|-----------------------------|------------------------------------------|-------------------------------------------------| | OAI-SearchBot | Search discovery | robots.txt User-agent | Prevents inclusion in ChatGPT search answers | Navigational links may still appear | | GPTBot | Model training | robots.txt User-agent | Content excluded from future training | Does not affect existing trained knowledge | | ChatGPT-User | User-initiated visits | Site access controls; robots.txt may not apply | A robots.txt rule is not a reliable block for user-triggered requests | Use OAI-SearchBot for search opt-out | Search and training preferences are independent. ChatGPT-User is different: user-triggered visits may not follow robots.txt, so its row should not be read as another automatic crawler setting. Two reading tips help when you share this with legal, security, or content stakeholders. First, the “consequence of blocking” column describes the intended effect of the applicable control for that agent, not a promise that every mention of your brand disappears from every answer surface. Second, the “limitation” column is where teams usually under-invest. Navigational links, prior training knowledge, and user-triggered fetches can still produce brand exposure even when a crawl path is closed. Plan measurement and expectations around those residual paths instead of assuming a single robots.txt line erases all presence. If your organization needs a default starting point, map each row to an owner. Search discovery usually sits with SEO or growth. Training preference often sits with legal or policy. User-initiated visits may involve product, support, or security if those fetches hit authenticated or sensitive paths. Clear ownership reduces the chance that one team’s disallow rule silently undoes another team’s visibility goal. ## OAI-SearchBot distinguishes search-answer inclusion from navigational links OAI-SearchBot handles search discovery. When allowed, pages become eligible for citation inside ChatGPT search answers. When blocked, the agent stops retrieving content for those answers. That is the control surface for teams whose primary question is whether their public pages can show up as sources in ChatGPT search. OpenAI's bot documentation states that opting out prevents inclusion in ChatGPT search answers, although navigational links can still appear. This distinction matters because some brand references may continue through other pathways even after the search directive is set to disallow. A user who already knows your domain, or an answer that points to a known destination without quoting your page body, is not the same event as your content being retrieved and cited as supporting material. Allowing the agent creates eligibility. It does not guarantee retrieval, a brand mention, ranking, citation, or revenue. Many other factors sit between “crawl allowed” and “your URL appears in a given answer,” including content quality, topical fit, freshness, competing sources, and how the prompt is phrased. Treat allow as a necessary condition for search-answer inclusion from your site, not a sufficient one. OpenAI's bot documentation also notes that search systems may take approximately 24 hours to adjust to a robots.txt change. That window is an operational expectation for crawl-policy propagation, not a service-level promise that a citation will appear inside a day. If you flip from disallow to allow, wait through that adjustment period before you judge whether discovery is working. If you flip from allow to disallow, use the same window before you conclude that search retrieval from your pages has stopped. For SEO and growth teams, the practical workflow is straightforward. Confirm OAI-SearchBot is allowed if ChatGPT search eligibility is a goal. Confirm it is disallowed if the goal is to keep page content out of those answers. Then measure with unbranded prompts and separate metrics, because navigational links and third-party descriptions of your brand can still surface even when your own pages are not being pulled into the answer body. ## GPTBot separates training preference from search visibility GPTBot concerns content that may be used for model training. OpenAI's bot documentation treats its robots.txt setting as independent of the search setting. That independence is what lets publishers express a training preference without automatically changing ChatGPT search eligibility, and the reverse. A site owner can therefore allow OAI-SearchBot while disallowing GPTBot. This combination permits search visibility without contributing new content to training runs. The reverse combination is also possible if a publisher prefers to block search discovery but still permit training use. Neither combination is a universal standard. Each is a policy choice that should match how the organization thinks about distribution, competitive risk, and long-term model exposure. The training preference does not retroactively remove knowledge already present in existing models. Blocking GPTBot affects only future training cycles that respect the current robots.txt file. Teams sometimes expect a disallow rule to erase prior brand facts, product descriptions, or public statements from every model response. That expectation overstates what a crawl directive can do. Historical training data, third-party coverage, and user-supplied context can still shape answers even when new crawls of your site are blocked for training. When you brief stakeholders, keep the language precise. Say that disallowing GPTBot signals that you do not want your site’s content used for future training that honors robots.txt. Do not say that blocking GPTBot removes every mention of the brand from ChatGPT, or that it is the same control as OAI-SearchBot. Those two claims are common sources of false confidence and of mis-configured files. If legal and SEO disagree, separate the tickets. Search eligibility is an OAI-SearchBot decision. Training preference is a GPTBot decision. Resolving them on different timelines is fine, as long as the live robots.txt file reflects the current approved state for each agent and someone owns the next review date. ## ChatGPT-User covers user-triggered visits and is not the search setting ChatGPT-User handles user-initiated page visits. OpenAI's bot documentation positions it as neither an automatic crawler nor the search opt-out control. Robots.txt directives may not apply to those visits in the same way they apply to scheduled crawlers, which is why teams that only edit crawl rules still see fetches when a person explicitly asks the product to open a page. Because ChatGPT-User responds to direct user requests, blocking it does not serve as a substitute for the OAI-SearchBot directive. Site owners who want to limit search inclusion must target the correct agent rather than assuming one rule covers all OpenAI traffic. If your logs show ChatGPT-User hits after you disallowed OAI-SearchBot, that pattern is consistent with user-triggered behavior, not proof that the search opt-out failed. Think about ChatGPT-User in the same category as other on-demand fetch paths. A person can paste a URL, ask for a summary of a public page, or follow a link that causes a retrieval. Those events are tied to a session and an intent, not to a background discovery crawl. Security and bot-management teams should still review how those requests are handled at the CDN and application layer, especially for paths that require authentication, leak internal detail, or are expensive to render. That review is complementary to robots.txt, not a replacement for the search and training directives. A useful internal test is to ask three questions in order. First, do we want eligibility in ChatGPT search answers from our own pages? That is OAI-SearchBot. Second, do we want new site content available for future training? That is GPTBot. Third, how should we treat explicit user fetches of public URLs? That is ChatGPT-User, plus whatever WAF, auth, and caching rules you already apply to human traffic. Answering them in that order keeps the search setting from being accidentally overloaded onto the user-agent that is not designed for search opt-out. Document the difference in your runbooks so on-call engineers do not “fix” search visibility by tightening ChatGPT-User alone. The fix for search inclusion is the OAI-SearchBot rule, verified live, with log checks against published IP ranges when you need stronger identity confidence. ## Example robots.txt that allows OAI-SearchBot and disallows GPTBot A publisher that wants search discovery while declining training use can apply the following pattern. Treat it as one supported option grounded in how OpenAI's bot documentation keeps search and training controls independent, not as a mandate for every site. ``` User-agent: OAI-SearchBot Allow: / Disallow: /private/ User-agent: GPTBot Disallow: / User-agent: * Disallow: /private/ ``` This configuration allows OAI-SearchBot across the public site, disallows GPTBot sitewide, and requests that both search and generic crawlers avoid a private path. Robots.txt is public guidance, not access control; private content still needs authentication. Other publishers may prefer different combinations based on their content strategy and risk tolerance. Some will allow both agents. Some will disallow both. Some will allow training on a documentation subdomain while disallowing it on pricing or research sections. The example simply illustrates one coherent approach for the common “visible in search, not used for new training” goal. Before you ship a change, place the file at the site root, serve it as plain text, and avoid accidental no-cache or auth walls that hide it from crawlers. Keep group order readable for humans. Put the most specific OpenAI agent blocks where operators will see them first, and avoid duplicate User-agent sections that disagree with each other. Conflicting Allow and Disallow lines for the same agent are a frequent source of “we thought we blocked it” incidents. If you maintain multiple brands or locales on one domain, decide whether the same training and search preferences apply everywhere. A single root robots.txt applies domain-wide unless you operate separate hosts. When preferences differ by product line, you may need host-level splits or path-level rules that your legal and SEO owners both accept. Path-level rules still require careful testing, because a broad Disallow on GPTBot combined with a narrow Allow can be misread by operators who only scan the first block. After deployment, do not stop at the editor preview. Fetch the public URL with a simple HTTP client, confirm the status code, and confirm the body matches what you approved. Then move into verification against logs and IP ranges so you are not trusting a user-agent string in isolation. ## Verification procedure for robots.txt changes After updating robots.txt, confirm that the file is publicly accessible and correctly formatted. Check the live response at the root of the domain to ensure no CDN or WAF layer is blocking the file, rewriting it, or serving a stale cached copy to some regions. Work through the checks in a fixed order so you can see where a failure starts: 1. Request `https://www.example.com/robots.txt` (and the apex host if it answers separately) and confirm a 200 response with the expected body. 2. Confirm Content-Type is suitable for a robots file and that the body is not an HTML interstitial, captcha page, or soft 404. 3. Review CDN and WAF rules for bot categories that might intercept OpenAI agents differently from your browser session. 4. Compare server access logs to the published OpenAI IP ranges for the agents you care about. OpenAI publishes range files such as [searchbot.json](https://openai.com/searchbot.json) and [gptbot.json](https://openai.com/gptbot.json) for this kind of verification. 5. Spot-check that disallowed paths are not being returned as full page bodies to the agents you intended to block, and that allowed paths remain reachable. A claimed user-agent string alone does not constitute identity verification. Any client can send `GPTBot` or `OAI-SearchBot` in a header. Pair the string with network identity, timing, and request patterns. When something looks wrong, capture the raw request metadata, the matching IP range entry, and the robots.txt bytes served at that moment. That bundle is what you need for a clean incident review. Also verify rendered public content, not only the HTML source. If key facts live behind tabs, client-side renders, or blocked assets, discovery quality can suffer even when the crawler is allowed. Eligibility to fetch is not the same as a clean extraction of the page you think you published. Our [AI crawlability guide](/blog/ai-crawlability-for-generative-search-a-step-by-step-guide/) covers this rendering check in more detail. If your primary templates depend on JavaScript for core copy, test the publicly rendered output the way a simple fetcher would see it, then decide whether server-rendered summaries are required for the pages you most want cited. Finally, record the timestamp of the robots.txt change. OpenAI's bot documentation notes that search systems may take approximately 24 hours to adjust. Build that delay into your verification calendar so you do not declare success or failure on an incomplete propagation window. ## Measure change with a stable unbranded prompt set Track outcomes by testing a fixed set of unbranded prompts over time. Unbranded prompts matter because branded queries mostly test whether the system already knows your name, not whether your pages are being retrieved as supporting sources for category problems your buyers actually type. Build a small prompt set that mirrors real buyer language. Keep the wording stable from week to week. For each prompt, record four separate fields: whether your content appears to have been retrieved, whether a citation to your URL appears, whether the brand is mentioned without a citation, and whether any referred session converts in your analytics. Keep these metrics separate. Retrieval does not equal citation. Citation does not equal brand mention. Brand mention does not equal referred conversion. Separate tracking prevents over-attribution when evaluating the effect of robots.txt changes. Run the set before you change robots.txt, again after the approximate adjustment window OpenAI describes for search systems (about 24 hours), and on a steady cadence after that. If you allow OAI-SearchBot for the first time, you are watching for eligibility to turn into retrieval and citation on prompts you already selected. If you disallow it, you are watching for citations that depended on your pages to fall off, while remembering that navigational links and non-site sources can still produce brand mentions. Avoid mixing training preference experiments into the same analysis unless you have a clear hypothesis. GPTBot rules change future training use. They are not a direct dial for next week’s ChatGPT search citations. Conflating the two will make your charts look noisy and your stakeholders skeptical. Store the prompt wording, country, engine, run date, answer and cited URLs together so later comparisons use the same conditions. Share results in the same vocabulary you used for the original decision. Report eligibility, retrieval, citation, mention, and conversion as different rows. That habit keeps executives from reading a mention-rate chart as if it were proof of crawl policy success, and it keeps SEO teams from promising citation lift that robots.txt alone cannot deliver. ## Frequently Asked Questions ### Does blocking GPTBot block search? No. GPTBot controls training use only. OAI-SearchBot controls search inclusion. The two directives remain independent, so a training disallow does not opt you out of ChatGPT search by itself. ### Can a brand still appear through other sources? Yes. Navigational links and other pathways may continue to surface references even after OAI-SearchBot is blocked. Prior knowledge and third-party sources can also mention a brand without fetching your pages. ### Does allowing search guarantee citations? No. Allowing the agent creates eligibility only. Retrieval, citation, brand mention, and conversions are separate outcomes and are not assured by an Allow rule. ## Practical next steps for visibility control Review the current robots.txt file against the documented agent purposes. Decide which combination of directives matches the site’s goals for search visibility and training use. Write that decision down with an owner and a review date so the next change is intentional. Apply the chosen rules on the public root file, then verify the live response, CDN and WAF behavior, and log evidence against published OpenAI IP ranges. Do not trust a spoofable user-agent string on its own. Monitor results through a stable unbranded prompt set, and keep content retrieval, citations, brand mentions, and referred conversions in separate columns so you can see what actually moved. Adjustments can be made as needed once the approximate 24-hour propagation window passes. If search eligibility is the goal, confirm OAI-SearchBot is allowed and that your strongest pages render cleanly without depending on blocked assets. If training decline is the goal, confirm GPTBot is disallowed without casually disabling search at the same time. If user-triggered fetches are the concern, handle them as a distinct path with the right application controls rather than overloading the search opt-out. The durable advantage is clarity. Three agents, three decisions, one verifiable file, and measurement that matches what robots.txt can and cannot change.