A business can publish excellent content, earn strong Google rankings, and still be missing from ChatGPT, Claude, and Perplexity answers for a reason nobody on the team has looked at: a few lines in a text file. Many sites block AI crawlers without anyone deciding to, through an old plugin setting, a copied template, or a security service switched on by default. The result is quiet. No error appears, traffic reports look normal, and competitors collect the citations instead.
Checking takes minutes, and fixing an accidental block is usually just as quick.
Three Different Jobs AI Bots Do on a Website
Not every AI bot visits for the same reason, and that distinction shapes every decision that follows. Some bots collect content that may be used to train future models. Others build a search index so an assistant can cite pages in its answers. A third group fetches a page in real time because a user asked a question that needs it. A clear approach to AI crawlers separates these jobs, because blocking a training bot has very different consequences from blocking a search bot.
A site can refuse training access and still remain fully visible in AI search. Blocking every AI bot with one broad rule removes both at once, and that’s where most visibility losses begin.
The User Agents Worth Knowing
Each AI company publishes the names its bots use, called user agents. These are the names that robots.txt rules refer to.
| User agent | Company | Main job | Effect of blocking it |
|---|---|---|---|
| GPTBot | OpenAI | Collects content for model training | Opts out of training, with no effect on ChatGPT search |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search | Pages can drop out of ChatGPT search answers |
| ChatGPT-User | OpenAI | Fetches pages when a user asks | OpenAI notes robots.txt rules may not apply to these user requests |
| ClaudeBot | Anthropic | Collects content for model training | Opts future content out of training |
| Claude-SearchBot | Anthropic | Indexes content for search results | Can reduce visibility in Claude’s search results |
| Claude-User | Anthropic | Fetches pages when a user asks | Stops Claude retrieving the page for user questions |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | Pages can disappear from Perplexity citations |
| Google-Extended | Controls use of content for Gemini training and grounding | No effect on Google Search, including AI Overviews |
Google’s case confuses many teams. AI Overviews are part of Google Search and rely on the regular Googlebot, so blocking Google-Extended doesn’t remove a site from them. Blocking Googlebot would, but it would also remove the site from Google Search entirely.
Reading robots.txt in Five Minutes
Open a browser and visit the domain followed by /robots.txt. The file is public, short, and readable without any technical background. Three patterns deserve attention.
The first is a blanket block. A rule like the one below tells every bot to stay away from the entire site. It sometimes survives from a staging site that was launched without cleanup.
User-agent: *
Disallow: /The second is a named block. Lines that name GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, or similar agents followed by Disallow: / shut those bots out completely. Some were added on purpose. Others arrived through a plugin or a template that someone copied years ago.
The third is a mixed setup that blocks search bots while allowing training bots, often the reverse of what the business wants. A balanced version for a company that wants AI search visibility but prefers to keep content out of training might look like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Partial blocks are easy to miss as well. A rule such as Disallow: /blog/ under a named AI bot hides an entire section while leaving the homepage open, so a quick glance suggests access is fine. Read every Disallow line beneath each AI user agent, not only the ones ending in a single slash, and confirm that the sections carrying the most useful content, such as guides, product pages, and pricing, stay open.
Anthropic also notes that rules must be placed in robots.txt at the top level of each subdomain, so a blog or shop on a separate subdomain needs its own check. SEO Sets reads robots.txt against a maintained list of documented AI user agents and separates the bots that fetch pages for answers from those that collect training data, which saves teams from tracking every new user agent by hand.
Hidden Blocks Outside robots.txt
A clean robots.txt file doesn’t guarantee access. Several blocks happen further upstream, before a bot ever reads the file.
Content delivery networks and firewalls are the most common source. In July 2025, Cloudflare began blocking AI bots by default for new domains on its network, asking site owners to choose whether to allow them. Sites that signed up after that change, or that switched on bot protection settings, may block AI search bots even though robots.txt says nothing about them.
Security plugins, hosting firewalls, and rate limits can have the same effect. A bot that receives a 403 error or a challenge page can’t read the content, so it can’t cite it. Anthropic advises against blocking its bots by IP address for a related reason: a blocked bot can’t even read robots.txt to learn what it’s allowed to do.
The only reliable way to catch these blocks is checking server or CDN logs. Filter requests by the user agents in the table above and look at the response codes. Steady 200 responses mean the bots are getting through. Repeated 403 or 429 responses mean something upstream is turning them away.
Choosing a Policy That Fits the Business
The right setup depends on how a business makes money from its content.
A software company or local service business usually gains from maximum AI search visibility, since a citation can put it on a buyer’s shortlist. Allowing search and user agents makes sense here, and the training decision can be made separately. Online stores sit in a similar position, since product and comparison prompts are where assistants recommend specific brands.
A publisher that earns revenue from original reporting or paid content may weigh things differently. Blocking training bots while allowing search bots keeps the brand present in AI answers without contributing content to model training.
Whatever the choice, it should be written down and checked after every site migration, CDN change, or security update, because those are the moments when accidental blocks appear. SEO Sets can verify a change by re-reading the live site afterward, which confirms whether a fix actually went live rather than assuming it did.
Questions Site Owners Ask About AI Bot Access
Does blocking GPTBot remove a site from ChatGPT search?
No. GPTBot covers training. ChatGPT search relies on OAI-SearchBot, which is controlled by its own rule in robots.txt.
How quickly do robots.txt changes take effect?
OpenAI says its systems adjust to robots.txt changes in about 24 hours. Other providers recrawl on their own schedules, so allow a few days before checking results.
Does an llms.txt file replace robots.txt?
No. llms.txt is a proposed convention for guiding AI tools to key content, and adoption by the major engines remains unproven. Access is still controlled by robots.txt and server settings.
The Takeaway
Missing AI citations often trace back to a technical decision that nobody remembers making. Reviewing robots.txt, understanding which AI crawlers handle training and which handle search, and checking server logs for upstream blocks gives any team a clear picture of whether AI assistants can actually reach its pages.
Access is only the first step toward appearing in AI answers, but without it, nothing else can work. SEO Sets checks AI bot access as part of a wider readiness review and tracks how often major assistants cite a site over time. Open your robots.txt file today, compare it with the table above, and ask your hosting or CDN provider whether AI bot blocking is switched on.
