Auditing AI Crawler Access in robots.txt

In this lesson

Module 3 looked at the sources behind AI answers. For your own site to be one of them, it first has to be readable. This module is about how open and readable your site is to AI systems, and robots.txt is the first step.

What robots.txt is

robots.txt is a plain text file at the root of your site (for example acmehelpdesk.example/robots.txt). It tells crawlers which parts of the site they may visit. Each crawler applies the rules itself. The file isn't a security measure and doesn't lock pages away.

The file is made of groups. Each group starts with one or more User-agent: lines, followed by Disallow: and Allow: rules.

The rule people miss most often: a crawler that has a group under its own name follows only that group. The User-agent: * group applies only to crawlers without a named group. This is defined in the robots.txt standard (RFC 9309). So a restriction in your * group doesn't carry over to a crawler that has its own group.

Two more details:

• Disallow: / closes the whole site. Disallow: /app/ closes only that path.
• robots.txt is per host. docs.acmehelpdesk.example needs its own file.

Documented AI crawlers (as of October 2026)

The names below come from the providers' own documentation. Names and roles can change. Check the provider's current documentation before you decide anything.

OpenAI

• GPTBot: crawls content that may be used to train OpenAI's models.
• OAI-SearchBot: crawls to surface sites in ChatGPT's search features. According to OpenAI, sites that block it won't be shown in ChatGPT search answers, though they can still appear as navigational links.
• ChatGPT-User: visits a page when a user asks for something in ChatGPT. Because the user starts the request, robots.txt rules may not apply.

Anthropic

• ClaudeBot: collects content that may contribute to model training.
• Claude-SearchBot: crawls to improve the quality of search results.
• Claude-User: fetches pages for a Claude user's question.

Perplexity

• PerplexityBot: crawls to surface and link sites in Perplexity search results. According to Perplexity, it isn't used for model training.
• Perplexity-User: fetches pages on a user's request. According to its documentation, it generally ignores robots.txt rules.

Google

• Google-Extended: not a separate crawler, but a control token used in robots.txt. It manages whether content may be used to train future Gemini models and for some grounding uses (basing answers on sources). According to Google, it doesn't affect inclusion or ranking in Google Search.
• For AI features inside Search, such as AI Overviews, Google points to the normal Googlebot rules for Search crawling.

In practice, crawlers serve three purposes: training, search and user requests. Which ones you allow is a business decision. This course won't tell you to "open everything". What matters is that the decision is deliberate.

Sample file (representative)

The file below is fictional. "Acme Helpdesk" is a fictional product.

User-agent: *
Disallow: /app/
Disallow: /login

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /docs/

Sitemap: https://acmehelpdesk.example/sitemap.xml

Reading it line by line

• The * group: The app and login pages are closed. That's a common, sensible setup.
• GPTBot and ClaudeBot: Consecutive User-agent: lines share one group, so both training crawlers are blocked from the whole site. That affects the training-data route from Module 1. If it was intended, that's fine.
• OAI-SearchBot: Blocked from the whole site. According to OpenAI, that means the site won't appear in ChatGPT search answers (it can still appear as a navigational link), which is probably not intended. Lines like this often come from copied "block all AI bots" lists.
• PerplexityBot: Only the help docs are blocked. Since this isn't a root block, an audit tool may not count it as a block at all. Yet for a question like "How do I set up email forwarding in Acme Helpdesk?", the most useful pages are closed. And because PerplexityBot has its own group, it doesn't follow the * group, so /app/ and /login are open to it.
• Claude-SearchBot: It has no group of its own, so it follows *. Everything except the app and login pages is open to it.
• Google-Extended: No group of its own; the * group applies, so everything outside /app/ and /login is allowed.

Does the file itself load?

How the file loads matters as much as its rules. According to RFC 9309:

• If robots.txt isn't found (a 4xx response), crawlers may access the site freely.
• If robots.txt returns a server error (a 5xx response), crawlers must treat the whole site as closed.

Some blocks also live outside robots.txt. Firewall and bot-protection settings can block a crawler that robots.txt allows. Ask your technical team about those settings too.

Corrected draft

Say Acme Helpdesk wants to block training crawlers only and stay open to search crawlers. The draft could look like this:

User-agent: *
Disallow: /app/
Disallow: /login

User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /app/
Disallow: /login

Sitemap: https://acmehelpdesk.example/sitemap.xml

This is an example. Adapt it to your own decision and check it with your technical team before you deploy it.

Provider documentation: OpenAI, Anthropic, Perplexity, Google.

Summary

• robots.txt is made of groups. A crawler with its own group doesn't follow the * group.
• AI crawlers serve three purposes: training, search and user requests. Blocking any of them should be a deliberate business decision.
• Blocking a search crawler can keep you out of that engine's search answers.
• Don't look only for root blocks. A path block on an important section matters too.
• Names change. Always check the provider's current documentation.
• A robots.txt that returns 5xx, and bot-protection settings, can also shut crawlers out.