What Should Robots.txt Say About AI Crawlers? 3 Copy-Paste Templates

Your robots.txt should say, token by token, which AI jobs you accept: training, AI search, and fetches a person asks for. Most AI robots.txt examples circulating miss that structure; they are copy-paste lists with misspelled tokens, rules that conflict with a User-agent: * group, and Googlebot blocked by accident. This page is the implementation layer: the tokens grouped by job, three clean templates, and the syntax rules that decide whether your file does what it says. Which posture to pick is a business decision, made bot by bot in the robots.txt decision framework; this page assumes you decided and gets the file right.

What should your robots.txt say about AI crawlers?

It should name each AI crawler by its job and allow or block it on purpose, because the jobs carry different costs: blocking training crawlers costs no search visibility, blocking AI-search crawlers takes you out of those engines' answers, and some user-triggered fetchers do not follow the file at all. Each operator documents its own split, and the token spellings below are verbatim from that documentation, read 2026-10-14:

Sources: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl and Bing's guidelines, which list blocking Bingbot in robots.txt among the things that can keep content out of Bing's index and grounding results. What each job means for the blocking decision is the subject of GPTBot vs OAI-SearchBot, Anthropic's three crawlers, and the Perplexity agents.

What does the allow-everything template look like?

It contains no AI groups at all. Under RFC 9309, a token with no matching group and no * restriction may crawl, so consent is expressed by silence:

# Template 1 — allow all crawlers (our posture, all 3 builds)
User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

This is what we run across our 3 production builds [our data], and the honest caveat is that it is a posture for sites that earn by being retrieved and cited. There is nothing clever to add: an empty Disallow: in the * group states the policy explicitly, and the Sitemap line is there because the file is fetched constantly anyway.

What does the block-training-only template look like?

One group per training token, with the search and user-fetch tokens deliberately absent:

# Template 2 — block training, keep AI-search crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Corpus bots: open archive and undocumented collector
User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

The logic comes straight from the operators' docs. OpenAI documents GPTBot as its training crawler and OAI-SearchBot, absent here, as the index behind ChatGPT search, and says each setting is independent. Google-Extended and Applebot-Extended are control tokens that never fetch a page: disallowing them tells Google and Apple not to train on what their existing crawlers fetch, and both companies say search inclusion is unaffected. The corpus groups are optional and stricter: CCBot feeds an open archive with diffuse downstream training uses, and Bytespider publishes no purpose statement at all.

Cloudflare's managed list, documented in October 2026, also disallows Amazonbot and meta-externalagent. If you follow the same posture, check those operators' own docs before copying the tokens.

What does the block-all template look like?

Every documented AI token gets a group, and Googlebot, bingbot and Applebot deliberately do not, because all three feed classic search:

# Template 3 — block all documented AI crawling (needs a CDN backstop)
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow:

Sitemap: https://www.example.com/sitemap.xml

Know what this file can and cannot do before shipping it. The documented training and search crawlers state they honor robots.txt, so those groups are effective on the operators' own terms. The ChatGPT-User and Perplexity-User groups are signals: OpenAI says robots.txt rules "may not apply" to user-initiated fetches, and Perplexity says its fetcher "generally ignores" them, so a genuinely complete block requires CDN or WAF rules keyed to the IP ranges OpenAI, Anthropic and Perplexity publish in the same documentation that defines the tokens. This template also costs you answer-engine visibility by design; if that surprises you, the decision page, not this one, is where to start.

What syntax mistakes break AI robots.txt files?

The group-matching rule causes nearly all of them. Under RFC 9309, a crawler obeys the group that names its product token, merging duplicate groups for the same token, and falls back to User-agent: * only when no group names it. So once GPTBot has a group, your * rules say nothing to GPTBot, in either direction. The two recurring bugs: a strict * group that everyone assumes covers AI bots which each have their own permissive group, and the reverse, where adding one AI-specific group silently exempts that bot from sitewide path restrictions it used to inherit.

Four smaller rules round out the checklist. Rules never combine across different tokens' groups, so write each group as if it were the only one the bot will read, because it is. Crawlers must be able to parse at least 500 kibibytes of robots.txt per RFC 9309, so a long per-bot file costs nothing. Crawl-delay is not in the RFC: Anthropic and Common Crawl document honoring it, Apple says Applebot does not, and OpenAI's and Perplexity's crawler pages don't mention it. And Apple documents a fallback worth knowing: if your rules don't mention Applebot but do mention Googlebot, Applebot follows the Googlebot rules.

Last, verify the served file, not the repo copy. We found a CDN-managed robots.txt silently overriding a build's allow-everything file on a production zone [our data]; one curl https://yourdomain.com/robots.txt compared byte-for-byte against your source is the cheapest audit in this whole stack.

How long does a robots.txt change take to work?

About a day, for the operators that say. OpenAI says search results can take about 24 hours to reflect a robots.txt update, Perplexity says changes may take up to 24 hours, and RFC 9309 says crawlers should not use a cached copy of the file for more than 24 hours unless it is unreachable.

Training opt-outs only look forward. Anthropic describes a ClaudeBot block as excluding the site's "future materials" from training, and Common Crawl's published archives keep what they already collected, so robots.txt cannot reach data gathered before the change.

What can't robots.txt control?

Three things, each with its own lever. First, Google's AI Overviews and AI Mode: since August 31, 2026, leaving them without leaving Search is the Search generative AI control in Search Console, not a robots.txt line (Google), and blocking Googlebot would remove you from Search entirely.

Second, user-triggered fetchers that do not follow the file, which only a CDN or WAF rule stops. Third, your CDN itself: a CDN can prepend its own rules to your robots.txt or block crawlers at the edge whatever your file says. On Cloudflare, since September 15, 2026, a Training setting of Block also stops Googlebot, Bingbot and Applebot, while "Disallow AI Training" blocks training crawlers and keeps those search crawlers (Cloudflare). What you are ultimately tuning, pages that earn the retrieval these rules permit, is covered in our generative engine optimization guide.