Cloudflare's AI Crawler Controls: What Each Setting Actually Blocks

Cloudflare sits in front of a meaningful share of the web, which means its AI-crawler settings are part of your robots policy whether you chose them or not. Those settings changed twice in 2026: on July 1, Cloudflare replaced its single "Block AI bots" toggle with separate Search, Agent and Training policies, and on September 15, it changed what "Block" means. This page maps what each setting actually blocks, per Cloudflare's own documentation and announcements, and then gives the audit we run, born from finding one of these settings live where nobody had chosen it [our data].

What do Cloudflare's AI crawler controls actually do?

They sort AI traffic by behavior, Search, Agent and Training, and let every customer, Free plan included, allow or block each one, write matching robots.txt rules, and block individual crawlers at the edge (Cloudflare changelog, July 1, 2026). Since September 15, 2026, the Training setting has four options, and the difference between two of them decides whether Google, Bing and Apple can still crawl you for search (Cloudflare, September 15, 2026).

What each Training option blocks, per Cloudflare's documentation and its September 15, 2026 announcement, read 2026-10-14:

Cloudflare's own summary of the change: "Block and Block on pages with ads now apply to mixed-use crawlers, including Applebot, Bingbot, and Googlebot, so either setting impacts search as well as training. To stop training and keep search, use Disallow AI Training." Disallow AI Training exists only for Training, not for Search or Agent.

The other settings, and what each one is:

Sources: Cloudflare's Block AI Bots, robots.txt setting, Manage AI crawlers and Bot Fight Mode documentation.

What changed on September 15, 2026?

Cloudflare added Disallow AI Training and widened Block: Block and Block on pages with ads now catch mixed-use crawlers, which until then had been exempt because blocking them also hurts search discoverability. Cloudflare's stated reason is that the new setting finally lets a site refuse training without refusing search, so Block can mean block.

Existing settings were migrated, not reset. Per Cloudflare's migration tables, a legacy Block AI Bots setting of Block, or Block on pages with ads, became Search allowed, Training set to Disallow AI Training, and Agent blocked on ad pages; a Training policy previously set to Block, or Block on pages with ads, became Disallow AI Training. Stopping Googlebot now takes a deliberate choice on the new controls: Block or Block on pages with ads for Training, or blocking Search itself.

New domains now get a preset based on one onboarding question. A site that says it monetizes with ads starts with Training on Disallow AI Training and Agent blocked on ad pages; any other site starts with Search, Training and Agent all allowed. Bot Preference Sync is on in both presets.

Correcting the record where Cloudflare's own pages disagree: the Block AI Bots documentation, last updated July 1, 2026, still says mixed-purpose crawlers will be blocked by "all configurations to block AI training, including the legacy 'Block AI bots' option." The September 15 announcement is the later statement, it is specific about the migration above, and it matches the robots.txt documentation updated October 8, 2026, which says mixed-use crawlers "are excluded" from the Disallow edge block. Trust the later two.

How does Cloudflare's robots.txt layer work?

Cloudflare intercepts requests for /robots.txt on a proxied hostname and, when your origin already has a file, prepends its managed rules to it, so crawlers receive one combined response (Cloudflare docs). Since September 15, 2026, Bot Preference Sync writes those rules from your Search, Agent and Training policies; your source file is never edited, which is exactly why the divergence is easy to miss.

Cloudflare's documented example shows what the managed block contains: a content-signals line (Content-signal: search=yes, ai-train=no, use=reference) and Disallow: / for 8 AI crawlers, namely Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent. The same docs say plainly that robots.txt compliance is voluntary and point to AI Crawl Control for enforcement.

We met that file in production. The managed robots.txt on one zone ran 2,895 lines and carried Disallow: / for those same 8 AI crawlers while the robots.txt in the build's git repo said allow [our data]. A site with that file live is opted out of the training pipelines of OpenAI, Anthropic, Google, Apple, Meta and the Common Crawl corpus at once, whatever its own robots.txt says. The full story, found one day before a live-domain cutover, is in the managed robots.txt trap.

What do the edge blocks and Bot Fight Mode add?

Enforcement, the layer robots.txt can never be. When you block a crawler in AI Crawl Control, Cloudflare creates or updates a WAF custom rule on your zone, and a blocked crawler gets no page whether it honors robots.txt or not (Cloudflare docs). Paid plans can choose the response, 403 Forbidden or 402 Payment Required. Outside Enterprise Bot Management, detection relies on user agent strings, so it catches self-identifying crawlers only.

Bot Fight Mode is broader machinery: free, zone-wide challenges for traffic matching known bot patterns, not scoped to AI. Cloudflare's docs say it cannot be customized with WAF custom rules and that Skip, Bypass and Allow actions have no effect on it, so you cannot carve out a crawler you want served. On a citation-seeking build, we treat it as guilty until its scope is confirmed [our data].

Cloudflare's crawler identification draws on the same network visibility behind its crawl-to-click research and the Radar bot insights it released on July 1, 2025 (Cloudflare changelog), so a crawler that declares itself honestly is identified by name, not guessed at.

How do you audit Cloudflare's AI settings?

Five checks, run before DNS ever points through the proxy, because every control above acts only on proxied traffic. These are the checks from our pre-cutover audit [our data], with the dashboard names updated to Cloudflare's current settings:

1. Security Settings, filtered to Bot traffic, then Configure AI bot policies: read Search, Agent and Training. A Training value of Block also blocks Googlebot, Bingbot and Applebot, and Block on pages with ads does so on ad pages; make sure that was the intent.
2. Check whether Bot Preference Sync is on, then fetch the served robots.txt and diff it. curl https://yourdomain.com/robots.txt should match what you expect byte for byte. We repeat the fetch with AI-crawler user agents, GPTBot, ClaudeBot, CCBot, PerplexityBot and OAI-SearchBot, spaced about 60 seconds apart, and flag any divergence.
3. Walk every per-crawler action in AI Crawl Control. Each crawler's allow or block state should match a row in your robots.txt decision framework; an action you cannot map to a decision is a finding.
4. Check Bot Fight Mode. Confirm its scope before trusting it near crawlers you want served, since WAF exceptions cannot override it.
5. Record evidence. Screenshot the settings and keep the curl outputs; re-run the whole pass right before any cutover that sends traffic through the proxy.

The verification habit matters more than the tooling: what your CDN serves is a claim to test, not a setting to remember. The deeper practice of reading crawler behavior from your own records is in server log analysis for AI bots.

Should you use Cloudflare's AI controls?

If your decision is to block AI training, yes, and use Disallow AI Training rather than Block unless you also want out of Google, Bing and Apple search. We say that as a shop whose own sites allow every documented AI crawler. The economics that drive publishers to block are real and measured: Cloudflare's crawl-to-click data put Anthropic's ratio at 38,066 pages crawled per referral in July 2025 (Cloudflare), and for a pageview-funded site, free edge enforcement against that is the correct tool. Our objection is not to the controls; it is to unaudited state.

If your decision is to allow, the posture behind should publishers block or allow AI crawlers and everything in our generative engine optimization guide, then confirm all three policies read Allow, because these settings are the first place your strategy can silently die. Either way, the rule is the same: the CDN is a co-author of your crawl policy, so read what it wrote.