What Is CCBot? What Blocking Common Crawl Actually Does

CCBot is the most misunderstood line in the AI section of robots.txt, because it is the only major AI-associated crawler that does not belong to an AI company. Block GPTBot and you have made a decision about OpenAI. Block CCBot and you have made a decision about an open archive and every pipeline that draws on it, most of which you cannot name. This page sets out what Common Crawl documents about its crawler, what blocking it does and does not change, and how to decide.

What is CCBot?

CCBot is the web crawler of Common Crawl, a 501(c)(3) nonprofit founded in 2007 that has collected web crawl data since 2008 and gives the archive away free (Common Crawl). CCBot does not feed one company's model or one engine's index; it feeds a public archive whose users range across AI training, academic research, and whatever else a downloadable copy of the web enables.

CCBot at a glance, per Common Crawl's own pages, read 2026-10-14:

Sources: Common Crawl's CCBot page, FAQ, about page and Opt-Out Ledger announcement.

Common Crawl is not an AI lab, but its archive sits underneath a lot of AI. Its own history page notes that Google's C4 corpus, built from a single Common Crawl snapshot, trained the T5 models in 2019, and that the GPT-3 paper in 2020 reported most of its training tokens came from filtered Common Crawl data.

What happens if you block CCBot?

Common Crawl stops crawling your site: its FAQ says that with Disallow: / in a CCBot group, "our crawler will stop crawling your website," and that it will keep checking whether your robots.txt has changed. Everything already published stays where it is.

What blocking does and does not change:

The downstream consequences are diffuse and invisible from where you stand. A model trainer who filters next year's corpus will not see your domain; a researcher building an evaluation set will not sample your pages. None of them will notify you, and none of their choices appear in any analytics property you own. Blocking GPTBot has a bounded, documented effect, one company's training pipeline per OpenAI's own docs; blocking CCBot edits you out of a shared input that an unknown number of systems draw from, the same corpus-shaped substrate that retrieval-based answer engines sit on top of. We can state the direction of that effect; nobody can state its size, and you should distrust anyone who quotes one.

For legal requests, Common Crawl publishes an Opt-Out Ledger listing every legal opt-out it has received, so downstream users can see what owners asked to exclude; for ordinary control of crawling, it points publishers to robots.txt.

Does CCBot respect robots.txt?

Yes, by Common Crawl's own documentation: CCBot checks robots.txt before fetching any page, obeys Disallow rules and the Crawl-delay parameter, and honors nofollow on links. The rules are ordinary RFC 9309 syntax. To block it entirely:

User-agent: CCBot
Disallow: /

To slow it instead, Common Crawl's own example limits CCBot to one request every 2 seconds:

User-agent: CCBot
Crawl-delay: 2

Under RFC 9309, robots.txt rules "are not a form of access authorization": compliance is the crawler operator's choice, and Common Crawl warns that other crawlers falsely identify themselves as CCBot. Those impostors have no reason to obey your CCBot rule, so verify before you trust the name: real CCBot requests come from dedicated IP ranges whose reverse DNS resolves under crawl.commoncrawl.org, and enforcement against fakes is a CDN or WAF job. In our log reviews the platform crawlers, ClaudeBot, GPTBot and PerplexityBot, are the regular AI visitors we track and verify; we have not published dated CCBot fetch counts, and when we do they will be log extracts with periods attached [our data].

How much does CCBot actually crawl?

Very little, by the one published network-wide measurement, and the volume is stable. Cloudflare's crawl-to-click study puts CCBot's share of AI-bot traffic on its network at 0.1% in July 2024 and the same 0.1% in July 2025, two orders of magnitude below the large platform training crawlers (Cloudflare).

Period and figures per Cloudflare's crawl-to-click analysis; platform crawlers omitted here, and the full roster is in the complete AI crawler list. Common Crawl adds its own load safeguards: CCBot backs off when your server answers with HTTP 429 or 5xx errors. Whatever you decide about CCBot, you are deciding about corpus inclusion, not bandwidth.

Are you blocking CCBot without knowing it?

Possibly. CCBot is 1 of the 8 AI crawlers in the managed robots.txt that Cloudflare documents, alongside GPTBot, ClaudeBot, Google-Extended and others (Cloudflare docs), and we have seen that default live in production. During a pre-cutover audit on our auto-finance authority rebuild, Cloudflare was serving a 2,895-line robots.txt with Disallow: / for 8 AI crawlers, CCBot among them, while the robots.txt in the build's git repo said allow [our data]. Nobody had decided to block Common Crawl; a dashboard default had.

The check takes one minute: fetch https://yourdomain.com/robots.txt and read what is actually served, not what your repo or CMS contains. If a CDN sits in front of your site, the served file is the only one that counts; how Cloudflare's 2026 settings write that file is mapped in what Cloudflare's AI crawler controls do.

Should you block CCBot?

It is the rare AI-crawler call where both answers are defensible, because the trade is licensing philosophy rather than measurable traffic. A publisher whose content is a licensed asset has a coherent case for blocking: an open, freely downloadable corpus is the least controlled form of distribution that exists, and 0.1% of bot traffic buys none of it back. A business that earns by being widely known, our situation, has the opposite case: the corpus is one more path into the systems that answer questions about your topic.

We allow CCBot across all 3 production builds [our data], consistent with the allow-everything posture argued in should publishers block or allow AI crawlers. Against our own interest: if your revenue depends on controlling copies of your content, our posture is wrong for you, and this is the block to make before any platform-specific one. Either way, decide it on purpose, and if you leave the door open, the pages behind it should be worth quoting, which is the entire subject of our generative engine optimization guide.