User-Agent String

A user-agent string is the text a client sends in the HTTP User-Agent request header to identify the software making the request: usually a product name and version, followed by details such as the rendering engine and operating system (MDN). Browsers send one, crawlers send one, and scripts send whatever their author typed.

The header is defined in RFC 9110, the HTTP Semantics standard, which says a user agent "SHOULD send a User-Agent header field in each request" (RFC 9110). The property that makes the rest of this library make sense is that the string is a claim, not an identity: nothing in HTTP verifies it, and every rule you write in robots.txt is addressed to whoever chooses to answer to that name.

What does a user-agent string look like?

A sequence of product identifiers and comments. RFC 9110 defines the value as one or more products, each a name with an optional /version, plus optional comments in parentheses, "listed in decreasing order of their significance"; its own example is CERN-LineMode/2.15 libwww/2.17b3. MDN gives the common browser format as Mozilla/5.0 (<system-information>) <platform> (<platform-details>) <extensions>.

A current desktop Chrome string from MDN, read part by part:

The zeros are deliberate. Under user-agent reduction, MDN explains, Chrome reports its major version correctly but always shows the minor version numbers as zeros, and on Android it reports a fixed platform version and a placeholder device model, K. Sites that need more detail can request it through User-Agent Client Hints, which arrive as separate Sec-CH-UA headers.

What is a user-agent string used for?

Servers read it to see what kind of client is asking. RFC 9110 lists the classic uses: identifying "the scope of reported interoperability problems," working around particular user agents' limitations, and "analytics regarding browser or operating system use." For a site owner in 2026, three more matter:

• Robots.txt matching. Crawlers look for the group that names their product token, a short name inside the full string.
• Log analysis. Counting requests by user agent shows which crawlers are working your site and how often.
• Bot controls. CDNs and WAFs match on it; Cloudflare's AI Crawl Control, for example, identifies AI crawlers by user agent string outside its Enterprise tier.

MDN adds a warning worth keeping: serving different content to different browsers based on the user agent "is usually a bad idea." For crawlers, the stakes are higher, as the cloaking section below explains.

What do crawler user-agent strings look like?

Mostly like browser strings with the crawler's product token and a URL explaining the bot. RFC 9309 asks that the identification string "describe the purpose of the crawler," and the major operators follow that convention. These are the strings as each operator documents them, read 2026-10-14:

Sources: Google, OpenAI, Perplexity, Apple and Common Crawl. Anthropic's help page names its 3 tokens, ClaudeBot, Claude-User and Claude-SearchBot, but does not publish full strings.

Versions move: OpenAI says the version number in its strings may change, Google's placeholder tracks the current Chrome release, and Apple says Applebot occasionally updates the browser version it advertises. That is the practical argument for matching on the token rather than the whole string. OpenAI also adds a robots.txt marker to the string when it fetches robots.txt itself, so site owners can separate that request from content requests in logs that do not record paths.

How does robots.txt match a user-agent?

By token, case-insensitively, to exactly one group of rules. RFC 9309 requires crawlers to use case-insensitive matching to find the group whose product token matches, so user-agent: gptbot and User-agent: GPTBot select the same rules. If more than one group matches the same token, the RFC requires those groups' rules to be merged into one.

Three fallbacks complete the logic, and they trip people in both directions. If no group matches the token, the crawler must obey the group with the * value. If there is no matching token and no * group, or no groups at all, no rules apply. And path matching runs on the opposite convention from token matching: the RFC says paths should be matched case-sensitively, starting from the first octet, with the most specific match winning and allow beating an equivalent disallow.

The consequence for a file you actually maintain: a * group is not a catch-all safety net for a bot you also named elsewhere. Name a token in its own group and that group is the only one it reads, which is exactly how a well-meaning "block everything, allow the good ones" file ends up allowing more than its author intended. Per-bot rule-writing is worked through in the robots.txt decision framework, and copy-paste files are in our AI robots.txt templates.

Why is a user-agent string a claim rather than an identity?

Because it costs one header to forge, and the popular bot names are the most useful camouflage on the web. A scraper sending GPTBot inherits whatever generosity your infrastructure extends to the real one: robots.txt allowances, WAF exceptions, rate-limit headroom. RFC 9110 itself anticipates masquerading and tells recipients they can assume a user agent that impersonates another wants responses tailored for the one it named.

The operators treat it the same way, which is the strongest evidence for the point. OpenAI, Anthropic and Perplexity publish crawler IP data so operators can separate their bots from impersonators; Apple and Common Crawl both document reverse-DNS checks, and Common Crawl warns outright that crawlers falsely identify themselves as CCBot. Google's crawlers overview documents identification by three signals: the user-agent header, the source IP, and the reverse-DNS hostname of that IP. The mechanics of running that check are in verifying real vs fake AI bots, and the class of client doing the fetching is defined in web crawler.

Should you serve different content based on the user-agent?

No. Varying the page an engine receives from the page a person receives is cloaking, and Bing's Webmaster Guidelines list it among the practices that "reduce ranking eligibility and grounding visibility, across Bing and Copilot experiences." The fact that the deciding input is a forgeable header should end the discussion on engineering grounds alone.

Reading the header is fine, and useful. Counting by token, rate-limiting a specific bot, or logging which agent fetched which URL are all ordinary operations. What breaks is branching the content on it, and there is no version of that trick that survives a spot check by anyone who curls your URL twice with two different headers. The page you want quoted should be the same page in every response, which is most of what our generative engine optimization guide argues for.

What should you do with the user-agent field in your logs?

Treat it as a label to group by, then verify before you believe it. Count fetches per token to see which platforms are working your site, keep the source IP on every line, and check membership against the operator's published ranges before any number leaves your machine.

Against our own interest: for most sites this is a spot check, not a project. If you are not about to publish crawler statistics, rewrite robots.txt, or configure CDN blocks, glancing at the top tokens and top source IPs is proportionate, and no tool you can buy makes a self-reported string more trustworthy than it is.