Introduction
For three decades, the way a machine identified itself on the web was a text field. A crawler wrote Googlebot into its User-Agent header, and the server believed it. No proof, no signature, no way to check. The whole convention rested on the idea that respectable software tells the truth about itself, and for a long time that was mostly good enough.
It stopped being good enough the moment automated traffic became the majority. In June 2026, Cloudflare put automated requests at 57.5% of HTML traffic across its network, the first time bots outnumbered humans in that measurement. DataDome logged 7.9 billion AI agent requests in January and February 2026 alone. And the identity crisis showed up in exactly the way you would predict from a system built on self-declaration: the most impersonated agent in that dataset, Meta-ExternalAgent, carried 16.4 million spoofed requests, while nearly 2.4% of everything claiming to be PerplexityBot was fraudulent.
Web Bot Auth is the answer the industry converged on. Instead of asking an agent who it is, the server checks a cryptographic signature on the request itself. This article walks through how that works on the wire, what a valid signature actually proves, where it sits next to the other ways machines authenticate to each other, and how much of it is real today rather than aspirational.
What Web Bot Auth Is, in One Paragraph
Web Bot Auth is a way for an automated HTTP client to sign its outbound requests with a private key, so that the receiving server can verify the request came from the holder of a specific key rather than from anyone who copied a User-Agent string. It is built on RFC 9421 HTTP Message Signatures, uses Ed25519 keys, and publishes public keys at a well-known URL so a verifier can fetch them without any prior relationship. The work lives in a dedicated IETF Web Bot Auth Working Group, and the core drafts are co-authored by Thibault Meunier of Cloudflare and Sandor Major of Google. Deployment is already ahead of standardisation, which is the usual order of events for anything Cloudflare ships at the edge.
Why the Old Ways of Identifying a Bot Failed
Every mechanism the web used before this one answers a slightly different question, and none of them answers "is this really who it says it is" cheaply enough to run on every request.
User-Agent strings are a claim, not evidence. Anyone can send ChatGPT-User and there is nothing in the request that contradicts it. This is not a subtle weakness. It is the entire attack, and it accounts for the impersonation numbers above.
Reverse DNS verification works, and Google and Bing have documented it for years. You take the client IP, resolve the PTR record, confirm the hostname belongs to the crawler's domain, then resolve that hostname forward and check it matches the original IP. Two DNS round trips per unknown client, on every request, at internet scale. It also fails the moment a legitimate agent runs from infrastructure that does not control its own reverse DNS, which describes most of the agent ecosystem born after 2024.
Published IP ranges shift the cost to maintenance. The crawler operator publishes a JSON list, the site owner ingests it and hopes the refresh cadence is fast enough. The list goes stale, cloud IP space rotates, and the whole scheme collapses for any agent that runs on shared infrastructure, because the address it exits from is also the address a thousand other things exit from. That is the same problem covered in IP reputation, clean versus toxic: an address describes a neighbourhood, not a tenant.
robots.txt is not identification at all. It is a posted request for good behaviour, standardised in RFC 9309, addressed to a client whose name you cannot verify. A crawler that lies about its name has already opted out of reading the file honestly.
The gap is structural. All four methods try to infer identity from something adjacent to the request, whether that is a string inside it, the address behind it, or a record somewhere else. A signature is the only approach that puts the proof inside the request.
The Mechanics: Signing an HTTP Request
Web Bot Auth does not invent its own cryptography. It profiles RFC 9421, the HTTP Message Signatures standard, and constrains it enough that two implementations built independently will interoperate.
An agent picks the components it will sign. The protocol requires at least one of @authority (the host being requested) or @target-uri (the full URL), so a captured signature cannot simply be pointed at a different site. It then fills in the signature parameters:
createdandexpires, a validity window. The drafts recommend no more than 24 hours, and short windows are the primary defence against replay.keyid, which is not a name chosen by the operator but a base64url JWK SHA-256 thumbprint of the public key itself. The identifier is derived from the key, so it cannot be claimed by anyone who does not hold that key.tag, which MUST be the literal stringweb-bot-auth. This is what separates bot authentication from every other use of HTTP Message Signatures on the same connection.nonce, optional but recommended, a random 64-byte value that lets a strict origin reject a replayed signature outright.
All of that travels in three headers. Signature carries the signature bytes themselves, base64 encoded. Signature-Input carries the recipe: the list of components that were signed, plus the parameters above, so the verifier can rebuild the exact string that was signed rather than guess at it. And Signature-Agent carries a single URL pointing at the place where the agent publishes its public keys. The first two are effectively mandatory. The third is optional but recommended, because without it a verifier that has never seen this key has nowhere to go and no way to check anything.
The verifier reconstructs the signed string from the named components, fetches the public key, and checks the signature. That is a single elliptic-curve verification, measured in microseconds, against a key that is almost always already in a local cache. Compare that to two DNS lookups.
One detail matters more than it looks. The Signature-Agent header must itself be covered by the signature when it is present. An intermediary cannot rewrite the agent's stated key location without breaking the signature, which closes the obvious downgrade path.
Reading the trust boundary correctly. The draft is blunt about this: Signature-Agent is a client-controlled hint, and verifiers MUST NOT grant trust based on its value alone. The header tells you where to look for a key. The key, and any binding you validated for it, is what policy attaches to.
Where the Keys Live: the Well-Known Directory
An agent that wants to be recognised by a site it has never contacted publishes its public keys at a fixed, standardised path on its own domain, /.well-known/http-message-signatures-directory. Anyone can fetch it, no registration required, which is the whole point: a verifier meeting an unknown agent for the first time still has somewhere to look.
The response is a JSON Web Key Set, served over HTTPS with a dedicated media type and a cache lifetime the drafts illustrate at 24 hours. Each key carries a validity window, so rotation is a publishing operation rather than a coordination problem. The operator serves old and new keys together, waits for caches to expire, then drops the old one.
This is not a hypothetical. Fetching that path on known agent domains at the time of writing returns live key material. OpenAI publishes a single Ed25519 key under chatgpt.com, complete with the thumbprint identifier, a start and end date for the key, and an extra field declaring the purpose of the traffic as AI.
Google publishes its signed agent traffic under the domain agent.bot.goog, and its directory currently carries five Ed25519 keys at once, which is what an active rotation schedule looks like from the outside. Browser automation platforms such as Browserbase serve the same well-known path.
The directory also carries a subtlety that is easy to miss. Serving keys over TLS proves that the domain published them, which makes the certificate on that domain the anchor of the whole scheme and something anyone can inspect with an SSL certificate lookup. The drafts go one step further and have each advertised key sign the directory response, which proves possession and stops a key set from being copied and re-served under someone else's authority. From that the verifier gets an authenticated binding between a domain and a key, and that binding is what survives rotation. The alternative, called opaque mode, skips the directory entirely and treats the raw key as the identity, which works but gives you no name to attach a reputation to.
Comparing the Ways a Machine Can Prove Itself
| Method | What it proves | Forgeable | Cost per request |
|---|---|---|---|
| User-Agent | Nothing. It is a self-declared label. | Trivially | Zero |
| Reverse DNS | The IP belongs to a domain the operator controls. | No | Two DNS round trips |
| IP allowlist | The request came from published address space, shared with everyone else there. | Only by co-tenants | Lookup, plus list maintenance |
| Web Bot Auth | The request was produced by the holder of a specific private key. | No | One signature verification |
What a Valid Signature Actually Proves
This is where careful reading pays off, because the protocol claims far less than the marketing around it.
A valid signature proves two things. The request was produced by a holder of the private key, within the window bounded by created and expires. And requests carrying the same keyid come from holders of the same key. That is it.
It does not say who operates the agent. It does not say the agent is benign. It does not say the request is authorized. The drafts state plainly that the protocol does not authenticate human users, does not provide anonymous authentication, and does not define authorization or delegation. Those are origin policy, and they always were.
What the signature buys you is continuity. A keyid is a stable handle you can log, rate limit, allowlist, or block, in exactly the way you already do with IP addresses. The difference is that a key cannot be borrowed by an impostor. The difference in the other direction is that a key is cheap to mint, so an agent blocked on one key can generate another or simply stop signing. The protocol does not pretend to solve that. It targets honest clients that want to be recognised across requests, and it leaves unsigned traffic exactly where it was.
That is a healthier framing than "cryptographic proof of a good bot," and it maps cleanly onto how detection already works. A signature answers who. It says nothing about what the client is doing, which is why the behavioural and transport layers do not go away. A signed agent that scrapes at 400 requests per second is still abusive, and an unsigned client still gets classified the old way, by TLS fingerprint, by TCP stack characteristics, and by the browser-level tells covered in fake browser detection.
Where This Sits in Machine Identity Management
Web Bot Auth is one answer inside a much larger shift. Non-human identities now outnumber human ones in most enterprises by a wide margin, and machine identity management has become its own discipline. It helps to see which problem each mechanism was built for, because they are not competitors.
API key management is the oldest pattern and the weakest. A shared secret in a header identifies a caller only as long as the secret stays secret, and a secret that travels in every request to a third party is one log file away from compromise. The Web Bot Auth drafts explicitly forbid the shared-secret HMAC mode of RFC 9421 for exactly this reason, because shared secrets break non-repudiation and make auditing hard.
OAuth client credentials and the newer MCP server auth model solve a different problem: an agent calling an API it has an account with. The Model Context Protocol specification now treats an MCP server as an OAuth 2.1 resource server, requires PKCE, and uses RFC 9728 protected resource metadata at /.well-known/oauth-protected-resource so that a client can discover which authorization server to talk to. That is AI agent authentication with a prior relationship, scoped tokens, and revocation. It is the right tool when the agent has an account. It is the wrong tool for an anonymous crawler hitting a site it has never seen.
Mutual TLS authentication proves identity at the transport layer, which is excellent inside a controlled boundary and awkward across the open web, where the client's certificate has to mean something to a verifier that has never heard of the client's certificate authority.
SPIFFE workload identity is the pattern that won inside the datacenter. Short-lived, cryptographically verifiable identity documents issued to workloads at startup, with workload identity binding tying the credential to what the workload actually is rather than to a secret someone pasted into a config file. It is now a graduated CNCF project and increasingly the substrate for agentic identity management in production systems.
Web Bot Auth fills the one gap none of those cover: a client and a server with no prior relationship at all, on the public web, where the verifier needs a cheap answer in the milliseconds before it decides whether to serve a page. That is why the design fetches keys from a well-known path instead of assuming enrolment, and why the trust model is deliberately thin.
The layer above all of them is still open. Proving that a request came from a specific agent does not tell you whether that agent may read this document, act on this account, or spend this budget. Fine-grained authorization for delegated agents, sometimes framed as agent zero trust or AI access control, is the part of the stack that is still being invented. Authentication is the prerequisite, not the answer.
Adoption: What Is Actually Live
Standardisation and deployment are running on different clocks, so it is worth separating them.
At the edge. Cloudflare validates Web Bot Auth signatures for its customers with no origin-side work, and on 1 July 2026 it folded signed agents and verified bots into a single model, distinguished by whether an agent is operated directly or on behalf of someone else. Its documentation lists three acceptable ways to be verified: message signatures, published IP lists, and reverse DNS. Signatures are the newest path, not yet the mandatory one. AWS WAF added Web Bot Auth support, and Akamai, Vercel and Shopify are in the same camp.
Among agents. OpenAI, Google and several browser automation platforms publish live key directories, as shown above. Google is explicit that this is experimental, that not every Google user agent uses the protocol, and that sites should keep using IP ranges, reverse DNS and user-agent strings during the rollout. That caveat is the single most useful sentence in the entire ecosystem right now.
At the IETF. A Web Bot Auth Working Group exists, with the protocol, directory and registry documents moving through it. The registry work, co-authored across Cloudflare and Amazon, adds a signature agent card, which is the beginning of a machine-readable description of who an agent is and what it is for. Alongside it, the AIPREF Working Group is standardising the vocabulary and the Content-Usage attachment that will let a publisher express what its content may be used for. Together those two efforts sketch a web where an agent proves its identity and then reads a machine-readable policy, instead of a web that guesses at both.
On the commercial side. Cloudflare's Pay Per Crawl work turns the same plumbing into an economic layer, answering an unpaid crawler with HTTP 402 Payment Required. From 15 September 2026, Cloudflare's defaults will block crawlers that mix search with AI training on ad-supported pages for new and free-tier sites, unless the operator separates them. Identity is the precondition for all of it, because you cannot charge, exempt, or throttle a client you cannot name.
Expect friction during the transition. Cloudflare's verifier currently expects the Signature-Agent header in the quoted-string form and rejects the dictionary form introduced in later protocol drafts, which is a small, concrete example of deployed reality lagging the specification. If you are implementing this today, match the verifier you are actually talking to.
What Web Bot Auth Does Not Fix
A signature is not a character reference. Several failure modes survive it intact.
A signed agent can still be hijacked. Prompt injection against an agentic browser makes the agent perform an attacker's instruction using the agent's own legitimate credentials, and every one of those requests will carry a perfectly valid signature. Authentication confirms the sender, not the sender's intentions.
Keys can be stolen. The drafts cover the response, which is to stop using the key, rotate, and notify. Between compromise and rotation, an attacker signs as you.
Blocking is cheap to escape. Rotating away from a blocked keyid costs nothing, which is why signatures are most useful for recognising clients that want to be recognised and much less useful as a punishment mechanism.
Small agents have no infrastructure. Publishing a key directory over HTTPS and rotating keys on a schedule is trivial for OpenAI and non-trivial for a two-person tool. If verification hardens into a requirement, the practical effect is centralisation, where a handful of large platforms become the only agents the web recognises.
Unsigned is not the same as malicious. Your own browser sends no signature, and neither does most legitimate software. The protocol says nothing about how origins should treat unsigned traffic, and that silence is deliberate.
FAQ
Q: What is the Signature-Agent header?
A: It is an HTTP header carrying the URL where an agent's public keys are published, for example Signature-Agent: sig="https://chatgpt.com". It is a hint that helps a verifier find key material. It is not proof of anything on its own, and the specification explicitly warns verifiers not to trust its value without validating a signature against the key it points to.
Q: How is this different from Cloudflare's Verified Bots?
A: Verified Bots is a programme, Web Bot Auth is a protocol. Cloudflare accepts three ways to earn verified status: message signatures, a published IP list, or reverse DNS. Web Bot Auth is the cryptographic one, and it is the only one that survives an operator moving to new infrastructure without republishing anything.
Q: Does this replace robots.txt?
A: No. They answer different questions. robots.txt says what a client should do, Web Bot Auth says who the client is. The IETF AIPREF work is the piece that modernises the first half, with a standard vocabulary and a Content-Usage signal for expressing how content may be used.
Q: Can a signature be forged or replayed?
A: Forged, no, not without the private key. Replayed, potentially, which is why signatures carry created and expires values with a recommended maximum lifetime of 24 hours, and why an origin can demand a unique nonce and reject anything it has seen before.
Q: Do I need to do anything as a normal user?
A: No. This is machine-to-machine plumbing. Your browser does not sign requests and is not expected to. Where it touches you indirectly is in bot mitigation quality: the better sites become at recognising real agents, the less they need to interrogate everyone else, which is the mechanism behind the CAPTCHA trap.
Q: I operate a crawler or an agent. What does adopting this involve?
A: This part is for operators of automated clients, not for anyone browsing normally. Four things are needed. Generate an Ed25519 keypair and keep the private half in whatever secret storage you already trust. Publish the public half as a JWK Set at the well-known directory path on a domain you control, served over HTTPS with the media type from the directory specification. Sign each outbound request following RFC 9421, with the tag set to web-bot-auth, the keyid derived as a thumbprint of your own key, a short created and expires window, and at least the request authority among the signed components. Finally, register that directory with the verifiers whose traffic decisions matter to you, which for Cloudflare means its bot submission form rather than anything automatic.
Two practical notes. Use a separate key per agent, since the drafts advise against reusing one key for different purposes, and reused keys merge two reputations into one. And check the exact header format your target verifier expects, because deployed verifiers and the current drafts do not agree yet on how Signature-Agent is written.
Q: What happens to unsigned bots?
A: Whatever each site decides. The protocol takes no position. In practice unsigned traffic keeps being classified the way it always was, by address reputation, transport fingerprint and behaviour, which means the least trustworthy signals are what an unsigned client gets judged on.
Recommendations
For site operators
- Treat a verified signature as identity, not as permission. Attach policy to the verified key or the verified domain binding, never to the
Signature-Agentstring alone. - Keep your existing layers. Signature verification tells you nothing about request rate, scraping patterns, or credential abuse.
- Cache key directories aggressively and fail closed on unknown keys rather than blocking on a network fetch in the request path.
For agent builders
- Sign now, before verification hardens from advantage into requirement. Early signers get expedited handling in verification programmes.
- One key per agent, per purpose. Reused keys collapse two reputations into one, and you will regret it the first time one of them is abused.
- Keep expiry windows short and rotate on a schedule, serving old and new keys together through one cache lifetime.
For privacy-minded readers
- Nothing here fingerprints you. Web Bot Auth is a public, deliberate identity claimed by software that wants to be recognised, which is the opposite of the covert identification covered in the rest of this blog.
- The relevant second-order effect is fewer challenges for humans as machines become easier to name, and that is worth watching rather than worrying about.
Final thought. For thirty years the web asked a machine "who are you" and simply believed the answer. Web Bot Auth does not make agents trustworthy. It makes them accountable, by giving them a name that cannot be borrowed. What each site does with that name is the interesting part, and it is only just being decided.
Curious what your own connection is broadcasting? Run the Privacy & Trust Index to see which signals your stack exposes before you ever say a word about who you are.