Implementing Web Bot Auth, end to end
A key, a published directory, a signed request, and a way to tell whether any of it worked.
On this page
Step by step
Why sites block you, and what signing does about it
A site deciding whether to serve you has three things to go on: the address you connected from, the User-Agent string, and how the request looks. The first two are claims anyone can copy. With Web Bot Auth you publish a public key at a domain you control and sign each request with the private key, so a site can check that the request was signed with a key that domain publishes. A short introduction to the standard is on the Agent Verifier page.
What signing gets you. Many sites do not verify signatures, so signing mainly prepares you to be recognised as more of them do. Doing it correctly matters all the same, because a wrong signature can cost more than none: a verifier that refuses it tells you nothing, and from your side a malformed signature and an unknown one look the same.
Step 1. Generate a key
Use Ed25519. It is the algorithm every deployed implementation supports, and its public key is small enough to sit comfortably in a directory. Keep the private key out of the repository and off the machine that serves your directory if you can.
The public half, as a JWK. Three members, and only these three matter:
{
"crv": "Ed25519",
"kty": "OKP",
"x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"
}The identifier for this key, the value that goes in keyid, is the RFC 7638 thumbprint: SHA-256 over the canonical JSON of exactly those three members, in lexicographic order (crv, kty, x), no whitespace, encoded base64url without padding.
Check yourself against a known answer first. Feed your thumbprint routine the Ed25519 test key above, the one from RFC 9421. It must produce exactly poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U. If it does not, stop here: every later failure will look like a rejected signature and you will debug the wrong thing. The two usual causes are including kid in the hashed JSON (it must not be there) and using base64 with padding instead of base64url.
Step 2. Publish a key directory
Serve a JWK Set over HTTPS at /.well-known/http-message-signatures-directory on the domain you want to be known by. That path is reserved, so a verifier reaching it knows the keys belong to that domain and not to whoever happened to upload a file somewhere.
GET /.well-known/http-message-signatures-directory
200 OK
Content-Type: application/http-message-signatures-directory+json
{
"keys": [
{ "crv": "Ed25519", "kty": "OKP", "x": "JrQLj5P…",
"kid": "poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U",
"nbf": 1786000000, "exp": 1793000000 }
]
}The key above is the RFC 9421 test key, so the numbers can be checked. Publish your own: the draft forbids test keys in production, and the Directory Verifier reports this one as a violation.
- Set the media type to
application/http-message-signatures-directory+json. The draft requires it. Plainapplication/jsonstill verifies at the signature check here, but the Directory Verifier reports it as a violation, and stricter verifiers may refuse it. - Leave
kidout, or set it to the thumbprint. The draft makes it optional, but akidthat is present must equal the thumbprint, and the Directory Verifier reports any other value as a violation. nbfandexpare optional and worth setting: they are how you retire a key without waiting for every cache to expire.- Leave
algout of the key. The draft restricts it to the HTTP Signature Algorithms registry, whose name for this curve ised25519. Browser and Workers verifiers import JWKs through WebCrypto, which throws on that value and accepts only the field absent or the JOSE spellings, so the one conformant value is the one that breaks the most common verifier. An operator reported a production directory refused by a major validator for exactly this. The draft’s own examples omit the field, the algorithm already travels inSignature-Input, and omitting it is accepted everywhere. - Serve it on the apex or the
wwwhost. Whichever you choose, name that exact host in the header and serve the directory there directly. A redirect between the two is not followed: the current draft requires every discovery fetch to answer200and forbids following redirects, because the address a verifier resolved is the identity it attributes the request to. A redirect is reported back to you as the status it was.
If you do not control a domain
The well-known path needs a domain to sit on. An agent that has none can point at the key file directly instead: put the full address of the file in the Signature-Agent member, with no parameters. The file may live at any address that answers 200 over HTTPS without a redirect, including a raw file on a code-hosting service.
Signature-Agent: sig1="https://example.com/keys.json"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1787000000;expires=1787000300;keyid="<RFC 7638 thumbprint>";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:BASE64_SIGNATURE:
signature base line for that member:
"signature-agent";key="sig1": "https://example.com/keys.json"- The body is the same JWK Set as above. No media type is required for it, because the draft does not define one for this address.
- Cover the
signature-agentmember keyed to your own signature label. The value that goes into the signature base is the member exactly as sent, here"https://example.com/keys.json". Earlier revisions of the draft asked for atype=jwks_uriparameter on this member. The current draft drops it and tells verifiers to ignore a member that carries one. This check still reads it, and a parameter you send belongs in the base too: leaving it out there fails asbad-signature, which looks like a cryptography problem although the cryptography is fine. - What it costs. A signature checked this way proves you hold the key. It attributes no domain to you, because a bare file has no origin behind it and no signed response to bind one. The answer says so:
keyDirectoryBindingcomes backbinding-not-checkedrather thanbinding-proven. Use it to get your signing right. To be recognised as somebody, the well-known path on a domain is the form that carries a name.
Step 3. Prove the directory is yours
The directory response can itself be signed by each key it advertises. That proves the party serving the file also holds the private keys, which stops a copied key set being re-served under someone else's name.
Cover the body as well. Signing the response is recommended, and a directory that signs is required to cover content-digest as well as the authority. Without the digest the signature proves the server holds a key, not which key set it published, and the Directory Verifier reports that as a violation.
Two details are easy to miss, and both cost time to find:
- The directory response is signed with the tag
"http-message-signatures-directory", a different tag from the one on request signatures. - It covers
("@authority";req)andcontent-digestover the body. The draft requires both.
A directory that signs for a shorter time than it asks to be cached has a problem: the proof expires while the cached copy is still being served. Verifiers that check freshness at use rather than at fetch, this one included, will report the proof as absent for most of that window.
Step 4. Publish an agent card
The signature ties the request to the keys a domain publishes. The card says what that domain is doing here: who runs the agent, why it fetches, the exact User-Agent it promises to send, where its keys live, and whom to write to. It is a small JSON file, and it is optional.
It is worth publishing anyway. A verifier holding your card can compare what you promised with what actually arrived, and the check on this site does that and reports the result. A site owner deciding whether to serve you also gets something to read besides a domain name.
The address of the card is up to you, and the draft treats that address as the card's own identifier: the client_id inside it must be identical to the URL it was fetched from. There is no well-known path to look a card up at. The registry draft, which defines the card, names three ways a verifier learns where a card is: the Signature-Agent header with type=cimd, a registry entry, or out of band. The one you control is the header, pointing at the card instead of at a key directory:
Signature-Agent: sig1="https://example.com/bot-card";type=cimdThe protocol draft tells verifiers to ignore a member that carries a parameter, so only a verifier that also implements the registry draft follows this one. This check does. Sent that way, the card is fetched here and its claims are reported back to you. Your keys then come from the card: give it a jwks_uri pointing at your key directory, or put the keys inline in jwks. The draft allows one or the other and rejects a card carrying both. A card with neither cannot be tied to the key that signed the request, so its claims are not read.
The card is fetched with a plain GET, no redirect is followed, and any status other than 200 is treated as a failure. That is the draft's rule, and the reason is that the address is the identity. The card published by this site sits at /.well-known/web-bot-auth-card, a working example to copy the shape from.
Step 5. Sign a request
A signed request carries three headers:
GET /api/v1/connection HTTP/1.1
Host: fp.packet.guru
Signature-Agent: sig1="https://your-agent.example/.well-known/http-message-signatures-directory"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");\
created=1786000000;expires=1786003600;\
keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";\
alg="ed25519";tag="web-bot-auth"
Signature: sig1=:BASE64_SIGNATURE:- Signature-Agent says where your keys are published. Send it as a dictionary member keyed to your signature label. The current draft makes the value the address of the key set itself, so for the directory from step 2 it is the full well-known URL, as in the example. Under earlier revisions the value was the bare origin,
sig1="https://your-agent.example", and the verifier added the path. That shape is still the common one on the wire, and so is the older bare quoted string without a member key. All of them get read here. - Signature-Input lists what you covered and the parameters. The tag must be
"web-bot-auth"or the signature is not read as an agent signature at all. - Signature is the signature itself.
Cover at least "@authority". Without a target component the signature is valid against any host and can be replayed elsewhere as your identity. Cover the signature-agent member keyed to your own label too, so the pointer to your keys cannot be swapped in transit. Covering "@method" and "@path" narrows it further, at the cost of a few bytes on each request.
The signature base you sign therefore looks like this:
"@authority": fp.packet.guru
"signature-agent";key="sig1": "https://your-agent.example/.well-known/http-message-signatures-directory"
"@signature-params": ("@authority" "signature-agent";key="sig1")\
;created=1786000000;expires=1786003600\
;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"\
;alg="ed25519";tag="web-bot-auth"Note the second line. A component written "signature-agent";key="sig1" is a dictionary member lookup: the value in the base is that member alone, quoted as it appears in the header, not the whole header field. Getting this wrong is a common way for a correct-looking implementation to fail.
Step 6. Check what you built
Send that request to the check, with no body and no parameters. It needs no account and no key, and the answer describes the request that was just made. To look at a request your agent already sent, paste it into the Agent Verifier form, and to check the directory from step 2 on its own, use the Directory Verifier.
curl https://fp.packet.guru/api/v1/connection \
-H 'Signature-Agent: sig1="https://your-agent.example/.well-known/http-message-signatures-directory"' \
-H 'Signature-Input: sig1=…' \
-H 'Signature: sig1=:…:'Read signals.agent.signature.outcome first. It carries the standard's own three verdicts: verified means the proof holds, invalid means a proof was presented and does not, and unverified means the check could not reach a decision, for example because the key directory could not be read or did not list the key.
When something is wrong, signals.agent.webBotAuthDetail names the one fault and the sentence saying what to change. Every reportable fault is listed, by subject, in the response reference.
The five mistakes that cost the most time
- Signing the whole header instead of the member. See the note under signing a request. Symptom: canonicalization-mismatch on an implementation you are certain is right, with a fix naming the line of the signature base to change.
- Hashing the wrong JSON for the thumbprint. Including
kid, or leaving whitespace in. Symptom: key-not-in-directory even though the key is plainly there. If the same wrong value is also published askid, the signature verifies with a note that the keyid is not the thumbprint, and the Directory Verifier reports thekid. - Using the wrong tag.
web-bot-authon requests,http-message-signatures-directoryon the directory response. Symptom: the answer reports no signature at all,signature.present: false, with a note carrying the bad-tag advice, or a directory proof that silently never counts. - A clock that is ahead. A
createdmore than five minutes in the future is rejected here, so a fast clock breaks every signature you send. Symptom: clock-ahead. - Naming a host that does not serve the directory. Apex versus
www, or a directory published somewhere else entirely. Symptom: anything in your key directory.
What signing will and will not get you
- A signature proves identity. Trust is separate. A valid signature proves the request was signed with a key your domain publishes. Anyone can publish keys at their own domain, so a valid signature from an unknown name tells a site who you are and nothing about whether to serve you.
- Recognition is a separate step. The large verifiers check signatures against a list of operators they have already admitted, and getting on such a list means applying to each of them. Signing correctly is required for that and does not replace the application.
- Paid access can depend on it. Besides allow and block, some networks let site owners charge automated callers per request at the edge. Those gateways can require the caller to authenticate with Web Bot Auth, so that the charge lands on a known account. Where that is switched on, a signature that fails to verify decides whether your agent gets its own terms or the default for strangers. Worth watching if your agent will pay for anything.
- Publishing your address ranges still helps. It is the other half of how you are recognised, and for most operators it is still the half that does the work.
Prompt: ask an agent to set signing up
Paste this into an agent that can edit files on the machine your service runs on. It works through the steps above and is written to stop at each one rather than run to the end on its own.
Read this before pasting it. It creates a signing key and changes what your service publishes. The key is generated on your machine and never sent anywhere, including to this site. The prompt asks for your approval at every step that writes something. Run it somewhere you can undo.
Set up Web Bot Auth signing for my service, following https://packet.guru/agents/guide.
Rules:
- Generate the Ed25519 key locally. It must never leave this machine, and never appear in a commit, a log or a message to anyone.
- Show me each change before you make it, and wait for my approval. That covers: the key, the key directory file, the directory-response signature, the agent card, and the signing code.
- Follow the guide's steps in order. If the guide and your own knowledge of the standard disagree, stop and show me the difference. Do not pick one on your own.
When everything is in place, verify it rather than assuming it worked: send one signed request to https://fp.packet.guru/api/v1/connection and report back
- whether the signature verified,
- the exact fault and fix sentence if it did not,
- what my key directory proved about its own key set,
- and what is left to do.
Do not tell me it works until that call says so.Further reading
- Web Bot Auth: How the Web Started Checking an Agent's ID How the signature, the key directory and the Signature-Agent header work, and who publishes keys.
- Agent Verifier: See Your Signed Request the Way a Server Sees It What breaks a setup in practice, from a signature base built a character differently to a directory a CDN refuses to serve.
- Replay Attacks: Why a Valid Signature Is Not a Fresh Request What a valid signature proves, what it leaves open, and why the created and expires window matters.
The standards behind this
For machines
- API catalog Every API here, each with its description and its documentation. Standard address, RFC 9727.
- AI catalog The AI artifacts this domain offers, for an agent discovering them on its own. It points to the MCP server card.
- openapi.json The response contract, machine-readable. The same source the reference page is built from.
- MCP server card What an MCP client reads before it connects: name, version, transport, and protocol versions.
- MCP registry entry What the official MCP registry publishes about this server: address, transport, and the tool it exposes.
- packet.guru key directory The public keys this site signs its own outbound requests with. A working example of the file you publish.
- packet.guru agent card What the fetcher on this site promises about itself, including the exact User-Agent it sends.