Implementing Web Bot Auth, end to end

Why sites block you, and what signing does about it

A site deciding whether to serve you has three things to go on: the address you connected from, the User-Agent string you sent, and how the request looks. The first two are claims. Anyone can send User-Agent: GPTBot, and plenty do, which is exactly why a site that trusts the string ends up trusting impostors and blocking you.

Web Bot Auth replaces the claim with a proof. You publish a public key at a domain you control, sign each request with the matching private key, and a site can then check the signature against the published key. What it proves is one thing properly: this request came from whoever controls that domain.

Step 1. Generate a key

Use Ed25519. It is the algorithm every deployed implementation supports, and its public key is small enough to sit comfortably in a directory. Keep the private key out of the repository and off the machine that serves your directory if you can.

The public half, as a JWK. Three members, and only these three matter:

{
  "crv": "Ed25519",
  "kty": "OKP",
  "x": "JrQLj5P_89iXES9-vFgrIy29clF9CC_oPPsw3c5D0bs"
}

The identifier for this key, the value that goes in keyid, is the RFC 7638 thumbprint: SHA-256 over the canonical JSON of exactly those three members, in lexicographic order (crv, kty, x), no whitespace, encoded base64url without padding.

Step 2. Publish a key directory

Serve a JWK Set over HTTPS at /.well-known/http-message-signatures-directory on the domain you want to be known by. That path is the whole discovery mechanism: it is reserved, so a verifier reaching it knows the keys belong to that domain and not to whoever happened to upload a file somewhere.

GET /.well-known/http-message-signatures-directory
200 OK
Content-Type: application/http-message-signatures-directory+json

{
  "keys": [
    { "crv": "Ed25519", "kty": "OKP", "x": "JrQLj5P…",
      "kid": "poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U",
      "nbf": 1786000000, "exp": 1793000000 }
  ]
}

• Set the media type to application/http-message-signatures-directory+json. Plain application/json gets named as a deviation here rather than refused, but some verifiers are stricter.
• Make kid equal to the thumbprint. It is not required, but both production directories in the world do it, and a verifier that looks up by kid will find your key without hashing anything.
• nbf and exp are optional and worth setting: they are how you retire a key without waiting for every cache to expire.
• Leave alg out of the key. The draft restricts it to the HTTP Signature Algorithms registry, whose name for this curve is ed25519. Browser and Workers verifiers import JWKs through WebCrypto, which throws on that value and accepts only the field absent or the JOSE spellings, so the one conformant value is the one that breaks the most common verifier. An operator reported a production directory refused by a major validator for exactly this. The draft’s own examples omit the field, the algorithm already travels in Signature-Input, and omitting it is accepted everywhere.
• Serve it on the apex or the www host. Whichever you choose, name that exact host in the header and serve the directory there directly. A redirect between the two is not followed: the current draft requires every discovery fetch to answer 200 and forbids following redirects, because the address a verifier resolved is the identity it attributes the request to. A redirect is reported back to you as the status it was.

If you do not control a domain

The well-known path needs a domain to sit on. An agent that has none can point at the key file directly instead, with type=jwks_uri on the Signature-Agent member. The file may live at any address that answers 200 over HTTPS without a redirect, including a raw file on a code-hosting service.

Signature-Agent: sig1="https://example.com/keys.json";type=jwks_uri
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");created=1787000000;expires=1787000300;keyid="<RFC 7638 thumbprint>";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:BASE64_SIGNATURE:

signature base line for that member:
"signature-agent";key="sig1": "https://example.com/keys.json";type=jwks_uri

• The body is the same JWK Set as above. No media type is required for it, because the draft does not define one for this address.
• Cover the signature-agent member keyed to your own signature label, and cover it including the type parameter. The value that goes into the signature base is "https://example.com/keys.json";type=jwks_uri, not the address on its own. Dropping the parameter is the commonest way this fails, and it fails as canonicalization-mismatch, which reads like a cryptography problem and is not one.
• What it costs. A signature checked this way proves you hold the key. It attributes no domain to you, because a bare file has no origin behind it and no signed response to bind one. The answer says so: keyDirectoryBinding comes back binding-not-checked rather than binding-proven. Use it to get your signing right. To be recognised as somebody, the well-known path on a domain is the form that carries a name.

Step 3. Prove the directory is yours

The directory response can itself be signed by each key it advertises. That proves the party serving the file also holds the private keys, which stops a copied key set being re-served under someone else's name.

Almost nobody does this, which is why doing it sets you apart. Most published directories cover only the response, not its body, so they prove the server holds a key without proving which key set it published. Covering the body is the stronger claim, and it costs one extra covered component.

Two details are easy to miss, and both cost time to find:

• The directory response is signed with the tag "http-message-signatures-directory", a different tag from the one on request signatures.
• It covers ("@authority";req), and to prove the key set rather than just possession, also content-digest over the body.

A directory that signs for a shorter time than it asks to be cached has a problem: the proof expires while the cached copy is still being served. Verifiers that check freshness at use rather than at fetch, this one included, will report the proof as absent for most of that window.

Step 4. Publish an agent card

The signature proves which domain you are. The card says what that domain is doing here: who runs the agent, why it fetches, the exact User-Agent it promises to send, where its keys live, and whom to write to. It is a small JSON file, and it is optional.

It is worth publishing anyway, because it turns a signature into an introduction. A verifier holding your card can compare what you promised with what actually arrived. The check on this site does that and reports the result. And a site owner deciding whether to serve you has something to read besides a domain name.

The address of the card is up to you, and the draft treats that address as the card's own identifier: the client_id inside it must be identical to the URL it was fetched from. There is no well-known path to look a card up at, so a verifier has to be told where yours is. The way the draft defines is the Signature-Agent header with type=cimd, pointing at the card instead of at a key directory:

Signature-Agent: sig1="https://example.com/bot-card";type=cimd

Sent that way, the card is fetched here and its claims are reported back to you. Your keys then come from the card: give it a jwks_uri pointing at your key directory, or put the keys inline in jwks. The draft allows one or the other and rejects a card carrying both. A card with neither cannot be tied to the key that signed the request, so its claims are not read.

The card is fetched with a plain GET, no redirect is followed, and any status other than 200 is treated as a failure. That is the draft's rule, and the reason is that the address is the identity. The card published by this site sits at /.well-known/web-bot-auth-card, a working example to copy the shape from.

Step 5. Sign a request

A signed request carries three headers:

GET /api/v1/connection HTTP/1.1
Host: fp.packet.guru
Signature-Agent: sig1="https://your-agent.example"
Signature-Input: sig1=("@authority" "signature-agent";key="sig1");\
                 created=1786000000;expires=1786003600;\
                 keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U";\
                 alg="ed25519";tag="web-bot-auth"
Signature: sig1=:BASE64_SIGNATURE:

• Signature-Agent says where your keys are published. Send it as a dictionary member keyed to your signature label, written sig1="https://…". A bare quoted string is the older form. It is still common on the wire, and both get read here.
• Signature-Input lists what you covered and the parameters. The tag must be "web-bot-auth" or the signature is not read as an agent signature at all.
• Signature is the signature itself.

Cover at least "@authority". Without a target component the signature is valid against any host and can be replayed elsewhere as your identity. Cover the signature-agent member keyed to your own label too, so the pointer to your keys cannot be swapped in transit. Covering "@method" and "@path" narrows it further and costs nothing.

The signature base you sign therefore looks like this:

"@authority": fp.packet.guru
"signature-agent";key="sig1": "https://your-agent.example"
"@signature-params": ("@authority" "signature-agent";key="sig1")\
 ;created=1786000000;expires=1786003600\
 ;keyid="poqkLGiymh_W0uP6PZFw-dvez3QJT5SolqXBCW38r0U"\
 ;alg="ed25519";tag="web-bot-auth"

Note the second line. A component written "signature-agent";key="sig1" is a dictionary member lookup: the value in the base is that member alone, quoted as it appears in the header, not the whole header field. Getting this wrong is the single most common way a correct-looking implementation fails.

Step 6. Check what you built

Send that request here. No body, no parameters, no account, no key. The answer describes the request that was just made.

curl https://fp.packet.guru/api/v1/connection \
  -H 'Signature-Agent: sig1="https://your-agent.example"' \
  -H 'Signature-Input: sig1=…' \
  -H 'Signature: sig1=:…:'

Read signals.agent.signature.outcome first. It carries the standard's own three verdicts: verified means the proof holds, invalid means a proof was presented and does not, and unverified means there was not enough to decide on. That last one is a statement about reach, never a finding against the caller.

When something is wrong, signals.agent.webBotAuthDetail names the one fault and the sentence saying what to change. Every reportable fault is listed, by subject, in the response reference.

The five mistakes that cost the most time

1. Signing the whole header instead of the member. See the note in step 4. Symptom: bad-signature on an implementation you are certain is right.
2. Hashing the wrong JSON for the thumbprint. Including kid, or leaving whitespace in. Symptom: key-not-in-directory even though the key is plainly there.
3. Using the wrong tag. web-bot-auth on requests, http-message-signatures-directory on the directory response. Symptom: bad-tag, or a directory proof that silently never counts.
4. A clock that is ahead. A created in the future is rejected by design, so a fast clock breaks every signature you send. Symptom: clock-ahead.
5. Naming a host that does not serve the directory. Apex versus www, or a directory published somewhere else entirely. Symptom: anything in your key directory.

What signing will and will not get you

• Identity, not trust. A valid signature proves you control the domain whose keys you published. Anyone can publish keys at their own domain, so a valid signature from a name nobody knows stays a valid signature from a name nobody knows.
• Recognition is a separate step. The large verifiers check signatures against a list of operators they have already admitted, and getting on such a list is an application, not a protocol feature. Signing correctly is a precondition for it, not a substitute.
• Paid access is starting to rest on it. Until recently a site had two answers for an automated caller, allow or block, and a signature bought nothing on its own. Networks are now offering site owners a third answer, charging per request at the edge, and those gateways can require the caller to authenticate with Web Bot Auth so that a price lands on a known account rather than on an anonymous payer. Where that is switched on, a signature that fails to verify stops being a missed nicety and becomes the difference between your own terms and whatever the default for strangers turns out to be. Worth watching if your agent will be paying for anything.
• Publishing your address ranges still helps. It is the other half of how you are recognised, and for most operators it is still the half that does the work.

Prompt: ask an agent to set signing up

Paste this into an agent that can edit files on the machine your service runs on. It works through the steps above and is written to stop at each one rather than run to the end on its own.

Set up Web Bot Auth signing for my service, following https://packet.guru/agents/guide.

Rules:
- Generate the Ed25519 key locally. It must never leave this machine, and never appear in a commit, a log or a message to anyone.
- Show me each change before you make it, and wait for my approval. That covers: the key, the key directory file, the directory-response signature, the agent card, and the signing code.
- Follow the guide's steps in order. If the guide and your own knowledge of the standard disagree, stop and show me the difference. Do not pick one on your own.

When everything is in place, verify it rather than assuming it worked: send one signed request to https://fp.packet.guru/api/v1/connection and report back
- whether the signature verified,
- the exact fault and fix sentence if it did not,
- what my key directory proved about its own key set,
- and what is left to do.

Do not tell me it works until that call says so.