Agent Readiness Checker

Check what AI agents and crawlers get from a website

Input

Reads robots.txt for each AI crawler, the home page without JavaScript, llms.txt, the sitemap, API and agent catalogs, MCP and A2A cards and a Web Bot Auth key directory.

About Agent Readiness Checker

Is my website AI ready?

A website is AI ready when AI agents, AI crawlers and language models can find it, read it and use what it publishes without a person at a browser. They read a site differently from people:

  • Each crawler follows the robots.txt rules written for its own name.
  • Many fetch a page without running JavaScript.
  • Some look for files written for machines, such as llms.txt, an API catalog or an MCP server card.

The Agent Readiness Checker reads all of this in one pass and returns an agent readiness report: two scores, cards with the details and a list of what to fix first.

Two scores

  • Access. Can AI agents and crawlers get in and read the site? Points go when robots.txt closes the site or blocks an AI search bot or a search engine, when the certificate is not valid, when the server or the home page answers with an error, when the home page leads to a sign-in page or has no text without JavaScript, or when a noindex hides it from search.
  • Standards. Are the files the site publishes for agents correct? Points go for every rule of a standard those files break, such as a wrong media type, a typo in the sitemap namespace or a link that leads nowhere.

Both scores start at 100. The colour of a score and of a card says what to do:

  • 90 to 100, green. Agents get in and find correct files. A green card was checked and is fine, a block the site owner chose included.
  • 60 to 89, orange. Worth a look. Part of the agents may be kept out, or a file skips a recommendation of its standard.
  • Below 60, red. Agents miss much of the site, or a file breaks a rule of its standard. Fix first lists the changes that bring the score back, the costliest first.
  • Grey. Nothing to check. A grey card is something the site does not publish, which costs nothing. "No files to check" under Standards means the site publishes no files for agents, which is not a fault: many sites need none. "Not measured" means the check could not run, usually because bot protection refused the scanner.

Some choices cost nothing: blocking AI training crawlers, leaving out a file the site has no use for, serving no Markdown copy.

What the checker looks at

  • Bot access. AI search bots such as OAI-SearchBot and PerplexityBot, AI training crawlers, assistants that fetch a page for a user, search engines, browser agents and other bots, each against its own robots.txt group, and the robots.txt file itself with its Content-Signal lines.
  • What agents see. The home page as the server sends it, without JavaScript: the text, the title and H1, JSON-LD structured data, noindex and snippet rules, a Markdown copy.
  • Files for agents. llms.txt and its links, the sitemap, an API catalog (RFC 9727), Agent Skills and AI catalogs, and whether these files agree with each other.
  • Agent protocols. An MCP server card and its sign-in, an A2A agent card, a Web Bot Auth key directory.

Each card shows what was found. A click opens the details: what the card checks, why a finding matters, how to fix it and what the site answered.

For the full report on one file, open the Server Card Checker or the Directory Verifier. To tell a real GPTBot from a fake one in server logs, use the Crawler Verifier. Which AI crawler feeds which AI product, and why ChatGPT or Perplexity may not see a site, is explained in how AI crawlers see your website.

How to use Agent Readiness Checker

  1. Enter the domain
    Type a domain or any address on the site. The checker reads the site the address leads to and says so when it redirects.
  2. Press Check site
    The report fills in section by section as each check finishes.
  3. Read the two scores
    Access shows whether AI agents and crawlers can get in and read the site, Standards whether the files it publishes for agents are correct. A score of 90 or more means nothing the checker found keeps agents out and the published files follow their standards.
  4. Start with Fix first
    Each line names a change and the points it costs, the costliest first. A click opens its card.
  5. Open a card, then check again
    A card says what it checks. Open it to see what was found, why it matters and how to fix it. Share sends a link with both scores. After a fix, run the check again to see the scores move.

Frequently Asked Questions

Enter the domain above and press Check site. The checker reads robots.txt for every known AI bot, the home page without JavaScript and the files agents look for, then gives two scores and a list of what to fix first.