WEB
Agent Readiness Checker
Check what AI agents and crawlers get from a website
Input
Reads robots.txt for each AI crawler, the home page without JavaScript, llms.txt, the sitemap, API and agent catalogs, MCP and A2A cards and a Web Bot Auth key directory.
About Agent Readiness Checker
Is my website AI ready?
A website is AI ready when AI agents, AI crawlers and language models can find it, read it and use what it publishes without a person at a browser. They read a site differently from people:
- Each crawler follows the robots.txt rules written for its own name.
- Many fetch a page without running JavaScript.
- Some look for files written for machines, such as llms.txt, an API catalog or an MCP server card.
The Agent Readiness Checker reads all of this in one pass and returns an agent readiness report: two scores, cards with the details and a list of what to fix first.
Two scores
- Access. Can AI agents and crawlers get in and read the site? Points go when robots.txt closes the site or blocks an AI search bot or a search engine, when the certificate is not valid, when the server or the home page answers with an error, when the home page leads to a sign-in page or has no text without JavaScript, or when a noindex hides it from search.
- Standards. Are the files the site publishes for agents correct? Points go for every rule of a standard those files break, such as a wrong media type, a typo in the sitemap namespace or a link that leads nowhere.
Both scores start at 100. The colour of a score and of a card says what to do:
- 90 to 100, green. Agents get in and find correct files. A green card was checked and is fine, a block the site owner chose included.
- 60 to 89, orange. Worth a look. Part of the agents may be kept out, or a file skips a recommendation of its standard.
- Below 60, red. Agents miss much of the site, or a file breaks a rule of its standard. Fix first lists the changes that bring the score back, the costliest first.
- Grey. Nothing to check. A grey card is something the site does not publish, which costs nothing. "No files to check" under Standards means the site publishes no files for agents, which is not a fault: many sites need none. "Not measured" means the check could not run, usually because bot protection refused the scanner.
Some choices cost nothing: blocking AI training crawlers, leaving out a file the site has no use for, serving no Markdown copy.
What the checker looks at
- Bot access. AI search bots such as OAI-SearchBot and PerplexityBot, AI training crawlers, assistants that fetch a page for a user, search engines, browser agents and other bots, each against its own robots.txt group, and the robots.txt file itself with its Content-Signal lines.
- What agents see. The home page as the server sends it, without JavaScript: the text, the title and H1, JSON-LD structured data, noindex and snippet rules, a Markdown copy.
- Files for agents. llms.txt and its links, the sitemap, an API catalog (RFC 9727), Agent Skills and AI catalogs, and whether these files agree with each other.
- Agent protocols. An MCP server card and its sign-in, an A2A agent card, a Web Bot Auth key directory.
Each card shows what was found. A click opens the details: what the card checks, why a finding matters, how to fix it and what the site answered.
For the full report on one file, open the Server Card Checker or the Directory Verifier. To tell a real GPTBot from a fake one in server logs, use the Crawler Verifier. Which AI crawler feeds which AI product, and why ChatGPT or Perplexity may not see a site, is explained in how AI crawlers see your website.
How to use Agent Readiness Checker
- Enter the domainType a domain or any address on the site. The checker reads the site the address leads to and says so when it redirects.
- Press Check siteThe report fills in section by section as each check finishes.
- Read the two scoresAccess shows whether AI agents and crawlers can get in and read the site, Standards whether the files it publishes for agents are correct. A score of 90 or more means nothing the checker found keeps agents out and the published files follow their standards.
- Start with Fix firstEach line names a change and the points it costs, the costliest first. A click opens its card.
- Open a card, then check againA card says what it checks. Open it to see what was found, why it matters and how to fix it. Share sends a link with both scores. After a fix, run the check again to see the scores move.
Frequently Asked Questions
- Enter the domain above and press Check site. The checker reads robots.txt for every known AI bot, the home page without JavaScript and the files agents look for, then gives two scores and a list of what to fix first.
- Let the AI search bots and search engines you want in through robots.txt, put the main content into the HTML instead of building it with JavaScript, and publish machine-readable files where they fit, such as llms.txt or an API catalog. A site that does this is AI friendly for crawlers and agents alike. Fix first lists the changes in the order they matter.
- The Bot access group shows which AI search bots, assistants and search engines robots.txt lets in, and what a block means for each operator. By OpenAI's own documentation, for example, a site that blocks OAI-SearchBot is not shown in ChatGPT search answers.
- Block it if the content should not be used to train AI models. By OpenAI's documentation, disallowing GPTBot in robots.txt keeps a site out of model training and does not take it out of ChatGPT search, which uses OAI-SearchBot. Anthropic separates ClaudeBot and Claude-SearchBot the same way, and Google-Extended controls Gemini training without affecting Google Search. Blocking training crawlers lowers neither score here.
- Both start at 100 and lose points for each problem. Access counts what keeps AI agents out or leaves them an empty page, Standards counts the rules broken in the files the site publishes for agents. Each card shows the points it took, and Fix first lists them all, the costliest first. Blocking training crawlers and leaving out a file the site has no use for cost nothing. The scores follow the published standards and change when the standards do.
- Bot protection answered with a challenge page instead of the site, so what crawlers get could not be seen. Allow the User-Agent packetguru-agent-readiness for a few minutes in the protection settings and check again. Unverified AI agents may be refused the same way.
- Only the files listed above, once per check, when a person starts it: robots.txt, the home page, llms.txt and a sample of its links on the same site, the sitemap and files under /.well-known/, and the MCP server a published card names. It follows robots.txt rules that name packetguru-agent-readiness. Rules written for User-agent: * do not apply to it, because a person starts every check. To keep it out, add User-agent: packetguru-agent-readiness and Disallow: / to robots.txt.