What an AI agent finds on your website: a five-step self-check
Florian Berger · Oct 26, 2026
Topic: Machine-readable website
When you ask an AI assistant to find a business, you send out a program that reads differently from a person: it first checks a few agreed addresses, then works its way through the pages. Which addresses those are, and what should be there, you can check on your own website in a few minutes — with nothing but a browser. Here is the tour, in the order I check things myself.
An AI agent is a program that searches, reads and acts on its own on someone's behalf: answering a question, comparing offers, sending an enquiry. To an AI agent, what matters is not how a page looks but what a machine can read from it. An agent-readable website therefore starts not with design but with these five places.
Step 1: robots.txt — the house rules
Type your address and add /robots.txt, for example yourbusiness.com/robots.txt. The file is described as an Internet standard (RFC 9309) and tells automated visitors which areas they may fetch.
What to look for:
- It exists and does not accidentally block everything (
Disallow: /for all visitors). - It names the sitemap, so programs find all your pages.
- If you want, it separates training from search. The large providers send different visitors for each: at OpenAI,
GPTBotcollects for training andOAI-SearchBotfor search; at Anthropic they areClaudeBotandClaude-SearchBot(OpenAI, Anthropic). Excluding only training keeps you findable in AI search.
Important: robots.txt is a request, not a lock. Reputable providers respect it; technically, the file prevents nothing. More in the glossary under robots.txt.
Step 2: llms.txt — the summary for language models
Add /llms.txt. It is a text file that describes, in plain Markdown, who you are and which pages matter. It is an industry proposal (llmstxt.org), not an adopted standard, and whether a given system reads it is open. What it can and cannot do is in the glossary under llms.txt.
What to look for: it exists, is not empty and matches your pages — name, offer, contact, the most important subpages. An llms.txt with an unfilled template left in it helps no one.
Step 3: The signposts under /.well-known/
Under the path /.well-known/ sit files that tell programs which interfaces a website offers. Three matter to agents:
/.well-known/api-catalog— a directory of a website's interfaces, standardised in RFC 9727./.well-known/mcp.json— describes access via the Model Context Protocol, through which an assistant can query content or, with permission, edit it. A community convention, not a standard./.well-known/security.txt— whom to contact about a security flaw (RFC 9116).
If you find nothing here, that is normal, not an omission. But don't create empty files: a signpost that points to no interface is worse than none.
Step 4: A text version of the page
Between a machine and your actual text there are often menus, banners and scripts. A clean text version, usually in Markdown, gives it only the content. You can check this in the source of a subpage (right-click, "View page source"): if there is a reference with rel="alternate" and type="text/markdown", the page offers a text version. If not, the program reads the HTML — that works too, just with more clutter. How an AI breaks a page into pieces I described in How an AI reads your website.
Step 5: The facts in structured form
Name, address, opening hours, services: these belong not only in the text but also as structured data following Schema.org. Whether your page serves any, and which, is shown by the Schema Markup Validator: enter the address, read the result. Which details really count is covered in Which facts should a machine be able to read about your business?
How rare this still is
With the ai-discovery-radar I measure every month which of these files websites actually serve. In the September 2026 run (10,667 domains, 9,206 of them reachable) it looked like this:
- robots.txt: 84.7% of the domains that could be checked
- llms.txt: 13.2%
- api-catalog under RFC 9727: 0.29%
/.well-known/mcp.json: 5 of 7,181 domains
Only what is actually served as a valid file is counted; error pages and placeholders drop out. What is measured is presence, not use: whether an agent reads a file, these numbers do not say. The findings from the September run are in a separate post, with denominators and method.
What I take from it
Steps 1 and 5 are the basics: they decide whether a machine reads you correctly at all. Steps 2 and 4 cost little if the website generates them from existing content instead of someone maintaining them by hand. Step 3 is only worth it if a real interface sits behind the signpost.
None of these files makes an AI recommend you. They make sure it can read you correctly when it looks.
On btlabs.dev you can look at all of them, for example robots.txt, llms.txt, the api-catalog and mcp.json. btlabs Core generates them from the same maintained data as the pages, and every page names the most important ones in its response.
Sources
- RFC 9309 — Robots Exclusion Protocol
- RFC 9727 — api-catalog
- RFC 9116 — security.txt
- llmstxt.org — the llms.txt proposal
- OpenAI — crawlers and what they do
- Anthropic — crawlers and what they do
- Berger, F. (2026): AI Discovery Radar, run monat-2026-09-block-a, ruleset 0.3.2. Zenodo, doi.org/10.5281/zenodo.22178282, CC BY 4.0
Frequently asked questions.
Do I absolutely need an llms.txt?
It is not a Google ranking signal (Google says so itself) — but it takes minutes to create and is useful as orientation for AI agents. A cheap addition, not a miracle cure.
How widespread it really is, we measure ourselves: as of October 2026, 14.1% of the domains probed serve an llms.txt. The measurement is open in the ai-discovery radar.
How do I tell whether my brand is machine-readable?
Open your stylesheet and ask whether a stranger could tell from it which colour is your brand colour — not which value sits where, but which role it plays. If your brand colour is called --colour-3 in the code, or sits unnamed beside two identical greys, a machine cannot find it.
What does my business gain if AI assistants can talk directly to my website?
Your details are ready exactly where people increasingly ask. Instead of an AI scraping your page and guessing, it reads exactly the content you've provided through an open interface — MCP (Model Context Protocol): services, opening hours, contact details. The aim: fewer incorrect statements about your business and correctly sourced citations — whether an AI uses this route is up to its provider. You stay in control — you decide which content is accessible, and every access is logged.
How do I get MCP access for my website, and what's included?
We set up MCP access for you whenever you want it — it's part of expanding your website, not a mandatory component. This lets you, or an AI assistant of your choice, work directly with your own website: maintaining content, querying your own data, automating recurring tasks. We agree together on what can be accessed — opening hours, services or contact details, for example — and you always keep an overview of who accesses what and when.
The effort is manageable: one-off setup, then the access runs alongside your website. It's most worthwhile once you update content regularly or have recurring questions about your own data — then an AI can take over the routine work, on request.
