What September shows: three findings from the radar
Berger+Team · Oct 2, 2026
Topic: Research
In September 2026, roughly one observable domain in seven or eight served an llms.txt. Files declaring usage reservations towards AI stayed below one percent, and files for AI agents came in at about one per thousand at most. Those are the three observations the published data of the ai-discovery radar can actually carry. Here they are with denominator, measurement date and working — and with what they do not say.
How the radar counts the adoption of machine-readable files is explained in the method report; we won't repeat it here. Just this much: what counts is whether a file can be retrieved and is formally valid. Not whether any system reads it.
The basis
Both data files come from the same run, 1 to 3 September 2026, computed under ruleset 0.3.2:
- Panel: 1,000 fixed domains, 853 of them reachable. The series that is measured the same way every month.
- Monthly run: 10,667 domains (block a with 9,667 plus the panel), 9,206 of them reachable. A larger base, narrower intervals.
Every route — that is, every path such as /llms.txt — has its own denominator n: only the hosts where we could actually look. A host that turns the request away, forbids it in robots.txt or does not answer is left out. In the panel, n therefore ranges from 641 to 761; in the monthly run from 6,725 to 8,221. Every percentage below comes with its 95% interval, and every one can be recomputed from the numerator and denominator in the files.
Observation 1: llms.txt sits just above 13% — in two separate samples
In September, 13.45% of the observable panel domains served an llms.txt: 90 of 669 (interval 11.1–16.2%). An llms.txt is a text file at the root of a website that tells language models, in short form, which content matters. It is an industry proposal, not an adopted standard (LLMs.txt in the glossary).
The figure holds up because it repeats. The panel is part of the monthly run. Take it out and you are left with the 9,667 domains of block a — where 845 of 6,404 observable hosts served an llms.txt: 13.19% (12.4–14.0%). Two separate draws from the same frame, the same picture: roughly one domain in seven or eight.
Two details from the full monthly run:
- The long version
/llms-full.txtwas served by 303 of 7,061, or 4.29%. - The location is unambiguous: 935 hits at the root (
/llms.txt), 2 in the/.well-known/llms.txtfolder (of 7,046).
What the figure does not mean: that 13% of "websites" are ready for AI. The sample is drawn from rankings of popular domains (Tranco and the Chrome UX Report), not from the whole web. And a file that exists says nothing about whether it is read, kept current, or achieves anything.
Observation 2: usage reservations in a dedicated file stay below one percent
Three files declare, in machine-readable form, what content may be used for. In the September monthly run, all three stayed below one percent of observable domains:
/rsl.xml, licence terms for the use of content: 53 of 6,795 = 0.78% (0.6–1.0%)/.well-known/tdmrep.json, a reservation against text and data mining: 28 of 7,282 = 0.38% (0.3–0.6%); valid declarations via HTTP header are included/ai.txt, content withheld from AI training: 19 of 7,053 = 0.27% (0.2–0.4%); in the/.well-known/ai.txtfolder it was 4 of 7,090
For comparison, each with its own denominator: 6,966 of 8,221 served a robots.txt, 84.73%. Each of the three reservation files is more than a hundred times rarer. A ranking among the three small values, however, is not something the data supports, because every row has its own denominator.
In the panel alone there are 2 to 5 hits per file. That is enough for "rare", not for a share worth quoting. Only the monthly run carries this observation.
Worth keeping in mind: robots.txt, ai.txt and TDMRep are signals, not technical locks. What a TDM reservation does and does not do is in the glossary.
Observation 3: files for AI agents sit between zero and about one per thousand
All nine routes aimed explicitly at AI agents and automated clients — agent cards, MCP descriptions, ai-catalog.json, plugin manifests and x402 — were at 0.11% or below in the monthly run. A selection:
/.well-known/ai-plugin.json: 8 of 7,038 = 0.11%/.well-known/agent-card.json: 5 of 7,194 = 0.07%/.well-known/mcp.json: 5 of 7,181 = 0.07%/.well-known/ai-catalog.json: 2 of 7,190 = 0.03%/.well-known/x402.json: 0 of 7,167
General interface files such as the API catalogue (0.29%) or the OAuth descriptions (about 5%) are not counted here, because they do not serve AI agents specifically.
This is where it becomes concrete why a zero in the panel is not an absence. In the September panel, ai-catalog.json stood at "0.0%", measured on 675 hosts. The 95% interval reaches up to 0.6%: with 675 hosts you cannot say more than "at most about six in a thousand". The run ten times the size narrows that to 0.0–0.1%. Rare, but not nothing.
It works the other way too. /.well-known/agent.json stands at 0.15% in the panel, and that is exactly one hit. A single host moves a panel share by about 0.15 percentage points — for rare files, the panel is too small to give shares.
What the data cannot tell you
Four questions come to mind, and the published files answer none of them:
- Is it growing? The panel was measured in August too, but under the older ruleset 0.3.1. For all 33 routes measured in both months, the intervals overlap. The first regular month-on-month comparison comes with the October run.
- Are operators blocking AI crawlers in robots.txt? The radar counts whether a robots.txt exists, not what it says. How many of them address the crawlers of AI providers is not in the published data.
- What about South Tyrol? The sample is drawn partly from top lists for Italy, Germany, Austria and Switzerland, but figures per country are not published. Nothing follows from it for businesses in South Tyrol.
- Does it help? Availability is not use. Whether AI systems fetch, read or follow these files is not something the radar measures.
What it means for your business
The numbers work against two lines you hear a lot. "Everyone has that by now" is not even true of robots.txt (84.73%), and after it and the sitemap (53.92%) the list drops below 14%. "Nobody uses that" rarely holds either: in the large run, only 4 of the 34 routes had no hit at all.
What the numbers do not do: tell you which file you need. That depends on what you want to tell machines. A table of contents for language models is one thing, a reservation against AI training another, an offer to agents a third. Adoption is no yardstick for that — in either direction.
Disclosure: btlabs Core ships several of the measured files out of the box, among them llms.txt, ai-catalog.json and a TDM reservation. Why we built them in rather than bolting them on later is in What is already built in. That is exactly why this post only contains numbers anyone can recompute from the files.
If you read one of these numbers differently from us, write to us via the form on the radar page. We are happy to run the numbers with you.
Sources and status
Measurement window 1–3 September 2026, ruleset 0.3.2, frame 2026-Q3. Data files panel-2026-09 and monat-2026-09-block-a in the public repository, dataset DOI 10.5281/zenodo.22178282, data licensed CC BY 4.0. The block-a-without-panel values are our own: numerator minus numerator, denominator minus denominator, because under the ruleset the panel is a subset of the monthly run. Their intervals are Wilson intervals, as in the radar.
Frequently asked questions.
Does 0 % in the panel mean nobody uses the format?
No. For rare routes, a zero across 1,000 domains is a sample-size floor: in the ten times larger block of the same month only 4 routes sit at zero instead of 16. From the October 2026 run the panel is therefore widened to 10,000 domains.
Do I absolutely need an llms.txt?
It is not a Google ranking signal (Google says so itself) — but it takes minutes to create and is useful as orientation for AI agents. A cheap addition, not a miracle cure.
How widespread it really is, we measure ourselves: as of October 2026, 14.1% of the domains probed serve an llms.txt. The measurement is open in the ai-discovery radar.
Can I stop AI from training on my content — without becoming invisible?
Yes — with a TDM opt-out (TDMRep): a machine-readable usage reservation that forbids text and data mining of your content for AI training. The legal basis is the EU framework, which obliges providers of large AI models to respect such reservations. Reading and citing by AI answer systems stays allowed — you remain visible and citable. You set the rule once in machine-readable form, instead of leaving it to a legal text alone. To be honest about it: this is a declaration, not a technical block — anyone who ignores it is not stopped by it. But the reservation is what makes it possible for you to invoke your rights at all.
What does the ai-discovery-radar measure?
The ai-discovery-radar measures how many websites in a frozen sample of 30,000 domains (a third of it each month, plus a fixed panel) actually serve 34 catalogued machine-readable discovery and consent files — robots.txt, llms.txt, TDMRep or agent cards, for example. What is counted is availability, not use.
