The ai-discovery radar

What gets measured

The radar checks which machine-readable files a website makes available to machines: robots.txt, which for thirty years has governed where automated visitors may go. llms.txt, a table of contents written for language models. ai-catalog.json, which describes the content and services a domain offers AI agents. 37 such routes are in the running measurement.

Then there is what we deliberately do not measure. Another 23 routes sit under observation: known to us, but not yet widespread enough to justify a request on every host in the sample. And 49 we examined and rejected, with a reason recorded entry by entry. A catalogue does not get better by admitting every format someone happens to mention.

Only public configuration files meant for machines are fetched — no page content, no images, no text. The measurement obeys each domain's robots.txt and identifies itself by name.

What the panel shows

As of November 2026, measured across a stratified sample of 8,603 reachable domains:

• 84.9% serve a robots.txt. The oldest standard is the only one nearly everyone honours.
• 14.8% serve an llms.txt. The format is young, and adoption reflects that.
• 0.1% serve an ai-catalog.json. Of the 37 routes probed, 5 returned not a single response across the entire panel.

Those zeros are not a measurement error. They are the result. Many of these formats are discussed as though they were established. On the open web, they are not.

Since October 2026 we measure a panel of 10,000 sources every month; the figures above are the latest run. The earlier panel of 1,000 sources continues as its own series until December. Comparisons are made only within one series. In November 2026 the panel is stable across the board compared with October; an arrow appears only when the 95% intervals of the two months do not overlap. The August 2026 exploration series across 16,554 domains remains documented as the baseline in the GitHub repository.

The measurement in numbers

• Panel: 10,000 sources, the same every month; in November 2026 8,603 reachable.
• Frame: 30,000 sources; each month a third is added, together with the panel about 16,800 sources per month.
• Routes: 37, under ruleset 0.4.0.
• 1,000 series: the same 1,000 sources since August 2026, as a separate series until December 2026.
• First full frame (September to November 2026): 25,851 reachable organisations, as a separate file in the dataset.

Why we publish this

We build websites meant to be found by people and by AI systems alike. That work needs numbers instead of assumptions: which route actually carries weight today, and which is a bet on tomorrow? Anyone who does not measure is selling guesswork.

So the method is public — sample, ruleset, confidence intervals, and the routes that came back at zero stay in the table. Anyone who doubts one of these figures can recompute it. That is the whole point.

The full method — classification model, denominator rule, sampling design, verification procedure and the measurement pitfalls that cost us data ourselves — is in the Technical Report v1.0 (Zenodo, DOI 10.5281/zenodo.22769680, CC BY 4.0). Anyone citing a share should have read it first.

Responsible for measurement, analysis and report: Florian Berger, Berger+Team, Bolzano · ORCID 0009-0008-5780-1147

Who does not want to be measured

No questions, no reason needed. An email with the domain is enough, or a single line in your own robots.txt. Excluded domains are skipped before any request is made.

View the radar on GitHub