# The ai-discovery radar

> An open monthly measurement of which discovery files websites make available to machines — robots.txt, llms.txt, ai-catalog.json and more. Measured, not estimated.

## btlabs Core · open measurement

## The ai-discovery radar

We measure what machines actually find on websites — and publish it, including the uncomfortable parts. A great deal is claimed about AI visibility and very little is measured. Yet the routes a website offers machines, so that it can be found and read correctly, can simply be counted. That is what we do: monthly, in public, with the method laid open.

## What gets measured

The radar checks which standardised files a website makes available to machines: robots.txt, which for thirty years has governed where automated visitors may go. llms.txt, a table of contents written for language models. ai-catalog.json, which describes the content and services a domain offers AI agents. **33** such routes are in the running measurement.

Then there is what we deliberately do **not** measure. Another **12** routes sit under observation: known to us, but not yet widespread enough to justify a request on every host in the sample. And **49** we examined and rejected, with a reason recorded entry by entry. A catalogue does not get better by admitting every format someone happens to mention.

Only public configuration files meant for machines are fetched — no page content, no images, no text. The measurement obeys each domain's robots.txt and identifies itself by name.

## What the first run shows

As of **August 2026**, measured across a stratified sample of **853** reachable domains:

- **77.0%** serve a robots.txt. The oldest standard is the only one nearly everyone honours.
- **10.2%** serve an llms.txt. The format is young, and adoption reflects that.
- **0.0%** serve an ai-catalog.json. Of the 33 routes probed, 16 returned not a single response across the entire panel.

Those zeros are not a measurement error. They are the result. Many of these formats are discussed as though they were established. On the open web, they are not.

## Why we publish this

We build websites meant to be found by people and by AI systems alike. That work needs numbers instead of assumptions: which route actually carries weight today, and which is a bet on tomorrow? Anyone who does not measure is selling guesswork.

So the method is public — sample, ruleset, confidence intervals, and the routes that came back at zero stay in the table. Anyone who doubts one of these figures can recompute it. That is the whole point.

## Who does not want to be measured

No questions, no reason needed. An email with the domain is enough, or a single line in your own robots.txt. Excluded domains are skipped **before** any request is made.

[View the radar on GitHub](https://github.com/flober81/ai-discovery-radar)

## Let’s talk straight

## Questions about the measurement?

Write to us — about the method, about a single figure, or if your domain should not be measured.

- Reply usually within 24 hours
- Opt out with no questions asked
- Method fully disclosed

No obligation · Straight answers · No bot

- Company name *
- Your name *
- Email address *
- What’s it about? *
- **Yes, run the free AI-readiness check for me.** It analyses — with no obligation — what AI models like ChatGPT currently know about my business, and where my blind spots are.
- I accept the [Privacy policy](https://btlabs.dev/en/privacy). *

→ Write to us

---
Source: https://btlabs.dev/en/ai-discovery-radar
Last-Modified: 2026-08-15T10:09:31.129Z
Languages: [de](https://btlabs.dev/llms/de/ai-discovery-radar) · [it](https://btlabs.dev/llms/it/ai-discovery-radar)
See also: [llms.txt](https://btlabs.dev/llms.txt) · [ai.txt (Policy)](https://btlabs.dev/ai.txt) · [identity.json](https://btlabs.dev/identity.json)