Mint line drawing on black: two identical stepped platforms with an almost identical pattern of glowing dots, joined by a dotted measuring bracket.

Cloudflare and AI training: what the radar shows in November

Berger+Team · Nov 18, 2026

Topic: Research

On many websites, Cloudflare writes a section into the robots.txt that can exclude AI training. In November 2026 it excluded training on 5.9% of the 1,000 popular websites we check every month. Among the websites running through Cloudflare, it was about one in seven. Usually it only blocks crawlers that collect text to train AI models. Search in ChatGPT or Perplexity and fetches made on behalf of users stay open. On a small share of sites it blocks more.

How to check your robots.txt

Three steps are enough, no technical knowledge needed:

  1. Open your website's address in the browser with /robots.txt at the end, for example [your-business].com/robots.txt.
  2. Look for the line "# BEGIN Cloudflare Managed content". If it is there, the section comes from your Cloudflare settings. If it contains "# Training crawlers" or "User-agent: GPTBot" with "Disallow: /", it excludes AI training.
  3. Check whether "# AI Agents" or "# AI Search crawlers" appear below it. If so, you are also asking AI assistants that open your page for a user, and the search of AI services, to stay away.

Whether you want that is your decision. You can change it in Cloudflare.

What the entry blocks and what it does not

The "# Training crawlers" section lists crawlers such as GPTBot, ClaudeBot, CCBot, Google-Extended and Bytespider. They collect content to train AI models. Googlebot is not affected, and neither is Google Search with its AI Overviews. Google-Extended controls training for Gemini and whether Gemini services may use content as a basis for answers.

In the larger panel of 10,000 domains, 420 of 439 entries in the new form contained only this section in November 2026. 19 also blocked AI agents, 6 of them AI search as well.

A robots.txt is a request to crawlers, not a technical block. Whether Cloudflare also turns the same crawlers away, we do not measure.

The numbers: the same 1,000 websites, three months

We count the websites whose robots.txt we could read, around 700 per month, and among them those where Cloudflare's section excludes AI training. The 95% interval is in brackets.

  • September 2026: 48 of 704, 6.8% (5.2–8.9%)
  • October 2026: 3 of 700, 0.4% (0.1–1.3%)
  • November 2026: 41 of 695, 5.9% (4.4–7.9%)

The rise from October to November is real: the intervals do not overlap. Over three months it is a return to the September level. The sample of 10,000 domains shows the same picture: 0.3% in October (21 of 6,896, 0.2–0.5%) and 6.4% in November (442 of 6,951, 5.8–7.0%).

All 41 entries come from websites that serve their robots.txt through Cloudflare. Of these 293 websites, the section excluded training on 41 in November, 14.0% (10.5–18.4%), about one in seven. In October it was 3 of 289, 1.0% (0.4–3.0%). On another 9 websites there was only a legal preamble from Cloudflare without any rule; the preamble itself excludes nothing.

In the larger panel of 10,000 domains the share is similar: of 2,901 websites that serve their robots.txt through Cloudflare, the section excluded training on 442 in November, 15.2% (14.0–16.6%), about one in seven. Without Cloudflare's section, another 826 websites there exclude training crawlers with their own lines, some of them by blocking all crawlers. That is more than through the section.

Where the movement comes from

Cloudflare had announced it would replace the old form of the entry on 15 September 2026. The drop in October fits that. The new form, with the line "# Training crawlers", has been spreading since around the end of October: in November, all 41 entries that exclude training had it. It matches a feature Cloudflare introduced on 21 August 2026, which writes the entries separately for search, agents and training. That the new form comes from there is likely, but not proven.

Whether a site owner chose the entry or it arrived as a default, we do not measure. The numbers show how widespread it is, not how many businesses decided against AI training.

What stayed the same

Apart from the Cloudflare entry, no file changed measurably in November. llms.txt is at 16.6% (110 of 661, 14.0–19.7%), after 13.5% in September (90 of 669, 11.1–16.2%). The direction is up, but the gap is still within the noise. robots.txt, sitemap.xml and security.txt stayed flat; files made specifically for AI agents remain isolated cases.

How we measured

We measured the same 1,000 domains, drawn from rankings of popular websites, on 1–3 September, 1–5 October and 1–5 November 2026. The comparison figures come from the panel, extended to 10,000 domains since October. The monthly shares per file are in the dataset v2026.11 (DOI 10.5281/zenodo.23998918). We counted the Cloudflare numbers from the raw measurement data. The domains are not a random sample of the web. Whether a crawler follows the robots.txt, the radar does not measure. What September showed on its own is in What September shows.

All monthly numbers, with interval and dataset, are on the radar page.

Frequently asked questions.

What does the ai-discovery-radar measure?

The ai-discovery-radar measures how many websites in a frozen sample of 30,000 domains (a third of it each month, plus a fixed panel) actually serve 37 catalogued machine-readable discovery and consent files — robots.txt, llms.txt, TDMRep or agent cards, for example. What is counted is availability, not use.

Do I absolutely need an llms.txt?

It is not a Google ranking signal (Google says so itself) — but it takes minutes to create and is useful as orientation for AI agents. A cheap addition, not a miracle cure.

How widespread it really is, we measure ourselves: as of November 2026, 14.8% of the domains probed in the 10,000-domain panel serve an llms.txt. The measurement is open in the ai-discovery-radar.

Does 0 % in the panel mean nobody uses the format?

No. For rare routes, a zero across 1,000 domains is a sample-size floor: in the ten times larger block of the same month only 4 routes sit at zero instead of 16. From the October 2026 run the panel is therefore widened to 10,000 domains.

Next step

Want to look at it together?

No pitch, no standard package — an honest take on your project. A reply, usually within 24h.

Start a conversation →