Visible to AI, protected from training: data sovereignty with TDMRep and the EU AI Act

There's an uncomfortable side effect when you make your website perfectly AI-readable: the very same clean structure that makes you visible in AI answers also turns you into the ideal target for data scrapers training AI models at scale. This raises a fundamental question every business has to answer: should your hard-earned expertise end up for free in the training material of global tech corporations?

The good news: you don't have to choose between visibility and data sovereignty. You can have both — if you use the right tools.

robots.txt isn't enough

robots.txt isn't enough because it's both a blunt instrument you'd have to maintain constantly and a way to shut an AI bot out completely at the network level – not just from training, but also from the real-time retrieval that answers a current customer query. The obvious reflex is still to lock out AI bots via the robots.txt.

First, it's a blunt instrument: you'd have to manually maintain hundreds of constantly changing bot names — GPTBot, ClaudeBot, PerplexityBot and their successors. A race you can't win.

Second, and more serious: if you block an AI bot via the robots.txt, you shut it out completely at the network level. This prevents not only the harvesting of training data, but also the real-time retrieval the AI uses to answer a current customer query. The result is total invisibility in the machine channel — you throw away visibility to prevent training. That's not protection, that's self-exclusion.

The clean separation: discovery yes, training no

What you need is granular control instead of a firewall: “Visible in AI search and for recommendations: yes. Mass model training with my data: no.”

That's exactly what the TDM Reservation Protocol (TDMRep) by the W3C Community Group delivers. It draws its force directly from European law:

• Article 4 of the EU Copyright Directive (CDSM Directive 2019/790) permits Text and Data Mining for commercial purposes in principle — unless the rightsholders have expressly reserved it by machine-readable means.
• The EU AI Act (Regulation 2024/1689) requires under Article 53 that providers of large AI models respect these machine-readable rights reservations — regardless of which country the training takes place in.

Your opt-out is therefore not just a request, but a legally anchored signal.

How it works

TDMRep works with two simple specifications:

• tdm-reservation — a yes/no value that declares whether you reserve the rights for AI training.
• tdm-policy — a reference to machine-readable terms (for example: use only with consent or against payment).

It can be implemented in three ways that do not interrupt regular AI search traffic:

• Central: a file at /.well-known/tdmrep.json with site-wide rules.
• Dynamic: the values directly in the HTTP header of the server response.
• Granular: as a meta tag in the HTML of individual pages, for example to protect a specific specialist article in a targeted way.

The decisive point

The W3C working group makes it explicitly clear: classic search and discoverability functions — explicitly including in AI search engines — do not fall under the training opt-out. You thereby decouple two things that robots.txt lumps together: discovery (you stay visible) and extraction (your knowledge does not flow into model training).

This is the practical core of data sovereignty: take part in the AI web without giving away your most valuable asset.

Why this matters for your business

This matters because your real first-party data, your local expert knowledge, and your concrete case examples — exactly what AI cannot invent — are the value worth protecting and playing deliberately, instead of giving it away into training material. In a world where AI generates generic text endlessly, the value of standard content drops to zero. What stays valuable? Exactly what an AI cannot invent: your real first-party data, your local expert knowledge, your concrete case examples. That's your moat — and you don't give it away into training material, you protect it and play it deliberately.

For a specialized business in South Tyrol this is a real opportunity: your concrete knowledge about your region, your customers, your niche is exactly the raw material AI searches need — and that you should provide on your own terms.

It's exactly this balance that btlabs Core builds in by default: from a single central data base, the visible website, the machine-readable data and the legally binding training reservations are generated in parallel — on your own EU server, with full data sovereignty.

Honest assessment: these standards are young, and not every AI provider respects them voluntarily yet. But they have a legal basis in the EU, the implementation costs little, and protecting your intellectual property is well worth the effort. You buy the house, you don't rent it — and that applies to your data too.

Sources & standards

• W3C TDM Reservation Protocol (TDMRep) — W3C Community Group, Final Report 2024. w3.org
• Directive (EU) 2019/790 (CDSM), Article 4 — Text and Data Mining. eur-lex.europa.eu
• Regulation (EU) 2024/1689 (AI Regulation / AI Act), Article 53 — Obligations for GPAI providers. artificialintelligenceact.eu
• We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — Ahrefs, 2026. ahrefs.com

Note: TDMRep, the EU AI Act and related standards are a young, evolving field. The legal basis in the EU is set; voluntary compliance by all providers is not yet.