robots.txt
robots.txt is a text file in a website’s root directory. It tells automated clients which areas they may enter and which they may not. It is the oldest of these routes and by far the most widely served.
robots.txt sits directly under the domain and is the first thing an automated client fetches. Line by line it states which client may enter which paths. The standard dates from the nineties and remains the only file of its kind that is present almost everywhere.
What matters most is what it is not: it is not an access control. An entry is a request, not a lock — anyone who ignores it is not prevented from proceeding. Confidential material belongs behind a login, not inside a disallow line. Conversely, a rule that is too strict can stop content from being found at all.
Since AI systems began reading the web, the file has taken on a second job: many providers declare their own token in it, so their access can be governed separately. Anyone who wants to decide whether and how their content appears in AI answers starts here.
Related terms
Frequently asked questions.
Can I stop AI from training on my content — without becoming invisible?
Yes — with a TDM opt-out (TDMRep): a machine-readable usage reservation that forbids text and data mining of your content for AI training. The legal basis is the EU framework, which obliges providers of large AI models to respect such reservations. Reading and citing by AI answer systems stays allowed — you remain visible and citable while your text does not permanently flow into models. You set the rule once in machine-readable form, instead of leaving it to a legal text alone.
Do I absolutely need an llms.txt?
It is not a Google ranking signal (Google says so itself) — but it takes minutes to create and is useful as orientation for AI agents. A cheap addition, not a miracle cure.
How widespread it really is, we measure ourselves: as of August 2026, 10.2% of the domains probed serve an llms.txt. The measurement is open in the ai-discovery radar.