# robots.txt

> How robots.txt sets limits for automated clients, why it is not an access control, and the role it has taken on since AI systems began reading the web.

Synonyms: robots.txt file, Robots Exclusion Standard, crawler control

robots.txt sits directly under the domain and is the first thing an automated client fetches. Line by line it states which client may enter which paths. The standard dates from the nineties and remains the only file of its kind that is present almost everywhere.

What matters most is what it is not: it is not an access control. An entry is a request, not a lock — anyone who ignores it is not prevented from proceeding. Confidential material belongs behind a login, not inside a disallow line. Conversely, a rule that is too strict can stop content from being found at all.

Since AI systems began reading the web, the file has taken on a second job: many providers declare their own token in it, so their access can be governed separately. Anyone who wants to decide whether and how their content appears in AI answers starts here.

---
Source: https://btlabs.dev/en/glossary/robots-txt
Last-Modified: 2026-08-16T04:56:20.712Z
Languages: [de](https://btlabs.dev/llms/de/glossary/robots-txt) · [it](https://btlabs.dev/llms/it/glossary/robots-txt)
See also: [llms.txt](https://btlabs.dev/llms.txt) · [ai.txt (Policy)](https://btlabs.dev/ai.txt) · [identity.json](https://btlabs.dev/identity.json)