# Crawler

> What a crawler does, why findability starts not with the text but with whether it arrives — and what changes when a page is fetched at the moment of a question.

Synonyms: web crawler, search engine bot, spider, crawling

A crawler fetches a page, reads the links it contains, and works its way through a website that way. Search engines build their index from this, archives their collection, AI providers their knowledge base or the material for a live answer. Every crawler identifies itself with a token by which it can be recognised and treated separately.

For a site owner this is not a side issue. Whatever a crawler cannot obtain — because it sits behind a login, only comes into being through scripts, or is simply served too slowly — does not exist for the service downstream. Findability therefore begins not with the text but with the question of whether the text arrives at all.

Two occasions are worth separating: scheduled collection in advance, and a fetch at the moment of a specific question. The second is the newer one — an assistant retrieves a page precisely when somebody wants to know something. In that moment, how fast and how clearly the page answers is what counts.

---
Source: https://btlabs.dev/en/glossary/crawler
Last-Modified: 2026-08-16T04:56:24.971Z
Languages: [de](https://btlabs.dev/llms/de/glossary/crawler) · [it](https://btlabs.dev/llms/it/glossary/crawler)
See also: [llms.txt](https://btlabs.dev/llms.txt) · [ai.txt (Policy)](https://btlabs.dev/ai.txt) · [identity.json](https://btlabs.dev/identity.json)