Crawler
A crawler is a program that fetches web pages automatically and follows their links — for search engines, archives or AI systems. Whatever it cannot fetch does not exist for the service behind it.
A crawler fetches a page, reads the links it contains, and works its way through a website that way. Search engines build their index from this, archives their collection, AI providers their knowledge base or the material for a live answer. Every crawler identifies itself with a token by which it can be recognised and treated separately.
For a site owner this is not a side issue. Whatever a crawler cannot obtain — because it sits behind a login, only comes into being through scripts, or is simply served too slowly — does not exist for the service downstream. Findability therefore begins not with the text but with the question of whether the text arrives at all.
Two occasions are worth separating: scheduled collection in advance, and a fetch at the moment of a specific question. The second is the newer one — an assistant retrieves a page precisely when somebody wants to know something. In that moment, how fast and how clearly the page answers is what counts.
Related terms
Frequently asked questions.
Is being AI-ready worth it for a small business — what changes concretely?
For small businesses in particular it is worth it, because this is where the leverage is greatest: if the big AI systems can read your website without errors and recommend it as a reliable source, you get found and named even alongside larger providers. Being AI-ready concretely means: clearly structured, fully multilingual content, structured data (JSON-LD) and fast, machine-readable HTML — so ChatGPT, Gemini and Perplexity classify you correctly. It is explicitly not about putting a chatbot on your page, but about the AI out there understanding you properly and recommending you.
How do I tell whether an agency genuinely understands how AI reads my website?
A simple test: ask about their last three projects and how AI visibility was handled. If the answer stops at meta tags, page speed and classic rankings, that only covers half the job — important, but not decisive for AI answer systems.
Anyone who genuinely understands how AI reads a website talks about more concrete things: a machine-readable profile such as `llms.txt`, structured data in JSON-LD format for facts like opening hours or services, and whether AI crawlers and agents actually have technical access to the site. These are the levers that decide whether an AI can cite or recommend your business correctly.