New page live: how Google finds it — and what you can do yourself

Picture this: on Monday a hotel in Merano puts the page for its new winter package online, in German, Italian and English. On Tuesday the owner types the name of the package into Google — and finds nothing.

That isn't a fault. It's how it normally works. Google doesn't add a new page the moment you publish it: it first has to discover it, crawl it and decide it deserves a place in the index. Here's how a new page gets onto Google, what you can do yourself — and what only Google decides.

Three steps before search.

Before a new page shows up in search results, Google has to discover it, crawl it and add it to its index — three separate steps, and each one can take time. The index is Google's catalogue of every page that can appear as a result at all.

• Discovery. Google learns that the address exists: through a link on a page it already knows, or through your sitemap.
• Crawling. A crawler, a program that fetches websites automatically, retrieves the page and reads its content. Google's is called Googlebot.
• Indexing. Google decides whether the page goes into the index — and with it, whether it can appear as a result at all.

In its explanation of how Google Search works, Google says so itself: there's no guarantee that a page will be crawled, indexed or shown, not even if it follows every guideline. Registering your website with Google and finding it in the results the next day: that's not how it works.

The sitemap tells Google what's there.

A sitemap is a machine-readable list of every address on your website that belongs in search — it helps Google discover them, but it doesn't promise indexing.

It does the most good in exactly the situation above. According to Google's guidance on sitemaps, the sites that benefit most are new ones with few links pointing to them from elsewhere. That describes almost every new website of a small business.

A good sitemap has three properties:

• Split by content type. Pages, posts, glossary: each type with its own list, gathered in an index.
• Every language version included. Publish in German, Italian and English and you have three times the addresses. Each one belongs on the list, or one language stays invisible for longer.
• Current on its own. A published page is in, a draft never is, a deleted page drops out.

The third is where things break most often. A sitemap maintained by hand or through an add-on goes stale with the first forgotten entry — and nobody notices.

In btlabs Core this is part of the foundation. Every content type with its own addresses has its own XML sitemap with all language versions, plus an index. You publish; the address is in. A draft never is. Your robots.txt names the sitemap, so Google finds it without you lifting a finger. What else runs by default for Google is on Found on Google.

Crawling takes time.

Google doesn't fetch every new address straight away; it decides for itself when, how often and how many pages of a site to crawl.

In its guide to asking Google to recrawl, Google puts this at anywhere from a few days to a few weeks. A brand-new website is one Google barely knows yet. Until a rhythm sets in, patience is part of the job.

Two things you can still influence: that the crawl succeeds, and that the page is worth indexing.

• The crawl succeeds. The page has to arrive as complete HTML, quickly and with no password in the way. In btlabs Core every page comes finished from the server, so even a crawler that doesn't run JavaScript sees every sentence. Why speed matters to machines too is covered in When Every Second of Load Time Counts.
• The page is worth it. Not every crawled page makes it into the index. Google lists low-quality content among the common reasons; among near-identical pages it picks one and usually shows only that. An offer page with three lines and a photo is found quickly — but has little to say.

Internal links are the shortest route.

The most reliable way for Google to discover a new page is a link from a page it already knows.

By its own description, Google finds pages mainly by following links. The sitemap complements that; it doesn't replace it. For the hotel in Merano, that means the winter package gets linked from the home page and the rooms page on the day it goes live, in all three languages. With link text that says what it's about — “Winter package with ski pass” rather than “click here”.

A page that no internal link points to is called an orphan page. Google can only reach it through the sitemap; visitors can't reach it at all.

In btlabs Core, the internal analysis in the visibility cockpit flags orphan pages like these, along with dead ends and broken links. Terms that are in the glossary get linked automatically in body text.

Search Console: check instead of guessing.

Google Search Console is the free tool where you can see whether Google was able to read your sitemap and whether a particular page is already in the index.

You set it up once, you or whoever looks after your website: add the site, verify ownership (with a DNS record or a meta tag), submit the sitemap and, for btlabs Core, grant read access. That's around five steps and roughly a quarter of an hour. You submit the sitemap once, not for every new page. From then on the Sitemaps report shows whether Google could read it and how many addresses it found in it.

For a single new page there's URL Inspection. Enter the address and you see whether it's on Google, when Google last crawled it and whether anything went wrong. From there you can also request indexing. That's a help, not a shortcut: Google stresses that a request guarantees neither immediate inclusion nor inclusion at all, that there's a quota, and that asking again doesn't speed anything up.

btlabs Core only reads from Search Console. Every night it pulls your search data into the cockpit, with a few days' lag, because Google delivers its figures late. The system doesn't file any requests with Google; if the button needs pressing, you press it. How much this search data reveals, right down to individual spellings, is shown in A machine reads literally.

IndexNow reaches Bing, not Google.

IndexNow is a protocol a website uses to report new addresses to search engines instantly — Google doesn't take part.

IndexNow is backed by Bing, Seznam, Yandex and others; the list is on indexnow.org. In btlabs Core it's a switch in the settings. Turn it on and every published address goes straight to the participating search engines. For Google, it stays with the sitemap and regular crawling.

Anyone who promises to report a page “to Google instantly” is promising too much. In 2023 Google even announced it would retire its old notification route for sitemaps, the so-called ping (Google Search Central Blog). What counts instead, Google says in the same post: a sitemap whose modification dates can be trusted.

Patience, not tricks.

After publishing, the most effective thing you can do is surprisingly little: check, link, wait.

1. Published, not a draft. And the switch that keeps a page out of search results is off.
2. Linked. From one or two existing pages, in every language.
3. Sitemap submitted. Once in Search Console, not again for every page.
4. For important pages: URL Inspection. Requesting indexing once is enough.
5. Check back after a few days. Not every hour.

What doesn't help: filing the same request five times, resubmitting the sitemap over and over even though it updates itself, or renaming the address after a week because the page hasn't shown up yet. btlabs Core does create the 301 redirect on its own. Google still has to crawl the new address first.

And nobody can sell you indexing. Whether and when a page lands in the index is Google's decision. You can clear the way; you can't set the date.

The one thing to take away.

A new page isn't on Google the moment it goes live. It gets discovered, crawled and maybe indexed. You have real influence over the first step: a sitemap that keeps itself current, internal links from day one, a Search Console that's set up. The rest is up to Google.

If you'd like to know how this looks for your website: write to us. You'll get an honest assessment, not a pitch.