How SEO crawling works: what Googlebot does from discovery to index

Published pages that never show up, a section the robot seems to ignore, an unreadable indexing report: these symptoms raise the same question. What does Googlebot actually do with your site, between the moment a URL exists and the moment it enters, or does not enter, the index?

This page is for the executive or the marketing, SEO or IT director who needs to understand SEO crawling before ordering fixes: it describes the mechanism, from discovery to indexing, and helps you decide where to look when crawling goes wrong.

What SEO crawling covers: discovery, crawling, indexing

Crawling is the condition of everything else: without it, Google discovers, understands and processes nothing. It forms the first layer of an SEO audit: before sorting fixes, you need to know how a URL is discovered, queued, fetched, rendered, then kept or not.

What a crawler observes of a site

To “crawl” means, literally, to scan: a website crawl extracts as much information as possible in order to understand the structure and check how robots reach the pages, independently of the CMS. A diagnostic crawler simulates that behaviour:

• it visits URLs and follows the links it meets;
• it collects HTTP statuses and spots redirects;
• it measures structuring elements: titles, canonicals, robots directives;
• it flags the areas that are poorly discovered or poorly understood: fragile site structure, insufficient internal linking, duplicated metadata.

Crawled is not indexed, and the reverse is true as well

Crawling means searching for and analysing content so that it can be displayed; indexing means deciding to add (or to keep) a URL in the index. A page can therefore be visited by Googlebot without being eligible for display in the SERPs, for well-known reasons:

• a noindex directive;
• duplication that leads Google to keep another canonical URL;
• content judged of little use;
• technical inconsistencies.

The converse is the least understood consequence, and the most expensive: preventing a URL from being crawled does not mechanically cause it to be de-indexed. If Google can no longer reach the page, it cannot see a noindex, a 301 redirect or a 410 code.

How Google crawls: who passes, at what pace, and what it has to load

Google uses automated systems, Googlebot among them, to discover and revisit URLs. The pace depends on the perceived importance of the pages, on their freshness and on technical constraints; nothing is instant, because the five stages of the journey are separated by queues.

Each stage has its brakes:

Rendering calls for a nuance: text content is processed first, then rendering, when it happens, can lead to a re-indexing of the final content. A page built on JavaScript is therefore evaluated in two stages.

Google prioritizes, and your server constraints weigh in that trade-off: recurring 5xx errors or high latency can lead it to reduce crawl pressure, and therefore to slow down how quickly updates are taken into account.

URL discovery: internal linking, external links, sitemap

Before crawling can happen, Google has to discover the URL. In practice, three sources dominate:

• internal linking (menus, contextual links, pagination): it determines the “natural” access paths and the depth of the pages;
• external links: they contribute to discovery and prioritization, and can keep a URL “alive” even when it becomes orphaned internally;
• the XML sitemap: it lists URLs to crawl as a priority or to revisit, with no guarantee of immediate crawling.

The nuance settles a great many disappointments: a sitemap does not force the robot to pass; it is a discovery and prioritization signal, not an “index now” button.

Rendering and resources: what Google has to load

If the main content or the links only appear after JavaScript has run, the analysis becomes more expensive, slower and prone to gaps between the initial HTML and the rendered content. Blocking resources too broadly in robots.txt (CSS, JS, fonts, critical images) degrades the rendering, and therefore the evaluation.

The fundamentals that make a site easier to crawl

Most failed crawls are explained by three families of causes, all of which can be read in the crawl:

• an architecture that pushes pages away: excessive depth, orphan pages, non-crawlable links, internal links pointing to non-indexable URLs;
• URLs that multiply: sorting, combined filters, internal search, sessions, UTMs;
• unstable server responses: redirect chains, 302s instead of 301s, 404s on pages that should exist, 5xx spikes.

Checking these points one by one (directives, statuses, canonicals, depth, pagination, rendering), then sorting them by impact, belongs to the technical SEO audit. The takeaway here: a deep page is revisited more rarely, and a page that is “expensive” to process forces Google to arbitrate.

URL management: parameters, facets and duplication

The key lever is not only to block, but to decide which URLs deserve to be indexable and canonical. Canonicalization consolidates indexing without removing crawling: canonicalized URLs are still crawled, and indiscriminate canonicalization, when internal linking pushes towards URLs that canonicalize elsewhere, makes Google reprocess pages for nothing.

To remove or merge URLs, three rules prevent the most frequent mistake:

• canonicalizing a URL A to B is not a “clean” deletion method if A and B are genuinely different;
• a 301 presupposes a relevant replacement, and external links on the deleted URL are not enough to justify one;
• with no equivalent, a 404 or a 410 is the right answer, and a 410 can speed up de-indexing.

The sitemap: what it does, what it does not do

An XML sitemap becomes genuinely useful in three typical cases:

• large sites;
• fresh content published often;
• deep pages that internal linking barely reaches.

On a small, well-linked and stable site, it mainly serves as a control rather than a discovery lever. Everywhere, it signals without compelling: the decision depends on perceived quality, on the consistency of the signals and on constraints.

An SEO-oriented sitemap reflects the indexing strategy, not an inventory of the site: URLs returning 200, indexable, canonical and useful. Three classic mistakes degrade the signal:

• including URLs that are redirected, in error or set to noindex;
• mixing in variants (parameters, facets) when the canonical points elsewhere;
• letting “technical” URLs (internal search, baskets, accounts) contaminate the file.

Align sitemap, canonicals and internal linking: a sitemap that pushes a URL your site treats as a variant manufactures useless crawling. Segment it by type (articles, categories, local pages), then compare the submitted URLs with the real state on Google's side: the “submitted” vs “indexed” gap often reveals duplication, canonicalization conflicts or an excess of variants.

Controlling robot access without degrading SEO

Robots.txt tells robots what they may request. Used well, it limits crawling of worthless areas; used too broadly, it closes business directories or rendering resources. The syntax and use cases of robots.txt are worth checking before any broad rule. An area closed to crawling must not be fed through internal linking, or you create dead ends.

Blocking crawling and blocking indexing do not answer the same intention: to make an already known URL disappear, first let Google re-crawl it so it can see the signal. For a staging environment or a private space, server authentication (of the .htpasswd kind) stops robots and users alike, more reliably than a robots.txt file, which stays public.

The three common directives are chosen according to the objective:

Concentrating crawling on what counts

Crawling is not infinite. Two movements steer it, in this order: remove what wastes it, then make what deserves revisiting easier to reach.

Spotting crawling that is badly spent

Badly spent crawling is rarely spotted with a single metric. Look instead for converging signals:

• the recurring crawling of parameters and variants with no value (sorting, combined filters);
• a high volume of “discovered” URLs for few URLs actually indexed;
• spikes of server errors (5xx) or slowdowns correlated with a drop in crawling, visible in the Crawl Stats report in Search Console.

The third signal deserves attention: an infrastructure incident can have a delayed SEO effect, through slower re-crawling. Preventing discovery (no internal link, no sitemap entry) is often cleaner than making up for it with directives. When the volume becomes the problem, the levers by URL family and the trade-off between server capacity and crawl demand belong to the crawl budget.

Prioritizing high-impact pages

Effective crawling serves your business strategy and favours:

• categories;
• conversion pages;
• pillar content;
• pages that already carry impressions and clicks.

On high-volume catalogues, frequent in e-commerce, the point is to bring those pages up in the internal linking and to limit the variants. Crawling is not an end in itself: it must serve access to the positions that count.

Running a crawl analysis: scope, cross-checking, decisions

A crawl quickly produces thousands of rows. It becomes an analysis with a scope decided in advance, a cross-check with what Google observes and an output in decisions.

Framing the crawl, then cross-checking what it shows

A useful crawl starts with a clear scope: which segment do you want to secure (blog, categories, products, local pages) and for what purpose? The most common objectives:

• uncover orphan pages;
• map the redirects;
• measure depth;
• identify duplication by parameters;
• check the rendering of a JavaScript template.

On very large sites, work in batches (directories, page types, templates): it is the only way to identify root causes, such as a global canonical rule or a resource block.

An external crawl shows what the robot can explore; Search Console shows what Google does: indexing states, exclusions, errors, sitemaps. A large volume of non-indexed pages is a warning, but the statuses differ:

Search Console has a limit: it tells you what Google observes and decides, but not always why an architecture generates so many parasitic URLs. Audience data, for its part, prevents you from fixing anomalies on pages with no traffic, no conversion and nothing at stake.

When an important page is almost never crawled

The diagnosis runs in three stages, in this order:

• discoverability: does the page receive internal links from strong pages, is it too deep, missing from the sitemap or caught in a canonicalization conflict?
• technical brakes: latency, 5xx errors, redirect chains;
• cross-checking with Search Console: its status tells you at which stage the page is stuck.

Thin content is only one hypothesis: the queue, unfavourable prioritization or server load explain a page left as “discovered” just as well.

The finding converts into actions by template, sorted with the audit's impact × effort × risk grid:

• quick wins: fix an over-broad robots rule, remove internal links pointing to redirects, repair a 404 pattern on a template;
• structural projects: rebuild the facets, consolidate the canonicals, simplify the pagination, make the content accessible without scripts;
• the objective: not “zero alerts”, but stability that lets Google discover, render and process the important pages over time.

Tracking crawling over time

A one-off crawl photographs a state; it does not see what breaks “silently” between two releases, which is where the most expensive regressions sit:

• redirects appearing on a whole template;
• an explosion of parameter URLs;
• new orphan pages;
• blocked resources;
• a rise in server errors.

A regular check catches them before indexing drops. The indicators to follow measure stability and focus:

• fewer parasitic URLs discovered;
• fewer redirect chains and internal 4xx;
• fewer pages excluded for duplication;
• an improved “submitted” vs “indexed” gap on the sitemap;
• rising impressions and clicks on the priority segments.

Measure in batches (templates, directories, page types) rather than page by page, then connect three planes:

• crawling: the crawl findings;
• indexing: what Google decides;
• value: what the pages bring in.

That cross-reading prioritizes the fixes that protect pages that matter and measures the real effect of an optimization, before and after, on comparable segments.

Continue reading

Two next steps, depending on what you are missing. You need the trace of what Googlebot actually requested: log analysis shows the visit frequency per section, the statuses returned and the crawl waste. You are looking for where to read coverage, exclusions and errors: the Google Search Console reports are interpreted one by one.

Comparing two successive crawl maps (new redirects, parameter URLs that have appeared, pages that have dropped out of the internal linking) resists manual work: that is what the audit and mapping module covers.

FAQ on site crawling and SEO crawling

How do you launch a site crawler step by step, without skewing the results?

First define the scope (domain, subdomains, directories) and a URL ceiling if the site is large. Start from representative entry URLs, checking that the crawler does not get lost in parameters. Finally export the URLs by status, depth and directive, so as to spot patterns rather than isolated cases.

How does Google crawling work, in practice?

Google discovers URLs through links and sitemaps, then places them in a queue. Googlebot then fetches the content and, depending on the case, the resources needed for a rendering that is sometimes deferred. Nothing is instant: the trade-off depends on perceived quality, authority, freshness and server constraints.

What is the difference between crawling and indexing?

Crawling is visiting a URL and fetching its resources; indexing is the decision to add that URL, or its canonical, to the index. A crawled page can remain non-indexed (noindex, duplication, insufficient quality), and without indexing it cannot rank in the SERPs.

Which tools should you use to crawl a website without multiplying solutions?

Two tools cover the essentials: Search Console and a diagnostic crawler. The first shows what Google sees and decides: coverage, exclusions, errors, sitemaps. The second maps the site like a robot (redirects, depth, canonicals, orphan pages). One says what Google does, the other why.

Does a sitemap guarantee crawling and indexing?

No, a sitemap informs Google of pages added or modified without forcing them to be crawled, and a crawled URL can stay out of the index. It becomes genuinely effective when it contains only URLs returning 200, indexable, canonical and aligned with the internal linking.

Do external links influence discovery and crawl frequency?

Yes, backlinks make URLs easier to discover and sometimes reinforce their perceived importance, which plays into prioritization. An external link can even keep an orphan page reachable, without replacing clean internal linking for stable crawling over time.

What are the SEO risks with a site heavily dependent on JavaScript?

A site heavily dependent on JavaScript costs more to render, which can delay or prevent indexing. Check the rendered HTML and the discoverability of internal links: a link that appears only after a script has run is not a reliable crawl path.