Googlebot in 2026: Mastering Crawling and Discovery

You run a B2B website and want to understand why some pages take a long time to appear in Google, or never appear at all. This page explains how Googlebot, Google's web crawler, works, building on our SEO guide and our technical SEO hub.

It helps you decide what to get crawled first, which directives to set, how to check what the crawler actually does and how to handle the crawlers of generative engines.

What Googlebot Is: Definition, Role, and Why It Matters

Googlebot is Google's official web crawler: it browses the web, follows links, fetches pages and resources, then feeds Google's indexing systems. A page it cannot reach (blocked, server errors, difficult to discover) may not be indexed.

Your goal is therefore not to have "everything crawled", but to make your highest-value content easy to discover and easy to understand: being discovered and understood comes before any ranking signal.

Crawl, Render, Index: What the Crawler Actually Does

You can break the workflow into three stages, with important nuances:

• Crawling: the crawler sends an HTTP request, downloads the HTML and fetches referenced resources separately (CSS, JavaScript, images), each subject to size limits.
• Rendering: if the page relies on JavaScript, Google may render it to view the final content, provided the required resources are not blocked.
• Indexing: Google analyses the content to determine its topic and decide whether to add it to the index.

At the time of writing, Googlebot crawls at most the first 15 MB of an HTML file. Beyond that, only the downloaded portion can be used: critical content and important links must sit within that portion.

Crawling: How the Crawler Reaches Your Pages and Prioritises URLs

Crawling follows a propagation logic: a page is visited, its links are followed, and then linked pages are discovered. According to Google Search Central, discovery mainly happens through links found on pages that have already been crawled:

• Internal linking: link to your key pages from strong pages that are already crawled (category pages, hubs, pillar guides).
• XML sitemap: it declares important URLs, but does not replace internal links.
• Update signals: consistent publishing and meaningful updates signal which sections deserve revisits.

Visit Frequency: What Changes Crawl Intensity

Google Search Central indicates that Googlebot does not access most websites more than once every few seconds on average. That cadence varies depending on three factors:

• Server capacity: slow responses, timeouts and 5xx errors reduce effective crawling.
• Perceived importance: heavily linked and regularly updated pages are crawled more readily.
• High volumes of similar URLs: parameters, facets, infinite filters and duplicates dilute crawl budget.

You cannot control this budget, but you can reduce the noise around priority pages.

Crawling vs Indexing: Understanding the Gaps That Block Visibility

Three misconceptions make diagnosis costly:

• Blocking crawling does not prevent appearance: a URL disallowed in robots.txt can appear in results if it is discovered in other ways.
• Being crawled does not mean being indexed: a visited page can be excluded (duplication, perceived quality, directives, inconsistencies).
• Being indexed does not mean being recrawled: a page can remain in the index without an immediate new visit.

So start by answering one question: is the page not visited (discovery, crawling), or visited and then excluded (indexing)?

Robots and Specialisations: Versions, User Agents, and Use Cases

The term "Googlebot" covers a family of crawlers, not to be confused with other search engines' bots, audit-tool crawlers or Google's non-Search agents (AdsBot-Google, Mediapartners-Google), which play no part in web indexing. A visit from a Google bot therefore does not prove indexing.

Googlebot Subtypes: Mobile, Desktop, Images, and News

Google Search Central distinguishes two subtypes for the web, Smartphone and Desktop, plus specialised agents visible in server logs. Each calls for a different check:

Mobile-first indexing now applies across the board: most crawl requests come from the Smartphone crawler, and the mobile content is what counts. A stripped-back mobile version therefore weakens understanding and indexing. robots.txt cannot distinguish Smartphone and Desktop via different tokens: only the HTTP user-agent header, in your logs, shows which subtype made the request.

If image or news traffic is strategic, check in your logs the share of requests from the relevant agent and the HTTP status codes returned.

User Agent: Recognising Profiles and Verifying a Legitimate Crawler

The user agent is the "ID card" sent in the HTTP header of every request; it helps you segment analysis by subtype. Google Search Central warns that it is often spoofed. Before filtering or blocking an IP that "claims" to be Google, verify it with a reverse DNS lookup: the hostname associated with the address must belong to Google, then a forward lookup of that hostname must return the same IP address.

Meta Directives and Access Control: robots.txt, Meta Tags, and HTTP Headers

Managing crawling and indexing relies on three complementary layers: the robots.txt file (site level), meta robots tags (page level) and the X-Robots-Tag header (server level). Each directive has a precise scope and a limit you need to know:

Golden rule: avoid contradictions between these layers.

robots.txt: Allow, Block, and Check Without Breaking Useful Crawling

robots.txt provides crawling instructions using User-agent groups and Disallow or Allow rules. Its useful applications:

• blocking low-value areas: internal search, endless filters, shopping baskets, back office;
• explicitly allowing, with Allow, resources required for rendering that sit in a blocked directory;
• targeting a specialised crawler, such as the image crawler, when a business reason justifies it.

The old robots.txt testing tool was retired in 2023 and replaced by the robots.txt report in Search Console. Check the file there after every change, then monitor your server logs: reduced crawling on blocked sections, no unintended side effects on commercial pages.

Best Practices for Control: Consistency Between Directives, Internal Linking, and Commercial Goals

First decide what must be visible (commercial pages, hubs, acquisition content), then control the rest with three rules:

• One tool per objective: robots.txt manages crawling, noindex manages indexing, authentication protects actual access.
• A readable noindex: to exclude a page, leave it crawlable, otherwise the crawler never reads the directive.
• Aligned internal linking: avoid heavily linking to pages you are excluding.

X-Robots-Tag takes over when you cannot edit the HTML, for example for generated documents.

Googlebot and the Crawlers of Generative Engines

Googlebot is no longer the only crawler that matters for your visibility. Generative engines send their own agents, each declared by a separate token in robots.txt:

• Google-Extended: a Google-specific token, separate from Googlebot;
• GPTBot: the crawler of OpenAI, the company behind ChatGPT;
• ClaudeBot: Anthropic's crawler;
• PerplexityBot: Perplexity's crawler.

These crawlers are managed independently: blocking an AI crawler in robots.txt does not affect Googlebot, and therefore does not affect your crawling and indexing in Google.

Most of these crawlers do not execute JavaScript: content that only exists after client-side rendering stays invisible to them, which makes server-side rendering the first safeguard. CSR, SSR and SSG options are compared in our article on JavaScript SEO.

Crawling by Googlebot nonetheless remains the foundation of generative visibility: 99% of AI Overviews cite the organic top 10 (Squid Impact, 2025). Other benchmarks are gathered in our GEO statistics, and the method in our guide to GEO (Generative Engine Optimization).

Building an Effective Crawling Strategy: Method, Priorities, and Use Cases

An effective crawling strategy aligns what you want discovered with what the crawler can access without friction: prioritisation, not completeness.

Defining Priority Pages: What You Want Discovered and Indexed First

Start with a short, business-led list:

• offer and category pages, which generate leads;
• pillar pages, which structure internal linking to deeper pages;
• high-demand pages: high impressions in Search Console and positions 4–10, the quick wins zone.

Then make sure these pages receive links from pages already being crawled. Update them when it genuinely improves them, by enriching, clarifying or consolidating, rather than simply changing a date.

Typical Scenarios: Launching New Pages, Redesigns, Highly Seasonal Content

Each situation calls for specific crawling actions:

Measuring Results: Indicators, Search Console, and Server Logs

You do not measure a bot, but the effects of controlled crawling, through four indicators:

• Index coverage: indexed, excluded or error pages, with exclusion reasons;
• Discovery delay: time between publishing, first crawl in the logs and first impressions in Search Console;
• Anomalies: spikes in 404 or 5xx errors, redirect loops, over-crawled sections with no business value;
• Performance: impressions, clicks, CTR and average position by page, then conversions in analytics.

Search Console: URL Inspection and Reading Trends

URL Inspection in Search Console checks a page's index status, validates a fix and shows which canonical Google selected; the Links report identifies important pages with weak internal linking. Its data is not real-time: look for trends over several weeks and cross-check them against your deployment dates.

Server Logs: What You Can Measure, and What You Should Not Over-Interpret

Logs show what is really happening: visited URLs, frequency, user agents, HTTP status codes and response times. They help you detect:

• over-crawling of parameters and facets;
• under-crawling of deep commercial pages;
• server errors and redirect chains that slow discovery.

More crawls do not mean better rankings: crawling is a prerequisite, not a performance guarantee.

Mistakes to Avoid: Blocks, Inconsistencies, and Crawl Traps

Crawling errors show up first through their symptoms. The following table links each one to where to observe it and the action to take:

Behind these symptoms, accidental blocks and inconsistent directives keep coming back:

• critical directories accidentally blocked in robots.txt;
• CSS or JavaScript needed for rendering blocked, hiding the primary content;
• an indexable staging environment or, conversely, production blocked by a copied configuration;
• a URL that is both blocked and set to noindex, so the crawler never reads the directive;
• internal links pointing to one URL variant whilst the canonical points to another;
• 301 chains that consume crawl resources and increase the risk of errors.

Finally, crawl traps come from URLs that are easy to generate, filters and sorting first: reduce indexable URLs to those with genuine user and business value.

Depending on Your Situation

Depending on the problem you face, continue with the right resource:

• Your pages are crawled but not indexed: exclusion causes and their fixes are covered in our guide to website indexing.
• You want to understand crawling beyond Google: our article on SEO crawling describes how crawlers work and crawl budget.
• Your mobile content lives in tabs or accordions: our article on hidden content and the Google cache explains how it is handled.
• You are new to crawl reports: our Google Search Console guide details URL Inspection and indexing reports.

To turn these crawling and indexing signals (Search Console, logs, anomalies) into action priorities, the Incremys SEO & GEO 360° audit combines technical, semantic and competitive diagnostics, then tracks fixes.

Googlebot FAQ

What is Googlebot and why does it matter?

Googlebot is Google's web crawler: it discovers URLs, fetches their content and resources, then feeds the indexing systems. It matters because a page it does not crawl properly may not be indexed, and therefore may not appear in search results at all.

What impact does crawling have on search rankings?

Crawling enables the discovery and understanding of your pages: without it, there is no reliable indexing and no ranking. It remains a prerequisite, not a ranking factor. A crawled page can be excluded from the index, and more visits from the crawler do not lead to better rankings.

How do you implement an effective crawling strategy?

An effective crawling strategy focuses the crawler on your priority pages: offers, hubs and high-demand content. Strengthen internal links to them, reduce unnecessary URLs such as parameters and duplicates, and keep robots.txt, meta directives and HTTP statuses consistent with one another.

How does Googlebot compare with alternatives?

Googlebot is the only crawler that feeds the Google Search index, which makes it decisive for SEO. Bing's crawlers feed their own index, audit-tool crawlers simulate a crawl, and Google's advertising agents such as AdsBot index nothing at all.

What mistakes should you avoid to prevent crawling issues?

The most damaging mistakes are accidental blocks in robots.txt and blocking resources required for rendering. Next come redirect chains, the multiplication of parameterised URLs and inconsistencies between canonicals, internal links and indexing directives, all of which waste crawl resources.

How do you set up proper control with robots.txt, meta directives and X-Robots-Tag?

Each tool has its role: robots.txt manages crawl access, meta robots tags manage indexing and snippet display page by page, and X-Robots-Tag applies these rules to non-HTML files. Never block a page set to noindex in robots.txt, because the crawler would not read the directive.

Should you block AI crawlers?

There is no single answer: blocking AI crawlers is a trade-off between visibility and control. Blocking them reduces your content's presence in generated answers; allowing them accepts its reuse. In every case, blocking GPTBot, ClaudeBot, PerplexityBot or Google-Extended affects neither Googlebot nor your indexing in Google.