Tech for Retail 2025 Workshop: From SEO to GEO – Gaining Visibility in the Era of Generative Engines

Back to blog

SEO crawl budget: when volume becomes the problem and what to decide

SEO

Discover Incremys

The 360° Next Gen SEO Platform

Request a demo
Last updated on

1/11/2026

Chapter 01

Example H2
Example H3
Example H4
Example H5
Example H6

SEO lead, e-commerce director or CIO of a high-volume site: this page helps you establish whether crawl budget is your problem, then decide, URL family by URL family, what to open, close or clean up. On a large catalogue, the point is to be crawled usefully: to bring the robots to the right URLs, at the right pace, without trapping them in low-value areas.

 

It sits within the framework of the SEO audit, which sets the prioritization grid and the reference cadence.

 

Does crawl budget apply to my site?

 

The question is settled before any project: a crawl optimization launched on a site that does not need one produces nothing. Three markers serve as a first grid:

 

  • roughly one million unique pages whose content changes moderately, in the order of once a week;
  • roughly 10,000 unique pages when the content changes very fast, daily;
  • a large volume of URLs listed as “discovered, currently not indexed” in Search Console, that is, known and not yet crawled.

 

These are markers, not admission thresholds. The “crawled, currently not indexed” state does not fall under the same diagnosis. The exposed profiles are always the same:

 

  • e-commerce: facets, sorting, pagination, variants;
  • marketplaces: highly dynamic inventory;
  • media sites: freshness;
  • directories: volume and duplication.

 

On these sites, a structural drift can create tens of thousands of “parasitic” URLs that capture crawling at the expense of the categories and products that matter.

 

Below those volumes, the budget is rarely the cause, without being ruled out: a slow or unstable site, or one rich in parameter URLs, can be poorly crawled at a few thousand URLs. But if a small site does not appear, look first for an over-broad directive, an inconsistent canonical or an unexpected HTTP status: that is the territory of the technical SEO audit.

 

What Google can crawl, and what it chooses to process

 

Crawl budget refers to the set of URLs a search engine can and wants to crawl on a site. Four stages can be distinguished within it, and confusing them leads to misreading the reports:

 

  • Discovery: Google finds the URL (links, sitemap, history).
  • Crawling: Googlebot fetches the resource.
  • Rendering: JavaScript processing and construction of the interpreted page; on JS-heavy sites, its cost multiplies the crawling cost and reduces the coverage that is possible.
  • Indexing: analysis, consolidation, then potential inclusion in the index; a crawled URL is therefore not an indexed URL.

 

The budget is also defined per hostname: your main domain and a subdomain do not have the same budget, a design decision in its own right for multi-subdomain architectures. How a URL enters the queue belongs to SEO crawling.

 

Capacity and demand: why adding servers is not enough

 

The budget results from two components that do not offset each other. Capacity depends on the robot's parallel connections and on the delay it imposes between two fetches to spare your servers. Demand reflects its interest in revisiting your URLs: size, freshness, perceived quality, relevance, authority.

 

Hence one consequence: if demand is low, Google crawls less, even when the server could take more. Adding infrastructure does not solve a problem of perception.

 

The signals that make demand vary

 

Three engines drive demand:

 

  • perceived inventory, often the most actionable lever: too many duplicated, deleted or unwanted URLs can lead the systems not to go through the rest of the site;
  • the authority of the URLs;
  • the freshness of the content.

 

A migration triggers a temporary surge in demand. That is a window, not an acquired right: if it creates redirect chains and duplicates, the processing cost increases at the worst possible moment.

 

Recognizing a genuine crawling problem

 

Before blocking anything, establish that it really is a crawling problem, and not one of perceived quality, architecture or intent: a diagnosis made too quickly produces blocks that change nothing.

 

Reading the indexing states without getting them wrong

 

Three points prevent misreading the indexing reports:

 

  • “Discovered, currently not indexed” designates URLs that are known and not yet crawled;
  • “Crawled, currently not indexed” designates pages that Google has chosen not to index, which is not proof of an insufficient budget: this state is cross-checked with recrawl delays, logs and crawl statistics;
  • two false positives recur: confusing “not indexed” with “technical problem”, and treating every discovered URL as a URL to be saved.

 

On a large site, the right question becomes: is this a URL that should exist in the indexing strategy? Answering no to it removes more projects than it creates.

 

The two symptoms that count

 

Two symptoms signal a genuine crawling problem: pages discovered but not crawled, or crawled too late, and changing content that is not revisited.

 

SymptomDominant causeWhat to look at
Pages discovered, not crawledPage too deep or badly linkedDistance from the home page and the hubs
Pages crawled too lateSet with low perceived value, typically facetsVolume of combinations generated
Price, stock, availability not revisitedPerceived freshness deficitRecrawl pace versus pace of change

 

Excessive depth acts as a priority penalty: the further a URL sits from the hubs, the fewer internal signals it receives. On freshness, the engine only re-crawls if it is “worth the cost”; massive duplication, slow templates, server instability and endless URLs reduce that perception.

 

Segment before deciding

 

On thousands, sometimes millions of URLs, you steer at the level of the template and the URL family, not the page: that is what makes a decision to block or to open defensible. Segment at minimum:

 

  • categories and subcategories;
  • product pages;
  • facets, filters, sorting and parameters, the main source of drift;
  • editorial content;
  • technical pages.

 

For each segment, then compare five dimensions:

 

  • the volume of URLs generated;
  • the expected indexability;
  • the crawl signals;
  • performance (TTFB, stability);
  • business value: organic performance, conversion, margin, availability, seasonality.

 

“Important for SEO” must stay aligned with “important for the business”. The diagnosis then isolates the URL families that meet at least one of these four criteria:

 

  • they are crawled frequently, but are not indexed or are excluded on a recurring basis;
  • they have no organic traffic and answer no searched intent;
  • they introduce duplicate content (close variants, parameters, pagination);
  • they degrade crawl capacity through slowness, errors or redirects.

 

The grid below sums up what is expected per family and the default decision.

 

URL familyExpected indexabilitySignal that should raise a flagDefault decision
Categories and subcategoriesIndexable and priorityRarely crawled despite demandReduce depth, strengthen internal linking
Product pagesIndexable, long tailDiscovered but not crawled, in bulkHandle in batches from a hub
Facets, filters and sortingA selection onlyThe number of URLs grows, the traffic does notDecide: SEO-first or UX-only
Editorial contentIndexableUpdates not reflectedCheck the perceived freshness
Technical pagesNot indexableCrawling of areas with no search valuePrevent discovery, not indexing alone

 

Where the budget is lost

 

Waste concentrates in three families of causes, to be taken in this order so as not to block first and understand afterwards.

 

Facets, parameters and sorting: the SEO-first or UX-only decision

 

Faceted navigation can produce an endless number of URLs (filter combinations, sorting, pagination) with largely redundant content: it is the classic robot trap, which wastes crawling at the expense of the strategic pages.

 

The right approach is not to “block everything” but to decide which combinations deserve indexing (strong search demand, a structuring range, seasonality) and to neutralize the rest through consistent rules: directives, canonicals, URL patterns, internal linking.

 

That decision is made once, in writing: which facets are “SEO-first” (indexable, in the sitemap, reinforced by internal linking) and which stay “UX-only”. Without an explicit ruling, the rule rebuilds itself at the first release, in the direction of opening up.

 

Worthless pages, duplication and the noindex trap

 

Internal search, the basket, the login, empty pages, sorting variants or an infinite scroll that duplicates existing links become crawl vacuum cleaners as soon as they are crawlable. The remedy is to close those areas, even when they are useful to the visitor.

 

One trap: noindex is not a crawl-saving solution. If Google has to crawl the URL to see the noindex, crawl time is consumed anyway. Each mechanism has its use:

 

MechanismWhat it preventsWhen to use it
Block in the crawl fileFetching the URLAreas with no value, sorting variants
noindexIndexing, not crawlingDe-indexing before any block
404 or 410Lasting recrawl of the URLPages deleted for good, rather than a block
Soft 404Nothing: it carries on consuming crawlingNever: to be eliminated
CanonicalNothing: it consolidates duplicatesGenuine duplicates, with consistent internal linking

 

On the duplication side, consolidate or eliminate so as to concentrate crawling on unique content rather than on unique URLs: reduce near-identical categories, limit variants with no search demand, rewrite templated content that is too close. Also be wary of canonicalizing “everywhere”, which makes pages get reprocessed when the internal linking pushes towards URLs that canonicalize elsewhere.

 

Redirect chains and instability: the hidden cost

 

Redirect chains are born of a migration, a change of URL rules or products redirected in cascade. Each hop consumes a request; across thousands of URLs, the coverage of the useful pages degrades.

 

Capacity follows “crawl health”: it rises when the site answers fast and steadily, it falls as soon as it slows down or returns errors. The slower the server and the rendering, the smaller the “useful window” left to cover a large inventory.

 

Crawl delay: what it really is

 

The expression covers two things: the Crawl-delay directive in the crawl file, a non-universal mechanism that Googlebot ignores, and the delay between two requests, a component of capacity that Google sets according to the server's response. The syntax of the robots.txt file is handled separately.

 

Less crawling can stem from three situations that do not have the same remedies:

 

SituationWhat happensDecision
Deliberate limitationYou have blocked or reduced certain URL familiesDesirable if those URLs are useless
ThrottlingGoogle lowers capacity in the face of slowness or 5xx errorsDocument the limit on the infrastructure side
Crawl-delay directive setGooglebot ignores it, other robots slow downDo not count on it for Google

 

Several observations document a limit on the infrastructure side:

 

  • a message of the “Hostload exceeded” kind in the URL inspection tool;
  • response times that degrade under load;
  • 5xx errors correlated with crawl peaks;
  • measured saturation.

 

It is on this body of evidence, and not on a single message, that an infrastructure reinforcement is decided. Rather than throttling blindly, four alternatives hold:

 

  • reduce parasitic URLs;
  • stabilize the responses;
  • improve rendering efficiency;
  • keep the sitemaps up to date with a reliable last-modified date.

 

You protect the infrastructure by making crawling more efficient, not by degrading the capacity for discovery. And the order matters: de-index first, block second, because a page blocked before it has been de-indexed can no longer leave the index.

 

The action plan at volume

 

Three workstreams run in parallel: steer crawling towards what counts, reduce what does not have to be crawled, and bring down the cost of each fetch. Their order is settled with the pillar's impact × effort × risk grid.

 

Concentrating crawling on the strategic pages

 

At high volume, internal linking works as a prioritization system: core categories, high-margin products, best sellers and pillar content receive more links, non-indexable URLs receive fewer. The patterns that hold at scale are well known:

 

  • hub pages per universe;
  • lateral and contextual links;
  • accessible pagination;
  • crawlable HTML link blocks, such as “top categories”.

 

Reducing the useless crawled surface

 

Three moves reduce the crawled surface:

 

  • prevent the discovery of what does not have to be crawled: UX-only areas are not injected into the crawlable internal linking;
  • standardize the parameters (order, case, separators, tracking) so that a page does not present itself under ten addresses;
  • treat the cause rather than the symptom, by stopping the explosion of URLs itself.

 

One reservation applies: blocking URLs can reduce their processing by other systems, and the “freed budget” is not automatically reallocated, unless you were at the ceiling on the server side. A block is therefore never announced as a crawl gain.

 

Cleaning up the redirects and stabilizing the response

 

On the server side, a “performance” backlog is a crawl backlog: fewer errors and less latency means more capacity. The rules to check are simple:

 

  • one redirect at most between an old URL and its final destination, ideally zero for internal links;
  • standardized versions (http and https, www and non-www, trailing slash);
  • a permanent redirect as soon as the destination is permanent, with internal linking pointing at the final URL;
  • a reduced rendering cost on JavaScript architectures (useless bundles, excessive hydration);
  • no blocking of the CSS, JavaScript and images needed to render the pages that count.

 

Steering over time and avoiding regressions

 

On large sites, the difficulty is deciding what to fix first and measuring the effect. A minimal dashboard is enough, provided it correlates four families of signals:

 

  • crawling: trends, server errors, redirects;
  • indexing states: valid, excluded, discovered or crawled without indexing;
  • technical performance: TTFB, stability, slow templates;
  • value: impressions, clicks, conversions on the business areas.

 

Regressions rarely come from bad intentions, but from a release. After every major release, check on the most crawled and most business-critical templates whether the deployment:

 

  • generates new parameter URLs;
  • multiplies near-identical pages through template duplication;
  • introduces redirect chains;
  • changes depth;
  • degrades latency and stability (5xx spikes).

 

A single badly framed facet rule can recreate a useless “perceived inventory” within days. A full analysis is relaunched on an event: a redesign, a migration or a change of rules (redirects, canonicals, sitemap), a seasonal peak, a massive addition of products or categories.

 

Continue reading

 

Depending on what the diagnosis revealed, two next steps stand out. You need to quantify what your URL families consume before blocking: the log analysis measures the visit frequency per section. The catalogue itself has become the subject: the e-commerce SEO audit arbitrates categories, product pages, facets and expired products.

 

Between two analyses, tracking, release after release, the volume of URLs generated per family, so as to catch a badly framed facet rule, is the task covered by the Incremys audit and mapping module.

 

FAQ on crawl budget and crawl delay in SEO

 

What is the definition of crawl budget in SEO?

 

Crawl budget is the set of URLs an engine such as Google can and wants to crawl on a site. It combines a capacity, limited by what your server can take, and a demand, which reflects the engine's interest in your pages. Being crawled does not guarantee being indexed.

 

What is crawl budget for and when does it become limiting?

 

Crawl budget becomes limiting when high-value URLs are not crawled fast enough: the robot wears itself out on parasitic URLs, or slowness and 5xx errors reduce its capacity. The markers: roughly one million pages updated weekly, or roughly 10,000 pages updated daily.

 

How do you know whether crawl budget is a problem on your site?

 

Crawl budget is a problem when several signals accumulate on the same templates: many URLs listed as “discovered, currently not indexed”, new pages slow to appear, updates not reflected, crawling concentrated on parameters, a rise in 5xx errors. No single signal settles it on its own.

 

How do you optimize without losing important pages?

 

You optimize without loss by proceeding through segmentation and rules, never through global actions. First decide which URL families must be indexable, then align internal linking, sitemap and directives. No broad block is set without a prior inventory, and a deleted page returns a 404 or a 410.

 

What is crawl delay in SEO?

 

Crawl delay refers to the delay a robot observes between two requests on the same site. With Google, it is part of crawl capacity, adjusted so as not to overload the server: it is a consequence of what your infrastructure returns, not a setting.

 

Does the crawl-delay directive in robots.txt really improve SEO?

 

The crawl-delay directive is not an SEO lever for Google, because Googlebot ignores it, even though other robots respect it. Google adjusts its pace according to the server's latency, errors and stability. The reliable lever is reducing useless URLs and improving performance.

 

Should filters and facets be blocked, or left crawlable on a large e-commerce site?

 

You should neither block everything nor open everything. Combinations that have search value and a clear intent are SEO-first and indexable; the rest stays UX-only and its combinatorial explosion is prevented. Blocking suits sorting variants, whereas noindex does not prevent crawling.

 

How do you prioritize fixes on a catalogue of hundreds of thousands of pages?

 

You prioritize by combining segmentation by template, business value and crawl signals, then ordering by impact, effort and risk. First come the templates crawled often but slow, the heavily duplicated URL families and the errors that recur from one release to the next.

Discover other items

See all

Next-Gen GEO/SEO starts here

Complete the form so we can contact you.

The new generation of SEO
is on!

Thank you for your request, we will get back to you as soon as possible.

Oops! Something went wrong while submitting the form.