Robots.txt: role, directives and AI crawlers to block or allow

The robots.txt file tells bots what they may crawl on your site. Yet it removes no page from the index, protects no data and only binds the bots that agree to read it. Badly written, it cuts business pages off from Google; too restrictive, it removes the site from ChatGPT or Perplexity answers. This guide explains its role, how to write its directives line by line and which AI crawlers to block without losing generative visibility.

Robots.txt: what it is for, and what it doesn't do

Robots.txt is a public text file, placed at the root of a website, that tells crawling bots which URLs they are allowed to access. Google highlights it: its main purpose is to manage crawl traffic and avoid overloading the server, not to prevent a page from appearing in search results.

To place its role, you need to distinguish the three stages a page goes through before it becomes visible:

• discovery: the bot finds the URL through internal links, external links or the sitemap;
• crawling: it downloads the page if robots.txt allows it and the server responds correctly;
• indexing: the engine decides whether to keep the page based on meta robots, X-Robots-Tag, canonicals and content quality.

Robots.txt only acts on the second stage: it allows or refuses access to paths, limits crawling of areas with no value (internal search, sorting, tracking) and declares sitemaps. Its limits explain most configuration errors:

• it does not prevent indexing: a blocked URL that is linked elsewhere can appear in results, without a description;
• it protects nothing: the file is public and even points to the directories you would like to hide;
• it is not enforceable: reputable bots respect it, a malicious bot ignores it.

In B2B, it channels crawling towards solution pages and expert content. With AI crawlers, it also decides who can read your content in order to cite it.

Where to place the file: root, subdomains, size and status codes

Bots look for the file at a fixed address, at the root of the host, for example www.yourdomain.co.uk/robots.txt over HTTPS. A few rules apply, at the time of writing:

• the name is written in lowercase, robots.txt, and the file is never placed in a subfolder;
• each host has its own file: www.yourdomain.co.uk, blog.yourdomain.co.uk and help.yourdomain.co.uk each have one, as does each protocol;
• Google reads at most 500 KiB and ignores anything beyond that limit;
• comments start with # and document the purpose of rules.

Version the file with the site's code: the team that takes it over during a migration will know why each rule exists. On a multilingual site, avoid blocking language directories by default; hreflang and sitemaps per language handle duplication better.

The HTTP status code returned by the file changes how Google crawls the whole site:

An empty file, or one reduced to User-agent: * followed by an empty Disallow:, allows everything: for crawling, it is the equivalent of having no file.

Directives: User-agent, Disallow, Allow, Sitemap

A file is made up of groups: one or more User-agent lines name the bots concerned, then the rules follow. Here are the directives and their support by Google at the time of writing:

When several rules target the same URL, priority follows three principles:

• the bot chooses the most specific User-agent group that names it, or failing that the * group;
• within that group, the rule with the longest, and therefore most specific, path wins;
• at equal length, Allow takes precedence over Disallow.

The following example blocks a directory whilst leaving one page accessible. Paths must use ASCII characters, as non-ASCII characters have to be encoded:

• User-agent: *
• Disallow: /private/
• Allow: /private/public-page.html

Two directives need clarifying. Google has no longer supported Noindex in robots.txt since 1 September 2019: to deindex, place noindex in a meta robots tag or in the X-Robots-Tag HTTP header, on a page the bot can crawl. If the page is blocked, it never sees the instruction.

Google also ignores Crawl-delay, which Bing and Yandex read. With Google, crawl rate is managed server-side or via Search Console.

Wildcards * and $: syntax and pitfalls

Google and Bing fully support two special characters, standardised by RFC 9309: the asterisk *, a wildcard that replaces any sequence of characters, and the dollar sign $, which anchors the rule to the end of the URL. Here are common uses, to adapt to your URL structure:

• Disallow: /*.pdf$: all PDF files, and only them;
• Disallow: /*?: all URLs with parameters, a broad rule to handle with care;
• Disallow: /*?sort=: sorting, when sort is the first parameter;
• Disallow: /*&sort=: sorting, when sort follows another parameter.

The most common pitfall lies in parameter order. In a URL, the first parameter follows a ? and the next ones an &: Disallow: /*?sort= therefore does not block /shoes?colour=black&sort=price. To cover both cases, double each rule. The same reasoning applies to tracking parameters:

• Disallow: /*?utm_
• Disallow: /*&utm_
• Disallow: /*?gclid=
• Disallow: /*&gclid=
• Disallow: /*?fbclid=
• Disallow: /*&fbclid=

Also encode paths with accented characters (/private/-style paths, no non-ASCII characters). Finally, an overly broad rule can block PDFs that generate leads: before deploying a pattern, test example URLs to "keep" and "exclude".

Ready-to-use example files

These templates adapt to your paths and domain. For a brochure site, you block no business pages, close two technical directories and declare the sitemap:

• User-agent: *
• Disallow: /cgi-bin/
• Disallow: /tmp/
• Sitemap: https://www.yourdomain.co.uk/sitemap.xml

For a B2B site with a blog, add internal search, marketing parameters in both forms and a sitemap index:

• User-agent: *
• Disallow: /search/
• Disallow: /*?utm_
• Disallow: /*&utm_
• Disallow: /*?gclid=
• Disallow: /*&gclid=
• Sitemap: https://www.yourdomain.co.uk/sitemap_index.xml

For an e-commerce site with faceted navigation, facets that match a search intent ("safety shoes size 8") remain crawlable; sorting, basket and account are closed:

• User-agent: *
• Disallow: /basket/
• Disallow: /my-account/
• Disallow: /*?sort=
• Disallow: /*&sort=
• Disallow: /*?order=
• Disallow: /*&order=

Before deploying, use your logs to spot the parameters that cause URL explosion and make sure you are not blocking a facet you want to rank.

For a WordPress site, the common approach is to close the admin area whilst leaving the AJAX file accessible, then block internal search:

• User-agent: *
• Disallow: /wp-admin/
• Allow: /wp-admin/admin-ajax.php
• Disallow: /?s=
• Sitemap: https://www.yourdomain.co.uk/sitemap.xml

Do not block /wp-includes/ or /wp-content/ without analysis: they are needed for rendering. On WooCommerce, target basket, account and parameters, never categories or products.

For a staging environment, the file blocks everything, but that is not enough: a URL linked elsewhere can be indexed without being crawled. Also protect access with a password or by IP:

• # staging: never deploy to production
• User-agent: *
• Disallow: /

The opposite risk costs more: this file goes to production and cuts off the whole site. Check /robots.txt with every release.

Crawl budget: when it really matters

Crawl budget is the number of URLs an engine is willing to crawl on a site over a given period. It only matters for very large sites: catalogues, help centres, platforms with hundreds of thousands of URLs. On a B2B site with a few hundred pages, robots.txt mainly serves to avoid noise. On a large site, here is the recommendation by page type:

Never block the CSS and JavaScript required for rendering: without them, Google misinterprets the structure and visible content. A rule such as Disallow: /*.js$ or blocking /assets/ without analysis degrades page understanding.

AI crawlers: which to block without losing generative visibility

AI crawlers are not a single block: some train models, others feed an answer engine that cites its sources, others act on a user's request. The table distinguishes them, at the time of writing:

The GEO doctrine comes down to this: you can refuse training without refusing search. Blocking GPTBot or ClaudeBot takes nothing away from your presence in answers; blocking OAI-SearchBot, Claude-SearchBot or PerplexityBot removes the site from answers, and therefore from citations.

This choice weighs more and more on acquisition: traffic from AI search grew by 527% in one year (Semrush, 2025). Further benchmarks are gathered in our GEO statistics.

To refuse training whilst remaining citable, a single group names several bots, with no search bot:

• User-agent: GPTBot
• User-agent: ClaudeBot
• User-agent: CCBot
• User-agent: Google-Extended
• Disallow: /

Google-Extended is often misunderstood: Googlebot crawls, and this token only indicates whether content may be used for Gemini. Blocking it therefore removes the site neither from AI Overviews nor from AI Mode, available in France since summer 2026, which rely on Googlebot.

The stakes are far from marginal: more than 50% of Google searches display an AI Overview (Squid Impact, 2025). Blocking Googlebot to protect yourself from AI would make the site disappear from both.

Robots.txt remains declarative: a bot can ignore the rule or spoof a user-agent. Against abusive scraping, add server filtering that returns a 403. Our guide to Generative Engine Optimization details the other levers of citability.

Finally, llms.txt does not replace robots.txt: it is a Markdown index of reference content offered to LLMs, with no directive value and little read to date, which our article on the llms.txt file presents in detail.

Robots.txt, meta robots, X-Robots-Tag, llms.txt: which tool for which need

Each mechanism acts at a different level: robots.txt on crawling, meta robots on indexing of HTML pages, X-Robots-Tag on indexing of any file, PDFs included. The table compares the five options:

To "allow crawling but prevent indexing", leave the page crawlable and add noindex via meta robots or X-Robots-Tag. For private content, only authentication truly protects it.

Common errors and checks in Search Console

Most incidents follow a few well-known scenarios:

To check the configuration, Google Search Console offers three reports, complemented by server logs:

• the robots.txt report, which replaces the old testing tool: files fetched per host and read errors;
• URL Inspection, which tells you whether a page is blocked and how Google renders it;
• the Page indexing report, which lists URLs blocked by robots.txt;
• server logs, which show which bots crawl what and with which HTTP status codes; a crawler completes the analysis.

After every change, open the file on each host, test a few sensitive URLs, then monitor crawling and the status of your sitemaps in Search Console.

Depending on your situation, these guides take the topic further:

• a complete SEO audit checks robots.txt alongside crawling and indexing;
• technical SEO covers rendering, HTTP status codes and architecture;
• for a faceted catalogue, the e-commerce SEO guide details parameter management;
• for a multilingual site, international SEO covers hreflang and sitemaps per language;
• to bring strategic pages forward, internal linking complements robots.txt.

In Incremys, the GEO & SEO platform, the 360 audit lets you spot blocked URLs, over-crawled areas and AI crawler access, alongside indexing, technical health and AI presence by keyword.

Robots.txt FAQ

Can you block ChatGPT without dropping out of ChatGPT search?

Yes. Block GPTBot, OpenAI's training bot, and let OAI-SearchBot access the site: your content is no longer used for training, but is still crawled for ChatGPT search and can be cited there. Blocking OAI-SearchBot, on the other hand, makes the site disappear from those results.

Is the robots.txt file mandatory?

No. Without a file, or if its address returns a 404 error, Google treats the whole site as crawlable. The file becomes useful as soon as the site has URL parameters, internal search, multiple environments or AI crawlers to filter. It also serves to declare your sitemaps.

Can you declare multiple sitemaps?

Yes. Add one Sitemap: line per file, with an absolute URL, independently of the User-agent groups. On a large or multilingual site, declare a sitemap index instead, referencing sitemaps by language or content type: a single line is then enough.

How do you protect a staging environment?

With authentication, not with robots.txt alone. A Disallow: / limits crawling, but a staging URL linked elsewhere can still be indexed. Protect access with a password or by IP address, then check with every release that the staging file has not been copied to production.

How long before a change is taken into account?

It is not immediate: it depends on when each bot fetches the file again. After an urgent fix, check the robots.txt report in Search Console, request re-crawling of key pages and verify them with URL Inspection.

Does blocking Google-Extended remove you from AI Overviews?

No. Google-Extended is a control token that indicates whether your content may be used for Gemini; it crawls nothing. AI Overviews and AI Mode rely on Googlebot, like classic search. Blocking Google-Extended therefore does not change your presence in those answers.

How do you deindex a PDF?

Leave the PDF crawlable and apply the X-Robots-Tag: noindex HTTP header server-side. A meta robots tag is impossible in a PDF, and a Disallow would prevent the bot from reading the header: the file could remain in results if it is linked elsewhere. For a confidential document, add access control.