Google Search Console and robots.txt: Diagnosing Crawl Issues in 2026

A strategic page shows the status "Blocked by robots.txt" and you do not know whether the rule is intentional or accidental? This page is for SEO managers, marketers and executives who need to decide quickly: unblock, leave as is, or hand the fix to the technical team.

It covers how to use Google Search Console and robots.txt to diagnose a block and deploy changes safely. The file's syntax and general rules are detailed in the robots.txt file guide.

The Role of robots.txt as Seen From Search Console

The robots.txt file instructs robots, including Googlebot, which areas to crawl or avoid, using the User-agent, Disallow and Allow directives. It comes into effect before any attempt to access a URL: a single misplaced directive can block key pages or essential resources. The complete Google Search Console guide places this report among the console's other tools.

Blocking crawling does not reliably de-index a page. The crawling → rendering → indexing cycle is described in the article on Google Search Console indexing.

What Search Console Can Confirm, and What It Cannot Deduce

Google Search Console (GSC) flags symptoms in three places:

• the Pages report, which lists URLs with the status "Blocked by robots.txt";
• the URL Inspection tool, which shows whether Googlebot was able to crawl a specific page;
• the robots.txt report, which shows the version of the file Google retrieved, its fetch date and its errors.

Search Console does not know your intent: it cannot tell a deliberate crawl optimisation from human error. Cross-check these signals with your deployment history.

Accessing the robots.txt Report and Interpreting Key Signals

The report is found under Settings, in the robots.txt section. At the time of writing, it brings together four pieces of information:

• the robots.txt files found for the top 20 hosts of the site;
• the last fetch date of each file;
• the errors and warnings detected during parsing;
• a button to request a recrawl of the file.

Which Properties and Hosts Are Affected

Ensure your GSC property covers the relevant variants: http and https, with or without www, subdomains. Each host serves its own file: the wrong property leads you to diagnose the wrong host.

Errors and Warnings: How to Prioritise

Rank the report's signals by urgency:

• file unavailable or cannot be parsed: restoring access comes before any detailed diagnosis;
• server error on a host: a hosting issue, to hand to the technical team;
• warnings (inconsistent or ignored directives): fix them first on business-critical areas.

Always relate a signal to a business risk before changing the file.

Testing a Blocked URL: A Diagnostic Method Within Search Console

The legacy robots.txt testing tool has been retired. Diagnostics now combine the robots.txt report, the URL Inspection tool and a reading of the file's rules.

Connecting URL Inspection and the Report, Then Isolating the Rule

Follow this sequence before drawing conclusions:

• inspect the URL to verify its crawl and indexing status;
• check in the report the version of the file Google retrieved and its fetch date;
• identify which rule applies to the path you are testing.

This way you avoid attributing to robots.txt an issue actually caused by a redirect, a server error or a non-existent URL. To isolate the rule, find which directive targets which user-agent and which of Allow and Disallow is the most specific: a block usually results from a set of rules rather than a single line.

Common Scenarios: CSS/JS, Images, URL Parameters, Entire Directories

Four blocks come up in most diagnoses:

• blocked CSS/JS: directly impacts rendering and how the page is understood;
• blocked images: visual value and certain signals are lost;
• URL parameters: useful against duplication, risky if key pages depend on them;
• entire directories: practical for excluding technical areas, dangerous if internal linking passes through them.

Correcting robots.txt Without Damaging SEO

The objective is to align crawling with your priorities: keep required resources accessible whilst limiting exploration of low-value areas. The following table links each symptom visible in GSC to its check and its fix:

Safe Changes: Unblock What Must Be Rendered, Limit What Dilutes Crawl

Unblock resources critical to rendering (CSS, JS) and block only genuinely unnecessary variants, such as filter combinations or technical endpoints, through minimal, traceable and reversible changes. Listing a sitemap in the file remains useful as long as it does not include URLs blocked from crawling; submitted vs indexed URLs are compared in the Google Search Console Sitemaps report.

Critical Mistakes to Avoid

Four mistakes cause most accidental blocks:

• Disallow: /: a global block, common when a staging file slips into production;
• invalid encoding or format: the file becomes unreadable to Googlebot;
• path inconsistencies: rules that do not match the actual site structure;
• variant confusion: one host fixed, another forgotten.

Formalise a deployment checklist: peer review, post-release check, then follow-up monitoring in Search Console.

Choosing the Right Lever: robots.txt, noindex or 403

robots.txt prevents crawling, not discovery: a blocked URL can appear in Google through external links, with minimal information. These three levers therefore do not have the same effect:

A noindex is only read if the page remains crawlable: do not combine both on the same URL. Diagnosing access refusals is covered in the article on the 403 error in Search Console.

Requesting a New Crawl and Verifying the Impact

After a correction, processing time varies. The robots.txt report lets you request a recrawl of the file: reserve it for urgent fixes, migrations and server incidents.

When to Trigger a Fetch: Urgent Fixes or Gradual Adjustments

Request a recrawl in three situations:

• a block affects a business-critical section;
• a redesign or migration has just gone live;
• the server has been through a period of instability.

For crawl budget optimisation, work iteratively instead.

Validating the Return to Normal

A return to normal is checked on three signals:

• a fall in "Blocked by robots.txt" URLs in the Pages report;
• inspection of a representative sample of URLs;
• the recovery of impressions and clicks for affected sections in the Performance report.

The Crawl stats report additionally confirms that Googlebot has resumed its requests. A weekly check of these reports catches a block before it costs you high-performing pages.

Advanced Scenarios: Complex Sites and robots.txt Governance

On complex setups, governance matters most: version control of the file, clear separation between staging and production, and cross-team reviews help prevent accidental blocks.

Multi-Host Setups, Staging and Migrations

Three situations cause most blocks on these sites:

• rules that differ from one host to another;
• a staging file pushed to production by mistake;
• outdated rules that no longer fit the structure after a migration.

The multi-host view of the robots.txt report makes these discrepancies easier to spot.

Wildcards, End-of-String Matches and Low-Value Pages

Wildcards and end-of-string matches increase precision but multiply side effects: document the intent of each rule and test it against concrete URLs.

Do not, however, hide pages that generate leads or carry business signals: let them be crawled, and apply explicit indexing rules where needed.

robots.txt and Generative Engine Crawlers

Generative engine crawlers are managed with the same tools as Googlebot:

• Google-Extended, GPTBot and similar crawlers are declared in the same file, each under its own user-agent;
• blocking an AI crawler does not affect Googlebot, as long as the rules targeting Googlebot do not change;
• a block shows up in the robots.txt report like any other.

AI Overviews and AI Mode rely on the Google index: a page that is not indexed cannot appear in them. To decide what you open up to generative engines, see the guide to Generative Engine Optimization (GEO).

Depending on Your Situation

Depending on the block you found, continue here.

• The page must leave the index, not just the crawl. robots.txt is not enough: diagnose the "Excluded by 'noindex' tag" status with the article on noindex in Google Search Console.
• A URL must disappear from results quickly. Use the right tool to remove a URL from Google's results.
• Googlebot is visiting less without any visible block. The three-part host status, described in the article on the Google Search Console exploration section, shows whether the server is at fault.

To link each robots.txt block to the value of the affected pages and prioritise fixes, the Incremys 360° SEO and GEO audit cross-references Search Console and Google Analytics data.

FAQ: robots.txt and Google Search Console

Where can you find the robots.txt report in Google Search Console?

The report is found under Settings, in the robots.txt section. At the time of writing, it shows the files for the top 20 hosts, the last fetch date, errors and warnings, and lets you request a recrawl. It replaces the legacy robots.txt testing tool, which Google has retired.

Why can a page still appear in Google if it is blocked by robots.txt?

A page blocked by robots.txt can appear in Google because the file prevents crawling, not discovery. Google finds the URL through external links and displays it with minimal information. To prevent indexing, keep the page crawlable with a noindex, or protect it with authentication if it is confidential.

How do you know whether Googlebot is blocked from an essential resource (JS/CSS)?

The URL Inspection tool reveals whether Googlebot was unable to fetch an essential resource. Then confirm that the directories containing your scripts and stylesheets are not caught by a Disallow directive. Finally, check in the robots.txt report that the version of the file Google retrieved includes your correction.

What should you do if the robots.txt file is missing (404) or unstable (5xx)?

A missing or unstable robots.txt is fixed on the hosting side first: the file must be available on each host. Once access is restored, check that the robots.txt report shows no error and request a recrawl of the file. Whilst the file remains unstable, any detailed diagnosis of blocks remains uncertain.

How can you prevent a robots.txt change from disrupting a redesign or migration?

A robots.txt change will not disrupt a migration if it is versioned, reviewed and checked after deployment. Separate staging and production files, require a review step, then check the robots.txt report and a sample of URLs in the URL Inspection tool. After that, monitor Google Analytics for any abnormal traffic drop.

Can you block AI crawlers without blocking Google?

Yes, blocking AI crawlers in robots.txt does not affect Googlebot. Google-Extended, GPTBot and other generative engine crawlers are each declared under their own user-agent. Make sure no general rule also targets Googlebot, then check the retrieved version in the Search Console robots.txt report.