Server log analysis: the proof of what Googlebot really requests, and what you decide from it

Your crawl says one thing, Search Console another, the infrastructure team a third. The server logs settle it: they record what was actually requested, by whom, when, and what was answered.

This page is for the marketing, SEO or IT director who commissions or reads a log analysis. It shows how to make the data reliable, isolate Googlebot, locate the waste and the errors, then turn those findings into a backlog and measurable checks.

What the logs show that nothing else shows

In an SEO audit, a “strategic” page only performs if it is discovered, crawled, understood and indexed. Server logs document crawling precisely: what the robots actually request from the server, how often and with what results (statuses, redirects, errors).

The aim is not to “produce tables” but to answer operational questions:

• Does Googlebot come often enough to the pages that count (business, conversion, authority)?
• Where does the crawl budget go (parameters, facets, redirects, worthless pages)?
• Which server errors (4xx/5xx) do the robots actually meet, and on which sections?
• What explains a gap between “pages crawled” and “pages that perform”?

Search Console gives trends, but does not replace a reading of the requests actually served by your infrastructure. Access logs record every request received, robots included. They allow you to:

• measure crawling at URL level rather than in aggregates;
• recover history if your files are retained, whereas tracking only starts when the tag is placed;
• isolate behaviour by user agent;
• connect the statuses seen by robots to accessibility problems.

Search Console tells you “Google had a problem” or “Google is crawling less”. The logs tell you where, when, how much, with which code and on which URL families. That measurement remains one source among those a technical SEO audit cross-checks. Each source produces a different deliverable:

Reading a log file: what counts, what misleads

In SEO, the main material is the web access logs: timestamped events that describe every request made to the server, human and robotic alike.

The fields that carry the information

A log entry most often contains six useful elements:

• Date and time: indispensable for rebuilding “crawl windows”, spotting peaks and comparing before and after.
• Requested resource: path and, often, query string, therefore the parameters.
• Response code (200, 301, 404, 500 and so on) and sometimes response size.
• User agent: the client's signature, robot or browser.
• Referrer: useful mainly on the human traffic side, sometimes revealing internal loops.
• IP: useful for validating certain robots, with caution and in compliance with internal constraints and the GDPR.

One point of method changes everything: you do not read “line by line”. The value comes from grouping and aggregating by field: URL patterns, statuses, user agents, sections.

What distorts the reading

The major risk is concluding on incomplete data. Five traps recur:

• a CDN or a reverse proxy that masks the real IP or part of the robot traffic;
• multiple servers: if you do not aggregate every machine, you measure a fraction of the crawl;
• duplicated sources, which cause double counting at centralization;
• parasitic bots, which inflate the volumes and blur the reading;
• missing fields, since some formats record neither the referrer, nor the query string, nor the response time.

Do not confuse the two types of logs either. Access logs say “Googlebot requested /product-x → 500”; error logs say “application exception, saturation, external dependency”. An actionable decision needs both: the crawl impact and the cause.

Getting data that stays comparable over time

Collection has two objectives: losing nothing critical to the crawl, and making the data comparable from one period to the next. The second is the one most often missed.

The preparation pipeline

Before any analysis, formalize four stages:

• Extraction: a defined period, every machine, every relevant source.
• Anonymization: according to your internal constraints, often on the IP on the human traffic side.
• Normalization: a single date format, encoding, homogeneous fields, URLs decoded or encoded consistently.
• Checks: volume per day, gaps in time, expected robot-to-human ratio, consistency of time zones.

A traceability tip: keep an untransformed raw sample, with restricted access, so that the transformations can be audited. Cover at least one crawl cycle, often two to four weeks, with a before and after if fixes have just been deployed, and list 3 to 5 priority sections with the expected outcome.

Formats and environments: the control points

Each server environment calls for its own checks:

• Apache: the “combined” formats generally include the URL, the status, the size, the referrer and the user agent; check the presence of the query string, indispensable for detecting parameters and facets, a usable timestamp (format and time zone) and the response time where available.
• Nginx: highly customized formats are an advantage if you log exactly what is needed, a risk if fields are missing; check URL encoding and that the statuses reflect the final response.
• IIS: standardize case and URL normalization when the application generates variants, on pain of fragmented counts, and cover every server behind a load balancer.

Set an explicit rotation policy, for example 30 to 90 days in detail, then aggregate. With multiple servers, inconsistent time zones or clocks create false “peaks” or false drops: detect the “missing” days before concluding that crawling has dropped.

Isolating Googlebot: segmentation and validation

Without segmentation, you mix robots, browsers, probes and parasitic bots, when the decision is first of all about Googlebot. Start with readable, auditable rules:

• a Googlebot segment, mobile variants included;
• an other robots segment, useful for measuring noise and load;
• a humans segment, kept apart, if you are trying to connect certain errors to the experience.

User-agent impersonation is common: a header announcing “Googlebot” proves nothing. Check that the IP belongs to Google, through the IP ranges Google publishes or a reverse DNS lookup followed by a forward one. It can be automated and is a matter of routine, not of incident handling alone. It confirms who is crawling, not why.

Then compare Googlebot mobile and desktop: section coverage, the distribution of statuses (an excess of 4xx or 5xx on the mobile side is a signal in itself) and access to critical resources if rendering depends on JavaScript.

Measuring where Google invests its time

The most actionable question is rarely “how many Googlebot requests in total?” but “where does Google put its crawl effort?”. It is handled by URL family, over comparable windows.

Splitting by URL families

Structure the reading by directories (/blog/, /products/, /category/, /help/), by templates (product pages, categories, tags, pagination, internal search, utility pages) and by technical variants (parameters, trailing slash, http/https, www or non-www). That level detects an under-crawled section without drowning the analysis in thousands of URLs. For each family, produce four measurements:

• coverage, the number of unique URLs crawled;
• median recency, that is the date of the last visit;
• recurrence, the number of returns over the period;
• the distribution of statuses.

Add the response time served to the robots if it is logged: it points to the slow templates.

Spotting an under-crawled section

The pace is read on recency (when was a URL last seen?), recurrence (does the robot come back regularly to the important pages?) and windows (do bursts of crawling follow a release or a redesign?). Over several weeks, it verifies that fixes free up budget for the benefit of the business pages.

Three signals point to under-crawling:

• high-value pages rarely seen by Googlebot;
• sections present in the sitemap and poorly linked whose visits remain rare (logs say that a URL was requested, not by which path it was discovered: the connection is confirmed by a crawl);
• crawling concentrated on “easy” pages, tags or filters, at the expense of the substantive pages.

The hypotheses to test next: excessive depth, orphan pages, duplication, internal links pointing to redirected URLs, over-prolific templates. Knowing why the engine arbitrates this way (perceived inventory, authority, freshness) belongs to SEO crawling.

Locating the waste and the access errors

Every useless request consumes capacity that does not go to the pages that must progress. Search Console's Crawl Stats report gives an aggregated view of it (responses, file types, Googlebot type); the logs quantify it URL by URL.

Where the budget goes

Waste concentrates on parameter URLs (sorting, tracking, filters), deep pagination, internal search, facet combinations, unstable URLs (slash, case) and A → B → C redirect chains. The logs quantify it: measure the share of Googlebot requests that end in a 3xx, then the most requested intermediate URLs, the figure that makes a URL governance project defensible.

The statuses that force an investigation

What matters is seeing what Googlebot really meets, not what the team thinks it is serving. Five situations trigger an immediate investigation:

A 404 or a 410 can be deliberate; a 403, legitimate or accidental. On 5xx errors, look at the concentration more than the volume (sections, templates, hours), to prevent instability from “teaching” the engine to slow its visits.

On 3xx, produce four views:

• the share of 3xx in the Googlebot crawl;
• the top source URLs still being requested;
• the top final destinations;
• chains and loops.

If your internal links still point to URLs returning 301 or 302, you are paying a permanent “crawl tax”. A 200 is not always good news either: empty pages and filters with no results pollute the coverage, and a very small response size points towards a soft 404 without proving one.

Keep, consolidate, limit, de-index

A “crawl budget” decision is not only technical. Once the waste has been located, four outcomes are chosen family by family: keep a URL with unique SEO value, consolidate several URLs serving the same intent, limit the crawling of a URL that is useful to users but has no SEO value, de-index one that pollutes the coverage. When the question becomes “which rule across a whole catalogue”, the trade-off belongs to the SEO crawl budget, which handles capacity, demand and facets at volume.

From finding to decision

A reading is worth only what it triggers: the translation of a symptom into a testable hypothesis, then the measurement that will prove the fix produced its effect.

Connecting symptom, hypothesis and action

These translations cover most cases:

By bringing the crawling recorded in the logs together with impressions, clicks and conversions, you identify the pages or templates that absorb crawling without producing value, and those that have value but lack crawling. From the first, you take crawling away; to the second, you give it.

Formalizing and validating

Structure the backlog in three levels, then order each level with the audit's impact × effort × risk grid:

• the quick wins: internal links to redirects, systematic 404 errors, short chains that are easy to fix;
• the structural projects: parameter and facet governance, consolidation of duplication, re-architecture of under-crawled sections;
• the post-fix checks: same dashboard, same segments, comparison over an identical window.

Validation is done on a similar window (same weekdays, same duration) and on the same segments. It expects three effects:

• a lower share of 3xx, 4xx and 5xx on Googlebot;
• fewer requests on the polluting URLs;
• a higher crawl frequency on the strategic pages.

Only in the medium term does Search Console show the effects on coverage, then on impressions and clicks. Measuring remains indispensable: a fix that is not measured often ends up contested or forgotten.

Tooling the reading without losing the rigour

Tooling has to move from millions of lines to a few actionable patterns, and redo the same calculation six months later. In a B2B context, five criteria decide:

• volume: ingesting heavy files and querying them without slowdown;
• security: access control, segmented rights, anonymization;
• traceability: audited transformations, saved views, history;
• exports: URL lists and aggregates meant for the backlog;
• collaboration: annotations, shared views, reproducibility.

The common approaches differ in where they hold:

With scripts, one reflex counts: version the script and freeze the normalization rules. Otherwise you cannot reliably compare two periods. An online analysis, for its part, is done on an anonymized extract with a documented scope: logs contain IPs, sensitive paths, sometimes identifiers.

Automation speeds up grouping into patterns, anomaly detection and classification by template. Three safeguards frame it: explainability, each grouping staying traceable to a rule; validation on samples (top URLs, top errors) before prioritizing; and bias: do not confuse “heavily crawled” with “important”; it is sometimes the opposite.

That leaves setting up the routine, at three paces:

• weekly: alerts on 5xx, on 429s, on 404 spikes in business sections and on the explosion of parameters;
• monthly: frequency trends per section, shares of 3xx, the movement of waste;
• post-deployment: a before-and-after comparison over a comparable crawl window.

Add annotations (a release, a redesign, a change to robots.txt or to the sitemap) to avoid mistaken interpretations.

Continue reading

Depending on your situation, two reads take the analysis further. Your logs point to slow templates and you have to decide whether the project is worth its cost: the website performance audit gives the speed measurement and the trade-off on the front-end project. You do not have access to the server logs yet, or want to see first what Google exposes on its own: the Google Search Console reports and how to read them give a first picture of coverage and exclusions.

Bringing the under-crawled sections recorded in the logs together with the impressions and clicks they generate, so as to sort out what deserves a project, is the task covered by the audit and mapping module.

FAQ on server log analysis

What is server log analysis in SEO?

Server log analysis uses access logs to measure how Googlebot actually crawls a site: which URLs are requested, how often, with which response codes. It serves to improve the accessibility of strategic pages and to reduce crawl waste.

How do you build a reliable diagnosis from a log file?

A reliable diagnosis follows six stages: collect, normalize URLs and timestamps, segment by user agent, aggregate by section and template, analyse statuses and frequency, then cross-check with Search Console and analytics. Skipping the normalization makes periods incomparable.

How do you analyse Googlebot's crawl and secure the identification?

Segment on the user agent, then validate the real origin of the requests through the IP ranges published by Google or a reverse DNS lookup followed by a forward check. As user-agent impersonation is common, this validation is automated and built into the routine rather than being kept for incidents.

How do you measure crawl frequency per section and detect under-crawling?

Under-crawling shows up when business pages have an old last visit and low recurrence, while secondary pages absorb most of the requests. It is measured by directory and template: unique URLs crawled, number of requests, recency and recurrence over a given period.

Which HTTP response codes should trigger an immediate investigation?

5xx errors, 429s, 404s and 410s on URLs that are still linked, 3xx in chains or loops and suspect 200s trigger an investigation. The decisive criterion remains the concentration on strategic pages, not the raw volume.

What are the differences between Apache, Nginx and IIS logs?

Apache, Nginx and IIS all produce usable access logs, but with different formats and logging options; Apache and Nginx mostly run on Linux, IIS on Windows. The method stays the same: the parsing adapts to each format and the structure is standardized at ingestion.

Which tool should you choose between a complete log analyser and an open-source analyser?

An open-source analyser fits when the volume is moderate and the team is comfortable with parsing. A more complete tool becomes necessary as soon as the volume explodes, security and traceability become critical, or exports and routines have to be industrialized.

When does artificial intelligence bring a real gain to the analysis?

Artificial intelligence brings a real gain on repetitive, high-volume tasks: grouping into patterns, detecting anomalies, classifying URLs by template. It stays controlled by the explainability of the groupings and validation on samples; otherwise it produces biased conclusions presented as measurements.