See What Googlebot Does Without a Log File Analyzer

Search log file analysis seo and every result is a tool vendor or a glossary entry. Screaming Frog's Log File Analyser, JetOctopus, Oncrawl, Botify. All of them are real products. None of them are your first move if you run a 60-page plumbing, dental, or law-firm site.

The reason 'log file analysis seo free' keeps showing up in related searches is that the demand is real and the SERP does not answer it. Here is the free answer, plus the honest line at which paying for a log analyzer starts to make sense.

What can you learn from server logs that Search Console won't tell you?

Raw logs give you one line per request — timestamp, IP, user agent, requested URL, status code, bytes — for every bot and human, with no sampling and no 90-day cap. Search Console's Crawl Stats gives you Googlebot only, aggregated, with a handful of example URLs per bucket.

That gap matters on a 500,000-URL ecommerce catalog with faceted navigation. On a service site with 60 pages, it mostly does not. Google says so itself in the Crawl Stats report documentation: 'This report is aimed at advanced users. If you have a site with fewer than a thousand pages, you should not need to use this report or worry about this level of crawling detail.'

Read that as permission, not a ban. The report is still the fastest free way to catch a server problem before it eats your rankings. Just don't buy a $250/year tool to look at a site Google can crawl in one afternoon.

Where is the Crawl Stats report and what does each chart mean?

Crawl Stats lives under Settings in Google Search Console, in the Crawling section, and it covers the last 90 days. One catch: it only exists on root-level properties — a Domain property or a root URL-prefix property. If your GSC property is a subfolder, you will not see it.

Three charts sit at the top, and four groupings sit below them. Each one answers a different question.

Above the charts sits Host status. Google's documentation defines it as a 90-day availability check: green means 'Google didn't encounter any significant crawl availability issues on your site in the past 90 days,' yellow means issues happened more than a week ago, and red means issues inside the last week. Red is the only one that should ruin your morning.

Which three crawl patterns mean something is actually broken?

Three patterns are worth acting on inside 48 hours; the rest of the noise in that report is normal variance. Google's own crawl budget guidance explains why: 'If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.' Your server's behavior directly sets your crawl ceiling.

A one-day blip of 503s during a host maintenance window is not an emergency. A week of them is. If host status is red and your rankings dipped in the same window, the cause is your server, not your content — and no amount of blog posting fixes it. That is exactly the kind of thing a technical SEO audit is supposed to catch before it costs you a quarter.

Is Googlebot wasting its visits on pages that don't matter?

Open the 'By purpose' grouping and the sample URLs under 'By response.' If more than a handful of the URLs Google is fetching are tag pages, author archives, paginated blog listings, or URLs with tracking parameters, Googlebot is spending its visits on pages that will never win you a customer.

This is the one thing a small site genuinely gets wrong. A WordPress site with 60 real pages can easily expose 400 crawlable URLs once you count /tag/emergency-plumber/, /author/admin/, /2025/03/, and every ?utm_source variant a newsletter ever produced.

The fix is boring and it works: noindex the archives you don't need, remove the internal links that feed them, and make sure your XML sitemap contains only the pages you actually want ranked. Then watch whether the sample URLs in Crawl Stats shift toward your service and location pages over the next month. That shift is the whole scoreboard.

Does a 60-page site have a crawl budget problem?

Almost certainly not. Google's crawl budget documentation is explicit about who needs to care: 'Large sites (1 million+ unique pages) with content that changes moderately often (once a week)' and 'Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily).' A 60-page HVAC site is not in either bucket.

There is one exception in that same document, and it is the one that bites small sites: sites with 'a large portion of their total URLs classified by Search Console as Discovered - currently not indexed.' If half your URLs sit in that state, the problem is not budget — it is that Google looked at your site and decided the pages weren't worth fetching. That is a quality and authority problem, and it is covered in our guide on reading Search Console like an operator.

So when an agency tells a 60-page contractor they need a crawl budget optimization retainer, they are selling you an enterprise problem you do not have. Ask them to show you the Crawl Stats screenshot that proves it.

How do you see whether AI crawlers like GPTBot are hitting your site?

Search Console will never show you this — it reports Googlebot only. AI crawler hits show up in exactly two places: your host's raw access logs, or your CDN's analytics (Cloudflare, Fastly, Vercel all expose bot traffic by user agent). Grep the log for the user-agent token and count the lines. That is the entire technique.

The tokens worth grepping for, taken from each vendor's own published documentation:

The distinction matters when you decide what to block. OpenAI's bot documentation says disallowing GPTBot 'indicates a site's content should not be used in training generative AI foundation models' — while 'sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.' Perplexity's docs draw the same line: PerplexityBot 'is not used to crawl content for AI foundation models,' it exists to link you in results.

Our position: block the training bots if you want to, but never block the search bots. A service business that blocks OAI-SearchBot and PerplexityBot to 'protect its content' has just opted out of the fastest-growing referral surface on the internet. That trade is covered in more detail on our GEO service page.

At what point is a paid log file analyser worth the money?

Roughly at 10,000 URLs, or the moment you can name a specific question Crawl Stats cannot answer. Below that, a paid analyzer tells you things you already know, and the setup cost — getting access logs off a managed host, parsing them, keeping them — is real work you are not being paid for.

Buy the tool when one of these is true. Otherwise, don't.

• You have 10,000+ URLs, or faceted/filtered URL generation you can't fully count
• You are mid-migration and need to prove Googlebot is following the new 301s, URL by URL
• A large share of your URLs sit in Discovered - currently not indexed and you need to know whether Google ever fetched them at all
• You need to audit AI crawler behavior over time, not just a one-off grep
• You must verify which bots claiming to be Googlebot are real, at scale

That last one has a free version. Google's Googlebot verification guide tells you to run a reverse DNS lookup on the IP, confirm the domain is googlebot.com, google.com, or googleusercontent.com, then run a forward lookup on that domain and confirm it resolves back to the same IP. Google also publishes IP range files (common-crawlers.json, special-crawlers.json, user-triggered-fetchers.json) if you want to automate it. Anything failing that check is a scraper wearing Googlebot's name.

For most service businesses the whole crawl question is 20 minutes a month in Crawl Stats and a periodic sanity check of what Google is actually fetching. If your host status is red, your crawl requests fell off a cliff, or half your URLs are stuck at Discovered - currently not indexed, that is a technical problem worth a real diagnosis — start with our SEO audit service or just Get my free audit and we'll tell you which of the three it is.