Googlebot Blocked by robots.txt: Find and Fix It Fast

robots.txt is two lines of text that can quietly delete a service business from Google. No error page, no warning email, no drop in your website's uptime monitor. Traffic just stops, and six weeks later somebody finally opens Search Console.

It is also one of the easiest technical problems to fix, because the fix is usually deleting one line. The hard part is knowing where the line came from, and understanding what robots.txt actually controls — because most people think it hides pages from Google, and it does not.

What does 'Blocked by robots.txt' actually stop Google from doing?

It stops Googlebot from requesting the page — one thing, nothing else. Google's robots.txt documentation is blunt about the limits: 'A robots.txt file tells search engine crawlers which URLs the crawler can access on your site,' and 'it is not a mechanism for keeping a web page out of Google.'

So a Disallow rule blocks crawling, not indexing. Google can still index a URL it has never fetched, based on links pointing at it. It just has nothing to show — no title it trusts, no description, no snippet.

Practically, here is what a Disallow line costs you on a blocked page:

• Google cannot read your headings, copy, prices, service areas, or CTAs on that page
• Google cannot see structured data on that page (LocalBusiness, FAQ, Service)
• Google cannot follow the internal links out of that page, so pages it links to lose that link signal
• Google cannot see a noindex tag on that page — the tag never gets read, so it never fires
• Google can still list the URL with no description if other pages link to it

That fourth one is the trap that burns people. Google's guidance on blocking indexing is explicit: 'We have to crawl your page in order to see meta tags and HTTP headers,' and it names robots.txt as the reason a noindex never fires — 'the robots.txt file is blocking the URL from Google web crawlers, so they can't see the tag.' Google adds that a disallowed URL can still be indexed if it is linked from other places on the web. Blocking and hiding are different jobs.

How do service businesses accidentally block their whole site?

Nearly every accidental site-wide block traces back to one of six sources, and the most common by far is a staging rule that shipped to production. Here is the forensic list, and how you tell which one you have.

The staging case is the expensive one. A developer adds Disallow: / to keep the build site out of Google, launch day arrives, the theme gets copied to production, and the two-line file rides along with it. The site works perfectly. It just does not exist to Google.

If your rankings vanished within days of a website relaunch, check robots.txt before you touch anything else. A full technical SEO audit starts here for exactly this reason — it is a five-minute check that occasionally explains a year of missing leads.

Is the WordPress 'Discourage search engines' box still ticked?

In WordPress 5.3 and later, that checkbox does NOT write a Disallow rule — it injects a noindex tag instead, which is a different failure with the same symptom. WordPress documents the behavior change directly: since 5.3 it 'causes meta name=robots content=noindex,nofollow to be generated into the head section' of your site. In WordPress 5.2 and earlier, the same checkbox made robots.txt return User-agent: * / Disallow: /.

So on a modern WordPress install with the box ticked, your robots.txt looks clean and your pages still will not rank. Search Console reports 'Excluded by noindex tag', not 'Blocked by robots.txt'. Two different reports, two different fixes, one identical checkbox.

Go to Settings > Reading in wp-admin and confirm 'Discourage search engines from indexing this site' is unchecked. Then view the page source of your homepage and search it for the word noindex. If it appears in a meta robots tag, something is still telling Google to stay away — the checkbox, a plugin, or a theme setting.

How do you check your robots.txt in 60 seconds?

Type your domain plus /robots.txt in a browser — Google tells you to 'place the robots.txt file in the top-level directory of a site,' so yoursite.com/robots.txt is the only place that counts. A file at yoursite.com/blog/robots.txt does nothing at all, because Google's crawlers 'don't check for robots.txt files in subdirectories.'

Then run these four checks in order:

• Read the file. Any line that says Disallow: / under User-agent: * or User-agent: Googlebot blocks the entire site. Disallow: with nothing after it blocks nothing — that is fine.
• Check case. Google states that the path value of a disallow rule 'is case-sensitive,' and its own matching table shows a rule for /fish does not match /Fish.asp. A block on /Services does not block /services.
• Run URL Inspection in Search Console on a page that should rank. If coverage says 'Blocked by robots.txt', you have your answer and Google will name the URL.
• Open the robots.txt report in Search Console. It shows which robots.txt files Google found, when they were last crawled, the file size, and any parsing issues.

Two more things worth knowing before you go rule-hunting. Per Google's robots.txt specification, Google enforces a robots.txt file size limit of 500 kibibytes and ignores content past that limit — so a bloated auto-generated file can silently drop the rules at the bottom. And when multiple rules match a URL, Google 'uses the most specific rule based on the length of the rule path,' with the least restrictive rule winning ties, which is why an Allow line can rescue one folder inside a blocked directory.

If Search Console is unfamiliar territory, our Google Search Console guide for service businesses walks through the reports that matter and the ones you can ignore.

Why can a blocked page still show up in Google with no description?

Because Google indexed the URL from links, not from the page. Google states it plainly: 'If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description.' You may also see the older phrasing — 'No information is available for this page' — on the result itself.

This is the single most misunderstood behavior in technical SEO, and it has an expensive consequence: robots.txt is the wrong tool for hiding a page. If you disallow a page you want gone, you freeze it in whatever half-indexed state it is in and remove Google's ability to read the noindex you were counting on.

Want a page out of Google? Let Google crawl it, and serve a noindex meta tag or X-Robots-Tag header. Want a page not crawled — an infinite filter, a print view, a calendar with 4,000 URL variants? That is a robots.txt job. Choosing between the directives is its own decision tree, covered in noindex vs nofollow vs disallow.

What should never go in robots.txt?

Anything you want kept private, anything you want removed from Google, and anything Google needs to render your page. robots.txt is a public file — every disallowed path you list is a directory you have advertised to the world.

• Do not Disallow: / on a live site. Ever. It is not a soft signal.
• Do not block /wp-content/ or your CSS and JS. Google needs to render the page; a blocked stylesheet can make your mobile layout look broken to Googlebot.
• Do not use robots.txt to hide a page from search results — it cannot do that, and it blocks the noindex tag that can.
• Do not list your admin URLs, staging subdomain, or client-portal paths as a way of 'protecting' them. Password-protect them instead.
• Do not block a URL you are also trying to remove — the removal needs a crawl to take effect.
• Do block genuine crawl waste: faceted-filter parameters, internal search results, endless calendar pages, and cart or booking-step URLs.

For a small service-business site — 20 to 200 pages — the honest answer is that you probably need almost nothing in robots.txt. A User-agent: * with an empty Disallow and a Sitemap line is a perfectly good file. Complexity here buys you nothing and costs you traffic when it goes wrong.

What do you do the moment you find a bad Disallow line?

Delete the line, deploy, then force Google to refetch the file — Google 'generally caches the contents of robots.txt file for up to 24 hours,' so the old rules can stay live in Google's memory even after your fix ships. Do these five things in order.

• Fix the source, not the file. If a plugin or CMS setting is generating the rule, editing the served file gets overwritten on the next update. Find the toggle.
• Reload yoursite.com/robots.txt in a private window and confirm the offending line is gone from the live response, not just from your editor.
• Request a recrawl in Search Console's robots.txt report — the report lets you request a recrawl for a file, which is exactly what you want after unblocking important URLs.
• Run URL Inspection on your homepage and top three money pages, then hit 'Request Indexing' on each. Do not spam it across 200 URLs.
• Check for a noindex tag too. A site that was blocked in robots.txt often has a noindex from the same launch mistake — and the noindex only starts working once Google can crawl again.

Recovery is not instant. Google has to recrawl the file, then recrawl the pages, then re-evaluate them. On a small site with a healthy sitemap, expect days to a few weeks — and expect the rankings that come back to be the ones you had, not better ones. Removing a block restores your baseline; it does not build authority.

One last operator note: put a check on it. A weekly automated fetch of /robots.txt that alerts you when the file changes costs nothing and would have caught every single case on the list above.

If your traffic fell off a cliff after a relaunch and you want someone to actually look at the file, the crawl logs, and the index reports rather than sell you a content package, that is what our technical SEO audit does — no lock-in, no ranking guarantees. Get my free audit.