Should You Block AI Crawlers? A Lead-Gen Answer

Every article ranking for this question was written for a publisher. Publishers sell attention: an AI crawler that takes the article and gives back nothing is a thief, so the advice is block everything. You are not a publisher. You are a plumber, a dentist, a law firm, a B2B founder — you are paying to be found. Your content is a sales asset, not inventory.

So when you copy a blocklist off a publisher blog and paste it into your robots.txt, you are not protecting an asset. You are quietly deleting yourself from the answer engines your next customer is asking. This post gives you the bot-by-bot split nobody makes, and the literal robots.txt a lead-gen site should run.

What is an AI crawler, and how is it different from Googlebot?

An AI crawler is a bot that fetches your pages for a language model or an AI answer engine instead of a classic ten-blue-links index — and as of June 2026, Cloudflare reports that 52% of crawler requests are now for AI training, up from 22% in spring 2025.

Googlebot is one bot with one job: crawl for Google Search. The AI companies split that job across several named bots, and each one is controlled separately in robots.txt. That is the whole point of the design — and the whole reason a blanket block is so destructive.

Cloudflare now sorts these into three behaviors: Search (indexing you so it can answer questions about you later), Agent (a bot fetching your page right now because a human asked), and Training (absorbing your content into the model). Only one of those three is what publishers are actually angry about.

Which AI bots are training crawlers and which are retrieval crawlers?

Four bots feed answers users see today — OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot — and four feed models that ship later: GPTBot, ClaudeBot, Google-Extended and Applebot-Extended. Confusing the two is the expensive mistake. Every row below comes from the operator's own documentation.

Read that table twice, because three pairs get conflated constantly. GPTBot is not OAI-SearchBot. ClaudeBot is not Claude-SearchBot. Google-Extended is not Googlebot — Google's own crawler docs state that Google-Extended 'does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.'

Applebot-Extended works the same way: Apple's documentation says it 'does not crawl webpages' at all, and pages that disallow it 'can still be included in search results.' These extended tokens are training switches, not visibility switches.

What happens to your leads if you block the retrieval bots?

You vanish from that engine's answers, and OpenAI says so in plain English: sites opted out of OAI-SearchBot 'will not be shown in ChatGPT search answers, though can still appear as navigational links' (OpenAI crawler docs). There is no partial credit and no appeal.

Anthropic is equally direct. Disabling Claude-SearchBot 'prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.' Perplexity's docs recommend allowing PerplexityBot precisely so your site appears in results.

The commercial cost is measurable. Seer Interactive analyzed 5.47 million queries across 53 brands and found that being cited inside an AI Overview delivers 120% more organic clicks per impression than appearing on the same SERP uncited — though still 38% fewer clicks than a SERP with no AI Overview at all (Seer Interactive, 2026). A retrieval block does not just cost you AI visibility. It costs you the click lift that citation carries.

If your pages are already invisible to these engines for other reasons — JavaScript rendering, a login wall, a CDN challenge page — a robots.txt fix will not save you. That is a separate diagnosis, and it belongs in a technical SEO audit.

Which bots should a lead-gen service business allow and block?

Allow all four retrieval bots and all three user-fetch agents, without exception; the only genuine decision is the four training tokens, and for most service businesses the honest answer is allow them too. Here is the reasoning, not just the ruling.

• Always allow the retrieval bots. OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are how a stranger with a problem finds you. Blocking them to protect a page about drain cleaning is self-harm.
• Always allow the user-fetch agents. ChatGPT-User, Claude-User and Perplexity-User only visit because a human asked for you by name or by need. That is the warmest traffic on the internet.
• Training is a judgment call, and it is close to a coin flip. Blocking GPTBot or ClaudeBot costs you nothing in today's answers. It also gains you nothing: your service page is not a copyrighted archive that a model can resell.
• Allow training if you want long-run brand memory. Models answer plenty of questions without a live search. Content absorbed in training is one of the few ways a small brand ends up in a model's default recall.
• Block training only if you publish something genuinely proprietary — a pricing methodology, a real dataset, an original diagnostic framework you sell. That is a real reason. 'An agency blog said to' is not.

Our position at Foundgrove: for a US service business at typical scale, block nothing. You have no leverage to license content, no ad inventory to protect, and everything to gain from being in the answer. If you disagree on training, block the four training tokens and stop there. We treat SEO, GEO and AEO as one program for exactly this reason — see how we handle it on the GEO service page.

What should your robots.txt actually say?

For the block-nothing posture, seven lines is the entire file — no AI user-agent groups at all, because the default is already allow. Explicit allows are optional but they document intent, which is worth something when a new marketer inherits the site.

• User-agent: *
• Allow: /
• Disallow: /wp-admin/
• Disallow: /cart/
• Disallow: /thank-you/
• Sitemap: https://example.com/sitemap.xml

If you have decided to block training and keep retrieval — the most defensible middle position — add exactly these four groups and nothing else. Four groups, four training tokens. Every other bot stays allowed.

• User-agent: GPTBot
• Disallow: /
• User-agent: ClaudeBot
• Disallow: /
• User-agent: Google-Extended
• Disallow: /
• User-agent: Applebot-Extended
• Disallow: /

Two syntax rules people get wrong. A user-agent token must be spelled exactly as the operator publishes it — 'GPT-Bot' or 'gptbot/1.0' matches nothing. And a group applies to one token: writing 'User-agent: GPTBot, ClaudeBot' on one line is not valid robots.txt and silently blocks neither.

Finally, robots.txt is a request, not a wall. Anthropic and Google both state their bots honor it. Perplexity states outright that Perplexity-User 'generally ignores robots.txt rules' because a user requested the fetch, and OpenAI's own docs say that because ChatGPT-User actions are initiated by a user, 'robots.txt rules may not apply.' If you truly need enforcement, that is a firewall job, not a text-file job.

Does Cloudflare's AI bot blocking override your robots.txt?

Yes — Cloudflare blocks at the network edge, before your robots.txt is ever consulted, and that toggle is where most accidental self-deletion happens. More than 20% of the web sits behind Cloudflare, so this is probably your site.

It gets sharper. In its July 2026 announcement, Cloudflare said that on September 15, 2026 multi-purpose crawlers that combine Search with Training — it names Googlebot, Applebot and BingBot — 'will be blocked by customers who have selected to block Training (either through the new options to manage AI traffic, or through the legacy Block AI bots service).'

Read that again with your business hat on. If someone on your team once ticked 'Block AI bots' in Cloudflare, that setting is scheduled to start blocking Googlebot. Cloudflare says owners can opt out of the new defaults in their zone Security settings before September 15. Go look at that toggle today.

Cloudflare's newer controls are actually good news if you use them deliberately: all customers, free tier included, can now allow Search while blocking Training and Agent separately, and AI Crawl Control shows which crawlers hit you and which ignored your directives.

Is there ever a good reason for a service business to block AI crawlers?

Three, and only three — none of which is 'AI is stealing my content.' Be honest about which one you actually have before you touch a config file.

• Server load. Aggressive crawling on a cheap shared host can degrade the site for real buyers. The fix is a Crawl-delay directive (Anthropic explicitly supports it) or edge rate limiting — not a block.
• Genuinely proprietary content. A licensed dataset, a paid course, an original research corpus. Put it behind a login. A robots.txt disallow is a signpost, not a lock.
• Client or patient confidentiality. Case pages, portals, intake forms, anything with PII. Those should be noindex and behind auth regardless of AI — this is basic hygiene, not an AI question.

Everything else is publisher anxiety borrowed by a business with the opposite economics. Cloudflare's own framing concedes the point for small sites: the problem is not just that someone could train on your content, it is that nobody can find you in the first place.

How do you verify the bots are actually reaching your pages?

Four checks, in this order, and all four are free. Do them before you change anything and again 48 hours after — OpenAI notes it can take roughly 24 hours from a robots.txt update for its systems to adjust.

• Open yoursite.com/robots.txt in a browser. Read every line out loud. If you see GPTBot, Google-Extended or a wildcard Disallow you did not intend, you found it in 30 seconds.
• Grep your server or CDN logs for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Googlebot. Zero hits from the retrieval bots over 30 days means something upstream is blocking them.
• Check the edge, not just the file. In Cloudflare, open AI Crawl Control and the Bot Management settings. A 403 at the edge never shows up in robots.txt.
• Ask the engine. Open ChatGPT search and Perplexity and prompt for your service in your city. If competitors are cited and you are not, you have a retrieval problem — which may be access, or may be that nothing on the page is worth citing.

That last check is the one people skip, and it is the one that separates an access problem from a content problem. Access gets you eligible. Being the clearest, most citable answer on the page is what actually gets you quoted — which is the whole discipline covered in our GEO guide.

And no, adding an llms.txt file does not fix any of this. It is not a permission system, no major engine treats it as one, and it cannot undo a robots.txt disallow or a Cloudflare edge block.

If you want a second pair of eyes on your robots.txt, your CDN bot settings and whether the AI engines can actually see and cite your money pages, that is exactly what our GEO program starts with. Get my free audit — we will tell you what is blocked and what it is costing you, whether or not you hire us.