Why ChatGPT Can't See Your Website: 7 Blockers

You typed your company name into ChatGPT and got a shrug. Or worse, it described a competitor. Your first instinct is that you need better content. Usually you don't. Usually the crawler never got a usable copy of your page in the first place.

This is a fault tree, not a strategy essay. Work down it in order. Each blocker has a test you can run in a terminal in under a minute, and a fix.

Why can't ChatGPT see your website at all?

Seven blockers cover nearly every case, and six of them are server-side — nothing to do with how well you write. The order below is the order to check them in, because the cheap tests come first.

If you only have time for one of these, do the first. It is by far the most common, and it is the one nobody looks for.

Do AI crawlers execute JavaScript, or do they see an empty page?

They see an empty page. Vercel and MERJ analyzed AI crawler traffic across Vercel's network and concluded that none of the major AI crawlers currently render JavaScript — that list explicitly includes OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Meta-ExternalAgent, ByteDance's Bytespider, and PerplexityBot (Vercel + MERJ).

The cruel detail: they do download your JavaScript files. Vercel measured 11.50% of ChatGPT's fetches and 23.84% of Claude's going to JS files. They fetch them and never run them. So your bundle burns crawl budget and returns nothing.

This is the opposite of Googlebot. Google's own documentation says Google Search runs JavaScript with an evergreen version of Chromium, queues every 200-status page for rendering, and indexes the rendered HTML (Google Search Central). Google adds the quiet warning that matters here: 'not all bots can run JavaScript.'

So a page can rank fine in Google and be a blank sheet of paper to ChatGPT. Same URL. Same content. Different reader.

The fix is server-side rendering — SSR, static generation, or plain old server-rendered HTML — for anything that carries the answer: your services, your prices, your service area, your hours, your FAQs. Client-side rendering is fine for a chat widget or a review carousel. It is not fine for the paragraph you want quoted. If you're deciding what to rebuild on, we compared the tradeoffs in Next.js vs WordPress for a marketing site, and it's the first thing we look at in a technical SEO audit.

Is Cloudflare or your host returning a 403 to AI crawlers?

Very possibly, and possibly without you ever touching a setting: on July 1, 2025, Cloudflare announced it was 'changing the default to block AI crawlers unless they pay creators for their content' (Cloudflare).

That is a defensible business position for a publisher selling ad impressions. For a plumber who wants ChatGPT to recommend them at 11pm, it is a self-inflicted wound. We break down which bots to allow and which to block in should you block AI crawlers — the short version is that a service business almost always wants the search and assistant bots in.

Is your robots.txt blocking bots you never meant to block?

Check for one line, because OpenAI runs three separate agents and blocking the wrong one costs you visibility you wanted. OpenAI's documentation is explicit: GPTBot is for model training, OAI-SearchBot is for search, ChatGPT-User is for user-triggered fetches, and 'each setting is independent of the others' (OpenAI).

The line that costs money: OpenAI states that 'sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.' You do not vanish — you get demoted from a cited answer to a bare link. A developer who copy-pasted a blanket AI-bot disallow to protect your content from training made that trade for you. Those are two different decisions and a lot of themes and plugins conflate them. Read every Disallow line in your robots.txt yourself.

Does your page need to be in Bing's index to be retrieved?

Not for ChatGPT specifically — OpenAI crawls with its own agent, OAI-SearchBot — but Bing is a second, separate retrieval surface you are probably ignoring. Microsoft's Bing Webmaster Tools now reports 'AI Performance,' which shows when your site is cited in AI-generated answers across Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations (public preview, February 2026 — Bing Webmaster Blog).

That makes Bing the only free dashboard that will tell you your pages are being cited by an AI assistant at all. Set it up. It takes twenty minutes and it is the closest thing to a GEO analytics tool that exists for free. More on that surface in Bing SEO for ChatGPT and Copilot citations.

While you're in there, wire up IndexNow. It's a free open protocol that pings participating engines the moment a page is added, updated, or deleted. You host one key file at your root, then POST your URLs — up to 10,000 per request (IndexNow docs). Bing's own guidance ties it directly to AI answers: fresher content means AI systems reference the current version of a page rather than a stale one.

One trap: an HTTP 200 back from IndexNow only means the engine received your URL. Their documentation says so in plain language. It is not a promise of indexation, and anyone selling you IndexNow as an indexing guarantee is selling you a receipt.

Is your answer trapped in an image, PDF, or video?

If your price list is a JPEG, your price list does not exist. Google's AI-features guidance lists 'making sure that important content is available in textual form' as a core requirement for AI Overviews and AI Mode eligibility (Google Search Central), and Bing's AI guidance says the same thing from the other side: clear headings, tables and FAQ sections make content easier for AI systems to reference accurately.

The service-business version of this failure is depressingly consistent. Hours in a header graphic. Service area on a map image with no text list. Financing terms in a downloadable PDF. A five-minute explainer video with no transcript. Every one of those is an answer an assistant would have used and couldn't read.

Rule of thumb: if you cannot select the text with your cursor and paste it into a notepad, a crawler cannot lift it into an answer. Put the fact in HTML, then use the image, PDF, or video to support it.

How do you test what an AI crawler actually sees on your page?

Four commands and one dashboard, in this order. This takes about ten minutes and it beats every guess you could make from your own browser, because your browser runs JavaScript and the crawler does not.

• Step 1 — Status check. Run: curl -s -o /dev/null -w '%{http_code}' -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot' https://yoursite.com/your-page — anything that isn't 200 is your blocker.
• Step 2 — Repeat with the OAI-SearchBot and ChatGPT-User agent strings from OpenAI's docs. A WAF rule often blocks one and not the others, which is why 'it worked when I tested it' happens.
• Step 3 — Raw HTML check. Save the response to a file with curl and search it for your H1 and your key paragraph. If the text is not in that file, no AI crawler can see it, full stop.
• Step 4 — Header check. Run curl -I on the same URL and look for an x-robots-tag: noindex header or a 401. Staging configs leak into production more often than anyone admits.
• Step 5 — Server logs. Grep your access logs for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. OpenAI publishes its IP ranges, so you can confirm a hit was genuine and not a spoofed user-agent.

Log analysis is the step people skip and the step that tells the truth. If the AI crawlers are hitting you and finding nothing, you have a rendering problem. If they never show up at all, you have an access problem. Those are different repairs, and the logs are the only place that distinction is visible. It's the same starting point as any real technical SEO audit.

How long after you fix this before an AI engine cites you?

Plan for weeks, not days — with one fast lane. OpenAI's docs say it can take roughly 24 hours from a robots.txt change for their systems to adjust, so unblocking is quick. Being crawled, retrieved, and then chosen as a source is a separate and much slower process, and nobody publishes an SLA for it.

Vercel's crawl data explains why. ChatGPT's crawler spent 34.82% of its fetches on 404 pages and Claude's 34.16%, against 8.22% for Googlebot. AI crawlers are still comparatively bad at finding the right URLs. Clean sitemaps, real redirects, and IndexNow pings are how you shorten that gap.

Fixing access is necessary, not sufficient. Ahrefs analyzed 863,000 SERPs and 4 million AI Overview URLs and found that just 38% of pages cited in AI Overviews also rank in Google's top 10 for the same query — down from about 76% a year earlier (Ahrefs, March 2026). Retrieval and ranking have partly decoupled. Getting crawlable puts you in the pool; being the clearest, most citable answer on the page is what gets you picked. That is the work in our GEO program.

And no, nobody can guarantee you a ChatGPT citation. Anyone who does is guessing with your money.

If you want someone to run the seven checks above against your actual domain and hand you the raw curl output plus a fix list — not a 40-page PDF — that's what a free audit is. No contract, no pitch deck. Get my free audit.