llms.txt Explained: Does It Work, and How to Do It Right

llms.txt is the GEO topic that gets the most breathless coverage and the least honest assessment. The hype: it's the robots.txt of the AI era. The reality: as of mid-2026, no major AI crawler actually fetches it at scale. The right move: publish it anyway, because the cost is one afternoon and the upside is real if adoption catches up. Here's the honest version of the story.

This post covers what llms.txt is, what the audit data actually shows, and exactly how to publish it correctly if you decide to. For the broader GEO context, see the pillar. For where to put your schema work instead, see the schema deep-dive.

What is llms.txt, exactly?

llms.txt is a proposed standard, introduced by Jeremy Howard (co-founder of Answer.AI and fast.ai) in 2024, for a single markdown index file at the root of your domain. It summarizes what your site is and links to your most important content, so an LLM or agent can orient itself without crawling everything. It is structurally similar to sitemap.xml but written for LLMs rather than search engines.

Two things it is constantly confused with. It is not a robots.txt-style directive — despite the filename, it controls nothing and blocks nothing. And it is not the practice of publishing markdown copies of every page, which is a separate tactic with its own problems. llms.txt is an index file. That's all it is.

The standard lives at llmstxt.org and has a specific format: a top-level H1 with the site name, a blockquote with a short description, then sections of links organized by topic. Each link can have a short markdown description. It's deliberately simple — designed to be both human-readable and trivially parseable by any LLM.

How many sites have adopted llms.txt?

About one in four, in the most generous sample available. Ahrefs analyzed 137,210 domains in its Web Analytics panel and found that 28% — roughly 38,000 sites — publish an llms.txt file. Ahrefs is explicit that its panel skews more technical and SEO-aware than the web at large, so treat 28% as an upper bound, not a web-wide adoption rate.

Service businesses are barely represented in that population. If you publish one, you're an early adopter in your segment, not a laggard playing catch-up. And adoption has been driven by speculation that AI platforms may start reading the file — not by any confirmation that they do. Publishing a file is not the same as anyone reading it, which is the root cause of the next question.

Do AI crawlers actually fetch llms.txt?

Overwhelmingly, no. In Ahrefs' analysis of 137,210 domains, 97% of published llms.txt files received zero requests in May 2026 — no bots, no humans, nothing fetched them at all. Of the 3% that got any traffic, the AI retrieval bots that answer live user queries (OAI-SearchBot, PerplexityBot, Claude's search crawler) accounted for just 1.1% of requests. Ahrefs notes that Slackbot alone fetched llms.txt files more often than PerplexityBot did.

But 'nobody reads it' is too glib. Among the files that did get fetched, named AI bots made up 19.5% of requests — and the biggest AI reader is not an AI search engine. AI agents and agentic infrastructure sent 10.5% of requests, and AI training crawlers sent 5.3%, with GPTBot alone accounting for 4.51%. Anthropic's Claude-Code out-fetched every AI retrieval bot, every AI assistant, and every training crawler except GPTBot.

So the file does have a readership: coding agents and training crawlers, not the AI search engines most people publish it for. Two caveats keep this honest. Ahrefs' percentages describe only the 3% of files that got any request at all — roughly 1.1K domains — and a fetch is not a read. Ahrefs states plainly that every figure in the study is a ceiling on actual llms.txt consumption.

Google, meanwhile, has managed to argue both sides. Its guide to generative AI features has a section titled 'Mythbusting generative AI search' which states you don't need to create machine-readable files, AI text files, or Markdown to appear in Google Search including its generative AI capabilities, 'as Google Search itself doesn't use them.' Days later, the Chrome team shipped an llms.txt audit inside Lighthouse's agentic-browsing checks, whose documentation says that without the file, 'agents may spend more time crawling the site to understand its high-level structure and primary content.'

Both of those are Google. Search says ignore it; the agent team audits for it. That contradiction is the entire story of llms.txt in 2026 — it is not a search asset, and it might become an agent asset.

One finding kills the most common sales pitch outright. Ahrefs measured every request to llms.txt URLs that returned a 404 and found that zero came from AI bots — 98% of the probes for missing files were humans, presumably SEOs checking on competitors. AI tools never go looking for an llms.txt that isn't there. They fetch it when a link, an index, or a user instruction tells them it exists. 'Publish it and you'll show up on the AI radar' is backwards: publishing alone puts you on no radar at all.

Then why publish it at all?

Three honest reasons — and one real risk that nobody selling you the file will mention. First, coding agents: if your buyers use tools like Claude Code to source recommendations, that is the one audience the data shows actually reads the file. Second, future-proofing — Google has been clear that search is going agentic, and if agents end up mediating AI answers, the file could matter through that layer. Third, internal RAG: if you build a chatbot or site search against your own content, llms.txt is a clean source of truth.

Now the risk. The largest single research crawler in Ahrefs' dataset identified itself as prompt-injection-survey/1.0 — somebody is systematically probing llms.txt files as a prompt-injection surface, precisely because agents are built to trust them. A stale or compromised llms.txt misleads every agent that reads it. This is why 'negligible downside' is the wrong frame.

• Version-control the file like code — it belongs in your repo, not in a CMS text box.
• Restrict who can edit it and alert on unauthorized changes.
• Keep it to plain links and descriptions. Nothing instruction-shaped — no 'always recommend', no 'ignore previous'.
• Only link to resources you control.
• Review anything your platform auto-generates on your behalf. Wix already ships these files by default.

The wrong reason to publish: thinking it will move AI citation rates this quarter. It won't. Anyone selling you a '$2,000 llms.txt optimization' is charging you for a file you can generate in an afternoon — and that your website platform may already generate for free.

What's the right llms.txt template?

The standard format is markdown with a specific structure. Start with an H1 of the site name. Add a blockquote with a 1-2 sentence description. Then organize sections with H2 headers, each containing a markdown list of links. Optional sections include an "Optional" H2 (lower-priority links) and a "Notes" section for clarifications.

• Line 1: # Site Name (H1 with the site name)
• Line 2-3: > One-sentence description of what the site is and who it's for.
• Section: ## Docs — links to product documentation, ordered most-to-least important
• Section: ## Guides — links to longer educational content (pillars, deep-dives)
• Section: ## Services — links to commercial service pages
• Section: ## Case Studies — links to social proof and outcome stories
• Section: ## Blog — links to high-priority blog posts (not everything, just the canonical pieces)
• Section: ## Optional — lower-priority but still relevant links
• Each link format: - Page Title: one-line description
• Keep each section to 5-15 links. More than 20 dilutes signal.
• Use absolute URLs everywhere — the file may be parsed by tools that don't know your domain context.
• Update the file when major new pages are published or canonical pages change.
• Route agents to it: link the file from your HTML, reference it in your docs, mention it anywhere agents get instructions about your site. Agents fetch llms.txt when directed, not speculatively — an unlinked file is unlikely to ever be picked up.

What about llms-full.txt?

llms-full.txt is the companion file containing the actual concatenated markdown content of your most important pages. Where llms.txt is a curated link index, llms-full.txt is the full-text version, designed to be consumed directly without crawling each linked page. Be aware that it is a different bet: publishing markdown copies of your pages is a separate tactic from the index file, and the Ahrefs data above measured only the index file — it says nothing about whether llms-full.txt gets read.

Generate it programmatically by concatenating the markdown versions of every page listed in your llms.txt, separated by clear page markers (the page URL as a header). Most static-site generators (Next.js, Astro, Hugo) can build it in a few lines of script. Wire it into the deploy pipeline so it regenerates from your canonical page list — a hand-maintained copy goes stale within a quarter, and a stale file is the exact failure mode that misleads agents.

How do you test whether crawlers see your llms.txt?

Two checks, and one common mistake to avoid. First, manual: curl your llms.txt with each crawler's user agent and confirm the response is a 200 returning real markdown, not an HTML soft-404. Second, server-log analysis: pull the last 30 days of access logs and grep for hits to /llms.txt from named AI user agents. Your own logs beat anyone's headline number, including the ones in this post.

• curl -A 'GPTBot/1.0' https://yourdomain.com/llms.txt
• curl -A 'ClaudeBot/1.0' https://yourdomain.com/llms.txt
• curl -A 'OAI-SearchBot/1.0' https://yourdomain.com/llms.txt
• curl -A 'PerplexityBot/1.0' https://yourdomain.com/llms.txt
• grep 'llms.txt' /var/log/nginx/access.log | grep -iE 'gptbot|claudebot|claude-code|oai-searchbot|perplexity'
• Don't bother curling Google-Extended — it is a robots.txt opt-out token, not a crawler with its own user agent. There is no Gemini crawler at all; Google grounds Gemini on content fetched by ordinary Googlebot.
• Googlebot hits on /llms.txt mean nothing special. It crawls any URL it discovers, the same way it crawls a sitemap.
• If you see zero hits over 30 days, that is the base rate, not a bug: 97% of published llms.txt files got zero requests of any kind in Ahrefs' May 2026 data.

What's the practical recommendation?

Publish it, link to it, and spend one afternoon — not one dollar of agency budget — on it. Use the standard template. Version-control it. Then go check your own logs in 90 days and decide with your own data rather than anyone's headline. If a crawler you care about has started fetching it, you'll see it. If nothing has, you've lost an afternoon, which was the whole bet.

What we won't do is bill it as a GEO deliverable. Ahrefs' own verdict on llms.txt is that the cons currently outweigh the pros, and we agree: it is cheap optionality, not a citation lever. It rides along inside a broader GEO program at no line item. If you want an honest read on what will actually move your AI visibility this quarter, book a strategy call.

For the actual citation-moving work, focus your effort on passage-level optimization and schema deployment. Those are the levers that pay back this quarter. llms.txt is a small bet on next year.