Is llms.txt worth it if almost nobody reads it?
Fetch studies say most llms.txt files sit untouched. Here is when agencies should still ship one, when to skip it, and what to measure instead of GEO theatre.

In 2026 the industry finally measured something practitioners suspected: most llms.txt files never get fetched. Ahrefs’ study of domains running its Web Analytics reported that about 97% of published files received zero traffic in the sample window, and that AI search bots barely show up among the few requests that do arrive. Otterly’s GEO experiment and other crawls of large domain sets point the same way, and Google Search has said it does not treat llms.txt as a robots-style control for ranking. So the honest agency question is no longer “how do we write the perfect file?” It is whether the file is worth the upkeep when almost nobody reads it.
We already separated three artefacts in robots.txt, llms.txt, and cats.txt are not the same job: crawl policy, optional agent map, and the joke that proves a 200 on a .txt is not a protocol. What follows is the decision layer on top of that taxonomy. For crawl speed and fetchability habits that still move risk, keep Why AI crawlers need fast, crawlable pages open (Apogee Watcher). For measurement without buying another GEO dashboard, use Monitor AI search optimisation without a subscription.
What “almost nobody reads llms.txt” actually means
“Unread” in these studies means no HTTP fetches of /llms.txt in the observation window, not “AI models refuse to cite you.” That distinction matters in client meetings. A zero-fetch file failed as a distribution channel; it did not prove your product pages are invisible in ChatGPT answers, and it did not prove Lighthouse’s experimental Agentic Browsing check is wrong to mention the file.
When fetches do happen, researchers keep finding a skewed audience: coding agents, researchers, and tools that study llms.txt itself, not a reliable pipeline of AI Overview crawlers hunting for your brand summary. Ahrefs also reported that bots do not go looking for a missing file, so publishing one does not create demand. At best it serves readers who were already pointed at your docs.
The base rate for a random marketing site is harsh. Most files sit untouched, and treating that silence as “we’re GEO-ready” is the expensive mistake.
What recent llms.txt studies measured (and what they did not)
Point at the artefacts, not the vibes:
- Ahrefs (137k sites with analytics): high share of domains that publish a file still see zero fetches; among the tiny share that are hit, coding agents outrank AI search bots; Lighthouse’s audit accounts for a tiny fraction of fetches.
- Otterly and similar GEO tests:
/llms.txtreceives a tiny slice of AI bot traffic relative to ordinary content URLs; presence does not reliably lift overall crawler interest. - Large-domain adoption crawls (for example Invisit on Tranco): a minority of sites serve a real file; major operators still do not document third-party
llms.txtas a Search or assistant ranking lever; Google Search ignores it for ranking controls.
None of that means the proposal is useless for your own agents. Separate research on docs sites shows token savings when an agent is instructed to start at /llms.txt. The lever is the prompt you control, not speculative discovery by someone else’s crawler.
When llms.txt is still worth shipping
Use this as a portfolio rule, not a slogan.
| Site type | Verdict | Why |
|---|---|---|
| API, SDK, or developer docs you own | Ship and maintain | Coding agents and internal assistants are the readers that show up; a curated Markdown map can cut token waste when the prompt says to open it first |
| CMS that auto-generates the file | Keep, then QA | Cheap to leave on; expensive if it lists retired URLs and wrong prices |
| Product marketing / SaaS homepage story | Optional, one afternoon max | Fine if you can point at pricing and docs mirrors; skip if nobody will update it |
| News, large blog, or huge catalogue | Skip for now | Curating thousands of changing URLs is real labour for near-zero fetch rates |
| “We need it for Google / AI Overviews” | Skip | Search does not use it as robots-for-AI; invest in crawlable HTML and field Core Web Vitals instead |
You block the AI bots in robots.txt that would need the map |
Skip or fix the conflict | A guide for agents you refuse at the gate has no compliant readers |
Worth it, then, when someone you control will fetch it, or when generation is free and the file stays accurate. Not worth it as a retainer line item sold as AI-search visibility. The table is the whole argument. The rest is how you defend the skip without sounding lazy.
Minimum maintainable llms.txt (or a clean skip)
If you ship one:
- Keep it short: H1, a few Markdown links to canonical docs, pricing, and contact; no dump of every blog URL.
- Link formats agents and Lighthouse audits expect (
[label](url)), not bare URLs alone, if you care about experimental Agentic Browsing checks. - Put an owner and a refresh trigger (release notes, pricing change). A stale file is worse than no file because agents trust confident wrong answers.
- Log fetches for a month. If nothing hits
/llms.txtexcept your own checkers, stop debating prose and either wire it into an agent prompt you own or delete the ticket.
If you skip, say so in the ticket with the study base rate and the Google Search stance. That is a stronger professional answer than a decorative file, and clients usually accept a sourced skip faster than a file nobody can prove is read.
What to measure instead of llms.txt for AI visibility
Agencies still need an AI-adjacent story clients will fund. Prefer work that shows up in logs and field data: crawlability and performance for the bots you allow (see the Watcher AI crawlers piece); scheduled lab and CrUX coverage on the URLs that earn revenue, not a homepage-only paste; and answer or citation checks you can repeat without a GEO SaaS tax (Watcher’s monitor-without-subscription angle). llms.txt can sit beside that stack as an optional docs affordance, but it should not be the centrepiece of the retainer narrative.
FAQ
Does Google Search use llms.txt?
Google has stated Search does not treat it as a ranking or robots-style control. Do not sell it as a Search Console lever.
Should we add llms.txt because Lighthouse Agentic Browsing mentions it?
Treat that as an experimental readiness check, not proof of traffic. Fix or ship the file if you already want agent docs; do not reorder the whole sprint for a fractional lab audit.
Is a zero-fetch file a security problem?
The bigger practical risk is a compromised or stale file that agents trust. If you publish, keep ownership and review. If you cannot, omit it.
Put the llms.txt ship-or-skip verdict in your runbook
Agency technical leads: before the next “add llms.txt for AI visibility” ticket, pick a row from the table above and write the verdict into the runbook. That is the same upkeep question from the opening, answered with a sourced ship or skip instead of another unchecked box. Use Why AI crawlers need fast, crawlable pages for fetchability work that still compounds, and Monitor AI search optimisation without a subscription when the client wants measurement without another dashboard. Leaving an unread file as the plan is how retainers fund silence. If you want scheduled PageSpeed and Core Web Vitals history across the portfolio while that debate settles, start a Watcher trial.
References
- We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read (Ahrefs)
- llms.txt and AI Visibility: Results from OtterlyAI's GEO Study (OtterlyAI)
- Do websites actually use llms.txt? We checked 20,000 (Invisit)
- llms.txt saves tokens, if you tell the agent to read it (a14y research)
- robots.txt, llms.txt, and cats.txt are not the same job (Apogee Watcher Hashnode)
- Why AI crawlers need fast, crawlable pages (Apogee Watcher)
- Monitor AI search optimisation without a subscription (Apogee Watcher)
About the author
- Apogee Information Systems operates at the intersection of digital transformation for international organisations, innovative e-learning, public service automation, and specialised commercial solutions. Currently building Apogee Watcher, a web performance tool for agencies and solopreneurs.
Comments
Join the discussionNo comments yet. Be the first to comment.
Industry insights & trends
Industry and trends angles for agencies: Chrome Stable, CWV updates, AI-shaped stacks, and what changes in client reporting.





