# Do AI engines like ChatGPT actually read llms.txt?

> llms.txt is a proposal for a Markdown file placed at a website’s root or another published path.

Source: https://bestaibottrackers.com/does-llms-txt-work · Publisher: Bot Trackers · Updated: 2026-11-13

## TL;DR
- `llms.txt` is a proposal for a Markdown overview of a website, not a crawler directive like `robots.txt`.[3]
- Google Search says it does not use `llms.txt` and that the file will neither help nor hurt Search visibility or rankings.[1]
- Server logs show occasional fetches by some agents, but retrieval does not prove parsing, indexing, citation or influence on generated answers.[7]
- OpenAI and Anthropic publish `llms.txt` for their own documentation, while third-party crawler support remains unpublished in the official guidance reviewed.[7]
- The defensible 2026 verdict is that `llms.txt` is an optional experiment, not a proven AI-visibility or SEO ranking signal.[7]

## Do AI engines actually read llms.txt?

**Sometimes, but not consistently—and a fetch does not prove that an AI engine uses the file.** The strongest current conclusion is that `llms.txt` is an optional documentation convention with uneven crawler adoption, not an established AI-search ranking or visibility signal. Google Search explicitly says it does not use `llms.txt` for Search or its generative AI features.[1]

### What is llms.txt?

`llms.txt` is a proposal for a Markdown file placed at a website’s root or another published path. It is intended to give language-model systems a concise overview of a site, with links to more detailed pages that an agent can inspect when needed.[2][3]

The proposal was authored by Jeremy Howard and introduced on September 3, 2024. Its current version was modified on August 10, 2026.[4][5] The format is not a crawler-control mechanism like `robots.txt`: it does not instruct a bot whether it may crawl a site, and it does not guarantee that a crawler will request, parse or use the file.[6][7]

A typical file might contain:

- A short description of the business or project
- The site’s main documentation or product areas
- Links to authoritative pages
- Optional context intended to help an AI system navigate the site

The proposal’s adoption page says that thousands of sites publish `llms.txt` and that some documentation platforms generate it.[3] That indicates growing availability, but it does not measure how often AI systems retrieve or use the files.

### What does Google say?

Google Search’s official guidance is unusually clear: websites do not need new machine-readable files, AI text files or Markdown files to appear in Google Search or its generative AI features.[1] Google says that Google Search itself does not use `llms.txt`, and that publishing one will neither help nor hurt Search visibility or rankings.[1]

Google may still discover, crawl or index many file types, but ordinary file retrieval does not mean that Google treats a file as a special input.[7] Therefore, a Googlebot request for `/llms.txt`, if one occurs, would not contradict Google’s published position that Google Search ignores the file as a special optimization signal.[7]

This distinction matters for the query **“llms txt SEO.”** There is no evidence in Google’s guidance that adding the file improves organic rankings, eligibility for AI Overviews or visibility in Google AI Mode.[8][7]

### Do ChatGPT and Claude crawlers read llms.txt?

OpenAI documents OAI-SearchBot as the crawler used to help surface websites in ChatGPT search features.[2] OpenAI also publishes an `llms.txt` file for its own developer documentation.[8][7] Neither fact establishes that OAI-SearchBot automatically discovers or parses third-party websites’ `llms.txt` files.[7]

Anthropic documents ClaudeBot, Claude-User and Claude-SearchBot, along with ways site owners can manage crawler access through `robots.txt`.[6] Anthropic also serves an `llms.txt` file for its own developer documentation.[9] The crawler guidance reviewed does not publish an explicit commitment that Anthropic’s systems consume third-party `llms.txt` files.[7][3]

A vendor publishing `llms.txt` for its own documentation is therefore weak evidence of general support. It shows that the vendor considers the format useful for at least one property; it does not show that the vendor’s production crawlers use the convention across the web.

### Which AI crawlers fetch llms.txt, according to log studies?

The available evidence shows occasional retrieval, with results varying substantially by crawler, site and discovery method.

| Study or source | Observed evidence | What it proves | What it does not prove |
|---|---|---|---|
| Google Search Central | Google says Google Search does not use `llms.txt` and that the file does not help or harm Search visibility.[1] | Google’s published position is an explicit “no” for Google Search. | It does not establish how every other Google product behaves. |
| Ahrefs analysis | Ahrefs reports that 28% of its analyzed domains published `llms.txt`; its traffic analysis found no requests to 97% of the measured files.[7] | Most files in that measured set received no observed AI request during the study. | It does not prove that an unobserved file was never accessed by another path or that a fetched file was not used. |
| Dejan.ai single-site logs | The report counted 1,150 requests for `/robots.txt` and four for `/llms.txt`; three were attributed to Googlebot, one to ClaudeBot and none to OpenAI.[5] | A few crawlers requested the file on one site. | It does not show that any crawler parsed, indexed or acted on the contents. |
| Nathan Hall preprint | Across six WordPress sites and 165 site-days, a Markdown resource at `/ai/llms.txt` received 1,549 requests after the URL was announced through `robots.txt` and page HTML; an unannounced root `/llms.txt` received two requests across seven sites.[4] | Explicitly announcing a resource can increase retrieval of that URL. | It does not establish recognition of the `llms.txt` convention or influence on generated answers. |
| OtterlyAI experiment | One website recorded more than 62,100 AI-bot visits and 84 requests to `/llms.txt`, approximately 0.1% of AI-bot visits.[10] | The file was fetched occasionally in that experiment. | It does not show improved citations, rankings or answer visibility. |
| Plan Conversion logs | In an 11-day log study, `llms.txt` was fetched on nine of 11 days, mostly by Exa, with occasional requests attributed to Anthropic, Amazon and ByteDance; OpenAI, Perplexity, Google and Bing did not fetch it in that log.[5] | Adoption differs sharply among crawlers, even on a site receiving substantial crawler traffic. | A limited observation window cannot establish permanent crawler behavior. |
| Common Crawl experiment | Common Crawl added `/llms.txt` and `/llms-full.txt` to the seed list for a large random sample of crawlable hosts in its July 2026 crawl.[7] | A crawler can test the convention deliberately at large scale. | Seed-list requests do not prove that downstream AI systems use the files. |

The studies are not directly interchangeable. Some measure one website, some measure selected domains, some announce the file through another mechanism and some deliberately request the path. Their percentages and request counts should not be combined into a single adoption rate.[5]

The consistent limitation is that server logs record HTTP retrieval. They do not reveal whether a system:

1. Parsed the Markdown
2. Stored the contents
3. Indexed the linked pages
4. Used the information during answer generation
5. Cited the site more often
6. Changed a ranking or recommendation

A request is therefore evidence of **fetching**, not evidence of **semantic use**.[7]

### Does llms.txt improve brand visibility in AI search answers?

There is no robust evidence that publishing `llms.txt` alone improves AI visibility. Google explicitly says it does not use the file, while third-party studies show sparse or discovery-dependent retrieval by other crawlers.[7]

The strongest positive interpretation is narrower: `llms.txt` can sometimes function as a useful, easily discovered summary when an agent or crawler is already looking for it. That may help a system navigate a site in specific circumstances, but current evidence does not show a reliable causal path from publishing the file to more mentions, citations, referrals or rankings.

The strongest negative interpretation is also too broad. A zero-request observation does not prove that no system ever saw the file. A system could have accessed it outside the measurement period, through a cached copy, through a user-directed fetch or after discovering a linked URL through another page.[6]

The practical answer to **“does llms.txt work?”** is therefore:

> It may be fetched by some agents, but broad automatic consumption and measurable AI-search benefits remain unproven.

### Why discovery matters more than the filename

The available log evidence suggests that discovery mechanics can matter more than the existence of the conventional root filename. In the Zenodo preprint, a non-root Markdown resource received many more requests after its URL was announced through `robots.txt` and page-level HTML, while an unannounced root file received almost none.[4]

That result should not be interpreted as a recommendation to rename the file or as proof that `robots.txt` formally supports `llms.txt`. It shows only that a crawler is more likely to retrieve a resource when the resource is exposed through a path it already follows.[7]

The same principle applies to ordinary AI visibility work: accessible, well-linked, crawlable and authoritative HTML remains more dependable than assuming an AI system will discover a special file.

### How should you implement llms.txt?

If the file is inexpensive to maintain and accurately summarizes the site, publishing it can be a reasonable experiment. It should be treated as supplementary documentation rather than a replacement for technical SEO, structured content or crawl controls.

1. **Create a concise Markdown file.** Place it at `/llms.txt` if using the conventional proposal path, and describe the organization, product or site in plain language.[3]
2. **Link to authoritative pages.** Include documentation, product, policy, support and other pages that genuinely support the summary.[7]
3. **Keep claims current.** Remove discontinued products, obsolete pricing and outdated descriptions.
4. **Use canonical page URLs.** Avoid sending agents to duplicate, parameterized or blocked versions of the same content.
5. **Make the file publicly retrievable.** Test it with an unauthenticated request and confirm that the server returns the intended Markdown.
6. **Do not use it as an access-control file.** Manage crawler permissions with `robots.txt`, authentication, server controls and applicable vendor tools; `llms.txt` is not a substitute.[10][7]
7. **Log requests separately.** Record user agent, timestamp, status code and requested path so that fetches can be distinguished from ordinary page traffic.
8. **Track outcomes, not just requests.** Compare citations, mentions, referrals and answer content over time, while controlling for other changes.
9. **Avoid unsupported success claims.** A request proves retrieval, not parsing, citation or ranking impact.[11][4]
10. **Prioritize the underlying pages.** Ensure important information is crawlable, internally linked, clear, current and supported by evidence.

A lightweight test can compare a period before and after publication, but it should not be treated as a controlled experiment unless crawler access, content changes, public announcements and other visibility factors are controlled. Even then, a change in citations cannot automatically be attributed to `llms.txt`.


What should a basic llms.txt file contain?

A practical file can include a short site description followed by links to the most authoritative pages. Keep it readable, avoid promotional filler and make sure every linked page is public, current and relevant. The proposal describes the file as an overview with links to more detailed, LLM-friendly resources.[3]



### What should brands track to measure AI search visibility instead?

For Google visibility, follow Google’s published guidance rather than treating `llms.txt` as an SEO shortcut.[7] The more durable work is to publish useful content, make important pages accessible to crawlers, explain entities and products clearly, maintain internal links and earn references from credible sources.

For AI-answer visibility, measure observable outcomes:

- Brand mentions across representative prompts
- Citation frequency and cited URLs
- The position of the brand within generated answers
- Referral sessions from AI services
- Conversions or assisted revenue from AI referrals
- Accuracy of product, pricing and policy information
- Changes after substantial content or technical updates

These metrics still require careful sampling because AI answers vary by prompt, location, account, model version and retrieval context. Even surfaces from the same company differ: [Finseo's own analysis of Google AI Mode and AI Overviews](https://www.finseo.ai/blog/ai-mode-vs-ai-overviews) found the two share only 13.7% of cited URLs on the same query, so each surface needs to be tracked separately. A single answer or isolated bot request cannot establish a trend.

### What is the evidence-based verdict?

`llms.txt` is neither a proven breakthrough nor an entirely imaginary mechanism. Some automated agents fetch it, especially when the resource is explicitly announced or otherwise discoverable. Other logs show no requests from major crawlers during the observation period, while Google explicitly says that Google Search ignores the file.[8][7]

The file can be maintained as a low-cost experiment or documentation aid, particularly for sites whose audiences include developers and AI agents. It should not receive priority over crawlable HTML, useful information, accurate structured data, internal linking, technical accessibility and independent authority.

For buyer-focused AI visibility programs, the correct classification is **experimental and optional**. Treat a fetch as a technical event to investigate—not as proof that an AI engine read, trusted, cited or ranked the site.

## FAQ
### Do AI engines actually read llms.txt?
Some bots have fetched `llms.txt`, but the evidence does not establish broad or consistent use.[7] A fetch shows that a URL was requested; it does not show that the contents were parsed, indexed, cited or used in an answer.[7]

### Does llms.txt improve Google rankings?
No evidence in Google’s official guidance supports that claim. Google says Google Search does not use `llms.txt` and that publishing one will neither help nor hurt Search visibility or rankings.[1]

### Does llms.txt help Google AI Overviews or AI Mode?
Google says that new machine-readable files and AI text files are not needed for its generative AI features.[1] The guidance therefore does not support treating `llms.txt` as a way to improve eligibility or visibility in Google AI Overviews or AI Mode.

### Do ChatGPT and OpenAI read llms.txt?
OpenAI documents OAI-SearchBot and publishes an `llms.txt` file for its own developer documentation.[7] The reviewed crawler documentation does not publish an explicit commitment that OpenAI automatically reads third-party sites’ `llms.txt` files.[7]

### Does Anthropic or Claude read llms.txt?
Anthropic documents several crawlers and publishes an `llms.txt` file for its own documentation.[5][7] That is not proof that Claude’s crawlers consume third-party files; the reviewed official guidance does not make that commitment.[7]

### Should every website publish llms.txt?
Not necessarily. Publishing a concise, accurate file can be a reasonable low-cost experiment, but it should not replace crawlable pages, internal linking, technical SEO or evidence-based content.[7] If maintaining the file would create stale or contradictory information, the effort may be better spent improving the pages that users and crawlers already need.

### Does llms.txt help brand visibility in Perplexity?
In an 11 day log study, `llms.txt` was fetched mostly by Exa. OpenAI, Perplexity, Google and Bing did not fetch it in that log.[5] There is no robust evidence that publishing `llms.txt` alone improves AI visibility.[7]

### Which AI crawlers have been observed fetching llms.txt?
One single site report counted four requests for `/llms.txt`: three attributed to Googlebot, one to ClaudeBot and none to OpenAI.[5] Another log study saw occasional requests attributed to Anthropic, Amazon and ByteDance, with most fetches coming from Exa.[5][7] These results vary by crawler, site and discovery method, so they should not be combined into a single adoption rate.[4]

### Can llms.txt increase brand mentions or citations in AI answers?
Current evidence does not show a reliable causal path from publishing the file to more mentions, citations, referrals or rankings. In the OtterlyAI experiment, `/llms.txt` received 84 requests out of more than 62,100 AI bot visits, approximately 0.1%, and the experiment did not show improved citations, rankings or answer visibility.[10]

## Sources
[1] Google's Guide to Optimizing for Generative AI Features on Google Search | Google Search Central  |  Documentation  |  Google for Developers — https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
[2] Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots
[3] The /llms.txt file, v2 – llms-txt — https://llmstxt.org/
[4] Discovery, not adoption: crawler retrieval of an llms.txt resource announced through robots.txt, across seven sites on one server, and what a server log can and cannot show | Zenodo — https://zenodo.org/records/22814844
[5] Google, Anthropic and OpenAI crawlers fetched dejan.ai's robots.txt 1,150 times and its llms.txt 4 times in 30 days — https://dejan.ai/blog/crawler-census-robots-llms-okf/
[6] Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
[7] We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read — https://ahrefs.com/blog/llmstxt-study
[8] Does llms.txt matter? We tracked 10 sites to find out — https://searchengineland.com/does-llms-txt-matter-467740
[9] docs.anthropic.com — https://docs.anthropic.com/llms.txt
[10] llms.txt and AI Visibility: Results from OtterlyAI's GEO Study — https://otterly.ai/blog/the-llms-txt-experiment
[11] LLMs.txt Does Not Boost AI Citations, New Analysis Finds — https://www.searchenginejournal.com/llms-txt-shows-no-clear-effect-on-ai-citations-based-on-300k-domains/561542/
