llms.txt is a proposed Markdown file at a website's root that gives AI agents a short summary of the site and links to its key pages. The proposal dates from September 2024, is not an official web standard, and requires only an H1 title.

Most business websites do not need the file to appear in Google Search or AI answers. Page passages, not site files, usually explain why a page ranks in Google and does not get cited by AI.

Google says publishing the file neither helps nor harms visibility in Google Search, because Google Search ignores it. Google needs no special markup for AI features either, and how schema markup affects AI citations depends on the passages beside the markup. Lighthouse checks for the file, while crawler logs show AI crawlers rarely fetch it.

AI answers quote passages from retrieved pages, which is how AI search engines choose the pages they cite. A business file should name the business, define it in one sentence and link the pages an AI agent needs. Shopify already serves a default file for every store, and a SaaS file should link pricing, features and documentation.

Publishing the file takes six steps, and crawler access still depends on robots.txt, which the AI Visibility Checker for crawler access reads. Pages with clear, verifiable passages matter more for AI answers, which is what AI citation readiness means. Crawler access, entity signals and citable passages make up AI Search Visibility for ChatGPT and AI Overviews.

What is llms.txt?

llms.txt is a proposed Markdown file at a website's root that gives AI agents a short summary of the site and links to its key pages. The address is /llms.txt, and the intended readers are large language models and the AI agents built on them.

An AI agent reads the file on demand, when the agent needs facts about a site while helping a user.

Many owners call the Markdown file "robots.txt for AI", but it works as a reading list that grants or blocks nothing. The proposal names software documentation as the heaviest use, because coding agents follow the links to API references.

Searchers also type llm.txt or llms txt, and all three spellings ask about one Markdown file. The proposal also fixes the order of the parts inside the reading list.

What does the file look like?

The Markdown file for AI agents requires only an H1 with the site name, then takes an optional summary, details and link lists.

The five parts, in order, are listed below:

  1. H1 title: the name of the site or project, the only required part.
  2. Blockquote summary: one or two sentences an AI agent reads before any link.
  3. Details: optional paragraphs or lists without headings that explain how to read the file.
  4. H2 link lists: sections of Markdown links in the form [name](url): note.
  5. Optional section: an H2 named "Optional" with links an agent can skip when short on context.

The minimal file below uses four of the five parts, in order.

# Sample Site
> One-sentence summary of what the site offers and for whom.

## Main pages
- [Services](https://example.com/services): what the business sells
- [Pricing](https://example.com/pricing): plans and prices

## Optional
- [Blog](https://example.com/blog): articles and news

Square brackets hold the link text, not a heading, and each note after a colon stays on one line. The format is fixed by a proposal, not by a standards body.

Is the file an official standard?

The Markdown reading list for AI agents is not an official web standard, because the proposal has no standards body behind it. Jeremy Howard of Answer.AI proposed the format in September 2024. The current version, v2, lets a file sit at any path, such as a /docs/ folder, and cover only the pages under that path.

The proposal reports that thousands of sites publish the file and that documentation platforms generate one automatically. OpenAI, Anthropic and Google's Gemini team publish their own files for their developer docs, which differs from their crawlers reading other sites' files. Publishing a file and needing one are separate questions.

Do you need an llms.txt file?

Most business websites do not need an llms.txt file, because Google Search ignores it and no major AI search platform has confirmed reading it. A short file still costs little, and documentation and API sites gain the most from one. Our verdict below depends on who would read the file on each type of site.

The table below gives our verdict for five types of website:

Site type Who reads the file there What the file should list Our verdict
Documentation or API site Coding agents and AI assistants that fetch docs API reference, quick start, guides Publish one
B2B SaaS website AI agents browsing for a user; coding agents for the docs Product, pricing, integrations, docs Publish a short file
Shopify store AI shopping agents Store summary, collections, policies Review the default Shopify serves
Local service business Few AI systems today Services, service area, contact Optional; check crawler access first
Blog or content site Few AI systems today Best guides by topic Optional

We publish a file on every site we build, because the cost is one short text file.

A business missing from AI answers should check its key pages before adding the file. The reason why a page ranks in Google and does not get cited by AI sits in its passages.

The seven causes are listed in why a page ranks in Google and does not get cited by AI. Google's own wording settles the first half of the verdict.

Does Google use llms.txt?

Google Search does not use the file, and Google says publishing one neither helps nor harms a site's visibility in Google Search. Google's guide to generative AI features on Google Search, last updated 10 July 2026, states the position directly:

"You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them."

The same guide says Google may crawl many file types without treating any of them in a special way. Publishing the file for AI agents therefore costs a site nothing in Google Search, according to Google. Those generative AI capabilities include AI Overviews and Google AI Mode for business websites.

Google makes the same point about markup and states that no special schema.org structured data is needed for AI features. Citations are a separate question, and how schema markup affects AI citations depends on the passages beside the markup.

The evidence on how schema markup affects AI citations covers Google, ChatGPT and Perplexity. A Google tool still checks for the file, which confuses many owners.

Why does Lighthouse check for the file?

Lighthouse checks for the file in its agentic browsing audits, and the audit fails only when the server returns an error. Lighthouse is the page audit tool built into Chrome, and its results do not feed Google Search rankings. Agentic browsing audits test how easily AI agents can read and use a site.

The two possible results of the audit in a Lighthouse report are listed below:

  • Not applicable: the site returns a 404 for the address, because the file is "optional at the moment", per Lighthouse documentation updated 5 May 2026.
  • Flagged: the server returns an error when Lighthouse requests the address.
sample Lighthouse agentic browsing result for a real estate website with no llms.txt file
Lighthouse marks a missing file as not applicable and flags only a server error.

The sample real estate site has no file, so the audit reads "Not applicable" and the score stays unchanged.

Some SEO audit tools list the absence as an issue, and a missing file is a tool warning, not a ranking penalty. AI assistants decide for themselves whether to fetch the reading list.

Do ChatGPT, Claude and Perplexity read the file?

No major AI search platform has confirmed that its crawlers read the file, and crawler logs show it is rarely fetched. Ahrefs and Semrush both reported in 2026 that no AI platform had confirmed reading the Markdown file.

EZY Research placed the file on 83 sites from 27 April to 19 July 2026 and logged every AI crawler request.

Fetches of the reading list against robots.txt fetches in that 12-week test are listed below:

  • GPTBot (OpenAI): 7 fetches against 3,990 robots.txt fetches.
  • ClaudeBot (Anthropic): 9 fetches against 3,120 robots.txt fetches.
  • PerplexityBot: 0 fetches against 775 robots.txt fetches.

Ahrefs found that about 97 percent of roughly 38,000 sites with a valid file got zero requests for it in May 2026. A site owner can repeat the check in server or CDN logs, filtered for the reading list's address and grouped by user agent.

AI answers quote passages from retrieved pages, which is how AI search engines choose the pages they cite. Retrieval, passage extraction and attribution are covered in how AI search engines choose the pages they cite. A business that still wants a file for AI agents should keep the reading list short and accurate.

What should a business website's llms.txt say?

A business website's llms.txt should name the business, define the business in one sentence and link the pages an AI agent needs to answer questions.

The six sections of a business file are listed below:

  1. Business name: the H1, written exactly as on the About page.
  2. One-sentence definition: a blockquote that states what the business is, for whom and where.
  3. Services or products: links to each service page or product collection, with a short note.
  4. Pricing, policies and contact: pricing, shipping, returns, booking and contact pages.
  5. Key guides: the five to ten guides that answer the questions buyers ask most.
  6. Optional: the blog, news and other pages an AI agent can skip.

The definition in the file should match the About page word for word, so every source describes the business the same way.

Digital Vikingz publishes its own file at its site root, with six H2 sections: Services, Free tools, Industries, Key guides, Proof and Optional. Every link in its Services and Free tools sections carries a one-line note. Stores and SaaS companies need different key pages in the same six sections.

What should a Shopify store's file say?

Shopify serves a default file for every store, written as instructions for AI shopping agents rather than as a store summary. Shopify's default mirrors the store's /agents.md file and covers the Shop skill, the Universal Commerce Protocol and store policies. Store owners can view that output by opening the file's address on their own domain.

Shopify lets stores customize the Markdown file with theme templates, a change Shopify announced on 28 May 2026. A custom llms.txt.liquid or agents.md.liquid template replaces the Shopify default for that address. A store that adds a summary should keep the shopping instructions and add the store definition, top collections and shipping and returns pages.

sample outdoor gear store on Shopify with its default agent instructions file beside a version that adds collections and policies
A Shopify store serves agent instructions by default and can add a store summary through theme templates.

The sample store keeps the default shopping instructions and adds four collections and two policy pages above the Optional section. How AI shopping agents pick products is covered in how ChatGPT Shopping recommends Shopify products. A SaaS company starts from an empty file instead.

What should a SaaS company's file say?

A SaaS company's file should define the product in one sentence and link pricing, features, integrations and the documentation. Documentation can carry a second file in the /docs/ folder, which covers only the pages under that path. Coding agents follow a docs file to API references, the use the proposal names first.

sample llms.txt for a B2B accounting software company with pricing, integrations and a separate docs file
A SaaS file links pricing and integrations and points coding agents to a separate docs file.

The sample SaaS file links pricing, two integration pages and the docs file, and moves the blog and changelog to Optional.

Comparison pages and customer stories belong in the file when buyers ask an AI assistant to compare vendors. The product definition should match the homepage and the About page, so AI agents meet one description of the product. Writing and publishing the reading list takes six steps on any platform.

How do you create and add an llms.txt file?

Creating a Markdown file for AI agents takes six steps, from writing the links to confirming that the live address returns text.

The six steps, from draft to live address, are listed below:

  1. Write the file in Markdown in any text editor.
  2. Start with an H1 business name and a one-sentence blockquote summary.
  3. Add H2 sections of links with absolute URLs and a one-line note each.
  4. Upload the file to the site root, the folder that holds robots.txt.
  5. Open the address and confirm a 200 status, a text/plain type and UTF-8 encoding.
  6. Update the file whenever a key page is added, moved or removed.

Shopify stores already serve a default and edit the output through theme templates under Online Store, Themes, Edit code. Webflow has an upload field for the file in its SEO site settings.

Yoast SEO for WordPress has a setting that creates the Markdown file and updates it weekly. Next.js sites serve the reading list from the public folder. A file that passes these checks still controls no crawler, which robots.txt does.

How is llms.txt different from robots.txt and sitemap.xml?

llms.txt points AI agents to key pages, robots.txt sets crawler access, and sitemap.xml lists URLs for search engines to index. robots.txt tells crawlers which URLs they may fetch, while the reading list is read on demand by an agent helping a user.

sitemap.xml lists the URLs a site wants search engines to crawl and index. robots.txt follows the Robots Exclusion Protocol, and sitemap.xml follows the sitemaps protocol. A Sitemap: line in robots.txt tells crawlers where sitemap.xml lives.

AI crawlers request robots.txt far more often than the reading list, as the EZY Research fetch counts above show. A site can publish all three files at once, because each file answers a different reader and none replaces another.

The table below compares the three files on five attributes:

File Main job Format Main readers Controls access
llms.txt Points AI agents to key pages Markdown AI agents and coding tools No
robots.txt Sets which crawlers may fetch which URLs Directive lines Search and AI crawlers Yes, for crawlers that obey
sitemap.xml Lists URLs for indexing XML Search engine crawlers No
sample dental clinic robots.txt rules beside the same clinic's llms.txt reading list
robots.txt decides which crawlers may fetch pages, while the reading list only names pages to read.

The sample clinic blocks one private folder in robots.txt and lists its treatment and booking pages in its reading list.

A robots.txt rule that blocks GPTBot or ClaudeBot stops those crawlers from fetching the blocked pages, whatever the reading list says. The AI Visibility Checker for crawler access reads robots.txt the way AI crawlers do.

Run the free AI Visibility Checker for crawler access before writing the file. The reading list also has a longer companion file.

What is llms-full.txt?

llms-full.txt is a companion file that holds the full text of a site's key pages in one Markdown file, instead of links. Documentation platforms generate llms-full.txt so a coding agent can load a whole manual in one request.

The companion file sits at the address /llms-full.txt, beside the reading list. Shopify serves an llms-full.txt for stores as well, and stores can customize it with the same theme templates.

The companion file suits manuals and API references, where AI agents need every page at once. A business website rarely needs llms-full.txt, because the key pages already exist as HTML. Neither file keeps content away from AI crawlers.

Does llms.txt block AI crawlers or AI training?

A reading list for AI agents does not block AI crawlers or AI training, because the list holds no access rules. Crawler and training controls sit in robots.txt user-agent rules, such as rules for GPTBot, ClaudeBot and Google-Extended. robots.txt rules work only for crawlers that choose to follow them.

Google's crawler documentation, last updated 30 October 2026, says Google-Extended manages Gemini training and grounding use, not inclusion or ranking in Google Search. Pages listed for AI agents stay open to every crawler that robots.txt allows.

Adding Disallow lines to the file does nothing, because no crawler treats the reading list as a rule file. The proposal describes its contents as information used on demand, not as a rule set. Crawler access and clear pages matter more than any file for AI answers.

Other questions about the file

Business owners ask six further questions about the file, on SEO impact, spelling, sitemaps, generators, robots.txt and updates.

Does a missing file hurt SEO?

Having no Markdown file for AI agents does not hurt SEO, because Google Search ignores such files and Lighthouse marks the gap as not applicable. Some SEO audit tools still list the gap as a warning, and a warning in a third-party tool is not a ranking signal.

Is llm.txt the same as llms.txt?

llm.txt is a common misspelling, and AI agents look only for the name with an s. A file saved as llm.txt sits at an address no agent checks, so add the s before uploading the file to the site root.

Should the file go in the sitemap?

A reading list for AI agents does not go in the sitemap, because the sitemap lists pages for search engines to index. AI agents find the file at the site root by convention, and the sitemap keeps listing the key pages themselves.

Should you use a generator for the file?

A generator is useful for a first draft, and a person should cut the draft to the key pages. Generators often copy the whole sitemap, and a validator checks only the format, not the choice of pages an AI agent needs.

Does robots.txt need a line pointing to the file?

robots.txt needs no line for the file, because the proposal places it at a fixed address in the site root. A sitemap line in robots.txt tells search engines where sitemap.xml lives, while AI agents request the reading list directly at the site root.

How often should you update the file?

A reading list for AI agents should be updated whenever a key page is added, moved or removed. Yoast SEO for WordPress refreshes its generated file once a week, and a hand-written list needs a check after every change to the site's pages.

What matters more than llms.txt for AI answers?

Pages that answer questions in clear, verifiable passages matter more for AI answers than any site file. AI systems cite passages from pages, so each answer should sit in one self-contained passage near the top of a section.

AI citation readiness is the state of a page whose sentences can each be verified against one passage and matched to what other trusted sources say. The page-level standard is set out in what AI citation readiness means.

Digital Vikingz builds crawler access, entity signals and citable passages in AI Search Visibility for ChatGPT and AI Overviews. Publish a short file for AI agents if the cost suits you, and spend the larger effort on the key pages AI answers quote.