llms.txt: What It Is, What It Does and Whether Your Business Needs One

By the FlyRank team, Manchester · Published 15 October 2026

TL;DR: llms.txt is a plain markdown file at the root of your website that lists your most useful pages for AI systems. It is a community proposal, not a formal standard, and no major search engine has confirmed that it improves rankings or AI citations. It costs very little to publish, so treat it as a low-risk tidy-up and never as a ranking lever. Crawl access, clear content and strong structure matter far more.

What is llms.txt?

llms.txt is a markdown file, served at /llms.txt, that gives AI systems a short, curated map of a website: what it is, and which pages are worth reading first.

The idea was proposed by Jeremy Howard in September 2024 and is documented at llmstxt.org. The proposal exists because language models read pages in limited chunks, and a full HTML page carries navigation, scripts and layout that get in the way of the content.

The file is written for machines but stays human-readable. It sits beside robots.txt and sitemap.xml in the site root, and it describes the site in a few lines rather than listing every URL.

The llmstxt.org page describes the project as still open to community input. It is an informal overview of a proposal, not a ratified standard from a body such as the W3C or the IETF.

Does llms.txt help you appear in AI answers?

There is no confirmed evidence that it does. Adoption exists, but no major AI search product has said that it uses llms.txt to decide what to cite.

Google is direct about its own products. Its documentation on AI features in Search says there are no additional requirements to appear in AI Overviews or AI Mode, and that site owners do not need new machine-readable files or AI text files. A page needs to be indexed and eligible to show with a snippet, and that is the whole test.

OpenAI documents its crawlers separately. Its bots documentation explains that OAI-SearchBot is the crawler behind ChatGPT search results and that robots.txt controls it. It does not mention llms.txt as an input.

The honest position is that llms.txt is an unproven hedge. Some tools, particularly in software documentation and developer workflows, do read the file when a person points them at it. That use is real, but it is different from a search engine using it to rank or cite you.

Key takeaway: publish llms.txt only if you accept that it may do nothing. Do not promise a client, or yourself, that it will lift visibility.

What goes in an llms.txt file?

A site name as an H1, a one-line summary in a blockquote, then H2 sections of markdown links with a short note on each page.

Per llmstxt.org, the H1 is the only required element. Everything else is optional, but the summary and the link lists are what make the file useful.

Choose pages that are indexable, canonical and live. Do not list redirects, thin pages, duplicates or anything private. Fifty links is a sensible ceiling for a small business site, and most need far fewer.

Write the notes as plain facts. A line such as “Services: SEO, answer engine optimisation and web design for businesses in Greater Manchester” tells a reader what the page covers without any marketing language.

How do you create and publish one?

Draft the file in a text editor, save it as llms.txt in UTF-8, upload it to the web root, and check that it returns a 200 status as plain text.

1. Pick the pages

   List your core service pages, your best guides and your contact page. Include only URLs that are indexable and return a 200 status.

2. Write the file

   Follow the structure in the table above. Keep it short, factual and consistent with the wording on your site and your Google Business Profile.

3. Upload it

   Place it at https://yourdomain.co.uk/llms.txt, in the same folder as robots.txt. In WordPress this usually means SFTP or a file-manager plugin, as the file is not a normal page.

4. Test it

   Load the URL in a browser, confirm it shows as plain text, and click every link. Re-test after any CMS, CDN or caching change.

5. Maintain it

   Update the file whenever you add or remove a core page. Add it to your monthly technical check.

Be careful with online generators. Some publish your output on a public link, so do not paste anything confidential into them. Writing the file by hand takes about ten minutes for most sites.

Do not invent robots.txt directives to point at the file. Lines such as “LLM: /llms.txt” are not a recognised part of the robots exclusion standard and should not replace real crawl rules.

How is llms.txt different from robots.txt and sitemap.xml?

robots.txt controls crawler access, sitemap.xml lists URLs for search engines, and llms.txt offers a curated reading list for language models. Only the first two are established standards that search engines document.

The practical consequence is that robots.txt is where AI visibility is actually decided. If OAI-SearchBot or Googlebot is blocked, a flawless llms.txt will not help.

OpenAI also lets you treat its crawlers separately. You can allow OAI-SearchBot for search visibility and disallow GPTBot, which relates to model training, so review these rules before assuming anything is open or closed.

Should a UK business bother?

Yes, if you have a spare half hour after the fundamentals are done. No, if publishing it would delay fixing crawl access, thin content or unclear service pages.

Order of work matters. First confirm that key pages are indexable and linked, that robots.txt does not block the crawlers you want, and that each page answers its main question near the top. Those steps have documented effects. llms.txt does not.

It makes most sense for sites with a lot of documentation, product information or help content, where a tidy index genuinely helps a person or tool that reads it. For a ten-page local services site, the gain is small and the cost is smaller.

If you want an independent view of what AI systems currently say about your business, our answer engine optimisation services start with an audit of crawl access, content and brand facts. To see where llms.txt sits among the wider tactics, read what answer engine optimisation is and how AEO differs from SEO. For the schema side, see schema markup for AI search.

Key takeaway: llms.txt is optional housekeeping. Do the proven work first, and measure results with a tracked prompt set, as described in measuring AI search visibility.