How to Create an llms.txt File for Your Website: A Practical Guide With a Free Generator

Direct answer: You can create an llms.txt file for your website with the llms.txt Generator tool: give it your sitemap, pick the pages that matter, and it builds a plain Markdown file with your site name as the heading, an optional summary, and grouped link lists, plus an optional llms-full.txt containing real page text. The file sits at your site root, ready for any system that reads it.

The honest framing comes first, because this topic attracts hype. The llms.txt format is a community proposal from 2024, published at llmstxt.org, and it is not an official standard. No standards body has adopted it, and there is no guarantee about which systems read it, how they use it, or whether they use it at all. Google has said that llms.txt is not used for Search or for AI Overviews. Anyone promising visibility, rankings, or traffic from publishing an llms.txt file is selling something the evidence does not support.

So why bother? Because the file is cheap to make, easy to maintain, and genuinely useful as a concise, human-readable map of a site. It gives any reader — human or machine — a short summary of what the site is and a curated list of its most important pages, in a format that is trivial to parse. Think of it as a well-written contents page for your site rather than a lever for traffic, and the file earns its place on those modest terms.

What an llms.txt file actually contains

The format is plain Markdown, which is part of its appeal: anyone can read it, edit it in a text editor, and understand it without special software. The only required part is an H1 heading with the site name. Everything else is optional, but the useful parts follow a clear pattern: an optional blockquote summary describing the site, optional notes with extra context, and then Markdown link lists grouped under section headings, each link pointing to a page with a short description of what that page covers.

The section grouping is what makes the file more useful than a raw sitemap dump. Instead of listing every page in crawl order, a good llms.txt groups pages by purpose — documentation, guides, tools, reference, support — so a reader can find the right corner of the site quickly. Within each section, links carry brief descriptions, because a bare URL tells a reader where a page is but not whether it is the page they need. The descriptions are the editorial work, and they are what separate a helpful file from a mechanical one.

One structural rule matters: the Optional section, when used, comes last. It holds links to pages that are useful but not essential — supplementary material, archives, older versions — and placing it last keeps the essential sections prominent. A file that buries its core pages behind optional material has its priorities backwards, and a reader skimming from the top will miss what matters.

Loading your sitemap: indexes, compressed files, and child sitemaps

The generator starts from your sitemap rather than asking you to type URLs by hand, which avoids typos and guarantees the links match pages that actually exist. It accepts a sitemap index as well as a single sitemap, and it handles compressed .xml.gz sitemaps, so sites whose sitemaps are generated in compressed form do not need to decompress anything first. Up to 25 child sitemaps are supported from an index, which covers most small and medium sites comfortably.

Starting from the sitemap also gives the tool something to check. It flags duplicate URLs, so a page listed twice — once with and once without a trailing slash, for example — shows up before it becomes a duplicate link in your file. It checks pages for errors, redirects, noindex directives, and robots blocks, so a page that should not be promoted — a redirect, a page you asked search engines not to index, a page blocked by robots rules — is flagged rather than silently included. Those checks turn a mechanical listing into an editorial review.

The practical workflow is to load the sitemap, review the flagged items, and then select pages deliberately. A site with hundreds of pages should not list them all: the file is a curated guide, not a second sitemap. Pick the pages a newcomer most needs — the homepage, the main guides, the key tools, the central reference pages — and leave the long tail out. The tool warns when selections grow past about 50 pages, which is a sensible nudge that the file is becoming a catalogue rather than a guide.

Real titles, real descriptions, and the 30-page fetch

For the first 30 selected pages, the tool fetches the real title and description from each page, so the link lists carry accurate descriptions instead of guesses. That matters because invented descriptions are worse than none: a description that promises content a page does not contain misleads every reader who follows it. Fetching real metadata keeps the file grounded in what the pages actually say, and it saves the manual work of opening thirty pages and copying their titles one by one.

Beyond those 30 pages, descriptions need your own input, which is another reason to keep selections focused. A file whose first thirty links carry real titles and whose remaining links carry thoughtful hand-written descriptions is a good file. A file whose links carry no descriptions at all is still valid, but it asks more of its readers, who must click through to learn what each page covers. Spend description effort on the pages that matter most, and let the rest be plain links.

The same honesty rule applies to the summary and notes. Describe the site as it is, not as you wish it were. A summary that claims coverage the site does not have will be discovered quickly by anyone who follows the links, and a file that overpromises undermines trust in the site it was meant to present. Plain, accurate language serves this file better than marketing language.

The optional llms-full.txt: real page text, failures reported

Alongside the main file, the tool can build an llms-full.txt, which contains the actual text of selected pages rather than just links to them. It covers the first 25 selected pages, pulling their real content into a single document. For a reader that wants substance rather than pointers — a researcher, a support agent, a system assembling context — the full file is the more useful of the two, because it answers questions directly instead of pointing at pages that might answer them.

The important guarantee is about failures: when a page's text cannot be fetched, the failure is reported, never papered over with invented content. A full file that silently invented a page's text would be actively harmful, presenting fabricated content as if it were the site's own words. Reporting the failure keeps the file honest and tells you which pages need attention — a page that cannot be fetched may be blocking automated readers, or may simply have moved, and either fact is worth knowing.

Not every site needs an llms-full.txt. For a small site, the main file plus its linked pages may be enough, and maintaining a full-text copy adds a second file to keep in sync. For a documentation site or a knowledge base, where the content itself is the product, the full file is the more valuable artefact. Decide based on whether your readers need pointers or substance, and remember that either file is only as current as its last regeneration: update both when the site's core pages change.

Autosave, honest limits, and a sane publishing routine

Work in the tool autosaves in your browser, so a half-built file survives a closed tab or an interrupted session. That is a small feature with a real payoff: curating a good file takes thought, and losing twenty minutes of selections to a browser crash is the kind of frustration that makes people abandon the task. Autosave removes that risk, and you can return to the file, refine the selections, and finish it in a second sitting without starting over.

The honest limits deserve repeating as a closing note, because they shape expectations. The format is a proposal, not a standard. Google does not use it for Search or AI Overviews. No visibility, ranking, or traffic outcome should be promised from it, and no one should pay for a service whose pitch depends on such a promise. What the file offers is a clean, maintained, human-readable map of your site — useful to readers, cheap to produce, and worth having on those terms alone.

A sane routine is to generate the file, place it at the site root, and revisit it when the site's core pages change — a new main guide, a retired tool, a restructured section. A quarterly glance is plenty for most sites. If your site's structure is documented elsewhere, the Schema Markup Generator describes individual pages for search engines, and the Readability Checker keeps the pages themselves easy to read. More site guides live on the Asad Webs blog.

Frequently Asked Questions