What Is llms.txt? A Practical Guide with Real Examples

You've met robots.txt — the file that tells search crawlers where they may and may not go. And sitemap.xml — the machine-readable list of every page on your site. Now there's a third file joining them: llms.txt.

It's a simple Markdown file that lives at the root of your site (yoursite.com/llms.txt) and tells AI tools — ChatGPT, Claude, Perplexity and the crawlers that feed them — what your site is about, which pages matter most, and what each one does. Think of it as the human-readable-but-machine-friendly table of contents for your website.

We run one ourselves at asadwebs.com/llms.txt. This guide explains the format with real excerpts from that file, the rules we follow when curating it, and the honest limits nobody should sell you past.

What llms.txt is — and what it isn't

Let's clear up the biggest misconception first, because it's the one most articles skip:

Google does not read llms.txt. It has no effect on your Google rankings, and it won't get you cited in Google's AI Overviews. The pages that AI search engines quote are the ones with clear structure, real guides and good schema — that's a content job, not a file job.

So why bother? Because the other AI engines — the ones behind ChatGPT, Claude and Perplexity — fetch web pages directly, and raw HTML is a noisy soup of navigation, footers, ads and scripts. A clean Markdown summary gives them your site's essentials in one read: who you are, what your best pages are, and what each page actually does. It's an emerging convention, not a web standard, and no honest person can promise you "AI visibility" from it. But as a curated front door for AI readers, it's cheap to make and easy to maintain — which is why we have one.

The anatomy of a good llms.txt

The format is plain Markdown. Here's how ours starts:

# AsadWebs

> AsadWebs (asadwebs.com) is a web design and development agency building
> websites, online stores and custom web applications. The site also offers
> 190 free online tools that run entirely in the visitor's browser - no
> sign-up, files are never uploaded - including PDF and image tools, media
> converters, text and SEO utilities, developer tools, unit converters and
> calculators. Every tool page carries a written guide with steps, a worked
> example and a FAQ.

## Key pages

- [AsadWebs - Home](https://asadwebs.com/): Web development, eCommerce and
  custom web applications.
- [Free Online Tools](https://asadwebs.com/tools): All 190 free
  browser-based tools: PDF, image, video, text, SEO and developer tools,
  converters and calculators.

Four parts, each with a job:

1. An H1 with the site name. One line, no marketing fluff.
2. A summary in a blockquote. Two to four sentences: who you are, what the site offers, and the one or two facts an AI should never get wrong about you (for us: the tools are free, run in the browser, and never upload your files).
3. Key pages first. The handful of pages that define the site — home, main hub, blog, shop, services.
4. Sections of links with one-line descriptions. Grouped by topic, each link followed by a short, factual description of what the page does:

## Tools - SEO & Web

- [Sitemap Generator](https://asadwebs.com/tools/sitemap-generator): Crawl
  your site and build a valid XML sitemap (up to 500 pages)
- [Robots.txt Generator](https://asadwebs.com/tools/robots-txt-generator):
  Build a robots.txt visually — rules per crawler, no syntax to memorise
- [LLMs.txt Generator](https://asadwebs.com/tools/llms-txt-generator):
  Build a curated llms.txt from your sitemap — best pages, checked and grouped

Notice the descriptions say what the visitor gets, not keyword-stuffed slogans. An AI quoting your page will lean on that sentence — make it the truest one-liner you can write.

The 5 curation rules we follow

This is where most llms.txt files go wrong. The format is trivial; the curation is the work. These are the rules baked into our own file and into the generator we built for it:

1. Curate, don't dump. An llms.txt works best as a short list of your best pages, not a copy of your whole sitemap. If you select more than about 50 pages, you're not curating anymore — you're duplicating the sitemap in a worse format. Keep the essentials; move the merely-useful to an ## Optional section at the end (or leave them out entirely).

2. Skip pages that shouldn't be listed. Before a URL earns a place, it should pass four checks: not marked noindex, not blocked in your robots.txt, not a redirect, and not an error page (404/500). Every URL in your sitemap is not automatically trustworthy — sitemaps go stale, and listing a dead page in your AI front door is worse than listing nothing.

3. Group by section, in your order. Sections (## Tools - PDF Tools, ## Tools - Finance) let an AI reader jump to the topic it needs. Keep the order deliberate — most important sections first — and use the same section names everywhere on the page.

4. Descriptions say what the page does. One line, factual, in the visitor's language. "Compress a PDF towards under 100 KB — measured ladder, honest result if unreachable" beats "Best free online PDF compressor tool 2026!!!" — and it's the sentence an AI will repeat about you.

5. Keep it fresh. Regenerate or review the file whenever you ship new pages. A front door that lists last year's tools is a small lie told at scale. We regenerate ours after every tool wave — it takes minutes with the right tool.

The optional second file: llms-full.txt

The convention allows a companion file, llms-full.txt, that holds the actual text of your selected pages converted to Markdown. It's heavier — it has to fetch every page and convert its HTML — but it's the difference between a table of contents and the book itself.

Use it when your pages contain genuinely useful reference content (guides, documentation, worked examples). Skip it for thin pages; a full file of empty pages teaches an AI nothing. Our generator builds it as an optional second export: it converts real pages to genuine Markdown and plainly reports any page it couldn't fetch, rather than inventing content.

Make yours in minutes

The manual way: create a Markdown file following the anatomy above, check each URL against the four skip-rules, and upload it as /llms.txt at your domain root. Then request indexing of a couple of your key pages so crawlers pick the file up in their normal passes.

The faster way: use our free LLMs.txt Generator. It loads your sitemap, warns you when you've selected more than ~50 pages, lets you drag sections into order and bulk-assign them, automatically flags noindex pages, robots-blocked URLs, redirects and error pages, imports an existing llms.txt so you're never starting over, autosaves your work in the browser, and exports both llms.txt and the optional llms-full.txt. Everything runs in your browser — your sitemap never leaves your machine.

Frequently Asked Questions

Does llms.txt help my Google rankings?

No. Google has said nothing about reading it, and there is no evidence it influences rankings or AI Overviews. Don't let anyone sell you an "llms.txt SEO package."

Then why does it exist?

For the AI systems that do fetch it directly — ChatGPT, Claude, Perplexity and their crawlers. Clean Markdown is dramatically easier for a language model to use accurately than a page full of navigation chrome.

Is it an official standard?

No — it's a proposed convention. That's fine; robots.txt started the same way. The cost of adopting it is one small text file.

How is it different from sitemap.xml?

A sitemap is exhaustive and machine-formatted for crawlers; llms.txt is curated and human-readable, meant to convey meaning, not just URLs. They complement each other — our generator reads your sitemap as the starting point, then you curate down from there.

Should every page be listed?

No — that's the most common mistake. Aim for your best pages, grouped by topic. The generator warns you past ~50 selections precisely so the file stays a curated front door rather than a second sitemap in a worse format.

Do I need llms-full.txt too?

Only if your pages hold real reference content worth quoting. The short file is the must-have; the full file is the nice-to-have.

We keep our own file honest and current — read it, steal the structure, and generate yours in a few minutes.