llms.txt
Last updated:
What llms.txt is
The llms.txt format was proposed by Jeremy Howard on September 3, 2024. It is a proposal, not an official standard. The idea is simple: instead of making an AI system dig through navigation menus, cookie banners and scripts, a site offers one text file in Markdown with the essentials and pointers to more detail.
The proposal explains the motivation: context windows are too small to hold most websites in full, and HTML pages wrap their information in navigation, ads and JavaScript that are hard to convert into clean text. The file is meant mainly for inference, when an AI assistant needs information while helping a user, rather than for training models. The proposal also suggests offering clean Markdown versions of important pages at the same URL with .md appended.
How it works
The file sits at the root of a site, as /llms.txt, although the current version of the proposal also allows placing it under a sub-path to cover the pages below it. Its sections come in a fixed order:
- An H1 heading with the name of the site or project. This is the only required part.
- A blockquote with a short summary that contains the key information needed to understand the rest of the file.
- Optional sections with more detail, such as paragraphs or lists.
- Sections under H2 headings with “file lists”: each item is a Markdown link with a name and a URL, optionally followed by a colon and a short note.
- An optional section named “Optional” for secondary links an AI agent can skip when it needs a shorter context.
Example
A small bakery could publish a file like this: an H1 with “Rosa's Bakery”, a blockquote describing it as a neighborhood family bakery that bakes sourdough bread and custom cakes to order, and a section “Key pages” with links to the menu, opening hours, cake orders and the delivery area, each with a one-line note. A section “Optional” could link to the bakery's history and its press page. When someone asks an AI assistant whether the bakery makes gluten-free cakes, an assistant that reads the file can go straight to the right page instead of guessing from fragments.
llms.txt, robots.txt and sitemap.xml
The three files are often mentioned together, but they do different jobs. robots.txt sets access rules, while llms.txt is read on demand, when an assistant needs information about a topic.
| File | What it does | Format |
|---|---|---|
| robots.txt | Tells automated crawlers which parts of a site they may access | Plain-text rules |
| sitemap.xml | Lists the pages of a site so search engines can find and index them | XML |
| llms.txt | Gives AI assistants a concise summary and links to the most useful content | Markdown |
Why it matters for businesses and chatbots
Many people now ask AI assistants about products and services. An llms.txt file is a low-cost way to give those assistants a correct, current summary of what you offer, including what you do not offer, which helps prevent wrong answers about your business. Publishing one does not guarantee that any AI system reads it or cites your site, so treat it as a complement to clear pages, not a replacement.
The same discipline helps a website chatbot. An AI chatbot that answers from your own content gives better answers when that content is clean, factual and well structured. A precise llms.txt and a precise knowledge base come from the same work: writing down, in plain language, what is true about your business.
llms.txt at intoCHAT
intoCHAT publishes its own llms.txt at https://www.intochat.ai/llms.txt. It summarizes what intoCHAT does, lists what it does not do, shows the plans with their prices and limits, and includes the one-line installation script. It also points AI assistants to a longer fact sheet at https://www.intochat.ai/en/llm-info. intoCHAT does not create llms.txt files for your website, but if you write one, you can upload the same text as a TXT or Markdown file to train your agent.