LLM-LD SOP

Appearance

LLMs.txt 1.0 Specification Fresh ​

Status: Draft | Published: February 2026 | License: CC BY 4.0 Companion to: LLM-LD 1.0 Specification, ADP 1.0 Specification

Abstract ​

LLMs.txt is a specification for a plain-text Markdown file that provides AI systems with a concise, human- and machine-readable orientation to a website. It serves as a lightweight manifest that identifies the site, declares its canonical authority, enumerates its machine-readable resources, summarizes its products and services, and provides contact information -- all in a single file that can be consumed in a fraction of a context window.

This specification defines:

  1. A file format (llms.txt) using Markdown with a predictable section structure
  2. Required and optional sections with defined semantics
  3. A metadata header block for machine-parseable fields
  4. Placement and discovery rules consistent with the LLM-LD ecosystem
  5. The relationship between llms.txt and companion structured-data resources

Table of Contents ​

  1. Introduction
  2. Terminology
  3. Relationship to Existing Standards
  4. File Placement and Discovery
  5. Document Structure
  6. Header Block
  7. Required Sections
  8. Recommended Sections
  9. Optional Sections
  10. Formatting Rules
  11. Size and Performance Considerations
  12. Security Considerations
  13. IANA Considerations
  14. Examples
  15. Changelog

1. Introduction ​

1.1 Background ​

AI systems consume web content in fundamentally different ways than human readers or traditional search crawlers. Large language models operating at inference time face strict context-window constraints that make crawling an entire website impractical. Retrieval-augmented generation (RAG) pipelines benefit from a single, authoritative summary that can orient the model before it decides which deeper resources to fetch. Autonomous agents need a quick overview of available actions, resources, and contact channels before they begin navigating.

In September 2024, Jeremy Howard proposed a convention of placing a Markdown file at /llms.txt to help LLMs navigate website content. Since then, the convention has been widely adopted: Anthropic, Perplexity, Zapier, and hundreds of other organizations publish llms.txt files.

This specification formalizes the emerging practice for business websites -- defining a predictable section structure, a metadata header block, and a defined vocabulary of sections.

1.2 Design Principles ​

  1. Plain text first. The file MUST be readable by any system that can fetch a text file.
  2. Orientation, not duplication. The file orients an AI system -- telling it what the site is, where the deeper resources live, and what actions are available.
  3. Predictable structure. Every conforming file uses the same section ordering with the same heading names.
  4. Minimal by default. A conforming file can be very short.
  5. Compatible with the LLM-LD ecosystem. References llm-index.json, the AI Discovery Page, entities.json, and knowledge.graph.jsonld.

2. Terminology ​

llms.txt -- The filename as specified by this standard.

Header Block -- The key-value metadata section immediately after the H1 heading.

Section -- A Markdown H2 heading plus its content, separated from other sections by horizontal rules (---).

Canonical Site -- The primary, authoritative URL for the organization.

3. Relationship to Existing Standards ​

3.1 llmstxt.org Convention ​

This specification extends the llmstxt.org convention (Jeremy Howard, September 2024) to the business use case. A file conforming to this specification is backwards compatible with the llmstxt.org convention.

3.2 robots.txt ​

llms.txt does not replace or conflict with robots.txt. They serve different purposes:

  • robots.txt: access control (what can crawlers fetch?)
  • llms.txt: orientation (what is this site, and what should you fetch first?)

3.3 llm-index.json ​

llms.txt is a plain-text summary. llm-index.json is the full structured data. llms.txt SHOULD point to llm-index.json in its START HERE section.

3.4 AI Discovery Page ​

The ADP is an HTML document for both human and machine audiences. llms.txt is plain-text only. Both serve as discovery entry points -- llms.txt is optimized for inference-time retrieval by AI systems; the ADP is optimized for crawlers that parse HTML.

4. File Placement and Discovery ​

4.1 Canonical Location ​

https://ai.{domain}/llms.txt

The file lives on the AI subdomain, not the main site. The main site's robots.txt blocks AI crawlers (except for /ai-discovery), so a llms.txt there would be invisible to its intended audience.

4.2 Alternative Locations ​

If no AI subdomain exists:

https://www.{domain}/llms.txt

4.3 robots.txt Pointer (Optional) ​

Add a pointer in the AI subdomain's robots.txt:

LLMs-Txt: https://ai.yoursite.com/llms.txt

4.4 ADP Link Relation ​

Add a <link> element to the AI Discovery Page:

html
<link rel="ai-manifest" type="text/plain"
      href="https://ai.yoursite.com/llms.txt" />

5. Document Structure ​

A conforming llms.txt has the following structure:

markdown
# {Site Name} AI Optimization Layer
{Header Block}

---

## PURPOSE
{Content}

---

## CANONICAL AUTHORITY
{Content}

---

## START HERE (CRAWLING GUIDANCE)
{Content}

---

[RECOMMENDED AND OPTIONAL SECTIONS]

---

## CONTACT
{Content}

---
EOF

6. Header Block ​

Immediately after the H1 heading, include metadata as key-value pairs on separate lines.

6.1 Required Fields ​

FieldDescriptionExample
VersionSpecification version1.0
Last-UpdatedISO 8601 date2026-02-07
Canonical-SitePrimary site URLhttps://www.yoursite.com

6.2 Recommended Fields ​

FieldDescriptionExample
Primary-LanguageBCP 47 language codeen-US
AI-MirrorAI subdomain URLhttps://ai.yoursite.com

6.3 Optional Fields ​

FieldDescriptionExample
GeneratorTool that created this fileaso-generator/2.0
Conformance-LevelLLM-LD conformance level3
markdown
# Acme Corp AI Optimization Layer
Version: 1.0
Last-Updated: 2026-02-07
Primary-Language: en-US
Canonical-Site: https://www.acmecorp.com
AI-Mirror: https://ai.acmecorp.com

7. Required Sections ​

7.1 PURPOSE ​

Declares what this file is for and which AI use cases it supports.

Standard use case list for business sites:

markdown
## PURPOSE
This is an AI-readable representation of {Business Name} content optimized for:
- LLM crawlers
- AI search systems
- Agentic retrieval workflows
- Entity-based knowledge extraction

Optional additions:

  • - AI shopping assistants (for e-commerce)
  • - AI appointment booking (if supported)

7.2 CANONICAL AUTHORITY ​

Names the one URL that is the authoritative source of truth.

markdown
## CANONICAL AUTHORITY
The canonical source of truth is always:
https://www.yoursite.com

Critical for preventing AI from trusting scraped copies, cached versions, or third-party reproductions.

7.3 START HERE (CRAWLING GUIDANCE) ​

The most important section. Points AI to structured data files in priority order.

markdown
## START HERE (CRAWLING GUIDANCE)

### 1. Full Site Intelligence (Recommended)
https://ai.yoursite.com/llm-index.json

### 2. Entity Index
https://ai.yoursite.com/entities.json

### 3. Knowledge Graph
https://ai.yoursite.com/knowledge.graph.jsonld

### 4. Sitemap
https://ai.yoursite.com/sitemap.xml

Minimum Requirement

At minimum, include llm-index.json. If you only list one resource, make it this one.

7.4 CONTACT ​

Organization name and at least one contact method.

markdown
## CONTACT
Acme Corp
info@acmecorp.com
https://www.acmecorp.com

8. Recommended Sections ​

8.1 PRODUCTS & SERVICES ​

markdown
## PRODUCTS & SERVICES
- SPYBOX: https://ai.acmecorp.com/products/spybox
- Statilitix: https://ai.acmecorp.com/products/statilitix

For large catalogs, list top items and add a fallback:

markdown
See entities.json for full catalog (240+ products).

8.2 ENTITY STATS ​

Quick quantitative summary.

markdown
## ENTITY STATS
- Total Entities: 47
- Products/Software: 3
- Services: 5
- Pages: 22

9. Optional Sections ​

9.1 ABOUT ​

2-4 sentence narrative about the organization.

9.2 KEY FACTS ​

Bullet list of important differentiators. Critical for local businesses.

markdown
## KEY FACTS
- Serving Tampa since 2006
- 4.8 star rating (127 reviews)
- Same-day emergencies available
- Most insurance accepted

9.3 ACTIONS ​

Links to key conversion actions.

markdown
## ACTIONS
- Start Free Trial: https://www.acmecorp.com/signup
- Book a Demo: https://www.acmecorp.com/demo
- Contact Sales: https://www.acmecorp.com/contact

9.4 FAQ ​

3-5 questions maximum. Short, factual answers.

markdown
## FAQ

### Do you accept insurance?
We accept most major dental insurance including Delta, Cigna, MetLife, and Aetna.

### What are your hours?
Monday-Friday 8am-5pm, Saturday 9am-2pm.

9.5 VERIFICATION ​

markdown
## VERIFICATION
Verified by: LLM Disco Directory
Conformance: Level 3 (Agent-Ready)

10. Formatting Rules ​

  • UTF-8 encoding, no BOM
  • Unix line endings (\n). No carriage returns.
  • Sections separated by --- (horizontal rules). One before each section, one after the last.
  • Sections in order: PURPOSE > CANONICAL AUTHORITY > START HERE > (optional sections) > CONTACT
  • End with EOF -- lets AI systems confirm they received the complete file
  • No HTML -- Plain Markdown only. HTML defeats the purpose.
  • Under 4 KB ideally. Hard limit is 20 KB.
markdown
...last section content...

---
EOF

11. Size and Performance Considerations ​

Target size: Under 4 KB.

Hard limit: 20 KB. If you're over, trim PRODUCTS & SERVICES and use the catalog fallback pattern.

HTTP headers:

Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=86400
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff

The Access-Control-Allow-Origin: * header lets AI agents fetch the file from any origin. Without it, browser-based AI tools may not be able to read it.

12. Security Considerations ​

No Private Data -- Do not include authentication tokens, API keys, internal pricing, or PII.

Content Injection -- This file may be consumed directly by language models. Avoid content that could cause prompt injection.

13. IANA Considerations ​

This specification requests registration of the ai-manifest link relation type.

Relation Name: ai-manifest

Description: Identifies a resource containing a plain-text AI manifest conforming to the LLMs.txt specification.

14. Examples ​

See the llms.txt Guide for complete worked examples including:

  • Minimal 4-section example
  • Complete SaaS company example
  • Complete local business (dental) example

15. Changelog ​

v1.0 (February 2026)

  • Initial draft specification
  • Formalizes organic llmstxt.org usage for business websites
  • Defines metadata header block
  • Defines 8 section types: PURPOSE, CANONICAL AUTHORITY, START HERE, ABOUT, PRODUCTS & SERVICES, ENTITY STATS, KEY FACTS, ACTIONS, FAQ, VERIFICATION, CONTACT
  • Integration with LLM-LD ecosystem (llm-index.json, ADP, entities.json)
  • LLM Disco Directory integration

llms.txt Guide | LLM-LD 1.0 Spec | ADP 1.0 Spec

LLM-LD is licensed under CC BY 4.0. Created by CAPXEL.