Configuring llms.txt Fresh

The plain-text manifest that tells AI systems who you are, what you offer, and where to find your structured data. One file, instant orientation.

What Is llms.txt?

llms.txt is a Markdown file you place at the root of your domain that gives AI systems a concise summary of your website. Think of it as a business card and a map combined -- it tells an AI crawler who you are, what your business does, and where to find the machine-readable resources it needs.

robots.txt tells crawlers where they can't go. sitemap.xml tells them what pages exist. llms.txt tells them who you are and where to start.

The llms.txt format has been adopted by thousands of businesses. But everyone's been doing it differently. The LLM-LD specification for llms.txt formalizes the business use case -- defining a predictable section structure, a metadata header block, and a defined vocabulary of sections.

Where It Fits in the Ecosystem

File	Format	Purpose	Typical Size
ai-discovery	HTML	Bridge page with links to AI resources	5-15 KB
llms.txt	Markdown	Quick orientation and resource pointers	1-4 KB
llm-index.json	JSON-LD	Complete structured data about your site	5-100 KB

An AI system with a small context window might only fetch llms.txt. A full RAG pipeline will use it to discover llm-index.json and go deeper.

---

File Structure at a Glance

Every llms.txt following this specification has the same predictable structure:

1. Required H1 Heading + Header Block -- Site name, version, last-updated date, and canonical URL
2. Required PURPOSE -- Declares what this file is for and which AI use cases it supports
3. Required CANONICAL AUTHORITY -- Names the one URL that is the authoritative source of truth
4. Required START HERE (CRAWLING GUIDANCE) -- Points AI to your structured data files, ordered by priority
5. Recommended PRODUCTS & SERVICES -- Lists your offerings with links
6. Recommended ENTITY STATS -- Quick quantitative summary
7. Optional ABOUT, KEY FACTS, VERIFICATION, ACTIONS, FAQ
8. Required CONTACT -- Organization name and at least one contact method

---

Minimal Example

The simplest valid llms.txt file -- just the four required sections:

# Acme Dental AI Optimization Layer
Version: 1.0
Last-Updated: 2026-02-07
Canonical-Site: https://www.acmedental.com

---

## PURPOSE
This is an AI-readable representation of Acme Dental content optimized for:
- LLM crawlers
- AI search systems
- Agentic retrieval workflows
- Entity-based knowledge extraction

---

## CANONICAL AUTHORITY
The canonical source of truth is always:
https://www.acmedental.com

---

## START HERE (CRAWLING GUIDANCE)

### 1. Full Site Intelligence (Recommended)
https://ai.acmedental.com/llm-index.json

---

## CONTACT
Acme Dental
https://www.acmedental.com

---
EOF

That's a valid file. Under 500 bytes. Any AI system that fetches this instantly knows: the business name, the canonical URL, and where to find the full structured data.

---

Complete Example -- SaaS Company

# Acme Corp AI Optimization Layer
Version: 1.0
Last-Updated: 2026-02-07
Primary-Language: en-US
Canonical-Site: https://www.acmecorp.com
AI-Mirror: https://ai.acmecorp.com

---

## PURPOSE
This is an AI-readable representation of Acme Corp content optimized for:
- LLM crawlers
- AI search systems
- Agentic retrieval workflows
- Entity-based knowledge extraction

---

## CANONICAL AUTHORITY
The canonical source of truth is always:
https://www.acmecorp.com

---

## START HERE (CRAWLING GUIDANCE)

### 1. Full Site Intelligence (Recommended)
https://ai.acmecorp.com/llm-index.json

### 2. Entity Index
https://ai.acmecorp.com/entities.json

### 3. Knowledge Graph
https://ai.acmecorp.com/knowledge.graph.jsonld

### 4. Sitemap
https://ai.acmecorp.com/sitemap.xml

---

## ABOUT
Acme Corp is a marketing technology company founded in 2020,
headquartered in Miami, Florida. The company builds AI-powered
tools for competitive intelligence, web analytics, and conversion
optimization, serving over 2,500 business customers globally.

---

## PRODUCTS & SERVICES
- SPYBOX: https://ai.acmecorp.com/products/spybox
- Statilitix: https://ai.acmecorp.com/products/statilitix
- SEO Consulting: https://ai.acmecorp.com/services/seo-consulting

---

## ENTITY STATS
- Total Entities: 47
- Products/Software: 3
- Services: 5
- Pages: 22

---

## KEY FACTS
- Founded in 2020
- Headquartered in Miami, FL
- 3 SaaS products
- 2,500+ customers
- SOC 2 Type II certified

---

## ACTIONS
- Start Free Trial: https://www.acmecorp.com/signup
- Book a Demo: https://www.acmecorp.com/demo
- Contact Sales: https://www.acmecorp.com/contact

---

## VERIFICATION
Verified by: LLM Disco Directory
Conformance: Level 3 (Agent-Ready)

---

## CONTACT
Acme Corp
info@acmecorp.com
https://www.acmecorp.com

---
EOF

---

Complete Example -- Local Business (Dental)

# Acme Dental AI Optimization Layer
Version: 1.0
Last-Updated: 2026-02-07
Primary-Language: en-US
Canonical-Site: https://www.acmedental.com
AI-Mirror: https://ai.acmedental.com

---

## PURPOSE
This is an AI-readable representation of Acme Dental content optimized for:
- LLM crawlers
- AI search systems
- Agentic retrieval workflows
- Entity-based knowledge extraction

---

## CANONICAL AUTHORITY
The canonical source of truth is always:
https://www.acmedental.com

---

## START HERE (CRAWLING GUIDANCE)

### 1. Full Site Intelligence (Recommended)
https://ai.acmedental.com/llm-index.json

### 2. Entity Index
https://ai.acmedental.com/entities.json

### 3. Sitemap
https://ai.acmedental.com/sitemap.xml

---

## PRODUCTS & SERVICES
- Dental Cleaning: https://ai.acmedental.com/services/cleaning
- Teeth Whitening: https://ai.acmedental.com/services/whitening
- Dental Implants: https://ai.acmedental.com/services/implants

---

## ENTITY STATS
- Total Entities: 12
- Services: 5
- Pages: 7

---

## KEY FACTS
- Serving Tampa since 2006
- 4.8 star rating (127 reviews)
- Same-day emergencies available
- Most insurance accepted
- Free parking on-site

---

## FAQ

### Do you accept my insurance?
We accept most major dental insurance including Delta, Cigna,
MetLife, and Aetna. Call us to verify your specific plan.

### Do you offer emergency appointments?
Yes. We reserve time for same-day emergencies. Call us
immediately if you are experiencing dental pain.

### What are your hours?
Monday-Friday 8am-5pm, Saturday 9am-2pm. Closed Sunday.

---

## CONTACT
Acme Dental
info@acmedental.com
+1-813-555-1234
https://www.acmedental.com

---
EOF

Why FAQ Matters for Local Businesses

When someone asks an AI "does Acme Dental accept Delta insurance?" -- the FAQ section gives the AI an instant, accurate answer without fetching any additional pages. This is the content that directly drives AI recommendations.

---

Section-by-Section Breakdown

The Header Block

Immediately after the H1 heading, include metadata as key-value pairs:

Field	Level	Example
Version	Required	1.0
Last-Updated	Required	2026-02-07
Canonical-Site	Required	https://www.yoursite.com
Primary-Language	Recommended	en-US
AI-Mirror	Recommended	https://ai.yoursite.com
Generator	Optional	aso-generator/2.0
Conformance-Level	Optional	3

START HERE (CRAWLING GUIDANCE)

This is the most important section. It tells AI systems exactly where to go next, in priority order:

1. llm-index.json -- Your full structured data. Always first.
2. entities.json -- Structured entity data
3. knowledge.graph.jsonld -- Entity relationships
4. sitemap.xml -- Full URL listing

At Minimum, Include llm-index.json

The whole point of this section is to get the AI to your structured data. If you only list one resource, make it this one.

PRODUCTS & SERVICES

List each product or service with a link. If you have too many (e-commerce catalogs), list your top items and add a fallback line:

## PRODUCTS & SERVICES
- Alpine Pro Tent (4-Person): https://ai.acmeoutdoors.com/products/alpine-pro-tent
- TrailMaster Hiking Boots: https://ai.acmeoutdoors.com/products/trailmaster-boots
See entities.json for full catalog (240+ products).

---

Deployment

Where to Put the File

Place llms.txt at the root of your AI subdomain:

https://ai.yoursite.com/llms.txt

The file lives on the AI subdomain only -- not on your main site.

HTTP Headers

Configure your server to return these headers:

Content-Type: text/plain; charset=utf-8
Cache-Control: public, max-age=86400
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff

CORS Matters

The Access-Control-Allow-Origin: * header lets AI agents fetch your file from any origin. Without it, browser-based AI tools and some crawlers won't be able to read it.

Connect It to Your AI Discovery Page

Add a <link> element to your ADP's <head>:

<link rel="ai-manifest" type="text/plain"
      href="https://ai.yoursite.com/llms.txt" />

---

Formatting Rules

• UTF-8 encoding, no BOM
• Unix line endings (\n). No carriage returns.
• Sections separated by --- (horizontal rules)
• Sections in order: PURPOSE > CANONICAL AUTHORITY > START HERE > (optional sections) > CONTACT
• End with EOF -- lets AI systems confirm they received the complete file
• No HTML -- Plain Markdown only
• Under 4 KB ideally. Hard limit is 20 KB.

Common Mistake

Forgetting the EOF marker. Without it, an AI system can't tell if the file was truncated during download.

---

Testing Your File

1. Visit ai.yoursite.com/llms.txt in a browser -- it should render as plain text
2. Check the H1 heading matches your business name
3. Verify all URLs in START HERE are reachable
4. Confirm the Canonical-Site header matches the URL in CANONICAL AUTHORITY
5. Check the file ends with EOF
6. Inspect response headers -- confirm Content-Type is text/plain and CORS is set

Building llm-index.json | AI Discovery Page