Configuring robots.txt for AI Fresh

How to direct AI crawlers to your AI-optimized content while keeping them away from your main site. Two files, clean separation.

The Two-File Strategy

When you set up an AI subdomain, you need two separate robots.txt files with opposite rules:

Main site (yoursite.com/robots.txt) -- Blocks AI crawlers, allows search engines
AI subdomain (ai.yoursite.com/robots.txt) -- Allows AI crawlers, blocks search engines

Your main site is for humans and Google. Your AI subdomain is for ChatGPT, Claude, and Perplexity. robots.txt enforces the separation.

Why Separate Sites?

The core idea behind the two-site architecture is independence. Your human-facing website and your AI-facing content have different audiences, different needs, and will evolve differently over time.

With a separate AI layer, you can:

Add AI-specific content that doesn't belong on your main site
Remove content that confuses AI without affecting what humans see
Experiment and iterate on your AI presence without touching your production website
Respond to new AI capabilities as they emerge

Think of It Like Responsive Design

You wouldn't serve the exact same layout to mobile and desktop -- you adapt to the device. The AI subdomain is the same concept: adapting your content to the consumer.

Why Block AI from Your Main Site?

This might seem counterintuitive -- don't you want AI to crawl your site? Yes, but you want AI to crawl the right version of your site.

Signal-to-noise ratio -- Your main site is built for humans: navigation menus, hero images, JavaScript frameworks, cookie banners. AI crawlers have to wade through all of this to extract the actual content. Your AI subdomain is pure signal.

Consistent structured data -- Your AI subdomain has guaranteed Schema.org markup on every page, plus llm-index.json, entities, and knowledge graphs. Your main site might have inconsistent or missing structured data.

Control over what AI learns -- With an AI subdomain, you're explicitly defining what AI should know about your business.

Avoid duplicate/conflicting information -- If AI crawls both your main site and your AI subdomain, it might find slightly different information. Blocking the main site ensures AI only sees your canonical, structured version.

Exception: AI Discovery Page

Your ADP at /ai-discovery should always be crawlable -- it's the bridge that points AI crawlers to your AI subdomain. Make sure to add Allow: /ai-discovery in your main site's robots.txt.

Traffic Flow

AI Crawler (GPTBot, ClaudeBot, etc.)
  -> yoursite.com (BLOCKED)
  -> /ai-discovery (ALLOWED - bridge)
  -> ai.yoursite.com (ALLOWED)

Search Crawler (Googlebot, Bingbot, etc.)
  -> yoursite.com (ALLOWED)

Main Site robots.txt

Place at yoursite.com/robots.txt.

# ============================================================
# ROBOTS.TXT FOR YOURSITE.COM (MAIN SITE)
# ============================================================
# This file BLOCKS AI crawlers and ALLOWS regular search crawlers.
# AI crawlers should use ai.yoursite.com instead.
#
# IMPORTANT: All crawlers are allowed to access /ai-discovery
# Last updated: February 2026
# ============================================================

# ============================================================
# OPENAI CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: GPTBot
Allow: /ai-discovery
Disallow: /

User-agent: ChatGPT-User
Allow: /ai-discovery
Disallow: /

User-agent: OAI-SearchBot
Allow: /ai-discovery
Disallow: /

# ============================================================
# ANTHROPIC (CLAUDE) CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: anthropic-ai
Allow: /ai-discovery
Disallow: /

User-agent: ClaudeBot
Allow: /ai-discovery
Disallow: /

User-agent: Claude-Web
Allow: /ai-discovery
Disallow: /

User-agent: Claude-SearchBot
Allow: /ai-discovery
Disallow: /

# ============================================================
# PERPLEXITY CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: PerplexityBot
Allow: /ai-discovery
Disallow: /

User-agent: Perplexity-User
Allow: /ai-discovery
Disallow: /

# ============================================================
# GOOGLE AI CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: Google-Extended
Allow: /ai-discovery
Disallow: /

User-agent: GoogleOther
Allow: /ai-discovery
Disallow: /

# ============================================================
# META AI CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: FacebookBot
Allow: /ai-discovery
Disallow: /

User-agent: meta-externalagent
Allow: /ai-discovery
Disallow: /

# ============================================================
# OTHER AI CRAWLERS - BLOCKED (except /ai-discovery)
# ============================================================
User-agent: Amazonbot
Allow: /ai-discovery
Disallow: /

User-agent: Applebot-Extended
Allow: /ai-discovery
Disallow: /

User-agent: Bytespider
Allow: /ai-discovery
Disallow: /

User-agent: CCBot
Allow: /ai-discovery
Disallow: /

User-agent: cohere-ai
Allow: /ai-discovery
Disallow: /

User-agent: Diffbot
Allow: /ai-discovery
Disallow: /

User-agent: DeepSeekBot
Allow: /ai-discovery
Disallow: /

User-agent: DuckAssistBot
Allow: /ai-discovery
Disallow: /

User-agent: YouBot
Allow: /ai-discovery
Disallow: /

# ============================================================
# REGULAR SEARCH CRAWLERS - ALLOWED
# ============================================================
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Slurp
Allow: /

User-agent: DuckDuckBot
Allow: /

# ============================================================
# DEFAULT - ALLOW (for unlisted crawlers)
# ============================================================
User-agent: *
Allow: /

# ============================================================
# SITEMAP
# ============================================================
Sitemap: https://yoursite.com/sitemap.xml

Why Allow: /ai-discovery Appears for Every AI Crawler

In robots.txt, specific user-agent rules override wildcard rules. If you block GPTBot with Disallow: /, it can't access anything -- including /ai-discovery -- even if you have a separate wildcard Allow. Each AI crawler needs its own explicit Allow: /ai-discovery line.

AI Subdomain robots.txt

Place at ai.yoursite.com/robots.txt. Does the opposite: welcomes all AI crawlers while blocking traditional search engines.

# ============================================================
# ROBOTS.TXT FOR AI.YOURSITE.COM (AI SUBDOMAIN)
# ============================================================
# This file ALLOWS all AI crawlers and BLOCKS regular search crawlers.
# Last updated: February 2026
# ============================================================

# OPENAI - ALLOWED
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

# ANTHROPIC - ALLOWED
User-agent: anthropic-ai
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

# PERPLEXITY - ALLOWED
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# GOOGLE AI - ALLOWED
User-agent: Google-Extended
Allow: /

User-agent: GoogleOther
Allow: /

User-agent: Gemini-AI
Allow: /

# MICROSOFT/BING AI - ALLOWED
User-agent: Copilot
Allow: /

# META AI - ALLOWED
User-agent: FacebookBot
Allow: /

User-agent: meta-externalagent
Allow: /

# OTHER AI/LLM - ALLOWED
User-agent: Amazonbot
Allow: /

User-agent: Applebot
Allow: /

User-agent: Bytespider
Allow: /

User-agent: CCBot
Allow: /

User-agent: cohere-ai
Allow: /

User-agent: DeepSeekBot
Allow: /

User-agent: Diffbot
Allow: /

User-agent: DuckAssistBot
Allow: /

User-agent: YouBot
Allow: /

# BLOCK REGULAR SEARCH ENGINE CRAWLERS
# These should index the main site, not the AI subdomain
User-agent: Googlebot
Disallow: /

User-agent: Googlebot-Image
Disallow: /

User-agent: Bingbot
Disallow: /

User-agent: msnbot
Disallow: /

User-agent: Slurp
Disallow: /

User-agent: DuckDuckBot
Disallow: /

# BLOCK EVERYTHING ELSE BY DEFAULT
User-agent: *
Disallow: /

# SITEMAP (AI SUBDOMAIN)
Sitemap: https://ai.yoursite.com/sitemap.xml

Quick Reference

Crawler	Main Site	AI Subdomain
GPTBot (OpenAI)	Blocked	Allowed
ClaudeBot (Anthropic)	Blocked	Allowed
PerplexityBot	Blocked	Allowed
Google-Extended (AI)	Blocked	Allowed
Googlebot (Search)	Allowed	Blocked
Bingbot (Search)	Allowed	Blocked
/ai-discovery page	Allowed for all	--

Current Major AI Crawler User-Agents

OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
Anthropic: anthropic-ai, ClaudeBot, Claude-Web, Claude-SearchBot
Google: Google-Extended, GoogleOther, Gemini-AI
Meta: FacebookBot, meta-externalagent
Perplexity: PerplexityBot, Perplexity-User
Others: Amazonbot, Applebot-Extended, Bytespider, CCBot, cohere-ai, DeepSeekBot

Testing Your Setup

Visit yoursite.com/robots.txt -- verify AI crawlers are blocked
Visit ai.yoursite.com/robots.txt -- verify AI crawlers are allowed
Check that /ai-discovery is explicitly allowed on the main site

Common Mistake

Getting the two files swapped. Double-check that the main site blocks AI and the AI subdomain allows AI.

Building an AI Discovery Page | AI Subdomain Setup

Configuring robots.txt for AI Fresh ​

The Two-File Strategy ​

Why Separate Sites? ​

Why Block AI from Your Main Site? ​

Traffic Flow ​

Main Site robots.txt ​

AI Subdomain robots.txt ​

Quick Reference ​

Current Major AI Crawler User-Agents ​

Testing Your Setup ​