# Stage 1 WS-3 (rev 9 M2 fix): explicit AI-bot allow + mirrored Disallow.
# RFC 9309 + Google grouping: specific UA blocks DO NOT inherit from
# `User-agent: *`. Each AI-bot block must restate the Disallow set or it
# silently has more permissive access than the wildcard. We're explicitly
# welcoming GPTBot / ClaudeBot / PerplexityBot / Google-Extended for
# GEO/AEO surfaces while keeping /api/* off-limits to all crawlers.

User-agent: *
Allow: /
Disallow: /api/

User-agent: GPTBot
Allow: /
Disallow: /api/

User-agent: ClaudeBot
Allow: /
Disallow: /api/

User-agent: PerplexityBot
Allow: /
Disallow: /api/

User-agent: Google-Extended
Allow: /
Disallow: /api/

# Stage 2 (2026-05-21, FB-21): explicit allow for search/citation crawlers.
# These are DIFFERENT bots from the training crawlers above — blocking GPTBot
# does NOT block OAI-SearchBot, and OpenAI documents that blocking
# OAI-SearchBot removes the site from ChatGPT Search answers. Anthropic now
# operates THREE crawlers: ClaudeBot (training), Claude-SearchBot (search
# index), Claude-User (user-initiated fetch). Deprecated strings Claude-Web
# and anthropic-ai no longer fire. Source:
# 200_Reference/seo-knowledge-base/SEO-tactical-2026Q2-external.md §2.3.

User-agent: OAI-SearchBot
Allow: /
Disallow: /api/

User-agent: ChatGPT-User
Allow: /
Disallow: /api/

User-agent: Claude-SearchBot
Allow: /
Disallow: /api/

User-agent: Claude-User
Allow: /
Disallow: /api/

User-agent: Perplexity-User
Allow: /
Disallow: /api/

Sitemap: https://www.luceagentic.com/sitemap-index.xml