Robots.txt and llms.txt for AI crawlers — a drop-in template
AI answer engines have their own crawlers. ChatGPT uses GPTBot. Claude uses ClaudeBot. Perplexity uses PerplexityBot. Each has its own rules. Here are drop-in robots.txt and llms.txt files that welcome the verified crawlers, plus the citations to vendor docs that prove each line is real.
Read more →