The world of AI is moving fast and so are the bots that crawl the web to feed large language models (LLMs), AI search engines, and generative tools. If you want your site’s content to be available for AI models like ChatGPT, Claude, Perplexity, or Gemini, you can explicitly grant permission by updating your robots.txt file.

In this guide, you’ll learn:

• Why AI bots matter in 2025
• Which user-agents to include
• A ready-to-use robots.txt template

Why AI Bots Need robots.txt Rules

Traditionally, robots.txt controlled search engines like Googlebot and Bingbot. Now, AI companies also respect these directives to decide whether your content can be indexed for AI search or used for model training.

• Search bots (AI answers/assistants): Allowing them means your content may appear in AI-powered search results (e.g., Perplexity, You.com).
• Training crawlers: Allowing them means your content may be used to train or fine-tune large AI models (e.g., GPTBot, ClaudeBot).

By updating your robots.txt, you can allow, block, or mix your policies across these bots.

Major AI User-Agents in 2025

OpenAI (ChatGPT / SearchGPT)

• GPTBot => training crawler
• ChatGPT-User => on-demand browsing in ChatGPT
• OAI-SearchBot => indexing for SearchGPT

Anthropic (Claude)

• ClaudeBot => model training
• Claude-Web / anthropic-ai => crawling/AI access

Perplexity

• PerplexityBot => AI search engine

Google (Gemini)

• Google-Extended => controls if Gemini can use your content for AI

Apple

• Applebot-Extended => controls Apple AI training usage

Amazon

• Amazonbot => AI + product/search crawling

Common Crawl

• CCBot => feeds public datasets used in AI training

AI Search Engines

• YouBot => You.com
• PhindBot => Phind AI search
• ExaBot => Exa.ai
• AndiBot => Andi search
• FirecrawlAgent => Firecrawl AI

Example: Allowing AI Bots

Here’s a robots.txt example that explicitly allows AI bots alongside traditional crawlers:

# Default: allow all crawlers
User-agent: *
Disallow:

# --- Explicitly ALLOW major AI/LLM bots ---

# OpenAI
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /

# Anthropic (Claude)
User-agent: ClaudeBot
Allow: /
User-agent: Claude-Web
Allow: /
User-agent: anthropic-ai
Allow: /

# Perplexity
User-agent: PerplexityBot
Allow: /

# Google AI usage token
User-agent: Google-Extended
Allow: /

# Apple AI usage token
User-agent: Applebot-Extended
Allow: /

# Amazon
User-agent: Amazonbot
Allow: /

# Common Crawl
User-agent: CCBot
Allow: /

# AI search engines
User-agent: YouBot
Allow: /
User-agent: PhindBot
Allow: /
User-agent: ExaBot
Allow: /
User-agent: AndiBot
Allow: /
User-agent: FirecrawlAgent
Allow: /

Optional: Balanced Policy (Allow AI Search, Block Training)

If you want to appear in AI search results but not have your content used for model training, use a mixed policy. For example:

# Allow AI search crawlers
User-agent: PerplexityBot
Allow: /
User-agent: YouBot
Allow: /
User-agent: PhindBot
Allow: /
User-agent: ExaBot
Allow: /
User-agent: AndiBot
Allow: /
User-agent: OAI-SearchBot
Allow: /

# Disallow training/aggregator crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /

Pro Tips

• Keep it updated: New AI bots appear frequently so review your list quarterly.
• Combine with enforcement: Some bots may ignore rules. If strict compliance matters, pair robots rules with IP/reverse-DNS checks or WAF bot controls.
• Test quickly: Host your robots.txt at https://<your-domain>/robots.txt and fetch it from a browser or with curl to verify.

Going Further with SEO

If you’re running an Optimizely CMS site and want deeper control over SEO, check out my open-source tool: SEOBOOST for Optimizely CMS.

AI is becoming a parallel discovery channel next to Google. Your robots.txt file is the simplest way to take control of how your content participates.

Problem Statement

In Optimizely CMS 12, dynamically generated pages inherit URL segments from their container pages. However, in certain cases, some pages are renamed, leading to undesired URL structures. The three primary challenges faced in this scenario are:

1. Ignoring the container page name segment from the URL.

2. Redirecting to the new page without using any add-ons or manually mapping old URLs to new ones.

3. Create a URL for the 'View on Website' hyperlink without including container pages in the CMS.

This approach ensures a cleaner URL structure and avoids unnecessary dependencies while maintaining flexibility in content organization.

SEOBOOST for Optimizely CMS 12 is a technical SEO add-on designed to give developers and editors more control over canonical links, hreflang alternate links, breadcrumbs, and robots.txt management. It builds on the earlier EPiServer-era version and adds broader support for real-world Optimizely routing and content scenarios.

Last reviewed: July 24, 2026

If you are looking for the older EPiServer-focused article published on May 7, 2019, you can still find it here: SEOBOOST for EPiServer 11.

What SEOBOOST adds

• robots.txt management through the CMS
• Canonical link generation with better route handling
• hreflang alternate links for multilingual and multi-site setups
• Breadcrumb structured data output
• Support for simple addresses, partial routing, shortcuts, and trailing slash behavior

What's new in the Optimizely CMS 12 version

This version expands SEOBOOST beyond the earlier EPiServer implementation by supporting more real-world routing scenarios and giving editors more flexibility over technical SEO settings.

robots.txt management

SEOBOOST can expose a CMS-managed robots.txt page type and route, allowing editors to manage crawler directives without changing code.

public void ConfigureServices(IServiceCollection services)
{
    services.AddSeoBoost(x =>
    {
        x.EnableRobotsTxtSupport = true;
    });
}

Canonical links

SEOBOOST supports canonical link generation across scenarios such as:

• partial routing
• simple addresses
• page shortcuts
• multi-site and multi-domain environments
• automatic trailing slash handling

You can also specify a custom canonical field in CMS:

public void ConfigureServices(IServiceCollection services)
{
    services.AddSeoBoost(x =>
    {
        x.CustomCanonicalTagFieldName = "CanonicalTag";
    });
}

hreflang alternate links

Like canonical links, SEOBOOST supports alternate link generation with better handling for multilingual and multi-domain Optimizely setups, helping teams create cleaner signals for localized content.

Installation

Install the package with:

dotnet add package SeoBoost

Example configuration

public void ConfigureServices(IServiceCollection services)
{
    services.AddSeoBoost(x =>
    {
        x.CustomCanonicalTagFieldName = "CanonicalTag";
        x.UseSimpleAddressAsPath = true;
        x.EnableRobotsTxtSupport = true;
        x.UseMirrorPageReference = true;
        x.UseSiteUrlAsHost = true;
    });
}

When SEOBOOST is useful

• When canonical URLs need to reflect real routing behavior
• When multilingual pages need stronger hreflang output
• When editors need CMS-managed robots.txt support
• When technical SEO needs to be handled more consistently across sites

Links

GitHub repository
NuGet package
Legacy EPiServer 11 post

FAQ

Is this the same as the older 2019 SEOBOOST post?
It is the newer Optimizely CMS 12 version with broader features and updated configuration options.

Does SEOBOOST only handle canonical tags?
No. It also supports hreflang alternate links, breadcrumbs, and robots.txt management.