robots.txt is 30 years old and it is more relevant than ever. In the AI search era, it serves a dual purpose: managing traditional search crawlers and governing which AI systems can access your content for training and retrieval. Get it wrong and you either block AI bots from content you want cited, or you grant unrestricted access to systems you'd prefer to limit. This guide gives you exact robots.txt syntax for every major AI bot, explains the precedence rules that govern how bots interpret your file, and shows you how to grant granular, path-level access without losing control of sensitive content.
Understanding robots.txt Precedence and AI Bot Behavior
The Robots Exclusion Protocol has clear precedence rules: a specific User-agent directive takes precedence over the wildcard User-agent: * directive for any bot that matches the specific rule. This means if you have 'User-agent: * / Disallow: /' (block everything) but also 'User-agent: GPTBot / Allow: /', GPTBot will be allowed while all other unrecognized bots are blocked. AI bots are required by their terms of service and community norms to honor robots.txt — OpenAI, Anthropic, Perplexity, and Google all explicitly state they respect robots.txt directives. However, their compliance is voluntary, not technically enforced. Bad actors spoofing AI bot User-agents will ignore your robots.txt regardless. Use robots.txt for access governance with legitimate AI systems, and use WAF rules or server-side authentication for content you need to truly protect. One important nuance: GPTBot distinguishes between training data access and real-time search access. OpenAI uses 'GPTBot' for training and 'OAI-SearchBot' for ChatGPT's browsing feature — you may want different rules for each.
- Specific User-agent rules always override wildcard User-agent: * rules
- GPTBot (training) and OAI-SearchBot (real-time search) are separate — configure both
- All major AI companies commit to honoring robots.txt in their published policies
- robots.txt is governance, not security — use WAF for truly sensitive content
- Verify your robots.txt is served at the exact path /robots.txt with 200 status
- robots.txt must be plain text (text/plain Content-Type) — not HTML or redirected
Complete AI Bot robots.txt Configuration: Production-Ready Syntax
Here is a complete, production-ready robots.txt configuration for a site that wants to allow all major AI bots to access content pages while blocking admin, app, and API routes. The file uses named User-agent blocks for each AI bot, each with identical Allow and Disallow rules. You would customize the Disallow paths for your specific site structure. The key AI User-agent strings to include: GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Gemini-extended (also listed as Google-Extended in some Google documentation), cohere-ai, Meta-ExternalAgent, YouBot, and Bytespider. For each, specify 'Allow: /blog/' (or whatever your content paths are), 'Allow: /resources/', 'Allow: /services/', and 'Disallow: /admin/', 'Disallow: /api/', 'Disallow: /app/', 'Disallow: /checkout/'. End your robots.txt with the Sitemap URL. Keep your existing Googlebot, Bingbot, and other traditional search engine rules in the same file — they coexist without conflict.
- User-agent: GPTBot → the OpenAI training crawler
- User-agent: OAI-SearchBot → ChatGPT's real-time browsing feature
- User-agent: ClaudeBot → Anthropic's Claude training and retrieval crawler
- User-agent: anthropic-ai → Anthropic's secondary research crawler
- User-agent: PerplexityBot → Perplexity AI real-time search crawler
- User-agent: Google-Extended → Google's AI training opt-out mechanism
- User-agent: cohere-ai → Cohere language model training crawler
- User-agent: Meta-ExternalAgent → Meta AI crawler
Granular Access Control: Allowing AI to Some Paths, Blocking Others
The power of named User-agent blocks is granular path-level control. For a SaaS company, a typical configuration would allow AI bots to access marketing content (blog, landing pages, case studies, pricing) while blocking authenticated app routes and API endpoints. For a media company, you might allow access to free articles but block paywalled content. For an e-commerce site, you might allow product pages and category pages but block checkout, account, and order management paths. When constructing path rules, be aware of the specificity rule: longer paths take precedence over shorter paths. 'Disallow: /admin/' will block all paths starting with /admin/. 'Allow: /admin/public/' will re-allow that specific subdirectory even if the broader /admin/ path is disallowed. Use this nesting to create precise access maps. Test your rules using Google's robots.txt Tester by substituting each AI bot's User-agent string. Also test edge cases: does your rule correctly handle trailing slashes? Does it handle query parameters? The robots.txt standard treats '?query=param' as part of the path, so 'Disallow: /search' will also block '/search?q=term'.
- Allow marketing content paths; Disallow app, API, admin, and checkout paths
- Use longer paths for exceptions to shorter Disallow rules
- Test rules for trailing slash variants: /blog and /blog/ are technically different
- Query strings are part of the path — 'Disallow: /search' blocks all search URLs
- Consider separate rule sets for training bots vs. real-time retrieval bots
- Document your AI bot access policy so your team understands the intent
The Google-Extended Directive: Opting Out of Google AI Training
Google introduced a special User-agent called Google-Extended specifically for AI training opt-outs. Blocking Google-Extended prevents your content from being used in Google's Gemini model training, while still allowing Googlebot to crawl and rank your pages normally. This is a notable policy decision — Google separated AI training from search crawling to give publishers more control. If you want to allow your content in Google Search but not in Gemini training data: 'User-agent: Google-Extended / Disallow: /'. Note that this does not block Gemini's real-time retrieval for AI Overviews — that is controlled separately. For publishers who want maximum AI citation potential from Google's products while protecting training data, the recommended configuration is: allow Googlebot fully, allow Google-Extended to your public content paths (so your content is in training data and thus cited more readily by Gemini), and block Google-Extended only from proprietary or paywalled content. The decision whether to allow or block Google-Extended is a strategic one: opt-in means Gemini learns from your content and may cite you more; opt-out keeps your content out of Google's AI training pipeline.
- Google-Extended controls AI training use, not regular Search crawling
- Blocking Google-Extended has zero impact on your Google Search rankings
- Allowing Google-Extended may improve Gemini citation rates for your content
- This is a strategic decision — consider your content licensing and business model
- Media publishers with licensing programs often block Google-Extended by default
- Document your Google-Extended policy in your content licensing terms
Monitoring Compliance and Handling Rogue Scrapers
Major AI companies — OpenAI, Anthropic, Perplexity, Google — publish their crawler policies and commit to honoring robots.txt. However, smaller or less reputable AI companies may not comply. Monitor your server access logs for User-agent strings claiming to be AI bots. For each, verify via reverse DNS lookup that the IP resolves to the company's domain. If you receive requests from IPs claiming to be GPTBot but resolving to unknown hosts, these are likely scrapers spoofing the User-agent to bypass blocks. Handle these at the WAF level, not in robots.txt. For legitimate bots you want to block (perhaps you've decided to opt out of a specific AI company's training), robots.txt is the appropriate tool for compliant companies. Create a dedicated monitoring dashboard that tracks AI bot crawl volume, response codes, and top crawled pages weekly. This gives you visibility into how AI systems prioritize your content and lets you identify crawl issues early.
- Verify AI bot legitimacy via reverse DNS lookup before configuring access rules
- Monitor server logs weekly for new AI bot User-agents and unusual crawl patterns
- Handle non-compliant scrapers at the WAF/firewall level, not robots.txt
- Alert on sudden spikes in bot traffic — could indicate aggressive crawl or attack
- Keep a change log for your robots.txt — track what changed and why
- Review and update AI bot rules quarterly as new AI systems launch
Your robots.txt file is the control plane for AI visibility. Spend two hours auditing and updating it with explicit named rules for every major AI bot, and you'll have resolved the most common AI crawlability issue in a single session. Use the Google-Extended decision as a forcing function to define your AI content strategy: do you want to be in AI training data or not? The technical implementation either way is trivial — the strategic clarity is the harder work. After updating your robots.txt, verify each bot's access using Google's testing tool, then monitor server logs for a week to confirm crawl activity from the bots you've allowed.
Frequently Asked Questions
If I block GPTBot, will my site still appear in ChatGPT answers?
Blocking GPTBot prevents OpenAI from crawling your site for training data, which reduces the likelihood your content is embedded in GPT model weights. However, ChatGPT's real-time browsing feature uses OAI-SearchBot — blocking GPTBot does not block real-time retrieval. For current ChatGPT answers, OAI-SearchBot matters more than GPTBot. To block all OpenAI access, you'd need to block both User-agent strings.
Does robots.txt affect my site's performance or SEO in any way?
robots.txt itself has negligible performance impact — it's a small plain-text file. Well-configured robots.txt can improve crawl efficiency by directing bots away from low-value pages (paginated archives, faceted navigation, admin routes), which preserves crawl budget for important pages. Misconfigured robots.txt that accidentally blocks important pages can significantly harm SEO — always test changes before deploying.
How often should I update my robots.txt for AI bots?
Review quarterly at minimum, and immediately when: a major new AI product launches (check for its crawler User-agent), you add new site sections that need access rules, or you change your AI content licensing strategy. Subscribe to OpenAI, Anthropic, and Perplexity's developer blogs to catch crawler policy changes. The AI search landscape is evolving rapidly — what was comprehensive six months ago may be missing several new bots today.