Dora NodaAbout 5.6 million websites now tell OpenAI's GPTBot to stay out in their robots.txt file, up from roughly 3.3 million at the start of July 2025 — an increase of almost 70 percent in five months, according to BuiltWith data reported by The Register. Read that number quickly and it looks like the whole web is slamming the door on AI crawlers. Read the actual census data and the picture inverts: publishers are not blocking AI. They are blocking training and allowing answering — often to two different crawlers run by the same company.
A Cloudflare-network snapshot from August 31, 2026 makes the split concrete. Training bots are blocked hard: GPTBot sits at a 2.33:1 block-to-allow ratio, ClaudeBot at 2.39:1, ByteDance's Bytespider at 5.80:1. But the answering crawlers — the bots that fetch pages to ground chat responses and citations — are welcomed: OAI-SearchBot at 0.94:1 (allowed more often than blocked) and ChatGPT-User at 1.12:1, as GeoParrot's census analysis summarizes. The web has quietly converged on an allowlist posture, one user-agent string at a time.
If you run a self-hosted PaaS, this split lands on you twice. First on your own docs site: an agent researching how to deploy on your platform has to get through your robots.txt before it ever reads a word of your documentation. Second, and more consequentially, on every tenant app your platform deploys: whatever robots.txt your buildpacks, scaffolds, or edge defaults ship becomes a decision about AI visibility that you made silently on someone else's behalf. This post gives you a concrete default for both surfaces, the evidence behind it, and the gotchas that break naive policies.
The three species of AI crawler
The single biggest source of bad robots.txt policy is treating "AI crawler" as one thing. Every major vendor now operates at least three distinct bots, and blocking the wrong one costs you visibility while blocking the right one costs you nothing:
| Purpose | OpenAI | Anthropic | Others |
|---|---|---|---|
| Training-data collection | GPTBot | ClaudeBot | CCBot (Common Crawl), Bytespider, Google-Extended, Applebot-Extended, Amazonbot |
| Search/answer indexing | OAI-SearchBot | Claude-SearchBot | PerplexityBot, DuckAssistBot |
| On-demand user fetch | ChatGPT-User | Claude-User | Perplexity-User |
OpenAI's trio has been stable for a while: GPTBot crawls for model training, OAI-SearchBot builds the search index, and ChatGPT-User fetches pages when a user asks about them. Anthropic formalized the same three-bot framework in 2026: ClaudeBot collects training data, Claude-SearchBot indexes content for search results, and Claude-User retrieves pages when someone asks Claude a question. Anthropic is explicit about the stakes — blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results."
Note the asymmetry that matters for everything below: training crawlers take your content to build someone else's model weights. Search and on-demand crawlers bring your content back to a human who is one click away from your site. A docs page blocked from GPTBot loses nothing; the same page blocked from OAI-SearchBot and Claude-SearchBot is invisible in the exact moment a developer asks an AI assistant how to do the thing your docs explain.
The defaults, up front
Here is the three-tier policy for a docs or marketing site. Block training, allow answering and search, allow on-demand fetch:
# AI-crawler policy: block training, welcome answering.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: *
Allow: /And here is the default your platform should ship in tenant scaffolds and buildpack templates — the same answering-friendly posture, with the training block left as a documented one-line choice rather than a silent verdict:
# Default AI-crawler policy for apps deployed on this platform.
# Answering/search crawlers are welcome so your public pages stay
# visible in AI search and citations. Training crawlers are allowed
# by default; disallow them below if your content should not feed
# model training.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
# Uncomment to also block AI training crawlers:
# User-agent: GPTBot
# Disallow: /
# User-agent: ClaudeBot
# Disallow: /
# User-agent: CCBot
# Disallow: /
# User-agent: Bytespider
# Disallow: /
User-agent: *
Allow: /Why do the two defaults differ on training? Because the two site owners are answering different questions. Your docs exist to be read as widely as possible, and training ingestion of documentation buys you nothing — blocking it is pure upside. A tenant's app might be a public blog, a storefront, or an internal dashboard behind auth; the platform cannot know which, so the tenant default welcomes the bots that drive traffic and citations while leaving the training decision visible, documented, and one uncomment away. Everything that follows is the evidence and the fine print for these two snippets.
What over-blocking actually costs
Two findings from the last year should end any temptation toward a blanket Disallow: / for every AI user-agent.
First, blocking AI crawlers measurably costs traffic. Research published December 31, 2025, by academics at Rutgers and Wharton found that publishers blocking AI crawlers via robots.txt saw a 23.1 percent decline in total monthly visits and a 13.9 percent decline in human-only browsing. Blocking also did not reliably reduce AI citation rates — the worst of both worlds: less traffic without meaningfully less AI reuse of the content.
Second, robots.txt is a request, not a lock, and a large minority of AI traffic treats it as optional. TollBit's Q4 2025 data, reported by Digiday, found that 30 percent of AI bot scrapes bypassed explicit robots.txt permissions, with OpenAI's ChatGPT-User the worst offender at 42 percent of its scrapes hitting sites that had explicitly blocked it. OpenAI's own documentation frames ChatGPT-User as user-initiated fetching rather than automatic crawling, which is why its operators argue robots.txt rules may not apply. If your threat model is "this content must never reach an AI system," robots.txt was never the control — you need authentication, edge enforcement, or both.
Together these findings draw the policy boundary cleanly. Use robots.txt to express a deliberate preference that well-behaved crawlers honor: keep training bots out, keep answering bots in. Do not mistake that preference for protection, and do not pay a 23 percent traffic tax to express a preference that 30 percent of scrapers ignore.
Four gotchas that break naive policies
1. A named group replaces *; it does not inherit from it. This is the most common robots.txt bug in AI-crawler policies. The moment you write User-agent: GPTBot, every rule under User-agent: * stops applying to GPTBot. If your catch-all group disallows /api/, /admin/, or /internal/, a bare User-agent: PerplexityBot / Allow: / group silently re-opens all of those paths to that crawler. Every named allow-group must restate the sensitive-path disallows from your catch-all. Audit any AI-crawler snippet — including the ones above — against your real * group before shipping.
2. Stale user-agent strings rot your policy. Anthropic retired the anthropic-ai and Claude-Web tokens; ClaudeBot now honors directives previously set for them, but the reverse is not true — a policy that allows anthropic-ai while blocking nothing else is guarding a door nobody uses. Crawler token registries drift every year; re-verify your named agents against vendor documentation on the same cadence you review TLS or headers.
3. Your edge may be overriding your file. Cloudflare-managed robots.txt blocks prepend Disallow: / for a fixed AI-crawler set (GPTBot, ClaudeBot, CCBot, Google-Extended, Amazonbot, Applebot-Extended, Bytespider, and others), and Cloudflare's "Block AI Scrapers and Crawlers" toggle 403s AI user-agents at the WAF regardless of what your file says. Since July 2025 that blocking has been the default for new Cloudflare customers, and a July 2026 policy update extends default blocking of Training and Agent-class crawlers on ad-carrying pages. If your docs sit behind Cloudflare, curl your live robots.txt with an AI user-agent and confirm the served file is the one you wrote — a repo-served policy means nothing if the edge injects its own above it.
4. On-demand fetches play by different rules. ChatGPT-User and Claude-User act when a specific user pastes your URL into a chat session. Blocking them in robots.txt does not stop a determined user from getting your public page summarized — the model will say it cannot browse, the user will paste the text, and you have bought nothing but a worse citation. This is why both defaults above allow the on-demand tier unconditionally: for public content, the fetch is happening with or without your blessing, and blessing it is what gets you cited.
The positive complement: llms.txt
Robots.txt tells crawlers what not to fetch. The convention it pairs with, llms.txt, tells AI systems what is most useful — a markdown index of your site's key pages served at /llms.txt, with an optional /llms-full.txt embedding full content so agents need fewer follow-up fetches. Jeremy Howard proposed it in September 2024; adoption has compounded since. Common Crawl's analysis of its July 2026 archive found 11.72 percent of sampled hosts serving /llms.txt, up from roughly 2 percent in the 2025 Web Almanac.
The pairing matters more than either file alone. An allowlist robots.txt without llms.txt welcomes agents into a maze; llms.txt behind a blanket AI block is a map to a locked building. For a docs site, the minimal useful llms.txt is short:
# Example Docs
> Deploy apps from git on infrastructure you own.
- [Quickstart](https://example.com/docs/quickstart): deploy your first app in five minutes.
- [Configuration reference](https://example.com/docs/config): every setting, with defaults.
- [CLI reference](https://example.com/docs/cli): command-by-command usage.
- [Troubleshooting](https://example.com/docs/troubleshooting): the ten errors everyone hits.Serve it, link your canonical pages, and keep it current on the same release process as the docs themselves. A stale llms.txt pointing at renamed pages is worse than none — it teaches agents your 404 page.
Why the tenant default must be explicit, not inherited
Return to the second surface: the robots.txt your platform ships for tenant apps. Today, most tenants inherit whatever the scaffold happened to include — a buildpack template written before GPTBot existed, a framework starter with User-agent: * / Allow: /, or nothing at all, which every crawler reads as "come on in." None of these is a decision; all of them have consequences now that training and answering crawlers diverge.
Three principles for the platform default:
Ship answering-friendly, not AI-hostile. A tenant's public pages gain traffic and citations from search and on-demand crawlers and lose nothing to them that a public page had. The Rutgers/Wharton traffic decline is the cost you would silently impose on every tenant with a block-everything scaffold. Defaults should favor the tenant's discoverability; restrictions should be the tenant's explicit choice.
Make the training decision visible. The commented training-block in the tenant snippet above is doing real work: it names the crawlers, states the tradeoff in plain language, and shows the exact change. A tenant who uncomments four lines has made an informed decision. A tenant whose scaffold shipped a silent Disallow: / for GPTBot has inherited your legal posture without ever seeing it — and will discover it only when something downstream behaves unexpectedly.
Document the enforcement boundary. Tell tenants, in the deploy docs next to the snippet, that robots.txt is advisory: roughly a third of AI scrapes ignore it, and anything that must not be ingested needs auth or edge rules. Platforms that let tenants believe a robots.txt line is access control are setting up the support ticket where a tenant learns otherwise in public.
One more consideration for the platform operator: your own docs should model the policy you recommend. If your docs site blocks OAI-SearchBot while your tenant guide says to allow it, tenants will — correctly — follow your example instead of your advice. Ship the three-tier policy on your own domain first.
The door, the map, and the meter
The direction of travel is clear. Crawlers keep splitting into finer-grained identities, publishers keep sorting them into training-blocked and answering-welcomed, and the infrastructure layer is starting to price what robots.txt could only permit or deny: Cloudflare's pay-per-crawl experiment is evolving toward payment tied to citations in generated answers, and licensing marketplaces now intermediate the transactions a binary allow/deny never could. The binary file is not going away — it remains the cheapest signal with the widest support — but it is becoming the floor of an AI-access policy, not the whole policy.
Set the floor deliberately. Block the bots that train on your words, welcome the bots that bring readers to them, hand agents a map with llms.txt, and never let a tenant inherit a verdict they never saw. The robots.txt default is a small file with an outsized vote on who your docs — and your tenants' apps — are for.
Bex.co is the open-source, AI-native Render alternative — push a git repo, get a running HTTPS service on machines you own. Star the repo on GitHub or deploy your first app today.
