Quick answer? You give an AI crawler permission the same way you’d give any visitor a key: through a small text file called robots.txt, (and many now say llms.txt) through explicit allow rules in your firewall or CDN, and increasingly through a cryptographic ID check called Web Bot Auth. There’s no single “grant access” button. Permission is a layered decision you make per crawler and per purpose (training, search, or live agent lookups), and starting September 15, 2026, part of that decision gets made for you by default unless you go change it.
I remember two years ago when this was a one-line call: allow Googlebot, block spam bots, move on. Not anymore. In 2026, a dozen-plus AI companies are hitting your site with different intentions, and treating them all the same either locks your product out of every AI answer engine or hands your entire site to a training dataset you never agreed to.
Quick definition first, since it trips people up: an AI bot is any automated program acting on behalf of an AI product, whether that’s a training crawler stockpiling data, a search bot indexing for AI answers, or a live agent (via an LLM) fetching a page mid-conversation.
AI website scraping (pulling page content at scale) is what all three do. What differs is what happens to the content afterward.
The Fast-scan Version
- First of all -permission is signaled, not enforced, by default. robots.txt is a request. Compliant bots (most major ones) honor it; non-compliant ones don’t. A UK study found 72% of tested sites had at least one AI crawler ignore their explicit disallow rules.
- Not all AI crawlers belong in one bucket. Cloudflare splits them into three categories:
A) Training (builds model datasets),
B) Search (powers AI answer engines), and
C) Agent (fetches a page in real time because a user just asked a question).
Your permission decision should differ by category, not treat them as one thing.
- A deadline just changed the default for you. From September 15, 2026, Cloudflare will block Training and Agent crawlers by default on any page carrying ads, catching Googlebot along with everyone else unless you opt out first.
- Robots.txt alone stops almost nobody who’s actually trying to get in. It’s a courtesy signal. Real enforcement means a firewall rule or a cryptographic identity check, not a text file.
- You can charge instead of just blocking. Pay-per-crawl and the emerging x402 micropayment protocol let you say “yes, but pay per visit” instead of a flat yes or no.
What Is an AI Agent Crawler, Actually?
This is worth separating from the AI web crawlers you already know. A traditional search crawler like Googlebot visits your page, indexes it, and moves on, aiming to help a human find that page later in a results list. Some AI web crawlers work the same way, just for a different index. Others are AI agents for scraping and answering in the same motion: there’s no separate index or lookup, just an immediate response that’s extracted from your site.
An AI agent crawler works differently in one specific way. It fetches your page in the moment, as a direct side effect of someone asking an AI assistant a question. A developer asks ChatGPT how your API handles authentication; ChatGPT-User goes and fetches your docs page right then, reads it, and folds it into the answer. The developer never clicks through to your site at all.
A few things follow from that:
The agent also generates no ad impressions and no clicks. It’s reading your page, not a person, so there’s nothing to serve an ad to. And it can visit far more pages per question than a human ever would: research testing commercial AI research tools against roughly 100,000 real queries found AI systems visiting 10 to 60 pages per query on average, with some tools reaching well over 100 pages for a single question. A human asking the same question visits one or two.
IMO the trickiest part is that it behaves like a real browser rather than a declared bot. Many agent products render JavaScript, manage cookies, and generate traffic that looks human to standard analytics tools. That’s exactly what makes permission harder to enforce than the setup sounds like it should be, and it’s the whole reason “just add a robots.txt rule” isn’t a complete answer.
Getting Permission for AI Agent Crawlers
This is the part you actually came for this post, I assume. Controlling crawler access isn’t a single switch. Here’s how site owners realistically grant, restrict, or negotiate access, roughly in order from easiest to set up to strongest enforcement.
- Set up an AI crawler robots.txt or llms.txt file, but treat it as a request, not a lock. Name the crawlers you want to allow or block (GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, etc.) and set Allow or Disallow per path. Every major AI operator has its own user-agent string, so you can be specific: blog open, customer portal closed.
- Use the Content Signals Policy for permission by use, not just by access. This newer robots.txt extension from Cloudflare (2025, free) adds three flags: search=, ai-input=, and ai-train=. It’s the closest thing to conditional permission that exists today, an agent can read your page to answer one question without that page becoming training data.
- Decide by crawler category, not by company. OpenAI alone runs three separately named bots, GPTBot (training), OAI-SearchBot (search), ChatGPT-User (live agent). Treating “OpenAI” as one yes-or-no throws away that distinction for nothing.
- You can’t fully block Google. Googlebot handles both search ranking and AI Overviews. Google-Extended only opts you out of training, not AI-generated summaries. Block Googlebot and you disappear from Google Search too. No clean split exists yet, though the UK’s CMA has been pushing for one since January 2026.
- Back it up with a firewall rule, and let Cloudflare AI Crawl Control handle the busywork. Robots.txt only stops bots that choose to obey it – and the likes of Perplexity have been known to ignore these instructions. But a firewall rule blocked by user-agent or IP before the request even lands, no cooperation required. Cloudflare AI Crawl Control wraps that enforcement with per-crawler monitoring in one dashboard, and it’s genuinely the next step most small teams should take once robots.txt stops being enough.
- Verify identity for agents that hide who they are. Some agents render like a normal browser and never announce themselves, so blocking by name misses them entirely. Web Bot Auth closes that gap by having the agent cryptographically sign its request, so you’re checking real proof instead of a fakeable label. ChatGPT’s agent already does this; the rest are catching up.
- Charge instead of blocking: If an agent sends real value back, like a Perplexity citation landing someone on your pricing page, blocking it outright wastes that. Pay-per-crawl returns an HTTP 402 instead of a flat no, letting you charge per visit rather than picking between free and nothing. Don’t expect this to catch up any time soon though, it’s very new.
How Autonomous AI Agents Are Changing the Threat Landscape
AI training crawlers were manageable because they were honest about it: a declared user-agent string, a scheduled crawl, blockable with a text file and some IP rules.
Agent traffic breaks that model in three ways.
- It’s continuous rather than scheduled: an agent fetches your page the instant someone asks a relevant question, with no crawl campaign to anticipate and block ahead of time.
- It’s browser-shaped rather than bot-shaped, since tools that control a full browser session, rendering JavaScript, managing cookies, produce traffic that’s functionally indistinguishable from a human on standard analytics.
- And it multiplies per question: one live query can trigger dozens of page fetches on your site as the agent gathers context, where a search crawler indexes a page once and reuses that index for thousands of later searches.
Modern AI coding assistants can already inspect project files, run commands and complete multi-step tasks with limited human input. The same agentic model is now expanding to websites, where automated systems can browse pages, gather information and act on a user’s behalf.
Ways to Measure the Non-human Traffic Exposure in Your Site
You can’t set sensible permission rules on traffic you can’t see. Before choosing a block, allow, or charge posture, measure your AI bot traffic: what’s actually hitting your site, from which platform, and how often.
Check your crawl-to-refer ratio per platform first. That’s how many pages a platform crawled for every visitor it actually sent back. Anthropic’s ClaudeBot crawled roughly 38,000 pages for every visitor it referred back as of mid-2025, down sharply from over a quarter-million to one earlier that year, but still lopsided. Google’s ratio, by comparison, has stayed closer to single or low double digits.
If you’re not already on a CDN with built-in bot visibility, pull server logs and filter for known AI user-agent strings. And don’t trust your analytics dashboard alone: standard tools like Google Analytics can’t reliably tell an agent session from a human one, especially when the agent renders JavaScript and manages cookies, so your traffic and conversion numbers may already be inflated by bot sessions you’re not accounting for.
If you handle any regulated data, extend measurement past your marketing pages. Client portals and account pages deserve their own check, since crawler access there can create compliance exposure, not just a content-licensing question.
Why Crawl Purpose Shifted so Fast
The type of crawler hitting your site has changed meaningfully in a year, which is exactly why a one-size permission rule stops working. Here’s the same quarter, one year apart:
| Crawl Purpose | Q2 2025 | Q2 2026 | Change |
| Training | 28.7% | 44.9% | +16.1 pts |
| Mixed purpose | 65.1% | 43.0% | −22.2 pts |
| Search | 4.6% | 9.1% | +4.5 pts |
| User action (agent) | 1.1% | 2.5% | +1.4 pts |
| Undeclared | 0.4% | 0.6% | +0.2 pts |
Training went from a minority use to the single largest declared purpose – so the assumption that AI crawling is mostly harmless indexing is no longer true. Meanwhile, the vague “mixed purpose” bucket, bots that don’t cleanly separate what they’re doing, has collapsed. This means more crawlers are now declaring a specific purpose instead of blending several, and that’s actually good news for you. A bot that tells you plainly it’s here to train a model is one you can make a clean call about.
When You Shouldn’t Give Permission
If your content is your competitive edge, don’t hand it to a competitor’s model for free. In a regulated sector like HealthTech or FinTech, crawler access to a client portal or account page creates compliance exposure no matter what the bot intends.
If you’ve already measured your crawl-to-refer ratio for a platform and it’s wildly one-sided, with no referral traffic or citation value coming back, that’s a data point, not a guess. And if you genuinely can’t tell AI crawler traffic from human traffic on a given page, pause before granting anything. Access you can’t monitor isn’t really a decision. It’s a blind spot you’ve agreed to live with.
The Bottom Line
There’s no single switch labeled “give AI agents permission.” It’s a stack: a text-file signal most reputable bots respect, a use-based flag for what they’re allowed to do with what they collect, a firewall rule for the ones that don’t respect the signal, and, for agents that can fake their identity entirely, a cryptographic check that verifies who’s actually asking. Treat this as an ongoing set of crawling policies rather than a one-time setup. New AI operators show up regularly, and a policy nobody revisits goes stale within months. Decide per crawler category, measure before you decide, and check your Cloudflare (or equivalent) defaults before September 15, 2026. That deadline will make the choice for you if you don’t make it first.
Frequently asked questions
Mainly to feed AI models a steady diet of real content, so they get more accurate, stay current, and can power things like chatbots and summaries. Without crawlers doing this constantly, most AI systems would be stuck working off old, static snapshots of the internet.
A search crawler like Googlebot indexes your page so a person can find it later in a results list. An AI crawler visits the same kind of page for a different reason entirely: to feed a model, not to help someone locate it afterward. Same basic move, reading a page, but the two have almost nothing in common past that.
Whatever’s publicly sitting on your pages: article text, product descriptions, code, images. Nothing exotic, just everything visible, and at a scale no human team could ever match, sometimes across millions of pages.
Depends which one. Older-style training crawlers just grab raw content and leave. Newer AI agents act a lot more like a person browsing: they render JavaScript, hold onto cookies, and can look identical to a human visit in your analytics.
Honestly, it depends, and that’s not a satisfying answer, but it’s the true one. Most are built to stick to public information rather than private data, but problems can still turn up depending on what’s actually sitting on the page they scrape. That’s the whole reason robots.txt, the Content Signals Policy, and firewall rules matter: they’re how you set the line yourself instead of hoping every crawler sets it responsibly for you.
