TL;DR: On 15 September 2026, Cloudflare changes the defaults for domains newly onboarded to its network. On pages that display ads, bots classified as Training or Agent will be blocked; bots classified as Search stay allowed. GPTBot and ClaudeBot are Training. ChatGPT-User and Claude-User are Agent. If you onboard a client to Cloudflare after that date and their article pages carry ads, the crawlers your AI-search work depends on can be turned away at the edge — with nothing in your dashboard to explain why. Existing zones keep their current settings. This is a five-minute check, and it is much cheaper to run before the client asks.
What actually changes
Cloudflare announced new AI traffic options that sort AI bots into three behaviours:
- Search — crawlers that index content so it can be surfaced and linked later.
- Agent — automated activity acting in real time for a person, such as a chat assistant fetching a page you asked about.
- Training — crawlers collecting content to train or fine-tune a model.
The changelog entry sets the date and the scope: from 15 September 2026, for domains newly onboarding to Cloudflare, Training and Agent are blocked by default on pages that display ads, while Search remains allowed. This applies across plans, including Free.
The reasoning is coherent from Cloudflare’s side. An ad on a page is a signal that the site owner meant a human to land there. Search sends humans back; training and real-time agent fetches generally do not.
Why this lands on AI-SEO work specifically
The crawlers that decide AI-search visibility do not sit neatly in one bucket. They split across all three:
| Crawler | Behaviour | After 15 Sept, on ad pages |
|---|---|---|
| GPTBot | Training | Blocked |
| ClaudeBot | Training | Blocked |
| ChatGPT-User | Agent | Blocked |
| Claude-User | Agent | Blocked |
| OAI-SearchBot | Search | Allowed |
| Claude-SearchBot | Search | Allowed |
| PerplexityBot | Search | Allowed |
And there is a detail worth reading twice: Cloudflare states that multi-purpose crawlers combining Search and Training are affected by the Training block. A crawler that does both does not get the benefit of the doubt.
So a publisher-shaped client — the kind with ads on every article — could end up serving their best content to some AI crawlers and a 403 to others, page by page, depending on which templates carry ad slots.
Who this does and does not affect
Not affected: zones already on Cloudflare before 15 September. Defaults change for new onboardings; existing configurations are not rewritten underneath you.
Affected: any client you move onto Cloudflare from that date, on any plan, whose pages display ads.
Worth checking anyway: clients already on Cloudflare where somebody has since enabled AI bot blocking by hand. That has been a one-click option for a while, and it is enabled far more often than agencies realise — usually by a well-meaning developer who read a headline about AI scraping.
The check, in order
- Fetch the page as the crawler, not as yourself.
curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" -I https://client.example/some-article. A403is the answer. So is a challenge page returning200with no article in it — check the body, not just the status. - Test a page with ads, not the homepage. This is the part that catches people. The rule keys on ad-bearing pages, and homepages frequently have no ad slots. A homepage probe will come back clean while every article is blocked.
- Test more than one crawler. Because the rule discriminates by behaviour, a single-crawler probe tells you almost nothing. Check one Training bot and one Search bot and compare.
- Read robots.txt separately. A
Disallow: /in the wrong group is a different failure with the same symptom, and it is far more common than the Cloudflare default. - If it is the Cloudflare default, it is a setting, not a rebuild. The controls live in the dashboard under AI crawl and bot options. You are changing a policy your client may have opinions about — which is a conversation to have with them, not a decision to make for them.
The part that is actually about you
An agency reselling AI-search work is exposed here in a specific way: the failure is invisible from inside the product. Citations stop climbing, or start slipping. Nothing errors. Nobody gets an alert. Three weeks later the client asks what they are paying for, and the honest answer — “your CDN started turning away the crawlers in September” — sounds like an excuse unless you can show it.
That is the argument for checking crawler reachability as a first-class part of the work rather than as debugging you do after a complaint. If a crawler cannot reach the page, nothing else in AI-SEO matters: the title, the schema, the answer block are all optimisations to a response that never gets delivered.
Key takeaways
- From 15 September 2026, newly onboarded Cloudflare domains block Training and Agent bots by default on pages with ads. Search bots stay allowed.
- GPTBot and ClaudeBot are in the blocked group. So are ChatGPT-User and Claude-User.
- Multi-purpose crawlers that combine Search and Training are caught by the Training block.
- Existing zones are unchanged — this is a default for new onboardings.
- Probe an ad-bearing page, not the homepage, and probe with more than one crawler UA.
- Preferium can fetch any page as an AI crawler and report what came back, so a block is something you can show a client rather than argue about. Continuous monitoring of this is on our roadmap, not shipped — until it lands, the check above is worth running by hand on the clients that matter.
Sources: Cloudflare changelog — New options to manage AI traffic and Your site, your rules: new AI traffic options for all customers.
