Cloudflare changes its AI crawler defaults on 15 September, and Googlebot is in scope

TL;DR: On 15 September, Cloudflare — the company that sits in front of a very large share of the world’s websites and decides which automated visitors get through — switches on a new default. Software that reads websites to train AI models, or to act on a user’s behalf, gets turned away from any page that shows ads. Software that reads websites to build a search index still gets in. That sounds tidy until you notice that Google’s reader does both jobs with one program. Cloudflare’s rule is to apply the strictest setting that matches. So a site owner who ticks “block AI training” to keep their content out of AI models can also, without being warned, shut Google Search out of their own site. That is not a hypothetical. In August someone did it and watched Google start getting refused at the door.

First, what we’re actually talking about

A crawler is a program that visits web pages and reads them, on its own, without a human clicking anything. Google runs one called Googlebot. It reads your pages so they can appear in search results. OpenAI, Anthropic, Perplexity and others run their own.

Historically a site owner had one crude lever for these: robots.txt, a plain text file at the root of the site that politely asks named crawlers to stay away. Polite is the operative word. It has no teeth, and it works at the level of “this crawler, yes or no”. It cannot say “read my site for search, but not for training”.

Cloudflare can. It sits between visitors and the server, so it doesn’t ask — it refuses the connection, usually with a 403, the HTTP code meaning “I know who you are and you’re not coming in”. And a year ago Cloudflare started sorting crawlers by purpose instead of by name.

What changes on 15 September

In its 1 July announcement, Cloudflare split AI crawler behaviour into three buckets:

• Search — collecting and indexing your content so it can answer questions about it later
• Agent — acting in real time on a specific person’s behalf, like a shopping assistant fetching a price
• Training — taking your content to train or fine-tune a model

From 15 September, domains coming onto Cloudflare get new defaults: Training and Agent blocked on pages that display ads, Search still allowed. Reporting on the announcement described existing free-tier sites that have never touched the setting being moved onto those defaults as well. Cloudflare’s own post tells existing customers they can set their preference in Security settings any time before the date.

The reasoning is straightforward. On a page that earns money from ads, a human reading it is the point, and a bot reading it isn’t. AI companies that want that content anyway are pointed at Cloudflare’s paid-access route rather than at the front door.

The part that catches Google

Crawlers don’t respect these categories. Googlebot is one program doing several jobs: it builds the search index, and the same fetched content feeds Google’s AI features.

Cloudflare’s stated approach is to judge a multi-purpose crawler on all of its behaviours, and to apply the most restrictive rule that applies. Search Engine Journal worked through the consequence the day after the announcement: if you block Training, you block every crawler that also trains. Googlebot trains. So does Bingbot. So does Applebot.

You wanted to keep your articles out of a language model. You also removed yourself from Google.

Someone already did this

On 4 August, Search Engine Journal reported a case from Reddit. A site owner set AI Training to Block. Googlebot and Bingbot immediately started getting 403s when they came for the sitemap — the file that tells a search engine which pages exist. Cloudflare’s own dashboard listed both as blocked. Turning the setting off restored access straight away.

Someone in the thread suggested the bots were fake, which is the usual explanation for a blocked Googlebot. The poster said they weren’t. Google’s John Mueller asked them to get in touch so the team could look at it.

Note what that failure looks like from the inside: nothing. No warning, no email, no red banner. Sitemap fetches fail quietly, new pages stop getting picked up, and the drop in traffic arrives weeks later with no obvious cause. By then nobody remembers a checkbox.

What to do this week

1. Open the Cloudflare dashboard for every site you run and look at Security → the AI bot policy settings. Read what Search, Agent and Training are each set to. If a site is on a free plan and nobody has touched this, it is exactly the case the new defaults apply to.
2. Decide the three categories separately. Blocking Training while allowing Search is a reasonable position. Getting there requires setting it deliberately, not blocking Training and hoping.
3. Check server logs for 403s served to Googlebot and Bingbot. This is the only reliable evidence. A 403 to a verified search crawler is never normal.
4. Use Search Console’s URL Inspection on a live page and on your sitemap. A fetch failure there confirms the problem from Google’s side rather than yours.
5. Do it for the whole portfolio, not one site. Agencies rarely have one Cloudflare account, and the default only applies where nobody has changed anything.

Steps 3 to 5 are where this gets expensive. Reading access logs for one site is twenty minutes; doing it across forty client sites, on a deadline, is a week nobody has. Preferium crawls every page of a site with 47 automated checks and scores it out of 1000, so a page returning 403 to a search crawler surfaces as a finding rather than as a traffic dip in October. After it deploys a fix, a real browser loads the live page to confirm the change actually took. The setting itself lives in your Cloudflare account and stays your decision — what the system removes is the part where nobody notices for six weeks. More on how the system works, or run a free AI scan against a single site first.

Key takeaways

• From 15 September 2026, Cloudflare blocks AI Training and Agent crawlers by default on ad-supported pages for domains newly onboarding; Search stays allowed.
• Multi-purpose crawlers are judged on all their behaviours under the strictest matching rule. Googlebot, Bingbot and Applebot all crawl for both search and training.
• Blocking Training can therefore block Google Search. A site owner hit this in August and Googlebot got 403s on the sitemap.
• The failure is silent. No warning is sent, and the traffic loss shows up weeks after the setting was changed.
• Before 15 September: check the AI bot settings on every domain, set the three categories individually, and look for 403s served to verified search crawlers in your logs.