Cloudflare's 15 September AI crawler defaults: what changed for Googlebot

Cloudflare’s September 2026 changes made “Disallow AI Training” distinct from blocking the crawler connection. The setting matters: a no-training preference and an access block can have different consequences for search. Check the actual value in each client zone.

Update, 19 September 2026: Earlier versions, published on 25 July, 5 August and 6 September, described the announced defaults. Cloudflare’s 15 September follow-up changed the Training setting. This corrected article replaces those versions; their addresses now lead here. The dated sources are listed below.

Preferences and access blocks

A crawler is a program that visits web pages and reads them on its own, without a human clicking anything. Google runs one called Googlebot, which reads your pages so they can appear in search results. OpenAI, Anthropic, Perplexity and others run their own.

Historically a site owner had one crude lever for these: robots.txt, a plain text file at the root of the site that politely asks named crawlers to stay away. It relies on crawlers respecting the instructions and is not an access-control mechanism.

Cloudflare can enforce an access block. It sits between visitors and the server, so it can refuse the connection, usually with a 403, the HTTP status for a forbidden request. On 1 July 2026 it started sorting AI crawlers by purpose instead of by name.

What Cloudflare announced on 1 July

The announcement split AI crawler behaviour into three groups:

• Search: collecting and indexing content so it can be surfaced and linked later.
• Agent: acting in real time on a specific person’s behalf, such as a chat assistant fetching a page you asked about.
• Training: taking content to train or fine-tune a model.

Each group can be set independently to allow, block, or block only on pages that display ads.

The July changelog entry announced new defaults for 15 September 2026: Training and Agent blocked on ad-bearing pages for new domains, with Search allowed, including on the Free plan. Existing customers could opt out before that date. The Training setting subsequently changed, as explained below.

Why the earlier setting could block search crawlers

The problem was the crawlers that do more than one job. Googlebot builds the search index, and the same fetched content feeds Google’s AI features. Bingbot and Applebot work the same way.

Cloudflare’s July rule was to judge a multi-purpose crawler on all of its behaviours and apply the most restrictive rule that matched. Search Engine Journal worked through the consequence the day after the announcement: block Training, and you block every crawler that also trains.

On 4 August Search Engine Journal reported a case from Reddit. A site owner set AI Training to Block, and Googlebot and Bingbot immediately started getting 403s on the sitemap. Cloudflare’s own dashboard listed both as blocked. Turning the setting off restored access at once. The account’s crawler logs were needed to identify the block.

What actually shipped on 15 September

Cloudflare’s 15 September post changed the Training half of the picture in four ways.

1. A new setting, Disallow AI Training. It publishes a no-training preference in robots.txt instead of refusing the connection. Crawlers Cloudflare classes as Accountable keep crawling for search under it.
2. Googlebot, Bingbot and Applebot are Accountable. So are the relevant crawlers from Amazon, Anthropic, Meta and OpenAI, which run search and training as separate bots. The press release describes Disallow AI Training as letting a site refuse AI training while staying in search results.
3. Existing settings were migrated. A zone that had Training set to Block, or Block on pages with ads, now has Disallow AI Training. A zone that never blocked AI bots now has Training allowed.
4. New domains are offered a preset, not forced into one. Ad-supported sites are offered Search on, Training set to Disallow AI Training, and Agents blocked on pages with ads. Other sites are offered all three allowed. Every setting can be changed during onboarding or at any time afterwards.

Two limits remain relevant to the September announcement:

• Block still means block. Block and Block on pages with ads apply to training crawlers, including mixed-use ones. Setting Training back to Block can therefore block Googlebot again.
• Bing’s training preference was not covered. At the time of the announcement, Disallow AI Training did not convey a no-training preference to Bing through robots.txt; Microsoft support was still pending.

What did not change is the Agent half. Under the ad-supported preset, crawlers acting for a person in real time, such as ChatGPT-User and Claude-User, are still blocked on pages that display ads.

What to check, per client

1. List the client sites using Cloudflare. Check each site’s zone rather than assuming the portfolio shares one configuration.
2. Read each zone’s AI crawler settings as they are now, not as you assume they were inherited. The value that matters most is Training: Disallow AI Training keeps Googlebot, Bingbot and Applebot crawling for search; Block does not.
3. Inspect an ad-bearing page, not only the homepage. Check the response body as well as the HTTP status. A 403 is a block; a challenge page can return 200 without containing the article.
4. Test more than one crawler. The rules discriminate by behaviour, so check one Search, one Training and one Agent user agent and compare.
5. Look for 403s served to verified Googlebot and Bingbot in the server logs, and use Search Console’s URL Inspection on a live page and on the sitemap. Investigate any such block that conflicts with the client’s intended search access.
6. Read robots.txt separately. With Disallow AI Training, Cloudflare now writes to it for you, and a Disallow: / in the wrong group is a different failure with the same symptom.
7. Confirm intent with the client in writing. Training preferences and search access are separate choices. Explain the effect of each setting before changing it, including which user-requested agent visits it may block. Our take on the AI Overviews opt-out applies here too: record the business reason and the expected consequence.

For an initial response check, send a request with a crawler-style user agent:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)" \
  https://client.example/some-article

This changes the user-agent header. It does not reproduce a verified OpenAI request or its network identity. Use verified crawler logs to confirm how real provider requests are handled.

Preferium AI Edge has AI crawler blocking detection: it tells you when a client’s own site is blocking ChatGPT, Claude or Perplexity, and separates a total block from training-only blocking.

Crawler View helps inspect the HTML returned to a crawler-style request; verified provider logs remain important for confirming real access. The Cloudflare setting itself lives in the client’s account and stays their decision. How crawler and agent visits are read.

Sources, read on 19 September 2026: Cloudflare, Have it both ways: stay discoverable in search while disallowing AI training (15 September 2026); Cloudflare, Cloudflare Helps End the Search-or-AI-Training Tradeoff (15 September 2026); Cloudflare changelog, New options to manage AI traffic (1 July 2026); Cloudflare, Your site, your rules: new AI traffic options for all customers (1 July 2026).