Cloudflare’s September 15, 2026 AI crawler changes are not just a security update. They affect who can fetch your pages, how AI training bots are handled, and whether your content can stay visible in traditional search while opting out of model training.
Cloudflare says it has deprecated the older “Block AI Bots” approach in favour of more granular Search, Training, and Agent controls. It has also introduced “Disallow AI Training,” a setting designed to let a site refuse AI training while keeping certain mixed-use crawlers available for search.
For website owners, the risk is simple: a setting chosen months ago for content protection may now map to a different crawler policy than you expect. The fix is not panic. It’s a settings review.
What changed on September 15, 2026
Cloudflare now separates AI-related crawling into three categories: Search, Training, and Agent. That replaces the older idea that one broad AI crawler setting can fit every use case.
For new domains, Cloudflare says customers will be offered two preset configurations, depending on whether the site earns money from advertising. Sites that do not monetize with ads start with Preference Sync enabled and Search, Training, and Agent set to Allow. Ad-supported sites start with Preference Sync enabled, Search set to Allow, Training set to Disallow AI Training, and Agent set to Block on pages with ads.
Existing domains are migrated based on current settings. If the legacy Block AI setting was disabled, Cloudflare says Search, Training, and Agent become Allow. If the legacy setting was set to Block or Block on pages with ads, Search becomes Allow, Training becomes Disallow AI Training, and Agent becomes Block on pages with ads.
Cloudflare also says domains that previously configured the granular controls will have the practical effect of their selections preserved. Previous Training selections of Block or Block on pages with ads move to Disallow AI Training.
The new categories matter more than the old toggle
The old “AI bots” label hid too many different behaviours inside one phrase. A crawler used for search discovery, a crawler used for model training, and a live agent fetching a page for a person are not doing the same job.
- Search means crawling to build a search index.
- Training means crawling to train or fine-tune a model.
- Agent means a user-directed agent visiting a page on behalf of a person, such as a chat fetch bot or browser-use agent.
That separation gives site owners more control, but it also adds a new place for mistakes. You may want your pages discoverable in Google, Bing, or Apple search results while refusing model training. You may want AI assistants to fetch product or service pages for live answers. Or you may want to block agent traffic on ad-supported pages because no human sees the ad.
Those are different decisions. They now need different settings.
“Disallow AI Training” is the setting to understand
Cloudflare’s new Disallow AI Training setting publishes a no-training preference through robots.txt via Bot Preference Sync. Cloudflare says Accountable mixed-use crawlers can remain allowed for search under this setting, while other training crawlers are blocked.
The key difference: Disallow AI Training expresses a training preference without fully removing Accountable mixed-use search crawlers. Block removes crawler access.
Cloudflare now says Block and Block on pages with ads apply to all training crawlers, including mixed-use crawlers. Cloudflare names Applebot, Bingbot, and Googlebot as examples. If you select Block for mixed-use crawlers, you may be blocking search crawling too.
Under Disallow AI Training, Cloudflare says Applebot, Bingbot, and Googlebot can keep crawling for search. Selecting Block stops them entirely. One caveat: Cloudflare says Bing does not yet support receiving the no-training preference through robots.txt at the domain level, so that preference is not currently communicated to Bing in the same way.
This is a visibility issue, not only a crawler issue
Traditional SEO still depends on crawlable pages. AI visibility also depends on whether answer engines, assistant fetchers, and retrieval bots can reach useful content when they need it.
Those systems overlap, but they are not identical. Letting one class through does not guarantee rankings, citations, referrals, or AI answers. Blocking one class does not mean every other bot is blocked.
We’ve covered this wider shift in how AI is changing search: the page that wins is no longer just the page that ranks. It also needs to be accessible, useful, structured, and trusted enough for search systems and AI systems to understand. If your CDN blocks the wrong crawler, strong content can still disappear from that path.
Who should check Cloudflare first
This change deserves attention from any site owner using Cloudflare, but a few groups should check sooner.
- Ad-supported blogs and publishers. Cloudflare’s recommended new-domain preset is more restrictive for ad-supported sites because ad revenue depends on a person seeing the page.
- Sites that enabled the old Block AI Bots setting. Cloudflare says legacy settings migrate, but the new labels are different enough that you should confirm what changed.
- Businesses trying to appear in AI answers. If you want AI assistants to fetch product pages, service pages, documentation, or articles, check whether Agent traffic is allowed where it should be.
- SEO teams watching crawl errors. If Google Search Console or Bing Webmaster Tools shows unexpected 4xx responses, CDN-level crawler controls belong on the troubleshooting list.
- WordPress sites managed by hosts, agencies, or security plugins. If someone else configured Cloudflare months ago, the business owner may not know which crawler settings are active.
What to check in your Cloudflare account
A quick audit can prevent a lot of confusion later. Do not rely on memory or old setup notes. Check the live settings.
- Find the AI crawler controls. Look for AI Crawl Control or the bot settings area in Cloudflare. Dashboard labels can change as Cloudflare migrates older features.
- Record your Search, Training, and Agent settings. Capture what each setting currently does, especially if legacy Block AI settings were enabled before September 15, 2026.
- Match settings to business goals. A public service business may want Search and Agent access open. A publisher may want Search allowed, Training disallowed, and Agent traffic restricted on pages with ads.
- Check Bot Preference Sync and robots.txt. Cloudflare says Bot Preference Sync updates robots.txt based on your AI bot configuration while preserving existing directives. Make sure the file reflects what you intend.
- Review WAF custom rules. Cloudflare’s own documentation says AI Crawl Control uses WAF custom rules, and other WAF rules can still affect crawlers you intend to allow.
- Test important pages directly. Compare normal browser requests with relevant crawler-user-agent tests for your homepage, key service pages, product pages, and articles. Treat those tests as diagnostics rather than exact simulations of verified crawler traffic, and review Cloudflare’s AI Crawl Control or Security Analytics to see how requests are actually being identified and handled. Record status codes and whether the returned HTML contains the expected content.
If robots.txt looks correct but a crawler still gets blocked, the issue may be at Cloudflare’s edge rather than your website origin. That is why origin logs alone may not show the full picture.
A simple policy framework
Do not copy a preset without thinking through the business model. The right policy depends on what your site needs from search, AI systems, and human visitors.
- If traditional search visibility matters, keep Search allowed unless you have a specific reason to restrict it.
- If AI assistant visibility matters, decide where Agent traffic should be allowed. Product pages, service pages, documentation, and support content may benefit from live retrieval.
- If you want to avoid model training, consider Disallow AI Training instead of a broad Block when you still want Accountable mixed-use crawlers to reach the site for search.
- If your pages carry ads, decide whether agents should fetch those pages without a human visit. Cloudflare’s ad-supported preset reflects that concern.
- If you have private or sensitive areas, crawler settings are not enough. Use proper authentication, access controls, and security rules.
For most small business websites, this is not a choice between “open to every bot” and “closed to AI.” It is a choice about which types of automated access support the business and which ones do not.
Do not treat robots.txt as the whole answer
Robots.txt is a signal. It tells crawlers what you prefer, and many legitimate crawlers respect it. But Cloudflare points out a limitation: robots.txt alone cannot identify who is crawling, determine why they are crawling, or stop a crawler that ignores the file.
Cloudflare’s edge controls and WAF rules can actually allow or block requests before they reach your server. That makes crawler policy part of infrastructure, not just SEO configuration.
Your audit should include Cloudflare analytics, WAF rules, AI Crawl Control settings, robots.txt, server logs, and search console reports. Looking at only one layer can leave you with the wrong answer.
What Tech Help Canada readers should do now
If you manage a site on Cloudflare, take 20 minutes today to record your current Search, Training, and Agent settings. The goal is not to allow everything or block everything. The goal is to match access rules to your content strategy: search visibility, AI visibility, content protection, and revenue model.
If you are updating content for AI search, start with content quality and structure, then check whether systems can reach it. Our guide to making content easier for AI search systems to understand covers the content side. Cloudflare’s change is the access-control side of the same conversation.
The safest assumption now: AI crawler policy is no longer a background security toggle. It is part of website operations, SEO, and AI visibility. Review it like you would review indexing rules, canonical tags, or firewall changes before they affect traffic.
I can suggest related Tech Help Canada articles based on the topic you’re reading now.
Tech Help Canada Staff researches, writes, and reviews practical content for business owners and professionals. Our coverage spans business, marketing, SEO, technology, and the tools and systems people use to grow and operate online. We focus on clear, useful information backed by research, hands-on experience, and editorial review. Learn more about our team and editorial standards. Need help with something? Contact Us
