How to Configure Cloudflare's New AI Crawler Categories to Protect Proprietary Business Data

How do I configure Cloudflare’s new AI crawler categories?

Cloudflare now classifies all AI bot traffic into three behavior-based categories — Search, Agent, and Training — and lets you independently control each one. For every category you can choose Allow, Block on pages that display ads, or Block on all pages (Source: Cloudflare). These controls are available to all Cloudflare customers, including those on the Free plan (Source: Cloudflare). You configure them in the Cloudflare dashboard under Security Settings > Configure AI bot policies.

That is the short answer. The longer answer involves what these controls actually protect, what they do not protect, why the September 15, 2026 defaults matter even if you already use Cloudflare, and how to think about the trade-off between AI-search visibility and content protection. This article walks through each of those, grounded in Cloudflare’s official documentation and RFC 9309 (the Robots Exclusion Protocol standard).

One thing to state clearly at the start: these controls govern public website pages served through Cloudflare. They do not protect private databases, CRMs, authentication-protected applications, private APIs, or data that is never publicly served on a web page. Those require authentication, authorization, and application-level security — not crawler category toggles.

The three categories: Search, Agent, and Training

Rather than relying on a single “AI bot” label, Cloudflare classifies bots by behavior — what a bot does on your site (Source: Cloudflare). This matters because not all AI traffic has the same effect on your business. Cloudflare lets you manage three AI-related behaviors directly:

Each blocking option blocks Verified bots classified with that behavior, plus additional unverified bots that fall under these classifications (Source: Cloudflare). A single bot can have more than one behavior (Source: Cloudflare), which is why the mixed-purpose category matters — and why blocking Training has implications beyond standalone training crawlers.

What these controls do and do not protect

Cloudflare’s AI crawler category controls are access-policy controls for publicly served web content. They determine whether identified AI bots can reach pages that Cloudflare serves on your behalf. They do not:

• Protect private data that is already behind authentication. Login-gated dashboards, client portals, member-only areas, admin panels, and database-backed applications are not accessible to web crawlers regardless of your category settings. Authentication and authorization already prevent this.
• Protect private APIs. API endpoints that require authentication tokens are not part of the public web and are not affected by crawler category policies.
• Protect data that was never publicly served. Databases, CRMs, file stores, internal systems — none of these are reached by web crawlers unless a public page exposes them.
• Replace application security. Crawler controls are a web-access layer, not a substitute for proper authentication, authorization, data minimization, input validation, or encryption.

The practical implication is that if proprietary business data is currently on a public page, blocking AI crawlers reduces its exposure to AI training and AI-search answers — but it does not make the data private. If the data is genuinely proprietary, the correct response is to move it behind authentication, where crawler settings become irrelevant.

Detection quality: free plan versus paid plans

Cloudflare’s free plan and paid plans differ significantly in how thoroughly they identify AI crawlers. The Cloudflare documentation states this directly: “On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings. This enables AI Crawl Control to detect well-known, self-identifying AI crawlers” (Source: Cloudflare). Upgrade your plan to enable a more thorough detection using Cloudflare’s Bot Management detection ID field (Source: Cloudflare).

In practice this means:

• Free plan: catches crawlers that accurately identify themselves in their User-Agent header. Crawlers that self-identify as AI bots — GPTBot, ClaudeBot, Google-Extended, and similar — are detected. Crawlers that do not self-identify or that spoof their user-agent may pass through.
• Paid plans with Bot Management: use a deeper detection mechanism that identifies bot behavior patterns, not just self-declared user-agent strings. This catches a wider range of AI crawlers, including those that do not accurately self-identify.

For a small business deciding which tier to use: if blocking AI crawlers is a priority and you want coverage beyond the self-identifying subset, the paid tier provides more comprehensive detection. If your concern is primarily about large, well-known AI models absorbing your content, the free plan covers the most recognizable crawlers.

robots.txt is a directive, not access control

The robots.txt protocol has always been a voluntary convention, not a security mechanism. RFC 9309, the IETF standard that formalizes the Robots Exclusion Protocol, states explicitly: “These rules are not a form of access authorization” (Source: IETF RFC 9309). The standard’s Security Considerations section goes further: “The Robots Exclusion Protocol is not a substitute for valid content security measures. Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable” (Source: IETF RFC 9309).

Cloudflare’s own documentation reinforces this. When you turn on the managed robots.txt setting, Cloudflare generates and maintains a robots.txt file that instructs known AI crawlers to stay away from your content — but “robots.txt compliance is voluntary. The file expresses your preferences, but it does not prevent crawlers from accessing your content at a technical level” (Source: Cloudflare). Some crawler operators may disregard your robots.txt directives and crawl your content regardless (Source: Cloudflare).

For actual enforcement, Cloudflare recommends AI Crawl Control: “If you want to enforce crawl blocking rather than request it, use AI Crawl Control” (Source: Cloudflare). When you block a crawler in AI Crawl Control, the system creates or updates a WAF custom rule on your zone to enforce that block (Source: Cloudflare).

The best practice is to use both together: robots.txt to express your preferences to the broader internet of compliant crawlers, and AI Crawl Control to enforce those preferences at the network edge via WAF rules. Cloudflare’s managed robots.txt is available on all plans and blocks eight named AI crawlers by default: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent (Source: Cloudflare).

The September 15, 2026 defaults — precisely

On September 15, 2026, Cloudflare will set updated defaults for new domains: bots classified as Training or as Agent will be blocked on pages that display ads, and Search will remain allowed (Source: Cloudflare). Three details matter here.

New domains only. The defaults apply to domains newly onboarding to Cloudflare after September 15. Existing domains are not automatically reconfigured. If you already have a Cloudflare account, your current settings remain unchanged unless you modify them.

Ad-supported pages. The default block applies specifically to pages that display ads. Cloudflare uses automated detection to identify ad-serving pages on your zone. Pages without ads are not affected by the default Training and Agent block — you would need to set these manually if you want them blocked on non-ad pages too.

Mixed-purpose crawlers. Before September 15, Cloudflare’s legacy “Block AI bots” option explicitly excluded mixed-purpose bots — crawlers used both for Training and for Search. After September 15, mixed-purpose crawlers that combine Search and Training will also be blocked by all configurations to block AI training, including the legacy “Block AI bots” option (Source: Cloudflare). This means blocking Training may also block the Search function of bots that serve both purposes. The legacy “Block AI bots” option itself is deprecating on September 15, 2026.

All customers can opt out of these new defaults at any time before September 15 (Source: Cloudflare).

Trade-offs: AI search visibility, citations, and lead generation

The practical trade-off at the heart of these controls is between AI-search exposure and content protection. Cloudflare frames this directly: Search covers crawlers that index your content so they can answer questions about it later, where you should expect referral traffic or other equitable compensation in return (Source: Cloudflare).

Allowing Search crawlers means your public content can appear in AI-generated answers, cited by AI assistants, and surfaced as referrals from AI search interfaces. This is meaningful for businesses whose lead generation depends on being found online — consultants, service providers, publishers, e-commerce stores.

Blocking Search crawlers means you sacrifice that potential visibility. For a business whose public content is primarily proprietary methodologies, unique frameworks, or IP-sensitive material, the trade-off may be worth it. For a business that depends on discoverability, it may not.

One important caveat: Cloudflare’s documentation states that Search crawlers generate referral traffic, but no public data quantifies how much traffic AI-search interfaces currently send compared to traditional search engines. The referral benefit is directional, not measured. Businesses making the Allow/Block decision for Search should treat AI-search visibility as a potential channel rather than a proven one, and weigh it against the specific sensitivity of their public content.

Training crawlers present a simpler calculation: they absorb your content into model training with no direct return. Unlike Search crawlers, Training crawlers do not promise citations, referrals, or traffic. Blocking Training is a low-cost protection that most businesses should consider.

Agent traffic occupies a middle ground. A user asking their AI assistant “Find me a cloud security consultant near me” may trigger an Agent crawler that fetches your product pages. Blocking Agent traffic might prevent legitimate user-initiated actions, depending on your business model.

Worked example: a consulting firm’s public website

Consider a five-person consulting firm with a marketing website, a public blog, a lead-generation form, and a gated client resource library behind login. Here is how the three categories apply:

Public marketing pages and blog — these are served through Cloudflare and are within scope of AI crawler controls. The firm’s methodology descriptions, case study summaries, and blog posts are all publicly accessible.

Gated client resource library — behind authentication. Not affected by crawler settings regardless of configuration. This content is already protected by login requirements.

Practical configuration posture:

Additionally: enable managed robots.txt to signal compliance to the broader crawler ecosystem, configure AI Crawl Control to enforce blocks via WAF rules for the Training category, and review the Crawlers tab monthly to observe which bots are requesting access and adjust accordingly.

This posture protects the firm’s training-relevant content while preserving search visibility and allowing legitimate agent-mediated discovery. The gated content is already protected by authentication and is unaffected by any of these settings.

A practical decision framework

For any small business working through these settings, the following sequence turns Cloudflare’s three categories into actionable configuration:

1. Scope. Identify which pages are public (within scope of crawler controls) and which are already authenticated or private (already protected, outside scope). Only public pages need attention here.

2. Search decision — Allow or Block? Allow if your business depends on online discoverability and your public content is not IP-sensitive. Block if your public pages contain proprietary methodologies, unique frameworks, or commercially sensitive information that AI-search surfacing could expose to competitors.

3. Training decision — Block for most businesses. Training crawlers absorb content with no direct return. The cost of blocking them is typically low — Training crawlers do not generate citations or referrals — and the benefit of limiting content absorption into training models is clear. Set to Block on all pages unless you have a specific licensing agreement that depends on training access.

4. Agent decision — Context-dependent. If your business benefits from users delegating tasks to AI assistants (product research, service comparison, information retrieval), Allow makes sense. If you see Agent traffic as primarily extractive or disruptive, Block. Monitor actual Agent traffic in the AI Crawl Control dashboard before deciding.

5. Detection tier. If you are on the Free plan and accept that detection is limited to self-identifying crawlers, proceed. If you need more comprehensive detection, evaluate a paid plan with Bot Management.

6. Layer robots.txt with enforcement. Enable managed robots.txt for protocol-level signaling, then configure AI Crawl Control to enforce your category choices via WAF rules. Using both together provides signaling to compliant crawlers and active blocking at the network level.

7. Review periodically. Check the Crawlers tab in AI Crawl Control for request volumes, trends, and robots.txt violations after your initial configuration. Adjust as patterns emerge.

What these controls leave uncertain

Several details relevant to configuration decisions are not publicly documented by Cloudflare as of July 2026:

• No exhaustive published list identifies which specific bots are classified as Search, Agent, or Training. The Crawlers tab in AI Crawl Control shows classifications for observed bots, but Cloudflare has not published a complete roster.
• The exact method used to detect “pages that display ads” — the trigger for the September 15 default block on ad-supported pages — is not described in public documentation. Cloudflare states it uses automated detection but does not detail the mechanism.
• Which specific bots Cloudflare treats as mixed-purpose is not enumerated. The documentation states the rule (Training controls apply to mixed-purpose crawlers) without listing the affected bots.
• Free-plan user-agent detection covers self-identifying crawlers only. The coverage gap between Free and paid Bot Management detection is not quantified.
• No public data measures how much referral traffic AI-search interfaces currently generate compared to traditional search engines. Cloudflare frames Search crawlers as generating referrals, but the magnitude is not established.

These gaps do not invalidate the configuration categories themselves. They mean that some decisions — particularly the Search Allow/Block trade-off — rest on directional reasoning rather than measured outcomes.

What not to rely on

Three common misconceptions deserve explicit correction:

robots.txt alone is not access control. As established in RFC 9309 and confirmed by Cloudflare, robots.txt is a protocol directive. Compliant crawlers honor it; non-compliant ones ignore it. Enforcement requires AI Crawl Control or equivalent WAF rules.

Free-plan detection misses non-self-identifying crawlers. If you are on the Free plan and rely solely on AI crawler category controls, you are covered for well-known, self-identifying bots. Crawlers that do not accurately declare themselves in their user-agent string may not be caught.

Crawler controls do not replace application security. Authentication, authorization, data minimization, encryption, and proper application-layer security protect your systems regardless of crawler behavior. If proprietary data is currently on a public page, the correct first action is to assess whether it belongs there — crawler controls are a second-layer consideration for content that is intentionally public.

When these controls do not apply

This article addresses AI crawler configuration for public websites served through Cloudflare. The category controls are not relevant when:

• All content is already behind authentication. If your site is a SaaS application, an internal tool, or a member-only portal with no public pages, AI crawler category settings have nothing to act on.
• You do not use Cloudflare. These controls are specific to Cloudflare’s infrastructure. Businesses using other CDN/security vendors have different (or no) equivalent category controls.
• Your public content has no AI-training value. A static informational page with generic copy and no proprietary information is unlikely to attract training crawlers regardless of settings.

Where to go from here

The configuration steps above translate directly into two practical follow-up resources: the Cloudflare AI crawler configuration checklist for the step-by-step dashboard actions, and the decision guide on which AI crawlers a small business should allow for the business reasoning behind each choice.

If the task of auditing your public pages, configuring Cloudflare settings, and monitoring AI crawler activity is adding to your administrative workload, Hallermann Consulting helps small businesses simplify repeatable website configuration and security tasks. A workflow audit can identify which parts of your website security process benefit from structured processes, and a technical care plan covers ongoing monitoring and configuration reviews.