Cloudflare AI Crawler Configuration Checklist

What steps configure Cloudflare AI crawler categories?

This checklist turns Cloudflare’s three-category system into a seven-step process you can run through once and repeat quarterly. Each step maps to a specific screen in the Cloudflare dashboard.

For the reasoning behind each step — including what these controls protect and what they do not — see the main article on how to configure Cloudflare’s AI crawler categories.

Step 1: Separate public from authenticated pages

AI crawler category settings apply only to pages that Cloudflare serves publicly. Login-gated dashboards, client portals, member-only areas, admin panels, and private APIs are already protected by authentication and are unaffected regardless of your settings.

Make a short list:

• Public pages — marketing site, blog, product pages, resource pages, documentation. These are in scope.
• Authenticated or gated pages — member areas, admin panels, client portals. These are already protected.

You only need to worry about the public pages.

Step 2: Open the AI bot policy page

Log in to the Cloudflare dashboard, select your account and domain, and go to Security Settings > Configure AI bot policies (Source: Cloudflare). This screen presents the three categories — Search, Agent, Training — each with a dropdown for your chosen action.

Step 3: Set each category

Each category offers three actions: Block on all pages, Block on pages with ads, or Allow (Source: Cloudflare). A practical starting point for most small businesses:

These are starting points. The decision guide on which AI crawlers a small business should allow covers the business reasoning in more detail.

Step 4: Enable managed robots.txt

Go to Security Settings > Bot traffic and enable “Set your preference to block training in robots.txt” (Source: Cloudflare). This feature is available on all plans.

What this does: Cloudflare prepends managed rules to your robots.txt (or creates one if you don’t have one), signaling known AI crawlers — Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot, and meta-externalagent — to stay away. This is protocol-level signaling, not enforcement.

Step 5: Review AI Crawl Control for per-crawler detail

Navigate to AI Crawl Control > Crawlers. This table shows every AI crawler requesting access to your site: the crawler name and operator, its category, request volumes with trend charts, and robots.txt violations (Source: Cloudflare).

Use the filters (Name, Operator, Category) to narrow the view. Pay attention to the robots.txt violations column — crawlers that violate your robots.txt directives are candidates for active blocking.

Step 6: Set per-crawler enforcement actions

For any crawler you want to explicitly allow or block, use the Actions column in the Crawlers table:

• Allow signals that you permit this crawler to access your content. You can still enforce your robots.txt directives for it.
• Block actively prevents access. Cloudflare creates or updates a WAF custom rule to enforce this block at the network edge (Source: Cloudflare).

Using both robots.txt and AI Crawl Control together — robots.txt for compliant crawlers, AI Crawl Control for active blocking — is the recommended layered approach (Source: Cloudflare).

Step 7: Revisit quarterly

AI crawler behavior evolves. New crawlers appear, operators change their policies, and your content strategy may shift. After your initial configuration month, schedule a quarterly review to:

• Check the Crawlers tab for new bots requesting access.
• Adjust category settings if your content or business model has changed.
• Verify that the mixed-purpose crawler impact aligns with your preferences, especially as the September 15, 2026 defaults take effect.
• Review robots.txt violations to catch non-compliant crawlers.

What this checklist does not cover

This checklist covers dashboard configuration for public websites served through Cloudflare. It does not address:

• Sites where all content is behind authentication (crawler settings have nothing to act on there).
• Sites not hosted on Cloudflare (these controls are Cloudflare-specific).
• Custom WAF rules beyond what AI Crawl Control generates automatically.
• Pay per crawl (currently in closed beta for Enterprise customers).

For those scenarios, see the main article’s discussion of detection tiers, WAF integration, and when crawler controls do not apply.

Where to go next

The reasoning behind each setting lives in the decision guide on which AI crawlers a small business should allow. Read that first if you are unsure which starting settings make sense for your business, then run this checklist to implement them.

If ongoing website configuration and security monitoring is adding to your administrative workload, Hallermann Consulting helps small businesses streamline repeatable security tasks. A workflow audit or technical care plan can identify which parts of your Cloudflare and security configuration benefit from structured, repeatable processes.