Is Cloudflare blocking the crawlers you actually want?

Start by proving where access stops.

Cloudflare may be part of the problem. It may also be doing exactly what you asked it to do.

Before you change a firewall rule, separate four questions:

A normal browser view proves that a person using that browser received the page. It does not prove Googlebot, Bingbot, OAI-SearchBot or another named crawler received the same response.

Google’s technical requirements make the boundary clear. Googlebot access, an HTTP 200 response and indexable content make a page eligible for indexing. They do not guarantee that Google will index it.

The first job is smaller than “fix AI visibility.” Find the first place where a named, legitimate request stops receiving the public page.

Identify the crawler and its purpose first.

Do not begin with “allow AI bots.” Begin with what you want the platform to do.

A crawler may collect pages for search. A user-triggered fetcher may open a page because someone asked a question. A training crawler may collect material for model development. Those are different uses, and the operators publish different controls for them.

The table below reflects primary platform documentation checked on 2026-08-21. Bot names, IP ranges and product rules can change. Recheck the linked source before changing a live firewall.

OpenAI’s crawler documentation explicitly says its settings are independent. Anthropic’s crawler documentation also separates training, search and user-directed retrieval. Perplexity publishes a crawler and WAF guide for its two named agents.

This is why one broad “AI bot” toggle cannot answer every access question.

Read the robots file that crawlers actually receive.

Open the exact live hostname at /robots.txt. Save the response with the UTC date, HTTP status and response body.

Do not rely only on the file in your CMS or origin server. Cloudflare’s managed robots setting can prepend managed instructions to an existing origin file or create a response when the origin has no file. The crawler acts on the public response it receives.

For the named crawler, record:

• the matching User-agent group;
• every applicable Allow and Disallow rule;
• whether a general group and a specific group conflict;
• whether Cloudflare’s managed setting is on;
• whether the file is different across the apex domain, www host or another subdomain;
• whether the page also sends noindex in HTML or an X-Robots-Tag header.

Robots is not a password. Google’s robots.txt guidance says it manages crawler access and load; it is not a secure way to hide a page. Different crawlers may also support or interpret instructions differently.

There is another common trap. A crawler usually needs to receive a page to read its noindex instruction. Blocking the URL in robots while expecting the platform to process page-level noindex can prevent that instruction from being seen.

Use authentication for material that must remain private. Use robots and platform-specific controls for public content-use preferences.

Check every Cloudflare control that can act.

Cloudflare is not one switch. Several products can inspect or change the same request.

Start with AI Crawl Control if it is available for the proxied zone. Its crawler table can show activity and the chosen action. Cloudflare warns that an “unsuccessful request” may come from another rule or response error, not only an AI Crawl Control block.

Then inspect WAF, bot, rate-limit and Access evidence for the same time and path. Cloudflare’s product-order documentation says AI Crawl Control crawler blocks run through WAF custom rules before Cloudflare bot solutions. That means changing a later bot setting may not repair an earlier block.

Cloudflare Access is different again. It is an identity-aware boundary for protected applications. If a private portal requires authorization, an unauthenticated crawler should not receive it. The right decision may be to keep that route closed and make a separate public information page clear and accessible.

Do not switch off Cloudflare for the whole site to see whether the problem disappears. That removes useful protection and gives you poor evidence about which control acted.

Compare the status, body, headers and rendered page.

An HTTP status is important, but it is not the whole response.

Google’s CDN crawling guidance distinguishes hard errors from soft blocks such as a human-verification page. A crawler receiving an HTTP 200 challenge has not received the same useful page as a normal visitor.

Save the redirect chain, response body marker, content type, robots headers and CF-Ray. In Cloudflare, compare the edge response with the origin response. If the edge returned 403 and the origin never received the request, investigate the edge rule. If both returned 200 but the rendered page is empty, investigate rendering and resource access.

Cloudflare Security Events helps identify acted-on requests, but it may use sampled data and one HTTP request can create several events. An absent event is not proof that no request occurred. Security Analytics or request logs may provide a broader view, depending on the plan.

For Google, inspect the tested page and rendered view in Search Console. Confirm that the main content appears—not a blank page, login, block message or challenge.

Verify identity without trusting the bot name.

Anyone can put Googlebot, Bingbot or an AI crawler name in a user-agent header. Do not create a security exception from that string alone.

Use the verification method the operator currently supports:

• Google documents reverse-and-forward DNS checks and publishes IP data for different crawler and fetcher classes.
• Bing provides a Verify Bingbot tool and warns that user-agent strings are easy to spoof.
• OpenAI publishes separate current IP files for OAI-SearchBot, GPTBot and ChatGPT-User.
• Anthropic publishes crawler IP information and says the list identifies Anthropic-origin requests.
• Perplexity recommends combining the named user agent with its current official IP sources in a WAF rule.

Cloudflare’s verified-bot documentation adds another useful signal, but still record the behavior or purpose you intend to allow.

Any exception should be narrow: verified identity, intended hostname, necessary public paths, expected request methods and the purpose you accept. Preserve rate limiting, WAF protection and authentication for everything outside that scope.

Test Google, Bing and each AI platform separately.

There is no single test for “all search and AI.” Use the platform’s own evidence when it provides one.

Google documents how to verify Google requests. It separates common crawlers, special crawlers and user-triggered fetchers because their roles and robots behavior can differ.

Bing’s crawler inventory identifies Bingbot as its main crawler and points owners to verification. Bing also explains that crawl access is not the only reason a page may be absent from its index.

For AI products, test the named crawler or fetcher the operator documents. Do not infer that passing OAI-SearchBot proves Claude-SearchBot access, or that allowing a training bot is required for search visibility.

Before changing crawler policy, use the robots.txt decision guide to separate public discovery, search use and training preferences. After access passes, the work moves to a different question: can the platform understand, select and use the page? The AI search mechanics guide explains that next layer. Mindflow’s AI Visibility service covers the broader commercial work.

Keep private information private.

Visibility work applies to public pages. It is not a reason to expose protected data.

Keep authentication and authorization on:

• admin and staging systems;
• patient portals and health records;
• legal client files and privileged material;
• SaaS customer workspaces, account data and internal documentation;
• eCommerce order histories, payment areas and customer accounts;
• home-service dispatch, customer records and internal operating systems.

Do not publish raw security logs in a report. They may contain IP addresses, private URLs, query strings, account identifiers or rule details. Preserve the evidence privately and publish only the conclusion needed for the decision.

If a commercial topic must be discoverable, create or improve a separate public page. A healthcare practice can publish a service explanation without exposing a patient portal. A SaaS company can publish product documentation without opening a customer’s workspace.

Cloudflare’s Access documentation describes Access as an identity-aware proxy. Treat that boundary as intentional unless the exact public path was included by mistake.

The public/private line changes by industry.

The diagnostic method stays consistent. The pages and protected records change.

Equal attention does not mean using the same allow rule everywhere. It means protecting the public discovery path and the private operating boundary in each industry.

Healthcare and legal teams may need additional privacy, security and professional review. That review should narrow the publishable surface, not turn robots.txt into a substitute for proper access control.

End with a dated answer and one safe next action.

Do not end the check with “looks fine.” Use a record another person can inspect.

If the evidence shows a block, change the smallest safe rule and run the same test again. If access passes, stop changing Cloudflare and investigate the next layer. If the evidence is incomplete, label the result unknown instead of guessing.

The broader technical SEO guide helps decide whether this is one contained fix or part of a larger problem. Mindflow’s Organic SEO service covers the full discovery path.

If you do not know where the response changes, request a Free Visibility Check. The check reviews a limited public sample and identifies the first visible issue worth investigating. It is not a full security audit, and it cannot promise indexing or appearance in an AI answer.

Primary sources

• Cloudflare AI Crawl Control
• Cloudflare managed robots.txt
• Cloudflare Security Events
• Google Search technical requirements
• Google guidance on CDNs and crawling
• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Bing crawler inventory