Decide what should be public before you write a rule
A robots.txt file is a public request to crawlers. It can say which paths a named crawler may request. It does not lock a page, protect a customer record or prove that only the right person can open a file.
That distinction comes from the Robots Exclusion Protocol. The standard says the rules are not access authorization. It also warns that listing a path can make the path easier to discover.
Start with a simpler question: Should a person be able to open this page without signing in?
- A service page, attorney profile, product page or software guide is normally public.
- A customer account, patient portal, client file, tenant workspace or order record is private.
- A staging site, admin area or internal report should not become public because a crawler behaves well.
Private content needs authentication and authorization. Use robots.txt for public-crawl choices and basic crawler hygiene, not for privacy.
This page stays focused on the access decision. The AI Search Guide explains the wider visibility system, and How AI Search Works follows a source from discovery through retrieval and answer building.
Crawling, indexing, retrieval and training are four decisions
The word “access” hides several different jobs.
| Decision | Plain-English question | Typical control | What the control cannot prove |
|---|---|---|---|
| Automatic crawling | May this named bot request these public paths on its own schedule? | robots.txt, plus CDN/WAF rules where needed | That the page will be indexed or used |
| Indexing and serving | May a search engine keep and show this page or its snippet? | Robots meta tag or X-Robots-Tag, where the platform supports it | That the page will rank or receive a click |
| User-requested retrieval | May an assistant fetch a public URL because a user asked? | Provider-specific behavior; sometimes robots, sometimes a separate access rule | That the answer will use or cite the page |
| Model training | May a provider collect public content for stated training uses? | A provider-specific crawler token or page rule | That blocking training changes search visibility |
The same company can document a different agent for each job. OpenAI separates its search crawler, training crawler and user-action agent. Anthropic documents three similar roles. Perplexity separates search crawling from user-requested retrieval. Google separates Google Search crawling, the Google-Extended product token and user-triggered fetchers.
Do not translate that into one universal rule. Some user-triggered fetchers generally ignore robots.txt. Others say they honor it. Some providers publish a training crawler. Microsoft currently documents page-level controls for Bing and Copilot rather than a separate training crawler in this source set.
An allow decision means the named system is eligible to request the public page under that control. It does not mean the system will crawl, index, cite or recommend it.
Start with a public-content access matrix
Do not begin with a downloaded list of bot names. Begin with your content classes and the business reason for keeping each one public or private.
| Content class | Public to people? | Search crawl and index | Answer-search crawl | User-requested retrieval | Training collection | Real security control | Decision |
|---|---|---|---|---|---|---|---|
| Public commercial page | Yes | Usually allow when the page should be found | Decide by named provider purpose | Decide with the provider’s stated behavior in mind | Separate owner decision | Normal WAF and abuse controls | PUBLIC SEARCH |
| Public research or reference | Yes | Usually allow | Usually allow when the source should be found | Usually allow | Separate rights and policy decision | Normal WAF and abuse controls | PUBLIC REFERENCE |
| Public page that should not appear in search | Yes | Allow the search crawler to read the page-level noindex | Provider-specific | Provider-specific | Separate decision | Normal WAF and abuse controls | PUBLIC, NOT INDEXED |
| Customer, patient, client or tenant portal | No | Not a robots decision | Not a robots decision | Not a robots decision | Not a robots decision | Authentication and authorization | PRIVATE |
| Staging, admin or internal tool | No | A robots block can reduce polite crawling only | Same | Same | Same | Authentication, network and application controls | PRIVATE / ADMIN |
| Page with unknown live behavior | Unknown | Test | Test | Test | Test | Inspect edge, origin and logs | INVESTIGATE |
Keep a short record beside the matrix:
- who owns the policy;
- the date of the decision;
- the live
robots.txtURL; - the CDN or security provider;
- the server or application that owns the underlying file;
- the named agents tested;
- the status and body each request received;
- the next review date.
This record prevents a later developer, plugin or CDN toggle from silently reversing the reason behind the file.
Provider names matter because their purposes differ
This matrix reflects the providers’ own documentation checked on 21 August 2026. Provider names, product roles and IP ranges can change. Recheck the linked source before changing production access.
| Provider control | Documented purpose | What the owner can decide | Important boundary |
|---|---|---|---|
Googlebot | Automatic crawling for Google Search | Allow or disallow paths for ordinary Search crawling | Access does not guarantee indexing, ranking or an AI Overview source link |
Google-Extended | A standalone control for stated Gemini training and grounding uses | Allow or disallow those uses independently | It has no separate HTTP user agent and does not affect Google Search inclusion or ranking |
| Google user-triggered fetchers | A user asks a Google product to fetch a resource or take an action | Use real access/security rules for private content | Google says these fetchers generally ignore robots.txt; its published list is not exhaustive |
Bingbot | Bing search crawling | Set Bing crawl rules and page-level indexing choices | A specific Bingbot group can take precedence over a generic group |
Bing NOCACHE / NOARCHIVE | Page-level choices affecting documented Copilot/Chat display and training uses | Set the supported meta tag or response header on the page | These are not robots.txt bot groups; review current Bing wording before deployment |
OAI-SearchBot | Surface websites in ChatGPT search features | Allow search crawling separately | A block can still leave a navigational title/link in stated cases; use page-level noindex for that separate goal |
GPTBot | Crawl content that may be used in OpenAI foundation-model training | Signal a separate training preference | This control does not manage ChatGPT Search |
ChatGPT-User | Certain actions initiated by a ChatGPT or Custom GPT user | Apply real access controls where needed | OpenAI says robots.txt may not apply because the action is user initiated |
Claude-SearchBot | Improve Claude search-result quality | Decide search-crawl access | Access remains eligibility, not a citation promise |
ClaudeBot | Collect content that could contribute to Anthropic model training | Signal a separate training preference | Apply the rule to each relevant subdomain |
Claude-User | Retrieve content at a Claude user’s direction | Decide user-directed retrieval access under Anthropic’s documented rules | Anthropic says its named bots honor robots directives |
PerplexityBot | Surface and link websites in Perplexity search results | Decide search-crawl access | Perplexity says this agent is not used to crawl for foundation-model training |
Perplexity-User | Fetch a page for a user action | Use an enforcement rule if a real block is required | Perplexity says it generally ignores robots.txt because a user requested the fetch |
Primary references: Google’s crawler list, Google’s user-triggered fetchers, Bing’s supported robots meta controls, OpenAI’s crawler documentation, Anthropic’s crawler documentation and Perplexity’s crawler documentation.
Google AI Overviews are part of Google Search. Google-Extended is not an AI Overview opt-in switch. Google’s current AI features guidance points owners back to ordinary Search eligibility and useful content.
Robots.txt and noindex can work against each other
Disallow and noindex solve different problems.
| Your goal | Appropriate starting control | Common mistake |
|---|---|---|
| Reduce automatic crawling of an unimportant public path | A correctly scoped robots rule | Assuming the URL can never appear anywhere |
| Keep a public page out of Google or Bing results | A supported page-level noindex meta tag or X-Robots-Tag | Blocking the crawler before it can read noindex |
| Keep information private | Authentication and authorization | Listing the secret path in a public robots file |
| Limit a search-result snippet | A supported snippet meta/header rule | Blocking the page and expecting the crawler to read the rule |
| Exclude stated training use | The current provider-specific control | Blocking every search crawler and losing desired discovery |
Google says a page must remain accessible to Googlebot for its noindex rule to be seen. Bing gives the same practical instruction: allow Bingbot to crawl a page that carries noindex. Google also says noindex inside robots.txt is not supported.
There is another subtlety. A disallowed URL can be discovered from links even when the crawler cannot read its content. Google may show the URL without a normal description. OpenAI’s publisher FAQ describes a similar title/link possibility in stated ChatGPT search conditions. If the goal is indexing or result display, use the provider’s page-level control and let the relevant crawler see it.
Read Google’s noindex guidance before combining the two controls.
The live edge response can differ from the origin file
The edge is the CDN or security layer a request reaches before your server. The origin is the server or application that owns the underlying file and page.
An owner may inspect the origin file and see Allow: /. A crawler can still receive a different answer.
Cloudflare’s current managed robots.txt documentation says it can prepend managed content to an existing origin file. If the origin has no file, Cloudflare can create one at the edge. Its crawler-control feature can also create or update a WAF rule that returns a block response.
That creates at least three independent questions:
- What policy did the owner intend?
- What does the live public
robots.txtresponse say? - What status and body does the named requester receive for the page?
Use this short test:
- Fetch the live
https://yourdomain.com/robots.txtresponse. - Compare it with the origin or CMS configuration.
- Request one public sample URL using the named user agent.
- Where the provider publishes IP ranges or a verification method, use it with the agent string. A user-agent string alone can be spoofed.
- Record the status, response body, redirect, challenge and relevant security/server log.
- Recheck after the provider’s documented cache or propagation period.
A 200 proves that the request received a response. A 403 proves that one layer refused it. Neither proves what a search or answer product did downstream.
If Cloudflare or a WAF is in the path, use the focused Cloudflare AI blocking check. It owns the deeper edge-versus-origin diagnostic.
Use authentication for private information
Authentication checks who is making a request. Authorization checks whether that person or system may open the specific resource. Those are the controls that protect private records.
The OWASP authorization guidance recommends denying by default where no permission matches and validating permissions on every request. RFC 9309 says robots.txt is not a substitute for valid security measures.
This matters in every industry:
- A patient portal can contain electronic protected health information. The HHS Security Rule requires appropriate safeguards where it applies.
- A legal client file or detailed intake record can contain confidential information. The ABA Model Rule 1.6 model requires reasonable efforts against unauthorized access, but the rules of the governing jurisdiction control.
- A SaaS tenant workspace can contain customer data, credentials or keys.
- An eCommerce account can contain addresses, payment-related data and order history.
- A home-service job record can contain a customer’s address, access details and private notes.
Do not paste private data or credentials into a crawler test. Use public sample URLs and redacted logs. Healthcare decisions need privacy and security review. Legal decisions need review under the rules that govern the firm. This page is not legal or clinical advice.
One access standard, five industry applications
The public/private rule stays the same. The useful public pages and the protected records change.
| Industry | Public pages usually considered for discovery | Keep behind real access control | Business outcome measured separately |
|---|---|---|---|
| Home Services | Service pages, legitimate market pages, warranties, financing explanations and public business facts | Customer addresses, job notes, access instructions, invoices and account records | Qualified calls and booked jobs |
| Healthcare | Approved provider, location, service, insurance and appointment information | Patient portals, forms containing patient data, clinical records and messages | Appropriate appointments |
| Legal | Jurisdiction-reviewed attorney, practice, office and consultation information | Client files, detailed intake, privileged communications and internal matter records | Qualified consultations and accepted matters |
| B2B SaaS | Product, use-case, integration, comparison and public documentation pages | Tenant workspaces, customer data, API keys, internal dashboards and private documentation | Qualified demos, opportunities and pipeline |
| eCommerce | Category, product, shipping, returns, care and merchant-information pages | Customer accounts, addresses, payment records, carts and order history | Completed and return-adjusted orders |
The outcome column does not belong in robots.txt. It belongs in measurement. A crawler request is not a booked job, appropriate appointment, accepted matter, qualified opportunity or completed order.
Change one policy, test it and keep the record
Crawler documentation changes. Edge products add controls. A plugin update can overwrite a file. Treat the decision like a small production change.
- Freeze the intended policy by content class and provider purpose.
- Save the current public file and relevant edge/origin settings.
- Change one control at a time.
- Wait the documented cache or propagation window.
- Test the live file and named sample requests.
- Verify the requester where the provider publishes a method.
- Record status, response body, time and relevant logs.
- Check indexing or source display later as a separate observation.
- Keep qualified enquiries and commercial results separate again.
- Set a review date and rollback owner.
The evidence chain should read:
rule published → file fetched → page requested → response delivered → indexing observed → source displayed → buyer acted
Each arrow needs its own evidence. A 2026 controlled study of ten AI assistants found that retrieval behavior and visible answer correctness could diverge. A system could access a page without surfacing it, or fail to use a page it was allowed to access. Other studies also found varied compliance in their own samples.
Those studies explain why testing matters. They do not tell you what happened on your site today.
Do not add an llms.txt file to solve this problem. It has a different proposed job. See Does llms.txt Work? for that separate evidence decision.
What Mindflow can check and what it cannot promise
For a crawler-access review, bring:
- the public
robots.txtURL; - a short list of public pages that should be reachable;
- the name of the CDN, WAF, CMS or hosting layer;
- safe log excerpts with credentials and private customer, patient, client or tenant data removed;
- the business reason for allowing or blocking search, user retrieval or training.
Mindflow can map the stated provider purpose, compare the public edge response with the intended policy, test public URLs, record allow/block/unknown results and tell you which control deserves attention first.
We cannot promise that an allowed page will be indexed, cited, recommended or converted into an enquiry or sale. We can make the access decision clearer and the evidence easier to check.
Request your Free Visibility Check or review the wider AI Visibility service.
Provider matrix and primary documentation checked 21 August 2026. Recheck the linked provider sources before changing a production access policy.