AI Crawler Reference: Which Controls Affect Training, Search and User-Requested Fetches?

Last checked: 19 September 2026. This reference describes published provider policies. Configuration examples have not been deployed or tested against live crawlers.

“Block AI bots” sounds like one decision. For a website owner, it usually means several.

You might want people to discover your tutorials through AI search, while declining their use for model training. You might also want an assistant to open a product page when a customer asks about it. Those preferences can require different controls.

Choose controls by purpose, then by provider. Training, search and user-requested fetching need separate decisions. The table below maps the documented options and their limits.

There is another distinction to keep in view: robots.txt expresses crawl preferences. It does not enforce authentication or make public content private. The Robots Exclusion Protocol, RFC 9309, explicitly separates its rules from access authorization.

The three reasons an AI service might access your website

1. Collecting material for model training

This concerns content that could help develop a model. A training preference answers a different question from whether a page should appear in a search result.

2. Finding content for search answers

An AI search service needs to discover and retrieve useful sources. Its search crawler may support an index that helps the service find your pages later.

3. Fetching a page because a user requested it

Someone might paste a URL into an assistant and ask for a summary. That request can trigger a fetch independently of routine crawling. The word “user” in a bot name does not establish whether robots.txt applies; the provider’s documentation does.

These purposes can share a crawl, as OpenAI documents.

AI crawler user agents and controls: detailed comparison

The entries below cover seven operators. They are a selected reference, not an exhaustive list of automated traffic.

How to read the table: A token is the name used to select a robots.txt group. A full HTTP User-Agent string may include browser details and a version. Some entries are usage-control tokens that do not identify a separate crawler. “Respects robots.txt” below means the provider’s published policy, not independently measured compliance.

Use the verification resources later in this article to check a caller’s identity. A matching name in your logs is not proof of origin.

The differences that are easiest to miss

Google-Extended also covers some grounding

Google describes Google-Extended as governing future Gemini training and grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Grounding means supplying retrieved information when a model prepares a response.

It is therefore too broad to describe this token as “training only.” It is also not the switch for Google AI Overviews or AI Mode. Those are Search features. Google says the Google-Extended preference does not affect Search inclusion or act as a Search ranking signal. Google’s crawler reference.

For Google Search, choose the preview control carefully:

• nosnippet prevents page content from being a direct input to AI Overviews and AI Mode. It also removes ordinary text snippets and video previews; an image thumbnail may remain.
• max-snippet:0 is equivalent to nosnippet. A positive value limits text length instead. Google documents exceptions for separately permitted uses, such as licensed content or supplied structured data.
• data-nosnippet excludes marked sections from snippets, rather than applying a whole-page restriction.

These are Google’s documented rules, not universal settings for every assistant. For a Google-specific page restriction, an example HTML-head tag is:

<meta name="googlebot" content="nosnippet">

User-requested fetching is provider-specific

The three “User” entries in the table are deliberately different. OpenAI says robots.txt may not apply; Perplexity says its fetcher generally ignores it; Anthropic documents robots.txt support for its bots, including Claude-User.

Check the individual fetcher when deciding whether assistants may open public pages for readers. OpenAI, Perplexity, Anthropic.

Apple’s training preference does not remove all answer uses

Applebot-Extended addresses training. Apple’s current help page separately describes nosnippet for excluding content from broad world-knowledge answers in its products. Turning off training alone should not be described as turning off every generative use. Apple’s Applebot documentation.

Bing’s noarchive and nocache are not interchangeable

Bing’s current help page documents these distinctions:

• noarchive: excludes content from Microsoft’s foundation-model training and prevents links in Chat/Copilot.
• nocache: permits URL, title and snippet display in Chat/Copilot, and permits those elements to be used for training.
• Both together: Bing applies the nocache treatment.

You can target Bing with a page-level tag such as:

<meta name="bingbot" content="noarchive">

This belongs in the page’s HTML head, not robots.txt. Bing also supports the directives through an X-Robots-Tag response header. Its published announcement says these controls do not remove pages from ordinary Bing search results. Current Bing help, Microsoft’s explanation.

Choose a policy before editing robots.txt

A short written policy is easier to maintain than a copied list of bot names.

For example: “Our public tutorials should remain discoverable through search. We decline the documented training uses where separate controls exist. We allow supported user-requested visits to public pages.”

That is a proposed publisher policy, not a promise that every platform offers the same options.

Make the decision at the content level too. A public help article, an unpublished client report and a subscriber-only dataset do not need the same access policy.

Blocking crawling is different from removing a search result

A robots.txt Disallow rule can stop Google from fetching a page while its URL remains discoverable through links elsewhere. For removal from Google’s index, use an appropriate noindex meta tag or response header and allow the crawler to read it. Putting noindex in robots.txt is not supported. This affects ordinary search visibility too. Google’s robots.txt limitations, Google’s noindex instructions.

For private content, require authentication. Neither hiding a snippet nor requesting removal from search prevents direct access to a public URL.

Example: decline selected training uses without blocking search crawlers

The following fragment targets three separately documented training controls:

# Merge these with any existing rules for the same tokens.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

It does not add a restriction for their corresponding search crawlers. Existing rules and firewall settings still apply. The scope comes from the individual OpenAI, Anthropic and Apple policies.

If you also want to decline the training and specified grounding covered by Google-Extended, add its separate group:

User-agent: Google-Extended
Disallow: /

Review that grounding trade-off before adding it. Google’s documented scope.

Common Crawl requires a separate decision about dataset collection:

User-agent: CCBot
Disallow: /

Common Crawl documents this rule for preventing future crawling by CCBot. It is not a control for every organization that might already hold a copy of a page. Common Crawl’s crawler guidance.

These are fragments to review and merge, not a replacement for your complete robots.txt. Keep your existing sitemap declaration and any intentional path rules. Review existing Allow exceptions within matching groups before assuming a new Disallow: / settles the policy.

Also check group selection. Google’s parser uses the most specific matching user-agent group and does not combine it with the wildcard group. Adding a specific group can therefore change which path restrictions apply. Google’s robots.txt specification.

For basic rule drafting, the SanishTech robots.txt generator provides a live preview. In our limited browser check on 19 September 2026, changing a sample path updated the displayed rule. The visible user-agent dropdown did not offer the AI tokens above, so add those groups manually in your text editor. This check did not test download behavior or crawler compliance.

Verify who is making the request

The HTTP User-Agent header describes the requesting software, but a caller can supply a misleading value. Validate the source using the operator’s published method. MDN’s User-Agent reference, Google’s verification guidance.

Google-Extended and Applebot-Extended are usage controls, not separate fetching agents. Do not expect requests under those names or use their absence from logs as evidence that a preference failed.

Verification documentation checked: 19 September 2026. Follow the live resources rather than copying today’s IP ranges permanently into a rule.

Keep a small log of the request time, path, claimed user agent, source-IP verification, response status and rule responsible for the decision. If you use a CDN, use its trusted client-IP information rather than blindly accepting a caller-supplied forwarding header.

Verification identifies the caller more reliably. It does not independently reveal how that organization later uses the retrieved content.

Check what your website actually serves

Treat the saved configuration and the live response as two things to compare.

1. Read the public robots.txt response. Confirm the root file contains the intended rules. Review each hostname and protocol you actually serve; a subdomain can need its own file.
2. Inspect a representative public page. Look at its HTML and response headers for indexing and preview directives.
3. Review CDN and hosting rules. A robots.txt allowance cannot make a request succeed when another layer blocks it.
4. Check real logs after the change. Separate robots.txt requests from page requests and verify the caller before interpreting either.

Google documents robots.txt scope by host, protocol and port. Robots.txt specification.

For a simple response inspection on a domain you manage, replace the example address:

curl -sS -D - https://example.com/robots.txt
curl -sS -D - https://example.com/public-article/

These commands show the initial headers and response body. They are examples, not tests performed for this article. Inspect any redirect before checking its destination.

A successful response to curl does not prove that a provider can fetch the page. A request with a copied bot name still comes from your machine, not the provider’s network.

The SanishTech SEO tools hub also links to page-level checking tools. Use those for additional inspection, while keeping provider identity and actual server responses as separate evidence.

Do you need a paid AI crawler blocking tool?

Editing robots.txt does not require an AI-service subscription. Network enforcement and reporting may involve hosting or security products.

Cloudflare’s AI Crawl Control is one option for enforcement and reporting. Its setup documentation requires the relevant traffic to pass through its proxy.

Pricing and availability checked on 19 September 2026 using Cloudflare’s pricing page and its AI Crawl Control help. These are published terms, not a paid-account test.

A September 2026 Cloudflare change to review

Cloudflare’s 15 September announcement distinguishes Disallow AI Training from Block. It says the latter now also applies to mixed-use crawlers, including Googlebot, Bingbot and Applebot, so selecting it can affect search discovery.

The announcement also says Bing’s support for a robots.txt-based training preference is targeted for early 2027. Cloudflare explicitly says its Disallow AI Training setting does not yet automatically convey that preference to Bing through robots.txt.

That is a published rollout announcement and future target, not confirmation of a feature in your account. Check the current dashboard and operator documentation before relying on the label. Cloudflare’s announcement.

Where llms.txt fits

An llms.txt file helps describe content for agents that use it. It is not a substitute for robots.txt, authentication or provider-specific controls. The current llms.txt proposal describes background information, guidance and links to useful content.

The HTTP Archive’s 2025 Generative AI chapter measures adoption of files and technologies. Adoption does not establish that adding a file increases rankings or citations.

Google also says special AI text files are not needed for its Search AI features. Google Search guidance.

Treat an llms.txt file as an optional publishing aid for compatible consumers. There is no verified citation improvement from adding one in this research.

Frequently asked questions

Does robots.txt remove content from existing training datasets?

Do not assume so. A future crawl restriction is not evidence of deletion from existing datasets or model weights. Anthropic, for example, describes its training opt-out in terms of future materials. Anthropic’s published policy.

Will allowing an AI crawler guarantee a citation?

No. Access is only one condition. Google explicitly says meeting its requirements does not guarantee crawling, indexing or serving a page. This research contains no experiment establishing a citation increase for any provider. Google’s eligibility guidance.

Why do requests still appear after I add a block?

First identify what was requested. A visit to robots.txt is different from a visit to restricted content. Other possibilities include cached preferences, a different user agent, a user-requested fetch or a caller ignoring the rule. Inspect the evidence before choosing an explanation. RFC 9309.

Is a crawl-to-refer ratio my site’s return on AI traffic?

No. Cloudflare Radar compares crawl requests with referrals for its observed traffic. Those network-level figures are not your website’s analytics. Measure your own visits and useful actions separately.

How this reference was researched

The provider policies, help pages, pricing information and linked SanishTech pages were checked on 19 September 2026. The main table includes entries whose purpose and controls could be documented from the named operators. Other crawlers, browser agents and third-party retrieval arrangements are outside its scope.

Bing’s current help was read in a browser. The only functional check was the generator preview described above. No live crawler compliance, paid-account features or ranking, citation, traffic and training-deletion outcomes were tested.

SanishTech owns the internal tools linked here. Their inclusion is for workflow relevance, not an independent comparison with competing tools.

Your next step: choose what your site allows

Before editing robots.txt or changing a firewall setting, answer three questions about your public content:

1. Do you want search services to discover it?
2. Which training or grounding uses do you want to decline?
3. Should assistants retrieve it when a user asks?

Start with one provider from the comparison table. Check its documented controls against your existing rules, save a backup and make the change that matches your policy. Then inspect the live response and review verified crawler requests in your logs.