For most small business websites, our recommended starting point is to allow the crawlers that support search and AI answers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot), while making a separate, deliberate choice about training-related controls such as GPTBot, ClaudeBot, Google-Extended and CCBot. Several providers separate search and training controls, but the effects differ: Google-Extended also controls some Gemini grounding uses. Blocking a search crawler can limit that provider's access to your pages and their eligibility for answers; it is not a universal way to remove every link or mention.

Allowing a crawler doesn't guarantee you'll be shown or cited. It only removes one reason you might not be.

What robots.txt does, and what it doesn't

A robots.txt file sits at the root of your site (for example https://www.example.com/robots.txt) and tells crawlers which parts of the site they may visit.

Google describes it as a tool "used primarily to manage crawler traffic to your site" and says it is "not a mechanism for keeping a web page out of Google" (Google: introduction to robots.txt, checked 2026-10-21). Three limits matter:

  • It's voluntary. Reputable crawlers follow it, but nothing forces a crawler to.
  • It's not security. Anyone can read your robots.txt, and blocked pages aren't hidden.
  • It controls crawling, not every kind of visit. Several AI companies say fetches made because a user asked a question may not follow robots.txt (see below).

Search crawlers, training controls and user-triggered fetchers

The most useful thing to understand is that several providers distinguish different jobs through bots or product controls:

  • Search or index crawlers collect pages so the tool can find and link to them when answering questions.
  • Training crawlers and product controls let you express preferences about future collection or use for training. Some controls cover additional uses, so check the provider's description.
  • User-triggered fetchers visit a page in real time because someone asked the AI a question.

Where a provider offers independent controls, you can make a different decision for each.

The main crawlers, company by company

OpenAI (ChatGPT) (OpenAI: overview of crawlers, checked 2026-10-21)

  • OAI-SearchBot: used for ChatGPT search. OpenAI says to use OAI-SearchBot in robots.txt to manage search opt-outs, and that it can take about 24 hours for changes to take effect. Opted-out sites are excluded from ChatGPT search answers, but may still appear as navigational links.
  • GPTBot: crawls content that may be used to train OpenAI's models. Disallowing it signals that your content shouldn't be used for training.
  • ChatGPT-User: visits pages when a ChatGPT user asks a question. Because these visits are user-initiated, OpenAI says "robots.txt rules may not apply". OpenAI also says ChatGPT-User isn't used to decide whether content appears in search.
  • Each setting is independent: allowing OAI-SearchBot and blocking GPTBot is a valid combination.

Google (Google: common crawlers, last updated 2026-10-15; AI features and your website; both checked 2026-10-21)

  • Googlebot: crawls for Google Search, including AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are the control for how your site is crawled for Search. Blocking Googlebot prevents crawling and can harm visibility across Search. It does not reliably remove already indexed URLs; robots.txt is not an indexing-removal tool.
  • Google-Extended: not a separate crawler but a robots.txt token. It controls whether content Google crawls can be used to train Gemini models and for grounding in some other Google AI products. Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". To limit what appears from your pages in Search, including AI features, Google points to nosnippet, data-nosnippet, max-snippet and noindex instead.

Google also offers a Search generative AI control in Search Console. Include is the default, but an Exclude setting can apply; check the current setting separately from robots.txt (Google control documentation).

Microsoft (Bing and Copilot) (Bing Webmaster Guidelines; Bing Webmaster blog, September 2023, checked 2026-10-21)

  • Bingbot: Bing's search crawler. Bing's current guidelines describe meta-tag controls for Copilot and grounding: NOCACHE limits use to the URL, title and snippet, while NOARCHIVE prevents use in Copilot responses and grounding results. Microsoft's 2023 announcement also describes the corresponding limits on future use of Bing-indexed content for model training. Importantly, it says that when both tags are present, NOCACHE applies. These tags do not by themselves exclude a page from ordinary Bing search, but appearance is not guaranteed.

Perplexity (Perplexity: crawlers, checked 2026-10-21)

  • PerplexityBot: surfaces and links websites in Perplexity's search results. Perplexity says it isn't used to crawl content for AI foundation models, and recommends allowing it if you want to appear in results.
  • Perplexity-User: visits pages when a user asks a question. Perplexity says this fetcher "generally ignores robots.txt rules" because a user requested it.

Anthropic (Claude) (Anthropic: does Anthropic crawl the web?, checked 2026-10-21)

  • ClaudeBot: collects web content that may be used for model training. Blocking it signals that future content should be excluded from training.
  • Claude-SearchBot: used to improve search results. Blocking it may reduce your visibility in Claude's search answers.
  • Claude-User: fetches pages when a Claude user asks a question. Blocking it stops Claude retrieving your pages for user queries.
  • Anthropic says its bots honour robots.txt and don't try to bypass CAPTCHAs.

Apple (Apple: about Applebot, checked 2026-10-21)

  • Applebot: crawls for search features in Siri, Spotlight and Safari.
  • Applebot-Extended: doesn't crawl pages. It controls whether content Applebot has collected can be used to train Apple's generative models. Apple says pages that disallow Applebot-Extended can still appear in search results.

Common Crawl (Common Crawl FAQ, checked 2026-10-21)

  • CCBot: builds a free, public archive of the web used by researchers and companies. Common Crawl's own About page describes use of its data, after filtering and processing, in language-model training. Common Crawl says CCBot obeys robots.txt directives, so disallowing it limits future crawling rather than withdrawing existing datasets.

A sensible starting point for a small business

This is an example, not a template to copy blindly. On its own, it leaves unlisted search crawlers unrestricted by robots.txt and expresses opt-outs for the listed training-related uses. Google-Extended also affects grounding in Gemini Apps and Grounding with Google Search on Vertex AI, not Google Search itself. Review each control before keeping or removing it, and preserve any existing rules your site needs.

# Search crawlers: allowed by default (not listed)

# Limit the listed training-related uses
# Google-Extended also affects some Gemini grounding uses
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Before you change anything:

  • Check what's already there. Open yourdomain.com/robots.txt in a browser. Your site builder or SEO plugin may already be generating one.
  • Avoid an accidental site-wide block. Disallow: / under User-agent: * blocks compliant crawlers using that group. Googlebot uses its most-specific matching group, so check any separate Googlebot rules too (Google's robots.txt specification).
  • Decide on training with your eyes open. Google says Google-Extended does not affect inclusion or ranking in Google Search. Apple says Applebot-Extended is not considered in Search ranking and disallowed pages can still appear in results. These assurances concern those specific controls, not every bot or CDN block. Whether to allow training remains a business choice.
  • Test after changes. Use Search Console's URL Inspection live test to check crawl access to key pages, and review the current robots.txt rules and any server-level blocks. A successful test does not guarantee indexing or visibility.

Check whether your host or CDN is blocking AI bots for you

Your robots.txt isn't the only place bots get blocked. Content delivery networks and security tools can block them at the server level, sometimes by default.

For example, Cloudflare announced in July 2026 that from 15 September 2026, new domains would block "Training" and "Agent" bots by default on pages that display ads, while keeping "Search" bots allowed. It also said that crawlers which do both search and training, including Googlebot, Applebot and Bingbot, would be blocked for customers who choose to block training (Cloudflare blog, July 2026, checked 2026-10-21).

If your site uses Cloudflare or a similar service, check its bot settings as well as your robots.txt. A setting that blocks "AI training" could have wider effects than you expect.

What blocking will and won't do

  • It can stop compliant crawlers from collecting disallowed pages after they process the updated rules; product tokens may instead control how already-crawled content is used.
  • It won't remove content already collected, or necessarily stop user-triggered fetches.
  • It won't stop your business being mentioned in AI answers based on other websites that talk about you.
  • Blocking a search crawler can limit search access or answer eligibility, but effects vary. OpenAI, for example, says OAI-SearchBot opt-outs may still appear as navigational links.

What's in your control, and what isn't

In your control: your robots.txt rules, robots meta tags, your CDN or security settings, and whether search crawlers can reach your important pages.

Not in your control: whether every crawler respects your rules, how user-triggered fetchers behave, and whether any AI tool chooses to show or cite your pages. Allowing appropriate crawlers supports access and eligibility for the uses described above; it does not promise a citation or control every mention.

Related guides