Guides/Technical fixesupdated 2026-08-04 · 5 min

Should You Block GPTBot, ClaudeBot, and PerplexityBot? (No — Here's Why)

The short answer

No. For a local service business, blocking AI crawlers removes you from the systems your customers are asking for recommendations. The advice to block them came from publishers protecting paid content — a completely different situation from a contractor whose website exists to be found.

Somewhere between 2023 and 2025 a lot of local business websites quietly disappeared from AI search, and most of the owners still do not know it happened.

The cause was a robots.txt file, edited by a security plugin or a well-meaning developer, following advice that was never meant for them.

Where the advice came from

Large publishers — news organizations, reference sites, anyone whose business is the content itself — had a real reason to block AI crawlers. Their content is the product. Letting a model ingest it for free undermines both their subscription business and their negotiating position for licensing deals.

That reasoning is sound for The New York Times. It is exactly backwards for a plumbing company.

Your website is not the product. It is an advertisement for the product. Its entire job is to be found. Blocking the crawlers that feed the systems your customers now ask for recommendations is like taking your business out of the phone book to stop people reading your ad for free.

The advice spread anyway. Security plugins added AI-crawler blocking as a default. Agencies added it as a checkbox service. Nobody told the owner what it cost.

The bots, and what each one does

User-agent Operator What it does
GPTBot OpenAI Crawls for training data
OAI-SearchBot OpenAI Crawls for ChatGPT search results
ChatGPT-User OpenAI Fetches a page when a user's question needs it live
ClaudeBot Anthropic Crawls for indexing and training
Claude-Web / Claude-User Anthropic User-triggered live fetching
PerplexityBot Perplexity Crawls for its search index
Perplexity-User Perplexity User-triggered live fetching
Google-Extended Google Controls use in Gemini and grounded AI answers
Applebot-Extended Apple Controls use in Apple Intelligence training
CCBot Common Crawl Open crawl used by many downstream models
Bytespider ByteDance Crawls for TikTok's AI products

The distinction between the training crawlers and the search crawlers matters. OAI-SearchBot, ChatGPT-User, PerplexityBot, and the user-triggered fetchers are how the assistants read your page to answer a live question about your business. Blocking those is the most directly damaging choice — it does not just keep you out of a future model, it keeps you out of the answer someone is getting right now.

Google-Extended deserves a specific note: it does not affect Google Search ranking or indexing at all. It only governs Gemini and grounded AI use. Blocking it therefore costs you AI visibility and buys you nothing.

How to check, in thirty seconds

Open https://yourdomain.com/robots.txt in a browser.

You are looking for anything like:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Or a blanket block:

User-agent: *
Disallow: /

That last one blocks everything including Google, and we do find it on live sites — usually left over from a staging environment that got pushed to production.

What your robots.txt should look like

For a typical local service business:

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

That is it. Allow everything, point at your sitemap.

If you have genuine areas that should not be indexed — an admin path, a customer portal, internal search results — disallow those specific paths for all agents:

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /portal/
Disallow: /?s=

Sitemap: https://yourdomain.com/sitemap.xml

Note that robots.txt is not a security mechanism. Anything genuinely private belongs behind authentication, not behind a Disallow line.

Where the block usually comes from

Security plugins. Several popular WordPress security and firewall plugins ship AI-crawler blocking, sometimes on by default. Check your plugin settings, not just the file — some plugins generate robots.txt dynamically and overwrite manual edits.

"AI protection" plugins. A category that appeared in 2024 marketed on protecting your content. For a local business it is a product that reduces your visibility for a fee.

Cloudflare. Cloudflare offers one-click AI bot blocking and has enabled AI crawler controls by default for some new zones. Check your Cloudflare dashboard under Security → Bots, not only your robots.txt — Cloudflare can block at the network level regardless of what your file says.

Your developer. Frequently added in good faith after reading publisher-oriented advice.

Your agency. Occasionally sold as a feature.

Also check that you are not blocked at the server level

robots.txt is voluntary and separate from actual blocking. A crawler can be allowed in robots.txt and still be refused by:

  • Cloudflare or another WAF bot rule
  • A firewall rule blocking unknown user agents
  • Aggressive rate limiting
  • A "verify you are human" challenge on every request

If your robots.txt is clean but AI assistants still seem to know nothing about your site, check these next.

The one legitimate concern, addressed

"But they are using my content without paying me."

For a publisher, that is a real grievance. For a local service business, consider what "your content" actually is: your services, your service area, your phone number, and a few hundred words about what you do.

You want that reproduced. That is marketing. An assistant that repeats your service area and phone number to a homeowner who needs a plumber has done you a favor, not an injury.

The businesses winning AI recommendations are not the ones protecting their service page copy. They are the ones making sure it is readable everywhere.

What to do this week

  1. Open yourdomain.com/robots.txt and read it.
  2. Remove any Disallow targeting an AI user-agent.
  3. Check your security plugin settings for AI blocking toggles.
  4. Check Cloudflare's bot settings if you use it.
  5. Confirm your sitemap line is present and correct.
  6. Re-check in a week — some plugins regenerate the file.

Then find out what they actually say

Once the crawlers can read you, the question becomes what the assistants say when a customer asks. Our free scan runs twenty-five real customer questions across ChatGPT, Claude, Gemini, Perplexity, and Google AI Overviews and shows you every verbatim answer.

About fifteen seconds, free, no account.

Common questions

How do I check if I'm blocking AI crawlers?
Open yourdomain.com/robots.txt in a browser and read it. Look for Disallow rules under user-agent lines naming GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Google-Extended, CCBot, Bytespider, or Applebot-Extended. It takes thirty seconds.
Why did people start blocking AI crawlers?
Large publishers with paid content and licensing leverage blocked them to protect their business model and negotiating position. That reasoning does not transfer to a business whose website exists to attract customers. Plugins and developers then applied the advice indiscriminately.
Does blocking Google-Extended hurt my search ranking?
No — Google-Extended does not affect Search ranking or indexing. It controls whether your content can be used in Gemini and grounded AI responses. So blocking it costs you AI visibility and buys you nothing, which is the worst possible trade for a local business.
Is there any case where a local business should block them?
Rarely. Genuinely proprietary content behind a login should not be crawlable by anything, which is a permissions question rather than a robots.txt one. For your public marketing pages, blocking is self-harm.
Stop guessing

See exactly what the five AIs say about your business.

Twenty-five real customer questions across Claude, ChatGPT, Gemini, Perplexity, and Google AI Overviews. Free, about fifteen seconds, no account.

Run the free scan

Keep reading