Which AI Crawlers Should You Allow or Block in robots.txt?

How do I control which AI bots crawl my site?

You use a small text file called robots.txt to tell each AI crawler whether it may access your site. You name the bot, then allow or disallow it. Every major AI company publishes the exact name of its crawler, so you can permit the ones you want and block the ones you do not, line by line.

This is one of the most common questions I get in 2026, and most advice online is either outdated or too blunt. Blocking every AI bot sounds safe, but it can quietly cut you out of the AI answers where your future customers are looking. The smart move is selective, not total.

Let me walk through the real crawler names, what each one does, and how to make this choice on purpose.

What is robots.txt, and what can it actually do?

Robots.txt is a plain text file at the root of your domain that tells crawlers which parts of your site they may access. You write rules that name a user-agent, which is the crawler's identity, and then allow or disallow paths for it. Well-behaved bots read this file before crawling and follow the rules.

It is a request, not a wall. Compliant crawlers from major companies honor robots.txt because their reputation depends on it. A badly behaved scraper can ignore the file entirely, so robots.txt controls the polite bots, not the rude ones. Understanding that limit keeps you from a false sense of security.

The file works by matching the user-agent name. That is why knowing the exact crawler names matters so much. If you block a name that does not exist, nothing happens, and if you miss a real one, it keeps crawling. Precision is the whole game here.

Which AI crawlers should you know by name?

The big ones each publish their crawler names openly. OpenAI documents GPTBot for training, OAI-SearchBot for its search results, and ChatGPT-User for user-triggered browsing. Anthropic runs ClaudeBot. Perplexity runs PerplexityBot and Perplexity-User. Google uses a separate token called Google-Extended for its AI models.

These names come straight from each company's own documentation, not from guesswork. OpenAI lists its bots on its platform docs, Anthropic explains ClaudeBot in its privacy center, and Perplexity publishes its crawlers in its developer docs. I always check the source, because crawler names change and stale lists are everywhere.

The important insight is that these bots do different jobs. Some gather training data. Some fetch pages to answer a live user question. Treating them all as one thing is the mistake that leads people to block bots they actually want, which is the opposite of helpful.

Should you block GPTBot and ClaudeBot?

Block them only if you do not want your content used to train AI models. GPTBot is OpenAI's training crawler, and ClaudeBot is Anthropic's general-purpose crawler for gathering training data from public sites. Disallowing them tells those companies not to use your pages to train future models.

Anthropic documents that ClaudeBot follows robots.txt and even supports a Crawl-delay setting to slow it down, and you block it with a User-agent ClaudeBot rule and Disallow slash. OpenAI says publishers should disallow GPTBot to exclude pages from potential training. Both honor the standard, so the block works for the polite version of the bot.

Whether you should is a values question, not a technical one. Some creators block training crawlers on principle. Others let them through, betting that being part of the models helps their brand appear in AI answers later. There is no single right answer, only your answer, and I help clients think it through rather than default to fear.

Wait, do you actually want some AI bots to crawl you?

Often, yes. If you care about showing up in AI answers, you may want the search and browsing bots to reach you. OpenAI's OAI-SearchBot and Perplexity's PerplexityBot fetch content to build the answers those tools show users. Block them, and you can remove yourself from results where people are actively searching.

This is the trap I see most. Someone reads block AI bots, adds a rule that disallows everything, and then wonders why they never get cited in ChatGPT or Perplexity. They blocked the very crawlers that could have surfaced them. The training bots and the answer bots are not the same choice.

Perplexity is clear that if you block PerplexityBot, it will not index your full text, though it may still show your domain and a brief factual summary. So blocking can reduce your presence without fully removing it. If getting cited matters to you, read my guide on how to get cited by ChatGPT, Perplexity, and Google AI before you block anything.

What does Google-Extended do, and will it hurt my SEO?

Google-Extended lets you opt out of AI training without touching your search rankings. Google documents it as a control for whether your content helps improve its Gemini and Vertex AI models, and it is clearly separate from Googlebot, which handles Search. Disallowing Google-Extended does not affect how you rank.

This is a genuinely useful design. You can tell Google not to use your content for generative AI training while still being fully crawled and ranked in normal search. Google states plainly that Google-Extended does not affect Google Search, so the two decisions are independent.

I point this out because people conflate the two and freeze. They worry that opting out of AI training will tank their SEO, so they do nothing. It will not. If you want to sit out generative training but keep your rankings, Google-Extended is exactly the lever, and it costs you nothing in Search.

What can robots.txt not do?

It cannot force a bad actor to behave, and it cannot hide a page from search. Robots.txt is an honor system that compliant crawlers respect, but a scraper that ignores it will crawl anyway. It also is not a security tool, so never use it to protect private data, which needs real access controls.

There is another common mix-up. Blocking a crawler in robots.txt is not the same as adding a noindex tag, and the two solve different problems. I untangled that exact confusion in my piece on noindex versus robots disallow for AI crawlers, because using the wrong one is a frequent and costly mistake.

Anthropic even warns that trying to block its bot by IP address can backfire, since that can stop the crawler from reading your robots.txt in the first place. The lesson is to use the standard, named-crawler method, not clever hacks that break the polite behavior you are relying on.

How should you set this up on a Webflow site?

Webflow lets you edit robots.txt from your site settings, so you add your crawler rules there and publish. You write one block per crawler you want to control, naming the user-agent and setting Disallow slash to block it fully, or leaving it allowed by simply not disallowing it. Then you publish the site for the file to go live.

My practical approach is to decide two things first: do you want to be in AI training, and do you want to appear in AI search. Those answers tell you which specific bots to allow or block, and only then do you write the file. Deciding the strategy before touching the file prevents the block-everything reflex.

After you publish, check the file directly by visiting your domain slash robots.txt in a browser. Confirm each rule reads exactly as you intended, because a typo in a user-agent name silently does nothing. A two-minute check saves you from a rule you thought was working but was not.

What should you do next?

Decide your two answers, training and AI search, then write a robots.txt that matches them using the real crawler names from each vendor's docs. Block the training bots you object to, keep the answer bots you want to be found by, and verify the live file in your browser.

Getting this right is equal parts technical and strategic, and the strategy is where most people go wrong. If you want help deciding which AI crawlers to welcome and which to turn away for your specific goals, reach out through pravinkumar.co and we can map it out together.