There is a new class of bot reading your website, and most teams have no policy for it. GPTBot, ClaudeBot, PerplexityBot, and a growing list of others crawl pages to train models, to power live answers, and to fetch a URL the moment a user pastes it into a chat. The question is no longer whether they visit. It’s whether you’ve decided, on purpose, what they’re allowed to do when they arrive.
This is a practitioner’s guide to that decision. What these crawlers are, what robots.txt actually controls, where it stops working, and how to set a policy you can defend.
The three jobs AI crawlers do
Lumping every AI bot into one bucket is the first mistake. They do different jobs, and you may feel differently about each.
Training crawlers collect pages to build or update a model’s weights. OpenAI’s GPTBot and Anthropic’s ClaudeBot are the common names here. What they take gets absorbed into a model and surfaces later as general knowledge, usually without a link back to you.
Live retrieval bots fetch pages in real time to answer a question a user is asking right now. OAI-SearchBot, PerplexityBot, and Google’s Google-Extended signal sit closer to this. When an assistant cites a source with a clickable link, retrieval is usually how it got there.
User-action fetchers grab a single URL because a person asked. ChatGPT-User and Claude-User are the pattern: someone pasted your link into a chat and the assistant went to read it. This isn’t crawling your site at scale. It’s one human, one page, on demand.
The reason this taxonomy matters: blocking training does nothing to your live-answer visibility, and blocking retrieval can quietly remove you from the exact surfaces you were hoping to win. Decide per job, not per company.
Blocking AI crawlers is a content licensing decision wearing a technical costume. Treat it like one.
What robots.txt actually controls
robots.txt is a plain-text file at the root of your domain that names user-agents and tells well-behaved bots which paths they may request. It’s the same mechanism Googlebot has honored for decades, and the major AI crawlers have published the agent names they respect.
A minimal policy that allows everything but blocks one training crawler looks like this:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /To block a list of them and leave search and retrieval alone, name each agent:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /Two things to hold onto. First, robots.txt is per user-agent, so you can allow the retrieval bot from a vendor while blocking that same vendor’s training bot. The agent strings are different, and the vendors document which is which. Second, the file is a request, not a wall. It works because reputable companies choose to honor it. Nothing in the protocol forces compliance.
The newer Content-Signal syntax inside robots.txt lets you express intent more precisely than allow or disallow, separating “you may crawl this” from “you may use it to train.” Adoption is early and uneven. Treat it as a clarifying layer on top of agent rules, not a replacement for them.
Where robots.txt stops working
Plenty of teams set a rule, assume it’s enforced, and never check. Here’s where that confidence is misplaced.
- It only binds bots that opt in. Scrapers that ignore the file, or that crawl under a generic agent string, sail straight past it. If you need a hard block, that lives at the network or firewall layer, not in a text file.
- It governs fetching, not memory. Disallowing
GPTBottoday does nothing about pages already absorbed into a model from earlier crawls or from third-party datasets like Common Crawl. The file is forward-looking only. - Disallow is not noindex. Blocking a path stops the fetch, but a URL can still appear in an index from inbound links. For AI surfaces there’s no reliable “don’t cite me” equivalent yet, so don’t expect one.
- User-action fetches often ignore it. When a person pastes your URL into an assistant, several vendors treat that as a user request rather than crawling, and fetch the page regardless of your crawl rules. You usually want this. It’s how your link gets read when someone is already interested.
The honest summary: robots.txt is a good tool for steering cooperative bots and a poor tool for stopping determined ones. Match the tool to the threat.
A decision framework
Skip the reflex to block everything, and skip the reflex to allow everything. Work through four questions instead.
- Is the content your distribution or your product? Marketing pages, guides, and documentation are how buyers find you. You generally want those read, quoted, and cited. A paywalled archive or proprietary dataset is the opposite case, and a candidate for blocking.
- Which job are you reacting to? If your worry is training without attribution, block the training agents and leave retrieval open. If you want to vanish from live answers too, that’s a different and more aggressive choice with real visibility cost.
- What do you lose by being absent? For most brands, presence in AI answers is becoming a discovery channel that behaves like organic search did fifteen years ago. Blocking retrieval to protect content nobody was monetizing is a bad trade.
- Can you enforce what you intend? If a block has to be airtight for legal or contractual reasons,
robots.txtalone won’t get you there. Pair it with server-side agent rules and accept the maintenance.
For a typical agency, SaaS, or content business, the answer that holds up is: allow retrieval and user-action bots, allow or block training bots based on how you feel about uncompensated model training, and put genuinely sensitive material behind auth where no crawl policy is needed. Most of the value is in being read by the bots that send traffic and citations back.
Verify, then revisit
A policy you don’t check is a guess. Three habits keep it honest.
Read your server logs and filter by user-agent. The agent strings are public, and your logs show who actually came, how often, and which paths they hit. This is the only ground truth for whether your rules are being honored.
Validate the file itself. A stray syntax error or a misplaced wildcard can block far more than you meant, or nothing at all. Confirm the live file matches your intent after every deploy, because a redesign that regenerates robots.txt is a common way to silently undo a rule.
Revisit on a schedule. New agents launch, vendors split crawling into more granular bots, and the Content-Signal conventions are still settling. A rule set that was correct in January can be incomplete by summer. This work pairs naturally with a broader technical SEO checklist, since the same logs and crawl behavior drive both.
If your aim is to be read well rather than blocked, the structural work matters more than the rules. A clean, fast, crawlable site with a clear llms.txt gives cooperative bots an easy path to your best material, and the habits that get you cited by AI assistants start from being legible in the first place.
How Strynal approaches AI crawler policy
We treat crawler policy as part of a site’s information architecture, not a config file someone edits in a panic after reading a headline. Before touching robots.txt, we ask what the site is for, which content earns distribution by being read, and what genuinely needs to stay private. Then the rules follow the strategy instead of fighting it.
Because strategy, design, and build sit together here, we can set the agent rules, structure the content so retrieval bots can lift it cleanly, and confirm in the logs that real bots behave as intended, all in one pass. The team that scopes the work is the team that ships it, so the policy on the page matches the intent in the brief.
If you want a clear read on which AI crawlers are reaching your site and whether your current rules are helping or quietly hurting you, see how we run AI visibility work, and we’ll tell you what we find.