How to check if your website is blocking AI crawlers

このコンテンツは英語です。

AI companies run several different bots, and each one does a different job: training models, building a search index, or fetching a page because a person asked a question. Many sites block some of them by accident, or allow all of them without ever deciding to. This guide explains how to read your robots.txt, what each user agent means, and what blocking does and does not achieve, based on the operators' own documentation as checked in October 2026.

Valtrinity 編。更新日：2026年10月18日。

Why it is worth checking

Your robots.txt file is a public set of instructions that tells crawlers which parts of your site they may request. For years it was mostly about search engines. Now it also carries decisions about AI: whether your content can be used to train models, whether it can appear in AI search answers, and whether assistants can fetch your pages when a user asks about them.

Those are three different choices, and a single rule can affect more than one of them. A broad block that you added years ago to deal with scrapers might be keeping you out of AI search results today. Equally, a site that wants its content left out of model training may have no rules at all, simply because nobody thought to add them.

Checking takes a few minutes and gives you an accurate picture of where you stand. The first step is to read your own file, and the rest of this guide explains how to interpret what you find.

The main AI user agents and what they do

A user agent is the name a crawler uses to identify itself, and robots.txt rules are matched against it. The operators below publish their own descriptions, and we have summarised them. Treat the operator's documentation as the source of truth, because names and behaviour can change.

• GPTBot (OpenAI): crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it signals that your content should not be used for that training.
• OAI-SearchBot (OpenAI): surfaces websites in ChatGPT's search features. If you opt out, your site will not appear in ChatGPT search answers, although it can still show as a navigational link.
• ChatGPT-User (OpenAI): visits pages when a user asks ChatGPT or a Custom GPT something, or uses GPT Actions. Because a person triggers it, OpenAI says robots.txt rules may not apply.
• ClaudeBot (Anthropic): collects web content that could help train Anthropic's generative AI models. Disabling it signals that your future content should be excluded from training datasets.
• Claude-SearchBot (Anthropic): crawls to improve the quality and relevance of search results for users. Blocking it stops indexing for that purpose and may reduce visibility in results.
• Claude-User (Anthropic): accesses a page when a Claude user asks a question that needs web content. Blocking it prevents retrieval for those requests.
• PerplexityBot (Perplexity): shows and links websites in Perplexity search results. Perplexity states it is not used to crawl content for AI foundation models.
• Perplexity-User (Perplexity): fetches individual pages on demand for a user's question. Because a user triggers the request, it generally does not follow robots.txt rules.
• Google-Extended (Google): a robots.txt token that controls whether content Google crawls can be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It does not affect inclusion in Google Search.

How to find and read your robots.txt

The file lives at the root of your host, so for example.com you would open example.com/robots.txt. Each subdomain has its own file, so if you run blog.example.com or shop.example.com you need to check those separately. Anthropic explicitly notes that rules must be repeated for each subdomain you want to cover.

A file is made of groups. Each group starts with one or more User-agent lines and is followed by Allow or Disallow rules. A crawler follows the group that matches its own name most specifically, and falls back to the group for the wildcard User-agent: * if nothing else matches. This is the detail that catches people out: if you have a wildcard group with Disallow: /, every bot that has no group of its own is blocked, including AI search bots you may want.

To read your file, go through it in order and write down three things. First, which groups name an AI user agent directly. Second, what the wildcard group says. Third, whether any Disallow rule covers the paths you care about, such as your blog or your product pages. A bot with no named group inherits the wildcard rules, and that is the combination that usually explains an unexpected result.

Example setups for common goals

If you want to stay out of model training but remain visible in AI search, you would block the training bots and leave the search bots alone. For OpenAI, add the two lines User-agent: GPTBot and Disallow: / and make sure OAI-SearchBot is not disallowed. For Anthropic, add User-agent: ClaudeBot and Disallow: / while leaving Claude-SearchBot allowed. OpenAI states that each of its settings is independent, so you can allow OAI-SearchBot while blocking GPTBot.

If you want to block an AI company entirely, add a group for each of its user agents. Anthropic says to use the same format for Claude-User and Claude-SearchBot as for ClaudeBot. For Google, add User-agent: Google-Extended and Disallow: / to opt out of Gemini training and grounding without any effect on how your pages rank in Search.

If you want to be fully open, you do not need to add anything, because crawlers are allowed by default. Some site owners still add explicit Allow groups for the bots they welcome, which makes their intent clear to anyone reading the file later.

Anthropic also supports the non-standard Crawl-delay extension if you want to slow a bot rather than block it. For example, a group with User-agent: ClaudeBot and Crawl-delay: 1 asks for a slower crawl. Not every crawler honours that directive, so treat it as a request.

What blocking does and does not do

Blocking in robots.txt is a request, not a technical barrier. Google's own documentation notes that crawlers are not forced to obey the file, and that well-behaved crawlers such as Googlebot do. The AI operators listed above state that their named crawlers respect it, but a robots.txt rule cannot stop a bot that chooses to ignore it.

Blocking is also forward-looking. Anthropic says that disabling ClaudeBot signals that your future content should be excluded from training datasets. OpenAI describes GPTBot in similar terms. Neither statement is a promise to remove material that was collected before you added the rule, so a block is best seen as controlling what happens from now on.

User-triggered fetchers are a separate case. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally does not follow robots.txt, because a person asked for the page. If you need to control those requests, robots.txt alone may not be enough, and you would look at server-level controls instead.

Finally, robots.txt is not the way to keep a page out of search results. Google states that it is not a mechanism for keeping a web page out of Google, and a blocked URL can still appear in results if other sites link to it. For that you use a noindex rule or password protection.

Common reasons a crawler is blocked by mistake

The most frequent cause is a wildcard group that disallows everything, left over from a staging site or an old plugin. Because bots without a named group inherit it, every AI search bot is blocked unless it has its own Allow rule.

A second cause is a typo or the wrong name. Rules are matched against the user agent token, so GPT-Bot is not the same as GPTBot. Use the exact spellings from the operator's documentation, and check for stray spaces or a rule placed above any User-agent line.

A third cause sits outside robots.txt. Your hosting provider, content delivery network or firewall may block automated traffic by default, which means a bot that robots.txt allows still gets an error. If your file looks right but you see no visits, check any bot-protection settings and your server logs.

• A wildcard group with Disallow: / that catches every bot without its own group
• Misspelt or outdated user agent names
• Rules in the wrong file, such as the root domain file when the content is on a subdomain
• A firewall or content delivery network rule that blocks bots before they reach robots.txt

How to verify the result

Once you have edited your file, open it at its address to confirm it is public and contains what you expect. Then run a checker that reads your file and reports on each AI user agent, so that you can see allowed or blocked status for the bots in one place without reading the rules by hand.

Allow time for the change to take effect. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust, so do not expect an instant change. To confirm real behaviour, look at your server logs for the user agent names over the following days. Anthropic publishes its crawler IP addresses at claude.com/crawling/bots.json, so you can confirm that a request claiming to be its crawler really is.

Anthropic also advises against blocking by IP address as an opt-out method, because it can stop the crawler reading your robots.txt in the first place. Use the file itself for your policy, and keep any network-level blocking for abusive traffic.

Making the decision that suits your site

There is no single right answer. A publisher that depends on direct visits may want to opt out of training while staying visible in AI search. A business that wants to be recommended by assistants may choose to allow everything. What matters is that the outcome reflects a decision rather than an accident.

Write down your policy in a sentence, such as allow search bots and user fetchers, block training bots. Then translate it into the file, check it with a tool, and review it every few months, since operators add and rename user agents over time. This guide reflects documentation as it stood in October 2026, so confirm any name against the operator's page before you rely on it.

このガイドで紹介したツール

よくあるご質問