AI Crawler Checker
Check how a site's robots.txt treats GPTBot, ClaudeBot, Google-Extended, Bingbot and more, for the homepage and any page you choose, and copy a ready-made rule set.
Deze inhoud is in het Engels.
Uitgewerkte voorbeelden
Rules for a named crawler
https://example.com typed in, with a robots.txt that blocks ExampleBot from the whole site
(Illustrative) ExampleBot is disallowed from / by the most specific group. No noai or noindex instruction found in the headers or page tags.
The most specific group wins, and robots.txt is a request that well-behaved crawlers follow.
Page asking not to be indexed
https://example.com/offer typed in, with a noindex tag in the page
(Illustrative) A noindex instruction found in the page tag, which asks search engines not to list this page.
The instruction is reported even if robots.txt allows the crawler to visit the page.
Over AI Crawler Checker
AI companies and search engines send automated crawlers to read websites, and your robots.txt file is where you tell them what they may fetch. Some crawlers collect pages to train AI models, some power AI search answers, some open a page only when a person asks, and others simply build a search index. It is easy to block too much or too little without realising. This tool reads your robots.txt and tells you, crawler by crawler, whether each one is allowed or blocked. It is a quick way to see your current position before you decide on any change, rather than editing the file blind and hoping for the best.
Enter a website and, if you like, a page path to test as well as the homepage. The rules are applied the way search engines apply them: the most specific user agent group wins, the longest matching Allow or Disallow rule wins, wildcards and end-of-path markers are understood, and Allow beats Disallow on a tie. We also report noindex and noai instructions sent in headers or page tags, and give you copy-ready rules to block or allow AI crawlers. Because the result lists every crawler side by side, it is easy to spot a rule that blocks more than you meant it to.
Zo gebruikt u AI Crawler Checker
- Enter your website address.
- Optionally add a page path such as /blog/my-post to test it as well as the homepage.
- Select Check crawlers.
- Read the table to see which crawlers are allowed or blocked and which rule decided it.
- Copy one of the ready-made rule sets into your robots.txt if you want to change the result.
Veelvoorkomend gebruik
- Deciding whether to opt out of AI training without hiding from search
- Checking that Googlebot and Bingbot can still reach your pages
- Testing whether a specific page or section is blocked
- Finding out why a page is not appearing in AI search answers
- Auditing a client site's robots.txt before a launch
Veelgestelde vragen
Will blocking AI crawlers affect my search ranking?
Blocking AI training crawlers such as GPTBot, ClaudeBot or CCBot does not affect normal search engines. Blocking Google-Extended also leaves Google Search alone. Blocking Googlebot or Bingbot would, so check the search engine rows before copying any rules. The ready-made rule sets here never touch search engines.
What is the difference between the crawler types?
Training crawlers collect pages that may be used to train AI models. Search crawlers find pages to show in AI search results. Assistant crawlers fetch a page only when a person asks an AI tool to. Search engines build the usual search index. You can allow some types and block others.
How are conflicting rules decided?
The crawler uses the group that names it most specifically, falling back to the star group if none does. Within that group the longest matching rule wins, so a long Allow can override a shorter Disallow. If an Allow and a Disallow match equally, Allow wins. Wildcards and a dollar sign at the end are supported.
Does robots.txt actually stop a crawler?
Only if the crawler chooses to obey it. Reputable crawlers do, but robots.txt is a request and not a lock. Some crawlers have been reported to ignore it. If you need to keep content from anyone, use passwords or server rules rather than relying on robots.txt alone.
What if my site has no robots.txt?
Then crawlers treat everything as allowed, and the table will show every crawler as allowed. If the file returns a server error rather than a simple not found, crawlers are told to assume everything is off limits until it works again, which is flagged in the result.
What are noindex and noai tags?
Noindex asks search engines not to list a page, and can be sent in a page tag or an X-Robots-Tag header. Noai and noimageai are informal hints some sites use to opt out of AI use, but there is no standard and many crawlers ignore them. We show any we find on the homepage and your chosen page.