Robots.txt Tester
Paste your robots.txt, choose a crawler name and list the paths or addresses to test. Each is marked allowed or blocked, with the rule that decided it.
Dieser Inhalt ist auf Englisch.
Durchgerechnete Beispiele
A blocked section
/admin/page
/admin/page: blocked by Disallow: /admin/
The Googlebot group applies, and Disallow: /admin/ matches.
An allowed exception
/admin/public/x.html
/admin/public/x.html: allowed by Allow: /admin/public/
The Allow rule is longer than the Disallow rule, so it wins.
Über Robots.txt Tester
This tester shows how a robots.txt file treats paths. Paste the file, enter a crawler name, and list the paths or addresses you want to test. Each one is marked allowed or blocked, with the rule that decided it.
The most specific group for the crawler name is used, with the group for * as a fallback. The longest matching rule wins and Allow wins a tie, with * and $ supported as wildcards. The tool tests the text you paste and does not fetch a live file.
So verwenden Sie Robots.txt Tester
- Paste the contents of your robots.txt file.
- Type the crawler name, such as Googlebot, or * for the default group.
- List the paths or full addresses to test, one per line, and read the result.
Typische Einsatzzwecke
- Checking a rule before you publish it
- Making sure an important page is not blocked
- Debugging why a crawler skips a section
- Explaining the effect of Allow and Disallow to a colleague
Häufige Fragen
How is the winning rule chosen?
The longest matching path wins, because it is the most specific. If an Allow and a Disallow match with the same length, Allow wins.
What do * and $ mean?
An asterisk matches any run of characters, and a dollar sign at the end of a rule matches only the end of the address.
What if no group names my crawler?
The group for * is used. If there is no group for either, everything is allowed.
Does blocking in robots.txt keep a page out of search results?
Not always. It stops crawling, but a blocked page can still be listed if other pages link to it. Use a noindex tag to keep a page out of results.