Sitemap & Robots.txt Checker
Enter a website to read its robots.txt rules, find its sitemaps and check that they are valid and that the pages they list actually load.
Questo contenuto è in inglese.
Esempi svolti
Healthy robots.txt and sitemap
https://example.com typed into the box
(Illustrative) robots.txt found, sitemap listed inside it, and the home page allowed for every crawler.
The sitemap is read from robots.txt and from the usual location, so both are compared.
Whole site blocked
https://example.com typed into the box, with robots.txt containing Disallow: /
(Illustrative) The home page is blocked for every crawler, shown as the top item in the to-do list.
A single rule like this can hide the whole site, so it is flagged first.
Informazioni su Sitemap & Robots.txt Checker
Robots.txt and the XML sitemap are the two files that tell search engines where they may go and which pages matter. A small mistake in either can hide a whole site, for example a single line that blocks everything, or a sitemap full of addresses that no longer exist. This checker fetches both files and reports exactly what it finds. For robots.txt it lists the rules for each crawler, flags a blanket block, a missing sitemap line, misspelt fields and rules placed in the wrong order, and tells you whether the home page is allowed.
It then discovers sitemaps from the robots.txt file and the usual location, and reads the sitemap or sitemap index. Up to three child sitemaps are opened and the first 200 addresses of each are examined. The XML shape, number of addresses, date formats, duplicates, unescaped characters, the standard namespace and host matching are all validated, and ten addresses are requested to see whether they load, redirect or fail. Everything comes back as findings in priority order, each with a plain instruction for fixing it, so the next step is always clear.
Come usare Sitemap & Robots.txt Checker
- Enter your domain or any address on the site.
- Select Check files and wait while both files and the sitemaps are read.
- Work through the high priority findings, which are the ones most likely to affect how the site is crawled.
- Fix the files on your site and run the check again to confirm.
Usi comuni
- Checking a site after launch to make sure nothing is blocked by mistake
- Confirming that a sitemap is valid before submitting it to a search engine
- Finding sitemap entries that redirect or return errors
- Reviewing the rules a site applies to different crawlers
- Auditing a client site as part of a technical review
Domande frequenti
What does Disallow: / mean?
It tells a crawler not to visit any page on the site. Under User-agent: * it applies to every crawler that follows the rules, so it will keep the site out of search results over time. It is common on test sites and sometimes left behind by accident at launch, which is why the checker marks it as a high priority problem.
Does a site need a robots.txt file?
No. If there is none, crawlers assume that everything may be crawled, and the checker reports this as a low priority note. It is still good practice to have one, mainly so you can point to your sitemap with a Sitemap line and keep crawlers away from pages that have no value in search, such as internal search results.
Where should the sitemap be listed?
Add a line such as Sitemap: https://example.com/sitemap.xml to robots.txt, using the full address. You can list several sitemaps. Many sites also keep the sitemap at the root, in a file called sitemap.xml, and this checker looks there as well as in robots.txt, and tries sitemap_index.xml if nothing else is found.
Which sitemap problems matter most?
The most serious are a file that is not valid XML, a sitemap that returns an error or a web page instead of XML, and addresses that are broken. Redirecting addresses, wrong date formats, duplicates and addresses on a different host are less serious but still worth cleaning up, because they reduce how much search engines trust the file.
Why are only some addresses checked?
To keep the check quick, up to three child sitemaps are opened, the first 200 addresses of each are read, and ten addresses spread across them are requested. A sample like this is a good signal of overall health. If it finds problems, fix those and check again, as the same cause usually affects many other entries.
What is a sitemap index?
It is a sitemap that lists other sitemaps instead of pages. Large sites use one because a single sitemap is limited to 50,000 addresses. The checker reads the index, then opens up to three of the sitemaps it lists and reports how many more there are, so very large sites are summarised rather than fully crawled.