How to check your sitemap and robots.txt

Ta treść jest po angielsku.

Search engines find your pages by following links and by reading two simple files. A sitemap lists the pages you want indexed. A robots.txt file tells crawlers which parts of the site they may visit. When either file is missing, wrong or out of date, important pages can be missed, or pages you never wanted public can appear in search results. This guide explains what each file does, how to check both with the sitemap and robots checker on Valtrinity, and how to fix the mistakes that most often cause problems.

Autor: Valtrinity. Zaktualizowano: 18 października 2026.

What a sitemap does

A sitemap is an XML file that lists the web addresses you want search engines to know about. It can also record when each page was last changed, which helps crawlers decide when to come back. A sitemap does not guarantee that a page will rank, but it makes discovery easier, particularly for large sites, new sites and pages that few other sites link to.

Most content management systems can generate a sitemap automatically. Check that the generated file includes the pages you care about and leaves out pages you do not, such as thank-you pages or internal search results.

• A sitemap lists pages you want found.
• It helps crawlers discover new or poorly linked pages.
• Check it includes the right pages and leaves out the rest.

What robots.txt does

The robots.txt file sits at the root of your domain, so the address is your domain followed by /robots.txt. It contains rules that tell crawlers which paths they may or may not visit. A line that begins with Disallow asks crawlers to stay out of a path, and a line beginning with Sitemap names the location of your sitemap.

Robots.txt is a request, not a lock. Well-behaved crawlers follow it, but it does not hide pages from people, and it does not stop a page appearing in search if other sites link to it. To keep a page out of search results, use a noindex instruction on the page itself.

• The file must be at the domain root.
• Disallow asks crawlers to skip a path.
• Sitemap points crawlers at your sitemap.
• It is a request, not a lock.

Check both files with the sitemap and robots checker

Open the sitemap and robots checker on Valtrinity and enter your domain. It reads the robots.txt file and the sitemap, and reports whether each exists, whether the sitemap is linked from robots.txt and whether any rules block pages that appear in the sitemap. The sitemap and robots checker is part of the Unlimited plan.

Read the result in order. A missing robots.txt is not always a problem, because crawlers will assume everything is allowed. A missing sitemap is more significant on a large site, where crawlers rely on it to find deep pages.

• Confirm robots.txt exists and loads.
• Confirm the sitemap exists and is linked from robots.txt.
• Look for rules that block pages in the sitemap.

Worked example: a rule that hides the whole site

A common mistake is copying a robots.txt from a development server. Suppose the live site contains the two lines User-agent: * and Disallow: /. Read literally, that tells every crawler to stay out of every page. Pages can drop out of search results over time, and the sitemap will list pages that the rules forbid.

The fix is to replace the rule with one that blocks only what you want hidden, such as Disallow: /admin/, and to make sure the Sitemap line points at the correct address. Then run the checker again. It should now show the sitemap pages as allowed.

• Disallow: / blocks the whole site.
• Block specific paths such as /admin/ instead.
• Recheck the sitemap pages after the change.

Common mistakes to look for

The sitemap should use the same address form as your site. If the site redirects www to the bare domain, the sitemap should list the final address, not the redirect. Mixing http and https addresses, or listing pages that return errors, makes the sitemap less useful. Remove those pages or fix them before you resubmit the sitemap.

Another common problem is a sitemap that is never updated. If new pages are added but the sitemap stays the same, crawlers may find those pages later than you would like. Most systems regenerate the sitemap automatically, so check the setting rather than editing the file by hand.

• Use the final address form, after redirects.
• Do not list error pages or redirects.
• Check the sitemap regenerates when pages change.

Keep both files in step after changes

Check robots.txt and the sitemap after any site redesign, domain change or hosting move. These are the moments when rules are most likely to be copied from the wrong place. A quick check after each change takes minutes and prevents weeks of lost visibility.

• Check after redesigns, domain changes and hosting moves.
• Keep a note of the last time each file was checked.

Narzędzia wspomniane w poradniku

Najczęstsze pytania