Robots.txt Generator
Choose a preset, add your own rules and sitemap, then copy or download a file that is ready to upload to the root of your site.
Worked examples
Block admin and add sitemap
All crawlers, disallow /admin/, sitemap https://example.co.uk/sitemap.xml
User-agent: * Disallow: /admin/ Sitemap: https://example.co.uk/sitemap.xml
The file asks well-behaved crawlers to skip the admin area; it is a request, not a lock, so it does not hide that area from visitors.
Ask one crawler to stay out
One named crawler, ExampleBot, disallow the whole site
User-agent: ExampleBot Disallow: /
Naming a single crawler limits the rule to that crawler, and the other crawlers follow the rules written for them.
About Robots.txt Generator
A robots.txt file sits at the top of your website and tells crawlers which areas they should not visit. Search engines read it before they crawl, so a few correct lines can keep admin areas, search result pages and duplicate content out of the way and point crawlers to your sitemap.
This generator gives you a clean starting point. Pick a preset for an open site, a closed site, a WordPress site or a site that allows search engines but asks AI training crawlers to stay away, then add your own rules and sitemap address. The file updates as you type and you can copy it or download it ready to upload.
How to use Robots.txt Generator
- Choose the preset that is closest to what you want.
- Add your sitemap address so crawlers can find all your pages.
- Type any extra rules, one per line, starting with Allow, Disallow or Crawl-delay.
- Copy the result or download robots.txt.
- Upload the file to the top folder of your site and test that it loads at yoursite.com/robots.txt.
Common uses
- Keeping admin and search result pages out of search engines
- Stopping AI training crawlers while staying visible in search
- Pointing crawlers to a sitemap
- Blocking a staging site completely while it is being built
Frequently asked questions
Does robots.txt hide pages from the public?
No. It is a polite request that well-behaved crawlers follow, but anyone can still open a blocked page and the file itself is public. Use a password or proper access controls for anything private.
Where does the robots.txt file go?
It must be in the top folder of the site so that it loads at the root address, for example example.com/robots.txt. A copy placed in a sub folder is ignored by crawlers.
Will blocking AI crawlers remove my pages from search?
The AI preset blocks named crawlers that collect content for training or answers, while leaving search engines such as Google and Bing alone. It does not remove pages from a search index and it only affects crawlers that obey the file.
Can a robots.txt file remove a page that is already in Google?
Not reliably. Blocking a page stops crawling but a page can stay in results if other sites link to it. To remove a page use a noindex tag or the removal tool in the search engine's own webmaster service.
What does Crawl-delay do?
It asks some crawlers to wait a number of seconds between requests. Google ignores it, but some other search engines respect it. Use it only if a crawler is putting too much load on your server.