robots.txt and Sitemap Generator
Build a robots.txt file with block and allow rules and optional blocks for AI crawlers, or turn a list of page addresses into an XML sitemap. Both are checked for common mistakes and ready to download.
One path on each line, such as /admin/. These are kept out of search.
Paths that stay open inside a blocked path.
One address on each line. Each starts with https://.
23 characters
On this device
No signup. Your input is never uploaded.
How to use the robots.txt and Sitemap Generator
Choose robots.txt or XML sitemap.
For robots.txt, list the paths to block and allow, and any sitemap addresses.
For a sitemap, paste your page addresses, one on each line.
Check the tips, then copy or download the file.
Questions about the robots.txt and Sitemap Generator
At the top of your site, so that it opens at yoursite.com/robots.txt. Crawlers look for it there, and only there.
It asks them not to. Well-behaved crawlers such as GPTBot and ClaudeBot follow it, but robots.txt is a request and not a lock, and companies rename their crawlers. The tool can add blocks for the common ones.
Not always. robots.txt stops a page being crawled, but Google can still list a blocked address it finds elsewhere. To keep a page out of results, use a noindex tag on a page that crawlers are allowed to reach.
Up to 50,000 addresses in one sitemap file. For more, split them into several sitemaps and list those in a sitemap index.