Robots.txt Generator

How to use it

1. Start from a preset (allow all, block all, block admin and API, or block AI crawlers), or from an empty group.
2. For each group, choose the crawlers (user-agents) and add Allow or Disallow paths that start with “/”.
3. Add your sitemap URLs as full https:// addresses.
4. Review the warnings, and check important URLs with “Test a URL path”.
5. Copy or download robots.txt and upload it to the root of your site, for example https://www.example.com/robots.txt.

How rules are written and tested

• Each group lists its user-agents and then its rules. A group with no rules gets “Allow: /”, because crawlers join consecutive user-agent lines into one group (RFC 9309).
• Paths must start with “/” or “*”. Spaces and non-ASCII characters are percent-encoded, and “#” isn't allowed because it starts a comment. Invalid lines are left out of the file until fixed.
• The tester follows Google's documented rules: a crawler uses the group that names it (several groups for the same crawler are combined), otherwise the * group. Among the rules that match, the longest path wins and Allow wins a tie. * matches any characters, $ marks the end of the URL, and paths are case-sensitive.
• /robots.txt itself is always allowed, and a path that no rule matches is allowed.
• Crawl-delay is written if you set it, but Google ignores it. Sitemap lines must use absolute URLs.
• The file is generated in your browser. This tool doesn't fetch your live robots.txt or any other website.