Technical SEO utility · Free browser-based tool
Robots.txt
generator.
Create a robots.txt file from clear, validated rules. Add crawler directives, allow and disallow paths, and sitemap locations, then copy or download the result.
Robots.txt configuration
Define a crawler group and generate a plain-text robots.txt file. Review every rule against your site's actual URL structure before publishing.
Use * for all compliant crawlers, or a crawler token such as Googlebot.
Not supported by Google. Other crawlers may interpret it differently.
Paths should begin with /. You can use * and $ pattern characters.
Avoid blocking resources that search engines need to render pages.
Use absolute HTTP or HTTPS sitemap URLs.
Generated file
Review before deploying to the site root.Important: robots.txt is a crawl-control convention, not access control and not a reliable deindexing method. Blocked URLs can still appear in search results. Use authentication for private content and an appropriate noindex strategy for pages that must not be indexed.