Robots.txt Generator
Create a valid robots.txt with allow/disallow rules and a sitemap reference — then download it.
Robots.txt Generator
🔒 Nothing you enter is sent anywhere or stored. All processing happens in your browser.
How it works
A robots.txt file sits at the root of your domain and tells search-engine crawlers which parts of your site they may or may not request. It's the first file most bots fetch, so a correct one keeps crawlers focused on your important pages and away from admin areas, carts, or duplicate paths.
Each rule block starts with a User-agent line naming the crawler (* matches all of them), followed by Allow and Disallow paths. A disallowed path blocks crawling of anything beneath it; an allow rule can carve out an exception. You can also add an optional Crawl-delay to slow aggressive bots, and a Sitemap line pointing crawlers to your XML sitemap.
This generator assembles those pieces into a syntactically valid file you can copy or download. A word of caution built into good practice: robots.txt controls crawling, not indexing — a blocked page can still appear in results if other sites link to it, so use a noindex meta tag when you truly want a page kept out of search. Everything is generated in your browser.