How to Create a Robots.txt That Won't Break Your SEO
The robots.txt file is one of the smallest files on a website and one of the easiest to get catastrophically wrong. A single misplaced line can tell search engines to ignore your entire site, and because the file works silently, the damage can go unnoticed until traffic disappears. Understanding what it actually does — and, crucially, what it does not do — is the key to using it safely.
What robots.txt does
Robots.txt is a plain text file that lives at the root of your domain, at yourdomain.com/robots.txt. It is the first thing most well-behaved crawlers request when they visit, and it contains instructions about which parts of your site they may crawl. Think of it as a polite notice at the entrance rather than a locked door: cooperative bots like Googlebot obey it, but it is a set of directions, not an enforcement mechanism. You can build a valid one with the robots.txt generator, which assembles the rules correctly and lets you download the file.
Crawling is not indexing
This is the single most important and most misunderstood point about robots.txt, so it deserves its own section. Crawling is when a search engine fetches and reads a page. Indexing is when it stores that page and makes it eligible to appear in search results. Robots.txt controls crawling, not indexing.
noindex meta tag on the page, so the crawler can read the instruction not to index it.This is exactly backwards from most people's intuition, and it causes a lot of confusion. The rule to remember: use robots.txt to manage crawl budget and keep bots out of pointless areas; use the noindex meta tag (covered in our Meta Tags Checklist 2026) to keep pages out of search results.
The syntax that matters
The file is organised into groups. Each group starts with a User-agent line naming the crawler the rules apply to — * means all crawlers. Under it, Disallow lines list paths bots should not crawl, and Allow lines carve out exceptions. A Disallow: /admin/ keeps crawlers out of everything under the admin folder; an empty Disallow: means nothing is blocked. Paths are matched from the start of the URL, so /admin blocks /admin, /admin/, and anything beneath. You can also add a Sitemap: line pointing to your XML sitemap, which helps crawlers discover all your URLs.
Mistakes that break SEO
The most devastating mistake is shipping a development site's robots.txt to production. Staging sites often use Disallow: / — which blocks the entire site — and if that file goes live, search engines are told to crawl nothing. Sites have vanished from Google for exactly this reason. Always check that your live robots.txt does not contain a blanket disallow unless you genuinely mean it.
Other common errors include blocking your CSS and JavaScript folders (which prevents Google from rendering your pages properly and can hurt ranking), using robots.txt to try to hide sensitive pages (it does the opposite by listing them publicly for anyone to read), and typos in paths that silently fail to match. Because the file is public, never treat it as a security measure — anyone can read exactly which directories you tried to hide.
A safe starting template
For most sites, a permissive robots.txt is the right choice. Allow everything, block only the genuinely useless areas like internal search results or cart pages, and point to your sitemap. A sensible starting point is: User-agent: * on one line, an empty Disallow: to permit full crawling, and a Sitemap: line with your sitemap URL. Add specific Disallow rules only for paths you are certain should not be crawled, and always keep CSS and JS accessible.
The safest habit is to generate the file deliberately rather than copy one from elsewhere, review every line, and re-check it whenever you launch or migrate a site. A correct robots.txt is nearly invisible — it quietly guides crawlers to your good content and away from clutter, without ever standing between your pages and the search results they deserve.
Frequently asked questions
Robots.txt is powerful precisely because it is trusted and obeyed, which means a mistake is obeyed too. Remember that it controls crawling and not indexing, never block your assets, never rely on it for privacy, and double-check for a stray Disallow: / before every launch.