To create a robots.txt file, open the free robots.txt generator, start from a template that matches your kind of site, adjust the User-agent, Disallow and Allow rules, add a Sitemap line pointing at your sitemap, then save the result as a plain text file named robots.txt at the root of your domain, so it loads at yourdomain.com/robots.txt. That is the whole build. The care belongs in what the rules say, because this small file steers how search engines crawl your site — and one careless line can steer them away from pages you need indexed.
Robots.txt is one of the oldest files on the web and still one of the most misunderstood. Site owners regularly ask it to do jobs it cannot do — hide a page from search results, protect private content, fix a rankings problem — and then blame the file when nothing changes. Used for its real job, crawl management, it is genuinely valuable: it keeps crawler time focused on pages that matter, keeps staging and admin areas out of the crawl queue, and tells every well-behaved crawler where your sitemap lives. This guide explains what the file can and cannot do, decodes the syntax in plain language, gives you three starter templates you can adapt today, and finishes with the five mistakes that most often block Google by accident.
What robots.txt Can and Cannot Do
The single most important distinction is this: robots.txt controls crawling, not indexing. A Disallow rule tells a crawler not to fetch a URL. It does not order search engines to remove that URL from their index, and it does not guarantee the URL stays out. If other pages link to a disallowed URL, a search engine can still list it — usually shown with a note that no page information is available, because the crawler never fetched the content. Owners who disallow a page hoping it will vanish from results are often surprised to find it listed anyway, now with an unhelpful snippet. Keeping a page out of the index is a different job, handled by a noindex instruction on the page itself or by password protection for genuinely private material.
| Job | Can robots.txt do it? | What to use instead, or as well |
|---|---|---|
| Stop crawlers wasting time on admin, search-results or filter pages | Yes — this is its core job | Nothing else needed; a Disallow rule is enough |
| Point crawlers to your sitemap | Yes | Also submit the sitemap in Search Console |
| Remove a page from search results | No | A noindex tag or header on the page, or delete the page |
| Hide private or sensitive content | No — the file itself is public, and it advertises the paths you list | Login protection, passwords, server access rules |
| Stop every bot on the internet | No — it is a voluntary standard | Server firewall rules and bot management for bad actors |
| Fix thin content or ranking problems | No | Better content and internal linking; crawling is not ranking |
Two consequences follow from that table. First, never list anything secret in robots.txt: anyone can open yourdomain.com/robots.txt in a browser, so a Disallow line for a private folder is effectively a signpost to it. Second, because the standard is voluntary, well-behaved crawlers such as Googlebot follow it, while scrapers and malicious bots may ignore it entirely. Treat the file as traffic direction for welcome visitors, never as a lock on the door.
The Syntax in Plain Language
A robots.txt file is plain text, one instruction per line, built from a handful of directives. User-agent names which crawler a group of rules applies to; an asterisk means every crawler. Rules that follow a User-agent line belong to that crawler until the next User-agent line begins a new group. Disallow names a path that crawler should not fetch, and Allow carves an exception back out of a disallowed area — useful when one useful file lives inside a blocked folder. Paths are relative to the domain root, so Disallow: /checkout/ covers the checkout path on every page under it. Two pattern characters do heavy lifting: the asterisk matches any sequence of characters inside a path, and the dollar sign anchors a match to the end of a URL, which is how you block URLs ending in a particular way without catching longer names that merely contain the same letters.
The Sitemap line sits outside any User-agent group and gives the full URL of your sitemap file. You may list more than one if your site splits its sitemap into an index of smaller files. If you do not have a sitemap yet, build one first with the free sitemap generator and then reference it here — the two files work as a pair, one saying what to skip, the other listing what matters most. Blank lines and lines beginning with a hash character are treated as spacing and comments, so you can annotate your file for the next person who edits it. That future reader will thank you: a robots.txt file written six months ago, with no comments, is a puzzle nobody wants to solve under pressure.
Practical tip: edit a copy, never the live file. Draft your changes in the generator, read every line aloud as an instruction to a crawler, and only then replace the live file. Most robots.txt disasters are one-line edits made directly on a production site.
How to Create Your File, Step by Step
- Start from the closest template. Open the robots.txt generator tool and pick the starter that matches your site — blog, store, or a full block for a staging copy — rather than writing from a blank page.
- Set the default group. Keep User-agent: * as your main group unless you have a specific crawler to treat differently. One clear default beats five half-remembered exceptions.
- Add your Disallow lines. Cover the areas crawlers gain nothing from: admin screens, internal search results, checkout and account paths, print views, and tracking-parameter duplicates your platform creates.
- Add Allow exceptions. If a blocked folder holds a file crawlers genuinely need — a stylesheet the whole site depends on is the classic case — allow that exact path explicitly.
- Add the Sitemap line. Paste the full sitemap URL, including https, at the end of the file.
- Save with the exact filename. The file must be named robots.txt, in plain text, uploaded to the domain root. A file named robots.txt.txt, or one sitting in a subfolder, is invisible to crawlers.
- Test before you trust it. Follow the testing routine in the next section while the old version is still fresh in your mind.
Three Starter Templates You Can Adapt
Template 1 — a typical WordPress site. This keeps crawlers out of the admin area while explicitly leaving the admin's AJAX endpoint reachable, because some front-end features rely on it:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://example.com/sitemap.xml
Template 2 — an online store. Stores generate endless low-value URLs: internal search results, filtered category views, checkout and account pages. Blocking those families concentrates crawling on products and categories:
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /?s= Disallow: /*?orderby= Disallow: /*&orderby= Sitemap: https://example.com/sitemap.xml
Template 3 — block-everything staging. A staging or development copy of your site should never be crawled or indexed, and the entire file is two lines:
User-agent: * Disallow: /
That last template carries the sharpest warning in this guide: a block-everything file that migrates from staging to the live site — bundled into a deployment by someone moving files in bulk — takes the whole site out of the crawl queue until somebody notices. Add a launch-day check for this exact accident to your process; our small business SEO checklist includes it among the pre-launch basics for the same reason. The chart below compares the three templates by size, which is a fair picture of their intent: the staging file is tiny because it says one thing, while the store file is longest because a store has the most crawl traps to close.
Illustrative comparison: lines of rules in the three starter templates above — hover or tap each bar for details.
Adapt, do not adopt blindly. Your store platform may generate different parameter patterns from the examples above; your WordPress setup may add plugin folders worth blocking or worth leaving alone. The right method is to look at the URLs your own site produces — your sitemap, your internal search, your filtered category pages — and write rules for the site you actually run. While you are reviewing crawl-facing files, it is worth checking the other signals search engines read before your content: our guide to checking and fixing meta tags pairs naturally with this one, and the Meta Tag Analyzer shows you exactly what a crawler sees in your page head.
Testing the File in Search Console
Publishing an untested robots.txt file is like posting a detour sign without driving the road. Work through four checks. First, open yourdomain.com/robots.txt in a private browser window and confirm the live file is the one you wrote — caching and upload mistakes are common, and a surprising number of sites serve an ancient default. Second, use the robots.txt testing facility available through Google Search Console and its linked testing tools to validate the file's syntax and to test specific URLs: paste in an important product page and confirm it reports allowed, then paste an admin URL and confirm it reports blocked. Third, test one URL from each family you intended to block and one from each family you intended to keep — templates fail at the edges, not the middle. Fourth, over the following days, watch Search Console's coverage and crawl reporting for unexpected drops in crawled pages, which are often the first symptom of an over-broad rule nobody caught in testing. Keep a saved copy of the previous working version every time you edit, so a bad change is a two-minute rollback rather than an investigation.
Five Mistakes That Block Google by Accident
- Blocking CSS and JavaScript. Search engines render pages to judge them, and a page whose stylesheets and scripts are disallowed renders as broken text. If your rules block the folders holding those assets, rankings can suffer for reasons that look like a design problem. Allow the asset paths your pages need.
- Using robots.txt to hide pages. As covered above, disallowed pages can still appear in results, and the file publicly lists what you hid. Use noindex for result removal and real access control for anything private.
- Wrong wildcards. A stray asterisk turns a narrow rule into a site-wide one: Disallow: /*.pdf$ blocks PDF files, while a misplaced pattern can match every URL containing a common letter sequence. Test wildcard rules against real URLs before publishing them.
- The staging file reaching production. The two-line block-everything template is correct on staging and catastrophic live. Separate the environments' files, and check robots.txt as a named step in every launch.
- Forgetting the file after a redesign. New platforms create new admin paths, new search URLs and new parameter patterns. A robots.txt written for the old site silently mis-serves the new one. Re-audit the file whenever the platform, theme or URL structure changes.
Conclusion: Small File, Written Carefully
A good robots.txt file is short, commented, tested and boring. It blocks the crawl traps, allows everything the rendering process needs, points to the sitemap, and never pretends to be a privacy tool or an indexing switch. Build yours from the closest template, test it in Search Console before you trust it, and re-read it after every site change. The ten minutes that routine takes is cheap insurance against the quietest way a site can lose search traffic: telling Google, in writing, to stay away.
Crawl problems, or a site search engines cannot read properly? Book a free consultation with Asad Webs — tell us what your Search Console is showing and we will reply with honest next steps for your technical SEO, with no obligation.
Frequently Asked Questions
Where does the robots.txt file belong?
At the root of your domain, so it loads at yourdomain.com/robots.txt. Crawlers only look for it in that one location, using the exact filename robots.txt, served as plain text. A copy inside a subfolder, or under a slightly different name, is never found and has no effect at all.
Does Disallow remove a page from Google search results?
No. Disallow stops crawlers fetching the URL; it does not remove the URL from the index. A disallowed page that other sites link to can still appear in results, usually without a useful description. To keep a page out of results, use a noindex instruction on the page itself, or protect it with a login.
What is the difference between robots.txt and a noindex tag?
They work at different stages. Robots.txt speaks before crawling: it says whether a crawler may fetch a URL at all. A noindex tag speaks after crawling: the crawler fetches the page, reads the instruction, and keeps the page out of the index. Note the interaction — if a page is disallowed, the crawler never fetches it, so it never sees a noindex tag on it.
How do I block a staging site completely?
Use the two-line block-everything file: User-agent: * followed by Disallow: / on the staging domain. That instructs every well-behaved crawler to fetch nothing. Keep that file strictly on the staging environment, and verify after every launch that the live site is not serving it.
Can I have different rules for different crawlers?
Yes. Each User-agent line starts a group of rules for that crawler, so you can allow one bot everywhere while restricting another to part of the site. Keep the asterisk group as your default, and add specific groups only when you have a clear reason — every extra group is another set of rules to test and maintain.
What does the Sitemap line do, and can I list more than one sitemap?
The Sitemap line tells crawlers where your sitemap file lives, using its full URL. You can include several Sitemap lines if your site publishes multiple sitemaps or a sitemap index. Listing the sitemap here complements submitting it in Search Console; doing both costs nothing and covers crawlers that only read the file.
How often should I review my robots.txt file?
Review it after any platform change, redesign, or URL-structure change, and give it a quick read a few times a year even when nothing has changed. The review is fast: confirm the live file matches your saved copy, confirm your most important pages test as allowed, and confirm no staging rules or stale paths have crept in.
Be the first to share a useful question or thought.