Honest note: only crawlers that choose to obey robots.txt will respect this.
One group per crawler, rules as rows. The file updates live as you type. A group with an empty Disallow means “allow everything”. Click a Disallow / Allow badge to switch it.
− lines are live on the site but not in your file. + lines are in your file but not live yet.
| Path | Result | Reason |
|---|
Enter any site address and we fetch its current /robots.txt and break it into groups. (This checks a live file — it does not read the builder above.)
Groups
Rules
Sitemaps
| Crawler | Owner | Status | Block |
|---|
Build a robots.txt file visually: pick a crawler, add Allow and Disallow rules, set a crawl delay, then copy or download the result. No syntax to memorise.
100% free · No sign-up · No daily limits · No watermarks
Build a robots.txt file visually: pick a crawler, add Allow and Disallow rules, set a crawl delay, then copy or download the result. No syntax to memorise.
📖 From the blog: Creating a robots.txt file: rules, templates and the mistakes that block Google
To keep every bot out of a checkout while still letting them see the shop around it: one group for all crawlers with Disallow: /checkout/ and Disallow: /account/, a second group for GPTBot with Disallow: / to keep AI training crawlers out entirely, and the sitemap line at the end. The generated file keeps the groups in the order you built them.
robots.txt is a plain-text protocol: groups of user-agent lines followed by Allow, Disallow and Crawl-delay rules, with the sitemap line independent of any group. The generator assembles exactly that text from your rows, live as you type: an empty group with no rules means "allow everything", and in real crawlers the most specific matching rule wins. The optional checker fetches an existing /robots.txt through our protected fetcher — robots files are plain text, which the fetcher accepts — and breaks it into its groups for review.
At the root of your host: https://example.com/robots.txt. It must be a plain-text file at exactly that address — one per host, including separate ones for subdomains.
Disallow stops crawling; noindex (a meta tag or header) stops indexing but requires the crawler to load the page. To remove a page from results, allow crawling and use noindex — blocking it in robots.txt hides the noindex from the crawler.
Yes — the Block AI crawlers panel does it in one click per bot, adding a group naming the AI crawler (for example GPTBot or ClaudeBot) with Disallow: / as its only rule; you can also add such a group by hand. Compliant AI crawlers honour it, but crawlers that ignore robots.txt will not be stopped by it.
It asks a crawler to wait that many seconds between requests. Google ignores it (use Search Console’s settings there), but Bing, Yandex and many smaller bots respect it. Leave it empty unless a bot is overloading you.
The file may genuinely be absent (every crawler then defaults to "everything allowed"), the site may have answered our fetch with an error, or the file may be served under a different host. Check the address in a browser to confirm.
It means allow everything. A group with no rules, or a Disallow with nothing after it, places no restriction on that crawler. If you meant to block a folder, the path has to be there — for example Disallow: /checkout/ — and paths are matched as prefixes starting with a slash.
No. robots.txt is a public request, not a lock: well-behaved crawlers obey it and others can ignore it, and anyone can read the file itself. Never list secret paths you have not otherwise secured, and use real access control for anything private.
What is missing, broken, or could be better on this tool? Tell us — we read every suggestion.
✅ Thanks — your suggestion has been received. Suggestions like yours shape what we build next.