Build your robots.txt group by group — one group per crawler, rules as rows. The file updates live as you type. A group with an empty Disallow means “allow everything”.
Enter any site address and we fetch its current /robots.txt and break it into groups. (This checks a live file — it does not read the builder above.)
Build a robots.txt file visually: pick a crawler, add Allow and Disallow rules, set a crawl delay, then copy or download the result. No syntax to memorise.
Build a robots.txt file visually: pick a crawler, add Allow and Disallow rules, set a crawl delay, then copy or download the result. No syntax to memorise.
To keep every bot out of a checkout while still letting them see the shop around it: one group for all crawlers with Disallow: /checkout/ and Disallow: /account/, a second group for GPTBot with Disallow: / to keep AI training crawlers out entirely, and the sitemap line at the end. The generated file keeps the groups in the order you built them.
robots.txt is a plain-text protocol: groups of user-agent lines followed by Allow, Disallow and Crawl-delay rules, with the sitemap line independent of any group. The generator assembles exactly that text from your rows, live as you type: an empty group with no rules means "allow everything", and in real crawlers the most specific matching rule wins. The optional checker fetches an existing /robots.txt through our protected fetcher — robots files are plain text, which the fetcher accepts — and breaks it into its groups for review.
At the root of your host: https://example.com/robots.txt. It must be a plain-text file at exactly that address — one per host, including separate ones for subdomains.
Disallow stops crawling; noindex (a meta tag or header) stops indexing but requires the crawler to load the page. To remove a page from results, allow crawling and use noindex — blocking it in robots.txt hides the noindex from the crawler.
Yes — add a group naming the AI crawler (for example GPTBot or ClaudeBot) with Disallow: / as its only rule. Compliant AI crawlers honour it, and the user-agent list in the generator includes the common AI bots.
It asks a crawler to wait that many seconds between requests. Google ignores it (use Search Console’s settings there), but Bing, Yandex and many smaller bots respect it. Leave it empty unless a bot is overloading you.
The file may genuinely be absent (every crawler then defaults to "everything allowed"), the site may have answered our fetch with an error, or the file may be served under a different host. Check the address in a browser to confirm.