Enter your homepage and the crawler follows links across your own host, page by page, then builds the sitemap. Free cap: 500 pages per crawl — larger sites need their CMS sitemap or a desktop crawler. Only crawl sites you own or are authorised to test.
Preparing…
Crawl your website from its homepage and turn the pages found into a valid XML sitemap you can download and submit to search engines. Free for up to 500 pages.
Crawl your website from its homepage and turn the pages found into a valid XML sitemap you can download and submit to search engines. Free for up to 500 pages.
A 40-page brochure site crawls in one or two batches: the homepage links to the service, about and contact pages, those link onwards, and the finished sitemap contains all 40 reachable URLs. If the blog is only linked from a JavaScript menu the crawler cannot read, its posts will be missing — the crawl doubles as an internal-linking check.
The crawler works breadth-first on our server: fetch a page, record it, extract its links, queue the ones on the same host that have not been seen, and repeat — in batches of up to 40 fetches with a short pause between requests and an identifying user-agent. Only pages that answer with a successful HTML response are listed, redirects are recorded at their final address, and files (images, PDFs, scripts) are never queued. The finished list is wrapped in the standard XML sitemap format.
It is a machine-readable list of your pages that helps search engines discover and schedule them. It does not guarantee indexing — pages still need to be worth indexing — but it removes discovery as a reason a page is missing.
500 pages is the free cap for this tool, chosen so a crawl stays quick and light on the crawled site. Sites with more pages usually have a CMS that already generates a sitemap.
The crawler only finds pages linked from pages it can reach, that return a successful HTML response. Check for orphaned pages, links built by JavaScript, and pages blocked behind logins or forms.
This generator crawls the site you give it as an owner testing their own property; it does not deliberately bypass technical blocks, and password-protected or non-HTML content is skipped. If a section of your site should never be listed, remove it from the XML before submitting.
Upload it to the root of your site so it loads at yourdomain.com/sitemap.xml, reference it in robots.txt, and submit the address in Search Console.