Free tool
robots.txt checker
Free robots.txt checker: fetch a site’s robots.txt, list Sitemap lines, and sample Disallow rules.
robots.txt tells well-behaved crawlers which paths they may fetch and often points to XML sitemaps. This checker fetches the file and summarizes Sitemap and Disallow lines.
What we extract
HTTP status, Sitemap: URLs, counts of Allow/Disallow rules, and a preview of the file body. Use it after a CMS change accidentally blocks / or /blog.
robots.txt is not security
Disallow does not hide private content from attackers. It is a crawl hint. Sensitive URLs need authentication, not only robots rules.