Free tool
robots.txt Tester
One line in robots.txt can hide your entire site from Google. This tester fetches your robots.txt, checks it for a sitewide crawl block, and confirms your XML sitemap is declared.
Free · No signup · Results in ~10 seconds
The fatal pattern
A "Disallow: /" under "User-agent: *" blocks every crawler from every page. It is the standard staging-site setting, and it reaches live sites far more often than you would expect.
Crawling is not indexing
robots.txt controls crawling, not indexing. A blocked URL can still appear in results with no description. To keep a page out of search entirely, allow crawling and use a noindex meta tag instead.
Declare your sitemap
Adding a Sitemap: line to robots.txt is a free, one-line way to help search engines discover every page you want indexed.
More free tools
- Free SEO Checker
- AEO / GEO Checker
- llms.txt Checker
- Meta Tag Checker
- Open Graph Checker
- Heading Structure Checker
- Canonical Tag Checker
- Hreflang Checker
- Structured Data Checker
- Page Speed Checker
- XML Sitemap Checker
- noindex Checker
- Image Alt Text Checker
- Redirect Checker
Related guides
Run the full 35-point audit.
Free · No signup · Results in ~10 seconds