Enter a website to read its robots.txt the way a search engine does. We show the rules for each crawler, the sitemaps it points to and flag the mistakes that quietly keep your pages out of Google, like a leftover "Disallow: /" from a staging site.
Free and without registration. The check is made from our server, so the timings include the network distance to it.
robots.txt is a plain text file at the root of a domain (/robots.txt) that tells search engine crawlers which parts of the site they may or may not request. It is a request, not a lock: well-behaved crawlers follow it, but it does not protect private pages.
No. Disallow stops Google from crawling the page, but a blocked URL can still appear in search results without a description if other pages link to it. To keep a page out of search, allow crawling and add a "noindex" meta tag or header instead.
A common cause is a "User-agent: * Disallow: /" rule left over from development, which asks every crawler to stay out of the entire site. This checker flags that rule at the top of the result.
One request shows the best case. Run a free load test with up to 1,000 concurrent connections, compare the results after every change and get an email when your site goes down.
Run a free load test