Robots.txt checker

Enter any URL on a site to fetch that site's robots.txt. You'll see its rules grouped by user agent, the sitemaps it lists, and you can test whether a path is allowed for a given crawler.