Free Robots.txt Tester – Check Crawl Rules Online

What a Robots.txt Tester Does and How Matching Works

A robots.txt tester takes your robots.txt file and a URL, then tells you whether a crawler is allowed to fetch that URL and exactly which rule decided the outcome. This tool parses the file the way Google does and reports the verdict, the matching line, and the reason. Getting robots.txt right is a core piece of technical SEO, and getting it wrong is expensive: one stray rule can block an important section or, at worst, hide the whole site. Developers shipping a new site, SEO specialists auditing crawl coverage, e-commerce managers taming filter URLs, and anyone who noticed a page vanish from search all use a tester to confirm a rule behaves before trusting it in production. The file is organized into groups. A group begins with one or more User-agent lines naming a crawler, followed by Allow and Disallow lines that apply to it, until the next User-agent line. The tester first picks the group that matches the crawler you are testing, preferring an exact name over the wildcard star group. Within that group it checks every Allow and Disallow rule against your path, supporting the two wildcards search engines understand: star matches any run of characters and the dollar sign anchors the end of the path. When more than one rule matches, Google's precedence rule applies, and the tester implements it exactly: the most specific rule wins, measured by how many literal characters its path contains, and on an exact tie the Allow rule beats the Disallow. This is the subtle logic most people get wrong when reading a file by eye. Here is a worked case. Suppose your file says Disallow: / followed by Allow: /public/ for all crawlers, and you test Googlebot against /public/report.html. A naive reading sees the blanket Disallow and assumes the page is blocked, but the tester reports ALLOWED, because /public/ has seven literal characters while / has one, so the longer, more specific Allow wins. Change the test path to /admin/settings and the verdict flips to BLOCKED by Disallow: /, since no Allow rule matches that path. In both cases the tool names the exact line responsible, so you are never guessing which rule fired. The practical scenarios are common. A store confirms that its faceted filter URLs like /shoes?color=red are blocked while the clean category page stays crawlable, so crawl budget is not wasted on near-duplicates. A developer tests a staging path before launch to make sure a Disallow will not accidentally catch a production route that shares a prefix. An SEO auditor pastes a client's file and tests the same URL against Googlebot, Bingbot, and GPTBot at once, spotting that one crawler is blocked while others are allowed. A publisher checks that a thank-you page is excluded from crawling without also blocking the article template beside it. A few reminders keep testing accurate. Remember that robots.txt controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it, so use a noindex tag when you truly need a page kept out of search. Do not disallow your CSS and JavaScript folders, since Google needs them to render pages the way visitors see them, and the tester will show them as blocked so you can catch that mistake before it hurts how your layout is assessed. Watch for the catastrophic Disallow: / under the star user-agent, which blocks everything and has wiped sites from search overnight, often left behind from a staging environment that was never cleaned up. Crawlers read the file only at the site root, so it must live at the exact root path or it is ignored entirely, and Google skips the Crawl-delay directive, both of which this tool notes for you. It is also worth testing several important URLs after every edit, since a small change to one rule can unexpectedly affect a path that shares a prefix. Everything runs locally in your browser, so you paste your file and test privately with nothing uploaded to any server.

🔥 Trending Tools