Robots.txt Validator & Tester
Paste a robots.txt file or fetch one from a live URL, get its directive syntax checked for issues, and test whether a specific URL path would be allowed or blocked for a given crawler user-agent.
常见问题
How does the path test decide allowed vs. blocked?
It implements the actual robots.txt matching algorithm search engines use: among every Allow/Disallow rule in the group that applies to your user-agent, the rule with the longest (most specific) matching path wins, regardless of the order the rules appear in. A tie between an Allow and Disallow of equal length is resolved in favor of Allow.
What do the wildcard and $ symbols mean in a path?
"*" matches any sequence of characters, so "/products/*" matches any path starting with /products/. A trailing "$" anchors the end of the path, so "/*.pdf$" matches paths ending in .pdf but not something like /file.pdf.html.
What syntax issues does this flag?
Missing colons after a directive name, directives that appear before any User-agent line, unknown directives, empty Sitemap URLs, non-numeric Crawl-delay values, and paths that don't start with a slash - the kinds of mistakes that can silently make a rule not work as intended.
Does this fetch my live robots.txt for me?
Yes, switch to "Fetch from URL", enter your domain, and the file is fetched server-side (avoiding browser CORS restrictions) and loaded straight into the editor for testing.