Articles · SEO Tools
See which Allow or Disallow rule actually wins
Conflicting Allow and Disallow lines are where teams get surprised — especially with /admin/ public exceptions or GPTBot-specific blocks. You need a tester that shows the verdict and the matched rule, not a vague “probably blocked.”
Why this is worth doing in the browser
DevNestro’s Robots.txt Tester parses groups, selects the most specific User-agent match (falling back to *), then applies longest path matching. On equal length, Allow is treated as less restrictive. Results are Allowed, Blocked, or No Matching Rule, plus the matched rule text.
How to use the tool
Paste robots.txt, enter a path and user-agent token, run the test, and adjust generator output until the verdict matches intent. Pair with AI Crawler Checker for the registry of known AI agents.
- No Matching Rule usually means allowed by default.
- Specific User-agent groups override * when present.
- Wildcards and end anchors are supported in path patterns.
- Remember preferences are not guarantees of crawler compliance.
Privacy
Testing is local. API: None.
Unsure which rule wins? Open Robots.txt Tester and check the matched rule before deploy.