Articles · SEO Tools
Compare robots.txt to a verified AI crawler registry
Publishers want clarity on how AI crawlers read robots.txt, but inventing user-agents is worse than listing none. The registry must cite official documentation, include last-verified dates, and state clearly that compliance is not guaranteed.
Why this is worth doing in the browser
DevNestro’s AI Crawler Checker keeps a central registry (Name, UserAgent, Organization, Purpose, Source, LastVerifiedDate, Enabled) drawn from OpenAI, Anthropic, Google, Apple, and Perplexity docs. Paste robots.txt or fetch /robots.txt safely, then see Allowed/Blocked/No Matching Rule per crawler for a path. Copy explains robots.txt is preference, not hard access control.
How to use the tool
Paste or fetch robots.txt, pick a path, review each crawler’s verdict and source link, then adjust Robots.txt Generator output. Re-verify registry dates when vendors change bots.
- Do not invent crawlers without official sources.
- Google-Extended and Applebot-Extended are control tokens, not separate log UAs.
- User-initiated fetchers may ignore robots.txt.
- Preferences ≠ guarantees.
Privacy
Paste mode is local; fetch mode uses the safe public fetcher for /robots.txt only.
Check AI crawler preferences against a verified registry before you publish a new robots.txt.