NewOpen Source
OpenSEO
★ 19kOpen-source SEO platform for humans and AI agents
Active repoRelease v0.1.8
MA
ToolEditorially reviewed
Mydentify AI Crawler Access Checker
Check robots.txt, page directives and AI crawler access signals
PricingFree
CategoryTechnical SEO
Latest releasev1.1.4
Added to directorySep 8, 2026
Overview
What is Mydentify AI Crawler Access Checker?
Mydentify is a free web checker for inspecting access rules for documented OpenAI and Anthropic crawlers. It checks path-specific robots.txt rules, page directives, response headers and HTTP responses to user-agent probes. Search, user-requested retrieval and training crawlers are reported separately. Probes originate from Mydentify infrastructure, not verified provider IP addresses, so results do not guarantee crawling, indexing, citations or traffic. An MIT-licensed companion CLI is available on GitHub.
Best forTechnical SEO teams and site owners checking access rules for OpenAI and Anthropic crawlers.
Primary workflowCheck access rules for AI crawlers
Delivery modelFree web checker with an MIT-licensed companion CLI
Pricing modelFree
Editorial notes
When to consider Mydentify AI Crawler Access Checker
Mydentify inspects the access rules and HTTP responses for a public page. Its checks cover robots.txt, page directives and documented AI user-agent names, with separate results for search, user-requested retrieval and training.
- Start here
- Enter one public page URL in the web checker. Review the path-specific robots.txt rules, page directives and response results for each crawler before deciding which access rules to change.
- Keep in mind
- Probes originate from Mydentify infrastructure, not verified provider IP addresses. Results cannot certify official crawler access or guarantee crawling, indexing, citations or traffic.
- Free or paid?
- Free web checker with no account required. The companion CLI is open source under the MIT license.
Official sources: Web checker and method ↗ · Companion CLI source ↗
Checked . Based on product documentation, not a hands-on benchmark.
Capabilities
What it helps you do
01
Path-specific robots.txt checks
02
Robots meta and X-Robots-Tag inspection
03
Documented AI user-agent response probes
04
Separate search, user-requested retrieval and training checks
Verified reviews
What users say about Mydentify AI Crawler Access Checker
Reviews come from signed-in accounts and are checked before publication.
Share a real experience
Write a verified review.
Sign in with Google or GitHub. We verify the account identity—not a purchase or employment relationship.
No published reviews yet.Be the first to share a useful, specific experience.
Categories
Practical questions
Mydentify AI Crawler Access Checker FAQ
What is Mydentify AI Crawler Access Checker?+
Mydentify is a free web checker for inspecting access rules for documented OpenAI and Anthropic crawlers. It checks path-specific robots.txt rules, page directives, response headers and HTTP responses to user-agent probes. Search, user-requested retrieval and training crawlers are reported separately. Probes originate from Mydentify infrastructure, not verified provider IP addresses, so results do not guarantee crawling, indexing, citations or traffic. An MIT-licensed companion CLI is available on GitHub.
Who is Mydentify AI Crawler Access Checker best for?+
Technical SEO teams and site owners checking access rules for OpenAI and Anthropic crawlers.
How is Mydentify AI Crawler Access Checker priced?+
Free web checker with no account required. The companion CLI is open source under the MIT license.
Keep exploring
