9 September 2026

Check robots.txt, page directives and AI crawler access signals

Mydentify is a free web checker for inspecting access rules for documented OpenAI and Anthropic crawlers. It checks path-specific robots.txt rules, page directives, response headers and HTTP responses to user-agent probes. Search, user-requested retrieval and training crawlers are reported separately. Probes originate from Mydentify infrastructure, not verified provider IP addresses, so results do not guarantee crawling, indexing, citations or traffic. An MIT-licensed companion CLI is available on GitHub.

Audit sitemap coverage and compare exact URL cohorts between releases.

A dependency-free Node.js CLI for checking sitemap declarations, image coverage, metadata quality and deterministic URL cohort changes between releases. It works locally without an account or API key. Sitemap Cohort Auditor audits sitemap declarations; it does not crawl every listed page or predict indexing, rankings or traffic. Maintained by Edilec under the MIT License.