A focused check,
not a mystery score.
We fetch the public page and its robots.txt file.
Rules are evaluated for the final URL path and query. Unavailable policies are marked unknown.
Indexability, canonical, metadata and structured-data signals are summarized.
Before you use it.
Does this verify actual provider access?+
No. We fetch the page once and evaluate robots.txt; we do not send a probe for each bot or use verified provider IP addresses. Allowed means the declared policy permits the path. It does not prove retrieval, indexing or citation.
Why can a user-requested bot ignore robots.txt?+
OpenAI and Perplexity document exceptions for user-initiated fetches. We show the declared rule separately from whether a provider promises to follow it. Use authentication for private content.
What is Google-Extended?+
Google-Extended is a robots.txt control token publishers can use to manage whether site content may help improve Gemini models and grounding.
Does this tool execute JavaScript?+
No. It inspects the initial HTML response. Content that exists only after client-side rendering may not appear in the report.