Separate access from visibility
A crawler access check answers a narrow question: do your rules and responses permit a particular bot to retrieve a particular URL? It does not establish that the provider has fetched the page, included it in search or cited it in an answer.
Start with one public page and an explicit policy. You might want search discovery while restricting model-training crawlers. Or you might want to investigate why a user-requested page fetch fails. Those are different checks, even when the product name is the same.
Name the bot, name the exact URL and describe the evidence. “robots.txt permits OAI-SearchBot for this path; a normal request returns the article; provider access remains unverified” is more useful than “AI ready.”
ChatGPT and Claude use different bots for different jobs
The following table covers search, training and user-requested retrieval. It is not a complete inventory of every bot either provider operates. Bot policies can change; check the linked provider documentation before editing production rules.
| Provider | Token | Documented role |
|---|---|---|
| OpenAI | OAI-SearchBot | Search discovery for ChatGPT. |
| OpenAI | GPTBot | Collects content that may be used for model training. |
| OpenAI | ChatGPT-User | User-triggered page retrieval; not the control for search inclusion. |
| Anthropic | Claude-SearchBot | Search discovery and relevance. |
| Anthropic | ClaudeBot | Collects content that may contribute to model training. |
| Anthropic | Claude-User | Retrieval initiated by a user’s request. |
OpenAI documents independent controls for its search and training crawlers. Its user-triggered requests are different: robots.txt rules may not apply to ChatGPT-User. Anthropic documents robots.txt controls for its listed bots, including user-requested retrieval. Do not assume the two providers use identical access policies. Sources: OpenAI bot documentation and Anthropic crawler documentation.
Consequently, a result for GPTBot tells you nothing by itself about OAI-SearchBot. Keep one row per token in your audit. Decide separately how you want to handle search, training and user-requested access.
Check four layers for the same URL
1. The robots.txt policy
Fetch robots.txt from the page’s exact scheme and hostname. Read the group for the token you are investigating and check the page’s path against its rules. Repeat the check for a subdomain or redirected destination when relevant; do not assume every host shares one policy.
The free AI crawler access checker is a starting point for selected tokens and page signals. Its current set includes GPTBot and ClaudeBot, but not OAI-SearchBot or Claude-SearchBot. Inspect those search-bot groups separately. The Mydentify listing provides another tool to evaluate; check its current coverage before comparing results.
2. The HTTP response and redirects
Request the exact public URL. Record the response, final destination and returned content. A redirect to a login page does not give a crawler the article. A successful status with a challenge screen does not give it the article either.
# Diagnostic request from your own network, not an OpenAI fetch.
curl -L --max-redirs 5 --max-time 30 \
-A 'OAI-SearchBot' \
-D response-headers.txt \
-o response-body.html \
'https://example.com/public-article'This command saves headers and body for inspection. Setting a user-agent string only changes the label on your request. It cannot reproduce a provider’s IP address, prove bot identity or demonstrate that the provider can reach your server.
3. CDN, firewall and bot-management behavior
Compare the same path and time window in your edge and origin logs. Look for denied requests, rate limits, challenges or inconsistent responses. Validate claimed provider traffic using its current identification guidance; a user-agent name by itself is not authentication.
If an intended crawler is blocked, adjust the narrow rule responsible and retest. Avoid disabling protection across the whole site as a diagnostic shortcut. Keep authenticated content behind actual access controls: robots.txt is a published crawling preference, not a way to secure private data.
4. The content that is actually available
Inspect the returned body for the page title and main content. Record relevant page directives and rendering requirements, while keeping their interpretation specific to the provider. A browser can display text after scripts run even when your initial HTML response contains very little.
Example: permit search, restrict training
The illustrative policy below permits the two search tokens and blocks the two training tokens. It intentionally leaves user-requested retrieval out of scope. It is not a replacement for your full robots.txt file.
# Illustrative policy for public content only.
# Repeat your existing path restrictions in each specific group.
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
Allow: /
Disallow: /account/
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /Merge this intent into the existing groups without duplicating or contradicting them. Preserve all relevant path restrictions: a named group should not accidentally open paths that were excluded in your wildcard policy. The example repeats an account-path restriction to make that review explicit; your own excluded paths may differ. Robots Exclusion Protocol group matching ↗
After publishing a rule change, fetch the live robots.txt, test representative allowed and excluded paths, and inspect later provider requests in logs. A local parser result describes the policy it read, not whether every edge location or crawler already uses that version.
Worked example: robots allows it, but access still fails
Hypothetical diagnostic sequence. A public guide is allowed for a search bot in robots.txt. Your local request returns the article with HTTP 200. Verified provider requests to that path receive HTTP 403 at the CDN.
The evidence points to an edge access rule, so changing the article’s title or adding more schema would not address this failure. Review the matching CDN event, adjust the intended exception, then look for a successful provider request containing the expected page response.
| Observation | What to investigate next |
|---|---|
| A checker reports “allowed” | Which tokens did it check, on which host and path? Did it inspect rules, make a request, or both? |
| Local request succeeds; provider request fails | Compare verified request identity, timestamp and edge rules. Local access is not equivalent to provider access. |
| HTTP 200 contains a challenge | Inspect the body and challenge policy; the status alone is insufficient. |
| Provider retrieved the page; no citation appears | Access has been observed. Investigate query relevance and visibility separately rather than repeatedly changing robots.txt. |
Keep an access record you can reproduce
- Exact URL, hostname, timestamp and intended policy.
- Bot token and provider documentation checked.
- Applicable robots.txt group and path rule.
- HTTP response, redirect chain and body from the diagnostic request.
- CDN or origin evidence, with bot identity verified where possible.
- What remains unknown, plus the next check after any change.
Once access works as intended, decide whether you need repeated visibility measurements. Our AI visibility tools guide covers a different task: observing how brands and pages appear in answers. Keep access checks and visibility measurements as separate evidence.
Common questions
Does blocking GPTBot also block ChatGPT search?
OpenAI documents separate search and training controls. Check OAI-SearchBot and the actual response path when investigating search access; a GPTBot rule alone is not enough.
Can I guarantee citations by allowing a crawler?
No. Permission and a successful response establish only part of the retrieval path. They do not guarantee selection, ranking or citation for a particular question.
Do I need to open private pages to these bots?
No. Define the policy for public pages and preserve authentication and other access controls for private content. Audit a public URL when assessing search discovery.