Definition
The canonical answer.
An AI crawler fetches web pages for a specific purpose, such as search indexing, model training, or retrieval requested by a user. OAI-SearchBot and PerplexityBot support search; GPTBot and ClaudeBot concern training. Google-Extended is a robots.txt policy token, not a separate crawler
Worked example
In practice.
A site can allow OAI-SearchBot while making a separate decision about GPTBot. Check robots.txt, server responses and verified operator identity together. Blocking direct retrieval can limit source access, but a brand may still appear through other sources or existing
Where this work happens: AI
Is this your constraint?
The free 30-minute consult tells you, live, on your