Definition
What it means.
An AI crawler fetches web pages for a specific purpose, such as search indexing, model training, or retrieval requested by a user. OAI-SearchBot and PerplexityBot support search; GPTBot and ClaudeBot concern training. Google-Extended is a robots.txt policy token with no crawler user agent of
Worked example
In practice.
A site can allow OAI-SearchBot while making a separate decision about GPTBot. Check robots.txt, server responses and verified operator identity together. Blocking direct retrieval can limit source access, but a brand may still appear through other sources or existing
Where this work happens: AI
Put it in context.
Bring your growth question to a free 30-minute consultation. We can work through what the numbers mean and which changes