Definition
The canonical answer.
An AI crawler is an automated agent that fetches web pages for AI systems, either to train models or to retrieve live content for answering queries. Common examples include GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. If AI crawlers cannot reach a site, the brand cannot appear or be cited in the answers those systems generate.
Worked example
In practice.
A site behind an aggressive bot-blocking rule returns errors to AI crawlers and silently disappears from AI answers while its analytics look normal. The fix is deliberate crawler policy: verify which bots reach a 200 response, allow legitimate AI crawlers, and monitor that access like uptime, because for AI search it is uptime.
Where this work happens: AI Search Optimization
Is this your constraint?
The 24-hour audit tells you, with a screen-recorded walkthrough of your own funnel.
Book a growth audit →