Canonical: https://www.strataigize.com/glossary/ai-crawler/
Description: An AI crawler fetches web pages for a specific purpose, such as search indexing, model training, or retrieval requested by a user.

[Home](https://www.strataigize.com/) / [Glossary](https://www.strataigize.com/glossary/) / AI crawler

AI Search

# AI crawler.

Definition

## What it _means._

An AI crawler fetches web pages for a specific purpose, such as search indexing, model training, or retrieval requested by a user. OAI-SearchBot and PerplexityBot support search; GPTBot and ClaudeBot concern training. Google-Extended is a robots.txt policy token with no crawler user agent of its own.

Worked example

## In _practice._

A site can allow OAI-SearchBot while making a separate decision about GPTBot. Check robots.txt, server responses and verified operator identity together. Blocking direct retrieval can limit source access, but a brand may still appear through other sources or existing index data.

Where this work happens: [AI Search Optimization](https://www.strataigize.com/generative-engine-optimization/)

Related terms

-   [AI citation](https://www.strataigize.com/glossary/ai-citation/)
-   [Technical SEO](https://www.strataigize.com/glossary/technical-seo/)
-   [AI visibility](https://www.strataigize.com/glossary/ai-visibility/)

## Put it in _context._

Bring your growth question to a free 30-minute consultation. We can work through what the numbers mean and which changes to consider.

[Book a growth consult →](https://www.strataigize.com/audit/)