AI search · Glossary
What is an AI crawler?
An AI crawler is a bot run by an AI company that fetches web pages to train language models, to index pages for AI search, or to read a page live when a user asks a question. OpenAI and Perplexity, for example, use separate user agents for these jobs, and robots.txt can treat each one differently.
Written by Noor KhanWhy AI crawler matters
Blocking the wrong bot can remove a site from AI answers. OpenAI's documentation says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, while disallowing GPTBot signals that content shouldn't be used to train its models.
How AI crawler works
- Training crawlers collect content for foundation models; GPTBot is OpenAI's.
- Search crawlers index pages so an assistant can cite them: OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity.
- User-triggered fetchers such as ChatGPT-User and Perplexity-User visit a page because someone asked, and robots.txt rules may not apply to them.
- Google-Extended is a robots.txt token, not a separate crawler. It controls use in Gemini training and grounding, and Google says it has no effect on Google Search.
How to apply it
- Check your server logs for AI user agents and the pages they request.
- Allow the search crawlers of the assistants you want to be cited by.
- Treat training crawlers as a separate business decision.
- Make sure key content is in the initial HTML, not only rendered by JavaScript.
Example
A telehealth SaaS company that wants to appear in ChatGPT recommendations but keep its content out of model training allows OAI-SearchBot and disallows GPTBot in robots.txt.
Common mistake
Blocking every AI bot with one blanket rule to protect content, then wondering why assistants never cite the site.
put this into practice