Technical SEO · Glossary
What is robots.txt?
robots.txt is a plain-text file at the root of a host that tells crawlers which URLs they may request. Google uses it to manage crawling, not indexing, so a blocked URL can still appear in search results without a description.
Written by Noor KhanWhy robots.txt matters
One wrong line can stop Googlebot, or an AI search crawler, from reaching a whole section of a site. Used well, it keeps crawlers away from filter URLs, internal search results and other pages that waste crawling.
How robots.txt works
- Each group of rules names a user agent, such as Googlebot or GPTBot, and lists the paths it may not crawl (Disallow) or may crawl (Allow).
- The file applies only to the protocol, host and port it sits on, so each subdomain needs its own.
- Googlebot and other reputable crawlers obey it, but others might not, so it is not a security control.
How to apply it
- Block low-value URL patterns such as faceted filters and internal search, never pages you want ranked.
- Use noindex or password protection to keep a page out of Google, not a Disallow rule.
- Set AI crawler access per user agent, for example allowing OAI-SearchBot while disallowing GPTBot.
- Check Search Console's robots.txt report for fetch errors and warnings after every change.
Example
An e-commerce store disallows /search and its ?color= and ?size= parameter URLs so Googlebot spends its crawling on product and category pages.
Common mistake
Disallowing a page to get it out of Google. The crawler can't see a noindex rule on a blocked page, so the URL can stay in search results.
put this into practice