Technical SEO · Glossary
What is crawl budget?
Crawl budget is the number of URLs a search engine is willing and able to crawl on a site in a given period, determined by crawl capacity (what the server can handle) and crawl demand (how much the search engine wants the content).
Written by Noor KhanWhy Crawl budget matters
Google says crawl budget mainly matters for very large sites (around a million pages or more) or sites with content that changes very often. For most small and medium sites, it's rarely the bottleneck.
How Crawl budget works
- Slow or error-prone servers reduce how much gets crawled.
- Popular, frequently updated pages are crawled more often.
- Duplicate URLs, parameters and redirect chains waste crawling on pages that don't matter.
How to apply it
- Check Crawl Stats in Search Console before assuming there's a problem.
- Remove or block low-value URL variations such as faceted filters.
- Keep the sitemap limited to canonical, indexable pages.
- Improve server response times.
Example
An e-commerce site with millions of filter combinations blocks unnecessary parameter URLs so Googlebot spends its time on product and category pages.
Common mistake
Worrying about crawl budget on a 50-page site, where the real issue is usually content quality or internal linking.
Not sure how Crawl budget applies to your site? Book a strategy call, request a free gap snapshot, or ask the assistant.