Search engines don’t crawl your site in full every time they visit. They work from a budget, and on a large site with thousands of URLs, that budget runs out before every page gets a look. Which pages get crawled, and how often, is not random.
What crawl budget actually is
Crawl budget is the number of URLs a search engine will fetch from your site within a given time window. Google’s crawlers operate at a rate they judge your server can sustain, and they stop when they reach that limit. Every URL they spend time on is one fewer for something else.
Two inputs shape the number: crawl rate limit (how fast Googlebot requests pages, adjusted based on server response signals) and crawl demand (how much Google wants to revisit a page, based on perceived popularity and update frequency). Both levers are real, but demand is the harder one to move directly.
The term is used loosely to mean “how much of my site gets crawled,” which is close enough to be practical. For a 200-page marketing site with fast servers, crawl budget is not a meaningful constraint. For an e-commerce catalogue with half a million SKUs, faceted navigation, and unmanaged URL parameters, it determines what gets indexed and what sits invisible.
Why large sites run into trouble
Three patterns account for most crawl budget waste on large sites.
Duplicate and near-duplicate URLs. Faceted navigation is the classic culprit. Filtering a product list by colour, size, price, or sort order generates new URLs with nearly identical content. An unmanaged facet system can produce hundreds of thousands of essentially worthless addresses. Crawlers fetch them because they have no signal not to, and important category and product pages lose their share of the budget.
URL parameters and session IDs. Tracking parameters, session tokens, and A/B test tags appended to URLs create unique-looking addresses for the same underlying page. One URL becomes ten. Each gets crawled, none adds value.
Thin, low-quality, or orphaned pages. Boilerplate tag archive pages, deep pagination with no real content, internal search result pages served as indexable URLs, and orphan pages with no inbound links all dilute the pool. A crawler treats a thin tag page and a revenue-driving category page with equal weight until you tell it otherwise.
The pages that matter most to your business should not be competing with auto-generated parameter variants for a share of the crawl.
How to diagnose the problem
Before fixing anything, confirm you have a problem. Two data sources tell the story clearly.
Server logs. Log analysis is the only way to see what Googlebot actually fetches, as opposed to what you assume it fetches. Pull the Googlebot lines from your server logs and group them by URL pattern. You will almost certainly find a pattern absorbing a disproportionate share of requests that no human would ever need to see: /search?q=, /products?page=12&sort=price, /tag/blue. That pattern is the waste to cut.
Google Search Console coverage report. The “Discovered, Not Indexed” and “Crawled, Not Currently Indexed” states are signals that pages are sitting in the queue without being processed. A large gap between submitted and indexed URLs, alongside a high volume of discovered-but-skipped pages, suggests the crawl is being absorbed by the wrong parts of the site.
You can also check crawl stats directly in Search Console under Settings. It shows the request volume per day and average response time over the past 90 days. A falling trend without a corresponding reduction in site size is worth investigating.
Fixes that actually help
Fix the source of waste first. Tactics applied on top of an unfixed facet problem are noise.
Block low-value URL patterns in robots.txt. Internal search results, filter combinations with no meaningful content, and session ID parameters should be blocked from crawling. A single well-placed Disallow rule can redirect thousands of daily Googlebot requests toward pages that matter. One caveat: robots.txt blocks crawling, not indexing. A page blocked in robots.txt can still be indexed if it is linked from elsewhere. Combine it with noindex where both outcomes matter.
Use canonical tags consistently. For pages that need to exist but are variants of a primary, the canonical points back to that primary. The crawler still visits the variant occasionally, but the signal is cleaner. Canonical is a hint rather than a directive; it works best alongside robots.txt controls and sitemap hygiene.
Clean up your XML sitemap. Your sitemap should list only canonicalized, indexable, live URLs returning 200 status codes. A sitemap full of redirects, noindex pages, and 404s tells Google the sitemap is not a reliable guide. When Google stops treating it as reliable, it stops using it as one. Sitemap hygiene is one of the items in the technical SEO checklist.
Audit your internal link footprint. Every internal link is a crawl path. If your site’s navigation, footer, and breadcrumbs point to a thousand filtered variants, Google follows them. Review which URLs your templates link to and make sure that list reflects which URLs you actually want crawled. This connects to the broader question of how URL structure shapes crawling and indexing.
Improve server response speed. Google’s crawl rate limit scales with how quickly pages respond. A server consistently returning responses under 200ms will see more requests than a slow one. This is not always achievable overnight, but it is a real variable and it is measurable in the crawl stats report.
The connection to AI visibility
Crawl budget has always shaped indexing by traditional search engines. It matters just as much for AI visibility. Generative crawlers (GPTBot, ClaudeBot, PerplexityBot) make their own fetch decisions, and they face the same quality signal problem: a site flooded with low-value URLs reads as less credible than one with focused, substantive pages. The overlap with structured data and entity clarity is real. Both disciplines are about making your important content easy to find and process, whether by a conventional crawler or a language model.
If your AI visibility strategy depends on being cited from pages that never get properly crawled, the citation never comes. Crawlability is the foundation of every channel, organic or AI-driven. Our AI visibility practice treats both as part of the same problem rather than separate workstreams.
How Strynal approaches crawl budget
We start with log analysis because it is the only source of truth for what is actually happening on a site. Tools and assumptions describe what should be happening. Logs describe what is.
From there, the work is methodical: map the waste, rank patterns by volume and value, fix the highest-impact issues first, and verify the improvement in the data. The changes are rarely complex, but they require someone to own them across the full stack, from server configuration through CMS template settings through canonical markup and sitemap generation.
For large e-commerce and publishing sites, crawl budget optimization is often the fastest way to get more of the right pages indexed without writing new content. It is unglamorous, but it shows up clearly in the numbers. The technical SEO checklist is a good place to assess where your site stands, and our AI visibility and organic search work picks up from there.