Crawl Budget for Small Site Networks: When It Actually Matters

For most small sites, crawl budget isn’t what holds back indexing. Honestly. It starts to matter when a site produces far more crawlable URLs than it has real pages, and that usually comes down to filters, sort orders or sloppy error handling. So if you run several small sites and keep reading that crawl budget limits indexing, you’re probably wondering: does this apply to me? Below I go through how Google defines the budget, who the guidance is actually written for, the traps that waste crawling on small sites, and what to fix instead.

What is crawl budget, according to Google?

Crawl budget is the set of URLs Google can and wants to crawl on a site. According to Google’s crawl budget documentation, the crawling infrastructure treats a site as a unique hostname, which means www.example.com and code.example.com get separate budgets. Two parts make up the budget:

• Crawl capacity limit: how much crawling your server can take without trouble.
• Crawl demand: how much Google actually wants to crawl.

Every site starts with the same default capacity limit, and it’s a conservative one. Google raises it over time if there’s demand and the site stays healthy. But here’s the catch. Even when that limit is never reached, low demand still means Google crawls the site less.

Does crawl budget affect small websites in a network?

Rarely. For a typical small site with a modest, stable set of pages, crawl budget is almost never the reason pages aren’t indexed. The guidance on managing crawl budget targets large sites and sites whose content changes very often (not a blog or a brochure site with a few dozen URLs that barely move). And in a network, each domain or subdomain is its own hostname with its own budget. Small sites don’t share one pooled allowance. Add another site to the network and the others don’t lose any crawling.

When crawl budget becomes a real problem on a small site

A small site can still burn crawling when it spits out lots of URLs that duplicate the same content. The usual suspects:

• faceted navigation and filter combinations
• differently sorted versions of the same page
• infinite scrolling pages that repeat content already on linked pages
• soft 404 pages, which return a normal response for missing content
• long redirect chains

Faceted navigation is the textbook case. Every combination of colour, size or price filter creates a new URL with near-identical content, and it adds up fast. If Google spends too much time on URLs it shouldn’t crawl, it might not get to the rest of the site, or might not raise the budget at all. Site-wide events like a site move work the other way round: they briefly push crawl demand up so Google can reprocess content under the new URLs.

How to keep crawling focused on the pages that count

Manage your URL inventory so Google spends its crawling on unique content, not unique URLs. Big difference. These steps apply to your own origin, CMS or NGINX config:

1. Consolidate duplicate content into one canonical page.
2. Block unimportant URLs, like sorted and filtered variants, in robots.txt when you can’t consolidate them.
3. Return a 404 or 410 status code for permanently removed pages, as recommended in Google’s advice on removed URLs.
4. Fix the soft 404s listed in the Page Indexing report in Search Console.
5. Keep sitemaps current and add the <lastmod> tag for updated content.
6. Shorten redirect chains so each old URL points straight at its final target.

Pick between removal and blocking carefully. People mix these up all the time. A removed page that returns an error status is a strong signal not to crawl it again. URLs blocked in robots.txt, on the other hand, sit in the crawl queue much longer and get recrawled the moment the block is lifted.

Why site health matters more than the size of the budget

Google only raises the crawl capacity limit for sites that stay healthy, so availability during crawling beats any setting you could tweak. When debugging, check first whether the site had availability issues while Googlebot was visiting. Only then look at making crawling more efficient. And low crawl demand? In my view it’s usually a content and signals problem, not a budget cap. Thin, duplicate or rarely updated pages give Google little reason to come back. Logs are the right tool to see where crawlers spend time on your server, and the rest of our crawl and server posts cover related setup questions.

A quick crawl budget check for a small site network

For each site, compare the pages you want indexed with the URLs Google keeps finding. Blame crawl budget only when the gap comes from URL bloat. Go through this per domain:

• filter or sort parameters that create crawlable URLs
• soft 404s in the Page Indexing report
• sitemap coverage and an accurate <lastmod>
• redirect chains longer than a single hop
• availability problems during crawling

Nothing turned up? Then missing pages usually point to content quality or weak internal linking, not to the crawler running out of time. My advice: fix duplicates and site health first, and worry about crawl budget only after that.

FAQ