Rate Limiting Bots Without Slowing Down Googlebot

To rate limit bots safely, verify Googlebot first and exempt it, then throttle everything else per IP with nginx limit_req and return 429 Too Many Requests to any client that goes over the limit. That’s the whole recipe. This guide is for admins whose origin is getting hammered by scrapers and who want to slow them down without losing search crawling. The catch? A blanket limit also catches the real Googlebot. And fake crawlers send the Googlebot user agent precisely to slip past your rules, so the user agent string on its own proves nothing.

Why a Blanket Rate Limit Can Hurt Your Crawling

Google reads a pile of 429, 500 or 503 responses as “slow down, please”, so a limit that catches Googlebot lowers the crawl rate for your whole hostname. Not just one path. According to Google’s guide to reducing crawl rate, the slowdown covers the entire hostname, such as subdomain.example.com. So URLs that return perfectly normal content get crawled less too, right along with the ones that threw errors. Crawling picks up again on its own once the errors drop. My take: a 429 sent to Googlebot should be a tool you reach for on purpose, never collateral damage from a scraper rule.

How Do You Verify Googlebot by IP Before Exempting It?

A request really comes from Googlebot only if its IP resolves to a Google hostname and that hostname resolves back to the same IP, or if the IP sits inside Google’s published crawler ranges. The DNS round trip looks like this:

1. Run a reverse DNS lookup on the requesting IP.
2. Check that the domain is googlebot.com, google.com or googleusercontent.com.
3. Run a forward DNS lookup on the hostname you got back.
4. Compare the result with the original IP. No match, no Google.

Hostnames follow patterns like crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com, as shown in Google’s instructions for verifying crawlers. Prefer ranges over DNS? You can match the IP against Google’s published lists instead. Just refresh them on a schedule (a cron job is fine) rather than hardcoding them and forgetting about it. If the reverse half of the check feels a bit fuzzy, here’s how PTR lookups work.

Setting Up nginx limit_req for Unverified Bots

nginx limit_req throttles requests per key, and an empty key is never counted, so verified crawlers skip the limit while everyone else gets tracked per IP. That little quirk does most of the work here. A minimal setup has four parts: a geo block that flags verified Google ranges, a map that turns the flag into an empty value or $binary_remote_addr, a limit_req_zone on that key, and limit_req with burst and nodelay in the location block.

geo $google_verified {
    default 0;
    include /etc/nginx/google-ranges.conf;  # refreshed by a cron job
}
map $google_verified $limit_key {
    1 "";
    0 $binary_remote_addr;
}
limit_req_zone $limit_key zone=bots:10m rate=YOUR_RATE;

server {
    limit_req_status 429;
    location / {
        limit_req zone=bots burst=YOUR_BURST nodelay;
    }
}

Set limit_req_status 429 so throttled clients get 429 Too Many Requests instead of nginx’s default 503, and send a Retry-After header along with it. Expensive paths like search, filters and feeds deserve their own tighter zone. Static assets can live with a looser one. These settings apply to your own origin or NGINX server. And the rates? They depend on your traffic and your hardware, so tune them against your logs. Copying numbers from some random blog post (this one included) is how you end up throttling real users.

Catching Fake Googlebots and Aggressive Scrapers

Anything that claims to be Googlebot in its user agent but fails the DNS or IP range check belongs in your strictest limit zone, or gets blocked outright. No second chances. In the access log, these are the tells:

• a Googlebot user agent coming from a non-Google IP,
• a high request rate from one IP or one subnet,
• crawling of parameterised URLs such as sort, filter and session variants,
• ignoring robots.txt, which real Google crawlers always respect for automatic crawls.

Cache reverse DNS results, otherwise verification tacks a lookup onto every single request. Not great. Also log throttled requests to a separate file so false positives are easy to spot. For reviewing that file, the same approach used for verifying Googlebot in logs works just fine.

What Changes When Traffic Passes Through a Reverse Proxy With Rotating IPs?

Behind a reverse proxy, the origin sees the proxy’s address instead of the crawler’s, so IP-based verification and per-IP limits on the origin break unless the real client IP gets passed along. Rotating proxy IPs make it messier. One visitor can show up under several addresses, and many visitors can share one. Your per-IP counters get skewed, and reverse DNS checks end up looking at the wrong host. The fix is a forwarded client IP header plus a trusted list of proxy addresses, which is exactly what the nginx real_ip module handles. But here’s the thing: your verification is only as reliable as that header. If your traffic runs through a reverse proxy with rotating IPs, check which address actually lands in your limit zones.

When You Actually Need to Reduce Googlebot Crawl Rate

If Googlebot itself is overloading the server, you can return 500, 503 or 429 to its requests for a short time, and Google will crawl less until the errors stop. Short is the key word. Google’s crawl rate documentation describes this as an urgent measure for a short period, for example a couple of hours, or 1-2 days. Can’t serve errors? Then you can file a special request to report an unusually high crawl rate. Don’t try to use it to ask for a higher rate, because that’s not what it’s for, and it may take several days to be evaluated anyway. Either way, this is a deliberate, time-limited slowdown. Keep it separate from the scraper limits above.

Summary

Order matters here: verify Googlebot, exempt it, throttle clients that fail verification with limit_req and 429, then keep an eye on the logs for mistakes. Behind a proxy with rotating addresses, per-IP logic on the origin is only as good as the forwarded client IP. There’s more in our crawling and indexing posts if you want to dig further. Rate limit bots this way and scrapers slow to a crawl while search crawling keeps its normal pace.

FAQ