Skip to content
SEO

Moving to a CDN or Proxy Without Blocking Googlebot

Hands typing on a laptop with a strategy slide open, planning a site move to a CDN or reverse proxy
In this article
  1. Why does crawling spike right after moving to a CDN?
  2. How CDN and WAF bot protection ends up blocking Googlebot
  3. Pre-switch checklist: cache warm-up before migration
  4. What to do if the origin server is overloaded by crawling
  5. After moving to a CDN: checking Googlebot in logs and Search Console
  6. Final checks before every domain goes live
  7. FAQ

Moving to a CDN changes how Googlebot crawls your site, and two things can go wrong. Your origin chokes on the first wave of cache misses, or the CDN starts challenging verified Googlebot and locks it out. So plan the switch so the origin can take the misses and Googlebot never sees a challenge, then keep an eye on your logs and the Crawl stats report. If you’re about to point several domains at a CDN or reverse proxy and you’re nervous about either problem, this guide covers both. The logic is the same whether you’re on a commercial CDN or pointing a domain at a proxy like Jalvo, an NGINX reverse proxy.

Why does crawling spike right after moving to a CDN?

Short version: the edge answers faster, so Google raises the crawl capacity limit. But the cache is still cold, so every first request goes back to the origin anyway. A quick bit of vocabulary. The origin server is the backend that holds your original content. The edge is the CDN node that answers visitors close to it. A cache miss is when the edge has no stored copy and has to fetch one from the origin.

According to Google’s guidance on moving to a CDN, its crawlers work out a crawl capacity limit (you’ll also see it called hostload). Slow responses push the crawl rate down. Fast, stable ones let it creep up, as long as there’s crawl demand. And a new CDN starts out empty, so the first request for every URL is a miss. That’s where it hurts. Real people bunch up on a handful of popular pages. Crawlers don’t. They go through every URL you have, and until the caches fill, your origin is doing most of the work.

How CDN and WAF bot protection ends up blocking Googlebot

This one is common. Someone turns on blanket bot rules (CAPTCHA, WAF challenges) with no exemptions, and Google’s crawlers get caught too. They fail the challenge and back off. A WAF, or web application firewall, filters requests before they reach your site. If it treats every automated client the same way, your pages can start dropping out of Search.

The fix is easy. Allow verified bots in your CDN and firewall settings, since most providers have a toggle or a managed rule for known or verified bots. Want to check a crawler yourself? Do a reverse DNS lookup and look for googlebot.com, google.com or googleusercontent.com, or match the address against the ranges on Google’s page on verifying requests. If there are bots you actually don’t want, go after those specific ones in robots.txt or in rules, not every automated client. I’d follow the approach in rate limiting bots without slowing Googlebot.

Pre-switch checklist: cache warm-up before migration

Before you send all traffic through, find the URLs that matter most and warm them through the CDN at a pace you control. That way the first crawler visits hit the cache, not the origin.

  1. Pull the most frequently crawled URLs from your origin access logs.
  2. Add your highest-traffic pages from analytics.
  3. Fetch that list through the CDN slowly. Otherwise the warm-up becomes its own load spike on the origin (kind of defeats the purpose).
  4. Check that the verified-bot exemption is on for every domain. Every single one.
  5. Go through your cache rules using caching rules that avoid stale pages, so nobody gets outdated HTML after you update something.

Move domains one at a time, not all at once. You can then watch origin load for each domain and stop before the next move if something looks off.

What to do if the origin server is overloaded by crawling

In an emergency you can return 429 or 503 for a short window, and crawlers will slow down. Google’s advice on handling the CDN transition points to its reduce crawl rate guidance, which keeps this to a few hours, 1-2 days at most. Don’t use 403 or 404 to throttle crawling. Those codes tell Google the content is forbidden or gone, and that’s not the message you want to send.

These status codes are a band-aid. What actually fixes origin overload during a crawl surge is a warm cache and enough backend capacity for whatever misses are left.

After moving to a CDN: checking Googlebot in logs and Search Console

Put your origin logs side by side with the Crawl stats report in Search Console. You want to see Googlebot reaching cached pages and getting no errors. The techniques in reading server logs for crawl problems help you tell real issues from noise. What I’d watch:

  • Cache misses from Googlebot vs. other bots in the origin logs.
  • Zero 403 responses or challenge pages served to verified Googlebot.
  • Origin response times, especially in the first few days.
  • Changes in crawl requests, response times and errors in the Crawl stats report.

Misses still high on one domain? Extend its warm-up list before you move the next one.

Final checks before every domain goes live

When moving to a CDN, keep the same order for every domain: read the origin logs, turn on the verified-bot exemption, warm the cache, switch DNS in a controlled way, monitor. Boring? A bit. But that sequence keeps the origin from drowning in misses and keeps Googlebot crawling, so faster edge servers actually help how Google sees your sites instead of getting in the way.

FAQ

Will moving to a CDN hurt my SEO?

Not if Googlebot gets through and your origin can handle the cold cache. Faster, stable responses can even push the crawl rate up over time. The trouble usually comes from bot challenges or an overloaded origin.

How do I know a request really comes from Googlebot?

Do a reverse DNS lookup and check that the host ends in googlebot.com, google.com or googleusercontent.com, then confirm it with a forward lookup. Or match the IP address against Google’s published crawler ranges. The user agent string on its own proves nothing. Anyone can fake it.

Can I use 403 to slow Googlebot during the migration?

No. A 403 tells crawlers the content is forbidden, and that can hurt indexing. Use 429 or 503, only for a short window, and fix the actual capacity or cache problem underneath.

ShareLinkedInX
Web Systems team

The team that builds and runs Jalvo.

Give each site its own IP.

Add your first domain in a few minutes.

Start now

Choose which cookies we may use. Necessary cookies keep the site and your login working and cannot be switched off.