Reverse Proxy Caching and SEO: No Stale Pages for Crawlers

Crawlers see stale pages when your proxy keeps handing out stored copies and never checks them against the origin. Fixing reverse proxy caching SEO means short TTLs for HTML, ETag and Last-Modified passed through untouched, and 304 responses handled properly. Is Google still showing old titles? Outdated canonicals, or redirects you pulled weeks ago? Or did crawlers keep getting a cached 404 or 5xx long after the origin was back up? Then one of these settings is almost always the culprit.

How does a reverse proxy end up serving stale content to Googlebot?

A proxy returns its stored response until its own TTL runs out, no matter what has changed on the origin in the meantime. Usually the cause is a config choice that looked harmless when it went in:

• a long proxy_cache_valid that covers any status code, not just 200
• proxy_ignore_headers Cache-Control Expires, which throws the origin’s freshness rules straight in the bin
• proxy_cache_use_stale switched on for every error, with no time limit
• a cache key without the host or scheme, so one site (or protocol) gets served another’s page

Errors are the nasty case. A two-minute outage can become a cached 5xx that bots keep hitting for hours. Want to confirm it? Compare what the origin returns with what bots actually received, and start with crawl problems in server logs.

Which Cache-Control headers should HTML pages send?

HTML wants a short max-age that matches how often it really changes. Static assets can sit in cache for a long time. CSS, JavaScript and images with versioned filenames are safe with long lifetimes, because a new release means a new URL anyway. Pages don’t work like that. Same URL, different content.

Google suggests using max-age to tell crawlers when to come back, set to the number of seconds you expect the content to stay put. Its post on how Google’s crawlers cache uses Cache-Control: max-age=94043 as the example. Three directives do the heavy lifting here. s-maxage applies only to shared caches like your proxy. max-age covers browsers and crawlers. And no-cache? Despite the name, it lets a copy be stored - it just has to be revalidated before every use.

Using ETag and Last-Modified so crawlers get 304 instead of old copies

Validators let a crawler ask “has this changed?”, and if it hasn’t, the server answers 304 Not Modified with no body. According to the Google Search Central guidance on crawler caching, Googlebot sends the stored ETag in If-None-Match, and a matching value should get a 304 with no HTTP body. If one URL serves several versions, say mobile and desktop, each one needs its own ETag. Last-Modified with If-Modified-Since works the same way.

This is exactly where proxies tend to break things. On-the-fly gzip or HTML rewriting can strip the ETag, weaken it, or leave a single tag shared by two different representations. Not great. The HTTP caching standard lets a cache merge the client’s entity tags with its own when it revalidates. So if the origin then replies 304 with a tag the client never sent, the cache has to build a full 200 from its stored copy. That’s correct revalidation (and it’s what stops the proxy from firing off a blind 304 for content the client never had).

A safe nginx proxy_cache configuration for crawlable sites

Cache HTML briefly, revalidate instead of refetching, and never hold on to error responses for long. The core directives:

1. proxy_cache_key $scheme$host$request_uri; keeps protocols and hostnames apart.
2. proxy_cache_valid set per status: short for 200, very short or none at all for 404 and 5xx.
3. proxy_cache_revalidate on; refreshes expired entries with conditional requests.
4. proxy_cache_use_stale updating error timeout; covers only those cases, with a lifetime you pick deliberately.
5. proxy_cache_background_update on; refreshes entries without making the visitor wait.
6. proxy_cache_lock on; sends just one request to the origin for a missing key.

Keep proxy_ignore_headers off for HTML unless the origin sends headers that are plain nonsense. Add add_header X-Cache-Status $upstream_cache_status; and log that value, so every bot request shows HIT, MISS or STALE. In practice that one log field saves a lot of guessing. All of this lives on your own origin or NGINX, by the way. A managed NGINX proxy setup can route your domains and IPs, but the caching rules are yours to set.

Purging and invalidating the cache when content changes

Hitting publish should clear the affected URLs straight away, not wait for the TTL to run out. You’ve got a few options: proxy_cache_purge via a third-party module or NGINX Plus, a version segment in the cache key, deleting the cache file for a given key, and a short TTL as the safety net. Purge related pages together - the article, its listing and category pages, the sitemap and the feed. Status changes need an immediate purge too. Page turned into a 404 or 410? Got a new 301? Crawlers will keep seeing the old status until that cached entry is gone.

Don’t vary cached pages by bot: keep crawlers and users on the same content

Giving Googlebot its own cached version based on user agent or IP flirts with cloaking, so crawlers should share the cache with visitors. Vary only on things that genuinely change the representation, like Accept-Encoding and maybe language. Never on the User-Agent string. Logged-in sessions and personalisation cookies get their own bypass rules, so private pages never land in the shared cache. More on the risks in our guide to staying clear of cloaking.

How to verify crawlers get fresh pages

Test from the outside with conditional requests, and from the inside with your logs:

• Run curl -I twice and compare X-Cache-Status across the two responses.
• Send If-None-Match or If-Modified-Since and expect a 304 with an empty body.
• Edit a page and check it now comes back as a 200 with a new ETag.
• Filter access logs for Googlebot and group requests by cache status and response code.
• Use URL Inspection in Search Console to see a live fetch.

Good reverse proxy caching SEO boils down to four habits: short TTLs for HTML, validators passed through intact, errors kept out of the cache, and a purge on every publish. That’s it, really. You’ll find related guides in our technical SEO category.

FAQ