Skip to content
SEO

Robots.txt Unreachable: How Server Errors Stop Google Crawling

Blue network cables plugged into a server switch, the path every crawler request depends on
In this article
  1. How Do robots.txt Status Codes Change What Google Does?
  2. Why a 403 for Googlebot Is the Wrong Way to Control Crawling
  3. Where Robots.txt Fetch Failed Errors Show in Search Console
  4. How to Check That Your Server Serves robots.txt Correctly
  5. Robots.txt Behind a Reverse Proxy: The Origin Still Has to Answer
  6. Robots.txt Unreachable: What to Remember
  7. FAQ

When robots.txt is unreachable because the server returns an error, Google doesn’t skip the file and carry on. It pauses crawling of the whole site until it can read the file again. If Search Console shows robots.txt fetch failures, or your site sits behind a firewall, proxy or hosting panel that sometimes answers /robots.txt with an error or an HTML page, start with the server, not the rules in the file. The cause is usually the server, the firewall or whatever sits in front of it.

How Do robots.txt Status Codes Change What Google Does?

The HTTP status code on /robots.txt decides whether Google reads the file, ignores it or stops crawling. Google’s robots.txt specification describes how each response class is handled, and notes that Google generally caches the contents of the file for up to 24 hours, so neither a fault nor a fix shows up instantly. The rules from the specification:

  • 2xx: the file is processed as provided.
  • 3xx: Google follows at least five redirect hops, then treats the result as a 404.
  • 4xx except 429: handled as if a valid robots.txt file didn’t exist, so Google assumes no crawl restrictions.
  • 5xx: for the first 12 hours, Google stops crawling the site but keeps trying to fetch the file; for the next 30 days it uses the last good version. After that, a generally available site is treated as having no robots.txt, and a site with general availability problems stops being crawled.
  • 503: results in fairly frequent retrying, so a short maintenance window hurts less than a long outage, but it still pauses crawling.
  • DNS or networking issues, such as timeouts, count as a server error.

Why a 403 for Googlebot Is the Wrong Way to Control Crawling

Returning a 403 on robots.txt to Googlebot doesn’t limit crawling. Google reads it as a missing file and assumes there are no restrictions, which is the opposite of what you wanted. The specification says it directly: don’t use 401 and 403 status codes for limiting the crawl rate. The block is rarely deliberate. It usually comes from your own stack: a firewall rule, a rate limit or a bot filter on the server or CDN that rejects the Googlebot user agent.

An accidental “Disallow: /” is a different problem. There the file is reachable and blocks everything, so you fix the rule in the file, not the server.

Where Robots.txt Fetch Failed Errors Show in Search Console

Look in the Crawl Stats report, under host status. Open it from the Search Console settings, check whether robots.txt fetching is flagged as having problems, and note when the failed requests happened. Repeated failures mean crawling has been paused or limited, following the rules above. Line those dates up against deploys, server restarts, DNS changes and proxy edits; a match is usually your answer.

How to Check That Your Server Serves robots.txt Correctly

Fetch /robots.txt from outside your network, with and without a Googlebot user agent, and confirm you get a 200 status and plain-text content. A test from inside the network, or from a logged-in admin session, can hide rules that apply only to outside traffic or to bots.

  1. Request the file from an external machine twice: once with a normal browser user agent and once with a Googlebot user agent string.
  2. Compare the status codes of both responses. Any difference needs an explanation.
  3. Check that the Content-Type header is text/plain, not text/html.
  4. Read the body and confirm it contains your rules, not a login form, a challenge page or a maintenance notice.

A firewall, rate limit or maintenance page should never answer robots.txt with a 403, a 5xx code, or an HTML page sent with a 200. Serve robots.txt as a static file straight from the origin or NGINX, outside any application that can crash, time out or redirect visitors to a login screen. Repeat the check after every server, DNS or proxy change, because that is when these problems tend to appear.

Robots.txt Behind a Reverse Proxy: The Origin Still Has to Answer

A reverse proxy passes requests along, so if the origin fails, robots.txt fails through the proxy too. Jalvo works this way: as an SEO reverse proxy per site, it forwards requests to your existing hosting and gives each site its own IP. The fix therefore lives on the origin, with a static robots.txt served reliably by your own hosting or NGINX. Once that works and you’re deciding how many IPs your sites need, you can compare the proxy plans.

Robots.txt Unreachable: What to Remember

A robots.txt unreachable error is a server problem first. 5xx responses and timeouts pause crawling, a 4xx other than 429 removes your crawl restrictions, and a 403 is never a way to control crawl rate. Serve the file statically, test it from outside with a Googlebot user agent, and check it again after every infrastructure change.

FAQ

Does Google stop crawling if robots.txt returns a 404?

No. Any 4xx other than 429 is treated as if no robots.txt existed, so Google assumes there are no crawl restrictions and keeps crawling. If you have no rules, that’s fine, but any Disallow lines you relied on stop applying.

How long can a robots.txt 503 last before it hurts crawling?

It affects crawling from the first failure, because Google stops crawling while it can’t fetch the file. Google retries a 503 fairly often, but crawling stays paused until the file is back, so keep maintenance windows short.

Is a ‘Disallow: /’ rule the same as an unreachable robots.txt?

No. With “Disallow: /” the file is reachable and Google reads it. The rule itself blocks the whole site, so you fix the rule in the file, not the server.

ShareLinkedInX
Web Systems team

The team that builds and runs Jalvo.

Give each site its own IP.

Add your first domain in a few minutes.

Start now

Choose which cookies we may use. Necessary cookies keep the site and your login working and cannot be switched off.