JavaScript SEO: How Googlebot Fetches and Renders Resources

Googlebot grabs the HTML of the parent URL first. Then the Web Rendering Service fetches the JavaScript and CSS that markup points to and builds the page the way a browser would. Text that only shows up after that step? It reaches the index only if every required resource is reachable. That’s JavaScript SEO in a nutshell, and it explains a symptom most of us have seen: the page looks complete in your browser, yet Google shows barely any of its text. The topic is fresh, too, because on December 3, 2024 Google opened its Crawling December series with a post on resource fetching and crawl budget.

How does Googlebot fetch a JavaScript page?

It starts with the initial HTML from the parent URL. The resources that HTML references get fetched afterwards, for rendering. Google’s post on crawling page resources spells it out: that initial data may point to JavaScript, CSS, images and videos, exactly as it does for a browser. Here’s the sequence:

1. Robots.txt check - a disallowed URL gets skipped, no request at all.
2. HTTP request - the crawler pulls the HTML of the page.
3. Render queue - the page sits and waits until Google’s resources allow processing.
4. Resource fetches - scripts, stylesheets and the data they call are downloaded.
5. Rendered HTML - this is what gets used for indexing and link discovery.

What the Web Rendering Service does with your resources

The Web Rendering Service (WRS for short) takes everything it downloaded and constructs the page as a user’s browser would. Under the hood it’s a headless Chromium that executes the JavaScript. Which pages get in line? Google’s JavaScript SEO basics say that pages with a 200 status code are queued for rendering unless a robots meta tag or header tells Google not to index them. The wait can be a few seconds. But it may take longer.

Google indexes the rendered HTML and then parses it again for links. So menus, pagination and related-content blocks built by scripts have to end up as crawlable internal links in that output. If they don’t, the pages behind them stay undiscovered.

Blocked JavaScript files and other reasons text goes missing

Google Search won’t render JavaScript from blocked files or on blocked pages. Put a robots.txt disallow on a script, stylesheet or API path and the rendered HTML comes out without the content they produce. And Googlebot rendering fails quietly here: the HTML shell gets indexed, the text never arrives. No alarm, nothing. Check these causes on your own origin first:

• disallowed /assets or /api paths in robots.txt,
• resource hosts that refuse bot requests,
• rate limits that block crawlers while they download scripts,
• API calls returning errors instead of data.

Googlebot relies on HTTP status codes to find out if something went wrong. A client-side app that answers with a success code for a missing or empty page creates the mess covered in soft 404 errors explained. Routing matters as well: URLs should be accessible through the History API, not fragments. And every fix here applies to all visitors alike. You want one working page, not a separate version for crawlers.

Page resources crawling and your crawl budget

Every resource fetched for rendering eats into the crawl budget of the hostname that hosts it. To soften that cost, WRS attempts to cache each JavaScript and CSS file referenced in pages it renders, for up to 30 days and regardless of HTTP caching directives, as the Crawling December post on resources states. Media counts too: fetches by Googlebot-Image and Googlebot-Video consume the site’s budget.

So what do you do with crawl budget for JS and CSS? Simple, really. The fewer resources your main content needs, the less budget goes to rendering and the more is left for other crawl tasks. Google also updated the post on December 6, 2024 to note a performance impact of serving resources from a different origin. In my opinion that’s worth weighing before you move assets to another hostname.

Why Google may run an outdated version of your script

WRS caches aggressively and may ignore caching headers. The side effect: it can render a page with old JavaScript or CSS. The fix is content fingerprinting. A hash of the file content becomes part of the filename, such as main.2bb85551.js, so each update produces a new URL and the stale copy is never requested again.

What you should avoid is cache-busting query parameters that change on every request without a content change. New URLs mean new fetches, and those burn budget on files Google already holds. Set up fingerprinting in your build tool, then make sure your own origin or NGINX actually serves the hashed files correctly.

How to check what Googlebot rendering actually produced

Open the rendered HTML in the URL Inspection Tool or the Rich Results Test and search it for the text that’s missing in Google. Only content visible in that output counts (for web components, the shadow DOM and light DOM are flattened into it). I’d work through the evidence in this order:

1. the rendered HTML itself,
2. the list of resources that could not be loaded,
3. robots.txt rules covering those files,
4. status codes of scripts and API endpoints.

In practice, client-side rendering SEO comes down to one habit: put the key text and links into the initial HTML where your framework allows it, for every visitor alike. Got a proxy sitting in front of the site? The picture stays the same - Jalvo forwards requests to your origin, so rendering and resource rules remain in the origin or CMS configuration.

The rule behind JavaScript SEO is short. Google can index only what its renderer can fetch and build. Unblock the scripts, stylesheets and API paths your content depends on, fingerprint the files so updates get fresh URLs, and verify the rendered HTML after each significant release.

FAQ