AI crawlers found zero of the pages linked only by JavaScript

In a 41-day experiment on one directory site, several crawlers reached pages linked in the initial HTML but none of the deeper pages linked only through JavaScript. The study is a useful test case for navigation, with limits on what it tells us about indexing or AI citations.

The difference in the page response

A crawler is a program that reads web pages the way a very fast, very literal person would — it opens a page, notes the links on it, and goes to those next. Links are one way to discover pages; sitemaps and other discovery routes can also matter.

The distinction is when the link becomes available:

• In the initial HTML: the server sends the link with the page. A crawler can follow it without running a script.
• Added by JavaScript: the browser creates the link after the page arrives. A crawler must run that script to see it.

A framework can use either approach, depending on how the site is configured and rendered.

A browser with JavaScript enabled can run that program. A crawler only does if somebody built it to, and it’s expensive — you have to launch what amounts to a whole browser for every page. Do not assume that a fetcher renders it.

The experiment

Vinicius Stanula, Associate Director of SEO at LOCOMOTIVE, took a Brazilian business directory of roughly 2,400 pages and cut its navigation in half. Of the 21 sections in the test hierarchy, 11 kept hard-coded HTML links and 10 had their links injected by JavaScript instead.

Everything else that could give a crawler a back door was switched off: no sitemap, no breadcrumbs, no internal panels pointing at the deeper pages. Then he logged every single bot request for 41 days — path, user agent, IP, timestamp — and let the crawlers do whatever they were going to do.

The write-up is on Search Engine Land, published 19 August.

The setup deliberately restricted alternative discovery routes. When a page is missing from an AI answer you can rarely tell whether the crawler couldn’t find it or found it and didn’t care. Here there was only one route in.

The numbers

For four crawlers, the difference was clear. These are counts of deeper pages reached in the two groups:

GPTBot fetched all 759 pages in the HTML group on day one; ClaudeBot did so a day later. Both visited the JavaScript group’s section pages but reached none of its deeper pages. Bingbot reached three JavaScript-linked pages, about 1% of that group.

Google’s crawlers behaved differently. GoogleOther, used for research and product work, reached 48% of the JavaScript group. Googlebot, which feeds Google Search, reached 2% of that group and 5% of the HTML group.

These logs establish which pages were fetched in this setup. They do not explain the providers’ reasons or show which pages later appeared in search results or AI answers.

An earlier source provides additional context. Vercel’s crawler study back in December 2024 found the same thing from the other direction: no major AI crawler renders JavaScript.

GPTBot and ClaudeBot do download JavaScript files — Claude in nearly a quarter of its requests — they just never run them. That observation comes from a different dataset and should be read with its date and scope attached.

Getting it back is slower than losing it

Stanula then converted the JavaScript links to plain HTML and kept logging. Recovery was not uniform. GPTBot swept 250 previously unreachable pages within 48 hours. Bingbot worked through its backlog over about a week. ClaudeBot mostly carried on recrawling pages it already knew about, and Googlebot had barely moved by day 41.

So the cost of the mistake isn’t symmetrical with the cost of the fix. Restoring links did not produce the same recrawl timing for every crawler in the test.

What to do about it

Turn JavaScript off and load a category page and a deep product or article page. Check whether navigation, category links and pagination still work. This helps reveal what a fetcher receives without rendering scripts; the next step is to inspect the server response.

Then check what the server actually sends. curl the URL and search the response for a link you know should be there. If the link is absent, a fetcher that does not render JavaScript cannot follow it from that response.

Where links are missing, render the navigation on the server. Focus on the links needed to discover the site’s pages; this does not require replacing the whole application.

Google’s guidance for generative AI features also keeps the ordinary search requirements in place: a page must be indexed and eligible to appear with a snippet. Readable navigation helps discovery, but does not guarantee indexing or a citation.