AI crawlers found zero of the pages linked only by JavaScript

TL;DR: Websites are held together by links. Some of those links are written directly into the page, so anyone reading the raw file can see them. Others only appear after the visitor’s browser runs a small program and builds them on the fly. Human visitors can’t tell the difference. The software that reads the web for ChatGPT and Claude very much can: in a 41-day test published on Search Engine Land, those readers found every page on the hard-coded side of a site and not one page on the other side. Zero, not “fewer”.

First, what are we actually talking about?

A crawler is a program that reads web pages the way a very fast, very literal person would — it opens a page, notes the links on it, and goes to those next. If nothing links to a page, that page effectively does not exist as far as that crawler is concerned.

There are two ways a link can end up on a page. It can be hard-coded: the address is sitting in the raw HTML file the server sends out, so it’s there the moment the file arrives. Or it can be injected by JavaScript: the file arrives without the link, and the browser then runs a small program that creates it. Modern site frameworks do the second one constantly, often without the developer choosing it.

A browser always runs that program. A crawler only does if somebody built it to, and it’s expensive — you have to launch what amounts to a whole browser for every page. Most crawlers don’t bother.

The experiment

Vinicius Stanula, Associate Director of SEO at LOCOMOTIVE, took a Brazilian business directory of roughly 2,400 pages and cut its navigation in half. Of the 21 sections in the test hierarchy, 11 kept hard-coded HTML links and 10 had their links injected by JavaScript instead. Everything else that could give a crawler a back door was switched off: no sitemap, no breadcrumbs, no internal panels pointing at the deeper pages. Then he logged every single bot request for 41 days — path, user agent, IP, timestamp — and let the crawlers do whatever they were going to do. The write-up is on Search Engine Land, published 19 August.

The setup matters, because it removes the usual excuse. When a page is missing from an AI answer you can rarely tell whether the crawler couldn’t find it or found it and didn’t care. Here there was only one route in.

The numbers

On the hard-coded side, the AI crawlers behaved impeccably. GPTBot, OpenAI’s crawler, swept the site on day one and pulled all 759 pages of the HTML group. ClaudeBot, Anthropic’s, did the same thing a day later — deep, systematic, complete. Amazonbot got 735. Meta’s crawler got 537.

On the JavaScript side, the same crawlers hit the ten section pages, found nothing that looked like a link, and stopped. GPTBot: zero of the deeper pages. ClaudeBot: zero. Amazonbot: zero. Meta: zero. Bingbot managed three, which rounds to about 1%.

Google’s crawlers were the only ones that ran the JavaScript at all, and even there the picture is worse than the usual reassurance suggests. GoogleOther — the fetcher Google uses for research and product work rather than for building the search index — reached 48% of the JavaScript pages. Googlebot, the crawler that actually feeds Google Search, reached 2%. Googlebot was thin on the HTML side too, at 5%, so the contrast isn’t really one perfect crawler against seven broken ones. Running JavaScript costs money, and the crawlers that matter for AI answers have decided not to spend it.

None of this is new behaviour, which is the uncomfortable part. Vercel’s crawler study back in December 2024 found the same thing from the other direction: no major AI crawler renders JavaScript. GPTBot and ClaudeBot do download JavaScript files — Claude in nearly a quarter of its requests — they just never run them. Different method, twenty months apart, same result.

Getting it back is slower than losing it

Stanula then converted the JavaScript links to plain HTML and kept logging. Recovery was not uniform. GPTBot swept 250 previously unreachable pages within 48 hours. Bingbot worked through its backlog over about a week. ClaudeBot mostly carried on recrawling pages it already knew about, and Googlebot had barely moved by day 41.

So the cost of the mistake isn’t symmetrical with the cost of the fix. A crawler that never saw a page has no reason to re-check the section it wrote off, and some of them clearly don’t.

What to do about it

Turn JavaScript off in your browser and load your own site. If the navigation disappears, or the category links vanish, or the pagination stops working, that is roughly the page GPTBot and ClaudeBot are reading. Do it on a category page and a deep product or article page, not just the homepage — homepages are usually the best-built page on any site.

Then check what the server actually sends. curl the URL and search the response for a link you know should be there. If it isn’t in that raw text, it isn’t in the crawler’s copy either.

Where you find gaps, render the navigation on the server. You do not have to convert an entire application; navigation links are the part that has to survive without JavaScript, and most frameworks can serve those in the initial HTML without much argument. Google’s own guidance for generative AI features says the same thing in duller language: a page must be indexed and eligible to appear with a snippet before it can show up in an AI answer, and JavaScript SEO best practices sit among the fundamentals that get it there.

Doing that check once is easy. Doing it across a few hundred templates, on a portfolio of client sites, after every framework upgrade that quietly changes how links are emitted — that’s the part that never gets done. Preferium crawls every page with 47 automated checks and scores the site 0–1000, and where it finds a fix it makes the change, deploys it, and re-checks the live page with a real browser afterwards. The re-check is the relevant bit here: a link that only exists after JavaScript runs and a link that exists in the HTML look identical in a screenshot, and only one of them is visible to the crawler that decides whether ChatGPT has heard of you. See what the scan actually checks.

Key takeaways

• In a 41-day logged test, GPTBot, ClaudeBot, Amazonbot and Meta’s crawler reached none of the pages that were linked only via JavaScript, while crawling the hard-coded half of the same site completely.
• Googlebot reached 2% of the JavaScript-linked pages. GoogleOther, which does not build the search index, reached 48%.
• “Google renders JavaScript” is true and almost irrelevant, because the AI answer engines are not Google.
• Recovery after fixing the links was uneven: GPTBot re-crawled 250 pages within 48 hours, Googlebot had barely responded 41 days later.
• The five-minute version of this audit: disable JavaScript, load a deep page, see whether the links are still there.