Do AI crawlers run JavaScript?

AEO

6 August 20269 min readAEO

There is a sentence in almost every AEO article, and I had it on this site too: most AI crawlers do not execute JavaScript, so client-side content is invisible to them. It is close enough to be useful and wrong enough to send you after the wrong problem.

The sentence collapses three different systems into one. Once you separate them, the advice gets narrower, cheaper and easier to act on.

Three fetch paths

Dedicated HTTP crawlers. OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic's ClaudeBot, Claude-User and Claude-SearchBot. PerplexityBot and Perplexity-User. Meta-ExternalAgent, Bytespider. These pull raw HTML. There's no evidence any of them run JavaScript, and I'll get to how thin that evidence is.

Rendering crawlers. Googlebot renders with headless Chromium and always has. Apple is even more explicit, and almost nobody quotes it: "Applebot may render the content of your website within a browser." That is Apple's own documentation, and it goes on to warn that blocking JS or CSS in robots.txt will stop it rendering properly.

Agentic browsers. Perplexity Comet, ChatGPT agent mode, Claude in Chrome. These drive a real browser. Everything runs.

Why the Google part matters most

Google's own guidance on AI features says a page qualifies as a supporting link in AI Overviews and AI Mode by being indexed and eligible to be shown in Google Search. Indexing is the whole bar. AI Overviews are served off the Search index, and that index is built by a crawler that renders.

So the honest version of the claim is narrower and more useful:

A client-rendered page can rank in Google, qualify for an AI Overview, and still be absent from ChatGPT, Claude and Perplexity.

That is a problem worth fixing. Most articles pitch a bigger one, total invisibility to AI, and the evidence does not support it. Google is where most AI-answer impressions happen, and Google can see you.

The evidence is thinner than the confidence

Chasing the citation changed how I talk about this. I went looking for the source of "AI crawlers do not execute JavaScript" and it is, essentially, one study: Vercel and MERJ, published 17 December 2024. Roughly 569 million GPTBot requests, 370 million from Claude. JS files were fetched and not executed.

Good study. But it's from December 2024, it ran mostly on a JavaScript framework's own documentation site, and there is no published replication. Twenty months later every 2026 article still cites it, several while claiming that vendor documentation confirms it.

It does not. I read OpenAI's bot documentation end to end. It lists four bots, their user agents, their robots.txt tokens and their IP ranges. It says nothing about rendering, JavaScript or the DOM.

The only vendor sentence I could find is Anthropic's, and it has a scope people miss. Their API docs say the web fetch tool "currently does not support websites dynamically rendered with JavaScript". That documents the developer-facing fetch tool. The ClaudeBot crawler fleet is a separate system, and the sentence says nothing about it. It is strong evidence about Anthropic's retrieval stack and weaker evidence about the crawlers.

The widely repeated "69% of AI crawlers cannot execute JavaScript" is worse. It is a traffic-share number from that same dataset, describing what proportion of requests came from non-rendering bots on those specific properties. It gets restated as a property of crawlers, which is a different claim that nobody has supported.

What I check on the page itself

Fetch your own page as each bot and read what comes back. It takes a minute and it beats every article on the subject, including this one.

curl -sL https://example.com \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  | grep -c "the sentence you need to be found for"

Zero means the words are not in the served HTML. That's the test that decides it, and it answers the question for your page.

Two things the usual advice misses. Ctrl-F on view-source resolves less than people think, because for a lot of patterns the text is in the source and still never gets used: content inside <details>, panels behind display:none, anything past a lazy-loading boundary. And nobody separates the user agents. GPTBot builds a training corpus, OAI-SearchBot builds a search index, ChatGPT-User fetches live when someone asks a question. They do structurally different jobs and get tested as one blob.

The robots.txt part people get wrong

Two robots.txt tokens look like crawlers. Both are usage permissions.

Google-Extended has no user agent and fetches nothing. Google's documentation is blunt: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". It governs Gemini training and grounding. Blocking it does not remove you from AI Overviews.
Applebot-Extended likewise "does not crawl webpages". It only decides how already-crawled data may be used.

Perplexity's own documentation says Perplexity-User "generally ignores robots.txt rules", on the grounds that a person asked for the page. Anthropic, by contrast, makes an unqualified plural commitment that its bots honour robots.txt, with no carve-out for the user-triggered one.

What this changes

If your site is server-rendered, none of this is your problem and you can stop reading AEO checklists. If it is a client-rendered app, the diagnosis is narrower than "AI cannot see you": a specific, nameable set of crawlers cannot, Google can, and the cheapest move is to serve the content those crawlers need as HTML.

The sentence I removed from this site was directionally right and precisely wrong. Precisely wrong is how you end up rebuilding something that was working.