What Vercel measured about AI crawlers

# What Vercel measured about AI crawlers

The numbers from one month of monitoring — which bots show up, how often, and what they actually fetch.

By AgentSite · 4 min read · Updated 2026-05-23

In December 2024, Vercel published an analysis of one month of AI-crawler traffic across their network — billions of requests, broken down by user-agent, fetch type, and behavior. The headline finding was that none of the major AI crawlers currently execute JavaScript. The full breakdown by bot is the part most citations of the post leave out.

This page catalogs the numbers. The interpretation is in [agent readability](/agent-readability); the practical diagnostic for a single site is in [SSR-junk and bot walls](/ssr-junk-bot-wall).

## The dataset

Vercel's analysis covered "the past month" of traffic across their own network plus monitoring of [nextjs.org](http://nextjs.org), published on December 17, 2024 ([Vercel, "The Rise of the AI Crawler"](https://vercel.com/blog/the-rise-of-the-ai-crawler)). The unit is HTTP requests reaching Vercel-hosted pages, classified by the declared user-agent of the requester.

The post is one of the few public data drops on the topic. Cloudflare's bot-traffic data, also from 2024, gives a complementary view at a different vantage point ([Cloudflare, "Declaring Your AIndependence," July 2024](https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/)). Where Vercel measured requests-to-pages, Cloudflare reported reach across sites — "AI bots accessed around 39% of the top one million Internet properties" in a single month, with GPTBot alone touching 35.46% of those sites.

## The numbers, by bot

| Bot | Requests in the month | Operator |
| --- | --- | --- |
| Googlebot (search + Gemini) | 4.5 billion | Google |
| GPTBot | 569 million | OpenAI (training) |
| Claude | 370 million | Anthropic |
| AppleBot | 314 million | Apple |
| PerplexityBot | 24.4 million | Perplexity |

Two things stand out. First, Googlebot is still roughly an order of magnitude larger than every AI crawler combined — search has not gone away, and a Google-friendly site still gets more bot traffic than any AI surface. Second, AppleBot's 314 million is high relative to AppleBot's visibility in citation-output dashboards; Apple Intelligence is crawling at meaningful scale even when its surfaced citations are quieter than ChatGPT's or Perplexity's.

PerplexityBot's 24.4 million is the smallest number on the list, but Perplexity is a live-retrieval engine — every PerplexityBot fetch is plausibly tied to a user query happening in real time. The volume ratio matters less than the latency surface.

## The JavaScript story

Vercel's framing is direct: "The results consistently show that none of the major AI crawlers currently render JavaScript."

A useful subtlety in the data: AI crawlers do _fetch_ JavaScript files at varying rates, but they do not _execute_ them. From the post:

-   ChatGPT crawler: 11.50% of requests fetched JavaScript resources
-   Claude crawler: 23.84% of requests fetched JavaScript resources

Fetching JS as a text resource is different from running it. The crawler downloads the file, scans it as text for any static signals it can extract (strings, URLs, perhaps function names), and moves on. Hydration never happens. The rendered DOM the user would see never exists for the bot.

The implication for a single-page-app site is what every other piece on this corpus repeats: the bot reaches the URL, receives an HTML shell containing roughly `<div id="app"></div>` plus a few hundred kilobytes of unexecuted scripts, and leaves with no extractable content. The crawler ran. It just ran against a page that, to it, was empty.

## What the data does not say

A few things worth not over-claiming:

-   **Per-page citation outcomes.** Vercel measured fetches reaching their network; they did not measure which fetched pages then appeared in AI-engine answers. The fetch is necessary but not sufficient for citation.
-   **Smaller bots.** The post tallies the largest crawlers by volume. Long-tail bots — small AI startups, regional engines, in-house RAG-pipeline crawlers — are not in the table.
-   **Time-of-day or geographic variance.** Aggregate monthly numbers smooth over hourly variation and per-region behavior.
-   **Bot verification.** Every number is based on declared user-agent strings. Forged user-agents would land in the same bucket as the real bot. Vercel's hosted environment removes the worst of this, but it is not a verified-bot count.
-   **What the bots did with the content.** The post measures input (fetches by Vercel's network); engines measure output (what they cite). The Robots Exclusion Protocol formalized in [RFC 9309](https://datatracker.ietf.org/doc/html/rfc9309) is the convention that makes the user-agent identification reliable in the first place — without it, the per-bot breakdown would be guesswork.

## Where this fits

The Vercel data is the canonical reference for "AI crawlers do not run JavaScript." It is cited inline across the corpus in [the five layers of AEO](/five-layer-aeo) (as the Layer-1 disqualifying gap), [SSR-junk and bot walls](/ssr-junk-bot-wall) (as the rationale for the byte-composition probe), [agent readability](/agent-readability) (as the underlying fact behind the two-bot framing), and [llms-txt](/llms-txt) (as the reason agents can't crawl page-by-page within a user-query latency window).

This page is the place to point to when a reader wants the actual numbers behind those one-line citations.