AgentSiteBot

# The crawler behind the curtain.

AgentSiteBot is the headless browser [agentsite](/) uses to render your single-page-app pages so they can be served back to LLM crawlers (GPTBot, ClaudeBot, PerplexityBot, and other agents that don't execute JavaScript). It only crawls domains a registered owner has connected to an account, and only after a token has been used.

## How to identify it

AgentSiteBot identifies itself with a deterministic User-Agent string:

User-Agentcopy

```
AgentSiteBot/1.0 (+https://agentsite.com/bot)
```

## How to allowlist it (Cloudflare)

If your site is behind Cloudflare and our renderer is being challenged, add a WAF custom rule:

1. Cloudflare → your zone → _Security → WAF → Custom rules_.
2. Match: `(http.user_agent contains "AgentSiteBot")`.
3. Action: **Skip** → Bot Fight Mode, Browser Integrity Check, Managed Challenge.
4. Deploy.

We don't yet publish a static IP range — agentsite's renderer runs on rotating egress, so allowlist by User-Agent is the supported method for now.

## How to block it

If you don't want AgentSiteBot to render your site, two options:

- Don't register the domain in agentsite — without a valid token + matching domain, our crawler never visits.
- Add a `Disallow` entry in your `robots.txt`:

robots.txtcopy

```
User-agent: AgentSiteBot
Disallow: /
```

## What it does _not_ do

- It does not run JavaScript that submits forms or POSTs.
- It does not follow links beyond the path it was asked to render.
- It does not log in, accept cookies meaningfully, or attempt to bypass auth walls.
- It does not crawl your site on a schedule of its own — it renders on demand when a request from your installed snippet (or a direct API caller) reaches us.

## Contact

Questions, abuse reports, allowlist help: [get in touch](/contact).

We use cookies to make this site work and to understand how it's used. [Learn more](https://www.cookiesandyou.com/)

Decline Got it