# How can you tell a real AI crawler from a spoofed one?

> A crawler user-agent string is a claim, not authentication; verify the source against the operator’s bot-specific published IP ranges.

Source: https://bestaibottrackers.com/spoofed-ai-crawlers · Publisher: Bot Trackers · Updated: 2026-11-15

## TL;DR
- A crawler user-agent string is a claim, not authentication; verify the source against the operator’s bot-specific published IP ranges.[3][2]
- Google’s documented DNS check requires reverse lookup, hostname validation and a forward lookup back to the original IP.[13]
- ASN ownership is supporting evidence only because legitimate and spoofed AI-crawler requests can share the same cloud provider.[2][3]
- Web Bot Auth provides cryptographic evidence only for products and requests that document signing; missing signatures do not prove spoofing.[6]
- Generic browser requests after a declared crawler is blocked can justify investigation, but the Cloudflare–Perplexity case shows that operator attribution may remain disputed.[5]

A real AI crawler should match more than its user-agent string. The strongest practical checks are an operator’s published IP ranges, forward-confirmed reverse DNS where documented, and—only for supported products—valid Web Bot Auth signatures; ASN information and request patterns are useful supporting evidence, not authentication on their own.[1]

This distinction matters because HUMAN found that 5.7% of AI-crawler-labelled requests in its customer-traffic sample were spoofed, while DataDome’s controlled 2026 test found that more than seven in ten tested sites allowed a spoofed AI agent or crawler without a challenge.[2] Those figures measure different things: observed production traffic versus controlled access tests, so neither should be treated as a market-wide prevalence estimate.[3]

### What counts as a spoofed or undeclared AI crawler?

A **spoofed AI crawler** presents a familiar identity—such as `GPTBot`, `ClaudeBot`, `ChatGPT-User` or `PerplexityBot`—but fails the operator’s network or DNS verification checks. HUMAN defines spoofed requests as those using a legitimate crawler user agent while coming from a nonmatching network source.[4]

An **undeclared crawler** is harder to authenticate because it may not identify itself as an AI service at all. Cloudflare described a case in which requests attributed to Perplexity allegedly changed from a declared crawler to a generic Chrome-on-macOS user agent after restrictions were applied.[5][6] Perplexity denied the attribution and said Cloudflare had confused its activity with unrelated Browserbase traffic.[7][8]

These categories should remain separate in a log pipeline:

- **Verified identity:** the request matches the applicable operator evidence.
- **Failed identity check:** the claimed identity does not match the relevant published range or DNS procedure.
- **Unattributed automation:** the request appears automated but cannot be reliably tied to a named operator.
- **Potentially undeclared activity:** a suspicious sequence warrants investigation but does not prove operator attribution.

A user-agent string alone is a claim, not proof of identity.[9][2]

## Which verification signals actually hold?

### Published IP ranges or JSON files

Published CIDR ranges are usually the most practical first check. The source IP must belong to the list for the specific crawler identity—not merely to a large cloud provider or an ASN associated with the operator.[3]

OpenAI publishes separate files for `GPTBot`, `OAI-SearchBot` and `ChatGPT-User`, reflecting different purposes.[2][10] Perplexity likewise publishes separate files for `PerplexityBot` and `Perplexity-User`, and recommends combining the matching user agent with the relevant IP list.[11][4]

Anthropic’s current Claude Help Center links a `bots.json` file containing crawler prefixes, despite an older Anthropic page saying that ranges were not published.[12][3] The current publication position should therefore be based on the live Claude documentation rather than the legacy statement.[2]

### Forward-confirmed reverse DNS

Reverse DNS is meaningful only when the full verification loop succeeds. Google’s documented procedure is:

1. Reverse-resolve the source IP.
2. Check that the returned hostname belongs to an accepted Google domain.
3. Resolve that hostname forward.
4. Confirm that the result includes the original source IP.[3][5]

A plausible PTR record by itself is insufficient. Google publishes separate address files for common crawlers, special-case crawlers and user-triggered fetchers, so a Google-owned address should not automatically be classified as `Googlebot`.[13][3]

Apple documents a comparable DNS and CIDR-based approach for Applebot, but its rules should not be generalized to other operators.[14]

### ASN

An ASN identifies the network associated with an IP address, but it does not authenticate a particular bot. HUMAN observed legitimate and impersonating `ChatGPT-User` traffic using the same Microsoft Azure ASN 8075.[4]

Use ASN information to identify anomalies, cluster requests and investigate infrastructure changes. Match the actual source IP against the operator’s published CIDRs before treating the claimed identity as verified.[4]

### Web Bot Auth signatures

Web Bot Auth can provide stronger cryptographic evidence, but coverage remains product-specific. The protocol is still an IETF Internet-Draft built on HTTP Message Signatures, rather than a completed RFC.[15]

OpenAI documents signed requests for ChatGPT Work’s cloud browser, not a blanket signing policy for every `GPTBot` request.[2][16] Google documents signatures for an experimental subset of Google-Agent requests and explicitly recommends retaining IP and DNS checks because coverage is incomplete.[17]

For `GPTBot`, `OAI-SearchBot`, `ClaudeBot`, `PerplexityBot` and `Applebot`, crawler-wide signing is not documented in the cited operator guidance.[9][2] Missing signatures therefore do not prove spoofing.

Even a valid signature proves only that a holder of the published key signed the covered message. It does not establish that the request is benign, authorised or entitled to access the resource.[18]

## How do the main operators publish verification evidence?

| Operator | Published IP ranges or JSON file | Reverse DNS available | Signed requests | Source |
|---|---|---|---|---|
| OpenAI | `gptbot.json`, `searchbot.json` and `chatgpt-user.json` are published separately.[10] | No official crawler reverse-DNS procedure is published in the cited overview.[3] | Documented for ChatGPT Work’s cloud browser; not established for all OpenAI crawlers.[3][16] | OpenAI crawler documentation.[1][3] |
| Perplexity | Separate `perplexitybot.json` and `perplexity-user.json` files.[3][4] | No reverse-DNS authentication procedure is published in the cited guidance.[7][8] | No crawler-wide signing guidance is published in the cited documentation.[1][17] | Perplexity crawler documentation.[4] |
| Google | Separate JSON files cover common crawlers, special-case crawlers and user-triggered fetchers.[3] | Yes; Google documents reverse lookup followed by forward confirmation.[13] | Experimental signing covers only a subset of Google-Agent requests.[17] | Google verification guidance.[17] |
| Apple | `applebot.json` publishes Applebot ranges.[14] | Yes; Apple documents DNS checks under its Applebot domains.[14] | No Applebot signing guidance is published in the cited documentation.[14] | Applebot guidance.[14] |
| Anthropic | Current Claude guidance links `claude.com/crawling/bots.json`.[3] | No reverse-DNS authentication procedure is published in the cited current guidance.[3] | No crawler-wide signing guidance is published in the cited documentation.[3] | Claude crawler guidance and JSON file.[12] |

The files can change, so production verification should refresh them rather than hard-coding an old list of addresses.[11] The correct comparison is also identity-specific: `ChatGPT-User` should not be validated against `GPTBot` ranges, and `Perplexity-User` should not be treated as `PerplexityBot`.[4][2]

## What log patterns suggest spoofing?

The following patterns justify investigation, but they do not all prove that a request is fake:

- A claimed bot’s source IP falls outside the operator’s published ranges.[4]
- A source IP belongs to the same ASN as legitimate traffic but misses the operator’s exact CIDRs.[4]
- A reverse-DNS hostname looks plausible but does not resolve forward to the original IP.[13]
- A declared crawler is blocked, followed by generic browser requests to the same restricted content.[5]
- Source IPs or ASNs rotate after access restrictions are applied.[6][5]
- Thousands of IPs each make only one or two requests.[4]
- A browser-looking request repeatedly fetches pages in a sequence consistent with automated research.[2]
- Claimed identities, source networks and requested resources do not fit the operator’s documented purpose.[3]

HUMAN reported more than 10,000 IPs in one spoofing campaign, with 61% making only one or two requests.[4] That pattern is consistent with distributed activity, but low request volume per IP is not proof of a particular operator.

A suspicious browser user agent is especially difficult to attribute. It may represent an undeclared crawler, a user-triggered fetch, a third-party browsing service or an unrelated scraper. Cloudflare attributed its controlled-domain observations to Perplexity, while Perplexity disputed that conclusion and identified Browserbase traffic as the source of the discrepancy.[11][7]

## How should you check a suspicious request?

1. **Preserve the complete evidence.** Record the timestamp, source IP, port, user agent, requested URL, method, status code, referrer, selected headers, TLS or HTTP fingerprints if available, and nearby requests from the same session or network.
2. **Classify the claimed identity.** Separate `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `PerplexityBot`, Googlebot, Applebot and other identities before selecting a verification method.[2]
3. **Check the exact published network range.** Download the operator’s current JSON or CIDR file and test the source IP against the correct bot-specific list.[4]
4. **Run the documented DNS or signature check.** For Google, complete both reverse and forward DNS verification; for supported signed products, validate the covered components against current public keys rather than merely checking whether signature headers exist.[1][4]
5. **Compare the request sequence and report the narrowest conclusion.** Label the request as verified, failed verification, or unattributed automation; retain operator attribution as unresolved unless the evidence supports it.

A failed IP match is strong evidence that the request does not originate from the documented range for that identity. It is not, by itself, proof of which other operator or scraper sent the request.


What did the documented stealth-crawling case show—and what did it not prove?

Cloudflare said it created two newly purchased, undiscoverable domains, restricted them with `robots.txt`, blocked Perplexity’s declared crawlers, and then observed requests using a generic Chrome-on-macOS user agent. Cloudflare also reported IP and ASN changes and said Perplexity returned detailed information matching the restricted pages.[5][11]

Perplexity denied stealth crawling and said Cloudflare had attributed unrelated Browserbase traffic to Perplexity. It said its own Browserbase use was limited and that the reported traffic volume did not represent its operation.[3]

The case showed a documented request sequence consistent with evasion and produced a public attribution dispute. It did not independently prove that every generic-browser request was operated by Perplexity, establish a universal stealth-crawler signature, or resolve the disagreement over third-party infrastructure.[15][1]



## What should a crawler policy do with uncertain traffic?

Authentication and permission are different decisions. A request can match an operator’s published IP range and still be subject to the site’s access policy; conversely, an unsigned request is not automatically malicious.[3]

A practical policy can use graduated handling:

- Allow verified requests that meet the site’s access rules.
- Rate-limit requests that fail verification but show low-risk behaviour.
- Challenge or block repeated automation from nonmatching networks.
- Log generic browser requests that access restricted or undiscoverable content.
- Escalate recurring sequences for attribution rather than assigning a vendor name immediately.

`robots.txt` is also not an authentication mechanism. It expresses crawl preferences, while IP, DNS and signature checks address the identity question. A disagreement about whether a crawler should have accessed a page does not, on its own, prove that the user agent was forged.[6]

## Why do benchmarks and reported percentages need careful reading?

HUMAN’s 5.7% figure describes AI-crawler-associated traffic observed across its customers during a two-week period; its exact total request count and customer sample size were not published.[4][2] DataDome’s 2026 result comes from a controlled test of 21,491 valid domains in June 2026, designed to measure whether spoofed agents could pass site defences.[19]

Those results answer different questions. A controlled pass rate is not the share of real production traffic that is fake, and a vendor’s observed customer-traffic percentage is not a universal Internet estimate.[2]

An academic study provides a further caution: Kim and colleagues analysed 40 days of traffic across 36 institution-owned websites and flagged possible spoofing through ASN anomalies, but explicitly said the method could not definitively establish user-agent spoofing or be generalised beyond those sites.[6]

The correct reporting pattern includes the observation window, population, detection method and denominator. Without those details, a precise-looking spoofing percentage can be misleading.

## FAQ
### How can I detect a fake GPTBot?
Compare the request’s source IP with OpenAI’s current `GPTBot` ranges rather than relying on the `GPTBot` user agent.[2][3] A source address outside the published list is a failed network-identity check, but it does not by itself identify the actual sender.

### How do I verify ClaudeBot?
Use the current Claude crawler guidance and its linked `bots.json` file to check whether the source IP belongs to an Anthropic-published prefix.[3] The older Anthropic page saying that ranges were not published is no longer an accurate summary of the current Claude documentation.[3][11]

### Is an AI crawler’s ASN enough to prove it is genuine?
No. HUMAN observed legitimate and spoofed `ChatGPT-User` requests in the same Microsoft Azure ASN, showing why hosting-provider ownership is weaker than an exact published CIDR match.[4] ASN data is useful context, but it should support—not replace—the relevant IP, DNS or signature check.

### Can reverse DNS prove that a crawler is real?
Only when the operator documents a complete DNS procedure and the check succeeds. Google requires reverse lookup, validation of the returned hostname and a forward lookup that returns the original IP; a plausible PTR record alone is insufficient.[13]

### Does a missing Web Bot Auth signature prove spoofing?
No. Signing coverage is limited and product-specific: OpenAI documents signatures for ChatGPT Work’s cloud browser, while Google documents signing for an experimental subset of Google-Agent requests.[17] The cited crawler guidance does not establish crawler-wide signing for GPTBot, ClaudeBot, PerplexityBot or Applebot.[2][20]

### Does a generic Chrome user agent prove an undeclared AI crawler?
No. It is a suspicious signal only when combined with request sequences, access to restricted or undiscoverable content, network changes or other evidence.[3] Cloudflare attributed one such case to Perplexity, but Perplexity disputed that attribution and cited unrelated Browserbase traffic.[7][8]

## Sources
[1] Web Bot Auth — https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth
[2] 1 State of Bot & Agent Security Report 2026 — https://datadome.co/wp-content/uploads/2026/09/State-of-Bot-Agent-Security-Report_2026-Final.pdf
[3] Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers — https://www.sysnet.ucsd.edu/~voelker/pubs/robots-imc25.pdf
[4] AI Crawler Spoofing: Attackers Impersonate ChatGPT & Perplexity — https://www.humansecurity.com/learn/blog/ai-crawler-spoofing-chatgpt-mistral-perplexity
[5] Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives — https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives
[6] Scrapers Selectively Respect robots.txt Directives:Evidence From a Large-Scale Empirical Study — https://arxiv.org/html/2505.21733v2
[7] Agents or Bots? Making Sense of AI on the Open Web — https://www.perplexity.ai/hub/blog/agents-or-bots-making-sense-of-ai-on-the-open-web
[8] Perplexity on X: "https://t.co/0kTPCUluVJ" / X — https://x.com/perplexity_ai/status/1952531537385456019
[9] AI Bot & Agent Traffic Trends 2026 | DataDome — https://datadome.co/resources/bot-and-agent-security-report/ai-traffic-trends
[10] Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots
[11] Perplexity Crawlers — https://docs.perplexity.ai/docs/resources/perplexity-crawlers
[12] Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-the-web
[13] Verify Requests from Google Crawlers and Fetchers | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers — https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests
[14] About Applebot - Apple Support — https://support.apple.com/en-us/119829
[15] draft-ietf-webbotauth-httpsig-protocol-01 - HTTP Message Signatures for automated traffic — https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol
[16] ChatGPT Work's Cloud browser allowlisting | OpenAI Help Center — https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting
[17] Google's Guide to Authenticating Requests with Web Bot Auth (Experimental) | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers — https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth
[18] Message Signatures are now part of our Verified Bots Program, simplifying bot authentication — https://blog.cloudflare.com/verified-bots-with-cryptography
[19] Bot & AI Agent Security Report Methodology | DataDome — https://datadome.co/resources/bot-and-agent-security-report/metholodogy/
[20] Global Bot Security Report 2025 Final 1 — https://datadome.co/wp-content/uploads/2025/09/Global-Bot-Security-Report_2025-Final-1.pdf
