Updated

Which AI crawlers exist and what are their user agents?

AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing.

10 minute readMarkdown versionFact-checked

Table of AI crawler tokens by operator, marked by job: model training, search index or user-triggered fetch

TL;DR

  • GPTBot crawls content that may be used to train OpenAI foundation models, while OAI-SearchBot supports ChatGPT Search and ChatGPT-User handles user-initiated visits.[1]
  • ClaudeBot is Anthropic’s model-training crawler, and Claude-SearchBot and Claude-User serve distinct search and user-requested access purposes.[5]
  • Google-Extended is a robots.txt control token rather than a separate HTTP user agent.[2]
  • PerplexityBot, CCBot, Amazonbot and DuckAssistBot each have documented robots.txt tokens, but their purposes and published IP information differ.[8][7]
  • Robots.txt is a policy signal rather than a guaranteed block, so crawler verification should combine user-agent, IP, DNS and request-behavior checks.[5]

AI crawlers are automated agents that fetch web pages for model training, search indexes, AI-generated answers, product discovery, advertising checks, or user-requested browsing. The most clearly documented agents include OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User; Anthropic’s ClaudeBot; PerplexityBot; Google’s Google-Extended robots token; Applebot-Extended; Meta-ExternalAgent; CCBot; Amazonbot; DuckAssistBot; and xAI’s Grok Bot.[1]

This reference separates three things that are often confused:

  • An HTTP user agent is the identifier sent in a request header.
  • A robots.txt token is the name used in a User-agent directive.
  • An IP range helps verify that a request actually came from the claimed operator.

A robots.txt token is not always a separate HTTP crawler. Google-Extended, for example, controls how Google-crawled content may be used for Gemini but does not identify a separate Google request agent.[2] Applebot-Extended is documented as a secondary user agent for content-use controls rather than as a separate published crawler network.[3]

AI crawler user agents at a glance

The table below lists the main AI-related crawlers and control tokens documented in the research sources. User-agent strings can change, so use each operator’s current documentation or IP file when implementing verification rules.

Operator HTTP user agent or robots token Main purpose robots.txt token Published IP information
OpenAI GPTBot/1.4 Crawls content that may be used to train OpenAI foundation models.[1] GPTBot OpenAI publishes a GPTBot IP list.[4]
OpenAI OAI-SearchBot/1.4 Surfaces websites in ChatGPT Search.[1] OAI-SearchBot OpenAI publishes a SearchBot IP list.[1]
OpenAI ChatGPT-User/1.0 Makes user-initiated visits from ChatGPT.[1] ChatGPT-User OpenAI publishes a ChatGPT-User IP list.[1]
OpenAI OAI-AdsBot/1.0 Checks landing pages submitted for ChatGPT advertisements.[1] OAI-AdsBot OpenAI publishes an AdsBot IP list.[1]
Anthropic ClaudeBot Collects public web content that could contribute to model training.[5] ClaudeBot Anthropic says it does not publish crawler IP ranges because it uses service-provider public IPs.[5]
Anthropic Claude-SearchBot Crawls to improve search-result quality.[5] Claude-SearchBot No separate published range is identified in the cited documentation.
Anthropic Claude-User Supports user-initiated access.[5] Claude-User No separate published range is identified in the cited documentation.
Perplexity Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) Crawls and indexes web content.[6] PerplexityBot Perplexity publishes a PerplexityBot list.[6][7]
Perplexity Perplexity-User Supports user actions in Perplexity.[7] Perplexity-User Do not assume the PerplexityBot list applies to this agent.[8][7]
Google Existing Google crawler user agents Crawls pages used by Google services.[2] Google-Extended No separate Google-Extended IP range exists because it is not a separate HTTP request identity.[2]
Apple Applebot Crawls publicly available web content.[9] Applebot Apple publishes an Applebot CIDR JSON file.[3]
Apple Applebot-Extended Provides controls over how content is used, including model-training opt-out.[3] Applebot-Extended A separate Applebot-Extended IP list is not identified.[3]
Meta meta-externalagent/1.1 Crawls for foundation-model training and product indexing.[10] meta-externalagent No IP range is published in the cited documentation.[10]
Meta Meta-ExternalFetcher Fetches individual links at a user’s request, including agentic tasks.[10] Meta-ExternalFetcher No IP range is published in the cited documentation.[10]
ByteDance Bytespider Listed by Cloudflare as an AI crawler.[4] Bytespider No first-party IP list was located for this reference.[4]
Common Crawl CCBot/2.0 (+https://commoncrawl.org/faq/) Collects web data for Common Crawl’s open crawl repository.[11] CCBot Common Crawl publishes a CCBot IP JSON file.[11]
Amazon Amazonbot/0.1 Improves Amazon products and services and may support Amazon AI model training.[12][5] Amazonbot Amazon publishes Amazonbot IP addresses.[13][12]
DuckDuckGo DuckAssistBot/1.2 Fetches pages for AI-assisted DuckDuckGo answers that cite sources.[14] DuckAssistBot DuckDuckGo publishes addresses and a JSON file.[14]
xAI Grok Bot The cited documentation provides security information, but this reference does not confirm a complete user-agent string or purpose.[10] Not confirmed here Not confirmed here.

What does each major AI crawler do?

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot

OpenAI documents several distinct agents rather than one universal crawler. GPTBot is associated with crawling content that may be used to train OpenAI foundation models, while OAI-SearchBot supports ChatGPT Search.[1]

ChatGPT-User is used for user-initiated visits, so it should not automatically be treated as a background training crawler. OAI-AdsBot checks landing pages submitted as ChatGPT advertisements, and OpenAI states that data collected by that bot is not used to train foundation models.[1]

The corresponding robots.txt tokens are GPTBot, OAI-SearchBot, ChatGPT-User and OAI-AdsBot. OpenAI treats GPTBot and OAI-SearchBot controls independently, so blocking one does not automatically block the other.[1]

Anthropic: ClaudeBot, Claude-SearchBot and Claude-User

ClaudeBot collects public web content that could contribute to model training. Claude-SearchBot crawls to improve search-result quality, while Claude-User supports access initiated by a Claude user.[5]

Anthropic documents the tokens ClaudeBot, Claude-SearchBot and Claude-User. Its guidance includes a Crawl-delay: 1 example for ClaudeBot and recommends applying rules to every relevant subdomain.[5]

Anthropic says it does not publish crawler IP ranges because its crawler uses public IP addresses supplied by service providers.[5] That makes user-agent and reverse-DNS-only identification less conclusive than a published operator-owned range.

PerplexityBot and Perplexity-User

PerplexityBot is used for crawling and indexing, while Perplexity-User supports user actions in Perplexity.[7] Perplexity’s documented example begins as follows:

Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity says PerplexityBot follows robots.txt. If text access is disallowed, it may still index a domain, headline and brief factual summary.[6] The documented PerplexityBot IP list should not automatically be applied to Perplexity-User because the cited material does not confirm that both use the same ranges.[1][8]

Google-Extended

Google-Extended is commonly described as an AI crawler, but technically it is a robots.txt control token rather than a standalone HTTP user agent. Google uses its existing crawler user agents, and Google-Extended controls whether content crawled by Google may be used to train future Gemini models and for grounding.[2]

A rule such as User-agent: Google-Extended therefore does not identify a separate request source. It is a policy control layered onto Google’s established crawling infrastructure.[2]

Applebot and Applebot-Extended

Applebot crawls publicly available web content, and Apple says it follows standard robots.txt directives.[9] Apple publishes an Applebot CIDR JSON file for identifying Applebot traffic.[3]

Applebot-Extended is documented as a secondary user agent that gives publishers controls over content use, including opting out of training Apple’s foundation models.[3] Apple does not identify a separate Applebot-Extended IP list in the cited documentation.

Meta-ExternalAgent and Meta-ExternalFetcher

Meta-ExternalAgent, shown in Meta’s documentation with the prefix meta-externalagent/1.1, crawls for foundation-model training and product indexing.[10] Meta-ExternalFetcher fetches individual links at a user’s request, including for agentic tasks.[10]

Meta identifies meta-externalagent and Meta-ExternalFetcher as the relevant robots.txt tokens. The cited Meta documentation does not provide IP ranges for either crawler, so server operators should not treat a matching name alone as proof of origin.[8]

Bytespider

Cloudflare lists Bytespider as an AI crawler associated with ByteDance.[4] The research did not locate ByteDance first-party documentation confirming a fuller user-agent string, purpose, robots token behavior or IP list.[4][1]

A 2025 academic measurement study reported that Bytespider ignored robots.txt directives in the researchers’ tests.[4] That result means a Disallow rule should not be treated as a guaranteed technical block for this crawler.

CCBot

CCBot is Common Crawl’s crawler. Common Crawl describes its work as maintaining an open repository of web crawl data and gives CCBot as the robots.txt token for opting out.[11]

Common Crawl publishes a CCBot IP JSON file and recommends verifying requests using published ranges and reverse DNS because other crawlers may falsely identify themselves as CCBot.[11]

Amazonbot

Amazonbot helps Amazon improve its products and services, provide more accurate customer information and potentially train Amazon AI models.[12] Amazon documents a user-agent example containing Amazonbot/0.1 and says the bot honors allow and disallow directives in robots.txt.[13][12]

Amazon fetches the host-level robots.txt file or uses a cached copy from the previous 30 days. If it cannot fetch the file, Amazon says it behaves as though the file does not exist.[12] Amazon publishes a separate Amazonbot IP-address page.[13]

DuckAssistBot

DuckAssistBot crawls pages in real time for AI-assisted DuckDuckGo answers that cite their sources.[14] The documented user agent is:

DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

DuckDuckGo uses DuckAssistBot as the robots.txt token. A disallow takes effect after 72 hours and does not affect organic search rankings or inclusion in search results.[14]

How should robots.txt rules be written?

Robots.txt can express the operator’s requested access policy, but it is not an authentication mechanism and cannot force every crawler to comply. The academic study found that most large AI companies tested respected robots.txt, but some applications and crawlers did not; the study specifically reported noncompliance by Bytespider.[1][4]

Use separate rules when the purposes of two agents differ. For example, a site might allow a search crawler while disallowing a training crawler, subject to each operator’s documented behavior.

Example robots.txt file

  1. Open the site’s robots.txt file at the root of the host.
  2. Add the operator-specific tokens you intend to control.
  3. Use Disallow: / to request that the named agent access no paths.
  4. Use an empty Disallow: to allow access.
  5. Test the file and monitor server logs after publishing it.
# Allow ChatGPT Search but request no training crawl
User-agent: OAI-SearchBot
Disallow:

User-agent: GPTBot
Disallow: /

# Request that Claude's training crawler does not fetch the site
User-agent: ClaudeBot
Disallow: /

# Request that PerplexityBot does not fetch page text
User-agent: PerplexityBot
Disallow: /

# Control Google's Gemini training and grounding use
User-agent: Google-Extended
Disallow: /

# Request no Apple model-training use
User-agent: Applebot-Extended
Disallow: /

# Request no Meta foundation-model or product-indexing crawl
User-agent: meta-externalagent
Disallow: /

# Opt out of Common Crawl
User-agent: CCBot
Disallow: /

# Request no Amazonbot access
User-agent: Amazonbot
Disallow: /

# Request no DuckDuckGo AI answer crawl
User-agent: DuckAssistBot
Disallow: /

This example expresses separate requests for each documented token; it does not prove that every request will stop. Operators should also review each vendor’s current policy because crawler names, purposes and enforcement practices can change.[8]

How can you verify an AI crawler’s identity?

A user-agent string is self-declared and can be spoofed. Common Crawl explicitly warns that other crawlers may falsely identify themselves as CCBot and recommends checking reverse DNS and published IP ranges.[11]

A practical verification process is:

  1. Record the complete request user-agent, source IP, timestamp, requested URL and response status.
  2. Compare the source IP with the operator’s current published JSON or CIDR list.
  3. Perform reverse-DNS verification where the operator documents an official hostname pattern.
  4. Forward-confirm the resulting hostname back to the same IP address.
  5. Check request behavior, including rate, path selection, robots.txt retrieval and repeated visits.
  6. Treat traffic as unverified if the user agent matches but the network origin does not.

This process is strongest for operators that publish IP ranges, including OpenAI, Perplexity, Apple, Common Crawl, Amazon and DuckDuckGo. It is less conclusive for Anthropic, which says it uses service-provider public IPs rather than publishing crawler ranges.[5][8]

What is the difference between blocking a user agent and blocking an IP range?

Blocking a user agent uses a request’s self-declared User-Agent header and is easy to configure in robots.txt or a web server. Blocking an IP range operates at the network layer and can be stronger when the range is current and belongs to the claimed operator, but it can also block legitimate traffic if ranges change or are shared. Neither method alone is a universal guarantee against spoofing or distributed crawling.[5]

What are the main limitations of AI crawler lists?

AI crawler references become outdated quickly because operators can add agents, change user-agent versions, move infrastructure or introduce new access modes. A list should therefore show the date checked and link to the operator’s current documentation rather than treating a name as permanent.

The most important limitations are:

  • A robots token may not correspond to a separate HTTP crawler, as with Google-Extended.[2]
  • User-initiated agents can have a different purpose from automated crawlers, as with ChatGPT-User, Claude-User and Perplexity-User.
  • A robots.txt rule is a policy signal, not a technical access barrier.[8]
  • A user-agent string can be forged.
  • Some operators publish IP ranges, while others do not.[8][5]
  • A crawler’s stated purpose may include several uses, such as search, indexing, product improvement or model training.[10][1]
  • A block can affect discoverability or AI-generated answers differently depending on the operator’s product behavior.[14]

For ongoing monitoring, combine robots.txt review with access logs, CDN bot controls, published IP lists and alerts for new user-agent strings. Allowing a crawler also doesn’t guarantee it can read your pages: Finseo’s own audit of AI crawler readability found that most AI crawlers never run JavaScript, and across 20,293 audited pages on 524 sites, 8.9% returned nothing readable.

Frequently asked questions

What is an AI crawler user agent?

An AI crawler user agent is the identifier an automated AI-related web client sends in an HTTP request. It can indicate that the client is fetching content for model training, search, indexing, product improvement, advertising checks or a user-requested action.[10][2]

What is the GPTBot user agent?

GPTBot is OpenAI’s documented crawler for content that may be used to train OpenAI foundation models.[1] A documented example uses `GPTBot/1.4`, and the corresponding robots.txt token is `GPTBot`.[1]

What is the ClaudeBot user agent?

ClaudeBot is Anthropic’s crawler for collecting public web content that could contribute to model training.[5] The robots.txt token is `ClaudeBot`; Anthropic also documents separate Claude-SearchBot and Claude-User agents for search and user-initiated access.[5]

Is Google-Extended a crawler?

Google-Extended is not a separate HTTP crawler user agent. It is a robots.txt control token that governs how content crawled by Google may be used for Gemini model training and grounding.[2]

Does robots.txt block AI crawlers?

Robots.txt communicates an access preference to compliant crawlers, but it is not a guaranteed technical block. Academic testing found that most large AI companies studied respected robots.txt, while some crawlers, including Bytespider in that study, did not.[10][4]

How can a website verify an AI crawler?

Compare the complete user-agent string with the operator’s documentation, then verify the source IP against a current published range where available. Reverse DNS and forward-confirmation checks add evidence, but a matching user-agent alone is not sufficient because crawlers can impersonate one another.[3][8]

Sources

  1. [1]Overview of OpenAI Crawlers — developers.openai.com
  2. [2]Google's common crawlers | Google Crawling Infrastructure  |  Crawling infrastructure  |  Google for Developers — developers.google.com
  3. [3]About Applebot - Apple Support — support.apple.com
  4. [4]Bot reference — developers.cloudflare.com
  5. [5]Does Anthropic crawl data from the web, and how can site owners block the crawler? | Claude Help Center — anthropic.com
  6. [6]How does Perplexity follow robots.txt? — perplexity.ai
  7. [7]Perplexity Crawlers — docs.perplexity.ai
  8. [8]Grok Bot security | SpaceXSI Docs — docs.x.ai
  9. [9]Applebot model training and individual privacy rights - Apple Support — support.apple.com
  10. [10]Meta Web Crawlers | Meta for Developers — developers.facebook.com
  11. [11]Common Crawl - CCBot — commoncrawl.org
  12. [12]About AmazonBot — developer.amazon.com
  13. [13]Amazon Web Search crawlers | Amazonbot — developer.amazon.com
  14. [14]Is DuckAssistBot related to DuckDuckGo? - DuckDuckGo Help Pages — duckduckgo.com

Related topics

Reference & analysisReference

Get new field reports by email

One email when a report is published or re-checked. No tracking pixels.

More from Bot Trackers