ClaudeBot is Anthropic's web crawler. It visits public web pages and collects content that may be used to train Claude, Anthropic's family of AI models. You can see it in your server logs under the user-agent token ClaudeBot, and you control it with robots.txt, like any well-behaved crawler.
That is the short answer. The longer one matters because ClaudeBot is only one of three Anthropic bots, each with a different job, and blocking the wrong one can cost you visibility in Claude's answers without protecting anything. This guide covers what each bot does, how to recognise them, whether to block ClaudeBot, and the robots.txt mistakes that make a block fail silently.
What does ClaudeBot do?
ClaudeBot collects training data. Anthropic says so directly in its system cards, the technical documents it publishes with each model. The Claude Opus 5.5 system card, dated September 22, 2026, describes it as "a general-purpose web crawler called ClaudeBot" used "to obtain training data from public websites."
The same passage sets out three limits that matter before you decide anything:
- It follows robots.txt. The card says the crawler adheres to industry-standard practice for the robots.txt instructions site operators publish.
- It stays out of gated pages. Anthropic states that it does not access password-protected pages, pages that need a sign-in, or pages behind CAPTCHA verification.
- It identifies itself. Site operators can tell when it has crawled their pages and signal their preferences.
In practice, ClaudeBot reads what any anonymous visitor can read. Your public blog posts, product pages, help articles and documentation are all in scope. Your customer dashboard, a members-only course, or an invoice behind a login are not.
What ClaudeBot does not do is answer questions in real time. When someone asks Claude about your pricing today, ClaudeBot is not the bot that fetches your pricing page. That job belongs to a different user agent, which is why the next section matters more than most guides admit.
ClaudeBot vs Claude-User vs Claude-SearchBot
Anthropic runs three bots, and its crawler help article (last updated April 7, 2026) describes each one along with what happens when you block it.
| Bot | What it does | What blocking it means |
|---|---|---|
| ClaudeBot | Collects public web content that could contribute to training Anthropic's models | Your future content is excluded from training datasets |
| Claude-SearchBot | Indexes content to improve the relevance and accuracy of Claude's search results | Your pages are not indexed for Claude's search, which may reduce your visibility in its answers |
| Claude-User | Fetches a page when a person asks Claude a question that needs it | Claude cannot retrieve your page for that person |
The split works like OpenAI's, which the llms.txt guide touches on: one bot for training, one for search, one for user-triggered fetches. Each obeys its own robots.txt group, so the decision is never "block Anthropic or allow Anthropic." It is three separate decisions.
Two older names still circulate in robots.txt generators and bot directories: anthropic-ai and Claude-Web. Neither appears on Anthropic's current help page, which lists only the three bots above. Keeping them in your file does no harm, but a file that blocks only those two tokens does not block ClaudeBot.
The categories also show up in infrastructure tools. Cloudflare's Radar directory lists Claude-SearchBot as a verified bot in its Search category, and Claude-User in its Agent category, as a bot whose activity end users can direct. That classification decides how CDN-level AI blocking treats each one, which we come back to below.
Is ClaudeBot the same as Clawdbot?
No. A large share of people searching "claudebot" are looking for something else entirely: Clawdbot, an open-source personal AI assistant that went viral in early 2026.
Clawdbot has nothing to do with Anthropic's crawler. Its creator, Peter Steinberger, explains in the announcement of its current name that "Clawd" began in November 2025 as a pun on Claude, until Anthropic's legal team asked him to reconsider. The project became Moltbot, then on January 29, 2026, OpenClaw. It runs on your own machine and works through chat apps such as WhatsApp, Telegram, Discord and Slack.
So if you saw "Clawdbot" in a YouTube video about Mac minis running an assistant around the clock, that is OpenClaw. If you saw "ClaudeBot" in your server logs, that is Anthropic's training crawler. The rest of this guide is about the crawler.
How do you spot ClaudeBot in your logs?
Search your access logs for the token ClaudeBot. Robots.txt matching is case-insensitive under the standard, so match it case-insensitively in your logs too. On a typical Nginx or Apache server, one command gives you a daily count:
grep -i "claudebot" /var/log/nginx/access.log | awk '{print $4}' | cut -d: -f1 | sort | uniq -c
Swap the token for claude-searchbot or claude-user to see the other two. Claude-User announces itself with a string like this one, recorded in Cloudflare's bot directory:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)
Check that it really is Anthropic
A user-agent string is just text, and anyone can send it. In August 2025, Cloudflare pointed out that Anthropic "still lags in verification, which makes it easier for bad actors to spoof its crawler and ignore robots.txt." A scraper pretending to be ClaudeBot will not care what your robots.txt says, and if you judge Anthropic by its traffic you will be judging the impostor.
Anthropic now offers a way to check. Its help article links to a published list of the IP ranges its crawlers use, and says a crawler whose source IP is on that list is coming from Anthropic. The list lives here, as plain JSON:
https://claude.com/crawling/bots.json
When we fetched it in October 2026, the file was dated October 7, 2026, and held a few dozen IPv4 prefixes, the largest being 216.73.216.0/22. It changes, so check against the live file rather than a copied list.
A worked example
Say you run a UK recipe site and your hosting dashboard shows a spike: 40,000 requests in a night from user agents containing "ClaudeBot." Check things in this order:
- Pull the source IPs for those requests from the access log.
- Compare them with bots.json. If most of them fall inside Anthropic's ranges, the traffic is real.
- If they do not match, you are dealing with a spoofer. Robots.txt will not help. Rate-limit or block those IPs at the firewall or CDN.
- If they do match, decide what you want (the next section) and say it in robots.txt.
That last step is what most people skip. Blocking Anthropic's real IP ranges at the firewall feels decisive, but Anthropic warns that IP blocks "may not work correctly or persistently guarantee an opt-out," because they stop the crawler from reading the robots.txt file that records your opt-out.
Should you block ClaudeBot?
That depends on what you sell. Blocking ClaudeBot removes your future content from Anthropic's training data. It does not remove you from Claude's search results, because those come from Claude-SearchBot and Claude-User. That separation makes the decision cleaner than it looks.
The case for blocking is mostly economic. Cloudflare measures a "crawl-to-refer ratio," which it defines as how many pages a platform crawls compared with how often it sends users back to a website. In its analysis of AI crawler traffic published in August 2025, Anthropic had the highest ratio of any major AI company.
The trend is real: the ratio fell from about 286,930 pages crawled per referred visit in January 2025 to about 38,066 in July, after Anthropic added web search to Claude in March 2025. Even so, 38,000 pages for one visit is a long way from OpenAI's 1,091 in the same month. Cloudflare also found ClaudeBot's share of AI and search crawler traffic grew from 6% to nearly 10% between July 2024 and July 2025. ClaudeBot takes a lot and, measured by clicks, gives back little.
The case for allowing it is about the future. A model trained on your explanations, product names and comparisons is more likely to describe your category the way you do. That benefit is hard to measure, and nobody can promise it.
Our view, by type of site:
| Your site | ClaudeBot (training) | Claude-SearchBot and Claude-User |
|---|---|---|
| News publisher or paid content business | Block, unless you have a licensing deal | Allow, if you want citations in Claude's answers |
| SaaS, D2C brand or local service business | Allow: your content is marketing | Allow |
| Documentation or developer tools | Allow | Allow |
| A small server struggling with load | Slow it with Crawl-delay before blocking | Allow |
The middle row covers most businesses reading this, from a Pune SaaS startup to a Manchester plumber. Your content exists to be found and repeated. Opting out of training protects very little for you, while blocking the search and user bots would cut you off from the citations that send visitors back. If AI answers are part of your plan, our guides on optimizing content for ChatGPT answers and on AEO vs SEO cover what earns citations once the bots can reach you.
Publishers face a different trade. Their pages are the product, and a 38,000-to-one exchange rate is a poor deal. Blocking ClaudeBot while allowing Claude-SearchBot and Claude-User keeps the referral door open and the training door shut.
Blocking ClaudeBot has no effect on Google. Googlebot reads its own robots.txt group and ignores the ClaudeBot one.
How to block or allow ClaudeBot in robots.txt
Robots.txt sits at the root of each host, for example https://example.co.uk/robots.txt. Anthropic's help article gives the format for a full block:
User-agent: ClaudeBot
Disallow: /
The most common setup we would recommend for a publisher blocks training and keeps search:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
To allow ClaudeBot but keep it out of some folders, list them:
User-agent: ClaudeBot
Disallow: /cart/
Disallow: /search/
Disallow: /account/
And to slow it down rather than block it, Anthropic supports the non-standard Crawl-delay directive, which it documents with a value of 1:
User-agent: ClaudeBot
Crawl-delay: 1
The trap: a ClaudeBot group ignores your * rules
This is the mistake that catches careful people. Under RFC 9309, the standard for robots.txt, a crawler looks for the group that names it and obeys that group. It falls back to the User-agent: * group only "if no matching group exists."
Picture an Indian D2C store whose robots.txt has always kept every bot out of /checkout/ and /admin/ with a wildcard group. The owner then adds a group to slow ClaudeBot down:
User-agent: *
Disallow: /checkout/
Disallow: /admin/
User-agent: ClaudeBot
Crawl-delay: 1
ClaudeBot now matches its own group, and that group has no Disallow lines. The checkout and admin rules no longer apply to it. The fix is to repeat every rule you still want inside the ClaudeBot group:
User-agent: ClaudeBot
Crawl-delay: 1
Disallow: /checkout/
Disallow: /admin/
Run the same check on every named group in your file, not only Anthropic's. Our technical SEO audit checklist is a good place to make it a recurring item.
Every subdomain needs its own file
Anthropic asks site owners to add the rules "for every subdomain that you wish to opt out from." A block on www.example.com/robots.txt says nothing about blog.example.com or shop.example.com. If your blog runs on a separate host, as many WordPress and Shopify setups do, it needs its own file.
Why ClaudeBot might ignore your rules (or never reach your site)
When your logs disagree with your robots.txt, the cause is usually one of these, roughly in order of how often we would expect to see them.
The rules have not been re-read yet. RFC 9309 lets crawlers cache robots.txt, and says they should not use a cached copy for more than 24 hours. Give a change a day before you decide it failed.
The traffic is not Anthropic. Check the IPs against bots.json, as above. A spoofer ignores robots.txt by design.
Your robots.txt returns a server error. This one cuts the other way. Under RFC 9309, if robots.txt is unreachable because of a 5xx error, a compliant crawler "MUST assume complete disallow." A misconfigured server can block ClaudeBot and Claude-SearchBot from your whole site without you writing a single rule. Open the file in a browser and confirm it returns a 200.
Your CDN is blocking AI bots for you. Cloudflare lets sites block AI bots by behaviour, under Security Settings > Configure AI bot policies, with separate presets for Search, Agent and Training. Since September 15, 2026, new domains that run ads are offered a preset that keeps Search allowed, sets Training to "Disallow AI Training" and blocks Agent bots on pages with ads; our breakdown of Cloudflare's AI crawler defaults covers what shipped and how it changed from the July announcement. If your site runs ads and sits behind Cloudflare, Claude-User could be blocked on those pages without a line in your robots.txt. Check the dashboard before you debug the file.
You blocked IPs instead of writing rules. As covered above, Anthropic's crawlers need to read robots.txt to record an opt-out. A firewall block can stop that from happening.
One more limit applies to every crawler, not only Anthropic's. The standard itself warns that robots.txt "is not a substitute for valid content security measures." If a page must stay private, put it behind a login. ClaudeBot does not access sign-in pages, and a login stops the crawlers that ignore robots.txt as well.
What to do if ClaudeBot is hammering your server
Heavy crawling is a real complaint, not a theoretical one. In July 2024, The Verge reported that ClaudeBot hit iFixit's website almost a million times in 24 hours. iFixit's terms of use already banned using its content to train AI models, which did nothing to slow the crawler. What the company did next was technical: its CEO said it added the crawl-delay extension to its robots.txt.
The lesson for a smaller site is that terms of service do not talk to crawlers. Robots.txt does. If ClaudeBot is straining your server:
- Confirm the traffic is Anthropic's using the IP list.
- Add
Crawl-delay: 1to the ClaudeBot group, or a higher number, and copy in any wildcard rules you still need. - Disallow the expensive paths: internal search results, faceted filter URLs, calendar pages that generate endless dates. These are often where crawlers burn the most requests.
- Wait at least a day for the new file to be read.
- If it persists, email
claudebot@anthropic.comfrom an address on the affected domain. Anthropic asks for this because reports from other addresses are hard to verify.
Fixing crawl waste helps with every bot at once. The same faceted URLs that soak up ClaudeBot's requests also eat Googlebot's attention, and a site full of them often ends up with pages stuck as crawled but not indexed.
Frequently asked questions
Is ClaudeBot a virus or something harmful?
No. ClaudeBot is a legitimate crawler operated by Anthropic, the company behind Claude. It reads public pages to collect training data and follows robots.txt. Traffic that claims to be ClaudeBot but comes from IPs outside Anthropic's published list is a different matter, and should be treated like any other unknown scraper.
Does ClaudeBot respect robots.txt?
Yes. Anthropic's system cards state that ClaudeBot adheres to robots.txt instructions, and its help article says robots.txt is the way to opt out. It also supports the non-standard Crawl-delay directive. Rules apply per bot, so a ClaudeBot block does not cover Claude-SearchBot or Claude-User.
Will blocking ClaudeBot remove my site from Claude's answers?
No. ClaudeBot only covers training. Claude's search results rely on Claude-SearchBot, and live page fetches for a user's question rely on Claude-User. Block those two and your visibility in Claude's answers will likely drop. Block only ClaudeBot and your future content is simply left out of training data.
Does blocking ClaudeBot remove my content from models already trained?
Anthropic describes the block as forward-looking: restricting ClaudeBot signals that your site's "future materials" should be excluded from training datasets. Its help article does not promise removal from models that have already been trained, so treat a block as applying from now on.
Does blocking ClaudeBot affect my Google rankings?
No. Googlebot follows its own robots.txt group, so rules written for ClaudeBot do not change how Google crawls or ranks your pages. The only way a ClaudeBot rule touches Google is by mistake, for example a typo that turns it into a wildcard rule, so check the file after every edit.
What happened to anthropic-ai and Claude-Web?
They are older user-agent tokens that still appear in many robots.txt templates. Anthropic's current crawler page lists only ClaudeBot, Claude-SearchBot and Claude-User. Leaving the old tokens in your file is harmless, but they are not a substitute for a ClaudeBot rule.
