AI visibility performs an important position for SEOs, and this begins with controlling AI crawlers. If AI crawlers can’t entry your pages, you’re invisible to AI discovery engines.
On the flip aspect, unmonitored AI crawlers can overwhelm servers with extreme requests, inflicting crashes and sudden internet hosting payments.
Person-agent strings are important for controlling which AI crawlers can entry your web site, however official documentation is typically outdated, incomplete, or lacking completely. So, we curated a verified checklist of AI crawlers from our precise server logs as a helpful reference.
Each user-agent is validated in opposition to official IP lists when accessible, making certain accuracy. We are going to keep and replace this checklist to catch new crawlers and adjustments to current ones.
The Full Verified AI Crawler Record (December 2025)
| Identify | Goal | Crawl Charge of SEJ (pages/hour) | Verified IP Record | Robots.txt disallow | Full Person Agent |
|---|---|---|---|---|---|
| GPTBot | AI coaching information assortment for GPT fashions (ChatGPT, GPT-4o) | 100 | Official IP List | Person-agent: GPTBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; GPTBot/1.3; +https://openai.com/gptbot) |
| ChatGPT-User | AI agent for real-time net looking when customers work together with ChatGPT | 2400 | Official IP List | Person-agent: ChatGPT-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); suitable; ChatGPT-Person/1.0; +https://openai.com/bot |
| OAI-SearchBot | AI search indexing for ChatGPT search options (not for coaching) | 150 | Official IP List | Person-agent: OAI-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; suitable; OAI-SearchBot/1.3; +https://openai.com/searchbot |
| ClaudeBot | AI coaching information assortment for Claude fashions | 500 | Official IP List | Person-agent: ClaudeBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; [email protected]) |
| Claude-User | AI agent for real-time net entry when Claude customers browse | <10 | Not accessible | Person-agent: Claude-Person Disallow: /sample-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-Person/1.0; [email protected]) |
| Claude-SearchBot | AI search indexing for Claude search capabilities | <10 | Not accessible | Person-agent: Claude-SearchBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Claude-SearchBot/1.0; +https://www.anthropic.com) |
| Google-CloudVertexBot | AI agent for Vertex AI Agent Builder (web site homeowners’ request solely) | <10 | Official IP List | Person-agent: Google-CloudVertexBot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Construct/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.7390.122 Cell Safari/537.36 (suitable; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) |
| Google-Extended | Token controlling AI coaching utilization of Googlebot-crawled content material. | Person-agent: Google-Prolonged Permit: / Disallow: /private-folder |
|||
| Gemini-Deep-Research | AI analysis agent for Google Gemini’s Deep Analysis characteristic | <10 | Official IP List | Person-agent: Gemini-Deep-Analysis Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Gemini-Deep-Analysis; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36 |
| Gemini’s chat when a consumer asks to open a webpage | <10 | ||||
| Bingbot | Powers Bing Search and Bing Chat (Copilot) AI solutions | 1300 | Official IP List | Person-agent: BingBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 |
| Applebot-Extended | Doesn’t crawl however controls how Apple makes use of Applebot information. | <10 | Official IP List | Person-agent: Applebot-Prolonged Permit: / Disallow: /private-folder |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Model/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot) |
| PerplexityBot | AI search indexing for Perplexity’s reply engine | 150 | Official IP List | Person-agent: PerplexityBot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
| Perplexity-User | AI agent for real-time looking when Perplexity customers request information | <10 | Official IP List | Person-agent: Perplexity-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Perplexity-Person/1.0; +https://perplexity.ai/perplexity-user) |
| Meta-ExternalAgent | AI coaching information assortment for Meta’s LLMs (Llama, and so on.) | 1100 | Not accessible | Person-agent: meta-externalagent Permit: / Disallow: /private-folder |
meta-externalagent/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) |
| Meta-WebIndexer | Used to enhance Meta AI search. | <10 | Not accessible | Person-agent: Meta-WebIndexer Permit: / Disallow: /private-folder |
meta-webindexer/1.1 (+https://builders.fb.com/docs/sharing/site owners/crawler) |
| Bytespider | AI coaching information for ByteDance’s LLMs for merchandise like TikTok | <10 | Not accessible | Person-agent: Bytespider Permit: / Disallow: /private-folder |
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Cell Safari/537.36 (suitable; Bytespider; https://zhanzhang.toutiao.com/) |
| Amazonbot | AI coaching for Alexa and different Amazon AI companies | 1050 | Not accessible | Person-agent: Amazonbot Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; Amazonbot/0.1; +https://developer.amazon.com/assist/amazonbot) Chrome/119.0.6045.214 Safari/537.36 |
| DuckAssistBot | AI search indexing for DuckDuckGo search engine | 20 | Official IP List | Person-agent: DuckAssistBot Permit: / Disallow: /private-folder |
DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) |
| MistralAI-Person | Mistral’s real-time quotation fetcher for “Le Chat” assistant | <10 | Not accessible | Person-agent: MistralAI-Person Permit: / Disallow: /private-folder |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; MistralAI-Person/1.0; +https://docs.mistral.ai/robots) |
| Webz.io | Knowledge extraction and net scraping utilized by different AI coaching corporations. Previously referred to as Omgili. | <10 | Not accessible | Person-agent: webzio Permit: / Disallow: /private-folder |
webzio (+https://webz.io/bot.html) |
| Diffbot | Knowledge extraction and net scraping utilized by corporations throughout the world. | <10 | Not accessible | Person-agent: Diffbot Permit: / Disallow: /private-folder |
Mozilla/5.0 (Home windows; U; Home windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) |
| ICC-Crawler | AI and machine studying information assortment | <10 | Not accessible | Person-agent: ICC-Crawler Permit: / Disallow: /private-folder |
ICC-Crawler/3.0 (Mozilla-compatible; ; https://ucri.nict.go.jp/en/icccrawler.html) |
| CCBot | Open-source net archive used as coaching information by a number of AI corporations | <10 | Official IP List | Person-agent: CCBot Permit: / Disallow: /private-folder |
CCBot/2.0 (https://commoncrawl.org/faq/) |
The user-agent strings above have all been verified in opposition to Search Engine Journal server logs.
Well-liked AI Agent Crawlers With Unidentifiable Person Agent
We’ve discovered that the following didn’t determine themselves:
- you.com.
- ChatGPT’s agent Operator.
- Bing’s Copilot chat.
- Grok.
- DeepSeek.
There is no method to observe this crawler from accessing webpages aside from by figuring out the express IP.
We arrange a lure web page (e.g., /specific-page-for-you-com/) and used the on-page chat to immediate you.com to go to it, permitting us to find the corresponding go to file and IP tackle in our server logs. Beneath is the screenshot:
What About Agentic AI Browsers?
Sadly, AI browsers resembling Comet or ChatGPT’s Atlas don’t differentiate themselves in the consumer agent string, and you may’t determine them in server logs and mix with regular customers’ visits.
This is disappointing for SEOs as a result of monitoring agentic browser visits to an internet site is vital for reporting POV.
How To Examine What’s Crawling Your Server
Some internet hosting corporations provide a consumer interface (UI) that makes it simple to entry and take a look at server logs, relying on what hosting service you are utilizing.
In case your internet hosting doesn’t provide this, you may get server log information (often situated /var/log/apache2/entry.log in Linux-based servers) through FTP or request it from your server assist to ship it to you.
After getting the log file, you may view and analyze it in both Google Sheets (if the file is in CSV format), Screaming Frog’s log analyzer, or, in case your log file is less than 100 MB, you may attempt analyzing it with Gemini AI.
How To Confirm Respectable Vs. Pretend Bots
Pretend crawlers can spoof legit consumer brokers to bypass restrictions and scrape content material aggressively. For instance, anybody can impersonate ClaudeBot from their laptop computer and provoke crawl request from the terminal. In your server log, you will note it as Claudebot is crawling it:
curl -A 'Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; suitable; ClaudeBot/1.0; [email protected])' https://instance.comVerification may also help to save server bandwidth and stop harvesting content material illegally. Essentially the most dependable verification methodology you may apply is checking the request IP.
Examine all IPs and scan to match if it’s considered one of the formally declared IPs listed above. In that case, you may enable the request; in any other case, block.
Varied kinds of firewalls can help you with this through allowlist verified IPs (which permits legit bot requests to move by way of), and all different requests impersonating AI crawlers of their consumer agent strings are blocked.
For instance, in WordPress, you should use Wordfence free plugin to allowlist legit IPs from the official lists (as above) and add blocking customized guidelines as beneath:
The allowlist rule is superior, and it’ll let legit crawlers move by way of and block any impersonation request which comes from totally different IPs.
Nevertheless, please word that it is doable to spoof an IP address, and in that case, when bot consumer agent and IPs are spoofed, you received’t find a way to block it.
Conclusion: Keep In Management Of AI Crawlers For Dependable AI Visibility
AI crawlers are now a part of our net ecosystem, and the bots listed right here characterize the main AI platforms at the moment indexing the net, though this checklist is seemingly to develop.
Examine your server logs repeatedly to see what’s really hitting your web site and be sure you inadvertently don’t block AI crawlers if visibility in AI search engines is vital for your online business. If you happen to don’t need AI crawlers to entry your content material, block them through robots.txt utilizing the user-agent title.
We’ll maintain this checklist up to date as new crawlers emerge and replace current ones, so we advocate you bookmark this URL, or revisit this article on an everyday foundation to maintain your AI crawler checklist up to date.
Extra Assets:
Featured Picture: BestForBest/Shutterstock
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
