Cloudflare revealed its sixth annual Year in Review, providing a complete appears to be like at Web site visitors, safety, and AI crawler exercise throughout 2025.
The report attracts on information from Cloudflare’s community, which spans greater than 330 cities throughout 125 international locations and handles over 81 million HTTP requests per second on common.
The AI crawler findings stand out. Googlebot crawled way more net pages than some other AI bot, reflecting Google’s dual-purpose method to crawling for each search indexing and AI coaching.
Googlebot Prime AI Crawler Site visitors
Cloudflare analyzed profitable requests for HTML content material from main AI crawlers throughout October and November 2025. The outcomes confirmed Googlebot reached 11.6% of distinctive net pages in the pattern.
That’s greater than 3 occasions the pages seen by OpenAI’s GPTBot at 3.6%. It’s practically 200 occasions greater than PerplexityBot, which crawled simply 0.06% of pages.
Bingbot got here in third at 2.6%, adopted by Meta-ExternalAgent and ClaudeBot at 2.4% every.
The report famous that as a result of Googlebot crawls for each search indexing and AI mannequin coaching, net publishers face a troublesome selection. Blocking Googlebot’s AI coaching means risking search discoverability.
Cloudflare wrote:
“As a result of Googlebot is used to crawl content material for each search indexing and AI mannequin coaching, and due to Google’s long-established dominance in search, Website online operators are primarily unable to block Googlebot’s AI coaching with out risking search discoverability.”
AI Bots Now Account For 4.2% of HTML Requests
All through 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests throughout Cloudflare’s buyer base. The share fluctuated between 2.4% in early April and 6.4% in late June.
Googlebot alone accounted for 4.5% of HTML requests, barely greater than all different AI bots mixed.
The share of human-generated HTML site visitors began 2025 at seven proportion factors beneath non-AI bot site visitors. By September, human site visitors started exceeding non-AI bot site visitors on some days. As of December 2, people generated 47% of HTML requests whereas non-AI bots generated 44%.
Crawl-to-Refer Ratios Present Large Variation
Cloudflare tracks how typically AI and search platforms ship site visitors to websites relative to how typically they crawl. A excessive ratio means heavy crawling with out sending customers again to supply websites.
Anthropic had the highest ratios amongst AI platforms, ranging from roughly 25,000:1 to 100,000:1 throughout the second half of the 12 months after stabilizing from earlier volatility.
OpenAI’s ratios reached as excessive as 3,700:1 in March. Perplexity maintained the lowest ratios amongst main AI platforms, usually beneath 400:1 and underneath 200:1 from September onward.
For comparability, Google’s search crawl-to-refer ratio stayed a lot decrease, usually between 3:1 and 30:1 all through the 12 months.
Consumer-Motion Crawling Grew Over 20X
Not all AI crawling is for mannequin coaching. “Consumer motion” crawling happens when bots go to websites in response to person questions posed to chatbots.
This class noticed the quickest progress in 2025. Consumer-action crawling quantity elevated greater than 15 occasions from January by way of early December. The pattern carefully matched the site visitors sample for OpenAI’s ChatGPT-Consumer bot, which visits pages when customers ask ChatGPT questions.
The expansion confirmed a weekly utilization sample beginning in mid-February, suggesting elevated use in colleges and workplaces. Exercise dropped throughout June by way of August when college students have been on break and professionals took holidays.
AI Crawlers Most Blocked In Robots.txt
Cloudflare analyzed robots.txt recordsdata throughout practically 3,900 of the high 10,000 domains. AI crawlers have been the most steadily blocked person brokers.
GPTBot, ClaudeBot, and CCBot had the highest variety of full disallow directives. These directives inform crawlers to keep away from whole websites.
Googlebot and Bingbot confirmed a distinct sample. Their disallow directives leaned closely towards partial blocks, possible centered on login endpoints and non-content areas fairly than full website blocking.
Civil Society Grew to become Most-Attacked Sector
For the first time, organizations in the “Folks and Society” vertical have been the most focused by assaults. This class consists of non secular establishments, nonprofits, civic organizations, and libraries.
The sector acquired 4.4% of world mitigated site visitors, up from underneath 2% at the begin of the 12 months. Assault share jumped to over 17% in late March and peaked at 23.2% in early July.
Many of those organizations are protected by Cloudflare’s Mission Galileo.
Playing and video games, the most-attacked vertical in 2024, noticed its share drop by greater than half to 2.6%.
Different Key Findings
Cloudflare’s report included a number of extra findings throughout site visitors, safety, and connectivity.
International Web site visitors grew 19% year-over-year. Progress stayed comparatively flat by way of mid-April, then accelerated after mid-August.
Submit-quantum encryption now secures 52% of human site visitors to Cloudflare, practically double the 29% share at the begin of the 12 months.
ChatGPT remained the high generative AI service globally. Google Gemini, Windsurf AI, Grok/xAI, and DeepSeek have been new entrants to the high 10.
Starlink site visitors doubled in 2025, with service launching in additional than 20 new international locations.
Practically half of the 174 main Web outages noticed globally have been brought on by government-directed shutdowns. Cable lower outages dropped practically 50%, whereas energy failure outages doubled.
European international locations dominated Web high quality metrics. Spain topped the record for total Web high quality, with common obtain speeds above 300 Mbps.
Why This Issues
The AI crawler information ought to impacts how you consider bot entry and site visitors.
Google’s dual-purpose crawler creates a aggressive benefit. You possibly can block different AI crawlers whereas retaining Googlebot entry for search visibility, however you possibly can’t separate Google’s search crawling from its AI coaching crawling.
The crawl-to-refer ratios assist quantify what publishers already suspected. AI platforms crawl closely however ship little site visitors again. The hole between crawling and referring varies broadly by platform.
The civil society assault information issues for those who work with nonprofits or advocacy organizations. These teams now face the highest charge of assaults.
Wanting Forward
Cloudflare expects AI metrics to change as the area continues to evolve. The corporate added a number of new AI-related datasets to this 12 months’s report that weren’t out there in earlier editions.
The crawl-to-refer ratios might change as AI platforms regulate their search options and referral conduct. OpenAI’s ratios already confirmed some decline by way of the 12 months as ChatGPT search utilization grew.
For robots.txt administration, the information exhibits most publishers are selecting partial blocks for main search crawlers whereas totally blocking AI-only crawlers. The year-end state of those directives supplies a baseline for monitoring how writer insurance policies evolve in 2026.
Featured Picture: Mamun_Sheikh/Shutterstock
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
