Cloudflare launched a function that converts HTML pages to markdown when AI programs request it. Websites on its community can now serve lighter content material to bots with out constructing separate pages.
The function, referred to as Markdown for Agents, works by means of HTTP content material negotiation. An AI crawler sends a request with Settle for: textual content/markdown in the header. Cloudflare intercepts it, fetches the authentic HTML from the origin server, converts it to markdown, and delivers the consequence.
The launch arrives days after Google’s John Mueller called the idea of serving markdown to AI bots “a stupid idea” and questioned whether or not bots may even parse markdown hyperlinks correctly.
What’s New
Cloudflare described the function as treating AI brokers as “first-class residents” alongside human guests. The corporate used its personal weblog put up for instance. The HTML model consumed 16,180 tokens whereas the markdown conversion used 3,150 tokens.
“Feeding uncooked HTML to an AI is like paying by the phrase to learn packaging as an alternative of the letter inside,” the firm wrote.
The conversion occurs at Cloudflare’s edge community, not at the origin server. Web sites allow it per zone by means of the dashboard, and it’s obtainable in beta at no further value for Professional, Enterprise, and Enterprise plan clients, plus SSL for SaaS clients.
Cloudflare famous that some AI coding instruments already ship the Settle for: textual content/markdown header. The corporate named Claude Code and OpenCode as examples.
Every transformed response consists of an x-markdown-tokens header that estimates the token rely of the markdown model. Builders can use this to handle context home windows or plan chunking methods.
Content material-Sign Defaults
Transformed responses embrace a Content material-Sign header set to ai-train=sure, search=sure, ai-input=sure by default, signaling the content material can be utilized for AI coaching, search use, and AI enter (together with agentic use). Whether or not a given bot honors these indicators relies upon on the bot operator. Cloudflare mentioned the function will provide customized Content material-Sign insurance policies in the future.
The Content material Alerts framework, which Cloudflare announced during Birthday Week 2025, lets website homeowners set preferences for a way their content material will get used. Enabling markdown conversion additionally applies a default utilization sign, not only a format change.
How This Differs From What Mueller Criticized
Mueller was criticizing a special observe. Some website homeowners construct separate markdown pages and serve them to AI consumer brokers by means of middleware. Mueller raised considerations about cloaking and damaged linking, and questioned whether or not bots may even parse markdown correctly.
Cloudflare’s function makes use of a special mechanism. As an alternative of detecting consumer brokers and serving alternate pages, it depends on content material negotiation. The identical URL serves totally different representations based mostly on what the consumer requests in the header.
Mueller’s feedback addressed user-agent-based serving, not content material negotiation. In a Reddit thread about Cloudflare’s function, Mueller responded with the identical place. He wrote, “Why make issues much more sophisticated (parallel model only for bots) quite than spending a little bit of time bettering the website for everybody?”
Google defines cloaking as exhibiting totally different content material to customers and search engines like google with the intent to manipulate rankings and mislead customers. The cloaking concern might apply otherwise right here. With user-agent sniffing, the server decides what to present based mostly on who’s asking. With content material negotiation, the consumer requests a format and the server responds. The content material is the identical information in a special format, not totally different content material for various guests.
The sensible consequence is nonetheless related from a crawler’s perspective. Googlebot requesting commonplace HTML would see a full webpage. An AI agent requesting markdown would see a stripped-down textual content model of the identical web page.
New Radar Monitoring
Cloudflare additionally added content material kind monitoring to Cloudflare Radar for AI bot visitors. The information exhibits the distribution of content material varieties returned to AI brokers and crawlers, damaged down by MIME kind.
You’ll be able to filter by particular person bot to see what content material varieties particular crawlers obtain. Cloudflare confirmed OAI-SearchBot for instance, displaying the quantity of markdown responses served to OpenAI’s search crawler.
The information is obtainable by means of Cloudflare’s public APIs and Knowledge Explorer.
Why This Issues
In case you already run your website by means of Cloudflare, you’ll be able to allow markdown conversion with a single toggle as an alternative of constructing separate markdown pages.
Enabling Markdown for Brokers additionally units the Content material-Sign header to ai-train=sure, search=sure, ai-input=sure by default. Publishers who’ve been cautious about AI entry to their content material ought to evaluate these defaults before toggling the function on.
Trying Forward
Cloudflare mentioned it plans to add customized Content material-Sign coverage choices to Markdown for Brokers in the future.
Mueller’s criticism targeted on separate markdown pages, not on commonplace content material negotiation. Google hasn’t addressed whether or not serving markdown by means of content material negotiation falls underneath its cloaking tips.
The function is opt-in and restricted to paid Cloudflare plans. Overview the Content material-Sign defaults before enabling it.
Disclaimer: This article is sourced from external platforms. OverBeta has not independently verified the information. Readers are advised to verify details before relying on them.
