Cloudflare has announced a new feature that automatically serves websites in Markdown to AI agents. With Markdown for Agents, the company is responding to the growing role of AI systems that process web content for search engines, summaries, and analyses.
The solution should make it easier and more efficient for AI agents to read information from websites without website owners having to modify their content.
The key is that Cloudflare can convert web pages at the edge from HTML to Markdown when a client requests it. This is done via an HTTP Accept header with the value text/markdown. When an AI agent sends this header, Cloudflare converts the HTML page to Markdown in real time and returns it. For regular visitors, the HTML display remains unchanged.
According to Cloudflare, Markdown is better suited for AI processing than HTML. HTML often includes a lot of extra code for styling, scripts, and layout, while Markdown offers a cleaner, more semantically clear representation of the content. This means less noise for language models and other text-analysis systems, resulting in more efficient processing and potentially lower compute costs.
Significant token savings
The token savings can be significant. Based on Cloudflare examples, The Register calculated that a simple heading in Markdown costs about three tokens, while the HTML variant quickly consumes twelve to fifteen tokens, even without additional markup such as div elements, navigation bars, and scripts. For a full blog post from Cloudflare itself, the difference would be even greater. According to The Register, the number of tokens drops from 16,180 in HTML to 3,150 in Markdown, a reduction of approximately 80 percent. This is relevant for AI services that work with limited context windows and token-based cost models.
When a website owner enables Markdown for Agents, Cloudflare adds an additional header, x-markdown-tokens, to the Markdown response. This header contains the calculated number of tokens in the returned content. This helps AI agents determine whether a document fits within the available context window or needs to be split up.
No separate Markdown versions required
The functionality applies to HTML pages and not to other document formats such as PDF files. The conversion occurs within the Cloudflare network, eliminating the need for publishers to maintain separate Markdown versions. In doing so, the company is positioning itself as an infrastructure provider for the internet, where not only human visitors but also AI agents are important target groups.
The move is notable because Cloudflare previously introduced mechanisms to charge AI crawlers for content access. Whereas the company previously positioned itself as a party that protects publishers from scraping, it now offers a way to make automated access more efficient. According to The Register, coding agents such as Claude Code and OpenCode already explicitly request Markdown in their Accept headers, indicating demand.
Markdown for Agents ties in with the previously introduced Content Signals Policy. This allows publishers to use robots.txt to indicate whether their content may be used for AI training, AI search applications, or AI input after training, for example, for retrieval augmented generation. Like robots.txt, these signals are voluntary and non-enforceable, but they provide a standardized way to communicate usage preferences.
With Markdown for Agents, Cloudflare is taking the next step in adapting internet infrastructure to the rise of AI traffic. While HTML has been the standard for browsers for decades, a parallel layer is now emerging, optimized for machine readers and the economic realities of token-based AI models.