Why the next AI hurdle for websites is supporting agents

Ninety-seven percent of top websites aren’t ready for AI agents, according to research from BrightEdge, an AI-powered organic search provider. But these agents are here, and they are banging on your website’s door. Is there a way to let them in without rebuilding your website to meet their needs? BrightEdge says yes, and it has it. BrightEdge says that for every 100 visits a site receives from human organic search, it receives, on average, 88 from AI agents, the bulk of which are from OpenAI. AI Agents have become digital assistants that help people get the information they need, including researching products, comparing options, and checking product availability. These agents don’t just hit your website once; instead, they often make multiple trips to your site to get everything they need for the end user. Says BrightEdge: > For digital marketers, that number [95% of agent interactions come from OpenAI] has a practical implication that goes beyond strategy. If you haven't already had conversations with your IT, dev, or ops teams about AI agents, there's a good chance those agents are currently being treated the same as any other bot -- meaning they may be throttled, blocked, or otherwise restricted from reaching your content. The systems that control this sit outside of marketing's hands, and closing that gap sooner rather than later is no longer optional. Agentic challenge Your website needs to support these agents. You can’t block them; they are your window to get found in AI search or to help a ChatGPT user (or other AI assistant) answer questions about your brand and products. But you also don’t necessarily want them to come to your website and take forever to find the information they need. Agents work best with simplicity. They like structure - like Markdown (a lightweight, structured text format that strips away much of the presentation and scripting of a normal web page). And your website is not written in Markdown. BrightEdge says that AI agents are creating a “second web.” The second web is a new layer of access that creates a second audience for your content. How well you create that second web (and how you do it) affects the quality of the interactions between agents and your content. Is it really that important to give agents easier access to content? From June 2025 to June 2026, BrightEdge analyzed ChatGPT recommendations across nine industries. It found that ChatGPT researched more but recommended brands less. Specifically, The number of sources cited per answer doubled. The number of brand recommendations fell by around 25%. As Jim Yu, CEO and co-founder of BrightEdge, notes: > AI agents are getting dramatically better at doing the work people once had to do themselves.They are researching, comparing and taking action on our behalf, making the point of interaction between an agent and a brand increasingly critical. The companies that can understand and optimize those interactions will have a significant advantage as this second audience grows. While it is important to ensure agents can get the right information, it’s not necessarily critical that information is delivered in Markdown format. For agents that specifically request it, yes, it needs to be available, but most agents can read a traditional web page fine, as long as it’s structured properly. In fact, Profound did anexperiment to see if serving Markdown to agents improved a brand’s visibility in AI search. It looked at 381 pages across six websites, with half the pages serving HTML and half serving Markdown to AI agents.Here’s what it found: > At first glance, the Markdown group looks slightly ahead: one more median visit per page, about 16% higher on average. But here's the catch: the median page (your typical page) only got one extra visit. The 16% mean difference is driven almost entirely by high-traffic pages in the tail of the distribution, pages that were already getting lots of bot attention saw modest gains, while the median page saw almost nothing. [..] Put simply: we can't say with confidence that Markdown actually drove more traffic. How SEO tool providers are supporting agents There’s more to supporting AI agents than implementing a schema, but that’s what most brands are doing today, says BrightEdge. Very few are taking the next step of serving content specially designed to support agent queries. That being said, Search Engine Optimization (SEO) tool providers are giving brands the tools they need to support AI agents, including BrightEdge. Here are a couple of them. Agent Edge from BrightEdge Agent Edge looks for agent requests at the Content Delivery Network (CDN) layer. When it gets an agent request, it serves content to the agent in Markdown, reducing the number of tokens an agent uses. The Markdown versions are copies of the HTML web pages with all the extra stuff taken out (like navigation, JavaScript, and so on). Marketers have a dashboard where they can select the web pages they want converted to Markdown, and Agent Edge will create the Markdown version (marketers can make edits as well). BrightEdge goes further, though, because Agent Edge can use BrightEdge Intelligence to anticipate follow-on questions the agent is likely to ask and build on the initial content with those answers. Enriching the response with additional important information means the agent doesn’t have to come back and request more information. What’s nice about BrightEdge is that you get SEO, AI search, and agent interactions in one platform, allowing you to see and optimize holistically across all three. seoClarity’s AI Bot Optimizer seoClarity also tracks AI agents and delivers content optimized as Markdown. Like BrightEdge, it is implemented at the CDN layer, intercepting all requests and delivering Markdown if an agent requests it (it’s called HTTP Content Negotiation). Brands can select which web pages to convert to Markdown and approve the Markdown version. Both the HTML and Markdown versions of a web page share the same canonical URL, making it easy to track and analyze traffic, with full auditability and logging. Cloudflare Markdown for Agents Cloudflare isn’t an SEO provider; it is an internet infrastructure and security company. A CDN is one of the products it provides. It also uses HTTP Content Negotiation, so when an agent sends a request and asks for a Markdown response, Cloudflare converts the web page in real time at the source and delivers it to the agent. To help brands learn how much Markdown is being delivered, Cloudflare added a content type to its Radar product, allowing marketers to see AI bot and crawler traffic. My take The more tools we have that help brands understand how to get found in AI search, the better. But marketers need to be careful and ask lots of questions about how features work and what results they can expect. And tool providers need to prove why they work the way they do. BrightEdge can track how AI agents interact with brands, what those agents are trying to access, and how that behavior is changing. That means they can show a brand the before-and-after of implementing Agent Edge. This is the kind of information brands need from their tool providers to know their investment is warranted.