How to optimize your website for AI agents (not just AI answers)

What does it mean to optimize your website for AI agents?

It means making your site usable by software that visits on a person's behalf, not only readable by engines that quote it. Answer engine work gets your brand named in ChatGPT, Perplexity or Google AI Overviews. Agent work makes sure that when an assistant opens your pricing page, fills in a demo form or adds a product to a cart, it succeeds.

The two overlap heavily. Crawlable HTML, clear headings and consistent facts help both, which is why our general guide on how to optimize for ai search is the right starting point. What changes with agents is that they click, type and submit, so form labels, button names, layout stability and bot rules suddenly matter to revenue, not just to visibility.

It helps to separate three kinds of visit, because each one needs a different decision.

Which AI agents visit your site, and how do they identify themselves?

Most announce themselves with a user-agent string, and the newer acting agents can also sign their requests cryptographically. Read each vendor's documentation rather than guessing from the name.

OpenAI's crawler documentation says OAI-SearchBot decides whether a site can appear in ChatGPT search answers, while ChatGPT-User visits pages when a user asks a question and, because the action is user-initiated, robots.txt rules may not apply. Google's crawler documentation lists Google-Agent as the fetcher used by agents hosted on Google infrastructure to navigate the web and act on a user's request. Google says user-triggered fetchers like it generally ignore robots.txt, that it uses IP ranges published in user-triggered-agents.json, and that it is experimenting with Web Bot Auth under the identity agent.bot.goog.

Web Bot Auth is the part most guides skip. It lets an agent sign each request with HTTP Message Signatures (RFC 9421) so your server or CDN can check a public key instead of trusting a user-agent string that anyone can fake. ContextBolt's Web Bot Auth guide, updated October 2026, reports that ChatGPT's cloud browser signs requests with a Signature-Agent value of https://chatgpt.com, that Cloudflare and AWS WAF verify these signatures at the edge, and that the IETF specification was still a draft as of August 2026. Treat it as production-ready plumbing on a standard that may still change.

The products behind these agents move fast. Digital Trends reported that OpenAI announced on 9 July 2026 that its ChatGPT Atlas browser would stop working on 9 August 2026, with agent browsing moving into the ChatGPT desktop app and Codex. Plan around the identities in vendor documentation, not around any single app.

Should you block or allow AI agents in robots.txt and your CDN?

Allow the search crawlers and answer fetchers that can send you buyers, make a deliberate choice on training crawlers, and manage acting agents at the CDN or firewall, because robots.txt alone will not stop them. This is the core of AI crawler access work.

Defaults are often the problem. Cloudflare changed its default in July 2025 to block AI crawlers unless they pay, and many sites now block agents they never meant to. Check the Cloudflare bot settings, any WAF rules and your rate limits, then confirm in server logs that the agents you want receive HTTP 200 rather than 403 or a challenge page.

Use the table below as a starting point, then adjust for content you sell or license.

What does the Lighthouse agentic browsing audit check?

It checks whether an agent could use your page: a sound accessibility tree, WebMCP tool definitions, an llms.txt file and low layout shift. It is a usability test for agents, not a search ranking report.

According to accessiBe's June 2026 explainer, Lighthouse 13.3 added the Agentic Browsing category in May 2026. Chrome's own documentation calls the category and its WebMCP support experimental and based on proposed standards, reports a pass ratio instead of a 0 to 100 score, and states that agents rely on the accessibility tree as their primary data model. That last point is the practical one: an unlabeled icon button or a div pretending to be a link is invisible or ambiguous to an agent.

Search Engine Journal reported in September 2026 that Lighthouse 13.5 added an Agentic Resource Discovery audit (ard-schema) that validates an ai-catalog.json manifest, found through a robots.txt Agentmap line, a link element, an HTTP Link header or the default /.well-known/ai-catalog.json path. The same report notes that nothing in the release ties it to Google Search rankings. If you do not publish APIs, tools or MCP servers, you can ignore that audit for now. The same goes for llms.txt: no major answer engine has confirmed using it, so treat a pass as cheap hygiene.

How do you make pages and forms usable by an agent?

Build pages the way an accessibility auditor would want them: real buttons and links, a visible label on every field, content and prices in the HTML, and nothing that moves or covers the page while it loads. Most of this is technical AEO work that also helps human visitors.

Start with the pages an agent is most likely to be sent to: pricing, product and category pages, comparison pages, demo or quote forms, store locators and booking flows. For B2B SaaS that usually means pricing tiers, integrations and the demo form. For online stores it means product data, stock, shipping and returns. For hotels and clinics it means availability and booking.

• Every interactive element has an accessible name: no unlabeled icon buttons, no placeholder text used as the only label.
• Prices, plan limits, stock and opening hours are in server-rendered HTML, not injected after a script runs or hidden in images.
• Forms have a guest path where possible, with clear error messages in text and no CAPTCHA on the first step for verified agents.
• Cookie banners, chat widgets and newsletter pop-ups do not cover the main content or trap focus.
• Cumulative Layout Shift is low, so a button does not move just as an agent clicks it.
• URLs are stable and descriptive, and filters produce crawlable URLs rather than state held only in the browser.
• Schema markup matches what is visible on the page, so product, price and organization facts agree everywhere.

Do agentic checkout and payment protocols matter yet?

For most businesses, not yet. They matter now if you sell products online at volume or sell paid data or APIs; everyone else should get the basics right and watch.

On the shopping side, Google's Universal Commerce Protocol, announced in January 2026, powers checkout in AI Mode and Gemini with the retailer remaining the seller. OpenAI moved Instant Checkout out of the chat and into Apps in March 2026, according to Digital Commerce 360, while its Agentic Commerce Protocol continues. Our ChatGPT shopping guide covers the feed and merchant side in detail.

On the access side, Cloudflare launched a closed beta of Monetization Gateway in October 2026. It lets site owners charge AI agents per request for websites, APIs, MCP tools and datasets, returning payment instructions with an HTTP 402 Payment Required status, with payments currently settling in USDC on the Base blockchain. That is relevant to publishers and data businesses. For a typical brand that wants agents to recommend and buy from it, putting a toll on the front door works against the goal.

How do you measure AI agent visits and their outcomes?

Use server or CDN logs as the source of truth, because most agents never run your analytics script. Segment requests by user agent, then confirm identity with signature verification or the published IP ranges, and track which URLs each agent requests and what status code it receives.

A useful weekly view has four numbers per agent family: requests, share of 200 responses, top URLs and form or cart endpoints reached. If OAI-SearchBot gets 403s on your pricing page, or Google-Agent reaches your demo form and stops, you have a specific fix. Keep a change log so you can match each fix to a change in the logs.

Then connect it to outcomes you already track. GA4 still shows human click-throughs from chatgpt.com, perplexity.ai and gemini.google.com. Bing Webmaster Tools AI Performance reports Copilot citations and citation share. A fixed prompt set, re-run monthly, shows whether assistants now recommend you. Our AI visibility tracking service combines those views.

What does a 90-day agent-readiness plan look like?

Fix access in the first month, usability in the second and measurement and protocol decisions in the third. The order matters, because nothing downstream helps an agent that receives a 403.

This is the same sequence we use in an AEO audit, with the agent checks added to the usual AEO checklist.

When should you not spend money on agent readiness?

Skip dedicated agent work if your basics are not done: if AI crawlers cannot reach you, your key pages are not indexed in Google and Bing, or your content does not answer buyer questions, fix those first. Agent readiness sits on top of that foundation.

Gartner predicts that 90% of B2B buying will be AI agent intermediated by 2028, which is a good reason to prepare, but it is a forecast, not a deadline. If you sell through relationships, have one long sales form and no public pricing, a light pass on access and form labels is enough this year. Be wary of anyone selling WebMCP builds, ai-catalog.json files or llms.txt as a route to more citations. None of them has been confirmed as a ranking or citation signal by Google, OpenAI or Perplexity.

If you want a second opinion, AEO Services Agency, the AEO and GEO practice of Greenwolf Tech Labs, offers a free AEO visibility check. Any follow-on work is quoted at a fixed price, and we measure a fixed prompt set and your agent logs before and after changes so the results can be verified. No one can promise a citation or an agent conversion; a site that agents can reach, read and use gives you the best odds.