Can AI Agents Read Your Website? What Changes When They Can't
When a buyer asks ChatGPT or Claude about your pricing, the agent tries to open your site. New data on what happens when it can't, and how to test yours.
Frequently asked questions
Can ChatGPT read my website?
Usually, if your content is in the HTML the server sends. When a user asks about a page, ChatGPT fetches it with the ChatGPT-User agent. Crawl-log analysis by Vercel and MERJ found that none of the major AI crawlers, OpenAI's included, executed JavaScript, so content that only appears after client-side rendering is often invisible to them. Firewall or CDN rules that block AI agents will also stop the fetch.
Do AI agents render JavaScript?
Mostly not. Vercel and MERJ's December 2024 analysis of crawler traffic found that OpenAI, Anthropic, Meta, ByteDance, and Perplexity crawlers did not render JavaScript. ChatGPT's crawler requested JavaScript files in 11.5% of fetches and Claude's in 23.8%, but neither ran them. Google's Gemini was the exception, because it uses Googlebot's rendering infrastructure.
Does robots.txt block ChatGPT-User?
Not reliably. OpenAI's crawler documentation says that because ChatGPT-User actions are initiated by a user, 'robots.txt rules may not apply.' Blocking OAI-SearchBot keeps a site out of ChatGPT search answers, and blocking GPTBot opts out of training. A CDN or firewall rule is what actually stops a user-triggered fetch.
Does my own website matter for AI visibility if most citations are third-party?
Yes, at a different step. Third-party sources dominate when an AI engine is building a shortlist from a category question. Once a buyer asks about your business specifically, agents try to read your own pages. In a 2026 preprint covering 37,927 agent journeys, answers about agent-readable businesses were built from the business's own pages 78% of the time, against 58% for businesses whose sites agents struggled to read.
What happens when an AI agent can't read my site?
It mostly leaves things out rather than making them up. In the same preprint, facts the user asked for went unmentioned 45% of the time in answers built from the wider web, against 29% in answers built from the business's own site. Facts stated wrongly barely moved, from 4% to 6%. Answers about hard-to-read sites were also four times more likely to say the agent could not access the site.