LEARN
User-Agent-Based Prerendering Is a Trap
IN ONE SENTENCE
User-agent-based prerendering serves AI retrieval crawlers an empty shell while the page renders perfectly in a browser — only comparing responses across agents reveals it.
Some sites decide what to return based on user agent: prerendered full HTML for known search engines, and a shell requiring script execution for everything else. AI retrieval crawlers usually fall into 'everything else' and receive the shell.
OUR POSITION
This is the most expensive configuration error we have seen: it consumes the return on every content investment while the team sees a working page and stable rankings and never suspects this layer. Detection costs one command, and every site should check it first.
Why the configuration exists
History: to let search engines that do not execute JavaScript read content, many sites added a branch that serves a prerendered version to crawlers.
The allowlist was written against the agents known at the time. Newer AI retrieval crawlers are not on it, fall into the default branch, and receive a shell that only fills in after scripts run.
How to detect it
Request the same URL with different user agents and compare response size and whether the body copy is present. A clear difference means branching.
Cross-check in server logs: 200 responses with unusually small bodies are the signature.
Check structured data while you are there — it has to appear in server-rendered HTML, since anything injected by a front-end script is invisible without script execution.
What to do instead
Do not branch on user agent. Return the same readable HTML to every request, via server-side rendering or static generation.
If prerendering genuinely has to stay, include retrieval crawlers in the allowlist and **add a standing check** — lists expire and new platforms appear.
Sources
- [1]Optimizing your website for generative AI features on Google Search．Google Search Central．2026-05-15
Updated 2026-08-10