GLOSSARY
CDN and Edge Layer
IN ONE SENTENCE
A CDN or edge layer handles requests before the origin and can block outside your site configuration.
A CDN or edge layer handles requests before the origin and can block outside your site configuration.
01
In detail
This is the most commonly missed layer in AI crawler access: robots.txt can allow while the edge blocks anyway. Cloudflare has blocked AI crawlers by default for new domains since 1 July 2025, and its rule does not separate retrieval from training.
Updated 2026-08-06