GLOSSARY
robots.txt
IN ONE SENTENCE
robots.txt is a text file at the site root declaring which crawlers may fetch which paths.
robots.txt is a text file at the site root declaring which crawlers may fetch which paths.
01
In detail
Two common mistakes: multi-domain sites sharing one static file with a hard-coded domain, so one domain advertises another's sitemap; and treating it as the only switch while a CDN blocks independently. Its Sitemap line also has to point at that domain's own sitemap.
Updated 2026-08-06