How to set Crawl-delay to control the crawler access frequency?

When it is necessary to control the frequency at which search engine crawlers access a website, this can be achieved by setting the Crawl-delay directive in the robots.txt file, which is used to define the interval time between two requests from crawlers. Setting format: In the robots.txt file, add the "Crawl-delay: [value]" directive for a specific User-agent, where the value is usually in seconds. For example, "User-agent: * Crawl-delay: 10" means all crawlers need to wait 10 seconds before making the next access; if targeting a specific crawler (such as Googlebot), it can be written as "User-agent: Googlebot Crawl-delay: 15". Notes: Different search engines have different support for Crawl-delay; some engines (such as Baidu) may prioritize their own crawling strategies. The value should be adjusted according to server load; too short an interval may increase server pressure, while too long may affect content crawling efficiency. Recommendation: After setting, monitor the crawler crawling status through search engine webmaster tools (such as Google Search Console), analyze the actual effect in combination with website access logs, and gradually optimize the interval value to balance server stability and content crawling efficiency.