How to set up robots.txt to allow AI crawlers from specific IP ranges to access?

When it is necessary to allow AI crawlers from specific IP ranges to access a website, robots.txt itself cannot directly restrict or allow access based on IP addresses, as it primarily controls crawler behavior through the User-Agent directive. In such cases, it is necessary to combine server configuration with robots.txt to achieve this. First, identify the User-Agent identifier of the target AI crawler (such as the crawler name of a specific AI company) and obtain its IP address range. Then, set up an IP whitelist at the server level (e.g., Apache's .htaccess or Nginx configuration) to only allow access from the target IP range; at the same time, add an `Allow` directive in robots.txt for that User-Agent (e.g., `User-agent: [AI crawler User-Agent] Allow: /`) to explicitly permit it to crawl content. After completing the configuration, tools can be used to test the effectiveness of the IP restrictions and whether the robots.txt rules are in effect. It is recommended to regularly check for updates to the AI crawler's IP range and User-Agent to ensure the configuration remains compatible and to avoid access issues caused by IP changes.