Will AI crawlers crawl resources prohibited in robots.txt?

In general, compliant AI crawlers will abide by the directives in robots.txt that prohibit access to resources, but some non-standard or malicious AI crawlers may ignore this protocol. Essentially, robots.txt is a "gentleman's agreement" through which websites convey crawling rules to crawlers; it has no legally binding force, and its effectiveness depends on the compliance意愿 of crawler developers. AI crawlers of major search engines (such as Googlebot and Bingbot) usually strictly follow robots.txt and will actively avoid paths marked as "Disallow"; however, some AI tools used for data collection or crawlers that do not follow industry norms may disregard the restrictions of robots.txt and directly crawl prohibited resources. To prevent AI crawlers from accessing sensitive content, in addition to configuring robots.txt, you can combine meta tags (such as `<meta name="robots" content="noindex">`) or server IP restrictions. Regularly checking server logs can help identify whether there are non-compliant AI crawlers accessing prohibited resources.