Bot Management

Cloudflare ties AI crawler choices to robots.txt Cloudflare has announced Bot Preference Sync, a feature that updates robots.txt based on a site owner’s AI bot settings for Search, Agent and Training traffic. The company says the tool is intended to keep published crawler preferences aligned...