Cloudflare ties AI crawler choices to robots.txt
Cloudflare has announced Bot Preference Sync, a feature that updates robots.txt based on a site owner’s AI bot settings for Search, Agent and Training traffic. The company says the tool is intended to keep published crawler preferences aligned with enforcement choices in the Cloudflare dashboard. The feature is planned for all plans, from Free to Enterprise, with availability expected in the coming week. For publishers and other site owners, the practical question is how much AI crawler access should be allowed, limited or blocked without breaking ordinary discovery.Why robots.txt alignment matters
Cloudflare frames Bot Preference Sync as a response to a common policy gap: a site can say one thing in robots.txt while enforcing another rule at the edge. The company says that mismatch can create confusion for site owners and may give some crawlers room to disregard stated preferences or test the boundary between preference and enforcement.Robots.txt has long been a way to publish crawling preferences, but Cloudflare separates that from stronger blocking through Bot Management. The new feature does not turn robots.txt into a hard access-control system by itself. Instead, it tries to reduce the operational burden of maintaining one static file for declared preferences and another set of dashboard rules for enforcement. The implication is administrative rather than merely cosmetic: one policy setting can be reflected in both the public crawler signal and Cloudflare’s traffic controls.
How Bot Preference Sync works
Bot Preference Sync uses the AI bot configuration already set at Cloudflare’s zone level and writes matching entries into robots.txt. Cloudflare says this applies to the categories Search, Agent and Training, which the company introduced as separate AI traffic use cases on July 1, 2026.If a customer already has a robots.txt file, Cloudflare says the new Bot Preference Sync material will be prepended to the existing contents. Existing Disallow directives are therefore maintained rather than overwritten. That detail matters for sites with legacy crawler rules, because the feature is designed to add Cloudflare-managed category policy without erasing locally maintained restrictions.
The company also says the feature can be turned on or off at any time. Customers with complex custom arrangements, such as exceptions for a particular company, may still need to manage their own file because Bot Preference Sync works at the category level rather than reading every individual custom rule.
Search, Agent and Training get different choices
Cloudflare’s model separates AI crawler behavior into Search, Agent and Training categories, rather than treating all AI-related access as one decision. For Search and Agent traffic, Cloudflare says customers can choose Allow, Block on pages that serve ads, or Block everywhere.Training is handled differently through a Disallow option. In Cloudflare’s description, Disallow writes a “no training” preference to robots.txt. Cooperating mixed-use crawlers can still access content for search indexing if they meet Cloudflare’s transparency conditions and honor the no-training preference.
This distinction is central to the product’s value proposition. A publisher may want its pages to remain visible in search while not being used for model training. An e-commerce site, by contrast, may decide broader AI crawler access is useful for product discovery. Cloudflare’s message is that one default policy will not fit every business model.
Transparency becomes a condition for mixed-use bots
Cloudflare says mixed-use crawlers create a problem when search, agent use and training are blended behind a single user agent. In that scenario, a site owner may not be able to distinguish the use case they want from the one they reject.For bot verification, Cloudflare says owners of bots that perform both Search and Training must provide additional information if they are to avoid being blocked when a site has selected Disallow Training. The stated requirements include respecting a no-training preference in robots.txt, giving site owners a way to opt out of AI summaries, providing URL-level visibility into which pages were made available for training, providing metrics on search results, and publicly showing that disallowing training does not hurt traditional search results.
Cloudflare says bots from leading AI models and service providers that meet these criteria are tracked publicly in the AI bot transparency section of Cloudflare Radar. The company also says crawlers that do not provide transparency will not receive the benefit of the doubt when training is disallowed.
Defaults differ for publishers and other sites
Cloudflare says Bot Preference Sync will be on by default for new customers, but the default policy outcome depends on the type of site and the options selected during onboarding. For ad-supported publishing sites, Cloudflare is adding an onboarding choice labeled “I monetize from pages with ads on this domain.” If selected, Training will be set to Disallow by default.That configuration is meant to let an ad-supported site stay available for search while keeping content out of model training, according to Cloudflare. The company says customers can change the setting at any time.
For new non-publisher customers, Cloudflare says no blocks or disallows will be added by default when a domain is onboarded. Those customers can later choose whether to block Search, Agent or Training, but the starting point does not add restrictions on their behalf.
Rollout and operational limits
Cloudflare says Bot Preference Sync will be available to all customers on every plan in the coming week. Existing customers using the legacy managed robots.txt feature will be prompted to review and confirm preferences to transition to Bot Preference Sync when it launches.The feature will use bots tracked in Cloudflare’s BotBase to periodically update the list of bots added to robots.txt when a customer chooses to Block or Disallow a category. Cloudflare says verified bots classified as Search, Agent and Training can be viewed in its public bots directory.
There is a clear limit: the feature is designed for category-wide policy decisions, not fine-grained contractual exceptions. Customers that need tailored treatment for a particular crawler can turn off the group-policy sync and maintain their own robots.txt to match custom policy.
Conclusion
Bot Preference Sync is a control-plane update rather than a new standard for the web. Its significance is that Cloudflare is trying to make declared crawler preferences and enforced AI bot rules less likely to drift apart.For site owners, the key decision remains strategic. Search visibility, AI assistant referrals, ad-supported page views and protection from training use can point in different directions. Cloudflare’s feature gives customers a simpler way to publish and enforce a category-level choice, but the burden of deciding that choice still belongs to the site owner.
Sources
Editorial Team - CoinBotLab
