Cloudflare has introduced several new measures aimed at combating artificial intelligence search robots. To begin with, every new domain client that registers with the company to manage their website traffic will now be asked if they want to allow AI bots or block them altogether. In 2024, the company released a free tool for blocking AI bots, but with this change, users can block them by default without having to interfere with the settings. Several large publishers, including Condé Nast, TIME, and The Associated Press, have already signed up to block search robots. In addition, Cloudflare has launched a private beta experiment called “pay per crawl” that will allow crawlers to access website content only if they pay for it.
Matthew Prince, CEO of Cloudflare, recently stated that publishers face an existential threat because people do not follow links to chatbot sources. If users don’t visit these sources, websites don’t get the advertising revenue they need to keep operating. “Original content is what makes the internet one of the greatest inventions of the last century, and it’s critical that creators keep creating it,” said Prince in a statement released with the company’s latest updates. “Artificially intelligent search engines are pulling content without limits. Our goal is to put the power back in the hands of creators while helping AI companies innovate. It’s about protecting the future of a free and dynamic Internet with a new model that works for everyone.”
Cloudflare believes that publishers should be able to charge AI bots for access if they want to, and pay-per-crawl is the first experiment to do just that. “Whenever an AI crawler requests content, it either indicates its intention to pay in the request headers for successful access (HTTP 200 response code) or receives a 402 “Payment Required” response with a price,” Cloudflare explained. The company registers these transactions and provides the appropriate technical infrastructure. Publishers will be able to allow certain crawlers to access their content for free if they wish, and for other crawlers they can set a fixed price per query on their websites.
The company states that pay-per-crawl is still in very early stages and it expects this tool to evolve in the future. The company also states that it supports the development of other markets and ways to charge search robots for content. For example, the marketplace may allow dynamic pricing, which will allow publishers to set different rates for different types of content.
