OpenAI has announced its intention to support a widespread signal within the robots.txt file. This new signal will enable websites to clearly communicate their preferences regarding the use of their content for training generative AI models.
This initiative aims to simplify the process for publishers and website operators by allowing them to convey their wishes simultaneously to multiple AI providers involved in developing this standard. Previously, site owners had to identify individual crawlers (bots) and configure unique rules for each provider, a process that will be significantly streamlined with the adoption of a unified signal.
OpenAI plans to implement this signal in the coming months and will share further implementation details later. The new signal is designed to complement, not replace, existing control mechanisms provided through robots.txt. OpenAI’s current crawler directives, which offer flexibility and control for web publishers, will remain in effect.
This move is rooted in the ongoing work of the IETF (Internet Engineering Task Force) working group focused on AI preferences. OpenAI joins other group participants who have announced this common signal today, reinforcing the company’s commitment to participating in IETF standards discussions. OpenAI will continue to adapt its practices in line with these discussions and the evolving landscape, ensuring appropriate control and fairness mechanisms are in place.
The company views this as a practical step toward fostering an open web where individuals can publish and discover information, and website owners have clearer, simpler ways to express their choices about how their content is used. OpenAI encourages other AI providers to also adopt and support this signal.
Source: OpenAI
