It is recommended to use Cloudflare new means Protect websites from automatically downloading their materials for training AI models: By blocking AI bots designed to collect training information, website owners can preserve them in search indexes.
Image source: cloudflare.com
Until now, website owners have had no choice: the same web bots collect information for AI training and search engines; websites whose content is prohibited for AI training have been removed from search indexes. The standard directives in the “robots.txt” file are not sufficient because they do not allow identification of robots and the directives themselves are advisory in nature. Cloudflare attempts to fix the problem at the network level, identifying bots and blocking them if necessary, but this also seems like too drastic a measure. As a result, the company entered into direct negotiations with bot operators so that they no longer leave website owners with such a choice.
To help website owners address transparency concerns, Cloudflare introduced the Accountable tag. To achieve this status, bot operators must provide website owners with the opportunity to refuse to train AI on source material, implement mechanisms to refuse inclusion of website material in search snippets, and ensure URL-level transparency – the ability to see which website pages are available for AI training and how the content is displayed in search results. Apple, Google, and Microsoft have agreed to meet criteria for accountability status—some features already implemented, some still in development.
Photo credit: Mohamed Nohassi / unsplash.com
Cloudflare categorizes bots by behavior type. It is divided into search engines, used to form indexes, used to train AI, and the third category is AI agents. The fourth category is hybrid bots, which are used both to index sites and to train artificial intelligence, creating a dilemma. In Cloudflare settings, you can now select the “Disable AI training” option for operator bots with “Responsible” status – with this directive, they are obliged to use them only for search, but not for AI training. The rest of the options remain the same: bots can open the site completely, close the site completely, or just block bots from accessing ad pages. User AI agents occupy a special position in that they can also download website material, but do not do so en masse, so they have not been banned individually because there is no unified standard.
In the future, Cloudflare plans to study the issue of AI summarization in search results in more detail – it’s a complex issue. More than half of users read AI snippets in search results, making their search 40% more likely to end. On the other hand, users who come from artificial intelligence snippet links have a conversion rate 3-5 times higher than those who come from regular search results. For a news website that relies on advertising, increasing the number of visitors is important; if the number of orders increases, the online store can survive with smaller traffic. Therefore, being able to avoid inclusion in AI reports is not the end goal. Website owners need to understand how this tool affects their business and have a way to control the amount of content used.
If you find an error, select it with your mouse and press CTRL+ENTER.
