With AI Audit, websites will get tools to monitor and possibly block data scraping bots. Scraping is used to collect data from websites and feed it to an AI model.
Cloudflare says that any website can encounter data scraping. Whether a commercial organization, news site, or small personal website, content can be scanned thousands of times a day without the website owner’s knowledge. Bots need this content to train AI models to improve their performance. However, content creators are often not compensated even though their content represents value.
AI Audit offers more control over data scraping bots. The goal is to achieve a transparent data exchange, where both websites and AI models can extract value from the data used for training.
“AI is a quickly evolving space, and many website owners need time to understand and analyze how AI bots are affecting their traffic or business. Many small sites don’t have the skills or bandwidth to manually block AI bots. The ability to block all AI bots in one click puts content creators back in control,” Cloudflare said of the free tools.
Statistics
Website owners can instantly see analytics to understand why, when, and how often AI models access their content. These analytics also differentiate between different bots. For example, one bot may generate text while neatly referencing the source, while others scrape without giving credit.
In addition, Cloudflare supports negotiations with model providers. More and more Web sites are entering into agreements with these vendors to receive fees for training their AI models. The AI Audit tab provides useful information for these negotiations, such as a fee for full-page scrapping.