Bots access some web content more often than people, according to new research by F5. Over half of all page visits for content come from automated sources, including AI bots collecting information.
The rise of generative AI has significantly impacted web traffic. F5 Labs analyzed 207 billion web and API transactions between November 2023 and September 2024. The results show that 50.04 percent of page requests for content come from automated sources, compared to 22.3 percent of web searches and 21.5 percent of ‘add to cart’ transactions.
The research indicates a significant increase in the type of web scraping used by LLM providers such as OpenAI, Anthropic and Perplexity. These bots continue to send requests even when they are blocked. Of the 21.22 billion transactions (10.2 percent) from various automated sources, 4.8 percent consisted of malicious bot traffic.
Different sectors affected differently
The patterns and prevalence of bot traffic vary by sector. On the web, the sectors most affected are the hospitality industry (44.6 percent), healthcare (32.6 percent) and e-commerce (22.7 percent). On mobile, the entertainment sector is the most affected at 23 percent, well ahead of e-commerce (4.5 percent) and quick-service restaurants (4.2 percent).
Various sectors are still struggling with account takeover attempts. On the web, more than a third (33.5 percent) of the login attempts at technology companies were account takeover attempts, followed by general retail (25.7 percent) and gaming (19.6 percent). In mobile applications, these attacks are most common among entertainment companies (24.7 percent) and e-commerce providers (23.8 percent).