Over the past decade, cybersecurity reports on the current web traffic due to both useful and malicious bots have continuously emerged and are highlighting the immense increase in such traffic. Bots are therefore responsible for a lot of so-called invalid traffic which can skew analytics and cost website owners traffic and advertising budget.
Imperva releases annual reports detailing the current statistics. Here is an excerpt that outlines the general problem:
Bad bot traffic levels rose for the fifth consecutive year, indicating an alarming trend. This increase is partly driven by the increasing popularity of Artificial Intelligence (AI) and Large Learning Models (LLMs). In 2023, bad bots accounted for 32% of all internet traffic – a 1.8% increase from 2022. The portion of good bot traffic also increased, albeit slightly less significantly, from 17.3% of all intent traffic in 2022 to 17.6% in 2023. Combined, 49.6% of all internet traffic in 2023 wasn’t human, as human traffic levels decreased to 50.4% of all traffic.
Source: Imperva 2024 Bad Bot Report
Lunio provides a similar report focused on the effects of bots on advertising. Here is a quote explaining the general issue:
Invalid traffic (IVT) refers to any website visits that don’t come from a real person with genuine interest. It can include bots (both good and bad), fake users, misattributed accidental clicks, malicious clicks from competing advertisers, and otherwise invalid visitors that have zero chance of converting to customers.
It instantly wastes budget - because those “visitors” will never convert.
IVT also distorts analytics, leading to unwise budget allocation.
Source: Lunio Wasted Ad Spend Report 2024
The good news is that publishers are not among the most highly targeted business groups.
At Metro Publisher we are committed to protecting client sites nonetheless and monitors the load on our databases caused by such traffic in order to keep bad bots in check. We block certain aggressive bad bots by default via the robots.txt file on all client sites depending on our findings.
If you would like to review the list of bots we block and or modify your site's robots.txt file yourself, please visit our help article on how to do so: Using Pages to Create a Custom Robots.txt File.
Please note that AI Assistants that are generally considered useful, such as ChatGPT, may also cause traffic loads to your site. ChatGPT and AI Search Crawlers such as Amazon's (Alexa) and Apple's (Siri) do provide reference links back to their sources. All AI doesn't need to be blocked, but it is a topic to consider since copyrighted content is used and republished. Information is fabricated, meaning that your content may be misquoted and still be referred back to your site as the source of the misquote.
A comprehensive list with descriptions of the various types of of normal and artificial agents circulating the web can be found here.
