traffic boom of 50% due to AI bots

In the heart of the digital universe of free knowledge, Wikimedia is today facing one of the most complex challenges of its recent history: the growing wave of AI bot bull that systematically plunder its contents. 

In particular, in recent months there has been an increase of 50% in the traffic generated by the so-called crawler AI, which is putting a strain on both the technical capacity and the economic sustainability of the platform.

The impact of artificial intelligence (AI) on digital infrastructure is growing: the Wikimedia case

Starting from January 2024, there has been a skyrocketing growth in the volume of data downloaded from platforms like Wikipedia and other Wikimedia projects. 

This increase is not attributable to greater participation by human users, but rather to a systematic and often poorly regulated use of automatic bots employed by companies that develop artificial intelligence models.

These tools, designed to collect and analyze large amounts of text, images, and other content, use Wikimedia as a primary data source for the training of their algorithms. 

An operation that, on one hand, demonstrates the centrality of the platform in the ecosystem of digital knowledge, on the other hand, exerts an unsustainable pressure on its IT infrastructures.

The problem does not lie solely in the quantity of data transferred. The real critical issue is represented by the way these bots access the contents. 

In most cases, in fact, the requests are directed to rare or little-visited pages, that is, those that do not fall within the caching systems. In other words, mechanisms that allow temporarily storing copies of the most consulted pages to speed up their loading.

When this happens, the requests must be handled directly by the central servers, resulting in a significant increase in workload and, above all, in costs. 

This scenario becomes particularly critical in conjunction with events of high media relevance, during which “human” traffic already reaches high levels.

Bots out of control: they ignore the rules, evade the blocks

Another alarming dimension of the phenomenon is represented by the behavior that is increasingly sophisticated and, at times, incorrect of the crawlers. Many of these bots, in fact, ignore the established conventions, evade automatic blocking systems, and disguise themselves to appear as legitimate users.

This type of conduct not only violates the norms of good network usage, but forces the technical teams of Wikimedia to a continuous monitoring and a constant use of resources to protect the infrastructure. 

Resources that could instead be allocated to enhancing the platform or enriching its content.

In response to this situation, the Wikimedia Foundation is trying not to limit itself to a technical or defensive reaction. The proposed solution goes beyond merely containing the problem and aims for a collaborative and sustainable management of free knowledge.

Thus, WE5 is born, a new strategic initiative aimed at promoting more equitable and responsible approaches in the acquisition and use of data hosted by the platform.

The project is presented as an invitation to tech companies and artificial intelligence developers. 

Specifically, an invitation to respect the rules, contribute to the network management costs, and ensure the survival of the infrastructure on which one of the main sources of free information in the world is based.

The entire affair raises a crucial question for the future of free access to knowledge: in an era where data has become the lifeblood of artificial intelligence, who pays for the preservation and distribution of that data?

Wikimedia, always driven by the principle of gratuity and sharing, now finds itself at the crossroads between openness and sustainability.

Without a change of course by the big tech and the actors who massively use the foundation’s content, the project might be forced to reduce accessibility or introduce stricter limits to safeguard its infrastructure.

An appeal for respect of the digital public good

The message that Wikimedia sends to the world is clear. That is, free knowledge is a common good and, as such, it must be treated with respect and responsibility.

The use for commercial purposes of the enormous informational assets made available by the foundation must occur in a transparent manner, in accordance with the rules and. Furthermore, if necessary, accompanied by forms of fair contribution.

In an increasingly digital landscape dominated by algorithms and automation, it is essential to ensure that access to knowledge is not compromised by the economic interests of a few. 

Only through an open dialogue between communities, institutions, and companies will it be possible to keep alive the dream of a free, accessible, and sustainable global encyclopedia.