Security network Cloudfare expanded access to its Bot Management and AI Crawl Control services to protect journalists from AI crawlers.
The services are now accessible through Cloudfare’s Project Galileo programme–an internet security platform used by around 750 journalists and independent news organisations.
The aim of Project Galileo, which was launched in 2014, was to protect groups such as non-profit organisations and human rights defenders from cyber attacks. It now includes AI crawler protection tools to address modern threats posed to these groups.
The Bot Management and AI Crawl Control services are available to journalists for free.
Keeping Out the Crawlers
Bot Management analyses web traffic and uses machine learning to identify which bots are “good” – such as search engine crawlers – and which are bad – such as those that attack websites and steal information.
Through Galileo, users can control which bots are permitted access to their site.
AI Crawl Control identifies crawlers and allows website owners to segment them by origin and activity. For example, crawlers can be distinguished as AI search, or AI scraper. Journalists can also see where the crawlers come from, whether it be Google, OpenAI, Anthropic, or elsewhere.
The company explained how the Crawl Control allows journalists to manage the traffic to their websites – blocking AI scrapers while maintaining a direct connection with readers, including those who arrive via clicks made within AI platforms.
Source: Cloudfare
Journalism in the Age of AI
Increased use of AI platforms for information has severely impacted traffic to all websites sites, but news platforms have been hit particularly hard. Recent data from Cloudfare showed that now, with the addition of features like AI Overviews, it is nearly 10 times more difficult for a content creator to get the same traffic they could 10 years ago.
This research included that, when users are using OpenAI’s services, it is 750 times more difficult for sites to get traffic than it was before through channels like Google. With Anthropic, it becomes 30,000 times more difficult.
Google’s AI Overviews, a now automatic feature with most search queries, give users a summarised response to their search. The summary pulls from various sources, and gives the user a digestible response without the need to click onto a link in the search results.
As less users click onto sites and read the content directly, publishers lose both ad revenue and the opportunity for a website visitor to subscribe.
Commenting on Cloudflare’s announcement, Meera Selva, CEO, Internews Europe, said, “In an era defined by AI and digital disruption, providing robust tools to independent media isn’t just support – it’s a lifeline.”
