In recent weeks, openDemocracy’s website has been repeatedly knocked offline, as if under enemy attack. The partial cause is a gargantuan barrage of automated bots from AI companies.
As people increasingly turn to chatbots and other large language models (LLMs) rather than searching or visiting sites directly, AI companies are using large-scale scrapers to gather the data to power their services.
Many of these scrapers are so sophisticated that it is hard, or impossible, to detect them in action. They often ignore the websites’ programmatic pleas not to be scraped, and are known to hit the more fragile parts of a website repeatedly.
For media sites, the AI pincer movement decimates traffic, while undermining our ability to serve the traffic we do get with these mammoth crawling operations.
Grant Slater is a core developer at OpenStreetMap.org, an online map tool co-created by mappers all over the world, who contribute and maintain data about roads, paths, cafés, stations, and more. He told openDemocracy he has seen a “dramatic shift” in the rise of scrapers over the past two years – to the point where OpenStreetMap is now “at constant war” with them.
“We assume they are scraping data to fuel AI LLMs or start-ups set up to supply the AI companies with training data. There's an unspoken arms race happening across the web: public-interest projects are being hit by industrial-scale AI scraping,” he said.
Slater explained that “the traffic often arrives through anonymous residential IPs”, referring to residential proxy networks that route internet traffic through intermediary servers using IP addresses assigned by internet service providers to real homeowners. This, he said, makes it “hard to distinguish ‘normal users’ from automated collection”.
“We're being forced into permanent defence mode. Residential proxy networks let AI scrapers hide in plain sight, rotate identities, and extract data at scale. That shifts real costs onto projects that exist to serve people, not feed training pipelines.
“Scrapers can rotate addresses endlessly to harvest as much data as possible for LLM training, while we have finite bandwidth, compute, and volunteer time. The result is a war of attrition we didn't choose – where keeping the site usable becomes a constant battle.”
The resources required to keep these marauding invaders at bay are another concern in the growing list of reasons to worry about AI. Websites may soon be forced to require hard proof of humanity from all visitors, such as insisting on users being signed in.
Web traffic dashboard by Cloudflare showing January 2026 bot activity.
“The actors extracting the most value from large-scale crawling are largely insulated from the costs it creates, which are absorbed by publishers. They must respond by restricting access in order to survive,” writes researcher Audrey Hingle in her paper Getting bots to respect boundaries.
“Over time,” Hingle continues, “this risks accelerating enclosure: more gated content, more limited access, and a web that is harder to participate in.”
Mind your information diet
The advance of AI is having an impact on how people stay informed, with people prompting chatbots for news stories and other general inquiries.
AI agents, in their various forms, are here to stay and will make some elements of news provision more personalised and insightful. But the big firms leading the charge are animated by commercial interests, with ChatGPT now launching ads in its products.
The toxicity of social media, where financial imperatives drive design, is a worrying template for what’s to come with AI. Hallucinations, deception, propaganda and sycophancy, as chatbots are known for, will also muddy the waters.
As trust online becomes an increasingly fragile quantity, web users must be savvy to ensure they are getting accurate information from honest sources, scrutinising where news comes from and consolidating direct links with reputable organisations, such as with email newsletters – like openDemocracy’s, which are available here.
Just as important as minding how chatbots curate your info diet is what you allow to be taken from you. The AI age continues the era of surveillance capitalism: your data is still the product. Now they know more about you than ever given the depth of interactions with AI bots.
With this in mind, Tony Curzon Price, erstwhile editor-in-chief of openDemocracy, launched the First International Data Union to let people use LLMs without operators like OpenAI hoarding valuable data from those chats.
“Our data, used both for our personal benefit and for collective benefit, is the one chokepoint that ordinary citizens have over platform and BigAI power. I set up FIDU to be the trusted steward of members’ interests in cyberspace,” Curzon Price explained. “The rapid AI-fication of the web makes it all the more urgent that we create meaningful counterweights to planetary-scale profit-driven data extraction we're seeing by the AI labs.”
Let's hope that innovations like this give the public more control of AI in their lives.
Oversight and pushback is essential
We can recognise the benefits that AI systems may bring and still feel that negative effects may outweigh the positive.
The competition to win the AI race is already producing multiple harmful outcomes. From the expansion of environmentally destructive data centres, to AI’s use for paedophilia or other sexual abuse, to twisted ‘therapy bots’ and the potential replacement of humans in society at large, the effects are startling and unravelling quickly.
Even industry leaders who do care about the outcomes are caught in this dynamic to produce superintelligence, which is also a bellicose national security contest as much as it is a commercial one. There is not even a clear endpoint to the struggle, though some have imagined where we may end up.
In the meantime, active citizens must drive political pressure to ensure the industry doesn’t overpollute, or create the next financial collapse, or poison the well of public information.
For now, stay independent and maintain connections with media you trust, away from corporate platforms of dubious integrity.
Follow openDemocracy directly on the open web
openDemocracy Email newsletters
openDemocracy Mastodon
openDemocracy Bluesky
