Why Bot Traffic is Both a Business Problem and a Security Problem
In one week last May, a botnet sent 12.34 million requests at Planetizen Courses. Newsletter subscribers clicked through to error pages. Most website owners treat traffic as a proxy for demand with more visitors meaning more interest and demand. But traffic is only a useful signal when it reflects real people. When a significant share of it is automated, the number on the dashboard stops telling you anything reliable about your actual audience.
That's the situation we encountered with Planetizen, the leading urban planning news, media, jobs and education platform that publishes more than 40 articles a week and has over 2 million unique visitors a year, and 100,000+ content pages. Over time, automated bot traffic grew to the point where it caused two distinct problems: corrupting their analytics data, and degrading site performance badly enough to affect real users. Traffic for the year was up 30% and no one questioned it.
Two Problems, One Source
How Bot Traffic Distorts Google Analytics Data
Not all bots interact with a site the same way - some automated traffic executes JavaScript the way a real browser does, which means it can trigger the tracking code that reports to platforms like Google Analytics and show up as sessions that look human. Other bots never touch the analytics layer at all. They hit the server directly, consume resources, and leave without ever running the tracking script, so they stay invisible in your reports even as they drive up load. Google Analytics does filter some of this automatically, but only bots that identify themselves and appear on a known-bot list. The traffic that causes real problems is the traffic that attempts to present itself as an ordinary browser, and then is counted as a session like any other.
The result is that "bot traffic" isn't one number you can look up in one place. It shows up differently at the content delivery network (CDN), the web server, and the analytics platform, and each layer counts it differently. The traffic that does reach the analytics layer inflates session counts and distorts engagement metrics, so audience data that should inform editorial strategy, product decisions, digital investments, and marketing spend starts reflecting activity that isn't coming from real people. And because the distortion is gradual, it often goes unnoticed for months.
A deep, well-organized, publicly accessible archive is exactly what automated scrapers are built to harvest, and Planetizen has 26 years of one. The same qualities that make the site valuable to readers make it an efficient target.
How Bot Traffic Slows Sites and Causes Outages
At high enough volume, bot traffic consumes server resources that should be serving real users, causing pages to slow down. In serious cases, when automated requests hit a site faster than it can handle, real users start seeing errors or can't load pages at all. Planetizen saw this firsthand: at one point a distributed botnet drove Cloudflare-mitigated traffic to the courses site to more than 10 million requests in a single week, overwhelming the origin server and causing gateway timeouts across the site. When that happened, it became a business problem that impacted user experience (and likelihood of someone returning to the site), and the bottom line via missed traffic and missed conversions. The timing made it worse - newsletter subscribers were clicking through to error pages. Class enrollments and job listing renewals run through the same site as the free editorial content, so the outage was both a reader experience problem and a revenue problem.
Additionally, the damage carried into paid acquisition. Planetizen imports GA4 conversion events into Google Ads, where they feed automated bidding. Once bot sessions entered GA4, the signals Google Ads was bidding against no longer described real human behavior. Customer acquisition cost doubled and return on ad spend fell sharply despite the campaigns not having changed. It also negatively impacted remarketing and audience lists.
Ultimately, these issues are becoming ever more prevalent with the rise in AI bots and scrapers - according to HUMAN Security, a leader in bot detection and mitigation, 2025 saw a 7,851%year-over-year growth in agentic AI traffic.
How We Diagnosed and Mitigated the Bot Traffic
The instinct when facing bot traffic is to block aggressively. But a blunt approach creates its own problems. Block too broadly, and you start turning away legitimate users: professionals using VPNs, subscribers clicking through from email newsletters, and visitors whose traffic patterns look unusual but who are real humans.
The work starts with understanding the traffic. Before touching any controls, we analyzed request patterns to identify anomalous sources and behaviors, separated known legitimate crawlers from suspicious automation, and mapped where the problem traffic was actually coming from. Only then did it make sense to introduce mitigation, and to do it in graduated steps rather than all at once. Not all bots are bad and the goal is not to block all automated traffic as plenty of it is necessary such as search engine crawlers like Googlebot and Bingbot which need access to index the site and keep it visible in search results. Uptime monitors and legitimate preview bots from social and messaging platforms all serve a purpose. The job isn't to shut the door on automation; it's to tell the difference between the bots and AI scrapers that should be there and the ones that shouldn't, and to treat each accordingly. Some are welcomed through, some are challenged, and only the clearly harmful ones are blocked outright.
We weren't solving one problem with one tool: we were addressing two different problems with two complementary mechanisms. The first, bot mitigation, cuts down the unwanted automated traffic. The second, edge caching, means keeping ready-made copies of the site's pages on Cloudflare's network of servers around the world, so that most visitors are served a saved copy instead of the request going all the way back to the main server each time. That keeps the load off the origin even for perfectly legitimate visitors.
On the mitigation side, we used Cloudflare, which sits between the internet and the website and inspects incoming traffic before it ever reaches the server. The framework works in layers:
- Confirmed bad actors are blocked outright.
- Suspicious but unconfirmed traffic receives a Cloudflare Managed Challenge, a lightweight browser challenge designed to distinguish automated requests from legitimate visitors while minimizing friction for real users.
- Verified search engine crawlers like Googlebot are explicitly allowed through so legitimate crawling isn't caught in the rules.
- Known legitimate sources like email newsletter platforms are exempted so subscriber click-throughs land cleanly.
The other critical part to understand about Cloudflare is setting up the rules correctly and understanding who likely represents a real user for you, e.g., for Planetizen we tuned rules around the geographies that don't match Planetizen's actual readership or purchasers
On the caching side, we addressed a separate issue. Even with bad traffic filtered out, a large share of anonymous page views were still being rendered fresh by the origin server rather than served from cache, which meant legitimate visitors and crawlers alike were adding avoidable load. We reconfigured how the site serves pages to anonymous visitors so that a large share of those requests are served from Cloudflare's global edge network instead of the origin. Bot mitigation cut the unwanted requests; edge caching cut how many of the remaining legitimate requests had to reach the origin at all.
Now, we regularly monitor the system and update it as new threats emerge, so the rules keep pace with the constantly evolving bot threats and traffic rather than going stale and leading to more corrupted data in the future.
Results: Clean Analytics and Near-Zero Timeouts
Analytics data that had been distorted by automated traffic became more reliable, giving the Planetizen team a more accurate picture of how real users actually interact with the site. Decisions about content strategy and audience development could be based on data that reflected real behavior rather than automated noise. Paid acquisition recovered alongside the data: within two months of the mitigation rules going live and GA4 reporting clean sessions again, CAC and ROAS on Planetizen's Google Ads campaigns returned to their pre-incident baselines.
Beyond ad optimization, the performance gains were also noticeable. Over the seven days of May 15 to 21, the site logged more than 532,000 gateway timeout errors (these included both bots and real users attempting to reach pages/articles but getting errors instead). After the edge caching change went live, that number dropped to near zero, and origin server load fell sharply alongside it.
The goal, to ensure that legitimate traffic was never the casualty of the crackdown, was successful. Verified Googlebot continued passing through the mitigation rules with minimal disruption, helping ensure that legitimate search crawling wasn't blocked, and real visitors kept moving through the site normally. The user experience wasn't impacted because very few real users had to prove their humanity by clicking on pictures of traffic lights or an "I'm human" checkbox, which is what a heavier-handed approach would have required.
Signs Your Site Has a Bot Traffic Problem
Bot traffic isn't a problem unique to large media platforms. Any organization with a public-facing website, especially one that publishes valuable content, runs a membership, or manages an online store, is a potential target. The signs aren't always obvious. Watch for:
- Unexplained traffic spikes that don't correspond to campaigns or news events
- Analytics data that doesn't match what you'd expect from your audience
- Performance issues that don't have a clear infrastructure explanation
- Server costs that seem higher than your traffic would justify
Treating this as purely a security issue misses half the problem: bot traffic corrupts the data you use to make business decisions and degrades the experience for the people you're trying to reach.
Want to check yourself? Pull up your CDN or server logs and your analytics platform side by side for the same seven-day window. If the request counts are wildly different, and they usually are, that gap is the automated traffic, and it is the reason this problem goes unnoticed for months at a time.
Urban Insight offers a bot traffic and performance audit for organizations that suspect their numbers are not telling the truth. We compare traffic across your CDN, web server, and analytics platform, separate the automated from the real, review your caching configuration and origin load, and hand you a prioritized set of recommendations with the tradeoffs spelled out. You get the written report either way, whether or not you bring us in to implement the changes.
If you are on Cloudflare and your site carries advertising, there is added urgency. Cloudflare has introduced new defaults that block AI training and agent crawlers on ad-supported pages for newly onboarded domains, while still allowing search crawlers. Major search bots like Googlebot are multi-purpose, so a site that chooses to block training crawlers can end up blocking Googlebot too, since Cloudflare applies the most restrictive matching rule. Any site on Cloudflare can review and adjust these settings, and it's worth checking your configuration now rather than after your search traffic drops.
