Automated traffic now makes up more than half of internet activity by several industry measures. The bigger challenge for website owners is no longer simply spotting a bot. It is deciding which automation is useful, which is harmful, and where to stop it before it affects real users.
If it feels as though you are being asked to prove you are human more often online, you are probably noticing one symptom of a much bigger shift.
Bots have always been part of the web. Search engines crawl pages, monitoring tools check whether sites are online, and software services make automated requests every second. What has changed is the scale, variety and behaviour of that traffic.
Across websites managed by our teams in the UK and US, automated traffic is becoming harder to treat as background noise. We are seeing heavier crawling, more requests hitting resource-intensive areas of sites, and patterns that are much harder to contain with simple IP blocking.
The industry data points in the same direction.
50%+
of internet traffic described as non-human by Cloudflare in 2026
300%+
growth in AI bot traffic reported across Akamai’s network
53%
of internet traffic made up of bots in Thales’ 2026 report
Want to see how that balance looks right now? Cloudflare Radar provides a live view of bot versus human web traffic worldwide here.
The bot problem has changed
The important question is no longer simply whether a visitor is automated. It is what that automation is trying to do.
Googlebot crawling your content is very different from a scraper repeatedly requesting thousands of expensive URLs. AI adds more categories again: training crawlers, search and retrieval bots, user-directed agents, and services that mix several purposes.
That means ‘block the bots’ is no longer a sensible strategy. Some automated traffic helps people find and use your website. Some is harmless but resource-hungry. Some is actively malicious. Website owners increasingly need to tell those behaviours apart.
Cloudflare’s June 2026 data makes the change particularly visible: 52% of crawler requests were associated with AI training, up from 22% in spring 2025.
What this looks like on a real website
On one large B2B website we manage, automated traffic had been building for months. The problem eventually concentrated around a content hub where visitors could filter content using URL parameters.
For a human, those filters were useful. For automated traffic, they created a huge number of possible URL combinations. Bots began requesting them at volume, repeatedly generating uncached, resource-intensive work for the site.
The traffic was also distributed. Individual IP addresses often appeared only briefly before another took their place, which made conventional IP-by-IP blocking increasingly ineffective.
What worked
We introduced targeted application-level protection around the affected area, returning a lightweight response when request volumes became excessive. That helped stabilise the site and reduced the amount of work the application had to perform.
What it taught us
Application-level protection can be highly effective, but the request has still reached your infrastructure. For persistent bot pressure, the next question is whether some of that decision-making should happen earlier.
Protect the right layer, not just the website
This is where edge protection becomes important. A reverse proxy or web application firewall can inspect traffic before it is forwarded to the origin server. Cloudflare is one example: when a site’s DNS records are proxied through its network, requests pass through Cloudflare before reaching the origin.
That creates opportunities to rate limit, challenge or block specific patterns closer to the edge, rather than asking the application itself to deal with every request. Modern rate limiting can also be scoped around characteristics such as URL paths and other request properties instead of relying on IP address alone.
The point is not that every website suddenly needs Cloudflare, or that an edge service replaces good hosting and application design. The point is that bot resilience works best in layers. If one part of a site is attracting expensive automated traffic, the most effective control may sit outside the application.
But blocking everything creates a different problem
For teams responsible for both websites and search visibility, this is the part that matters most.
Search crawlers need to reach content. SEO platforms and uptime tools make automated requests. AI search and retrieval services increasingly rely on access to the open web. A blanket challenge or block can protect infrastructure while accidentally reducing discoverability, breaking monitoring or interfering with legitimate integrations.
Good bot management is therefore not about blocking more traffic. It is about making better decisions about automated traffic.
What website owners should do in 2026
- Measure before you block. Use server logs, hosting analytics, CDN data and monitoring to understand where automated requests are actually going.
- Find the expensive surfaces. Search, filters, APIs, login areas and parameter-heavy URLs can behave very differently under automated load.
- Use layered controls. Caching, application safeguards, WAF rules, rate limiting and managed challenges each solve different parts of the problem.
- Keep reviewing the rules. Bot behaviour changes quickly. A rule that works today can become ineffective, or unnecessarily restrictive, later.
The web now has two audiences
The web is not going to become less automated. Search engines, AI systems, software agents, monitoring services and malicious bots are all competing for access to the same infrastructure.
For website owners, that makes bot management part of performance and architecture, not just security. The goal is not a bot-free website. It is a website that stays fast and available for real users while giving useful machines the access they need.
At Noble Performance, we increasingly look at bot behaviour alongside hosting, website performance, SEO and technical architecture. If unexplained traffic or server load is affecting your site, understanding what is making those requests, and how far those requests are getting, is a good place to start.
Because in 2026, building a website for humans is only half the story.
