The problem: AI bots don't browse like humans
Cloudflare has published a detailed look at how AI-bot traffic is reshaping the demands on its CDN cache infrastructure. The numbers are significant. AI bots now generate over 10 billion requests per week across Cloudflare's network. And the way those bots interact with cached content is fundamentally different from how humans browse the web.
Human visitors tend to follow predictable patterns. They land on a homepage, click through a few pages, and leave. CDN caches have been optimised for exactly this behaviour for decades. Popular pages stay warm in cache. Less popular pages expire and get fetched from the origin server when needed.
AI crawlers behave differently. They tend to systematically traverse entire sites, requesting pages that humans rarely visit. They often ignore conventional caching signals. They request content in bulk, at speed, and without the natural pauses that human browsing introduces. The result is a different load profile that existing cache strategies were never designed to handle.
Why this matters for businesses
If you run a website or web application, this shift affects you in a few concrete ways.
- Origin load is increasing. AI bots requesting deep, rarely cached pages means more requests hitting your origin servers directly. That drives up compute costs and can degrade performance for real users.
- Cache hit ratios are dropping. Traditional cache strategies assume a long tail of infrequently accessed content. When bots systematically crawl that long tail, the cache becomes less effective at shielding your infrastructure.
- Bandwidth costs are climbing. High-volume, deep crawling generates significant egress traffic. For organisations running on cloud infrastructure, that translates directly to higher bills.
- User experience is at risk. If bot traffic is competing with human traffic for the same origin resources, response times for real customers can suffer.
Cloudflare's response is to rethink how cache behaves when the requestor is an AI bot versus a human visitor. The goal is to serve both audiences well, without one degrading the experience of the other.
What organisations should consider
Cloudflare's architectural changes will help at the CDN layer, but this is a broader issue that every organisation with a web presence should be thinking about.
- Understand your bot traffic. Review your access logs and analytics. Know what proportion of your traffic is coming from AI crawlers, and which pages they are hitting. You cannot manage what you do not measure.
- Review your caching strategy. If your cache configuration was last tuned for human browsing patterns, it may need revisiting. Consider how deep-crawl traffic interacts with your cache TTLs and eviction policies.
- Set clear bot policies. Decide whether you want AI bots crawling your content, and on what terms. Tools like robots.txt, rate limiting, and Cloudflare's own bot management features give you control. Use them deliberately.
- Monitor origin server health. Watch for unexplained increases in origin load or response times. These can be early indicators that bot traffic is bypassing your cache layer.
- Factor AI traffic into capacity planning. If your infrastructure was sized for human traffic volumes, the addition of aggressive AI crawling may push you past your headroom sooner than expected.
Metaphor's perspective
This is a good example of how AI is changing infrastructure requirements in ways that are easy to overlook. Most organisations are focused on how they can use AI. Fewer are thinking about how AI is using them.
The traffic patterns Cloudflare describes are real, and they are growing. We see this across our client base. Organisations that proactively review their caching, bot management, and origin protection strategies will be better positioned than those who wait until performance degrades or costs spike.
If you are unsure how AI-bot traffic is affecting your infrastructure, or whether your current CDN and caching setup is fit for purpose, that is a conversation worth having now rather than later.
Sources: Why we're rethinking cache for the AI era