It’s not your model — AI has a data movement problem
Network World ·
For the past two years, the checkbook has been wide open as enterprises have poured billions into AI models, compute, licenses and GPUs. Yet despite those massive investments, a growing number of organizations are running into the same problem: they can’t reliably access what they just built. When those expensive AI workflows collapse, it’s easy for leaders to assume the models aren’t working. However, the blame often lies elsewhere: leaders consistently underestimate the connectivity requirements beneath their AI strategy. According to recent research , 59% of organizations now report bandwidth issues, up from 43% just a year ago. Latency challenges have nearly doubled, surging from 32% to 53%. Enterprises are paying for that gap in performance, in user experience and often in unnecessary spend aimed at the wrong part of the stack. Addressing a data movement problem (not a compute problem) An AI conversation from two years ago and an AI conversation today would be practically unrecognizable from each other. We’ve moved from “is this technology real?” to “we’re fully committed, now make it work.” For most of that shift, the dialogue has understandably centered on the structures of the models — but a model is only as effective as the infrastructure connecting it to users, applications, agents and the other models on which it depends. Enterprise AI is increasingly bogged down by data movement problems. Traditional enterprise networks were built around a predictable human transaction: someone searches for something, the system fetches it, the transaction ends. AI breaks that pattern entirely. One statistic still makes my jaw drop every time I say it out loud: a typical AI agent can generate up to 450% more total network traffic per task than a human performing the equivalent work . That’s not just more traffic. It’s a different kind of traffic: distributed, bursty, concurrent and generated at machine speed rather than human speed. Which is why a successful AI pilot might prove your model works but doesn’t prove your enterprise can run it. I’ve even watched this play out live. At Cisco Live earlier this year, engineers were already measuring this data movement shift in real time on their own live network projections. The engineers were watching agentic workloads generate traffic spikes that looked nothing like the human traffic patterns their infrastructure was designed around. It wasn’t a hypothetical for them. It was showing up on the screen in front of us. This same pattern plays out every day across large enterprises rolling out AI strategy. In the initial environment, it performs exactly as expected. Users are sitting in close proximity to the cloud region, bandwidth is ample and usage is still modest because the organization is not yet fully using the tool. Everyone declares success. Then it goes global. Employees in Latin America, parts of Asia or Africa suddenly see dramatically slower responses. The instinctive conclusion for most leaders is that the AI platform itself is underperforming, so the logical response is to spend more. This means more compute, more licenses and more CapEx aimed at expanding the platform. But that can be the wrong lever. When you examine the full end-to-end transaction, the model usually isn’t the problem. Take the extraordinary scale of AI traffic as it moves from a branch office to a centralized security stack, then to a cloud region, then to a data-hosted backend somewhere — and then add inconsistent local internet performance, packet loss and congestion — and the AI experience deteriorates. The challenge also varies widely depending on where in the world you’re standing. At an AI symposium in Africa recently, the attendees might have been expected to discuss which model to use, but the conversation instead was “how do we actually deploy this.” In Latin America, the discussion I hear most often from customers isn’t about raw bandwidth, but latency, specifically, because of how the region’s infrastructure routes traffic back to major cloud regions. Silicon Valley and São Paulo are not the same infrastructure conversation, and treating them as though they were is exactly how well-intentioned rollouts stall. This is not because the model got worse, but because everything underneath it was never built to carry this kind of load across this kind of distance. When an AI application performs poorly, our instinct is to look up the stack, but the answer is often underneath it. Why a bigger pipe doesn’t fix the problem Bandwidth and latency may get treated as the same problem, but they are not. Bandwidth is volume, and AI increases it because organizations are moving richer data, like documents, images, video and telemetry. Latency is time, and it’s where the harder problem lives. AI workflows are chained: an agent calls a model, retrieves enterprise data, invokes an API to another system and that system may call another model in turn. The cumulative delay across that chain isn’t predictable, because you don’t know in advance what’s being asked or how far it must travel. That’s why simply buying a bigger circuit doesn’t solve the problem. Compute or GPU availability are not typically the bottleneck. What matters are congestion routing, packet loss, jitter and where security inspection happens along the path. Even hyperscaler infrastructure doesn’t eliminate this. The system’s speed depends on whether a request can traverse the full chain, from user to network to security to execution and back, without the delay compounding at every hop. When I talk to customers who are certain Azure or Google Cloud must be the bottleneck, the infrastructure on their end is rarely the issue. If anything, the infrastructure is usually excellent, because that’s what it’s built to deliver. The real question is whether a request can survive the full round trip: from user, through the network, through security inspection, out to that infrastructure and back again, without losing time at every handoff along the way. As compute continues to improve, the path to accessing it is becoming an increasingly important part of the performance equation. Questions before you spend another dollar There’s no one-size-fits-all fix, but there are questions every leader should be able to answer before scaling an AI deployment: Where do your AI users, models and data sit — locally, regionally or centrally? What’s the data sovereignty picture, and can traffic legally and technically move across the regions your rollout depends on? Are you assessing the actual end-to-end network conditions experienced by AI users — including latency, jitter, packet loss and routing — or relying on average network metrics that can conceal problems along specific routes or in particular locations? Do the team running your AI strategy and the team running your network infrastructure talk to each other, or are they operating in separate silos? And critically: what does performance look like in your hardest geography, not your best one? Until you can answer those questions, you will struggle to scale, regardless of how well the pilot went. Enterprises risk spending billions on a Formula 1 engine and then trying to race it on a road never designed for that speed. They won’t get the performance they paid for, not because the engine is wrong, but because the infrastructure underneath it cannot support what they are asking it to do. AI strategy and network strategy can no longer be separate conversations. They must be elements of the same conversation, from the very beginning.
For the past two years, the checkbook has been wide open as enterprises have poured billions into AI models, compute, licenses and GPUs. Yet despite those massive investments, a growing number of organizations are running into the same problem: they can’t reliably access what they just built. When those expensive AI workflows collapse, it’s easy for leaders to assume the models aren’t working. However, the blame often lies elsewhere: leaders consistently underestimate the connectivity requirements beneath their AI strategy. According to recent research , 59% of organizations now report bandwidth issues, up from 43% just a year ago. Latency challenges have nearly doubled, surging from 32% to 53%. Enterprises are paying for that gap in performance, in user experience and often in unnecessary spend aimed at the wrong part of the stack. Addressing a data movement problem (not a compute problem) An AI conversation from two years ago and an AI conversation today would be practically unrecognizable from each other. We’ve moved from “is this technology real?” to “we’re fully committed, now make it work.” For most of that shift, the dialogue has understandably centered on the structures of the models — but a model is only as effective as the infrastructure connecting it to users, applications, agents and the other models on which it depends. Enterprise AI is increasingly bogged down by data movement problems. Traditional enterprise networks were built around a predictable human transaction: someone searches for something, the system fetches it, the transaction ends. AI breaks that pattern entirely. One statistic still makes my jaw drop every time I say it out loud: a typical AI agent can generate up to 450% more total network traffic per task than a human performing the equivalent work . That’s not just more traffic. It’s a different kind of traffic: distributed, bursty, concurrent and generated at machine speed rather than human speed. Which is why a successful AI pilot might prove your model works but doesn’t prove your enterprise can run it. I’ve even watched this play out live. At Cisco Live earlier this year, engineers were already measuring this data movement shift in real time on their own live network projections. The engineers were watching agentic workloads generate traffic spikes that looked nothing like the human traffic patterns their infrastructure was designed around. It wasn’t a hypothetical for them. It was showing up on the screen in front of us. This same pattern plays out every day across large enterprises rolling out AI strategy. In the initial environment, it performs exactly as expected. Users are sitting in close proximity to the cloud region, bandwidth is ample and usage is still modest because the organization is not yet fully using the tool. Everyone declares success. Then it goes global. Employees in Latin America, parts of Asia or Africa suddenly see dramatically slower responses. The instinctive conclusion for most leaders is that the AI platform itself is underperforming, so the logical response is to spend more. This means more compute, more licenses and more CapEx aimed at expanding the platform. But that can be the wrong lever. When you examine the full end-to-end transaction, the model usually isn’t the problem. Take the extraordinary scale of AI traffic as it moves from a branch office to a centralized security stack, then to a cloud region, then to a data-hosted backend somewhere — and then add inconsistent local internet performance, packet loss and congestion — and the AI experience deteriorates. The challenge also varies widely depending on where in the world you’re standing. At an AI symposium in Africa recently, the attendees might have been expected to discuss which model to use, but the conversation instead was “how do we actually deploy this.” In Latin America, the discussion I hear most often from customers isn’t about raw bandwidth, but latency, specifically, because of how the region’s infrastructure routes traffic back to major cloud regions. Silicon Valley and São Paulo are not the same infrastructure conversation, and treating them as though they were is exactly how well-intentioned rollouts stall. This is not because the model got worse, but because everything underneath it was never built to carry this kind of load across this kind of distance. When an AI application performs poorly, our instinct is to look up the stack, but the answer is often underneath it. Why a bigger pipe doesn’t fix the problem Bandwidth and latency may get treated as the same problem, but they are not. Bandwidth is volume, and AI increases it because organizations are moving richer data, like documents, images, video and telemetry. Latency is time, and it’s where the harder problem lives. AI workflows are chained: an agent calls a model, retrieves enterprise data, invokes an API to another system and that system may call another model in turn. The cumulative delay across that chain isn’t predictable, because you don’t know in advance what’s being asked or how far it must travel. That’s why simply buying a bigger circuit doesn’t solve the problem. Compute or GPU availability are not typically the bottleneck. What matters are congestion routing, packet loss, jitter and where security inspection happens along the path. Even hyperscaler infrastructure doesn’t eliminate this. The system’s speed depends on whether a request can traverse the full chain, from user to network to security to execution and back, without the delay compounding at every hop. When I talk to customers who are certain Azure or Google Cloud must be the bottleneck, the infrastructure on their end is rarely the issue. If anything, the infrastructure is usually excellent, because that’s what it’s built to deliver. The real question is whether a request can survive the full round trip: from user, through the network, through security inspection, out to that infrastructure and back again, without losing time at every handoff along the way. As compute continues to improve, the path to accessing it is becoming an increasingly important part of the performance equation. Questions before you spend another dollar There’s no one-size-fits-all fix, but there are questions every leader should be able to answer before scaling an AI deployment: Where do your AI users, models and data sit — locally, regionally or centrally? What’s the data sovereignty picture, and can traffic legally and technically move across the regions your rollout depends on? Are you assessing the actual end-to-end network conditions experienced by AI users — including latency, jitter, packet loss and routing — or relying on average network metrics that can conceal problems along specific routes or in particular locations? Do the team running your AI strategy and the team running your network infrastructure talk to each other, or are they operating in separate silos? And critically: what does performance look like in your hardest geography, not your best one? Until you can answer those questions, you will struggle to scale, regardless of how well the pilot went. Enterprises risk spending billions on a Formula 1 engine and then trying to race it on a road never designed for that speed. They won’t get the performance they paid for, not because the engine is wrong, but because the infrastructure underneath it cannot support what they are asking it to do. AI strategy and network strategy can no longer be separate conversations. They must be elements of the same conversation, from the very beginning.