Enterprises are moving fast on AI infrastructure—so fast that many can’t yet answer a basic question: what does the compute actually cost, and what return is it producing? A new VentureBeat Pulse Research survey of 107 enterprises (100+ employees) paints a picture of a “compute gap”: heavy, fast-moving investment in AI capacity running ahead of the measurement systems needed to steer unit economics. The result is a market where organizations are preparing to re-platform their infrastructure while still operating with low GPU utilization, incomplete cost tracking, and an emerging inference bottleneck that many barely recognize.
This isn’t a story about sticker prices or whether one vendor’s rate card looks cheaper than another’s. In fact, the survey suggests the opposite: when enterprises choose infrastructure, they prioritize integration and total cost of ownership (TCO), not headline token pricing. That would be reassuring—except the same respondents report that fewer than half can rigorously track compute costs and ROI. So the decisions are being made on economic criteria that many organizations can’t yet measure with confidence.
The maturity mismatch is the first clue. Only 21% of enterprises say they run AI in production at scale. Another 37% have some workloads in production, but not across the organization. A further 38% are still experimenting, running proofs of concept rather than operating at scale. And 4% aren’t running AI workloads at all. In other words, most organizations are still in the phase where compute footprints are about to grow quickly—yet their ability to instrument and manage those footprints is lagging behind.
That lag matters because the next wave of spending is not simply incremental. Enterprises aren’t just adding more of what they already use; they’re planning to evaluate infrastructure categories they barely touch today. The survey shows a sharp tension between current deployment stacks and near-term evaluation priorities.
Right now, the stack is familiar: hyperscalers and model-provider APIs. Google Cloud is the most-used platform overall, cited by 48% of respondents. Microsoft Azure follows at 29%, with AWS and Oracle Cloud both at 22%. On the model side, Gemini is used by 41% of enterprises, OpenAI by 40%, and Anthropic by 12%. Meanwhile, specialized “neocloud” GPU providers—CoreWeave, Lambda, Crusoe, Nebius, Together, Fireworks, and peers—register at near-zero levels among current usage, with each provider at less than 2% in this sample. Only 6% run their own on-prem or co-located GPU clusters, and 4% use a custom open-source self-managed stack.
This is important context for interpreting the next finding. If today’s deployments are mostly hyperscaler-and-API, then the planned evaluations reveal where the next dollars are likely to go—and why the compute gap could widen rather than close.
Over the next 12 months, the single largest planned evaluation area is AI-specialized clouds, cited by 45% of enterprises. These are the very categories that show up as almost absent in current usage. Nearly a third (32%) plan to evaluate non-NVIDIA accelerators such as AWS Trainium, Google TPU, AMD Instinct, Intel Gaudi, and in-house ASICs. Another 28% plan to evaluate next-generation Nvidia GPUs, including Blackwell (GB300). Decentralized or distributed compute networks are also on the radar for 16%, and sovereign or region-specific compute is cited by 11%.
Put plainly: enterprises are preparing to move meaningful portions of AI compute off the general-purpose cloud, even though their current stack is dominated by hyperscalers and major model APIs. This isn’t just a small optimization cycle. It reads like the leading edge of a re-platforming.
And the re-platforming intent is unusually aggressive. When asked whether and when they plan to switch or add an infrastructure provider, 64% of enterprises say they intend to do so within 12 months. Even more striking, 38% plan to change within the next quarter. For a category as foundational as compute, that level of churn intent is high—suggesting that many organizations see the current setup as temporary, not settled.
Yet the switching wave doesn’t appear to be driven by a simple desire to chase lower token prices. When enterprises were asked what matters most when selecting an AI infrastructure provider, integration with the existing cloud and data stack came out on top at 41%. Total cost of ownership (TCO) was second at 35%. Performance—latency and throughput—was cited by 24%. Security/compliance, autoscaling for spiky workloads, and GPU access/availability were each part of the mix at 19%. Cost per 1M tokens was the least-cited factor at 8%.
This ordering is rational. Token price is often a misleading proxy for real cost, especially when workloads vary widely in batch size, concurrency, caching behavior, and orchestration overhead. Integration and TCO are closer to how compute actually behaves inside an enterprise environment. But here’s the catch: the survey indicates that many enterprises can’t yet measure TCO rigorously.
Only 44% of respondents say they track compute cost and ROI rigorously. Another 39% track it only partially. A fifth (20%) can’t quantify it yet, and 6% say it isn’t a priority. That means more than half of enterprises either lack full visibility into compute economics or don’t have the instrumentation maturity to quantify them reliably.
This is where the compute gap becomes concrete rather than theoretical. If you can’t measure unit economics, you can’t confidently steer them. You can still make decisions based on assumptions, vendor claims, or partial metrics—but you’re operating with blind spots. And those blind spots become more dangerous when the compute footprint is expanding and the architecture is shifting.
One of the clearest signals of the gap is GPU utilization. The survey asked enterprises what share of their GPU capacity they actually utilize. The results show widespread underutilization. Among enterprises that operate GPUs, 83% report GPU utilization of 50% or less. More specifically, 37% run at 26–50% utilization, 34% run at 10–25%, and 15% run under 10%. Only 12% report utilization above 50%. There’s also an instrumentation gap: 8% say they don’t measure utilization at all, and an additional 7% consume via API and run no GPUs of their own.
Idle accelerators are expensive accelerators. Underutilization doesn’t just waste money; it also distorts the economics of future decisions. If your current fleet is sitting at half capacity or less, then the marginal cost of “more compute” may look different than it would under efficient scheduling and workload packing. It also suggests that some organizations may be buying capacity faster than they can operationalize it—turning procurement into a bottleneck rather than a solution.
The survey’s findings imply that many enterprises are in a loop: they buy compute, but they don’t fully instrument it; they run workloads, but utilization remains low; they plan to switch providers, but the economic ledger is incomplete. That combination makes it harder to validate whether a new provider will truly improve unit economics—or whether it will simply shift costs without improving control.
There’s another layer to this story: the next inference bottleneck is arriving, and many enterprises aren’t prepared for it. As inference scales, the constraint is shifting from raw GPU compute toward memory bandwidth and KV-cache capacity. In transformer-based inference, KV-cache grows with sequence length and affects how much state must be stored and accessed during generation. When memory becomes the limiting factor, the economics and architecture of inference change dramatically. You can’t treat inference as “just more GPU.”
Yet awareness is mixed. When asked which approach they would rely on as the binding constraint in inference as it shifts from compute to memory, responses were fragmented. Dell (PowerScale / Project Lightning) led at 31%, Nvidia (Dynamo / ICMSP) followed at 16%, and 18% said they are not aware of this constraint or haven’t addressed inference-memory limits yet. Another 10% cited Hammerspace (Tier Zero), with 9% citing DDN (Infinia). The remaining responses split across open-source KV-cache tooling, model-level efficiency techniques, VAST Data, WEKA, and other options.
This fragmentation is itself a signal. It suggests that the market for inference-memory solutions is still unsettled, and enterprises are still forming their mental models of what will matter most. But it also suggests that many organizations may be approaching inference scaling with a compute-first mindset, even though the cost drivers are shifting.
If the compute gap is widening, it’s partly because the industry is moving through multiple transitions at once: from experimentation to scaled production, from hyperscaler-only stacks to specialized compute, and from compute-bound inference to memory-bound inference. Each transition changes the economics. And each transition requires measurement discipline to manage.
So what does “compute gap” mean in practice? It’s not just that enterprises spend money quickly. It’s that they spend money faster than they can build the instrumentation and operational processes needed to understand what they bought, how it’s being used, and what it’s costing per outcome.
Consider the survey’s buying logic. Integration and TCO are the top decision factors. But if only 44% track compute cost and ROI rigorously, then many buyers are effectively outsourcing parts of their economic judgment to vendors, estimates, or incomplete internal reporting. That might still work in early pilots, but it becomes risky when workloads scale and architectures evolve.
Now add the underutilization pattern. With 83% reporting GPU utilization at 50% or less, many enterprises likely have scheduling inefficiencies, workload fragmentation, or operational constraints that prevent full utilization. Those issues can inflate effective cost per useful compute. They can also mask the true performance characteristics of a provider, because the system isn’t running at the conditions under which it would be evaluated.
Finally, add the inference-memory frontier. If enterprises aren’t aware