The AI compute gap: Enterprise... Note
VentureBeat

The AI compute gap: Enterprises are buying infrastructure faster than they can measure what it costs

AI infrastructure spending is rapidly increasing, outpacing organizations' ability to understand and manage its economic implications. Currently, most AI workloads run on established hyperscalers and model provider APIs. However, a significant future investment is directed towards specialized compute, a sector most enterprises are not yet utilizing but plan to explore within the year. Procurement decisions prioritize integration with existing systems and overall cost of ownership over headline token prices. This is problematic as most companies lack clear unit economics and report low GPU utilization rates.The research highlights a "compute gap," defined by aggressive investment in AI infrastructure without sufficient visibility into its costs. While only about one-fifth of organizations are running AI at scale, their spending intentions are growing rapidly, with a strong focus on AI-specialized clouds. Existing compute resources are underutilized, with 83% reporting 50% or less GPU utilization. Furthermore, less than half of enterprises can accurately track their AI compute costs.Enterprises are also not settled on their current infrastructure vendors, with a majority planning to switch or add providers within twelve months. When selecting new vendors, integration and total cost of ownership are primary drivers, not per-token pricing. A significant portion of enterprises are unaware of or have not addressed the emerging constraint of memory bandwidth scaling in inference. The current AI infrastructure landscape is characterized by substantial investment growth alongside a lack of economic transparency and underutilized existing resources. This dynamic suggests a period of significant vendor evaluation and potential re-platforming in the near future.
CdXz5zHNQW_rcHYwGo3x2.png